Skip to contentSkip to conversation

gitsm:// question

DISCUSSION Böszörményi Zoltán 3 participants 8 messages Oct 18, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi,

I have a working python3-pytorch recipe which originally used this:

=======================================================
addtask do_git_submodules after do_unpack before do_patch

do_git_submodules[network] = "1"

do_git_submodules () {
        cd ${S}
        git submodule update --init --recursive
}
=======================================================

I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing:

SRC_URI = " \
git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 \
...
gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm \
gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit \
gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo \
gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass \
gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest \
gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch \
...
"

As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.

The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.

I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://

Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?

As an easy way out, is there a flag to git:// to process its submodules automatically?

Thanks in advance,
Zoltán Böszörményi

#1 lore raw

Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.

Quoted 76 lines · Zoltan Boszormenyi

Hi,

I have a working python3-pytorch recipe which originally used this:

=======================================================
addtask do_git_submodules after do_unpack before do_patch

do_git_submodules[network] = "1"

do_git_submodules () {
         cd ${S}
         git submodule update --init --recursive
}

=======================================================

I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing:

SRC_URI = " \
git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 \
...
gitsm://
github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
\
gitsm:// github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
\
gitsm:// github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo \
gitsm:// github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
\
gitsm:// github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest \
gitsm:// github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch \
...
"

As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.

The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.

I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://

Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?

As an easy way out, is there a flag to git:// to process its submodules automatically?

Thanks in advance,
Zoltán Böszörményi

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#189388):
https://lists.openembedded.org/g/openembedded-core/message/189388
Mute This Topic: https://lists.openembedded.org/mt/102038382/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

Thanks, I will try it.

2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:

Quoted 87 lines

Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.

On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:

    Hi,

    I have a working python3-pytorch recipe which originally used this:

    =======================================================
    addtask do_git_submodules after do_unpack before do_patch

    do_git_submodules[network] = "1"

    do_git_submodules () {
             cd ${S}
             git submodule update --init --recursive
    }
    =======================================================

    I would like to replace the above with a series of gitsm:// SRC_URI lines.
    Here's a subset of submodules that shows the problem I am seeing:

    SRC_URI = " \
    git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
    <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \
    ...
    gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
    <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm>

    \
    gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
    <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit>

    \
    gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
    <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo>

    \
    gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
    <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass>

    \
    gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
    <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest>

    \
    gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
    <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch>

    \
    ...
    "

    As you can see, there are recursively placed git submodules
    and do_fetch fails with an error message about SRCREV_FORMAT
    having to be set.

    The above is a limited subset of the complete list and
    some of the repositories (like googletest, gloo, glog and pybind11)
    occurs multiple times as child submodules of upper ones,
    with different SRCREV values in different leaf submodules.

    I have not found a conclusive example to use SRCREV_FORMAT
    with git:// + gitsm://

    Can someone enlighten me how to use SRCREV_FORMAT properly
    for this case? Should I stick to the proven working nonstandard way
    and just run git submodule update --init --recursive?

    As an easy way out, is there a flag to git:// to process its
    submodules automatically?

    Thanks in advance,
    Zoltán Böszörményi


    -=-=-=-=-=-=-=-=-=-=-=-
    Links: You receive all messages sent to this group.
    View/Reply Online (#189388):
    https://lists.openembedded.org/g/openembedded-core/message/189388
    Mute This Topic: https://lists.openembedded.org/mt/102038382/3617156
    Group Owner: [email protected]
    <mailto:openembedded-core%[email protected]>
    Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
    [[email protected]]
    -=-=-=-=-=-=-=-=-=-=-=-
#3 lore raw
Quoted 58 lines · Zoltan Boszormenyi

Thanks, I will try it.

2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:

Quoted 54 lines

Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.

On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:

    Hi,

    I have a working python3-pytorch recipe which originally used this:

    =======================================================
    addtask do_git_submodules after do_unpack before do_patch

    do_git_submodules[network] = "1"

    do_git_submodules () {
             cd ${S}
             git submodule update --init --recursive
    }
    =======================================================

    I would like to replace the above with a series of gitsm:// SRC_URI lines.
    Here's a subset of submodules that shows the problem I am seeing:

    SRC_URI = " \
    git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
    <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \
    ...
    gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
    <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm>

    \
    gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
    <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit>

    \
    gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
    <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo>

    \
    gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
    <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass>

    \
    gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
    <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest>

    \
    gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
    <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch>

    \
    ...
    "

Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:

SRC_URI = " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 "

will do all the magic for you :)

You can look at the vulkan-sample recipes in poky for an example of
how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master

Cheers
Julien

Quoted 36 lines
Quoted 17 lines

As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.

The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.

I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://

Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?

Quoted 9 lines

As an easy way out, is there a flag to git:// to process its submodules automatically?

Thanks in advance,
Zoltán Böszörményi

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw

2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta:

Quoted 58 lines · Zoltan Boszormenyi
Quoted 57 lines

Thanks, I will try it.

2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:

Quoted 54 lines

Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.

On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:

     Hi,

     I have a working python3-pytorch recipe which originally used this:

     =======================================================
     addtask do_git_submodules after do_unpack before do_patch

     do_git_submodules[network] = "1"

     do_git_submodules () {
              cd ${S}
              git submodule update --init --recursive
     }
     =======================================================

     I would like to replace the above with a series of gitsm:// SRC_URI lines.
     Here's a subset of submodules that shows the problem I am seeing:

     SRC_URI = " \
     git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
     <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \
     ...
     gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
     <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm>

     \
     gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
     <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit>

     \
     gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
     <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo>

     \
     gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
     <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass>

     \
     gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
     <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest>

     \
     gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
     <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch>

     \
     ...
     "
Quoted 7 lines

Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:

SRC_URI = " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 "

will do all the magic for you :)

When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't.

The problems were:
* PyTorch source releases are not on https://pypi.org/project/torch
   This would have been the most convenient.
* Yocto complains about using the (unstable) release tarball URL from Github.
* Using the git:// SRC_URI did not populate the submodules' directories. This was noticed because some submodules need patches to build under Yocto. Without checking out submodules manually, those patches fail.

Just checked the last one on mickledore, still a problem.

Have Yocto master changed this?

Quoted 41 lines

You can look at the vulkan-sample recipes in poky for an example of
how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master

Cheers
Julien

Quoted 34 lines
Quoted 16 lines

As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.

The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.

I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://

Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?

Quoted 9 lines

As an easy way out, is there a flag to git:// to process its submodules automatically?

Thanks in advance,
Zoltán Böszörményi

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#5 lore raw
Quoted 78 lines · Böszörményi Zoltán

2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta:

Quoted 58 lines · Zoltan Boszormenyi
Quoted 57 lines

Thanks, I will try it.

2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:

Quoted 54 lines

Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.

On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:

     Hi,

     I have a working python3-pytorch recipe which originally used this:

     =======================================================
     addtask do_git_submodules after do_unpack before do_patch

     do_git_submodules[network] = "1"

     do_git_submodules () {
              cd ${S}
              git submodule update --init --recursive
     }
     =======================================================

     I would like to replace the above with a series of gitsm:// SRC_URI lines.
     Here's a subset of submodules that shows the problem I am seeing:

     SRC_URI = " \
     git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
     <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \
     ...
     gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
     <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm>

     \
     gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
     <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit>

     \
     gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
     <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo>

     \
     gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
     <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass>

     \
     gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
     <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest>

     \
     gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
     <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch>

     \
     ...
     "
Quoted 7 lines

Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:

SRC_URI = " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 "

will do all the magic for you :)

When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't.

The problems were:
* PyTorch source releases are not on https://pypi.org/project/torch
This would have been the most convenient.
* Yocto complains about using the (unstable) release tarball URL from Github.
* Using the git:// SRC_URI did not populate the submodules' directories. This was noticed because some submodules need patches to build under Yocto. Without checking out submodules manually, those patches fail.

Quoted 4 lines

Just checked the last one on mickledore, still a problem.

Have Yocto master changed this?

I used to have a kirkstone recipe using gitsm properly to fetch the submodules. Can you reproduce your issue on master? And maybe share it to help debugging?

Cheers
Julien

Quoted 42 lines
Quoted 41 lines

You can look at the vulkan-sample recipes in poky for an example of
how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master

Cheers
Julien

Quoted 34 lines
Quoted 16 lines

As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.

The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.

I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://

Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?

Quoted 9 lines

As an easy way out, is there a flag to git:// to process its submodules automatically?

Thanks in advance,
Zoltán Böszörményi

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
Quoted 1 line

Using the git:// SRC_URI did not populate the submodules' directories.

That's expected, you need to use gitsm:// to do that (at least the ones listed in https://github.com/pytorch/pytorch/blob/main/.gitmodules).

I did not realize that the other gitsm:// you were added were actually submodules of pytorch itself, but Julien is right, it should work with one gitsm:// in SRC_URI (even with patches applied to various submodules in do_patch).

Quoted 169 lines · Julien Stephan

Le mer. 18 oct. 2023 à 16:12, Böszörményi Zoltán <[email protected]> a écrit :

Quoted 3 lines

2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta:

Quoted 1 line

Le mer. 18 oct. 2023 à 15:25, Zoltan Boszormenyi <[email protected]>

Quoted 4 lines · a écrit :
Quoted 4 lines
Quoted 4 lines

Thanks, I will try it.

2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:

Quoted 1 line

Just add the name to pytorch repo url and use only that in

SRCREV_FORMAT, you probably

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

don't care about representing the other repos in SRCPV if you're

updating them only

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

together with main pytorch update.

On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <

Quoted 4 lines · [email protected]> wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi,

I have a working python3-pytorch recipe which originally used

this:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

=======================================================
addtask do_git_submodules after do_unpack before do_patch

do_git_submodules[network] = "1"

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
     do_git_submodules () {
              cd ${S}
              git submodule update --init --recursive
     }
     =======================================================

     I would like to replace the above with a series of gitsm://

SRC_URI lines.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Here's a subset of submodules that shows the problem I am

seeing:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
     SRC_URI = " \
     git://

github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     <

http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
     ...
     gitsm://

github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     <

http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

     \
     gitsm://

github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     <

http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

     \
     gitsm://

github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     <

http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

     \
     gitsm://

github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     <

http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

     \
     gitsm://

github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     <

http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

     \
     gitsm://

github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
     <

http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch

Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

     \
     ...
     "

Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:

SRC_URI = " gitsm://

github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1

Quoted 10 lines
Quoted 3 lines

"

will do all the magic for you :)

When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't.

The problems were:
* PyTorch source releases are not on https://pypi.org/project/torch
This would have been the most convenient.
* Yocto complains about using the (unstable) release tarball URL from

Github.

Quoted 2 lines

* Using the git:// SRC_URI did not populate the submodules' directories. This was noticed because some submodules need patches to build under

Yocto.

Quoted 6 lines

Without checking out submodules manually, those patches fail.

Just checked the last one on mickledore, still a problem.

Have Yocto master changed this?

I used to have a kirkstone recipe using gitsm properly to fetch the submodules. Can you reproduce your issue on master? And maybe share it to help debugging?

Cheers
Julien

Quoted 2 lines
Quoted 2 lines

You can look at the vulkan-sample recipes in poky for an example of how to use it:

https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master

Quoted 11 lines
Quoted 11 lines

Cheers
Julien

Quoted 6 lines
Quoted 6 lines

As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.

The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and

pybind11)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.

I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://

Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard

way

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

and just run git submodule update --init --recursive?

As an easy way out, is there a flag to git:// to process its submodules automatically?

Thanks in advance,
Zoltán Böszörményi

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#189390):

https://lists.openembedded.org/g/openembedded-core/message/189390

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub

[[email protected]]

Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

Using a name for the main git:// SRC_URI entry and setting SRCREV_FORMAT for it works.

The rest were partly my bugs: I had to fix two copy&paste errors in that massive list of gitsm:// SRC_URI entries.

I also had to change a couple of them to use git:// due to the submodules' incorrect declaration in their parent repos. This is detected by Yocto and gave me the error below, indicating the repos in question.

ERROR: python3-pytorch do_fetch: Fetcher failure: Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake.Consider using git fetcher instead.

2023. 10. 18. 15:25 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta:

Quoted 90 lines

Thanks, I will try it.

2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:

Quoted 69 lines

Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.

On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:

    Hi,

    I have a working python3-pytorch recipe which originally used this:

    =======================================================
    addtask do_git_submodules after do_unpack before do_patch

    do_git_submodules[network] = "1"

    do_git_submodules () {
             cd ${S}
             git submodule update --init --recursive }
    =======================================================

    I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing:

    SRC_URI = " \
git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
<http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \
    ...
gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
<http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm>

    \
gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
<http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit>

    \
gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
<http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo>

    \
gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
<http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass>

    \
gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
<http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest>

    \
gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
<http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch>

    \
    ...
    "

    As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.

    The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.

    I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://

    Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?

Quoted 8 lines

    As an easy way out, is there a flag to git:// to process its submodules automatically?

    Thanks in advance,
    Zoltán Böszörményi

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/3617728
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-