gitsm:// question
Hi,
I have a working python3-pytorch recipe which originally used this:
=======================================================
addtask do_git_submodules after do_unpack before do_patch
do_git_submodules[network] = "1"
do_git_submodules () {
cd ${S}
git submodule update --init --recursive
}
=======================================================
I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing:
SRC_URI = " \git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 \
...
gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm \
gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit \
gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo \
gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass \
gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest \
gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch \
...
"
As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set.
The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm://
Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?
As an easy way out, is there a flag to git:// to process its submodules automatically?
Thanks in advance,
Zoltán Böszörményi
Just add the name to pytorch repo url and use only that in SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.
Quoted 76 lines · Zoltan Boszormenyi
Hi,
I have a working python3-pytorch recipe which originally used this:
=======================================================
addtaskdo_git_submodulesafterdo_unpackbeforedo_patchdo_git_submodules[network] = "1"
do_git_submodules () { cd ${S} git submodule update --init --recursive }=======================================================
I would like to replace the above with a series of gitsm://
SRC_URIlines. Here's a subset of submodules that shows the problem I am seeing:
SRC_URI= " \git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1\
...
gitsm://
github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
\
gitsm:// github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
\
gitsm:// github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo \
gitsm:// github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
\
gitsm:// github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest \
gitsm:// github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch \
...
"As you can see, there are recursively placed git submodules and
do_fetchfails with an error message aboutSRCREV_FORMAThaving to be set.The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use
SRCREV_FORMATwithgit://+ gitsm://Can someone enlighten me how to use
SRCREV_FORMATproperly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?As an easy way out, is there a flag to
git://to process its submodules automatically?Thanks in advance,
Zoltán Böszörményi-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#189388):
https://lists.openembedded.org/g/openembedded-core/message/189388
Mute This Topic: https://lists.openembedded.org/mt/102038382/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Thanks, I will try it.
2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:
Quoted 87 lines
Just add the name to pytorch repo url and use only that in
SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:
Hi, I have a working python3-pytorch recipe which originally used this: ======================================================= addtask do_git_submodules after do_unpack before do_patch do_git_submodules[network] = "1" do_git_submodules () { cd ${S} git submodule update --init --recursive } ======================================================= I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing: SRC_URI = " \ git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ ... gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> \ gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> \ gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> \ gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> \ gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> \ gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> \ ... " As you can see, there are recursively placed git submodules and do_fetch fails with an error message about SRCREV_FORMAT having to be set. The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules. I have not found a conclusive example to use SRCREV_FORMAT with git:// + gitsm:// Can someone enlighten me how to use SRCREV_FORMAT properly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive? As an easy way out, is there a flag to git:// to process its submodules automatically? Thanks in advance, Zoltán Böszörményi -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#189388): https://lists.openembedded.org/g/openembedded-core/message/189388 Mute This Topic: https://lists.openembedded.org/mt/102038382/3617156 Group Owner: [email protected] <mailto:openembedded-core%[email protected]> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Quoted 58 lines · Zoltan Boszormenyi
Thanks, I will try it.
2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:
Quoted 54 lines
Just add the name to pytorch repo url and use only that in
SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:
Hi, I have a working python3-pytorch recipe which originally used this: ======================================================= addtask do_git_submodules after do_unpack before do_patch do_git_submodules[network] = "1" do_git_submodules () { cd ${S} git submodule update --init --recursive } ======================================================= I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing: SRC_URI = " \ git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ ... gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> \ gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> \ gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> \ gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> \ gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> \ gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> \ ... "
Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:
SRC_URI = " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 "
will do all the magic for you :)
You can look at the vulkan-sample recipes in poky for an example of
how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=master
Cheers
Julien
Quoted 36 lines
Quoted 17 lines
As you can see, there are recursively placed git submodules and
do_fetchfails with an error message aboutSRCREV_FORMAThaving to be set.The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use
SRCREV_FORMATwithgit://+ gitsm://Can someone enlighten me how to use
SRCREV_FORMATproperly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?Quoted 9 lines
As an easy way out, is there a flag to
git://to process its submodules automatically?Thanks in advance,
Zoltán Böszörményi-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta:
Quoted 58 lines · Zoltan Boszormenyi
Quoted 57 lines
Thanks, I will try it.
2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:
Quoted 54 lines
Just add the name to pytorch repo url and use only that in
SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:
Hi, I have a working python3-pytorch recipe which originally used this: ======================================================= addtask do_git_submodules after do_unpack before do_patch do_git_submodules[network] = "1" do_git_submodules () { cd ${S} git submodule update --init --recursive } ======================================================= I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing: SRC_URI = " \ git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ ... gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> \ gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> \ gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> \ gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> \ gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> \ gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> \ ... "
Quoted 7 lines
Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:
SRC_URI= " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 "will do all the magic for you :)
When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't.
The problems were:
* PyTorch source releases are not on https://pypi.org/project/torch
This would have been the most convenient.
* Yocto complains about using the (unstable) release tarball URL from Github.
* Using the git:// SRC_URI did not populate the submodules' directories. This was noticed because some submodules need patches to build under Yocto. Without checking out submodules manually, those patches fail.
Just checked the last one on mickledore, still a problem.
Have Yocto master changed this?
Quoted 41 lines
You can look at the vulkan-sample recipes in poky for an example of
how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=masterCheers
JulienQuoted 34 lines
Quoted 16 lines
As you can see, there are recursively placed git submodules and
do_fetchfails with an error message aboutSRCREV_FORMAThaving to be set.The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use
SRCREV_FORMATwithgit://+ gitsm://Can someone enlighten me how to use
SRCREV_FORMATproperly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?Quoted 9 lines
As an easy way out, is there a flag to
git://to process its submodules automatically?Thanks in advance,
Zoltán Böszörményi-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 78 lines · Böszörményi Zoltán
2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta:
Quoted 58 lines · Zoltan Boszormenyi
Quoted 57 lines
Thanks, I will try it.
2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:
Quoted 54 lines
Just add the name to pytorch repo url and use only that in
SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:
Hi, I have a working python3-pytorch recipe which originally used this: ======================================================= addtask do_git_submodules after do_unpack before do_patch do_git_submodules[network] = "1" do_git_submodules () { cd ${S} git submodule update --init --recursive } ======================================================= I would like to replace the above with a series of gitsm:// SRC_URI lines. Here's a subset of submodules that shows the problem I am seeing: SRC_URI = " \ git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 <http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \ ... gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm <http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm> \ gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit <http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit> \ gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo <http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo> \ gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass <http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass> \ gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest <http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest> \ gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch <http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch> \ ... "Quoted 7 lines
Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:
SRC_URI= " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1 "will do all the magic for you :)
When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't.
The problems were:
* PyTorch source releases are not on https://pypi.org/project/torch
This would have been the most convenient.
* Yocto complains about using the (unstable) release tarball URL from Github.
* Using thegit://SRC_URIdid not populate the submodules' directories. This was noticed because some submodules need patches to build under Yocto. Without checking out submodules manually, those patches fail.
Quoted 4 lines
Just checked the last one on mickledore, still a problem.
Have Yocto master changed this?
I used to have a kirkstone recipe using gitsm properly to fetch the submodules. Can you reproduce your issue on master? And maybe share it to help debugging?
Cheers
Julien
Quoted 42 lines
Quoted 41 lines
You can look at the vulkan-sample recipes in poky for an example of
how to use it: https://git.openembedded.org/openembedded-core/tree/meta/recipes-graphics/vulkan/vulkan-samples_git.bb?h=masterCheers
JulienQuoted 34 lines
Quoted 16 lines
As you can see, there are recursively placed git submodules and
do_fetchfails with an error message aboutSRCREV_FORMAThaving to be set.The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use
SRCREV_FORMATwithgit://+ gitsm://Can someone enlighten me how to use
SRCREV_FORMATproperly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?Quoted 9 lines
As an easy way out, is there a flag to
git://to process its submodules automatically?Thanks in advance,
Zoltán Böszörményi-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 1 line
Using the
git://SRC_URIdid not populate the submodules' directories.
That's expected, you need to use gitsm:// to do that (at least the ones listed in https://github.com/pytorch/pytorch/blob/main/.gitmodules).
I did not realize that the other gitsm:// you were added were actually submodules of pytorch itself, but Julien is right, it should work with one gitsm:// in SRC_URI (even with patches applied to various submodules in do_patch).
Quoted 169 lines · Julien Stephan
Le mer. 18 oct. 2023 à 16:12, Böszörményi Zoltán <[email protected]> a écrit :
Quoted 3 lines
2023. 10. 18. 15:28 keltezéssel, Julien Stephan írta:
Quoted 1 line
Le mer. 18 oct. 2023 à 15:25, Zoltan Boszormenyi <[email protected]>
Quoted 4 lines · a écrit :
Quoted 4 lines
Quoted 4 lines
Thanks, I will try it.
2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:
Quoted 1 line
Just add the name to pytorch repo url and use only that in
SRCREV_FORMAT, you probablyQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
don't care about representing the other repos in SRCPV if you're
updating them only
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
together with main pytorch update.
On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <
Quoted 4 lines · [email protected]> wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hi,
I have a working python3-pytorch recipe which originally used
this:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
=======================================================
addtaskdo_git_submodulesafterdo_unpackbeforedo_patchdo_git_submodules[network] = "1"
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
do_git_submodules () { cd ${S} git submodule update --init --recursive } ======================================================= I would like to replace the above with a series of gitsm://
SRC_URIlines.Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Here's a subset of submodules that shows the problem I am
seeing:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
SRC_URI = " \ git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
... gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
\ gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
\ gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
\ gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
\ gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
\ gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
\ ... "Hi Zoltan
This is not how you are supposed to use gitsm.. This is much simpler:
SRC_URI= " gitsm://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
Quoted 10 lines
Quoted 3 lines
"
will do all the magic for you :)
When I first created that recipe (Yocto 4.0, PyTorch 1.13.0), it didn't.
The problems were:
* PyTorch source releases are not on https://pypi.org/project/torch
This would have been the most convenient.
* Yocto complains about using the (unstable) release tarball URL fromGithub.
Quoted 2 lines
* Using the
git://SRC_URIdid not populate the submodules' directories. This was noticed because some submodules need patches to build underYocto.
Quoted 6 lines
Without checking out submodules manually, those patches fail.
Just checked the last one on mickledore, still a problem.
Have Yocto master changed this?
I used to have a kirkstone recipe using gitsm properly to fetch the submodules. Can you reproduce your issue on master? And maybe share it to help debugging?
Cheers
JulienQuoted 2 lines
Quoted 2 lines
You can look at the vulkan-sample recipes in poky for an example of how to use it:
Quoted 11 lines
Quoted 11 lines
Cheers
JulienQuoted 6 lines
Quoted 6 lines
As you can see, there are recursively placed git submodules and
do_fetchfails with an error message aboutSRCREV_FORMAThaving to be set.The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and
pybind11)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use
SRCREV_FORMATwithgit://+ gitsm://Can someone enlighten me how to use
SRCREV_FORMATproperly for this case? Should I stick to the proven working nonstandardway
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
and just run git submodule update --init --recursive?
As an easy way out, is there a flag to
git://to process its submodules automatically?Thanks in advance,
Zoltán BöszörményiQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#189390):https://lists.openembedded.org/g/openembedded-core/message/189390
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Mute This Topic: https://lists.openembedded.org/mt/102038382/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsubQuoted 3 lines
Quoted 2 lines
Quoted 2 lines
-=-=-=-=-=-=-=-=-=-=-=-
A message that is not in the archive (sent off-list or not yet received): [email protected]
Using a name for the main git:// SRC_URI entry and setting SRCREV_FORMAT for it works.
The rest were partly my bugs: I had to fix two copy&paste errors in that massive list of gitsm:// SRC_URI entries.
I also had to change a couple of them to use git:// due to the submodules' incorrect declaration in their parent repos. This is detected by Yocto and gave me the error below, indicating the repos in question.
ERROR: python3-pytorch do_fetch: Fetcher failure: Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake.Consider using git fetcher instead.
2023. 10. 18. 15:25 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta:
Quoted 90 lines
Thanks, I will try it.
2023. 10. 18. 15:16 keltezéssel, Martin Jansa írta:
Quoted 69 lines
Just add the name to pytorch repo url and use only that in
SRCREV_FORMAT, you probably don't care about representing the other repos in SRCPV if you're updating them only together with main pytorch update.On Wed, Oct 18, 2023 at 3:05 PM Zoltan Boszormenyi <[email protected]> wrote:
Hi,
I have a working python3-pytorch recipe which originally used this:
=======================================================
addtaskdo_git_submodulesafterdo_unpackbeforedo_patchdo_git_submodules[network] = "1"
do_git_submodules() {
cd ${S}
git submodule update --init --recursive }
=======================================================I would like to replace the above with a series of gitsm://
SRC_URIlines. Here's a subset of submodules that shows the problem I am seeing:
SRC_URI= " \git://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1
<http://github.com/pytorch/pytorch.git;protocol=https;branch=release/2.1> \
...
gitsm://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm
<http://github.com/pytorch/FBGEMM.git;protocol=https;name=fbgemm;nobranch=1;destsuffix=third_party/fbgemm>\
gitsm://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit
<http://github.com/asmjit/asmjit.git;protocol=https;name=fbgemmasmjit;destsuffix=third_party/fbgemm/third_party/asmjit>\
gitsm://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo
<http://github.com/pytorch/cpuinfo.git;protocol=https;name=fbgemmcpuinfo;destsuffix=third_party/fbgemm/third_party/cpuinfo>\
gitsm://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass
<http://github.com/NVIDIA/cutlass.git;protocol=https;name=fbgemmcutlass;destsuffix=third_party/fbgemm/third_party/cutlass>\
gitsm://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest
<http://github.com/google/googletest.git;protocol=https;name=fbgemmgtest;destsuffix=third_party/fbgemm/third_party/googletest>\
gitsm://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch
<http://github.com/ROCmSoftwarePlatform/hipify_torch.git;protocol=https;name=fbgemmhiptorch;destsuffix=third_party/fbgemm/third_party/hipify_torch>\
...
"As you can see, there are recursively placed git submodules and
do_fetchfails with an error message aboutSRCREV_FORMAThaving to be set.The above is a limited subset of the complete list and some of the repositories (like googletest, gloo, glog and pybind11) occurs multiple times as child submodules of upper ones, with different SRCREV values in different leaf submodules.
I have not found a conclusive example to use
SRCREV_FORMATwithgit://+ gitsm://Can someone enlighten me how to use
SRCREV_FORMATproperly for this case? Should I stick to the proven working nonstandard way and just run git submodule update --init --recursive?Quoted 8 lines
As an easy way out, is there a flag to
git://to process its submodules automatically?Thanks in advance,
Zoltán Böszörményi-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189390): https://lists.openembedded.org/g/openembedded-core/message/189390
Mute This Topic: https://lists.openembedded.org/mt/102038382/3617728
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-