python3-ninja to speed up cmake builds of python modules
Hi,
Recently, I had to create recipes for a few python modules, among which a few used cmake to build their C or C++ pieces. I saw from their build log that they complained about the missing (optional) "ninja" python module, and falling back to the traditional compile-one-file-at-a-time method.
Do we want to speed up building python modules that exploit this or using ninja is too rare to bother?
Thanks,
Zoltán Böszörményi
Quoted 11 lines · Zoltan Boszormenyi via lists.openembedded.org
Hi,
Recently, I had to create recipes for a few python modules, among which a few used cmake to build their C or C++ pieces. I saw from their build log that they complained about the missing (optional) "ninja" python module, and falling back to the traditional compile-one-file-at-a-time method.
Do we want to speed up building python modules that exploit this or using ninja is too rare to bother?
Ninja is used by cmake.bbclass and meson.bbclass, so absolutely use ninja if the build supports it.
Ross
2023. 03. 08. 12:43 keltezéssel, Ross Burton írta:
Quoted 9 lines · Zoltan Boszormenyi via lists.openembedded.org
Quoted 8 lines
Hi,
Recently, I had to create recipes for a few python modules, among which a few used cmake to build their C or C++ pieces. I saw from their build log that they complained about the missing (optional) "ninja" python module, and falling back to the traditional compile-one-file-at-a-time method.
Quoted 3 lines
Quoted 2 lines
Do we want to speed up building python modules that exploit this or using ninja is too rare to bother?
Ninja is used by cmake.bbclass and meson.bbclass, so absolutely use ninja if the build supports it.
Cool.
However, I am talking about this below when trying to build ByteTrack from https://github.com/ifzhang/ByteTrack
which needs pytorch to build. Pytorch also has some build extensions itself:
python3-bytetrack/0.0.1-r0/temp/log.do_compile:/data2/yocto-4.0/tmp-sicom-glibc/work/corei7-64-oe-linux/python3-bytetrack/0.0.1-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/torch/utils/cpp_extension.py:415: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
The code in pytorch does "import ninja" conditionally, so it expects it as a python module. Also, this specific package and anything that uses pytorch for its build (like torchvision) complains about the same.
My question was out of curiosity, not as a real need.
Quoted 7 lines · Böszörményi Zoltán
The code in pytorch does "import ninja" conditionally, so it expects it as a python module. Also, this specific package and anything that uses pytorch for its build (like torchvision) complains about the same.
That would be:
https://pypi.org/project/ninja/
https://github.com/scikit-build/ninja-python-distributions
Which is a bit icky. We have a ninja binary already, I wonder if we can build just the python pieces there.
Ross
2023. 03. 08. 13:40 keltezéssel, Ross Burton írta:
Quoted 7 lines
Quoted 6 lines · Böszörményi Zoltán
The code in pytorch does "import ninja" conditionally, so it expects it as a python module. Also, this specific package and anything that uses pytorch for its build (like torchvision) complains about the same.
Quoted 6 lines
That would be:
https://pypi.org/project/ninja/
https://github.com/scikit-build/ninja-python-distributionsWhich is a bit icky. We have a ninja binary already, I wonder if we can build just the python pieces there.
Is there a way to distinguish between the download files for cmake vs python3-cmake
ninja vs python3-ninja
?
The file names from pypi.org are exactly the same as the source downloads of cmake and ninja of the same versions but obviously with different checksums.