SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host
A message that is not in the archive (sent off-list or not yet received): [email protected]
Ping Joshua
//Hongxu
________________________________
From: [email protected] <[email protected]> on behalf of hongxu via lists.openembedded.org <[email protected]>
Sent: Wednesday, November 20, 2024 1:50 PM
To: [email protected] <[email protected]>; [email protected] <[email protected]>; Jia, Hongxu <[email protected]>
Subject: [OE-core] [RR V2][NULL][0/3] SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host
Summary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host
* Git logs
[oe-core]
commit 5ebf6f058259bc073e83bd2fa2b3b116e6156d60
Author: Hongxu Jia <[email protected]> Date: Sat Nov 9 17:16:31 2024 +0800
oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build
$ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source
2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries:
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/meta-openembedded/meta-oe/lib
2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source)
2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message
2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok
2024-11-20 05:21:12,798 - oe-selftest - INFO - ----------------------------------------------------------------------
2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s
2024-11-20 05:21:12,798 - oe-selftest - INFO - OK
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS:
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s)
2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY:
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
Signed-off-by: Hongxu Jia <[email protected]>commit 3753b084cffcbef374093a442d4e692821d1a29b
Author: Hongxu Jia <[email protected]> Date: Sat Nov 9 14:18:26 2024 +0800
sbom30.py: reduce redundant spdxid symlinks to save inode on host
In order to support all in-scope SPDX data within a single JSON-LD file for SPDX 3.0.1, Yocto's SBOM:
- In native/target/nativesdk recipe, created spdxid-hash symlink for each element to point to the JSON-LD file that contains element details;
- In image recipe, use spdxid-hash symlink to collect element details from varies of JSON-LD files
While SPDX_INCLUDE_SOURCES = "1", it adds sources to JSON-LD file and create 2N+ spdxid-hash symlinks for N source files. (N for software_File, N for hasDeclaredLicense's Relationship)
For large numbers of source files, adding an extra symlink -> real file will occupy one more inode (per file), which will need a slot in the OS's inode cache. In this situation, disk performance is slow and inode is used up quickly
After commit [sbom30/spdx30: add link prefix and name to namespace of spdxId and alias] applied, the namespace of spdxId and alias in recipe and package jsonld differs. Use it to create symlink to jsonld, take recipe shadow, package shadow and package shadow-src for example:
For recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/recipe-shadow.spdx.json -> ../recipes/shadow.spdx.json
For package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow.spdx.json -> ../packages/shadow.spdx.json
In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow-src.spdx.json -> ../packages/shadow-src.spdx.json
Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043
echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf
Without this commit:
$ time bitbake core-image-minimal
real 100m17.769s
user 0m24.516s
sys 0m4.334s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824
With this commit:
$ time bitbake core-image-minimal
real 85m12.994s
user 0m20.423s
sys 0m4.228s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043
commit 0ea8bcaa9a8fe63f1935e02968fbaa541734024b
Author: Hongxu Jia <[email protected]> Date: Tue Nov 19 20:12:37 2024 +0800
sbom30/spdx30: add link prefix and name to namespace of spdxId and alias
In order to simple reference the SPDX ID to instead of making jsonld hash path for each element, only creating one symlink for one file and referencing it multiple times, add link prefix and name to the namespace of spdxId and alias to replace ${PN} to avoid namespace conflict between recipe, packages and images.
Take recipe shadow, package shadow and package shadow-src for example:
Without this commit, spdxId and alias in recipe and package jsonld have the same namespace
spdxId: http://spdx.org/spdxdocs/shadow-xxx/...
alias: shadow/UNIHASH/...
After apply this commit, the namespace of spdxId in recipe and package jsonld differs:
In recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...
In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...
In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...
Then will use namespace of spdxId and alias to create link for jsonld file, one symlink for one jsonld file, referenced by elements multiple times
====== Testing ======
* Commands
Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043
echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf
Without this commit:
$ time bitbake core-image-minimal real 100m17.769s user 0m24.516s sys 0m4.334s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824
With this commit:
$ time bitbake core-image-minimal real 85m12.994s user 0m20.423s sys 0m4.228s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043
$ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source 2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries: 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/meta-openembedded/meta-oe/lib 2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source) 2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message 2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok 2024-11-20 05:21:12,798 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s 2024-11-20 05:21:12,798 - oe-selftest - INFO - OK 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS: 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s) 2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY: 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
Ya, I need to look this over closely to try to understand your solution :)
Quoted 205 lines · Hongxu
Ping Joshua
//Hongxu
________________________________
From: [email protected] <[email protected]> on behalf of hongxu via lists.openembedded.org <[email protected]>
Sent: Wednesday, November 20, 2024 1:50 PM
To: [email protected] <[email protected]>; [email protected] <[email protected]>; Jia, Hongxu <[email protected]>
Subject: [OE-core] [RR V2][NULL][0/3] SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on hostSummary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host
* Git logs
[oe-core]
commit5ebf6f058259bc073e83bd2fa2b3b116e6156d60Author: Hongxu Jia <[email protected]> Date: Sat Nov 9 17:16:31 2024 +0800oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source 2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries: 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/meta-openembedded/meta-oe/lib 2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source) 2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message 2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok 2024-11-20 05:21:12,798 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s 2024-11-20 05:21:12,798 - oe-selftest - INFO - OK 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS: 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s) 2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY: 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) Signed-off-by: Hongxu Jia <[email protected]>commit
3753b084cffcbef374093a442d4e692821d1a29bAuthor: Hongxu Jia <[email protected]> Date: Sat Nov 9 14:18:26 2024 +0800
sbom30.py: reduce redundant spdxid symlinks to save inode on hostIn order to support all in-scope SPDX data within a single JSON-LD file for SPDX 3.0.1, Yocto's SBOM:
- In native/target/nativesdk recipe, created spdxid-hash symlink for each element to point to the JSON-LD file that contains element details;
- In image recipe, use spdxid-hash symlink to collect element details from varies of JSON-LD filesWhile
SPDX_INCLUDE_SOURCES= "1", it adds sources to JSON-LD file and create 2N+ spdxid-hash symlinks for N source files. (N forsoftware_File, N for hasDeclaredLicense's Relationship)For large numbers of source files, adding an extra symlink -> real file will occupy one more inode (per file), which will need a slot in the OS's inode cache. In this situation, disk performance is slow and inode is used up quickly
After commit [sbom30/spdx30: add link prefix and name to namespace of spdxId and alias] applied, the namespace of spdxId and alias in recipe and package jsonld differs. Use it to create symlink to jsonld, take recipe shadow, package shadow and package shadow-src for example:
For recipe jsonld
tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.jsonspdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...
symlink:tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/recipe-shadow.spdx.json->../recipes/shadow.spdx.jsonFor package jsonld
tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.jsonspdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...
symlink:tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow.spdx.json->../packages/shadow.spdx.jsonIn package jsonld
tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.jsonspdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...
symlink:tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow-src.spdx.json->../packages/shadow-src.spdx.jsonBuild core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043
echo '
SPDX_INCLUDE_SOURCES= "1"' >> local.confWithout this commit:
$ time bitbake core-image-minimal
real 100m17.769s
user 0m24.516s
sys 0m4.334s$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824
With this commit:
$ time bitbake core-image-minimal
real 85m12.994s
user 0m20.423s
sys 0m4.228s$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043
Signed-off-by Hongxu Jia <[email protected]>commit
0ea8bcaa9a8fe63f1935e02968fbaa541734024bAuthor: Hongxu Jia <[email protected]> Date: Tue Nov 19 20:12:37 2024 +0800sbom30/spdx30: add link prefix and name to namespace of spdxId and alias
In order to simple reference the SPDX ID to instead of making jsonld hash path for each element, only creating one symlink for one file and referencing it multiple times, add link prefix and name to the namespace of spdxId and alias to replace ${PN} to avoid namespace conflict between recipe, packages and images.
Take recipe shadow, package shadow and package shadow-src for example:
Without this commit, spdxId and alias in recipe and package jsonld have the same namespacespdxId: http://spdx.org/spdxdocs/shadow-xxx/...
alias: shadow/UNIHASH/...After apply this commit, the namespace of spdxId in recipe and package jsonld differs:
In recipe jsonldtmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.jsonspdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...In package jsonld
tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.jsonspdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...In package jsonld
tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.jsonspdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...Then will use namespace of spdxId and alias to create link for jsonld file, one symlink for one jsonld file, referenced by elements multiple times
Signed-off-by Hongxu Jia <[email protected]>====== Testing ======
* CommandsBuild core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043
echo '
SPDX_INCLUDE_SOURCES= "1"' >> local.confWithout this commit:
$ time bitbake core-image-minimal real 100m17.769s user 0m24.516s sys 0m4.334s$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824With this commit:
$ time bitbake core-image-minimal real 85m12.994s user 0m20.423s sys 0m4.228s$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043$ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source 2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries: 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/meta-openembedded/meta-oe/lib 2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source) 2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message 2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok 2024-11-20 05:21:12,798 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s 2024-11-20 05:21:12,798 - oe-selftest - INFO - OK 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS: 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s) 2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY: 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)