Skip to contentSkip to conversation

SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host

PATCH Jia, Hongxu 2 participants 2 messages Nov 21, 2024 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Ping Joshua

//Hongxu

________________________________
From: [email protected] <[email protected]> on behalf of hongxu via lists.openembedded.org <[email protected]>
Sent: Wednesday, November 20, 2024 1:50 PM
To: [email protected] <[email protected]>; [email protected] <[email protected]>; Jia, Hongxu <[email protected]>
Subject: [OE-core] [RR V2][NULL][0/3] SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host

Summary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host

* Git logs
[oe-core]
commit 5ebf6f058259bc073e83bd2fa2b3b116e6156d60

Author: Hongxu Jia <[email protected]>
Date:   Sat Nov 9 17:16:31 2024 +0800
    oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build

    $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source
    2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries:
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta/lib
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-yocto-bsp/lib
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-selftest/lib
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/meta-openembedded/meta-oe/lib
    2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable
    2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf
    2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
    2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source)
    2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message
    2024-11-20 05:14:08,115 - oe-selftest - INFO -  ... ok
    2024-11-20 05:21:12,798 - oe-selftest - INFO - ----------------------------------------------------------------------
    2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s
    2024-11-20 05:21:12,798 - oe-selftest - INFO - OK
    2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS:
    2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s)
    2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY:
    2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s
    2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)

    Signed-off-by: Hongxu Jia <[email protected]>

commit 3753b084cffcbef374093a442d4e692821d1a29b

Author: Hongxu Jia <[email protected]>
Date:   Sat Nov 9 14:18:26 2024 +0800

sbom30.py: reduce redundant spdxid symlinks to save inode on host

In order to support all in-scope SPDX data within a single JSON-LD file for SPDX 3.0.1, Yocto's SBOM:
- In native/target/nativesdk recipe, created spdxid-hash symlink for each element to point to the JSON-LD file that contains element details;
- In image recipe, use spdxid-hash symlink to collect element details from varies of JSON-LD files

While SPDX_INCLUDE_SOURCES = "1", it adds sources to JSON-LD file and create 2N+ spdxid-hash symlinks for N source files. (N for software_File, N for hasDeclaredLicense's Relationship)

For large numbers of source files, adding an extra symlink -> real file will occupy one more inode (per file), which will need a slot in the OS's inode cache. In this situation, disk performance is slow and inode is used up quickly

After commit [sbom30/spdx30: add link prefix and name to namespace of spdxId and alias] applied, the namespace of spdxId and alias in recipe and package jsonld differs. Use it to create symlink to jsonld, take recipe shadow, package shadow and package shadow-src for example:

For recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/recipe-shadow.spdx.json -> ../recipes/shadow.spdx.json

For package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow.spdx.json -> ../packages/shadow.spdx.json

In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow-src.spdx.json -> ../packages/shadow-src.spdx.json

Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043

echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf

Without this commit:
$ time bitbake core-image-minimal
real 100m17.769s
user 0m24.516s
sys 0m4.334s

$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824

With this commit:
$ time bitbake core-image-minimal
real 85m12.994s
user 0m20.423s
sys 0m4.228s

$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043

commit 0ea8bcaa9a8fe63f1935e02968fbaa541734024b

Author: Hongxu Jia <[email protected]>
Date:   Tue Nov 19 20:12:37 2024 +0800

sbom30/spdx30: add link prefix and name to namespace of spdxId and alias

In order to simple reference the SPDX ID to instead of making jsonld hash path for each element, only creating one symlink for one file and referencing it multiple times, add link prefix and name to the namespace of spdxId and alias to replace ${PN} to avoid namespace conflict between recipe, packages and images.

Take recipe shadow, package shadow and package shadow-src for example:
Without this commit, spdxId and alias in recipe and package jsonld have the same namespace

spdxId: http://spdx.org/spdxdocs/shadow-xxx/...
alias: shadow/UNIHASH/...

After apply this commit, the namespace of spdxId in recipe and package jsonld differs:
In recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...

In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...

In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...

Then will use namespace of spdxId and alias to create link for jsonld file, one symlink for one jsonld file, referenced by elements multiple times

====== Testing ======
* Commands

Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043

echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf

Without this commit:

$ time bitbake core-image-minimal
real    100m17.769s
user    0m24.516s
sys     0m4.334s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l
7281824

With this commit:

$ time bitbake core-image-minimal
real    85m12.994s
user    0m20.423s
sys     0m4.228s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l
6043
$ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source
2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries:
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-yocto-bsp/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-selftest/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/meta-openembedded/meta-oe/lib
2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source)
2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message
2024-11-20 05:14:08,115 - oe-selftest - INFO -  ... ok
2024-11-20 05:21:12,798 - oe-selftest - INFO - ----------------------------------------------------------------------
2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s
2024-11-20 05:21:12,798 - oe-selftest - INFO - OK
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS:
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s)
2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY:
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
#2 lore raw

Ya, I need to look this over closely to try to understand your solution :)

Quoted 205 lines · Hongxu

Ping Joshua

//Hongxu

________________________________
From: [email protected] <[email protected]> on behalf of hongxu via lists.openembedded.org <[email protected]>
Sent: Wednesday, November 20, 2024 1:50 PM
To: [email protected] <[email protected]>; [email protected] <[email protected]>; Jia, Hongxu <[email protected]>
Subject: [OE-core] [RR V2][NULL][0/3] SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host

Summary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host

* Git logs
[oe-core]
commit 5ebf6f058259bc073e83bd2fa2b3b116e6156d60

Author: Hongxu Jia <[email protected]>
Date:   Sat Nov 9 17:16:31 2024 +0800
    oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build

    $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source
    2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries:
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta/lib
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-yocto-bsp/lib
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-selftest/lib
    2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/meta-openembedded/meta-oe/lib
    2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable
    2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf
    2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
    2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source)
    2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message
    2024-11-20 05:14:08,115 - oe-selftest - INFO -  ... ok
    2024-11-20 05:21:12,798 - oe-selftest - INFO - ----------------------------------------------------------------------
    2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s
    2024-11-20 05:21:12,798 - oe-selftest - INFO - OK
    2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS:
    2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s)
    2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY:
    2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s
    2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)

    Signed-off-by: Hongxu Jia <[email protected]>

commit 3753b084cffcbef374093a442d4e692821d1a29b

Author: Hongxu Jia <[email protected]>
Date:   Sat Nov 9 14:18:26 2024 +0800

sbom30.py: reduce redundant spdxid symlinks to save inode on host

In order to support all in-scope SPDX data within a single JSON-LD file for SPDX 3.0.1, Yocto's SBOM:
- In native/target/nativesdk recipe, created spdxid-hash symlink for each element to point to the JSON-LD file that contains element details;
- In image recipe, use spdxid-hash symlink to collect element details from varies of JSON-LD files

While SPDX_INCLUDE_SOURCES = "1", it adds sources to JSON-LD file and create 2N+ spdxid-hash symlinks for N source files. (N for software_File, N for hasDeclaredLicense's Relationship)

For large numbers of source files, adding an extra symlink -> real file will occupy one more inode (per file), which will need a slot in the OS's inode cache. In this situation, disk performance is slow and inode is used up quickly

After commit [sbom30/spdx30: add link prefix and name to namespace of spdxId and alias] applied, the namespace of spdxId and alias in recipe and package jsonld differs. Use it to create symlink to jsonld, take recipe shadow, package shadow and package shadow-src for example:

For recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/recipe-shadow.spdx.json -> ../recipes/shadow.spdx.json

For package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow.spdx.json -> ../packages/shadow.spdx.json

In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...
symlink: tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow-src.spdx.json -> ../packages/shadow-src.spdx.json

Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043

echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf

Without this commit:
$ time bitbake core-image-minimal
real 100m17.769s
user 0m24.516s
sys 0m4.334s

$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l 7281824

With this commit:
$ time bitbake core-image-minimal
real 85m12.994s
user 0m20.423s
sys 0m4.228s

$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l 6043

commit 0ea8bcaa9a8fe63f1935e02968fbaa541734024b

Author: Hongxu Jia <[email protected]>
Date:   Tue Nov 19 20:12:37 2024 +0800

sbom30/spdx30: add link prefix and name to namespace of spdxId and alias

In order to simple reference the SPDX ID to instead of making jsonld hash path for each element, only creating one symlink for one file and referencing it multiple times, add link prefix and name to the namespace of spdxId and alias to replace ${PN} to avoid namespace conflict between recipe, packages and images.

Take recipe shadow, package shadow and package shadow-src for example:
Without this commit, spdxId and alias in recipe and package jsonld have the same namespace

spdxId: http://spdx.org/spdxdocs/shadow-xxx/...
alias: shadow/UNIHASH/...

After apply this commit, the namespace of spdxId in recipe and package jsonld differs:
In recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...

In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...

In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json

spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...

Then will use namespace of spdxId and alias to create link for jsonld file, one symlink for one jsonld file, referenced by elements multiple times

====== Testing ======
* Commands

Build core-image-minimal with/without this commit, comparing the spdxid-link number, 7 281 824 -> 6 043

echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf

Without this commit:

$ time bitbake core-image-minimal
real    100m17.769s
user    0m24.516s
sys     0m4.334s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l
7281824

With this commit:

$ time bitbake core-image-minimal
real    85m12.994s
user    0m20.423s
sys     0m4.228s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l
6043
$ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source
2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries:
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-yocto-bsp/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/poky/meta-selftest/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -  path-to/meta-openembedded/meta-oe/lib
2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is valid/parsable
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" in path-to/build_spdx3-st/conf/local.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - test_core_image_minimal_include_source (spdx.SPDX30Check.test_core_image_minimal_include_source)
2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message
2024-11-20 05:14:08,115 - oe-selftest - INFO -  ... ok
2024-11-20 05:21:12,798 - oe-selftest - INFO - ----------------------------------------------------------------------
2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s
2024-11-20 05:21:12,798 - oe-selftest - INFO - OK
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS:
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s)
2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY:
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 6706.272s
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)