generate-cve-exclusions: Add a new bbclass
v4 cover generate-cve-exclusions: Add a new bbclass
Changes since v3:
- Patch 2/4:
* Add variables to control offline mode, source URI and SRCREV for deterministic testing (GENERATE_CVE_EXCLUSIONS_SRC_URI, GENERATE_CVE_EXCLUSIONS_SRCREV, GENERATE_CVE_EXCLUSIONS_NETWORK).
* Updated generate_cve_exclusions task scheduling to be executed before do_cve_check.
Changes since v2:
- Patch 4/4: Inherit the new bbclass in linux-yocto.inc instead of individual recipes.
Changes since v1:
- Patch 2/4: Removed the mandatory execution of the generate-cve-exclusions class on every build. It now needs to be manually run using:
bitbake -c generate-cve-exclusions <kernel-recipe>
ValentinBoudevin (4):
generate-cve-exclusions: Add --output-json option
generate-cve-exclusions: Add a .bbclass
generate-cve-exclusions: Move python script
linux: Add inherit on generate-cve-exclusions
meta/classes/generate-cve-exclusions.bbclass | 97 +++++++++++++++++++
meta/recipes-kernel/linux/linux-yocto.inc | 3 +
.../contrib}/generate-cve-exclusions.py | 64 +++++++++---
3 files changed, 150 insertions(+), 14 deletions(-)
create mode 100644 meta/classes/generate-cve-exclusions.bbclass
rename {meta/recipes-kernel/linux => scripts/contrib}/generate-cve-exclusions.py (71%)PATCH v4 1/4 generate-cve-exclusions: Add --output-json option
This option "--output-json" can be used to return a json file instead of the standard .inc file provided.
The JSON file can easily be manipulated contrary to the .inc file.
Example output structure of the JSON file:
```json
{
"cve_status": {
"CVE-2019-25160": {
"active": false,
"message": "fixed-version: Fixed from version 5.0"
},
"CVE-2019-25162": {
"active": false,
"message": "fixed-version: Fixed from version 6.0"
},
...
```Also, this commit doesn't affect or modify any existing behaviour of the script.
--- .../linux/generate-cve-exclusions.py | 64 +++++++++++++++---- 1 file changed, 50 insertions(+), 14 deletions(-)
meta/recipes-kernel/linux/generate-cve-exclusions.py +50 -14
@@ -91,6 +91,7 @@ def main(argp=None): parser = argparse.ArgumentParser() parser.add_argument("datadir", type=pathlib.Path, help="Path to a clone of https://github.com/CVEProject/cvelistV5 or https://git.kernel.org/pub/scm/linux/security/vulns.git") parser.add_argument("version", type=Version, help="Kernel version number to generate data for, such as 6.1.38") + parser.add_argument("--output-json", action="store_true", help="Return CVE_STATUS mapping as JSON") args = parser.parse_args(argp) datadir = args.datadir.resolve()
@@ -99,7 +100,10 @@ def main(argp=None): data_version = subprocess.check_output(("git", "describe", "--tags", "HEAD"), cwd=datadir, text=True) - print(f""" + cve_status = {} + + if not args.output_json: + print(f""" # Auto-generated CVE metadata, DO NOT EDIT BY HAND. # Generated at {datetime.datetime.now(datetime.timezone.utc)} for kernel version {version} # From {datadir.name} {data_version}
@@ -131,26 +135,58 @@ do_cve_check[prefuncs] += "check_kernel_cve_status_version" continue first_affected, fixed, backport_ver = get_fixed_versions(cve_info, base_version) if not fixed: - print(f"# {cve} has no known resolution") + cve_status[cve] = { + "active": True, + "message": "no known resolution" + } + if not args.output_json: + print(f"# {cve} has no known resolution") elif first_affected and version < first_affected: - print(f'CVE_STATUS[{cve}] = "fixed-version: only affects {first_affected} onwards"') + cve_status[cve] = { + "active": False, + "message": f"fixed-version: only affects {first_affected} onwards" + } + if not args.output_json: + print(f'CVE_STATUS[{cve}] = "fixed-version: only affects {first_affected} onwards"') elif fixed <= version: - print( - f'CVE_STATUS[{cve}] = "fixed-version: Fixed from version {fixed}"' - ) + cve_status[cve] = { + "active": False, + "message": f"fixed-version: Fixed from version {fixed}" + } + if not args.output_json: + print(f'CVE_STATUS[{cve}] = "fixed-version: Fixed from version {fixed}"') else: if backport_ver: if backport_ver <= version: - print( - f'CVE_STATUS[{cve}] = "cpe-stable-backport: Backported in {backport_ver}"' - ) + cve_status[cve] = { + "active": False, + "message": f"cpe-stable-backport: Backported in {backport_ver}" + } + if not args.output_json: + print(f'CVE_STATUS[{cve}] = "cpe-stable-backport: Backported in {backport_ver}"') else: - print(f"# {cve} may need backporting (fixed from {backport_ver})") + cve_status[cve] = { + "active": True, + "message": f"May need backporting (fixed from {backport_ver})" + } + if not args.output_json: + print(f"# {cve} may need backporting (fixed from {backport_ver})") else: - print(f"# {cve} needs backporting (fixed from {fixed})") - - print() - + cve_status[cve] = { + "active": True, + "message": f"#Needs backporting (fixed from {fixed})" + } + if not args.output_json: + print(f"# {cve} needs backporting (fixed from {fixed})") + + if not args.output_json: + print() + + # Emit structured output if --ret-struct was requested + if args.output_json: + print(json.dumps({ + "cve_status": cve_status, + }, indent=2)) if __name__ == "__main__": main()
PATCH v4 2/4 generate-cve-exclusions: Add a .bbclass
In reply to ValentinBoudevinAdd a .bbclass to generate-cve-exclusions to use this script at every run.
Two steps for testing:
1) Inherit this class in the kernel recipe with "inherit generate-cve-exclusions.bbclass"
2) Use the following command to generate a cvelistV5 entry with a JSON file in in ${WORKDIR}/cvelistV5/ :
"bitbake linux-yocto -c generate-cve-exclusions"
The JSON file can then be parsed in the following run by cve-check.
This class contains several methods:
*do_clone_cvelistV5: Clone the cvelistV5 repo in
${WORKDIR}/cvelistV5/git
(e.g. bitbake-builds/poky-master/build/tmp/work/qemux86_64-poky-linux/ linux-yocto/6.18.1+git/cvelistV5/git)
*do_generate_cve_exclusions: Use the script generate-cve-exclusions.py. It uses the new "--output-json" argument to generate a JSON file as an output stored in ${WORKDIR}/cvelistV5//cve-exclusion_${LINUX_VERSION}.json
*do_cve_check:prepend: Parse the previously generated JSON file to set the variable CVE_STATUS corretly
The class also provides some variables:*GENERATE_CVE_EXCLUSIONS_SRC_URI and GENERATE_CVE_EXCLUSIONS_SRCREV can be used to change the source repository or fix a commit with SRCREV (usefull for deterministic testing) *GENERATE_CVE_EXCLUSIONS_NETWORK can be set to 0 to provide an offline mode based on DL_DIR directory. *GENERATE_CVE_EXCLUSIONS_WORKDIR path used as a working directory for this class
--- meta/classes/generate-cve-exclusions.bbclass | 97 ++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 meta/classes/generate-cve-exclusions.bbclass
meta/classes/generate-cve-exclusions.bbclass new file +97 -0
@@ -0,0 +1,97 @@ +GENERATE_CVE_EXCLUSIONS_SRC_URI ?= "git://github.com/CVEProject/cvelistV5.git;branch=main;protocol=https;destsuffix=git" +GENERATE_CVE_EXCLUSIONS_SRCREV ?= "${@bb.fetch2.get_autorev(d)}" +GENERATE_CVE_EXCLUSIONS_NETWORK ?= "1" +GENERATE_CVE_EXCLUSIONS_WORKDIR ?= "${WORKDIR}/cvelistV5" + +SRC_URI:append = " ${GENERATE_CVE_EXCLUSIONS_SRC_URI};name=generate-cve-exclusions" +SRCREV_generate-cve-exclusions = "${GENERATE_CVE_EXCLUSIONS_SRCREV}" + +python do_clone_cvelistV5() { + import subprocess + import shutil, os + network_allowed = d.getVar("GENERATE_CVE_EXCLUSIONS_NETWORK") == "1" + rootdir = d.getVar("GENERATE_CVE_EXCLUSIONS_WORKDIR") + # Remove existing unpacked directory if any + if os.path.exists(rootdir): + shutil.rmtree(rootdir) + # Prepare fetcher + src_uri_list = (d.getVar('SRC_URI') or "").split() + fetcher = bb.fetch2.Fetch(src_uri_list, d) + # Clone only if network is allowed + if network_allowed: + fetcher.download() + else: + # Offline mode without network access + bb.note("GENERATE_CVE_EXCLUSIONS_NETWORK=0: Skipping online fetch. Checking local downloads in DL_DIR...") + have_sources = False + dl_dir = d.getVar("DL_DIR") + srcrev = d.getVar("SRCREV") + # Check SRCREV is NOT set to AUTOREV + if srcrev.strip() in ("${AUTOREV}", "AUTOINC"): + bb.warn("Offline mode but SRCREV is set to AUTOREV/AUTOINC. Cannot proceed without network access.") + return + # Loop through the fetcher's expanded URL data + for ud in fetcher.expanded_urldata(): + ud.setup_localpath(d) + # Check mirror tarballs first + for mirror_fname in ud.mirrortarballs: + mirror_path = os.path.join(dl_dir, mirror_fname) + if os.path.exists(mirror_path): + bb.note(f"Found mirror tarball: {mirror_path}") + have_sources = True + break + # If no mirror, check original download path + if not have_sources and ud.localpath and os.path.exists(ud.localpath): + bb.note(f"Found local download: {ud.localpath}") + have_sources = True + if not have_sources: + bb.warn("Offline mode but required source is missing.\n"f"SRC_URI = {ud.url}") + return + # Unpack into the standard work directory + fetcher.unpack(rootdir) + # Remove the folder ${PN} set by unpack + subdirs = [d for d in os.listdir(rootdir) if os.path.isdir(os.path.join(rootdir, d))] + if len(subdirs) == 1: + srcdir = os.path.join(rootdir, subdirs[0]) + for f in os.listdir(srcdir): + shutil.move(os.path.join(srcdir, f), rootdir) + shutil.rmtree(srcdir) + bb.note("Vulnerabilities repo unpacked into: %s" % rootdir) +} +do_clone_cvelistV5[network] = "${GENERATE_CVE_EXCLUSIONS_NETWORK}" +do_clone_cvelistV5[nostamp] = "1" +do_clone_cvelistV5[doc] = "Clone CVE information from the CVE Project: https://github.com/CVEProject/cvelistV5.git" +addtask clone_cvelistV5 before do_generate_cve_exclusions + +do_generate_cve_exclusions() { + generate_cve_exclusions_script=$(find ${COREBASE} -name "generate-cve-exclusions.py") + if [ -z "${generate_cve_exclusions_script}" ]; then + bbfatal "generate-cve-exclusions.py not found in ${COREBASE}." + fi + python3 "${generate_cve_exclusions_script}" \ + "${GENERATE_CVE_EXCLUSIONS_WORKDIR}/git" \ + ${LINUX_VERSION} \ + --output-json > ${GENERATE_CVE_EXCLUSIONS_WORKDIR}/cve-exclusion_${LINUX_VERSION}.json +} +do_generate_cve_exclusions[nostamp] = "1" +do_generate_cve_exclusions[doc] = "Generate CVE exclusions for the kernel build. (e.g., cve-exclusion_6.12.inc)" +addtask generate_cve_exclusions after do_clone_cvelistV5 before do_cve_check + +python do_cve_check:prepend() { + import os + import json + workdir = d.getVar("GENERATE_CVE_EXCLUSIONS_WORKDIR") + kernel_version = d.getVar("LINUX_VERSION") + json_input_file = os.path.join(workdir, "cve-exclusion_%s.json" % kernel_version) + if os.path.exists(json_input_file): + with open(json_input_file, 'r', encoding='utf-8') as f: + cve_data = json.load(f) + cve_status_dict = cve_data.get("cve_status", {}) + count = 0 + for cve_id, info in cve_status_dict.items(): + if info.get("active", True): + continue + d.setVarFlag("CVE_STATUS", cve_id, info.get("message", "")) + count += 1 + bb.note("Loaded %d CVE_STATUS entries from JSON output for kernel %s" % (count, kernel_version)) +}
PATCH v4 3/4 generate-cve-exclusions: Move python script
In reply to ValentinBoudevinThe script should be located with other scripts in scripts/contrib instead of staying in meta/classes/.
Update the new .bbclass to match this modification
---
meta/classes/generate-cve-exclusions.bbclass | 2 +-
.../linux => scripts/contrib}/generate-cve-exclusions.py | 0
2 files changed, 1 insertion(+), 1 deletion(-)
rename {meta/recipes-kernel/linux => scripts/contrib}/generate-cve-exclusions.py (100%)meta/classes/generate-cve-exclusions.bbclass +1 -1
@@ -64,7 +64,7 @@ do_clone_cvelistV5[doc] = "Clone CVE information from the CVE Project: https://g addtask clone_cvelistV5 before do_generate_cve_exclusions do_generate_cve_exclusions() { - generate_cve_exclusions_script=$(find ${COREBASE} -name "generate-cve-exclusions.py") + generate_cve_exclusions_script=${COREBASE}/scripts/contrib/generate-cve-exclusions.py if [ -z "${generate_cve_exclusions_script}" ]; then bbfatal "generate-cve-exclusions.py not found in ${COREBASE}." fi
scripts/contrib/generate-cve-exclusions.py renamed from meta/recipes-kernel/linux/generate-cve-exclusions.py +0 -0
PATCH v4 4/4 linux: Add inherit on generate-cve-exclusions
In reply to ValentinBoudevinUpdate linux-yocto.inc to inherit the new generate-cve-exclusions class.
--- meta/recipes-kernel/linux/linux-yocto.inc | 3 +++ 1 file changed, 3 insertions(+)
meta/recipes-kernel/linux/linux-yocto.inc +3 -0
@@ -5,6 +5,9 @@ HOMEPAGE = "https://www.yoctoproject.org/" LIC_FILES_CHKSUM ?= "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7" +# Generate Dynamic CVE Exclusions +inherit generate-cve-exclusions + UPSTREAM_CHECK_GITTAGREGEX = "(?P<pver>\d+\.\d+(\.\d+)*)" RECIPE_NO_UPDATE_REASON = "Recipe is updated through a separate process"
Quoted 13 lines · vboudevin via lists.openembedded.org
Changes since v3:
- Patch 2/4:
* Add variables to control offline mode, source URI and SRCREV for deterministic testing (GENERATE_CVE_EXCLUSIONS_SRC_URI,GENERATE_CVE_EXCLUSIONS_SRCREV,GENERATE_CVE_EXCLUSIONS_NETWORK).
* Updatedgenerate_cve_exclusionstask scheduling to be executed beforedo_cve_check.Changes since v2:
- Patch 4/4: Inherit the new bbclass in linux-yocto.inc instead of individual recipes.
Quoted 5 lines
Changes since v1:
- Patch 2/4: Removed the mandatory execution of the generate-cve-exclusions class on every build. It now needs to be manually run using:
bitbake -c generate-cve-exclusions <kernel-recipe>
I think it will be good to update documentation as well mentioning new variables and the step
https://docs.yoctoproject.org/dev/ref-manual/variables.html
https://docs.yoctoproject.org/dev/dev-manual/vulnerabilities.html
Quoted 22 lines
ValentinBoudevin (4):
generate-cve-exclusions: Add --output-json option
generate-cve-exclusions: Add a .bbclass
generate-cve-exclusions: Move python script
linux: Add inherit on generate-cve-exclusionsmeta/classes/generate-cve-exclusions.bbclass | 97 +++++++++++++++++++ meta/recipes-kernel/linux/linux-yocto.inc | 3 + .../contrib}/generate-cve-exclusions.py | 64 +++++++++--- 3 files changed, 150 insertions(+), 14 deletions(-) create mode 100644 meta/classes/generate-cve-exclusions.bbclass rename {meta/recipes-kernel/linux => scripts/contrib}/generate-cve-exclusions.py (71%)-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#229427): https://lists.openembedded.org/g/openembedded-core/message/229427
Mute This Topic: https://lists.openembedded.org/mt/117285139/3619737
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v4 2/4] generate-cve-exclusions: Add a .bbclass
In reply to ValentinBoudevinHi,
Quoted 6 lines · vboudevin via lists.openembedded.org
Add a .bbclass to generate-cve-exclusions to use this script at every run.
Two steps for testing:
1) Inherit this class in the kernel recipe with "inherit generate-cve-exclusions.bbclass"
If this class is only meant to be used in a kernel recipe, maybe "kernel-generate-cve-exclusions" would be more appropriate name for it?
Not having "kernel" in it would make the class name sound like it could be used elsewhere.
Antonin
Signature · 3 lines
Antonin Godard, Bootlin Embedded Linux and Kernel engineering https://bootlin.com