cve_check.py: Add new method get_ignored_cves
PATCH 1/3 cve_check.py: Add new method get_ignored_cves
Add new method get_ignored_cves in cve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc
While calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()
--- meta/lib/oe/cve_check.py | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
meta/lib/oe/cve_check.py +38 -0
@@ -163,3 +163,41 @@ def cve_check_merge_jsons(output, data): return output["package"].append(data["package"][0]) + +def get_ignored_cves(paths, cves): + """ + Get ignored CVEs from the recipe and exlude the CVEs from + meta/conf/distro/include/cve-extra-exclusions.inc + + While calling this method use below code to get argument values + paths = d.getVar('PATH').split(':') + cves = d.getVar('CVE_CHECK_IGNORE').split() + """ + import os + + cve_extra_exclusion_inc_file = "../meta/conf/distro/include/cve-extra-exclusions.inc" + for path in paths: + check_for_correct_file_path = os.path.join(path, cve_extra_exclusion_inc_file) + if os.path.isfile(check_for_correct_file_path): + inc_file = check_for_correct_file_path + else: + continue + + cve_check_ignored = set() + ignored_cves = set() + with open(inc_file) as f: + lines = f.readlines() + for line in lines: + if line.strip(): + # Ignore the comments from cve-extra-exclusions.inc + if not re.search("^#", line): + cve_match = re.findall(r'CVE\-\d{4}\-\d+', line) + for cve in cve_match: + ignored_cves.add(cve) + + for cve in cves: + if cve not in ignored_cves: + cve_check_ignored.add(cve) + ignored_cves_from_recipe = " ".join(cve_check_ignored) + + return ignored_cves_from_recipe
Signature · 1 line
2.17.1
PATCH 2/3 cve-export.bbclass: Add a new class to get patched and ignored CVEs from the build
This class executes an anonymous function which sets the below variables
CVE_IGNORED = CVEs that are ignored in recipes CVE_PATCHED = CVEs that are fixed by applying patches
It does not consider CVEs that are ignored in poky/meta/conf/distro/include/cve-extra-exclusions.inc and only provide CVEs that are ignored in the recipe.
Default values are set for CVE_PRODUCT and CVE_VERSION to BPN and PV respectively.
Considered setting these values so that anyone can get below information about the CVE from the build.
CVE_PRODUCT
CVE_VERSION
CVE_IGNORED
CVE_PATCHED
--- meta/classes/cve-export.bbclass | 37 +++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 meta/classes/cve-export.bbclass
meta/classes/cve-export.bbclass new file +37 -0
@@ -0,0 +1,37 @@ +# This class is used to get patched and ignored CVEs from the build +# +# To use this class inherit it in the local.conf file. +# +# It executes an anonymous function which sets below variables +# +# CVE_IGNORED = CVEs those are ignored in recipes +# CVE_PATCHED = CVEs those are fixed by applying patches +# +# It does not consider all the CVEs that are ignored in +# poky/meta/conf/distro/include/cve-extra-exclusions.inc +# and only provide CVEs that are ignored in the recipe. +# +# The product name sets default to BPN and version sets default to +# PV but it can be overriden per recipe, to get the value of +# product and version use d.getVar() + +CVE_PRODUCT ??= "${BPN}" +CVE_VERSION ??= "${PV}" +CVE_CHECK_IGNORE ?= "" + +python __anonymous () { + import re + from oe.cve_check import get_patched_cves + from oe.cve_check import get_ignored_cves + + # Check if cve-extra-exclusions.inc file is included or not + if re.search('cve-extra-exclusions.inc', d.getVar('BBINCLUDED')): + paths = d.getVar('PATH').split(':') + cves = d.getVar('CVE_CHECK_IGNORE').split() + ignored_cves = get_ignored_cves(paths, cves) + else: + ignored_cves = " ".join(d.getVar('CVE_CHECK_IGNORE')) + + d.setVar('CVE_IGNORED', ignored_cves) + d.setVar('CVE_PATCHED', " ".join(get_patched_cves(d))) +}
Signature · 1 line
2.17.1
PATCH 3/3 cve_export.py: Add new selftest for cve-export.bbclass
In reply to Akash HadkeThis test compares values of variables CVE_CHECK_IGNORE with CVE_IGNORED that is exported from cve-export.bbclass
--- meta/lib/oeqa/selftest/cases/cve_export.py | 24 ++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/cve_export.py
meta/lib/oeqa/selftest/cases/cve_export.py new file +24 -0
@@ -0,0 +1,24 @@ +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import bitbake, get_bb_var +from oe.cve_check import get_ignored_cves + +class CveExportTest (OESelftestTestCase): + + def test_ignored_cves(self): + """ + This test compares the values of variables CVE_CHECK_IGNORE with + CVE_IGNORED that is exported from cve-export.bbclass + """ + target = 'unzip' + feature = 'INHERIT += "cve-export"\n' + feature += 'require conf/distro/include/cve-extra-exclusions.inc' + self.write_config(feature) + + res = bitbake("%s -c fetch" % target, ignore_status=True) + self.assertEqual(res.status, 0, "\nCouldn't build.\nbitbake output %s" % res.output) + + paths = get_bb_var('PATH', target).split(':') + cves = get_bb_var('CVE_CHECK_IGNORE', target).split() + final_cve = get_ignored_cves(paths, cves) + ignore_cves_from_exported_variable = get_bb_var('CVE_IGNORED', target) + self.assertEqual(final_cve, ignore_cves_from_exported_variable.strip(), "Ignored CVEs are not matching")
Signature · 1 line
2.17.1
Quoted 9 lines · akash hadke via lists.openembedded.org
Add new method
get_ignored_cvesincve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.incWhile calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()
Hello Akash,
While looking into this patch set I'm wondering what is your use case. It seems to be to get a list
of ignored and patched CVEs. This is already available from the cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure we can come with a simple
post-processing script to do the same.
BTW Why do assume people always include meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.
Regards,
Marta
Hello Marta,
Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.
Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the CVE_CHECK_IGNORED list even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc
Best Regards,
Akash
________________________________
From: Marta Rybczynska <[email protected]>
Sent: 17 May 2022 14:42
To: Akash Hadke <[email protected]>
Cc: OE-core <[email protected]>; Ranjitsinh Rathod <[email protected]>; Akash Hadke <[email protected]>
Subject: Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves
Caution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Quoted 9 lines · akash hadke via lists.openembedded.org
Add new method
get_ignored_cvesincve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.incWhile calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()
Hello Akash,
While looking into this patch set I'm wondering what is your use case. It seems to be to get a list
of ignored and patched CVEs. This is already available from the cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure we can come with a simple
post-processing script to do the same.
BTW Why do assume people always include meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.
Regards,
Marta
Quoted 4 lines · Akash Hadke
Hello Marta,
Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables
CVE_IGNOREDandCVE_PATCHEDwith those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.
Hello again Akash,
What you'd like to do is to see the difference in ignored and patched CVEs in buildhistory? Do I get it right?
Quoted 2 lines
Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the
CVE_CHECK_IGNOREDlist even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc
I think I understand the idea. The point I'm making is that if someone does not include the cve-extra-exclusions.inc in their distro, the code will still use it and filter out CVEs they still see when doing cve-check.
Kind regards,
Marta
Quoted 41 lines
Best Regards,
Akash
________________________________
From: Marta Rybczynska <[email protected]>
Sent: 17 May 2022 14:42
To: Akash Hadke <[email protected]>
Cc: OE-core <[email protected]>; Ranjitsinh Rathod <[email protected]>; Akash Hadke <[email protected]>
Subject: Re: [OE-core] [poky][master][PATCH 1/3]cve_check.py: Add new methodget_ignored_cvesCaution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Quoted 9 lines · akash hadke via lists.openembedded.org
Add new method
get_ignored_cvesincve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.incWhile calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()Hello Akash,
While looking into this patch set I'm wondering what is your use case. It seems to be to get a list
of ignored and patched CVEs. This is already available from the cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure we can come with a simple
post-processing script to do the same.BTW Why do assume people always include meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.Regards,
Marta
Hi Marta,
If you see the code from the succeeding patch to this https://lists.openembedded.org/g/openembedded-core/message/165502 here I have checked if cve-extra-exclusions.inc is included or not. If it is not included then the code will not get executed.
Quoted 5 lines · akash hadke via lists.openembedded.org
Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables
CVE_IGNOREDandCVE_PATCHEDwith those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.
I'd really prefer to have one cve handling class where we can configure it to get the data different people need rather than multiple difference cve classes which are going to confuse people. Could we have a way to disable NVD data from the cve-check class?
Cheers,
Richard
Hi Richard,
I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve-check it executes the cve_check task that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.
Quoted 7 lines · akash hadke via lists.openembedded.org
Hi Richard,
I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve- check it executes the
cve_checktask that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.
I guess the task dependencies are a bit of an issue there. There are probably ways to make those configurable, it would just take a little work.
I really do want to encourage us to work together on common cve tooling rather than having several partial implementations so I can't take this patch series.
Cheers,
Richard
Quoted 11 lines · Richard Purdie
Quoted 7 lines · akash hadke via lists.openembedded.org
Hi Richard,
I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve- check it executes the
cve_checktask that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.
Quoted 8 lines
I guess the task dependencies are a bit of an issue there. There are probably ways to make those configurable, it would just take a little work.
I really do want to encourage us to work together on common cve tooling rather than having several partial implementations so I can't take this patch series.
I plan to cut those dependencies and make different stages optional. In my case the first use
is to be able to run multiple cve-checks with the same database (guaranteed without updates)
or run cve-check with some given known database (for testing). We can add an option to make the actual check with the database optional.
Exporting the data to the buildhistory looks like a useful thing to do, too.
I can draft the split after YPS.
Kind regards,
Marta