Skip to contentSkip to conversation

cve_check.py: Add new method get_ignored_cves

PATCH In review Akash Hadke 3 participants 11 messages May 11, 2022 – May 18, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 cve_check.py: Add new method get_ignored_cves

Add new method get_ignored_cves in cve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc

While calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()

---
 meta/lib/oe/cve_check.py | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
meta/lib/oe/cve_check.py +38 -0
index dc7d2e2826..d96d47b737 100644
@@ -163,3 +163,41 @@ def cve_check_merge_jsons(output, data):
             return
 
     output["package"].append(data["package"][0])
+
+def get_ignored_cves(paths, cves):
+    """
+    Get ignored CVEs from the recipe and exlude the CVEs from
+    meta/conf/distro/include/cve-extra-exclusions.inc
+
+    While calling this method use below code to get argument values
+    paths = d.getVar('PATH').split(':')
+    cves = d.getVar('CVE_CHECK_IGNORE').split()
+    """
+    import os
+
+    cve_extra_exclusion_inc_file = "../meta/conf/distro/include/cve-extra-exclusions.inc"
+    for path in paths:
+        check_for_correct_file_path = os.path.join(path, cve_extra_exclusion_inc_file)
+        if os.path.isfile(check_for_correct_file_path):
+            inc_file = check_for_correct_file_path
+        else:
+            continue
+
+    cve_check_ignored = set()
+    ignored_cves = set()
+    with open(inc_file) as f:
+        lines = f.readlines()
+        for line in lines:
+            if line.strip():
+                # Ignore the comments from cve-extra-exclusions.inc
+                if not re.search("^#", line):
+                    cve_match = re.findall(r'CVE\-\d{4}\-\d+', line)
+                    for cve in cve_match:
+                        ignored_cves.add(cve)
+
+    for cve in cves:
+        if cve not in ignored_cves:
+            cve_check_ignored.add(cve)
+    ignored_cves_from_recipe = " ".join(cve_check_ignored)
+
+    return ignored_cves_from_recipe
Signature · 1 line
2.17.1
#1 lore raw

PATCH 2/3 cve-export.bbclass: Add a new class to get patched and ignored CVEs from the build

This class executes an anonymous function which sets the below variables
CVE_IGNORED = CVEs that are ignored in recipes CVE_PATCHED = CVEs that are fixed by applying patches

It does not consider CVEs that are ignored in poky/meta/conf/distro/include/cve-extra-exclusions.inc and only provide CVEs that are ignored in the recipe.

Default values are set for CVE_PRODUCT and CVE_VERSION to BPN and PV respectively.

Considered setting these values so that anyone can get below information about the CVE from the build.
CVE_PRODUCT
CVE_VERSION
CVE_IGNORED
CVE_PATCHED

---
 meta/classes/cve-export.bbclass | 37 +++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100644 meta/classes/cve-export.bbclass
meta/classes/cve-export.bbclass new file +37 -0
new file mode 100644
index 0000000000..5ed5760970
@@ -0,0 +1,37 @@
+# This class is used to get patched and ignored CVEs from the build
+#
+# To use this class inherit it in the local.conf file.
+#
+# It executes an anonymous function which sets below variables
+#
+# CVE_IGNORED = CVEs those are ignored in recipes
+# CVE_PATCHED = CVEs those are fixed by applying patches
+#
+# It does not consider all the CVEs that are ignored in
+# poky/meta/conf/distro/include/cve-extra-exclusions.inc
+# and only provide CVEs that are ignored in the recipe.
+#
+# The product name sets default to BPN and version sets default to
+# PV but it can be overriden per recipe, to get the value of
+# product and version use d.getVar()
+
+CVE_PRODUCT ??= "${BPN}"
+CVE_VERSION ??= "${PV}"
+CVE_CHECK_IGNORE ?= ""
+
+python __anonymous () {
+    import re
+    from oe.cve_check import get_patched_cves
+    from oe.cve_check import get_ignored_cves
+
+    # Check if cve-extra-exclusions.inc file is included or not
+    if re.search('cve-extra-exclusions.inc', d.getVar('BBINCLUDED')):
+        paths = d.getVar('PATH').split(':')
+        cves = d.getVar('CVE_CHECK_IGNORE').split()
+        ignored_cves = get_ignored_cves(paths, cves)
+    else:
+        ignored_cves = " ".join(d.getVar('CVE_CHECK_IGNORE'))
+
+    d.setVar('CVE_IGNORED', ignored_cves)
+    d.setVar('CVE_PATCHED', " ".join(get_patched_cves(d)))
+}
Signature · 1 line
2.17.1
#2 lore raw

PATCH 3/3 cve_export.py: Add new selftest for cve-export.bbclass

In reply to Akash Hadke

This test compares values of variables CVE_CHECK_IGNORE with CVE_IGNORED that is exported from cve-export.bbclass

---
 meta/lib/oeqa/selftest/cases/cve_export.py | 24 ++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/cve_export.py
meta/lib/oeqa/selftest/cases/cve_export.py new file +24 -0
new file mode 100644
index 0000000000..7c7fd07957
@@ -0,0 +1,24 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var
+from oe.cve_check import get_ignored_cves
+
+class CveExportTest (OESelftestTestCase):
+
+    def test_ignored_cves(self):
+        """
+        This test compares the values of variables CVE_CHECK_IGNORE with
+        CVE_IGNORED that is exported from cve-export.bbclass
+        """
+        target = 'unzip'
+        feature = 'INHERIT += "cve-export"\n'
+        feature += 'require conf/distro/include/cve-extra-exclusions.inc'
+        self.write_config(feature)
+
+        res = bitbake("%s -c fetch" % target, ignore_status=True)
+        self.assertEqual(res.status, 0, "\nCouldn't build.\nbitbake output %s" % res.output)
+
+        paths = get_bb_var('PATH', target).split(':')
+        cves = get_bb_var('CVE_CHECK_IGNORE', target).split()
+        final_cve = get_ignored_cves(paths, cves)
+        ignore_cves_from_exported_variable = get_bb_var('CVE_IGNORED', target)
+        self.assertEqual(final_cve, ignore_cves_from_exported_variable.strip(), "Ignored CVEs are not matching")
Signature · 1 line
2.17.1
#3 lore raw
In reply to Akash Hadke
Quoted 9 lines · akash hadke via lists.openembedded.org

Add new method get_ignored_cves in cve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc

While calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()

Hello Akash,
While looking into this patch set I'm wondering what is your use case. It seems to be to get a list
of ignored and patched CVEs. This is already available from the cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure we can come with a simple
post-processing script to do the same.

BTW Why do assume people always include meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.

Regards,
Marta

#4 lore raw

Hello Marta,

Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.

Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the CVE_CHECK_IGNORED list even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc ​

Best Regards,
Akash
________________________________
From: Marta Rybczynska <[email protected]>
Sent: 17 May 2022 14:42
To: Akash Hadke <[email protected]>
Cc: OE-core <[email protected]>; Ranjitsinh Rathod <[email protected]>; Akash Hadke <[email protected]>
Subject: Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

Caution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 9 lines · akash hadke via lists.openembedded.org

Add new method get_ignored_cves in cve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc

While calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()

Hello Akash,
While looking into this patch set I'm wondering what is your use case. It seems to be to get a list
of ignored and patched CVEs. This is already available from the cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure we can come with a simple
post-processing script to do the same.

BTW Why do assume people always include meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.

Regards,
Marta

#5 lore raw
Quoted 4 lines · Akash Hadke

Hello Marta,

Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.

Hello again Akash,
What you'd like to do is to see the difference in ignored and patched CVEs in buildhistory? Do I get it right?

Quoted 2 lines

Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the CVE_CHECK_IGNORED list even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc

I think I understand the idea. The point I'm making is that if someone does not include the cve-extra-exclusions.inc in their distro, the code will still use it and filter out CVEs they still see when doing cve-check.

Kind regards,
Marta

Quoted 41 lines

Best Regards,
Akash
________________________________
From: Marta Rybczynska <[email protected]>
Sent: 17 May 2022 14:42
To: Akash Hadke <[email protected]>
Cc: OE-core <[email protected]>; Ranjitsinh Rathod <[email protected]>; Akash Hadke <[email protected]>
Subject: Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

Caution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 9 lines · akash hadke via lists.openembedded.org

Add new method get_ignored_cves in cve_check.py
to get ignored CVEs from recipe by excluding distro-wide ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc

While calling this method use below code to get argument values paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()

Hello Akash,
While looking into this patch set I'm wondering what is your use case. It seems to be to get a list
of ignored and patched CVEs. This is already available from the cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure we can come with a simple
post-processing script to do the same.

BTW Why do assume people always include meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.

Regards,
Marta

#7 lore raw
In reply to Akash Hadke
Quoted 5 lines · akash hadke via lists.openembedded.org

Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.

I'd really prefer to have one cve handling class where we can configure it to get the data different people need rather than multiple difference cve classes which are going to confuse people. Could we have a way to disable NVD data from the cve-check class?

Cheers,

Richard

#8 lore raw

Hi Richard,

I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve-check it executes the cve_check task that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.

#9 lore raw
Quoted 7 lines · akash hadke via lists.openembedded.org

Hi Richard,

I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve- check it executes the cve_check task that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.

I guess the task dependencies are a bit of an issue there. There are probably ways to make those configurable, it would just take a little work.

I really do want to encourage us to work together on common cve tooling rather than having several partial implementations so I can't take this patch series.

Cheers,

Richard

#10 lore raw
Quoted 11 lines · Richard Purdie
Quoted 7 lines · akash hadke via lists.openembedded.org

Hi Richard,

I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve- check it executes the cve_check task that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.

Quoted 8 lines

I guess the task dependencies are a bit of an issue there. There are probably ways to make those configurable, it would just take a little work.

I really do want to encourage us to work together on common cve tooling rather than having several partial implementations so I can't take this patch series.

I plan to cut those dependencies and make different stages optional. In my case the first use
is to be able to run multiple cve-checks with the same database (guaranteed without updates)
or run cve-check with some given known database (for testing). We can add an option to make the actual check with the database optional.

Exporting the data to the buildhistory looks like a useful thing to do, too.

I can draft the split after YPS.

Kind regards,
Marta