severe issue in CVE checker
All, Marta, Richard,
while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this: https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81
It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.
The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)
I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:
- in a build where cve-check found a vulnerability for acl:
$ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
CVE-2009-4411
sqlite>- in another build where cve-check did not found any vulnerability for the very same version of acl:
$ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
sqlite>so I listed both CVE db files in those two builds and this is what I got:
$ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
-rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56
build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
-rw-r--r-- 1 ubuntu ubuntu 28672 Oct 11 10:00
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.dbThe two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.
I checked the log for the second build, and I found this:
WARNING: cve-update-db-native-1.0-r0 do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: task do_fetch: Succeeded
Fetch fails, but do_fetch task succeeds.
So I looked into the recipe's code, and I found this: https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81
It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.
IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native do_fecth should fail, and it turn all do_cve_check tasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless
Regards,
Alberto
Thanks for the information, can you send a patch?
Alex
Quoted 80 lines · Alberto Pianon
All, Marta, Richard,
while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.
The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)
I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:
- in a build where cve-check found a vulnerability for acl:
$ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; CVE-2009-4411 sqlite>- in another build where cve-check did not found any vulnerability for the very same version of acl:
$ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; sqlite>so I listed both CVE db files in those two builds and this is what I got:
$ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db -rw-r--r-- 1 ubuntu ubuntu 28672 Oct 11 10:00 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.dbThe two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.
I checked the log for the second build, and I found this:
WARNING: cve-update-db-native-1.0-r0
do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: taskdo_fetch: SucceededFetch fails, but
do_fetchtask succeeds.So I looked into the recipe's code, and I found this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.
IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native
do_fecthshould fail, and it turn alldo_cve_checktasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly uselessRegards,
Alberto
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171653): https://lists.openembedded.org/g/openembedded-core/message/171653
Mute This Topic: https://lists.openembedded.org/mt/94276393/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
I'll be looking into how to fix it. My current idea is to make the download a transaction, so do not update the database until we're sure the download is complete. Plus a warning that we have had an issue, I think.
In a next step we can retry a number of times.
Regards
Marta
Quoted 93 lines · On Wed, 12 Oct 2022, 16:50 Alexander Kanavin, <[email protected]> wrote:
Thanks for the information, can you send a patch?
Alex
Quoted 7 lines · Alberto Pianon
All, Marta, Richard,
while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81
Quoted 41 lines
It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.
The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)
I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:
- in a build where cve-check found a vulnerability for acl:
$ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; CVE-2009-4411 sqlite>- in another build where cve-check did not found any vulnerability for the very same version of acl:
$ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; sqlite>so I listed both CVE db files in those two builds and this is what I got:
$ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db -rw-r--r-- 1 ubuntu ubuntu 28672 Oct 11 10:00 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.dbThe two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.
I checked the log for the second build, and I found this:
Quoted 8 lines
WARNING: cve-update-db-native-1.0-r0
do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: taskdo_fetch: SucceededFetch fails, but
do_fetchtask succeeds.So I looked into the recipe's code, and I found this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81
Quoted 12 lines
It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.
IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native
do_fecthshould fail, and it turn alldo_cve_checktasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly uselessQuoted 6 lines
Regards,
Alberto
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#171657):
https://lists.openembedded.org/g/openembedded-core/message/171657
Mute This Topic: https://lists.openembedded.org/mt/94276393/5827677
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hi,
Can the downloads be turned into normal do_fetch() SRC_URI downloads including caches in yocto infrastructure?
There are many issues around CVE checking that it's really a process where a lot of details and uncertainties just need to be accepted. It's far from a perfect and users just need to accept this.
Cheers,
-Mikko
Quoted 107 lines · Marta Rybczynska
I'll be looking into how to fix it. My current idea is to make the download a transaction, so do not update the database until we're sure the download is complete. Plus a warning that we have had an issue, I think.
In a next step we can retry a number of times.
Regards
MartaQuoted 58 lines · On Wed, 12 Oct 2022, 16:50 Alexander Kanavin, <[email protected]> wrote:
Thanks for the information, can you send a patch?
Alex
Quoted 7 lines · Alberto Pianon
All, Marta, Richard,
while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81
Quoted 45 lines
It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.
The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)
I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:
- in a build where cve-check found a vulnerability for acl:
$ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; CVE-2009-4411 sqlite>- in another build where cve-check did not found any vulnerability for the very same version of acl:
$ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; sqlite>so I listed both CVE db files in those two builds and this is what I got:
$ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db -rw-r--r-- 1 ubuntu ubuntu 28672 Oct 11 10:00 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.dbThe two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.
I checked the log for the second build, and I found this:
WARNING: cve-update-db-native-1.0-r0
do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: taskdo_fetch: SucceededQuoted 27 lines
Quoted 4 lines
Fetch fails, but
do_fetchtask succeeds.So I looked into the recipe's code, and I found this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81
Quoted 18 lines
It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.
IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native
do_fecthshould fail, and it turn alldo_cve_checktasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly uselessRegards,
Alberto
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171658): https://lists.openembedded.org/g/openembedded-core/message/171658
Mute This Topic: https://lists.openembedded.org/mt/94276393/7159507
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 4 lines · Alberto Pianon via lists.openembedded.org
It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.
IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native
do_fecthshould fail, and it turn alldo_cve_checktasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless
Yes, that’s a bug.
Please do file a bug (bugzilla.yoctoproject.org), and ideally send a patch if you can.
Ross
Reported as https://bugzilla.yoctoproject.org/show_bug.cgi?id=14929
Quoted 22 lines · Ross Burton
Quoted 4 lines · Alberto Pianon via lists.openembedded.org
It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.
IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native
do_fecthshould fail, and it turn alldo_cve_checktasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly uselessYes, that’s a bug.
Please do file a bug (bugzilla.yoctoproject.org), and ideally send a patch if you can.
Ross
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171668): https://lists.openembedded.org/g/openembedded-core/message/171668
Mute This Topic: https://lists.openembedded.org/mt/94276393/5827677
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-