Skip to contentSkip to conversation

severe issue in CVE checker

DISCUSSION Alberto Pianon 5 participants 6 messages Oct 12, 2022 – Oct 13, 2022

lore
Share & tools
lore
#0 lore raw

All, Marta, Richard,

while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this: https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.

The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)

I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:

- in a build where cve-check found a vulnerability for acl:

    $ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
    sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
    CVE-2009-4411
    sqlite>

- in another build where cve-check did not found any vulnerability for the very same version of acl:

    $ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
    sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
    sqlite>

so I listed both CVE db files in those two builds and this is what I got:

    $ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db 
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
    -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56 
build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
    -rw-r--r-- 1 ubuntu ubuntu     28672 Oct 11 10:00 
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db

The two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.

I checked the log for the second build, and I found this:

WARNING: cve-update-db-native-1.0-r0 do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: task do_fetch: Succeeded

Fetch fails, but do_fetch task succeeds.

So I looked into the recipe's code, and I found this: https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.

IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native do_fecth should fail, and it turn all do_cve_check tasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless

Regards,

Alberto

#1 lore raw

Thanks for the information, can you send a patch?

Alex

Quoted 80 lines · Alberto Pianon

All, Marta, Richard,

while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.

The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)

I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:

- in a build where cve-check found a vulnerability for acl:

     $ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
     sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
     CVE-2009-4411
     sqlite>

- in another build where cve-check did not found any vulnerability for the very same version of acl:

     $ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
     sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
     sqlite>

so I listed both CVE db files in those two builds and this is what I got:

     $ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
     -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56
build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
     -rw-r--r-- 1 ubuntu ubuntu     28672 Oct 11 10:00
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db

The two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.

I checked the log for the second build, and I found this:

WARNING: cve-update-db-native-1.0-r0 do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: task do_fetch: Succeeded

Fetch fails, but do_fetch task succeeds.

So I looked into the recipe's code, and I found this:
https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.

IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native do_fecth should fail, and it turn all do_cve_check tasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless

Regards,

Alberto

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171653): https://lists.openembedded.org/g/openembedded-core/message/171653
Mute This Topic: https://lists.openembedded.org/mt/94276393/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

I'll be looking into how to fix it. My current idea is to make the download a transaction, so do not update the database until we're sure the download is complete. Plus a warning that we have had an issue, I think.

In a next step we can retry a number of times.

Regards
Marta

Quoted 93 lines · On Wed, 12 Oct 2022, 16:50 Alexander Kanavin, <[email protected]> wrote:

Thanks for the information, can you send a patch?

Alex

Quoted 7 lines · Alberto Pianon

All, Marta, Richard,

while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this:

https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

Quoted 41 lines

It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.

The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)

I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:

- in a build where cve-check found a vulnerability for acl:

     $ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
     sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
     CVE-2009-4411
     sqlite>

- in another build where cve-check did not found any vulnerability for the very same version of acl:

     $ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
     sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
     sqlite>

so I listed both CVE db files in those two builds and this is what I got:

     $ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
     -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56
build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
     -rw-r--r-- 1 ubuntu ubuntu     28672 Oct 11 10:00
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db

The two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.

I checked the log for the second build, and I found this:

Quoted 8 lines

WARNING: cve-update-db-native-1.0-r0 do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: task do_fetch: Succeeded

Fetch fails, but do_fetch task succeeds.

So I looked into the recipe's code, and I found this:

https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

Quoted 12 lines

It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.

IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native do_fecth should fail, and it turn all do_cve_check tasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless

Quoted 6 lines

Regards,

Alberto

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#171657):
https://lists.openembedded.org/g/openembedded-core/message/171657
Mute This Topic: https://lists.openembedded.org/mt/94276393/5827677
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw

Hi,

Can the downloads be turned into normal do_fetch() SRC_URI downloads including caches in yocto infrastructure?

There are many issues around CVE checking that it's really a process where a lot of details and uncertainties just need to be accepted. It's far from a perfect and users just need to accept this.

Cheers,

-Mikko

Quoted 107 lines · Marta Rybczynska

I'll be looking into how to fix it. My current idea is to make the download a transaction, so do not update the database until we're sure the download is complete. Plus a warning that we have had an issue, I think.

In a next step we can retry a number of times.

Regards
Marta

Quoted 58 lines · On Wed, 12 Oct 2022, 16:50 Alexander Kanavin, <[email protected]> wrote:

Thanks for the information, can you send a patch?

Alex

Quoted 7 lines · Alberto Pianon

All, Marta, Richard,

while implementing stats aggregation for CVE metadata in the Oniro project, I encountered a severe issue in Yocto's CVE checker, apparently due to this:

https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

Quoted 45 lines

It appears that when cve-update-db-native fails to fetch some years of NIST CVE db, it just issues a warning but goes on anyway.

The result is that in some builds, randomly (depending on NIST webserver timeouts or other connection problems), CVE db is not complete, so the CVE check returns false negatives (i.e. no vulnerabilities found in some components even if such vulnerabilities do exist)

I ran into such problem because in Oniro we need aggregate data from different builds for a large target matrix; I added a check to check that CVE metadata for each component are the same in all builds, and it failed, so I tried to figure out the cause and I found this:

- in a build where cve-check found a vulnerability for acl:

     $ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
     sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
     CVE-2009-4411
     sqlite>

- in another build where cve-check did not found any vulnerability for the very same version of acl:

     $ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
     sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl";
     sqlite>

so I listed both CVE db files in those two builds and this is what I got:

     $ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db
     -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56
build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db
     -rw-r--r-- 1 ubuntu ubuntu     28672 Oct 11 10:00
build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db

The two CVE db files were fetched just about 1h apart, but the second file is apparently incomplete.

I checked the log for the second build, and I found this:

WARNING: cve-update-db-native-1.0-r0 do_fetch: Failed to fetch CVE
data ([Errno 99] Cannot assign requested address)
NOTE: recipe cve-update-db-native-1.0-r0: task do_fetch: Succeeded

Quoted 27 lines
Quoted 4 lines

Fetch fails, but do_fetch task succeeds.

So I looked into the recipe's code, and I found this:

https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81

Quoted 18 lines

It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.

IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native do_fecth should fail, and it turn all do_cve_check tasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless

Regards,

Alberto

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171658): https://lists.openembedded.org/g/openembedded-core/message/171658
Mute This Topic: https://lists.openembedded.org/mt/94276393/7159507
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw
In reply to Alberto Pianon
Quoted 4 lines · Alberto Pianon via lists.openembedded.org

It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.

IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native do_fecth should fail, and it turn all do_cve_check tasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless

Yes, that’s a bug.

Please do file a bug (bugzilla.yoctoproject.org), and ideally send a patch if you can.

Ross

#5 lore raw

Reported as https://bugzilla.yoctoproject.org/show_bug.cgi?id=14929

Quoted 22 lines · Ross Burton
Quoted 4 lines · Alberto Pianon via lists.openembedded.org

It iterates over NIST CVE db years, but if some year fail to download, it goes on anyway, and it still merges the successful downloads into nvdcve_1.1.db, without returning error.

IMHO this is a severe issue because it may silently lead to false negatives in the CVE check. If some downloads fail due to timeouts or other connection problems, cve-check should retry a number of times, and if any download still fails, cve-update-db-native do_fecth should fail, and it turn all do_cve_check tasks should fail, since doing a CVE check against a corrupted/incomplete CVE database is clearly useless

Yes, that’s a bug.

Please do file a bug (bugzilla.yoctoproject.org), and ideally send a patch if you can.

Ross

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171668): https://lists.openembedded.org/g/openembedded-core/message/171668
Mute This Topic: https://lists.openembedded.org/mt/94276393/5827677
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-