Skip to contentSkip to conversation

cve-check-update-nvd2-native: Incremement DL_DIR database location

PATCH Richard Purdie 2 participants 8 messages Oct 11, 2024 – Oct 14, 2024 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 26 lines · Richard Purdie via lists.openembedded.org

We're seeing a lot of sqlite database corruption issues in our automated testing. It is unclear why this is happening. There were process imrpovements implemented in master and it is unclear if older releases are somehow making those changes ineffective or if the problem is elsewhere.

By changing the location in DL_DIR, we split the two sets of accesses to be separate and can isolate whether the master changes really did improve things or not. If successful, we may consider backporting those changes to the stable releases.

---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +0 -0
index 2d23d28c3e2..a7e568e3079 100644
@@ -34,7 +34,7 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000"
  # Number of attempts for each http query to nvd server before giving up

 CVE_DB_UPDATE_ATTEMPTS ?= "5"

meta/recipes-core/meta/cve-update-nvd2-native.bb +1 -1
-CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}"
+CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}"
Quoted 2 lines

 CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of failures on the new cluster, hopefully this will help isolate the problem.

Cheers,

Richard

#2 lore raw
Quoted 10 lines · Richard Purdie via lists.openembedded.org

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of failures on the new cluster, hopefully this will help isolate the problem.

The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.

Regards,
Marta

#3 lore raw
Quoted 17 lines · Marta Rybczynska
Quoted 11 lines · Richard Purdie via lists.openembedded.org

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of
failures on the new cluster, hopefully this will help isolate the problem.

Quoted 4 lines

The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.

https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio

Exception: sqlite3.DatabaseError: database disk image is malformed

Cheers,

Richard

#4 lore raw
Quoted 24 lines · Richard Purdie
Quoted 17 lines · Marta Rybczynska
Quoted 11 lines · Richard Purdie via lists.openembedded.org

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of
failures on the new cluster, hopefully this will help isolate the problem.

Quoted 4 lines

The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.

Quoted 5 lines

https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio

Exception: sqlite3.DatabaseError: database disk image is malformed

Managed to get the same error with the sqlite command line.

Marta

#5 lore raw
Quoted 26 lines · Marta Rybczynska
Quoted 23 lines · Richard Purdie
Quoted 21 lines · Marta Rybczynska
Quoted 11 lines · Richard Purdie via lists.openembedded.org

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of
failures on the new cluster, hopefully this will help isolate the problem.

The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.

Quoted 8 lines
Quoted 5 lines

https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio

Exception: sqlite3.DatabaseError: database disk image is malformed

Managed to get the same error with the sqlite command line.

I merged this patch as it would rule out a lot of things. The error recurred:

https://valkyrie.yoctoproject.org//#/builders/76/builds/235/steps/14/logs/stdio

This means that it happens with the access patterns in master alone. I'm also noticing there seems to be a trend for this to happen on alma9 (older centos systems?).

Running those tests manually on that worker passed the tests ok.

It has python 3.9.18 with sqlite3 not available as an external command.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14899

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw
Quoted 42 lines · Richard Purdie via lists.openembedded.org
Quoted 41 lines · Marta Rybczynska
Quoted 35 lines · Richard Purdie
Quoted 21 lines · Marta Rybczynska

On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via lists.openembedded.org <[email protected]>

Quoted 5 lines · wrote:

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

Quoted 9 lines

I'm not sure why this is happening but the older releases are all
accessing the DB over NFS which makes me nervous. We're seeing a
lot of
failures on the new cluster, hopefully this will help isolate the
problem.

Quoted 6 lines

The database isn't completely corrupted, it looks like it is partial,
however. What was the error
that you got in that build? Any or just strange results? I'm running
a verification vs a fresh database.

https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio

Exception: sqlite3.DatabaseError: database disk image is malformed

Quoted 4 lines
Quoted 1 line

Managed to get the same error with the sqlite command line.

I merged this patch as it would rule out a lot of things.

It looks like I didn't do this. I will do so, then we can rule out a few things.

Cheers,

Richard

#8 lore raw
Quoted 45 lines · Richard Purdie
Quoted 27 lines · Richard Purdie via lists.openembedded.org
Quoted 26 lines · Marta Rybczynska
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Marta Rybczynska

On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via lists.openembedded.org <[email protected]>

Quoted 14 lines · wrote:

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

I'm not sure why this is happening but the older releases are all
accessing the DB over NFS which makes me nervous. We're seeing a
lot of
failures on the new cluster, hopefully this will help isolate the
problem.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

The database isn't completely corrupted, it looks like it is partial,
however. What was the error
that you got in that build? Any or just strange results? I'm running
a verification vs a fresh database.

https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Exception: sqlite3.DatabaseError: database disk image is malformed

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Managed to get the same error with the sqlite command line.

I merged this patch as it would rule out a lot of things.

It looks like I didn't do this. I will do so, then we can rule out a few things.

I've analysed the corrupted file a bit. This is somewhat complex as I do not have a "golden" copy with the
exact same content. However, what I can see:
1. This is not a partial download, quite new CVEs from 2024 are there (as from other years)
2. Damaged records come from various years (a theory to check: if they all have been recently modified)
3. Only part of the database is broken and both NVD and PRODUCTS tables. You can read various CVEs depending on how you format your SELECT (getting all cve_ids works fine, for example)

If you suspect there are jobs accessing the file that shouldn't, what about installing inotify hooks on the file?

As I've never seen such a corruption on my side, I could imagine either an external job, or an effect of a re-download of the database during tests (but this should use locks).

Regards,
Marta

#9 lore raw
Quoted 11 lines · Marta Rybczynska

I've analysed the corrupted file a bit. This is somewhat complex as I do not have a "golden" copy with the
exact same content. However, what I can see:
1.  This is not a partial download, quite new CVEs from 2024 are there (as from other years)
2. Damaged records come from various years (a theory to check: if they all have been recently modified)
3. Only part of the database is broken and both NVD and PRODUCTS tables. You can read various CVEs depending on how you format your SELECT (getting all cve_ids works fine, for example)

Quoted 6 lines

If you suspect there are jobs accessing the file that shouldn't, what about installing inotify hooks on the file?

As I've never seen such a corruption on my side, I could imagine either an external job, or an effect of a re-download of the database during tests (but this should use locks).

That is all useful data, thanks.

I can add that on the autobuilders, the file is on NFS since DL_DIR is. Any worker can therefore in theory access it and inotify could be tricky to setup and monitor correctly in that setup.

In your tests, do you run multiple releases against the database? I'm wondering if older sqlite versions in one of the older releases may be triggering this somehow it if tries to update the database?

Cheers,

Richard