Latest AB-INT unexplained mystery failure
A message that is not in the archive (sent off-list or not yet received): [email protected]
It was suggested I should write down a list of other issues we have on the autobuilder right now. Some of these are related to the transition to new infrastructure, some are previously seen issues occurring more frequently, some are entirely new. I appreciate these should all have bugs. I'm copying various people including those helping with SWAT so we can transition those we don't get solved into bugzilla.
a) insane do_package_qa intermittent failures
---------------------------------------------
Two levels of probems. The cachedpath code returns False instead of exceptions so there was unexpected API changes. The recent insane changes also stopped handling DEBIAN files correctly leading to races. This is perhaps the best backtrace we have:
https://valkyrie.yoctoproject.org//#/builders/43/builds/246
Who/Plan: RP has a partial revert for the first part to fix builds. Ross is aware and working on the second.
b) docs build failing in missing inkscape
-----------------------------------------
e.g. https://valkyrie.yoctoproject.org//#/builders/34/builds/9
We could install inkscape on all the workers but I'm reluctant to do so as it starts to add large dependency chains potentially and reduces our host dependency checking. We could to with other tools to generate pdf and epub docs during release and docs builds too.
Proposal is to add a new buildtools-docs target to the AB which can pull in the meta-oe layer and build a more fully features docs buildtools. This would also be useful for the screenshot QA imagemagick tests we want to add.
Who/Plan: TBD
c) source mirroring failing on AB
---------------------------------
https://valkyrie.yoctoproject.org//#/builders/82/builds/15
https://valkyrie.yoctoproject.org//#/builders/82/builds/16
Sources are still mirrored off typhoon but we've stopped that cluster and are running off valkyrie. This could mean sources don't appear "fast" in the mirror until that mirroring moves to the new NAS.
Who/Plan: Michael to transition mirroring to run of valkyrie NAS
d) CDN artefacts are failing
Same deal as sources, we need to move the mirroring to be based off valkyrie's NAS.
Who/Plan: Michael to transition CDN to run of valkyrie NAS
e) bitbake server timeout issues
--------------------------------
https://valkyrie.yoctoproject.org//#/builders/48/builds/185/steps/14/logs/stdio
I've saved
https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/
- the json logs are something we've not had before for that
- the key message in cookerdaemon is "Idle loop didn't finish queued
commands after 30s, exiting."
- that message is there twice, it failed twice
- can we decode the json logs and get timestamps
- looks like it happens in particular in the siggen code?
Who/Plan: TBD
f) CVE database corruption
--------------------------
See list discussion:
https://lists.openembedded.org/g/openembedded-core/message/205715
Has a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14899
I'm at a loss on this one. Started to wonder if an rsync job is trampling the file. Check with Michael.
Who/Plan: Ask Michael about rsync job
g) Toaster test issues
----------------------
Toaster testing is more intermittent on the new faster workers. Have some patches in progress but is has highlighted issues with the tests.
Help in changing "assertTrue(X in Y)" to "assertIn(X, Y)" in the toaster tests would be welcome. I've made a few patches, more are needed.
Help in being able to delete a project from the database before starting tests would also be useful. Having trouble doing it in the tests themselves due to database locking and can't work out the way to call the rest DELETE API from selenium yet.
My plan is to length the timeouts and drop all the sleep/poll calls, clean up the timeouts. This will make the tests much faster too. This does mean adding "wait for alert to display" code. Have tried doing this but tests need fixing.
Who/Plan: RP has ideas but would welcome help. RP needs to send "debugging toaster tests email with tips".
h) weird networking issues causing test failures
See original email in this thread:
https://lists.openembedded.org/g/openembedded-core/message/205723
RP is at a loss.
Who/Plan: TBD
i) rust toolchain test failures mips/ppc
e.g.: https://valkyrie.yoctoproject.org//#/builders/21/builds/226
Who/Plan: Need to write email asking for mips/ppc help. Can Adrian/Chuck help?
j) SPDX build warnings
RP hasn't merged Joshua's patch. Need to include in next testing run
Who/Plan: RP to test and merge patch
k) ssh test still causing failures
RP's fix looks to be incorrect, changed the wrong number. Correct fix queued in master-next
Who/Plan: RP to test and merge patch
I'm sure there are more but I've put the ones I have in my head down for now. I'm pretty sure people find fixing one issue painful, trying to keep track of this many in my head is bad enough without trying to fix them! Help on any of these is welcome.
Cheers,
Richard
Quoted 6 lines · Richard Purdie
It was suggested I should write down a list of other issues we have on the autobuilder right now. Some of these are related to the transition to new infrastructure, some are previously seen issues occurring more frequently, some are entirely new. I appreciate these should all have bugs. I'm copying various people including those helping with SWAT so we can transition those we don't get solved into bugzilla.
I thought I’d send an update to this mail:
Quoted 2 lines
a) insane
do_package_qaintermittent failures
——————————————————————
This has now been fixed.
Quoted 2 lines
b) docs build failing in missing inkscape
-----------------------------------------
This was the AB building more than we expected, and is now resolved.
Quoted 2 lines
c) source mirroring failing on AB
————————————————
This is still in progress.
Quoted 1 line
d) CDN artefacts are failing
This is still in progress.
Quoted 2 lines
e) bitbake server timeout issues
--------------------------------
Still TDB.
Quoted 2 lines
f) CVE database corruption
--------------------------
Prognosis is good, appears to be fixed in master.
Quoted 2 lines
g) Toaster test issues
----------------------
RP has discovered his inner web developer and is making good progress on these.
Quoted 1 line
i) rust toolchain test failures mips/ppc
These are proving tricky to reproduce on demand. As the platforms are not supported upstream I think we should skip the rust tests entirely.
Quoted 1 line
j) SPDX build warnings
Appears to be resolved with Josh’s patch.
Quoted 1 line
k) ssh test still causing failures
Fixed.
Cheers,
Ross
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Richard Purdie via lists.openembedded.org
c) source mirroring failing on AB
---------------------------------https://valkyrie.yoctoproject.org//#/builders/82/builds/15
https://valkyrie.yoctoproject.org//#/builders/82/builds/16Sources are still mirrored off typhoon but we've stopped that cluster and are running off valkyrie. This could mean sources don't appear "fast" in the mirror until that mirroring moves to the new NAS.
Who/Plan: Michael to transition mirroring to run of valkyrie NAS
I talked with Michael and there is no automated rsync. Software has been updated since the last one and that is why things are not updating. As the mirroring is transitioned this issue will go away.
Quoted 7 lines
f) CVE database corruption
--------------------------See list discussion:
https://lists.openembedded.org/g/openembedded-core/message/205715
Quoted 8 lines
Has a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14899
I'm at a loss on this one. Started to wonder if an rsync job is trampling the file. Check with Michael.
Who/Plan: Ask Michael about rsync job
There is no issue with any rsync job. I did realise I'd not merged the patch I thought I had though. I will do so and see if that isolates the issue.
Cheers,
Richard
Quoted 9 lines · Richard Purdie via lists.openembedded.org
e) bitbake server timeout issues
--------------------------------https://valkyrie.yoctoproject.org//#/builders/48/builds/185/steps/14/logs/stdio
I've saved
https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/
Quoted 9 lines
- the json logs are something we've not had before for that
- the key message in cookerdaemon is "Idle loop didn't finish queued commands after 30s, exiting."
- that message is there twice, it failed twice
- can we decode the json logs and get timestamps
- looks like it happens in particular in the siggen code?Who/Plan: TBD
Happened again:
https://valkyrie.yoctoproject.org/#/builders/48/builds/192
fedora41 again. Logs into:
https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/2
Cheers,
Richard
l) libz reproducibiity failure
https://valkyrie.yoctoproject.org/#/builders/37/builds/265
https://valkyrie.yoctoproject.org/#/builders/37/builds/266
https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241014-68a7dqot/packages/diff-html/
This looks familiar somehow. Looks like a 'configure' issue of some kind (it isn't autoconf).
Who/Plan: TBD
Quoted 18 lines · Richard Purdie via lists.openembedded.org
It was suggested I should write down a list of other issues we have on
the autobuilder right now. Some of these are related to the transition
to new infrastructure, some are previously seen issues occurring more frequently, some are entirely new. I appreciate these should all have bugs. I'm copying various people including those helping with SWAT so we can transition those we don't get solved into bugzilla.a) insane
do_package_qaintermittent failures
---------------------------------------------Two levels of probems. The cachedpath code returns False instead of exceptions so there was unexpected API changes. The recent insane changes also stopped handling DEBIAN files correctly leading to races.
This is perhaps the best backtrace we have:
Quoted 4 lines
https://valkyrie.yoctoproject.org//#/builders/43/builds/246
Who/Plan: RP has a partial revert for the first part to fix builds. Ross is aware and working on the second.
I tested Ross' patch for this and unfortunate it caused a new problem to appear.
I accidentally triggered two a-full builds at once, there was also a scarthgap build running at the time.
The patch from Ross caused all the do_package_qa tasks to rerun at once in parallel which caused writes to the hash equivalence server. This caused widespread timeout errors and took out all the builds including the scarthgap one in parallel.
https://valkyrie.yoctoproject.org/#/builders/4/builds/257
https://valkyrie.yoctoproject.org/#/builders/2/builds/253
https://valkyrie.yoctoproject.org/#/builders/89/builds/236/steps/16/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/40/builds/242/steps/11/logs/stdio
and many more.
Joshua: Is there a performance issue with many writes in parallel to hashequiv?
The stats info off the server was:
"requests": {
"average": 0.042532518429623176,
"max_time": 3.0534732329833787,
"num": 73325629,
"stdev": 0.06959859907598737,
"total_time": 3118723.6668062117
}Cheers,
Richard
Quoted 30 lines · Richard Purdie via lists.openembedded.org
I've spent quite a bit of time trying to reproduce/debug this:
https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_apt: FAILED (218.02s) 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
on debian11-vk-1.
I've a successful test and a failed test testimage output for comparison:
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465along with qemu serial output:
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106I've tried rerunning that exact series of tests on that worker with no
success reproducing the issue. The image was overwritten by a later test so we can't retest the exact image. I've checked the journal on that system and there isn't anything related around the time this happened. The two failures were on different network interfaces and both interfaces work on later tests.
Quoted 1 line
I'm open to ideas...
https://valkyrie.yoctoproject.org//#/builders/35/builds/229/steps/14/logs/stdio
debian11-vk-2 this time. A different runtime test, same kind of network ing glitch.
Does debian11 have some kind of network management that is interfering?
Cheers,
Richard
Quoted 22 lines · Richard Purdie via lists.openembedded.org
I've spent quite a bit of time trying to reproduce/debug this:
https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_apt: FAILED (218.02s) 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
on debian11-vk-1.
I've a successful test and a failed test testimage output for comparison:
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465along with qemu serial output:
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106
Quoted 7 lines
I've tried rerunning that exact series of tests on that worker with no
success reproducing the issue. The image was overwritten by a later test so we can't retest the exact image. I've checked the journal on that system and there isn't anything related around the time this happened. The two failures were on different network interfaces and both interfaces work on later tests.
This is happening on most builds on debian11 for my test branch. If I add new changes, the issue doesn't happen so it looks to be timing related and needs a populated sstate cache.
I put auditing into runqemu's tap locks codepaths and the devices are being locked/released correctly, there is no duplicate device usage.
I also went through the non-tap/tun codepaths and I can't spot any issues, we do use slirp in some tests.
Since we have a trigger point where we know it is failing (the runtime ping and ssh tests), I added a os.system("ps awx") into it. That gave me a process dump of what was running when this happens.
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3603268
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3902903
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.4042146
From those we can conclude that there is only one qemu-system-* running using the appropriate interface. I couldn't spot any other particularly untoward processes running. That tells us a lot of things it isn't I guess.
In addition to the ps, I've now added a netstat -tunape to see if we can see something holding a network connection open. I'm wondering about our httpserver used in some tests, or debuginfod, or something. That build is ongoing.
Quoted 1 line
I'm open to ideas...
I still am, this one is proving to be a pain...
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 59 lines · Richard Purdie via lists.openembedded.org
Quoted 22 lines · Richard Purdie via lists.openembedded.org
I've spent quite a bit of time trying to reproduce/debug this:
https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_apt: FAILED (218.02s) 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
on debian11-vk-1.
I've a successful test and a failed test testimage output for comparison:
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465along with qemu serial output:
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106Quoted 8 lines
I've tried rerunning that exact series of tests on that worker with no
success reproducing the issue. The image was overwritten by a later test so we can't retest the exact image. I've checked the journal on
that system and there isn't anything related around the time this happened. The two failures were on different network interfaces and both interfaces work on later tests.This is happening on most builds on debian11 for my test branch. If I add new changes, the issue doesn't happen so it looks to be timing related and needs a populated sstate cache.
I put auditing into runqemu's tap locks codepaths and the devices are being locked/released correctly, there is no duplicate device usage.
I also went through the non-tap/tun codepaths and I can't spot any issues, we do use slirp in some tests.
Since we have a trigger point where we know it is failing (the runtime
ping and ssh tests), I added aos.system("ps awx")into it. That gave me a process dump of what was running when this happens.https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3603268
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3902903
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.4042146From those we can conclude that there is only one qemu-system-* running
using the appropriate interface. I couldn't spot any other particularly
untoward processes running. That tells us a lot of things it isn't I guess.
Quoted 4 lines
In addition to the ps, I've now added a netstat -tunape to see if we can see something holding a network connection open. I'm wondering about our httpserver used in some tests, or debuginfod, or something. That build is ongoing.
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/5/log.do_testimage.123265
This has netstat from the host but also uses the serial connection to run ping on the host/server IPs and ifconfig.
This tells is the interfaces have the addresses we expect on both sides and that packets are tx'd but not rx'd on the other side.
Creative ideas on further debugging welcome, I'm not sure where from here. I think it narrows down the issue to the host kernel or qemu and these qemu binaries are working on the other builders. The host is a 5.10.223 kernel:
$ cat /proc/version Linux version 5.10.0-32-amd64 ([email protected]) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)
Cheers,
Richard
Quoted 7 lines · Richard Purdie via lists.openembedded.org
Creative ideas on further debugging welcome, I'm not sure where from here. I think it narrows down the issue to the host kernel or qemu and these qemu binaries are working on the other builders. The host is a 5.10.223 kernel:
$ cat /proc/version Linux version 5.10.0-32-amd64 ([email protected]) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)
There's an option to use 6.1 kernel on bullseye:
https://packages.debian.org/bullseye-backports/linux-image-amd64
so I'd give that a shot, if only to confirm (or refute) it's definitely 5.10 on the host.
Alex
Quoted 17 lines · Alexander Kanavin
Quoted 8 lines · Richard Purdie via lists.openembedded.org
Creative ideas on further debugging welcome, I'm not sure where from
here. I think it narrows down the issue to the host kernel or qemu and
these qemu binaries are working on the other builders. The host is a
5.10.223 kernel:Quoted 4 lines
$ cat /proc/version Linux version 5.10.0-32-amd64 ([email protected]) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)
Quoted 5 lines
There's an option to use 6.1 kernel on bullseye:
https://packages.debian.org/bullseye-backports/linux-image-amd64so I'd give that a shot, if only to confirm (or refute) it's definitely 5.10 on the host.
Michael was able to switch debian11-vk-1 to the 6.1 kernel (thanks!) and I've run 5 builds successfully on it, the single build I tried on debian11-vk3 at the same time failed.
I'm therefore willing to say this is some bug in the 5.10 kernel, switch our other two workers to 6.1 and call things "resolved".
Yes, ideally we'd track it down and get it fixed properly but we're not really setup to debug Debian bugs in old releases and we have other more pressing problems.
Cheers,
Richard
Quoted 24 lines · Richard Purdie
Quoted 17 lines · Alexander Kanavin
Quoted 8 lines · Richard Purdie via lists.openembedded.org
Creative ideas on further debugging welcome, I'm not sure where from
here. I think it narrows down the issue to the host kernel or qemu and
these qemu binaries are working on the other builders. The host is a
5.10.223 kernel:Quoted 4 lines
$ cat /proc/version Linux version 5.10.0-32-amd64 ([email protected]) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)Quoted 5 lines
There's an option to use 6.1 kernel on bullseye:
https://packages.debian.org/bullseye-backports/linux-image-amd64so I'd give that a shot, if only to confirm (or refute) it's definitely 5.10 on the host.
Quoted 8 lines
Michael was able to switch debian11-vk-1 to the 6.1 kernel (thanks!) and I've run 5 builds successfully on it, the single build I tried on debian11-vk3 at the same time failed.
I'm therefore willing to say this is some bug in the 5.10 kernel, switch our other two workers to 6.1 and call things "resolved".
All three Debian 11 Bullseye workers are set up with 6.1.0 kernels now.
Quoted 8 lines
Yes, ideally we'd track it down and get it fixed properly but we're not really setup to debug Debian bugs in old releases and we have other more pressing problems.
Cheers,
Richard
Signature · 3 lines
Michael Halstead Linux Foundation / Yocto Project Systems Operations Engineer
Quoted 28 lines · Richard Purdie via lists.openembedded.org
Quoted 17 lines · Alexander Kanavin
Quoted 8 lines · Richard Purdie via lists.openembedded.org
Creative ideas on further debugging welcome, I'm not sure where from
here. I think it narrows down the issue to the host kernel or qemu and
these qemu binaries are working on the other builders. The host is a
5.10.223 kernel:Quoted 4 lines
$ cat /proc/version Linux version 5.10.0-32-amd64 ([email protected]) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)Quoted 5 lines
There's an option to use 6.1 kernel on bullseye:
https://packages.debian.org/bullseye-backports/linux-image-amd64so I'd give that a shot, if only to confirm (or refute) it's definitely 5.10 on the host.
Michael was able to switch debian11-vk-1 to the 6.1 kernel (thanks!) and I've run 5 builds successfully on it, the single build I tried on debian11-vk3 at the same time failed.
Quoted 6 lines
I'm therefore willing to say this is some bug in the 5.10 kernel, switch our other two workers to 6.1 and call things "resolved".
Yes, ideally we'd track it down and get it fixed properly but we're not really setup to debug Debian bugs in old releases and we have other more pressing problems.
I agree with this. I was wondering if there is some aspects that Debian community can be made aware of the problem ?
Quoted 17 lines
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#206146):
https://lists.openembedded.org/g/openembedded-core/message/206146
Mute This Topic: https://lists.openembedded.org/mt/108984574/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 6 lines · Khem Raj
Quoted 3 lines
Yes, ideally we'd track it down and get it fixed properly but we're not really setup to debug Debian bugs in old releases and we have other more pressing problems.
I agree with this. I was wondering if there is some aspects that Debian community can be made aware of the problem ?
If we can't make it easy for them to reproduce (and we can't), then reporting is not likely to achieve much. WONTFIX WORKSFORME and all that.
Alex
Quoted 9 lines · Alexander Kanavin
Quoted 7 lines · Khem Raj
Quoted 5 lines
Yes, ideally we'd track it down and get it fixed properly but we're not
really setup to debug Debian bugs in old releases and we have other
more pressing problems.
Quoted 6 lines
Quoted 2 lines
I agree with this. I was wondering if there is some aspects that Debian community can be made aware of the problem ?
If we can't make it easy for them to reproduce (and we can't), then reporting is not likely to achieve much. WONTFIX WORKSFORME and all that.
It is also in Debian 11 (old stable), not Debian 12 so it is getting fairly old now. I suspect there will be less interest in the older release.
Cheers,
Richard