Fix for the APIC hangs in qemux86-64
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which fixes the issue but backporting it to the autobuilder's host kernels will be a pain.
As a workaround I'd like to enable x2apic on qemux86-64. Since that works via MSRs it avoids the problem.
Would it be a big issue to set CONFIG_X86_X2APIC=y for qemux86-64?
Cheers,
Richard
Quoted 9 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which fixes the issue but backporting it to the autobuilder's host kernels will be a pain.
Quoted 4 lines
As a workaround I'd like to enable x2apic on qemux86-64. Since that works via MSRs it avoids the problem.
Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-64?
Nope. No big deal at all. I've used the same config i the past when doing work with rt-KVM, so I know it well.
I'll prep a commit, and send it along as part of a rebased set of kernel patches.
Cheers,
Bruce
Quoted 7 lines
Cheers,
Richard
Quoted 16 lines · Bruce Ashfield
Quoted 15 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which fixes
the issue but backporting it to the autobuilder's host kernels will be
a pain.As a workaround I'd like to enable x2apic on qemux86-64. Since that works via MSRs it avoids the problem.
Quoted 6 lines
Quoted 1 line
Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-64?Nope. No big deal at all. I've used the same config i the past when doing work with rt-KVM, so I know it well.
I'll prep a commit, and send it along as part of a rebased set of kernel patches.
Thanks, much appreciated.
Did we get to the bottom of the 4.12 selftest failures?
We may also need to backport the defconfig change back to the older releases so we don't get the hangs in those builds either...
Cheers,
Richard
Quoted 21 lines · Richard Purdie
Quoted 20 lines · Bruce Ashfield
Quoted 12 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which fixes
the issue but backporting it to the autobuilder's host kernels will be
a pain.Quoted 4 lines
As a workaround I'd like to enable x2apic on qemux86-64. Since that works via MSRs it avoids the problem.
Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-64?Nope. No big deal at all. I've used the same config i the past when doing work with rt-KVM, so I know it well.
Quoted 7 lines
Quoted 2 lines
I'll prep a commit, and send it along as part of a rebased set of kernel patches.
Thanks, much appreciated.
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
Quoted 2 lines
We may also need to backport the defconfig change back to the older releases so we don't get the hangs in those builds either...
Not a problem either. I'll prepare several.
Bruce
Quoted 5 lines
Cheers,
Richard
Quoted 33 lines · Bruce Ashfield
Quoted 32 lines · Richard Purdie
Quoted 21 lines · Bruce Ashfield
Quoted 19 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels will
be
a pain.As a workaround I'd like to enable x2apic on qemux86-64. Since that
works via MSRs it avoids the problem.Quoted 7 lines
Quoted 2 lines
Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86- 64?Nope. No big deal at all. I've used the same config i the past when doing work with rt-KVM, so I know it well.
I'll prep a commit, and send it along as part of a rebased set of kernel patches.
Thanks, much appreciated.
Quoted 8 lines
Quoted 2 lines
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first to speed it up a bit).
It may tell you to clear SANITY_TESTED_DISTROS, I think it auto adds meta-selftest now. You may also need to set DISPLAY to something (e.g. a dummy vncserver).
Not sure what is different about the serial qemu is using verses your own tests...
Quoted 3 lines
> We may also need to backport the defconfig change back to the older > releases so we don't get the hangs in those builds either... Not a problem either. I'll prepare several.>
Thanks!
Cheers,
Richard
Quoted 1 line
Quoted 40 lines · Richard Purdie
Quoted 39 lines · Bruce Ashfield
Quoted 25 lines · Richard Purdie
Quoted 23 lines · Bruce Ashfield
Quoted 21 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels will
be
a pain.As a workaround I'd like to enable x2apic on qemux86-64. Since that
works via MSRs it avoids the problem.Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86- 64?Quoted 9 lines
Quoted 5 lines
Nope. No big deal at all. I've used the same config i the past when doing work with rt-KVM, so I know it well.
I'll prep a commit, and send it along as part of a rebased set of kernel patches.
Thanks, much appreciated.
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Quoted 6 lines
Quoted 2 lines
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
Yah, that's where it went off the rails for me, so I figure that I'm doing something stupid:
yow-bashfiel-d4 [/home/bruc...poky/build]> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy Traceback (most recent call last):
File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module>
from oeqa.utils import load_test_components
ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n'
Quoted 7 lines
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first to speed it up a bit).
It may tell you to clear
SANITY_TESTED_DISTROS, I think it auto adds meta-selftest now. You may also need to set DISPLAY to something (e.g. a dummy vncserver).
I'll keep smashing on it.
Bruce
Quoted 14 lines
Not sure what is different about the serial qemu is using verses your own tests...
Quoted 3 lines
> We may also need to backport the defconfig change back to the older > releases so we don't get the hangs in those builds either... Not a problem either. I'll prepare several.>
Thanks!
Cheers,
Richard
Quoted 1 line
Quoted 6 lines · Bruce Ashfield
yow-bashfiel-d4 [
/home/bruc...poky/build]> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
Traceback (most recent call last):
File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module>
from oeqa.utils importload_test_components
ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n'
You've been switching between master and a pre-py3 branch, and Python is stupid so the py3 binary is trying to run py2 compiled binaries.
find poky -name \*.pyc -delete will fix it.
Ross
Quoted 11 lines · Ross
On 13 December 2017 at 14:17, Bruce Ashfield <[email protected] <mailto:[email protected]>> wrote:
yow-bashfiel-d4 [
/home/bruc...poky/build]> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
Traceback (most recent call last):
File "/home/bruce/poky/scripts/oe-selftest", line 43, in <module> from oeqa.utils importload_test_components
ImportError: bad magic number in 'oeqa': b'\x03\xf3\r\n'You've been switching between master and a pre-py3 branch, and Python is stupid so the py3 binary is trying to run py2 compiled binaries.
Yah. My build directory is three years old :D
Quoted 2 lines
find poky -name \*.pyc -delete will fix it.
Thanks. I'd have *never* come up with that!!!
Bruce
Quoted 2 lines
Ross
Quoted 45 lines · Richard Purdie
Quoted 33 lines · Bruce Ashfield
Quoted 32 lines · Richard Purdie
Quoted 21 lines · Bruce Ashfield
Quoted 19 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels will
be
a pain.As a workaround I'd like to enable x2apic on qemux86-64. Since that
works via MSRs it avoids the problem.Quoted 7 lines
Quoted 2 lines
Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86- 64?Nope. No big deal at all. I've used the same config i the past when doing work with rt-KVM, so I know it well.
I'll prep a commit, and send it along as part of a rebased set of kernel patches.
Thanks, much appreciated.
Quoted 8 lines
Quoted 2 lines
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
Quoted 9 lines
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first to speed it up a bit).
It may tell you to clear
SANITY_TESTED_DISTROS, I think it auto adds meta-selftest now. You may also need to set DISPLAY to something (e.g. a dummy vncserver).
With the cleanup that Ross gave me, I'm now able to see the issue.
I'll start debugging on Thursday.
One additional question: what's the right way to find the logs of that run ?
I'm getting a bit of noise from:
----------------
0155: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-1.pyc' -> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-1.pyc'
ERROR: gnu-efi-3.0.6-r0 do_package: Function failed: extend_recipe_sysroot
ERROR: Logfile of failure stored in: /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495
--------------
So I'm not seeing where to find it.
Bruce
Quoted 13 lines
Not sure what is different about the serial qemu is using verses your own tests...
Quoted 3 lines
> We may also need to backport the defconfig change back to the older > releases so we don't get the hangs in those builds either... Not a problem either. I'll prepare several.>
Thanks!
Cheers,
Richard
Quoted 1 line
Quoted 90 lines · Bruce Ashfield
Quoted 61 lines · Richard Purdie
Quoted 43 lines · Bruce Ashfield
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our
autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels
will
be
a pain.Quoted 8 lines
As a workaround I'd like to enable x2apic on qemux86-64. Since
that
works via MSRs it avoids the problem.Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-
64?Quoted 7 lines
Nope. No big deal at all. I've used the same config i the past
when doing work with rt-KVM, so I know it well.I'll prep a commit, and send it along as part of a rebased set
of kernel patches.Thanks, much appreciated.
Quoted 8 lines
Quoted 2 lines
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
to speed it up a bit).Quoted 6 lines
It may tell you to clear
SANITY_TESTED_DISTROS, I think it auto adds
meta-selftest now. You may also need to set DISPLAY to something (e.g.
a dummy vncserver).With the cleanup that Ross gave me, I'm now able to see the issue.
I'll start debugging on Thursday.
One additional question: what's the right way to find the logs of that
run ?I'm getting a bit of noise from:
----------------
0155: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'
-> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot- native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'
Quoted 5 lines
ERROR: gnu-efi-3.0.6-r0
do_package: Function failed:extend_recipe_sysroot
ERROR: Logfile of failure stored in:/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495
You've deleted pyc files from python itself which is bad. I'd cleansstate python-native and rebuilt it to fix that. Ross meant onto to run it on the poky meta directory...
Cheers,
Richard
Quoted 97 lines · Richard Purdie
Quoted 90 lines · Bruce Ashfield
Quoted 61 lines · Richard Purdie
Quoted 43 lines · Bruce Ashfield
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our
autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels
will
be
a pain.Quoted 8 lines
As a workaround I'd like to enable x2apic on qemux86-64. Since
that
works via MSRs it avoids the problem.Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-
64?Quoted 7 lines
Nope. No big deal at all. I've used the same config i the past
when doing work with rt-KVM, so I know it well.I'll prep a commit, and send it along as part of a rebased set
of kernel patches.Thanks, much appreciated.
Quoted 8 lines
Quoted 2 lines
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
to speed it up a bit).Quoted 6 lines
It may tell you to clear
SANITY_TESTED_DISTROS, I think it auto adds
meta-selftest now. You may also need to set DISPLAY to something (e.g.
a dummy vncserver).With the cleanup that Ross gave me, I'm now able to see the issue.
I'll start debugging on Thursday.
One additional question: what's the right way to find the logs of that
run ?I'm getting a bit of noise from:
----------------
0155: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'
-> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot- native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'Quoted 5 lines
ERROR: gnu-efi-3.0.6-r0
do_package: Function failed:extend_recipe_sysroot
ERROR: Logfile of failure stored in:/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495
Quoted 3 lines
You've deleted pyc files from python itself which is bad. I'd cleansstate python-native and rebuilt it to fix that. Ross meant onto to run it on the poky meta directory...
Yah. I clued into that right as I hit send. I'll see if I can save the build now :D
Bruce
Quoted 6 lines
Cheers,
Richard
Quoted 97 lines · Richard Purdie
Quoted 90 lines · Bruce Ashfield
Quoted 61 lines · Richard Purdie
Quoted 43 lines · Bruce Ashfield
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our
autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels
will
be
a pain.Quoted 8 lines
As a workaround I'd like to enable x2apic on qemux86-64. Since
that
works via MSRs it avoids the problem.Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-
64?Quoted 7 lines
Nope. No big deal at all. I've used the same config i the past
when doing work with rt-KVM, so I know it well.I'll prep a commit, and send it along as part of a rebased set
of kernel patches.Thanks, much appreciated.
Quoted 8 lines
Quoted 2 lines
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
to speed it up a bit).Quoted 6 lines
It may tell you to clear
SANITY_TESTED_DISTROS, I think it auto adds
meta-selftest now. You may also need to set DISPLAY to something (e.g.
a dummy vncserver).With the cleanup that Ross gave me, I'm now able to see the issue.
I'll start debugging on Thursday.
One additional question: what's the right way to find the logs of that
run ?I'm getting a bit of noise from:
----------------
0155: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'
-> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot- native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'Quoted 5 lines
ERROR: gnu-efi-3.0.6-r0
do_package: Function failed:extend_recipe_sysroot
ERROR: Logfile of failure stored in:/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495
Quoted 4 lines
You've deleted pyc files from python itself which is bad. I'd cleansstate python-native and rebuilt it to fix that. Ross meant onto to run it on the poky meta directory...
Is this what I should expect for the error ?
It looked like qemu timed out, but that is probably just the log in issue, correct ?
A manual log into the same image worked!
Bruce
-------------------
32105 3290 python3 /home/bruce/poky/scripts/oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
32520 32105 /bin/sh -c runqemu nographic /home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 bootparams="console=tty1 console=ttyS0,115200n8 printk.time=1" qemuparams="-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105" 32521 32520 python3 /home/bruce/poky/scripts/runqemu nographic /home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 bootparams=console=tty1 console=ttyS0,115200n8 printk.time=1 qemuparams=-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105 32522 32105 python3 /home/bruce/poky/scripts/oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
32527 32521 /bin/sh -c sudo /home/bruce/poky/scripts/runqemu-ifup 1000 1000 /home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin
32528 32527 sudo /home/bruce/poky/scripts/runqemu-ifup 1000 1000 /home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin
32529 2 [kworker/u8:1]
32530 32105 ps axww -o pid,ppid,command
Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201712131950_qemu Sending SIGTERM to runqemu
Sending SIGKILL to runqemu
Output from runqemu:
runqemu - INFO - Continuing with the following parameters:
runqemu - INFO - Setting up tap interface under sudo
----------------------------------------------------------------------
2017-12-13 19:52:57,550 - oe-selftest - INFO - Ran 1 test in 257.941s 2017-12-13 19:52:57,550 - oe-selftest - INFO - FAILED 2017-12-13 19:52:57,550 - oe-selftest - INFO - (errors=1) 2017-12-13 19:52:57,550 - oe-selftest - INFO - RESULTS: 2017-12-13 19:52:57,551 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: ERROR 2017-12-13 19:52:57,551 - oe-selftest - INFO - SUMMARY: 2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 257.946s 2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed
Quoted 5 lines
Cheers,
Richard
Quoted 97 lines · Richard Purdie
Quoted 90 lines · Bruce Ashfield
Quoted 61 lines · Richard Purdie
Quoted 43 lines · Bruce Ashfield
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our
autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels
will
be
a pain.Quoted 8 lines
As a workaround I'd like to enable x2apic on qemux86-64. Since
that
works via MSRs it avoids the problem.Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-
64?Quoted 7 lines
Nope. No big deal at all. I've used the same config i the past
when doing work with rt-KVM, so I know it well.I'll prep a commit, and send it along as part of a rebased set
of kernel patches.Thanks, much appreciated.
Quoted 8 lines
Quoted 2 lines
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
to speed it up a bit).Quoted 6 lines
It may tell you to clear
SANITY_TESTED_DISTROS, I think it auto adds
meta-selftest now. You may also need to set DISPLAY to something (e.g.
a dummy vncserver).With the cleanup that Ross gave me, I'm now able to see the issue.
I'll start debugging on Thursday.
One additional question: what's the right way to find the logs of that
run ?I'm getting a bit of noise from:
----------------
0155: else:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'
-> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot- native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'Quoted 5 lines
ERROR: gnu-efi-3.0.6-r0
do_package: Function failed:extend_recipe_sysroot
ERROR: Logfile of failure stored in:/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495
Quoted 4 lines
You've deleted pyc files from python itself which is bad. I'd cleansstate python-native and rebuilt it to fix that. Ross meant onto to run it on the poky meta directory...
I'm fully up and running now. I did a boot last night, and did see a failure in the selftest.
I then booted it manually (runqemu directly), logged in .. and now I'm seeing passing selftests. grr.
https://pastebin.com/iA0vxrmD
https://pastebin.com/ved4xvbQ
Bruce
Quoted 5 lines
Cheers,
Richard
Whoops. Maybe I should have been clearer... Sorry!
Ross
Quoted 112 lines · Richard Purdie
Quoted 90 lines · Bruce Ashfield
Quoted 61 lines · Richard Purdie
Quoted 43 lines · Bruce Ashfield
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
Hi Bruce,
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
We've root caused the hangs we're seeing in qemux86-64 on our
autobuilder workers. We've identified a commit on 4.15-rc3 which
fixes
the issue but backporting it to the autobuilder's host kernels
will
be
a pain.Quoted 8 lines
As a workaround I'd like to enable x2apic on qemux86-64. Since
that
works via MSRs it avoids the problem.Would it be a big issue to set
CONFIG_X86_X2APIC=yfor qemux86-
64?Quoted 7 lines
Nope. No big deal at all. I've used the same config i the past
when doing work with rt-KVM, so I know it well.I'll prep a commit, and send it along as part of a rebased set
of kernel patches.Thanks, much appreciated.
Quoted 8 lines
Quoted 2 lines
Did we get to the bottom of the 4.12 selftest failures?
Not yet no. I was distracted for a day, but need to learn how to run them myself. I was able to log in via serial in my normal boot flow, so that didn't get me anywhere.
Is the selftest HOWO in the yocto docs ? I'll google it up this morning.
No HOWTO but I did give you the command. In a poky checkout run:
oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS first
to speed it up a bit).Quoted 6 lines
It may tell you to clear
SANITY_TESTED_DISTROS, I think it auto adds
meta-selftest now. You may also need to set DISPLAY to something (e.g.
a dummy vncserver).With the cleanup that Ross gave me, I'm now able to see the issue.
I'll start debugging on Thursday.
One additional question: what's the right way to find the logs of that
run ?I'm getting a bit of noise from:
----------------
0155: else:Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'
-> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/recipe-sysroot- native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt- 1.pyc'Quoted 5 lines
ERROR: gnu-efi-3.0.6-r0
do_package: Function failed:extend_recipe_sysroot
ERROR: Logfile of failure stored in:/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-r0/temp/log.do_package.10495You've deleted pyc files from python itself which is bad. I'd cleansstate python-native and rebuilt it to fix that. Ross meant onto to run it on the poky meta directory...
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core