qemux86/conf: TEST ONLY. Drop smp boot
PATCH 1/2 qemux86/conf: TEST ONLY. Drop smp boot
From: Bruce Ashfield <[email protected]>
We are seeing intermittent boot hangs with x86. Once of the last messages in failing logs is secondary cpu startup.
Dropping us down to smp for testing to see if the hangs start to go away with smp removed.
--- meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/machine/include/x86/qemuboot-x86.inc +1 -1
@@ -1,6 +1,6 @@ # For runqemu IMAGE_CLASSES += "qemuboot" -QB_SMP ?= "-smp 4" +QB_SMP ?= "" QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off" QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
Signature · 1 line
2.39.2
PATCH 2/2 qemux86*: TEST only set preferred provider to linux-yocto-dev
From: Bruce Ashfield <[email protected]>
We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.
--- meta/conf/machine/include/qemu.inc | 2 ++ 1 file changed, 2 insertions(+)
meta/conf/machine/include/qemu.inc +2 -0
@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" # Use a common kernel recipe for all QEMU machines PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto" +PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev" +PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev" EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
Signature · 1 line
2.39.2
Re: [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev
Quoted 16 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:
From: Bruce Ashfield <[email protected]>
We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/conf/machine/include/qemu.inc | 2 ++ 1 file changed, 2 insertions(+)meta/conf/machine/include/qemu.inc +0
-0
meta/conf/machine/include/qemu.inc RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" · line 28
Quoted 8 lines
meta/conf/machine/include/qemu.inc +2
-0@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" # Use a common kernel recipe for all QEMU machines PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto" +PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev" +PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev" EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
Unfortunately this has other issues:
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, in test_kernel_module
self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
insmod: ERROR: could not insert module hellomod.ko: Invalid module format
But also:
https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio
which shows ping failures, as if the networking isn't working. I checked the boot logs and the kernel is booting but for some reason networking doesn't become live and ping fails.
I can see messages saying the "IP-Config: Complete:" with appropriate network config in the kernel boot log so it is all the more puzzling:
https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146
The ping failures repeated in many other targets in that build.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev
Quoted 54 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:
From: Bruce Ashfield <[email protected]>
We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/conf/machine/include/qemu.inc | 2 ++ 1 file changed, 2 insertions(+)meta/conf/machine/include/qemu.inc +0
-0meta/conf/machine/include/qemu.inc RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" · line 28
Quoted 8 lines
meta/conf/machine/include/qemu.inc +2
-0@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" # Use a common kernel recipe for all QEMU machines PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto" +PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev" +PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev" EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"Unfortunately this has other issues:
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, inwrapped_f
returnfunc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, inwrapped_f
returnfunc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, inwrapped_f
returnfunc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, intest_kernel_moduleself.assertEqual(status, 0, msg='\n'.join([cmd, output]))AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
insmod: ERROR: could not insert module hellomod.ko: Invalid module formatBut also:
https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio
which shows ping failures, as if the networking isn't working. I checked the boot logs and the kernel is booting but for some reason networking doesn't become live and ping fails.
Quoted 6 lines
I can see messages saying the "IP-Config: Complete:" with appropriate network config in the kernel boot log so it is all the more puzzling:
https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146
The ping failures repeated in many other targets in that build.
There is also a perf buildpath warning:
https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/12/logs/warnings
which we'll likely have to fix at some point.
Cheers,
Richard
Re: [OE-core] [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev
Quoted 33 lines · Richard Purdie
Quoted 30 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:
From: Bruce Ashfield <[email protected]>
We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/conf/machine/include/qemu.inc | 2 ++ 1 file changed, 2 insertions(+)meta/conf/machine/include/qemu.inc +0
-0meta/conf/machine/include/qemu.inc RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" · line 28
Quoted 8 lines
meta/conf/machine/include/qemu.inc +2
-0@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" # Use a common kernel recipe for all QEMU machines PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto" +PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev" +PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev" EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"Unfortunately this has other issues:
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio
Quoted 16 lines
Quoted 12 lines
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, inwrapped_f
returnfunc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, inwrapped_f
returnfunc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, inwrapped_f
returnfunc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, intest_kernel_moduleQuoted 4 lines
self.assertEqual(status, 0, msg='\n'.join([cmd, output]))AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
insmod: ERROR: could not insert module hellomod.ko: Invalid module format
I'll fix this up as part of my -dev kernel work to get it ready as a reference kernel.
Quoted 8 lines
Quoted 8 lines
But also:
https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio
which shows ping failures, as if the networking isn't working. I checked the boot logs and the kernel is booting but for some reason networking doesn't become live and ping fails.
Quoted 6 lines
Quoted 6 lines
I can see messages saying the "IP-Config: Complete:" with appropriate network config in the kernel boot log so it is all the more puzzling:
https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146
The ping failures repeated in many other targets in that build.
This is a strange one. Not seeing that here, but I'll see if some sort of networking configuration needs tweaking.
Quoted 6 lines
There is also a perf buildpath warning:
https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/12/logs/warnings
which we'll likely have to fix at some point.
I'll fix that right away.
Bruce
Quoted 4 lines
Cheers,
Richard
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 19 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:
From: Bruce Ashfield <[email protected]>
We are seeing intermittent boot hangs with x86. Once of the last messages in failing logs is secondary cpu startup.
Dropping us down to smp for testing to see if the hangs start to go away with smp removed.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/conf/machine/include/x86/qemuboot-x86.inc +0
-0
meta/conf/machine/include/x86/qemuboot-x86.inc line 6
Quoted 7 lines
meta/conf/machine/include/x86/qemuboot-x86.inc +1
-1@@ -1,6 +1,6 @@ # For runqemu IMAGE_CLASSES += "qemuboot" -QB_SMP ?= "-smp 4" +QB_SMP ?= "" QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off" QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
FWIW, this patch didn't change much. With it applied we still see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5168/steps/12/logs/stdio
[ 0.634735] Freeing SMP alternatives memory: 52K [ 12.275510] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) [274491.955544] cblist_init_generic: Setting adjustable number of callback queues. [274492.008510] cblist_init_generic: Setting shift to 0 and lim to 1. [279820.791509] cblist_init_generic: Setting shift to 0 and lim to 1. [282104.735510] cblist_init_generic: Setting shift to 0 and lim to 1. [386831.154510] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only. [386850.632510] rcu: Hierarchical SRCU implementation. [386850.668510] rcu: Max phase no-delay instances is 400. [386852.898509] smp: Bringing up secondary CPUs ... [386852.946510] smp: Brought up 1 node, 1 CPU [386852.992510] smpboot: Max logical packages: 1 [386853.018509] smpboot: Total of 1 processors activated (5199.99 BogoMIPS) [386855.257509] devtmpfs: initialized [386857.982509] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [386858.041510] futex hash table entries: 256 (order: 2, 16384 bytes, linear) [386858.691509] pinctrl core: initialized pinctrl subsystem [386862.836510] NET: Registered PF_NETLINK/PF_ROUTE protocol family [386864.720510] thermal_sys: Registered thermal governor 'step_wise' [386864.725509] thermal_sys: Registered thermal governor 'user_space' [386864.864510] cpuidle: using governor menu [386865.662509] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xb0000000-0xbfffffff] (base 0xb0000000) [386865.693510] PCI: MMCONFIG at [mem 0xb0000000-0xbfffffff] reserved in E820 [386865.900509] PCI: Using configuration type 1 for base access [386878.550509] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
Cheers,
Richard
Quoted 15 lines · Richard Purdie
Quoted 13 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:
From: Bruce Ashfield <[email protected]>
We are seeing intermittent boot hangs with x86. Once of the last messages in failing logs is secondary cpu startup.
Dropping us down to smp for testing to see if the hangs start to go away with smp removed.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines
meta/conf/machine/include/x86/qemuboot-x86.inc line 6
Quoted 13 lines
meta/conf/machine/include/x86/qemuboot-x86.inc +1
-1@@ -1,6 +1,6 @@ # For runqemu IMAGE_CLASSES += "qemuboot" -QB_SMP ?= "-smp 4" +QB_SMP ?= "" QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off" QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"FWIW, this patch didn't change much. With it applied we still see:
ok. so at least we can rule out it being specific to secondary cpu startup.
Bruce
Quoted 34 lines
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5168/steps/12/logs/stdio
[ 0.634735] Freeing SMP alternatives memory: 52K [ 12.275510] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9) [274491.955544] cblist_init_generic: Setting adjustable number of callback queues. [274492.008510] cblist_init_generic: Setting shift to 0 and lim to 1. [279820.791509] cblist_init_generic: Setting shift to 0 and lim to 1. [282104.735510] cblist_init_generic: Setting shift to 0 and lim to 1. [386831.154510] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only. [386850.632510] rcu: Hierarchical SRCU implementation. [386850.668510] rcu: Max phase no-delay instances is 400. [386852.898509] smp: Bringing up secondary CPUs ... [386852.946510] smp: Brought up 1 node, 1 CPU [386852.992510] smpboot: Max logical packages: 1 [386853.018509] smpboot: Total of 1 processors activated (5199.99 BogoMIPS) [386855.257509] devtmpfs: initialized [386857.982509] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [386858.041510] futex hash table entries: 256 (order: 2, 16384 bytes, linear) [386858.691509] pinctrl core: initialized pinctrl subsystem [386862.836510] NET: Registered PF_NETLINK/PF_ROUTE protocol family [386864.720510] thermal_sys: Registered thermal governor 'step_wise' [386864.725509] thermal_sys: Registered thermal governor 'user_space' [386864.864510] cpuidle: using governor menu [386865.662509] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xb0000000-0xbfffffff] (base 0xb0000000) [386865.693510] PCI: MMCONFIG at [mem 0xb0000000-0xbfffffff] reserved in E820 [386865.900509] PCI: Using configuration type 1 for base access [386878.550509] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.Cheers,
Richard
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 3 lines · [Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot] On 09/06/2023 (Fri…
Quoted 1 line · On Fri, Jun 9, 2023 at 2:06???AM Richard Purdie…
[...]
Quoted 4 lines
Quoted 1 line
FWIW, this patch didn't change much. With it applied we still see:
ok. so at least we can rule out it being specific to secondary cpu startup.
I'll own some of that. I was looking forward in the dmesg where secondary CPU init was, and ignoring looking backwards.
I have a new thought. I still could be completely off base. I updated the case with my theory.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15138
Might totally be another wild goose chase. Who knows...
Paul.
--
Quoted 16 lines · Paul Gortmaker
Quoted 3 lines · [Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot] On 09/06/2023 (Fri…
Quoted 1 line · On Fri, Jun 9, 2023 at 2:06???AM Richard Purdie…
[...]
Quoted 4 lines
Quoted 1 line
FWIW, this patch didn't change much. With it applied we still see:
ok. so at least we can rule out it being specific to secondary cpu startup.
I'll own some of that. I was looking forward in the dmesg where secondary CPU init was, and ignoring looking backwards.
Quoted 6 lines
I have a new thought. I still could be completely off base. I updated the case with my theory.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15138
Might totally be another wild goose chase. Who knows...
I was leaning toward leaving these on as it tests things as most users would end up using them. Whether this is the right thing to do, not sure.
As an extra data point, we have this failure with 6.1.28:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5262/steps/14/logs/stdio
which means the issue is probably between 6.1.26 and 6.1.28. I'm testing with 6.1.27 and haven't seen it there yet.
Cheers,
Richard