Skip to contentSkip to conversation

qemux86/conf: TEST ONLY. Drop smp boot

PATCH In review [email protected] 3 participants 9 messages Jun 8, 2023 – Jun 12, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 qemux86/conf: TEST ONLY. Drop smp boot

From: Bruce Ashfield <[email protected]>

We are seeing intermittent boot hangs with x86. Once of the last messages in failing logs is secondary cpu startup.

Dropping us down to smp for testing to see if the hangs start to go away with smp removed.

---
 meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/machine/include/x86/qemuboot-x86.inc +1 -1
index 6ae03633ae..54a12119a7 100644
@@ -1,6 +1,6 @@
 # For runqemu
 IMAGE_CLASSES += "qemuboot"
-QB_SMP ?= "-smp 4"
+QB_SMP ?= ""
 QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 
Signature · 1 line
2.39.2
#1 lore raw

PATCH 2/2 qemux86*: TEST only set preferred provider to linux-yocto-dev

From: Bruce Ashfield <[email protected]>

We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.

---
 meta/conf/machine/include/qemu.inc | 2 ++
 1 file changed, 2 insertions(+)
meta/conf/machine/include/qemu.inc +2 -0
index 0d71bcbbad..121e8474c1 100644
@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
 
 # Use a common kernel recipe for all QEMU machines
 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
+PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
+PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
 
 EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
 
Signature · 1 line
2.39.2
#2 lore raw

Re: [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev

Quoted 16 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:

From: Bruce Ashfield <[email protected]>

We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.

---
 meta/conf/machine/include/qemu.inc | 2 ++
 1 file changed, 2 insertions(+)
meta/conf/machine/include/qemu.inc +0 -0
index 0d71bcbbad..121e8474c1 100644

meta/conf/machine/include/qemu.inc RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" · line 28

Quoted 8 lines
meta/conf/machine/include/qemu.inc +2 -0
@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
 
 # Use a common kernel recipe for all QEMU machines
 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
+PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
+PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
 
 EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
 

Unfortunately this has other issues:

https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio

Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, in test_kernel_module

    self.assertEqual(status, 0, msg='\n'.join([cmd, output]))

AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
insmod: ERROR: could not insert module hellomod.ko: Invalid module format

But also:

https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio

which shows ping failures, as if the networking isn't working. I checked the boot logs and the kernel is booting but for some reason networking doesn't become live and ping fails.

I can see messages saying the "IP-Config: Complete:" with appropriate network config in the kernel boot log so it is all the more puzzling:

https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146

The ping failures repeated in many other targets in that build.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

Re: [OE-core] [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev

Quoted 54 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:

From: Bruce Ashfield <[email protected]>

We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.

---
 meta/conf/machine/include/qemu.inc | 2 ++
 1 file changed, 2 insertions(+)
meta/conf/machine/include/qemu.inc +0 -0
index 0d71bcbbad..121e8474c1 100644

meta/conf/machine/include/qemu.inc RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" · line 28

Quoted 8 lines
meta/conf/machine/include/qemu.inc +2 -0
@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
 
 # Use a common kernel recipe for all QEMU machines
 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
+PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
+PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
 
 EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
 

Unfortunately this has other issues:

https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio

Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, in test_kernel_module

    self.assertEqual(status, 0, msg='\n'.join([cmd, output]))

AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
insmod: ERROR: could not insert module hellomod.ko: Invalid module format

But also:

https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio

which shows ping failures, as if the networking isn't working. I checked the boot logs and the kernel is booting but for some reason networking doesn't become live and ping fails.

Quoted 6 lines

I can see messages saying the "IP-Config: Complete:" with appropriate network config in the kernel boot log so it is all the more puzzling:

https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146

The ping failures repeated in many other targets in that build.

There is also a perf buildpath warning:

https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/12/logs/warnings

which we'll likely have to fix at some point.

Cheers,

Richard

#5 lore raw

Re: [OE-core] [PATCH 2/2] qemux86*: TEST only set preferred provider to linux-yocto-dev

Quoted 33 lines · Richard Purdie
Quoted 30 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:

From: Bruce Ashfield <[email protected]>

We are seeing intermittent boot hangs on x86. To try and determine if this is a 6.1-stable only issue, set the prferred provider for qemux86 to linux-yocto-dev.

---
 meta/conf/machine/include/qemu.inc | 2 ++
 1 file changed, 2 insertions(+)
meta/conf/machine/include/qemu.inc +0 -0
index 0d71bcbbad..121e8474c1 100644

meta/conf/machine/include/qemu.inc RDEPENDS:${KERNEL_PACKAGE_NAME}-base = "" · line 28

Quoted 8 lines
meta/conf/machine/include/qemu.inc +2 -0
@@ -21,6 +21,8 @@ RDEPENDS:${KERNEL_PACKAGE_NAME}-base = ""
 
 # Use a common kernel recipe for all QEMU machines
 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
+PREFERRED_PROVIDER_virtual/kernel:qemux86 = "linux-yocto-dev"
+PREFERRED_PROVIDER_virtual/kernel:qemux86-64 = "linux-yocto-dev"
 
 EXTRA_IMAGEDEPENDS += "qemu-system-native qemu-helper-native:do_addto_recipe_sysroot"
 

Unfortunately this has other issues:

https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7276/steps/13/logs/stdio

Quoted 16 lines
Quoted 12 lines

Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/pokybuild/yocto-worker/musl-qemux86-64/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 48, in test_kernel_module

Quoted 4 lines
    self.assertEqual(status, 0, msg='\n'.join([cmd, output]))

AssertionError: 1 != 0 : cd /tmp && insmod hellomod.ko
insmod: ERROR: could not insert module hellomod.ko: Invalid module format

I'll fix this up as part of my -dev kernel work to get it ready as a reference kernel.

Quoted 8 lines
Quoted 8 lines

But also:

https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/13/logs/stdio

which shows ping failures, as if the networking isn't working. I checked the boot logs and the kernel is booting but for some reason networking doesn't become live and ping fails.

Quoted 6 lines
Quoted 6 lines

I can see messages saying the "IP-Config: Complete:" with appropriate network config in the kernel boot log so it is all the more puzzling:

https://autobuilder.yocto.io/pub/failed-builds-data/qemu_boot_log.20230609064146

The ping failures repeated in many other targets in that build.

This is a strange one. Not seeing that here, but I'll see if some sort of networking configuration needs tweaking.

Quoted 6 lines

There is also a perf buildpath warning:

https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/6105/steps/12/logs/warnings

which we'll likely have to fix at some point.

I'll fix that right away.

Bruce

Quoted 4 lines

Cheers,

Richard

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#6 lore raw
In reply to [email protected]
Quoted 19 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:

From: Bruce Ashfield <[email protected]>

We are seeing intermittent boot hangs with x86. Once of the last messages in failing logs is secondary cpu startup.

Dropping us down to smp for testing to see if the hangs start to go away with smp removed.

---
 meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/machine/include/x86/qemuboot-x86.inc +0 -0
index 6ae03633ae..54a12119a7 100644

meta/conf/machine/include/x86/qemuboot-x86.inc line 6

Quoted 7 lines
meta/conf/machine/include/x86/qemuboot-x86.inc +1 -1
@@ -1,6 +1,6 @@
 # For runqemu
 IMAGE_CLASSES += "qemuboot"
-QB_SMP ?= "-smp 4"
+QB_SMP ?= ""
 QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 

FWIW, this patch didn't change much. With it applied we still see:

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5168/steps/12/logs/stdio

[    0.634735] Freeing SMP alternatives memory: 52K
[   12.275510] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9)
[274491.955544] cblist_init_generic: Setting adjustable number of callback queues.
[274492.008510] cblist_init_generic: Setting shift to 0 and lim to 1.
[279820.791509] cblist_init_generic: Setting shift to 0 and lim to 1.
[282104.735510] cblist_init_generic: Setting shift to 0 and lim to 1.
[386831.154510] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.
[386850.632510] rcu: Hierarchical SRCU implementation.
[386850.668510] rcu: 	Max phase no-delay instances is 400.
[386852.898509] smp: Bringing up secondary CPUs ...
[386852.946510] smp: Brought up 1 node, 1 CPU
[386852.992510] smpboot: Max logical packages: 1
[386853.018509] smpboot: Total of 1 processors activated (5199.99 BogoMIPS)
[386855.257509] devtmpfs: initialized
[386857.982509] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[386858.041510] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[386858.691509] pinctrl core: initialized pinctrl subsystem
[386862.836510] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[386864.720510] thermal_sys: Registered thermal governor 'step_wise'
[386864.725509] thermal_sys: Registered thermal governor 'user_space'
[386864.864510] cpuidle: using governor menu
[386865.662509] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xb0000000-0xbfffffff] (base 0xb0000000)
[386865.693510] PCI: MMCONFIG at [mem 0xb0000000-0xbfffffff] reserved in E820
[386865.900509] PCI: Using configuration type 1 for base access
[386878.550509] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.

Cheers,

Richard

#7 lore raw
Quoted 15 lines · Richard Purdie
Quoted 13 lines · On Thu, 2023-06-08 at 19:39 +0000, [email protected] wrote:

From: Bruce Ashfield <[email protected]>

We are seeing intermittent boot hangs with x86. Once of the last messages in failing logs is secondary cpu startup.

Dropping us down to smp for testing to see if the hangs start to go away with smp removed.

---
 meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines

meta/conf/machine/include/x86/qemuboot-x86.inc line 6

Quoted 13 lines
meta/conf/machine/include/x86/qemuboot-x86.inc +1 -1
index 6ae03633ae..54a12119a7 100644
@@ -1,6 +1,6 @@
 # For runqemu
 IMAGE_CLASSES += "qemuboot"
-QB_SMP ?= "-smp 4"
+QB_SMP ?= ""
 QB_CPU:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 QB_CPU_KVM:x86 ?= "-cpu IvyBridge -machine q35,i8042=off"
 

FWIW, this patch didn't change much. With it applied we still see:

ok. so at least we can rule out it being specific to secondary cpu startup.

Bruce

Quoted 34 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/5168/steps/12/logs/stdio

[    0.634735] Freeing SMP alternatives memory: 52K
[   12.275510] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (family: 0x6, model: 0x3a, stepping: 0x9)
[274491.955544] cblist_init_generic: Setting adjustable number of callback queues.
[274492.008510] cblist_init_generic: Setting shift to 0 and lim to 1.
[279820.791509] cblist_init_generic: Setting shift to 0 and lim to 1.
[282104.735510] cblist_init_generic: Setting shift to 0 and lim to 1.
[386831.154510] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.
[386850.632510] rcu: Hierarchical SRCU implementation.
[386850.668510] rcu:    Max phase no-delay instances is 400.
[386852.898509] smp: Bringing up secondary CPUs ...
[386852.946510] smp: Brought up 1 node, 1 CPU
[386852.992510] smpboot: Max logical packages: 1
[386853.018509] smpboot: Total of 1 processors activated (5199.99 BogoMIPS)
[386855.257509] devtmpfs: initialized
[386857.982509] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[386858.041510] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[386858.691509] pinctrl core: initialized pinctrl subsystem
[386862.836510] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[386864.720510] thermal_sys: Registered thermal governor 'step_wise'
[386864.725509] thermal_sys: Registered thermal governor 'user_space'
[386864.864510] cpuidle: using governor menu
[386865.662509] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xb0000000-0xbfffffff] (base 0xb0000000)
[386865.693510] PCI: MMCONFIG at [mem 0xb0000000-0xbfffffff] reserved in E820
[386865.900509] PCI: Using configuration type 1 for base access
[386878.550509] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.

Cheers,

Richard

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#8 lore raw
Quoted 3 lines · [Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot] On 09/06/2023 (Fri…
Quoted 1 line · On Fri, Jun 9, 2023 at 2:06???AM Richard Purdie…

[...]

Quoted 4 lines
Quoted 1 line

FWIW, this patch didn't change much. With it applied we still see:

ok. so at least we can rule out it being specific to secondary cpu startup.

I'll own some of that. I was looking forward in the dmesg where secondary CPU init was, and ignoring looking backwards.

I have a new thought. I still could be completely off base. I updated the case with my theory.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15138

Might totally be another wild goose chase. Who knows...

Paul.
--

#9 lore raw
Quoted 16 lines · Paul Gortmaker
Quoted 3 lines · [Re: [PATCH 1/2] qemux86/conf: TEST ONLY. Drop smp boot] On 09/06/2023 (Fri…
Quoted 1 line · On Fri, Jun 9, 2023 at 2:06???AM Richard Purdie…

[...]

Quoted 4 lines
Quoted 1 line

FWIW, this patch didn't change much. With it applied we still see:

ok. so at least we can rule out it being specific to secondary cpu startup.

I'll own some of that. I was looking forward in the dmesg where secondary CPU init was, and ignoring looking backwards.

Quoted 6 lines

I have a new thought. I still could be completely off base. I updated the case with my theory.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15138

Might totally be another wild goose chase. Who knows...

I was leaning toward leaving these on as it tests things as most users would end up using them. Whether this is the right thing to do, not sure.

As an extra data point, we have this failure with 6.1.28:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5262/steps/14/logs/stdio

which means the issue is probably between 6.1.26 and 6.1.28. I'm testing with 6.1.27 and haven't seen it there yet.

Cheers,

Richard