2.6.29-rt1+ irqbalance = OOPS
Hello,
Starting the irqbalance daemon ( http://www.irqbalance.org/ ) on 2.6.29-rt1 causes the system to OOPS and freeze. irqbalance works fine with previous realtime kernels (2.6.24.7-rt27).
The OOPS seems to be caused by the PID of the [timer] IRQ-1. However the OOPS message is too long and I can't read the beginning lines on the terminal. The system hangs so I can't scroll back and it's not written to any log. SysRq does also not work after the OOPS.
A bit off topic, but how can I capture those OOPSes?
I could get a larger screen and a digital camera ;) The lkcd.sf.net patch does not apply to 2.6.29. Running a realtime kernel in qemu does not make much sense, although it may be sufficient to reproduce this OOPS. Is there a way to use USB as serial console using a hub and two computers? I guess I'll need a dongle or sth. Besides would a printk() to a serial port work even if some IRQ handler hangs?
Any ideas? links?
robin
Quoted 7 lines · Robin Gareus
Hello,
Starting the irqbalance daemon ( http://www.irqbalance.org/ ) on 2.6.29-rt1 causes the system to OOPS and freeze. irqbalance works fine with previous realtime kernels (2.6.24.7-rt27).
The OOPS seems to be caused by the PID of the [timer] IRQ-1
This in of itself might be part of the problem. I didn't think the timer was supposed to be threaded. Thomas?
Quoted 7 lines
. However the
OOPS message is too long and I can't read the beginning lines on the terminal. The system hangs so I can't scroll back and it's not written to any log. SysRq does also not work after the OOPS.A bit off topic, but how can I capture those OOPSes?
A serial-console or netconsole is generally better than a VT for catching the whole output.
Quoted 5 lines
I could get a larger screen and a digital camera ;) The lkcd.sf.net patch does not apply to 2.6.29. Running a realtime kernel in qemu does not make much sense, although it may be sufficient to reproduce this OOPS. Is there a way to use USB as serial console using a hub and two computers?
I think so, though I have never tried.
Quoted 3 lines
I guess I'll need a dongle or sth. Besides would a
printk()to a serial port work even if some IRQ handler hangs?
Usually yes. Of course, it depends on what is actually broken, but *most* issues still allow the printks to work.
Good luck!
-Greg
Quoted 10 lines · Gregory Haskins
Quoted 7 lines · Robin Gareus
Hello,
Starting the irqbalance daemon ( http://www.irqbalance.org/ ) on 2.6.29-rt1 causes the system to OOPS and freeze. irqbalance works fine with previous realtime kernels (2.6.24.7-rt27).
The OOPS seems to be caused by the PID of the [timer] IRQ-1
This in of itself might be part of the problem. I didn't think the timer was supposed to be threaded. Thomas?
OOPS ;) my bad. IRQ-0 is the timer and it's indeed not threaded.
IRQ-1 is i8042
The OOPS can be triggered by launching irqbalance. pressing any key. For whatever reason the OOPS also takes place after a random amount of time without pressing any key..
robin
Quoted 12 lines · Robin Gareus
Quoted 10 lines · Gregory Haskins
Quoted 7 lines · Robin Gareus
Hello,
Starting the irqbalance daemon ( http://www.irqbalance.org/ ) on 2.6.29-rt1 causes the system to OOPS and freeze. irqbalance works fine with previous realtime kernels (2.6.24.7-rt27).
The OOPS seems to be caused by the PID of the [timer] IRQ-1
This in of itself might be part of the problem. I didn't think the timer was supposed to be threaded. Thomas?
Quoted 7 lines
OOPS ;) my bad. IRQ-0 is the timer and it's indeed not threaded.
IRQ-1 is i8042
The OOPS can be triggered by launching irqbalance. pressing any key. For whatever reason the OOPS also takes place after a random amount of time without pressing any key..
Hmm. Works fine here. Can you please provide the output of
# cat /proc/interrupts
Thanks,
tglx
Quoted 16 lines · Thomas Gleixner
Quoted 15 lines · Robin Gareus
Quoted 10 lines · Gregory Haskins
Quoted 7 lines · Robin Gareus
Hello,
Starting the irqbalance daemon ( http://www.irqbalance.org/ ) on 2.6.29-rt1 causes the system to OOPS and freeze. irqbalance works fine with previous realtime kernels (2.6.24.7-rt27).
The OOPS seems to be caused by the PID of the [timer] IRQ-1
This in of itself might be part of the problem. I didn't think the timer was supposed to be threaded. Thomas?
OOPS ;) my bad. IRQ-0 is the timer and it's indeed not threaded.
IRQ-1 is i8042
Quoted 7 lines
Quoted 3 lines
The OOPS can be triggered by launching irqbalance. then pressing any key. For whatever reason the OOPS also takes place after a random amount of time without pressing any key..
Hmm. Works fine here. Can you please provide the output of
# cat /proc/interrupts
CPU0 CPU1
0: 2592396 0 IO-APIC-edge timer 1: 31901 0 IO-APIC-edge i8042 8: 79 0 IO-APIC-edge rtc0 9: 294004 0 IO-APIC-fasteoi acpi 12: 55544 0 IO-APIC-edge i8042 14: 28478 0 IO-APIC-edge ata_piix 15: 0 0 IO-APIC-edge ata_piix 16: 0 0 IO-APIC-fasteoi uhci_hcd:usb1, yenta, i915@pci:0000:00:02.0 17: 166 0 IO-APIC-fasteoi uhci_hcd:usb2, HDA Intel, ohci1394 18: 0 0 IO-APIC-fasteoi uhci_hcd:usb4, mmc0 19: 66 0 IO-APIC-fasteoi ehci_hcd:usb3, uhci_hcd:usb5 28: 1504 0 PCI-MSI-edge iwl3945 NMI: 0 0 Non-maskable interrupts LOC: 398393 1538764 Local timer interrupts SPU: 0 0 Spurious interrupts CNT: 0 0 Performance counter interrupts RES: 120526 181396 Rescheduling interrupts CAL: 42459 30754 Function call interrupts TLB: 839 1056 TLB shootdowns TRM: 0 0 Thermal event interrupts ERR: 0 MIS: 0
The corresponding .config is available at
http://rg42.org/_media/wiki/kernel/config-2.6.29-rt1.txt
I just tested Fernando's CCRMA 2.6.29-rt1 kernel which is x686 plus generic x86-compat (mine is MCORE2) and irqbalance does not produce an OOPS there.
I'll see if can get a dump of the OOPS on Monday using netconsole.
HTH,
robin
Quoted 2 lines · Robin Gareus
The corresponding .config is available at
http://rg42.org/_media/wiki/kernel/config-2.6.29-rt1.txt
BTW. This is a vanilla 2.6.29 with only the 2.6.29-rt1 patch applied.
Quoted 5 lines
I just tested Fernando's CCRMA 2.6.29-rt1 kernel which is x686 plus generic x86-compat (mine is MCORE2) and irqbalance does not produce an OOPS there.
I'll see if can get a dump of the OOPS on Monday using netconsole.
If the netconsole module is loaded, the OOPS is produced without the Call and Stack Trace (both on local terminal and remote) and the cause is no longer the keyboard-interrupt but ethernet (IRQ 28)
Starting irqbalance still reliably hangs this system (Thinkpad X60s).
------------[ cut here ]------------ kernel BUG at kernel/sched_rt.c:1602! invalid opcode: 0000 [#1] PREEMPT SMP kernel BUG at kernel/sched_rt.c:1602! invalid opcode: 0000 [#1] PREEMPT SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/irq Modules linked in: netconsole ipv6 nfsd exportfs nfs lockd nfs_acl
auth_rpcgss sunrpc deflate zlib_deflate twofish twofish_common camellia serpent blowfish des_generic cbc aes_i586 aes_generic xcbc rmd160 sha256_generic sha1_generic
last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/irq
Modules linked in: netconsole ipv6 nfsd exportfs nfs lockd nfs_acl
auth_rpcgss sunrpc deflate zlib_deflate twofish twofish_common camellia serpent blowfish des_generic cbc aes_i586 aes_generic xcbc rmd160 sha256_generic sha1_generic
-=-=-=-=-
# lspci ..
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode])
# cat /proc/interrupts
CPU0 CPU1
0: 416869 0 IO-APIC-edge timer 1: 2921 0 IO-APIC-edge i8042 8: 48 0 IO-APIC-edge rtc0 9: 36105 0 IO-APIC-fasteoi acpi 12: 23294 0 IO-APIC-edge i8042 14: 19922 0 IO-APIC-edge ata_piix 15: 0 0 IO-APIC-edge ata_piix 16: 0 0 IO-APIC-fasteoi uhci_hcd:usb2, yenta, i915@pci:0000:00:02.0 17: 51665 0 IO-APIC-fasteoi uhci_hcd:usb3, ohci1394, HDA Intel 18: 0 0 IO-APIC-fasteoi uhci_hcd:usb4, mmc0 19: 28 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5 28: 2669 0 PCI-MSI-edge eth1 29: 449 0 PCI-MSI-edge iwl3945 NMI: 0 0 Non-maskable interrupts LOC: 79098 252004 Local timer interrupts SPU: 0 0 Spurious interrupts CNT: 0 0 Performance counter interrupts RES: 86816 141218 Rescheduling interrupts CAL: 6852 5954 Function call interrupts TLB: 268 266 TLB shootdowns TRM: 0 0 Thermal event interrupts ERR: 0 MIS: 0
Quoted 10 lines · Robin Gareus
Quoted 2 lines · Robin Gareus
The corresponding .config is available at
http://rg42.org/_media/wiki/kernel/config-2.6.29-rt1.txtBTW. This is a vanilla 2.6.29 with only the 2.6.29-rt1 patch applied.
Quoted 3 lines
I just tested Fernando's CCRMA 2.6.29-rt1 kernel which is x686 plus generic x86-compat (mine is MCORE2) and irqbalance does not produce an OOPS there.
Ah, so there is some delta in the .configs which exposes this problem. I got your .config and try to reproduce.
Quoted 6 lines
Quoted 1 line
I'll see if can get a dump of the OOPS on Monday using netconsole.
If the netconsole module is loaded, the OOPS is produced without the Call and Stack Trace (both on local terminal and remote) and the cause is no longer the keyboard-interrupt but ethernet (IRQ 28)
Quoted 6 lines
Starting
irqbalancestill reliably hangs this system (Thinkpad X60s).------------[ cut here ]------------ kernel BUG at kernel/sched_rt.c:1602!invalid opcode: 0000 [#1] PREEMPT SMP kernel BUG at
kernel/sched_rt.c:1602!
Unfortunately this does not tell us much. It tells us that something went wrong, but not where and when :)
Thanks,
tglx
I'm just testing 2.6.29-rt3 and irqbalance works fine again. Many thanks Peter Zijlstra.
cheers,
robin