2.6.20-git13 kernel BUG at /mnt/md0/devel/linux-git/kernel/time/tick-sched.c:168
Hi,
This looks like a tickless stuff
------------[ cut here ]------------ kernel BUG at /mnt/md0/devel/linux-git/kernel/time/tick-sched.c:168! invalid opcode: 0000 [#1] PREEMPT SMP Modules linked in: rtc unix CPU: 0 EIP: 0060:[<c0139ea0>] Not tainted VLI EFLAGS: 00010002 (2.6.20 #53)
EIP is at tick_nohz_stop_sched_tick+0x5d/0x1d9
eax: 00000000 ebx: c6021100 ecx: 00000001 edx: c03e74e0
esi: c6022c40 edi: 00000000 ebp: c041dfac esp: c041df78
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068
Process swapper (pid: 0, ti=c041d000 task=c03e74e0 task.ti=c041d000)
Stack: 00000000 c03e7608 c6021b20 00000000 c041dfac 00000296 00000001 c602007b
c01000d8 c0451304 c6021100 c0102b42 00000000 c041dfc0 c01023b4 c602ee80
c0453ae4 c602b864 c041dfc8 c010141d c041dff8 c0421a52 c038d4da c04211ed
Call Trace:
[<c01050f9>] show_trace_log_lvl+0x1a/0x2f
[<c01051ab>] show_stack_log_lvl+0x9d/0xa5
[<c01053a0>] show_registers+0x1ed/0x32c
[<c01055f7>] die+0x118/0x22f
[<c0105787>] do_trap+0x79/0x91
[<c0106017>] do_invalid_op+0x97/0xa1
[<c0311abc>] error_code+0x7c/0x84
[<c01023b4>] cpu_idle+0x1d/0xfb
[<c010141d>] rest_init+0x37/0x3a
[<c0421a52>] start_kernel+0x459/0x461
[<00000000>] 0x0
=======================
Code: f0 ff ff 8b 40 08 a8 08 0f 85 74 01 00 00 e8 74 e7 0b 00 89 c7 bb 80 13 45 c0 e8 68 e7 0b 00 03 1c 85 00 a6 41 c0 83 3b 00 74 04 <0f> 0b eb fe 8d 45 ec e8 bf c0 ff ff 8b 4d ec 8b 5d f0 83 7e 3c
EIP: [<c0139ea0>] tick_nohz_stop_sched_tick+0x5d/0x1d9 SS:ESP 0068:c041df78
Kernel panic - not syncing: Attempted to kill the idle task!
BUG: at /mnt/md0/devel/linux-git/arch/i386/kernel/smp.c:547 smp_call_function()
[<c01050f9>] show_trace_log_lvl+0x1a/0x2f
[<c01057e0>] show_trace+0x12/0x14
[<c0105892>] dump_stack+0x16/0x18
[<c0112792>] smp_call_function+0x61/0x102
[<c0112851>] smp_send_stop+0x1e/0x31
[<c0121ae3>] panic+0x5e/0xfb
[<c012463b>] do_exit+0x7a/0x702
[<c01056e8>] die+0x209/0x22f
[<c0105787>] do_trap+0x79/0x91
[<c0106017>] do_invalid_op+0x97/0xa1
[<c0311abc>] error_code+0x7c/0x84
[<c01023b4>] cpu_idle+0x1d/0xfb
[<c010141d>] rest_init+0x37/0x3a
[<c0421a52>] start_kernel+0x459/0x461
[<00000000>] 0x0
=======================0xc0139ea0 is in tick_nohz_stop_sched_tick (/mnt/md0/devel/linux-git/kernel/time/tick-sched.c:168). 163
164 if (need_resched())
165 goto end;
166
167 cpu = smp_processor_id();
168 BUG_ON(local_softirq_pending());
169
170 now = ktime_get();
171 /*
172 * When called from irq_exit we need to account the idle sleep time
http://www.stardust.webpages.pl/files/tbf/euridica/2.6.20-git13/boot.log
http://www.stardust.webpages.pl/files/tbf/euridica/2.6.20-git13/git-config
Regards,
Michal
Signature · 5 lines
Michal K. K. Piotrowski LTG - Linux Testers Group (PL) (http://www.stardust.webpages.pl/ltg/) LTG - Linux Testers Group (EN) (http://www.stardust.webpages.pl/linux_testers_group_en/)
Quoted 3 lines · Michal Piotrowski
Hi,
This looks like a tickless stuff
Yup.
Quoted 7 lines
0xc0139ea0is intick_nohz_stop_sched_tick(/mnt/md0/devel/linux-git/kernel/time/tick-sched.c:168). 163
164 if (need_resched())
165 goto end;
166
167 cpu =smp_processor_id();
168 BUG_ON(local_softirq_pending());
Hmm, the BUG_ON is inside of an interrupt disabled region, so we should have bailed out early in the need_resched() check above (because we are in the idle task context according to the stack trace).
Is this reproducible ?
tglx
Quoted 15 lines · Thomas Gleixner
Quoted 3 lines · Michal Piotrowski
Hi,
This looks like a tickless stuff
Yup.
Quoted 7 lines
0xc0139ea0is intick_nohz_stop_sched_tick(/mnt/md0/devel/linux-git/kernel/time/tick-sched.c:168). 163
164 if (need_resched())
165 goto end;
166
167 cpu =smp_processor_id();
168 BUG_ON(local_softirq_pending());
Quoted 5 lines
Hmm, the
BUG_ONis inside of an interrupt disabled region, so we should have bailed out early in theneed_resched()check above (because we are in the idle task context according to the stack trace).Is this reproducible ?
Yes.
Quoted 5 lines
tglx
Regards,
Michal
Signature · 5 lines
Michal K. K. Piotrowski LTG - Linux Testers Group (PL) (http://www.stardust.webpages.pl/ltg/) LTG - Linux Testers Group (EN) (http://www.stardust.webpages.pl/linux_testers_group_en/)
Quoted 6 lines · Alex Riesen
Quoted 6 lines
Quoted 5 lines
164 if (
need_resched())
165 goto end;
166
167 cpu =smp_processor_id();
168 BUG_ON(local_softirq_pending());
Quoted 7 lines
Quoted 5 lines
Hmm, the
BUG_ONis inside of an interrupt disabled region, so we should have bailed out early in theneed_resched()check above (because we are in the idle task context according to the stack trace).Is this reproducible ?
Seen this too (Ubuntu, P4/ht-SMT, SATA, typed from screen):
Can you please apply the patch below, so we can at least see, which softirq is pending. This should trigger independently of hrtimers and dynticks. You can keep it compiled in and disable it at the kernel commandline with "nohz=off" and / or "highres=off"
tglx
arch/i386/kernel/process.c +5 -0
@@ -236,6 +236,11 @@ void cpu_idle(void)* Otherwise, idle callbacks can misfire. */ local_irq_disable (); +if (local_softirq_pending () && !need_resched ()) +printk (KERN_ERR +"Idle: local softirq pending: %04x" , +local_softirq_pending ()); +enter_idle ();idle ();__exit_idle ();
kernel/time/tick-sched.c +0 -1
@@ -165,7 +165,6 @@ void tick_nohz_stop_sched_tick(void) goto end; cpu =smp_processor_id (); -BUG_ON (local_softirq_pending ()); now =ktime_get ();/*
Quoted 8 lines · Alex Riesen
Quoted 4 lines
Can you please apply the patch below, so we can at least see, which softirq is pending. This should trigger independently of hrtimers and dynticks. You can keep it compiled in and disable it at the kernel commandline with "nohz=off" and / or "highres=off"
It did, only one time:
Idle: local softirq pending: 0020<6>USB Universal Host Controller Interface driver v3.0
0x20 is the TASKLET_SOFTIRQ. I have no idea yet, how this can happen.
Can you please check, if this happens when you add "nohz=off" to the kernel command line.
tglx
Quoted 16 lines · Alex Riesen
Thomas Gleixner, Sat, Feb 17, 2007 16:14:17 +0100:
Quoted 15 lines · Alex Riesen
Quoted 13 lines
Quoted 11 lines
Quoted 5 lines
164 if (
need_resched())
165 goto end;
166
167 cpu =smp_processor_id();
168 BUG_ON(local_softirq_pending());Hmm, the
BUG_ONis inside of an interrupt disabled region, so we should have bailed out early in theneed_resched()check above (because we are in the idle task context according to the stack trace).Is this reproducible ?
Seen this too (Ubuntu, P4/ht-SMT, SATA, typed from screen):
Quoted 9 lines
Quoted 4 lines
Can you please apply the patch below, so we can at least see, which softirq is pending. This should trigger independently of hrtimers and dynticks. You can keep it compiled in and disable it at the kernel commandline with "nohz=off" and / or "highres=off"
It did, only one time:
Idle: local softirq pending: 0020<6>USB Universal Host Controller Interface driver v3.0
sudo cat /var/log/messages | grep Idle
Feb 17 17:35:34 bitis-gabonica kernel: Idle: local softirq pending: 0020<6>hdd: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33) Feb 17 19:20:01 bitis-gabonica kernel: Idle: local softirq pending:
0020<3>Idle: local softirq pending: 0020<3>Idle: local softirq
pending: 0020<7>PM: Removing info for No Bus:vcs7
cat /proc/interrupts
CPU0 CPU1
0: 232838 0 IO-APIC-edge timer 1: 401 0 IO-APIC-edge i8042 7: 0 0 IO-APIC-edge parport0 8: 1 0 IO-APIC-edge rtc 9: 1 0 IO-APIC-fasteoi acpi 12: 4 0 IO-APIC-edge i8042 14: 319 0 IO-APIC-edge ide0 15: 1612 0 IO-APIC-edge ide1 16: 16494 0 IO-APIC-fasteoi uhci_hcd:usb3, libata 17: 4670 0 IO-APIC-fasteoi uhci_hcd:usb1, uhci_hcd:usb4 18: 0 0 IO-APIC-fasteoi uhci_hcd:usb2 19: 2 0 IO-APIC-fasteoi ehci_hcd:usb5 20: 2822 0 IO-APIC-fasteoi Intel ICH5 21: 2881 0 IO-APIC-fasteoi eth1 22: 58 0 IO-APIC-fasteoi eth0 NMI: 0 0 LOC: 232562 232846 ERR: 0 MIS: 0
I can confirm that this is ICH5 SATA controller problem.
Regards,
Michal
Signature · 5 lines
Michal K. K. Piotrowski LTG - Linux Testers Group (PL) (http://www.stardust.webpages.pl/ltg/) LTG - Linux Testers Group (EN) (http://www.stardust.webpages.pl/linux_testers_group_en/)
Quoted 9 lines · Michal Piotrowski
sudo cat
/var/log/messages| grep IdleFeb 17 17:35:34 bitis-gabonica kernel: Idle: local softirq pending: 0020<6>hdd: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33) Feb 17 19:20:01 bitis-gabonica kernel: Idle: local softirq pending:0020<3>Idle: local softirq pending: 0020<3>Idle: local softirq
pending: 0020<7>PM: Removing info for No Bus:vcs7>I can confirm that this is ICH5 SATA controller problem.
The arch/i386/kernel/process.c part of the patch should apply to 2.6.20 as well. Can you check if the problem is there too ?
tglx
Quoted 5 lines · Alex Riesen
Quoted 2 lines
The
arch/i386/kernel/process.cpart of the patch should apply to 2.6.20 as well. Can you check if the problem is there too ?It does not apply and does not look trivially hackable.
The code forcpu_idlewas introduced in2ff2d3d7"i386: add idle notifier".
Here you go.
tglx
arch/i386/kernel/process.c +7 -0
@@ -189,6 +189,13 @@ void cpu_idle(void)play_dead ();__get_cpu_var (irq_stat).idle_timestamp = jiffies; + +local_irq_disable (); +if (local_softirq_pending () && !need_resched ()) +printk (KERN_ERR +"Idle: local softirq pending: %04x\n" , +local_softirq_pending ()); +local_irq_enable ();idle (); }preempt_enable_no_resched ();
Quoted 5 lines · Alex Riesen
Thomas Gleixner, Sun, Feb 18, 2007 13:36:56 +0100:
Quoted 4 lines · Alex Riesen
Quoted 3 lines
Quoted 2 lines
The
arch/i386/kernel/process.cpart of the patch should apply to 2.6.20 as well. Can you check if the problem is there too ?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
It does not apply and does not look trivially hackable.
The code forcpu_idlewas introduced in2ff2d3d7"i386: add idle notifier".Here you go.
Ok, I confirm the message in vanilla v2.6.20
dmesg | grep Idle
Idle: local softirq pending: 0020
uname -a
Linux bitis-gabonica.linuxtestersgroup.org 2.6.20 #1 SMP PREEMPT Sun
Feb 18 15:20:27 CET 2007 i686 i686 i386 GNU/Linux
Regards,
Michal
Signature · 5 lines
Michal K. K. Piotrowski LTG - Linux Testers Group (PL) (http://www.stardust.webpages.pl/ltg/) LTG - Linux Testers Group (EN) (http://www.stardust.webpages.pl/linux_testers_group_en/)
Michal Piotrowski napisał(a):
Quoted 60 lines · Alex Riesen
Quoted 23 lines
Thomas Gleixner, Sat, Feb 17, 2007 16:14:17 +0100:
Quoted 8 lines · Alex Riesen
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
164 if (
need_resched())
165 goto end;
166
167 cpu =smp_processor_id();
168 BUG_ON(local_softirq_pending());Hmm, the
BUG_ONis inside of an interrupt disabled region, so weshould
Quoted 1 line
Quoted 1 line
Quoted 1 line
have bailed out early in the
need_resched()check above (becausewe are
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
in the idle task context according to the stack trace).
Is this reproducible ?
Seen this too (Ubuntu, P4/ht-SMT, SATA, typed from screen):
Can you please apply the patch below, so we can at least see, which softirq is pending. This should trigger independently of hrtimers and dynticks. You can keep it compiled in and disable it at the kernel commandline with "nohz=off" and / or "highres=off"
Quoted 5 lines
It did, only one time:
Idle: local softirq pending: 0020<6>USB Universal Host Controller Interface driver v3.0
sudo cat
/var/log/messages| grep IdleFeb 17 17:35:34 bitis-gabonica kernel: Idle: local softirq pending: 0020<6>hdd: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33) Feb 17 19:20:01 bitis-gabonica kernel: Idle: local softirq pending:0020<3>Idle: local softirq pending: 0020<3>Idle: local softirq
pending: 0020<7>PM: Removing info for No Bus:vcs7cat
/proc/interrupts
CPU0 CPU10: 232838 0 IO-APIC-edge timer 1: 401 0 IO-APIC-edge i8042 7: 0 0 IO-APIC-edge parport0 8: 1 0 IO-APIC-edge rtc 9: 1 0 IO-APIC-fasteoi acpi 12: 4 0 IO-APIC-edge i8042 14: 319 0 IO-APIC-edge ide0 15: 1612 0 IO-APIC-edge ide1 16: 16494 0 IO-APIC-fasteoi uhci_hcd:usb3, libata 17: 4670 0 IO-APIC-fasteoi uhci_hcd:usb1, uhci_hcd:usb4 18: 0 0 IO-APIC-fasteoi uhci_hcd:usb2 19: 2 0 IO-APIC-fasteoi ehci_hcd:usb5 20: 2822 0 IO-APIC-fasteoi Intel ICH5 21: 2881 0 IO-APIC-fasteoi eth1 22: 58 0 IO-APIC-fasteoi eth0 NMI: 0 0 LOC: 232562 232846 ERR: 0 MIS: 0
Quoted 1 line
I can confirm that this is ICH5 SATA controller problem.
Here is something interesting
cat /var/log/messages | tail -n 300 | grep NOHZ
Feb 20 20:09:39 bitis-gabonica kernel: NOHZ: local_softirq_pending 20 Feb 20 20:09:39 bitis-gabonica kernel: NOHZ: local_softirq_pending 20 Feb 20 21:10:01 bitis-gabonica kernel: NOHZ: local_softirq_pending 20 Feb 20 23:20:01 bitis-gabonica kernel: NOHZ: local_softirq_pending 20 Feb 21 05:10:28 bitis-gabonica kernel: NOHZ: local_softirq_pending 02 Feb 21 05:10:46 bitis-gabonica kernel: NOHZ: local_softirq_pending 02 ^^^^^^^^^
Feb 21 05:10:57 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:11:48 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:12:02 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:12:02 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:12:23 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:12:29 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:13:27 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:13:49 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:13:54 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:14:48 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:15:11 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:15:13 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:15:16 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:16:18 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:16:31 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:17:03 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:17:06 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:17:09 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:17:12 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
Feb 21 05:17:17 bitis-gabonica kernel: NOHZ: local_softirq_pending 20
Feb 21 05:17:18 bitis-gabonica kernel: NOHZ: local_softirq_pending 02
^^^^^^^^23 times in 7 minutes.
Regards,
Michal
--
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)