Skip to contentSkip to conversation

18rc1 soft lockup

DISCUSSION Dave Jones 3 participants 12 messages Jul 11, 2006 – Jul 14, 2006

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Just saw this during boot of a HT P4 box.

BUG: soft lockup detected on CPU#0!
 [<c04051af>] show_trace_log_lvl+0x54/0xfd
 [<c0405766>] show_trace+0xd/0x10
 [<c0405885>] dump_stack+0x19/0x1b
 [<c0450ec7>] softlockup_tick+0xa5/0xb9
 [<c042d496>] run_local_timers+0x12/0x14
 [<c042d81b>] update_process_times+0x3c/0x61
 [<c04179e0>] smp_apic_timer_interrupt+0x6d/0x75
 [<c0404ada>] apic_timer_interrupt+0x2a/0x30
BUG: soft lockup detected on CPU#1!
 [<c04051af>] show_trace_log_lvl+0x54/0xfd
 [<c0405766>] show_trace+0xd/0x10
 [<c0405885>] dump_stack+0x19/0x1b
 [<c0450ec7>] softlockup_tick+0xa5/0xb9
 [<c042d496>] run_local_timers+0x12/0x14
 [<c042d81b>] update_process_times+0x3c/0x61
 [<c04179e0>] smp_apic_timer_interrupt+0x6d/0x75
 [<c0404ada>] apic_timer_interrupt+0x2a/0x30
BUG: soft lockup detected on CPU#0!
 [<c04051af>] show_trace_log_lvl+0x54/0xfd
 [<c0405766>] show_trace+0xd/0x10
 [<c0405885>] dump_stack+0x19/0x1b
 [<c0450ec7>] softlockup_tick+0xa5/0xb9
 [<c042d496>] run_local_timers+0x12/0x14
 [<c042d81b>] update_process_times+0x3c/0x61
 [<c04179e0>] smp_apic_timer_interrupt+0x6d/0x75
 [<c0404ada>] apic_timer_interrupt+0x2a/0x30

It then continued booting just fine..

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#1 lore raw
Quoted 11 lines · Dave Jones

Just saw this during boot of a HT P4 box.

BUG: soft lockup detected on CPU#0!
 [<c04051af>] show_trace_log_lvl+0x54/0xfd
 [<c0405766>] show_trace+0xd/0x10
 [<c0405885>] dump_stack+0x19/0x1b
 [<c0450ec7>] softlockup_tick+0xa5/0xb9
 [<c042d496>] run_local_timers+0x12/0x14
 [<c042d81b>] update_process_times+0x3c/0x61
 [<c04179e0>] smp_apic_timer_interrupt+0x6d/0x75
 [<c0404ada>] apic_timer_interrupt+0x2a/0x30

That's clocksource_adjust/lost tick bug. Roman's fix landed in Linus' -git yesterday.

thanks
-john

author
#2 lore raw
Quoted 13 lines · john stultz
Quoted 11 lines · Dave Jones

Just saw this during boot of a HT P4 box.

BUG: soft lockup detected on CPU#0!
 [<c04051af>] show_trace_log_lvl+0x54/0xfd
 [<c0405766>] show_trace+0xd/0x10
 [<c0405885>] dump_stack+0x19/0x1b
 [<c0450ec7>] softlockup_tick+0xa5/0xb9
 [<c042d496>] run_local_timers+0x12/0x14
 [<c042d81b>] update_process_times+0x3c/0x61
 [<c04179e0>] smp_apic_timer_interrupt+0x6d/0x75
 [<c0404ada>] apic_timer_interrupt+0x2a/0x30
Quoted 2 lines

That's clocksource_adjust/lost tick bug. Roman's fix landed in Linus' -git yesterday.

Ah, that was actually a .18rc1-git3 tree. I notice a git4 just appeared, I'll try and reproduce with that.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
author
#3 lore raw
Quoted 14 lines · Dave Jones
Quoted 13 lines · john stultz
Quoted 11 lines · Dave Jones

Just saw this during boot of a HT P4 box.

BUG: soft lockup detected on CPU#0!
 [<c04051af>] show_trace_log_lvl+0x54/0xfd
 [<c0405766>] show_trace+0xd/0x10
 [<c0405885>] dump_stack+0x19/0x1b
 [<c0450ec7>] softlockup_tick+0xa5/0xb9
 [<c042d496>] run_local_timers+0x12/0x14
 [<c042d81b>] update_process_times+0x3c/0x61
 [<c04179e0>] smp_apic_timer_interrupt+0x6d/0x75
 [<c0404ada>] apic_timer_interrupt+0x2a/0x30
Quoted 5 lines
Quoted 2 lines

That's clocksource_adjust/lost tick bug. Roman's fix landed in Linus' -git yesterday.

Ah, that was actually a .18rc1-git3 tree. I notice a git4 just appeared, I'll try and reproduce with that.

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#4 lore raw
Quoted 24 lines · Dave Jones
Quoted 14 lines · Dave Jones
Quoted 13 lines · john stultz
Quoted 11 lines · Dave Jones

Just saw this during boot of a HT P4 box.

BUG: soft lockup detected on CPU#0!
 [<c04051af>] show_trace_log_lvl+0x54/0xfd
 [<c0405766>] show_trace+0xd/0x10
 [<c0405885>] dump_stack+0x19/0x1b
 [<c0450ec7>] softlockup_tick+0xa5/0xb9
 [<c042d496>] run_local_timers+0x12/0x14
 [<c042d81b>] update_process_times+0x3c/0x61
 [<c04179e0>] smp_apic_timer_interrupt+0x6d/0x75
 [<c0404ada>] apic_timer_interrupt+0x2a/0x30
Quoted 5 lines
Quoted 2 lines

That's clocksource_adjust/lost tick bug. Roman's fix landed in Linus' -git yesterday.

Ah, that was actually a .18rc1-git3 tree. I notice a git4 just appeared, I'll try and reproduce with that.

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

Quoted 11 lines
BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70

Hmmm.. grumble. Was this on bootup, or after some time period?

I'm looking into it.

thanks for the report
-john

author
#5 lore raw
Quoted 15 lines · john stultz
Quoted 14 lines

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70
Quoted 1 line

Hmmm.. grumble. Was this on bootup, or after some time period?

Right at the end of boot up, between the switch from runlevel 3 to 5.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#6 lore raw

Hi,

Quoted 17 lines · Dave Jones
Quoted 15 lines · john stultz
Quoted 14 lines

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70
Quoted 3 lines
Quoted 1 line

Hmmm.. grumble. Was this on bootup, or after some time period?

Right at the end of boot up, between the switch from runlevel 3 to 5.

When it waits, a SysRq+T might be useful.

bye, Roman

author
#7 lore raw
Quoted 21 lines · Roman Zippel

Hi,

Quoted 17 lines · Dave Jones
Quoted 15 lines · john stultz
Quoted 14 lines

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Hmmm.. grumble. Was this on bootup, or after some time period?

Right at the end of boot up, between the switch from runlevel 3 to 5.

When it waits, a SysRq+T might be useful.

it doesn't wait. this scrolls by within a split second.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#8 lore raw
Quoted 22 lines · Dave Jones
Quoted 21 lines · Roman Zippel

Hi,

Quoted 17 lines · Dave Jones
Quoted 15 lines · john stultz
Quoted 14 lines

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Hmmm.. grumble. Was this on bootup, or after some time period?

Right at the end of boot up, between the switch from runlevel 3 to 5.

When it waits, a SysRq+T might be useful.

it doesn't wait. this scrolls by within a split second.

Huh. I wonder if the x86-64 lost-tick compensation code is being triggered. Could you boot w/ "report_lost_ticks" and see if it spits anything out right before this?

thanks
-john

#9 lore raw
In reply to john stultz

Hi,

Quoted 15 lines · john stultz
Quoted 14 lines

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70
Quoted 3 lines

Hmmm.. grumble. Was this on bootup, or after some time period?

I'm looking into it.

I don't quite understand how this is clock related, soft lockup uses jiffies and there is nothing clock related in the trace???

bye, Roman

#10 lore raw
Quoted 19 lines · Roman Zippel

Hi,

Quoted 15 lines · john stultz
Quoted 14 lines

Just when I thought it had gotten fixed..
2.6.18rc1-git6 this time on x86-64..

BUG: soft lockup detected on CPU#3!

Call Trace:
 [<ffffffff80270865>] show_trace+0xaa/0x23d
 [<ffffffff80270a0d>] dump_stack+0x15/0x17
 [<ffffffff802c44e6>] softlockup_tick+0xd5/0xea
 [<ffffffff80250bea>] run_local_timers+0x13/0x15
 [<ffffffff8029cc1d>] update_process_times+0x4c/0x79
 [<ffffffff8027bfeb>] smp_local_timer_interrupt+0x2b/0x50
 [<ffffffff8027c766>] smp_apic_timer_interrupt+0x58/0x62
 [<ffffffff802628ae>] apic_timer_interrupt+0x6a/0x70
Quoted 6 lines
Quoted 3 lines

Hmmm.. grumble. Was this on bootup, or after some time period?

I'm looking into it.

I don't quite understand how this is clock related, soft lockup uses jiffies and there is nothing clock related in the trace???

Hmmm. Well, its easy to check:

Dave, could you comment out the "clocksource_adjust(...)" line in kernel/timer.c::update_wall_time() just to check if its the same issue?

thanks
-john

author
#11 lore raw
Quoted 7 lines · john stultz
Quoted 2 lines

I don't quite understand how this is clock related, soft lockup uses jiffies and there is nothing clock related in the trace???

Hmmm. Well, its easy to check:

Dave, could you comment out the "clocksource_adjust(...)" line in kernel/timer.c::update_wall_time() just to check if its the same issue?

I'll try, but just like every other bug I've hit together, it's non-deterministic. I'll do a half dozen boots to see if turns up again.

Whatever happened to the good old days of reproducable bugs? :)

Dave

Signature · 1 line
http://www.codemonkey.org.uk