watchdog kicked in while shrinking inactive list.

DISCUSSION Dave Jones 4 participants 11 messages Sep 25, 2014 – Oct 9, 2014 Memory management

Share & tools

Dave Jones

author

2014-09-25 14:14 UTC #0 lore raw

Got this on a box that had been fuzzing for 12 hours or so. There's also some timer stuff going on htere, so cc'ing the usual suspects.

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
CPU: 3 PID: 16759 Comm: trinity-c25 Not tainted 3.17.0-rc6+ #59
 0000000000000000 00000000ff743b71 ffff880244e06c00 ffffffffa1818ac4
 ffffffffa1c633c0 ffff880244e06c80 ffffffffa18130a3 ffff880200000010
 ffff880244e06c90 ffff880244e06c30 00000000ff743b71 ffffffffa117ead5
Call Trace:
 <NMI>  [<ffffffffa1818ac4>] dump_stack+0x4e/0x7a
 [<ffffffffa18130a3>] panic+0xd7/0x20a
 [<ffffffffa117ead5>] ? __perf_event_header__init_id+0xe5/0xf0
 [<ffffffffa1146220>] ? restart_watchdog_hrtimer+0x50/0x50
 [<ffffffffa1146338>] watchdog_overflow_callback+0x118/0x120
 [<ffffffffa118792c>] __perf_event_overflow+0xac/0x350
 [<ffffffffa1019ece>] ? x86_perf_event_set_period+0xde/0x150
 [<ffffffffa1188844>] perf_event_overflow+0x14/0x20
 [<ffffffffa101fc26>] intel_pmu_handle_irq+0x206/0x410
 [<ffffffffa1018d2b>] perf_event_nmi_handler+0x2b/0x50
 [<ffffffffa1007752>] nmi_handle+0xd2/0x3b0
 [<ffffffffa1007685>] ? nmi_handle+0x5/0x3b0
 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
 [<ffffffffa1007c62>] default_do_nmi+0x72/0x1c0
 [<ffffffffa1007e68>] do_nmi+0xb8/0xf0
 [<ffffffffa18268aa>] end_repeat_nmi+0x1e/0x2e
 [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
 [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
 <<EOE>>  <IRQ>  [<ffffffffa1823ac4>] _raw_spin_unlock_irqrestore+0x24/0x70
 [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
 [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
 [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
 [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
 [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
 [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
 [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
 [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
 [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
 <EOI>  [<ffffffffa1823b42>] ? _raw_spin_unlock_irq+0x32/0x60
 [<ffffffffa11a795b>] shrink_inactive_list+0x1cb/0x620
 [<ffffffffa11a8663>] shrink_lruvec+0x563/0x6b0
 [<ffffffffa119720f>] ? zone_watermark_ok+0x1f/0x30
 [<ffffffffa11a87fe>] shrink_zone+0x4e/0x130
 [<ffffffffa11a905f>] shrink_zones.constprop.63+0x20f/0x410
 [<ffffffffa11a93bb>] try_to_free_pages+0x15b/0x3e0
 [<ffffffffa119bacb>] __alloc_pages_nodemask+0x7eb/0xc20
 [<ffffffffa11e3756>] alloc_pages_current+0x106/0x1e0
 [<ffffffffa11ec905>] ? new_slab+0x2b5/0x390
 [<ffffffffa11ec905>] new_slab+0x2b5/0x390
 [<ffffffffa1817152>] __slab_alloc+0x348/0x56f
 [<ffffffffa11f2116>] ? kmem_cache_alloc+0x256/0x330
 [<ffffffffa16b31dd>] ? sock_alloc_inode+0x1d/0xc0
 [<ffffffffa10cb687>] ? __lock_is_held+0x57/0x80
 [<ffffffffa16b31dd>] ? sock_alloc_inode+0x1d/0xc0
 [<ffffffffa11f2116>] kmem_cache_alloc+0x256/0x330
 [<ffffffffa16b31dd>] sock_alloc_inode+0x1d/0xc0
 [<ffffffffa1223c4d>] alloc_inode+0x1d/0xa0
 [<ffffffffa1225df1>] new_inode_pseudo+0x11/0x60
 [<ffffffffa16b2dac>] sock_alloc+0x1c/0x90
 [<ffffffffa16b3c4d>] sock_create_lite+0x4d/0x90
 [<ffffffffa1709d71>] __netlink_kernel_create+0x71/0x280
 [<ffffffffa16ed337>] diag_net_init+0x47/0x80
 [<ffffffffa16ed2b0>] ? diag_net_exit+0x30/0x30
 [<ffffffffa16c85f1>] ops_init+0x41/0x190
 [<ffffffffa16c87c3>] setup_net+0x83/0x140
 [<ffffffffa16c8d7e>] copy_net_ns+0x7e/0x130
 [<ffffffffa109f279>] create_new_namespaces+0xf9/0x190
 [<ffffffffa109f5fa>] unshare_nsproxy_namespaces+0x5a/0xc0
 [<ffffffffa1077883>] SyS_unshare+0x173/0x330
 [<ffffffffa1824a24>] tracesys+0xdd/0xe2

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Thomas Gleixner

2014-09-25 14:35 UTC #1 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 2 lines · Dave Jones

Got this on a box that had been fuzzing for 12 hours or so. There's also some timer stuff going on htere, so cc'ing the usual suspects.

And it's a hrtimer lockup

Quoted 6 lines

[<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300 [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
<<EOE>> <IRQ> [<ffffffffa1823ac4>] _raw_spin_unlock_irqrestore+0x24/0x70

Quoted 16 lines

 [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
 [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
 [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
 [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
 [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
 [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
 [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
 [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
 [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80

hrtimer_interrupt
tick_sched_timer
tick_sched_handle
update_process_times
irq_work_run
irq_work_run_list
nohz_full_kick_work_func
__tick_nohz_full_check
tick_nohz_restart
hrtimer_cancel

And that hrtimer_cancel is:

static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)

{
	hrtimer_cancel(&ts->sched_timer);

Now, that's really bad because we are in the timer callback of ts->sched_timer. So hrtimer_cancel will loop forever waiting for the callback to complete.

Frederic !?!?

Thanks,

tglx

Frederic Weisbecker

2014-09-25 23:47 UTC #2 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 44 lines · Thomas Gleixner

Quoted 2 lines · Dave Jones
Got this on a box that had been fuzzing for 12 hours or so. There's also some timer stuff going on htere, so cc'ing the usual suspects.
And it's a hrtimer lockup
Quoted 6 lines
[<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300 [<ffffffffa10f95f8>] ? hrtimer_try_to_cancel+0x58/0x1f0 [<ffffffffa10d103d>] ? lock_release+0x1d/0x300
<<EOE>> <IRQ> [<ffffffffa1823ac4>] _raw_spin_unlock_irqrestore+0x24/0x70
Quoted 16 lines
 [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
 [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
 [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
 [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
 [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
 [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
 [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
 [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
 [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
hrtimer_interrupt
tick_sched_timer
tick_sched_handle
update_process_times
irq_work_run
irq_work_run_list
nohz_full_kick_work_func
__tick_nohz_full_check
tick_nohz_restart
hrtimer_cancel
And that hrtimer_cancel is:
static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)

Quoted 8 lines

{
	hrtimer_cancel(&ts->sched_timer);
Now, that's really bad because we are in the timer callback of ts->sched_timer. So hrtimer_cancel will loop forever waiting for the callback to complete.
Frederic !?!?

Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"

I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3

Thanks.

Dave Jones

author

2014-10-02 18:24 UTC #3 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 40 lines · Frederic Weisbecker

Quoted 31 lines
Quoted 8 lines
 [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
 [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
 [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
 [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
 [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
Quoted 8 lines
 [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
 [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
 [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
 [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
hrtimer_interrupt
tick_sched_timer
tick_sched_handle
update_process_times
irq_work_run
irq_work_run_list
nohz_full_kick_work_func
__tick_nohz_full_check
tick_nohz_restart
hrtimer_cancel
And that hrtimer_cancel is:
static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
Quoted 8 lines
{
	hrtimer_cancel(&ts->sched_timer);
Now, that's really bad because we are in the timer callback of ts->sched_timer. So hrtimer_cancel will loop forever waiting for the callback to complete.
Frederic !?!?

Quoted 7 lines

Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3

I'm now hitting this with such regularity that it's preventing me from tracking down other bugs. Can we get this merged soon please ?

Dave

Mike Lothian

2014-10-02 22:01 UTC #4 lore raw

Re: hrtimer deadlock caused by nohz_full

I think this might be the bug I reported here
https://bugzilla.kernel.org/show_bug.cgi?id=84131

Quoted 60 lines · Dave Jones

Quoted 40 lines · Frederic Weisbecker
Quoted 31 lines
Quoted 8 lines
 [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
 [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
 [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
 [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
 [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
Quoted 8 lines
 [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
 [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
 [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
 [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
hrtimer_interrupt
tick_sched_timer
tick_sched_handle
update_process_times
irq_work_run
irq_work_run_list
nohz_full_kick_work_func
__tick_nohz_full_check
tick_nohz_restart
hrtimer_cancel
And that hrtimer_cancel is:
static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
Quoted 8 lines
{
   hrtimer_cancel(&ts->sched_timer);
Now, that's really bad because we are in the timer callback of ts->sched_timer. So hrtimer_cancel will loop forever waiting for the callback to complete.
Frederic !?!?
Quoted 7 lines
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3
I'm now hitting this with such regularity that it's preventing me from tracking down other bugs. Can we get this merged soon please ?
Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Frederic Weisbecker

2014-10-07 15:30 UTC #5 lore raw

Re: hrtimer deadlock caused by nohz_full

In reply to Dave Jones

Quoted 44 lines · Dave Jones

Quoted 42 lines · Frederic Weisbecker
Quoted 31 lines
Quoted 8 lines
 [<ffffffffa10f95f8>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffffa10f97aa>] hrtimer_cancel+0x1a/0x30
 [<ffffffffa110a0e7>] tick_nohz_restart+0x17/0x90
 [<ffffffffa110af38>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffffa110af5e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffffa117c9bf>] irq_work_run_list+0x4f/0x70
 [<ffffffffa117ca0a>] irq_work_run+0x2a/0x60
 [<ffffffffa10f82eb>] update_process_times+0x5b/0x70
Quoted 8 lines
 [<ffffffffa1109dc5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffffa110a0b1>] tick_sched_timer+0x41/0x60
 [<ffffffffa10f8c71>] __run_hrtimer+0x81/0x480
 [<ffffffffa110a070>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffffa10f9b27>] hrtimer_interrupt+0x107/0x260
 [<ffffffffa10331a4>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffffa182734f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffffa182576f>] apic_timer_interrupt+0x6f/0x80
hrtimer_interrupt
tick_sched_timer
tick_sched_handle
update_process_times
irq_work_run
irq_work_run_list
nohz_full_kick_work_func
__tick_nohz_full_check
tick_nohz_restart
hrtimer_cancel
And that hrtimer_cancel is:
static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
Quoted 8 lines
{
	hrtimer_cancel(&ts->sched_timer);
Now, that's really bad because we are in the timer callback of ts->sched_timer. So hrtimer_cancel will loop forever waiting for the callback to complete.
Frederic !?!?
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"

Quoted 8 lines

Quoted 5 lines
I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3
I'm now hitting this with such regularity that it's preventing me from tracking down other bugs. Can we get this merged soon please ?

I asked Thomas to pull but he probably missed it. So I did a formal pull request to Ingo a few days later but he told me that the pull request was made too late as the merge window was too close. I told him it fixes annoying bugs but that was on IRC so this probably got lost.

It will probably be delayed to 3.18
Sorry I don't have much control myself on merging patches upstream.

Dave Jones

author

2014-10-07 17:38 UTC #6 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 11 lines · Frederic Weisbecker

Quoted 10 lines
Quoted 7 lines
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3
I'm now hitting this with such regularity that it's preventing me from tracking down other bugs. Can we get this merged soon please ?

Quoted 7 lines

I asked Thomas to pull but he probably missed it. So I did a formal pull request to Ingo a few days later but he told me that the pull request was made too late as the merge window was too close. I told him it fixes annoying bugs but that was on IRC so this probably got lost.
It will probably be delayed to 3.18
Sorry I don't have much control myself on merging patches upstream.

I got fed up of hitting it this morning and pull this into 3.17, and got a bunch of rejects, as that tree is against rc5. Is there an updated patchset somewhere else?

Dave

Thomas Gleixner

2014-10-07 17:49 UTC #7 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 10 lines · Dave Jones

Quoted 8 lines · Frederic Weisbecker
Quoted 8 lines
Quoted 7 lines
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3

Quoted 7 lines

Quoted 7 lines
Quoted 2 lines
I'm now hitting this with such regularity that it's preventing me from tracking down other bugs. Can we get this merged soon please ?
I asked Thomas to pull but he probably missed it. So I did a formal pull request to Ingo a few days later but he told me that the pull request was made too late as the merge window was too close. I told him it fixes annoying bugs but that was on IRC so this probably got lost.

Aarg. Lemme get this done.

Thanks,

tglx

Dave Jones

author

2014-10-09 02:03 UTC #8 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 15 lines · Thomas Gleixner

Quoted 13 lines · Dave Jones
Quoted 11 lines · Frederic Weisbecker
Quoted 10 lines
Quoted 7 lines
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3
I'm now hitting this with such regularity that it's preventing me from tracking down other bugs. Can we get this merged soon please ?

Quoted 6 lines

Quoted 4 lines
Quoted 4 lines
I asked Thomas to pull but he probably missed it. So I did a formal pull request to Ingo a few days later but he told me that the pull request was made too late as the merge window was too close. I told him it fixes annoying bugs but that was on IRC so this probably got lost.
Aarg. Lemme get this done.

So, time for sad news.
This still doesn't solve the problem for me.

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
CPU: 2 PID: 8193 Comm: kworker/u8:1 Not tainted 3.17.0+ #67
Workqueue: netns cleanup_net
 0000000000000000 000000002a473fb8 ffff880244c06c00 ffffffff9f81ea56
 ffffffff9fc63e00 ffff880244c06c80 ffffffff9f819003 0000000000000010
 ffff880244c06c90 ffff880244c06c30 000000002a473fb8 0000000000000000
Call Trace:
 <NMI>  [<ffffffff9f81ea56>] dump_stack+0x4e/0x7a
 [<ffffffff9f819003>] panic+0xd7/0x20a
 [<ffffffff9f149f30>] ? restart_watchdog_hrtimer+0x50/0x50
 [<ffffffff9f14a048>] watchdog_overflow_callback+0x118/0x120
 [<ffffffff9f18b69c>] __perf_event_overflow+0xac/0x350
 [<ffffffff9f019ece>] ? x86_perf_event_set_period+0xde/0x150
 [<ffffffff9f18c5b4>] perf_event_overflow+0x14/0x20
 [<ffffffff9f01fc26>] intel_pmu_handle_irq+0x206/0x410
 [<ffffffff9f018d2b>] perf_event_nmi_handler+0x2b/0x50
 [<ffffffff9f007752>] nmi_handle+0xd2/0x3b0
 [<ffffffff9f007685>] ? nmi_handle+0x5/0x3b0
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f007c62>] default_do_nmi+0x72/0x1c0
 [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f007e68>] do_nmi+0xb8/0xf0
 [<ffffffff9f82c9ea>] end_repeat_nmi+0x1e/0x2e
 [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 <<EOE>>  <IRQ>  [<ffffffff9f0cf880>] lock_release_holdtime.part.29+0x50/0x160
 [<ffffffff9f0d4f75>] lock_release+0x235/0x300
 [<ffffffff9f829bf4>] _raw_spin_unlock_irqrestore+0x24/0x70
 [<ffffffff9f0fd318>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f0fd4ca>] hrtimer_cancel+0x1a/0x30
 [<ffffffff9f10de07>] tick_nohz_restart+0x17/0x90
 [<ffffffff9f10ec58>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffff9f10ec7e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffff9f1806df>] irq_work_run_list+0x4f/0x70
 [<ffffffff9f18091b>] irq_work_tick+0x2b/0x50
 [<ffffffff9f0fc00b>] update_process_times+0x5b/0x70
 [<ffffffff9f10dae5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffff9f10ddd1>] tick_sched_timer+0x41/0x60
 [<ffffffff9f0fc991>] __run_hrtimer+0x81/0x480
 [<ffffffff9f10dd90>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffff9f0fd847>] hrtimer_interrupt+0x107/0x260
 [<ffffffff9f033274>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffff9f82d48f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffff9f82b8af>] apic_timer_interrupt+0x6f/0x80
 <EOI>  [<ffffffffc0415436>] ? cache_clean+0x66/0x2f0 [sunrpc]
 [<ffffffffc04156f7>] cache_flush+0x37/0x70 [sunrpc]
 [<ffffffffc041577f>] cache_purge+0x4f/0x80 [sunrpc]
 [<ffffffffc040c80a>] svcauth_unix_purge+0xca/0x1c0 [sunrpc]
 [<ffffffffc040c745>] ? svcauth_unix_purge+0x5/0x1c0 [sunrpc]
 [<ffffffffc04d4f72>] nfsd_export_shutdown+0x112/0x250 [nfsd]
 [<ffffffffc04d4e65>] ? nfsd_export_shutdown+0x5/0x250 [nfsd]
 [<ffffffffc04c972a>] nfsd_exit_net+0x1a/0x20 [nfsd]
 [<ffffffff9f6cdf39>] ops_exit_list.isra.1+0x39/0x60
 [<ffffffff9f6ceae0>] cleanup_net+0x100/0x1e0
 [<ffffffff9f097b30>] process_one_work+0x1e0/0x6b0
 [<ffffffff9f097ac7>] ? process_one_work+0x177/0x6b0
 [<ffffffff9f09811b>] worker_thread+0x11b/0x490
 [<ffffffff9f098000>] ? process_one_work+0x6b0/0x6b0
 [<ffffffff9f09ee46>] kthread+0xf6/0x110
 [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250
 [<ffffffff9f82a8ac>] ret_from_fork+0x7c/0xb0
 [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250

Here's what I have on top of Linus' current tree..
http://paste.fedoraproject.org/140505/20075141/raw/

Do I miss some other essential patch ?

Dave

Frederic Weisbecker

2014-10-09 03:30 UTC #9 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 91 lines · Dave Jones

Quoted 15 lines · Thomas Gleixner

Quoted 13 lines · Dave Jones
Quoted 11 lines · Frederic Weisbecker
Quoted 10 lines
Quoted 7 lines
Right, this patchset fixes it: "[PATCH 0/8] nohz: Fix nohz kick irq work on tick v3"
I was about to make the pull request, the branch is acked by peterz. Would you like to pull it? It's all merge window material.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/fixes-v3
I'm now hitting this with such regularity that it's preventing me from tracking down other bugs. Can we get this merged soon please ?

Quoted 6 lines

Quoted 4 lines
Quoted 4 lines
I asked Thomas to pull but he probably missed it. So I did a formal pull request to Ingo a few days later but he told me that the pull request was made too late as the merge window was too close. I told him it fixes annoying bugs but that was on IRC so this probably got lost.
Aarg. Lemme get this done.

So, time for sad news.
This still doesn't solve the problem for me.

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
CPU: 2 PID: 8193 Comm: kworker/u8:1 Not tainted 3.17.0+ #67
Workqueue: netns cleanup_net
 0000000000000000 000000002a473fb8 ffff880244c06c00 ffffffff9f81ea56
 ffffffff9fc63e00 ffff880244c06c80 ffffffff9f819003 0000000000000010
 ffff880244c06c90 ffff880244c06c30 000000002a473fb8 0000000000000000
Call Trace:
 <NMI>  [<ffffffff9f81ea56>] dump_stack+0x4e/0x7a
 [<ffffffff9f819003>] panic+0xd7/0x20a
 [<ffffffff9f149f30>] ? restart_watchdog_hrtimer+0x50/0x50
 [<ffffffff9f14a048>] watchdog_overflow_callback+0x118/0x120
 [<ffffffff9f18b69c>] __perf_event_overflow+0xac/0x350
 [<ffffffff9f019ece>] ? x86_perf_event_set_period+0xde/0x150
 [<ffffffff9f18c5b4>] perf_event_overflow+0x14/0x20
 [<ffffffff9f01fc26>] intel_pmu_handle_irq+0x206/0x410
 [<ffffffff9f018d2b>] perf_event_nmi_handler+0x2b/0x50
 [<ffffffff9f007752>] nmi_handle+0xd2/0x3b0
 [<ffffffff9f007685>] ? nmi_handle+0x5/0x3b0
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f007c62>] default_do_nmi+0x72/0x1c0
 [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f007e68>] do_nmi+0xb8/0xf0
 [<ffffffff9f82c9ea>] end_repeat_nmi+0x1e/0x2e
 [<ffffffff9f0fd318>] ? hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 [<ffffffff9f0cef39>] ? get_lock_stats+0x19/0x60
 <<EOE>>  <IRQ>  [<ffffffff9f0cf880>] lock_release_holdtime.part.29+0x50/0x160
 [<ffffffff9f0d4f75>] lock_release+0x235/0x300
 [<ffffffff9f829bf4>] _raw_spin_unlock_irqrestore+0x24/0x70
 [<ffffffff9f0fd318>] hrtimer_try_to_cancel+0x58/0x1f0
 [<ffffffff9f0fd4ca>] hrtimer_cancel+0x1a/0x30
 [<ffffffff9f10de07>] tick_nohz_restart+0x17/0x90
 [<ffffffff9f10ec58>] __tick_nohz_full_check+0xc8/0xe0
 [<ffffffff9f10ec7e>] nohz_full_kick_work_func+0xe/0x10
 [<ffffffff9f1806df>] irq_work_run_list+0x4f/0x70
 [<ffffffff9f18091b>] irq_work_tick+0x2b/0x50
 [<ffffffff9f0fc00b>] update_process_times+0x5b/0x70
 [<ffffffff9f10dae5>] tick_sched_handle.isra.21+0x25/0x60
 [<ffffffff9f10ddd1>] tick_sched_timer+0x41/0x60
 [<ffffffff9f0fc991>] __run_hrtimer+0x81/0x480
 [<ffffffff9f10dd90>] ? tick_sched_do_timer+0x90/0x90
 [<ffffffff9f0fd847>] hrtimer_interrupt+0x107/0x260
 [<ffffffff9f033274>] local_apic_timer_interrupt+0x34/0x60
 [<ffffffff9f82d48f>] smp_apic_timer_interrupt+0x3f/0x60
 [<ffffffff9f82b8af>] apic_timer_interrupt+0x6f/0x80
 <EOI>  [<ffffffffc0415436>] ? cache_clean+0x66/0x2f0 [sunrpc]
 [<ffffffffc04156f7>] cache_flush+0x37/0x70 [sunrpc]
 [<ffffffffc041577f>] cache_purge+0x4f/0x80 [sunrpc]
 [<ffffffffc040c80a>] svcauth_unix_purge+0xca/0x1c0 [sunrpc]
 [<ffffffffc040c745>] ? svcauth_unix_purge+0x5/0x1c0 [sunrpc]
 [<ffffffffc04d4f72>] nfsd_export_shutdown+0x112/0x250 [nfsd]
 [<ffffffffc04d4e65>] ? nfsd_export_shutdown+0x5/0x250 [nfsd]
 [<ffffffffc04c972a>] nfsd_exit_net+0x1a/0x20 [nfsd]
 [<ffffffff9f6cdf39>] ops_exit_list.isra.1+0x39/0x60
 [<ffffffff9f6ceae0>] cleanup_net+0x100/0x1e0
 [<ffffffff9f097b30>] process_one_work+0x1e0/0x6b0
 [<ffffffff9f097ac7>] ? process_one_work+0x177/0x6b0
 [<ffffffff9f09811b>] worker_thread+0x11b/0x490
 [<ffffffff9f098000>] ? process_one_work+0x6b0/0x6b0
 [<ffffffff9f09ee46>] kthread+0xf6/0x110
 [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250
 [<ffffffff9f82a8ac>] ret_from_fork+0x7c/0xb0
 [<ffffffff9f09ed50>] ? kthread_create_on_node+0x250/0x250

Quoted 4 lines

Here's what I have on top of Linus' current tree..
http://paste.fedoraproject.org/140505/20075141/raw/
Do I miss some other essential patch ?

If you have used the diff in that URL then you've missed:

x86: Tell irq work about self IPI support

Without that the whole patchset has no effect.

Dave Jones

author

2014-10-09 03:50 UTC #10 lore raw

Re: hrtimer deadlock caused by nohz_full

Quoted 7 lines · Frederic Weisbecker

Quoted 1 line
Do I miss some other essential patch ?
If you have used the diff in that URL then you've missed:
x86: Tell irq work about self IPI support
Without that the whole patchset has no effect.

Ok. Thanks, I'll rerun with that.

Dave