/proc dcache deadlock in do_exit
Hi,
this patch fixes a sles9 system hang in start_this_handle from a customer with some heavy workload where all tasks are waiting on kjournald to commit the transaction, but kjournald waits on t_updates to go down to zero (it never does). This was reported as a lowmem shortage deadlock but when checking the debug data I noticed the VM wasn't under pressure at all (well it was really under vm pressure, because lots of tasks hanged in the VM prune_dcache methods trying to flush dirty inodes, but no task was hanging in GFP_NOFS mode, the holder of the journal handle should have if this was a vm issue in the first place). No task was apparently holding the leftover handle in the committing transaction, so I deduced t_updates was stuck to 1 because a journal_stop was never run by some path (this turned out to be correct). With a debug patch adding proper reverse links and stack trace logging in ext3 deployed in production, I found journal_stop is never run because mark_inode_dirty_sync is called inside release_task called by do_exit. (that was quite fun because I would have never thought about this subtleness, I thought a regular path in ext3 had a bug and it forgot to call journal_stop)
do_exit->release_task->mark_inode_dirty_sync->schedule() (will never come back to run journal_stop)
The reason is that shrink_dcache_parent is racy by design (feature not a bug) and it can do blocking I/O in some case, but the point is that calling shrink_dcache_parent at the last stage of do_exit isn't safe for self-reaping tasks.
I guess the memory pressure of the unbalanced highmem system allowed to trigger this more easily.
Now mainline doesn't have this line in iput (like sles9 has):
if (inode->i_state & I_DIRTY_DELAYED) mark_inode_dirty_sync(inode);
so it will probably not crash with ext3, but for example ext2 implements an I/O-blocking ext2_put_inode that will lead to similar screwups with ext2_free_blocks never coming back and it's definitely wrong to call blocking-IO paths inside do_exit. So this should fix a subtle bug in mainline too (not verified in practice though). The equivalent fix for ext3 is also not verified yet to fix the problem in sles9 but I don't have doubt it will (it usually takes days to crash, so it'll take weeks to be sure).
An alternate fix would be to offload that work to a kernel thread, but I don't think a reschedule for this is worth it, the vm should be able to collect those entries for the synchronous release_task.
fs/proc/base.c +2 -1
@@ -2265,7 +2265,8 @@ static void proc_flush_task_mnt(struct v name.len =snprintf (buf,sizeof (buf),"%d" , pid); dentry =d_hash_and_lookup (mnt->mnt_root, &name);if (dentry) { -shrink_dcache_parent (dentry); +if (!(current->flags &PF_EXITING )) +shrink_dcache_parent (dentry);d_drop (dentry);dput (dentry); }
here the debugging code I written to track this deadlock down if anyone is interested. (I should really have used unsigned long _stack to make life easier).
Index: sles9/fs/jbd/transaction.c
fs/jbd/transaction.c renamed from fs/jbd/transaction.c.~1~ +17 -1
@@ -222,7 +222,19 @@ repeat_locked: handle->h_transaction = transaction; transaction->t_outstanding_credits += nblocks; transaction->t_updates++; - transaction->t_handle_count++; + handle->id = transaction->t_handle_count++; +if (handle->id <NR_HANDLES ) + transaction->handles[handle->id] = handle; + { +char _stack; +char * start = &_stack, * end, * stack; + end = (char *) (((unsigned long ) start+THREAD_SIZE -1 ) & (-THREAD_SIZE )); + stack = (char *) ((unsigned long ) start +2048 ); +if (stack > end) + stack = end; +memcpy (handle->stack, start, stack-start); + } +jbd_debug (4 ,"Handle %p given %d credits (total %d, free %d)\n" , handle, nblocks, transaction->t_outstanding_credits,__log_space_left (journal));
@@ -398,6 +410,8 @@ int journal_restart(handle_t *handle, inspin_lock (&transaction->t_handle_lock); transaction->t_outstanding_credits -= handle->h_buffer_credits; transaction->t_updates--; +if (handle->id <NR_HANDLES ) + transaction->handles[handle->id] =NULL ;if (!transaction->t_updates)wake_up (&journal->j_wait_updates);
@@ -1362,6 +1376,8 @@ int journal_stop(handle_t *handle)spin_lock (&transaction->t_handle_lock); transaction->t_outstanding_credits -= handle->h_buffer_credits; transaction->t_updates--; +if (handle->id <NR_HANDLES ) + transaction->handles[handle->id] =NULL ;if (!transaction->t_updates) {wake_up (&journal->j_wait_updates);if (journal->j_barrier_count)
Index: sles9/include/linux/jbd.h
include/linux/jbd.h renamed from include/linux/jbd.h.~1~ +4 -0
@@ -386,6 +386,8 @@ struct handle_s {/* Which compound transaction is this update a part of? */ transaction_t *h_transaction; +int id; +char stack[2048 ];/* Number of remaining buffers we are allowed to dirty: */ int h_buffer_credits;
@@ -569,6 +571,8 @@ struct transaction_s* How many handles used this transaction? [t_handle_lock] */ int t_handle_count; +#define NR_HANDLES 400 +handle_t * handles[NR_HANDLES ];/* * Protects the callback list
Quoted 21 lines · On Tue, 27 Nov 2007 14:20:22 +0100 Andrea Arcangeli <[email protected]> wrote:
Hi,
this patch fixes a sles9 system hang in
start_this_handlefrom a customer with some heavy workload where all tasks are waiting on kjournald to commit the transaction, but kjournald waits ont_updatesto go down to zero (it never does). This was reported as a lowmem shortage deadlock but when checking the debug data I noticed the VM wasn't under pressure at all (well it was really under vm pressure, because lots of tasks hanged in the VMprune_dcachemethods trying to flush dirty inodes, but no task was hanging inGFP_NOFSmode, the holder of the journal handle should have if this was a vm issue in the first place). No task was apparently holding the leftover handle in the committing transaction, so I deducedt_updateswas stuck to 1 because ajournal_stopwas never run by some path (this turned out to be correct). With a debug patch adding proper reverse links and stack trace logging in ext3 deployed in production, I foundjournal_stopis never run becausemark_inode_dirty_syncis called insiderelease_taskcalled bydo_exit. (that was quite fun because I would have never thought about this subtleness, I thought a regular path in ext3 had a bug and it forgot to calljournal_stop)
Quoted 2 lines
do_exit->release_task->mark_inode_dirty_sync->schedule()(will never come back to runjournal_stop)
I don't see why the schedule() will not return? Because the task has PF_EXITING set? Doesn't TASK_DEAD do that?
Quoted 32 lines
The reason is that
shrink_dcache_parentis racy by design (feature not a bug) and it can do blocking I/O in some case, but the point is that callingshrink_dcache_parentat the last stage ofdo_exitisn't safe for self-reaping tasks.I guess the memory pressure of the unbalanced highmem system allowed to trigger this more easily.
Now mainline doesn't have this line in iput (like sles9 has):
if (inode->i_state & I_DIRTY_DELAYED) mark_inode_dirty_sync(inode);so it will probably not crash with ext3, but for example ext2 implements an I/O-blocking
ext2_put_inodethat will lead to similar screwups withext2_free_blocksnever coming back and it's definitely wrong to call blocking-IO paths insidedo_exit. So this should fix a subtle bug in mainline too (not verified in practice though). The equivalent fix for ext3 is also not verified yet to fix the problem in sles9 but I don't have doubt it will (it usually takes days to crash, so it'll take weeks to be sure).An alternate fix would be to offload that work to a kernel thread, but I don't think a reschedule for this is worth it, the vm should be able to collect those entries for the synchronous
release_task.Signed-off-by Andrea Arcangeli <[email protected]>
fs/proc/base.c static void proc_flush_task_mnt(struct v · line 2272
Quoted 9 lines
fs/proc/base.c +2
-1@@ -2265,7 +2265,8 @@ static void proc_flush_task_mnt(struct v name.len =snprintf (buf,sizeof (buf),"%d" , pid); dentry =d_hash_and_lookup (mnt->mnt_root, &name);if (dentry) { -shrink_dcache_parent (dentry); +if (!(current->flags &PF_EXITING )) +shrink_dcache_parent (dentry);d_drop (dentry);dput (dentry); }
What are the implications of not running shrink_dcache_parent() on the exit path sometimes? We'll leave procfs stuff behind? Will they be reaped by memory pressure later on?
Quoted 2 lines · Andrew Morton
I don't see why the
schedule()will not return? Because the task hasPF_EXITINGset? Doesn'tTASK_DEADdo that?
Ouch, I assumed you couldn't sleep safely anymore in release_task given it's the function that will free the task structure itself and there was no preempt related action anywhere close to it! delayed_put_task_struct can be called if a quiescent point is reached and any scheduling would exactly allow it to run (it requires quite a bit of a race, with local irq triggering a reschedule and the timer irq invoking the tasklet to run to free the task struct before do_exit finishes and all other cpus in quiescent state too).
So a corollary question is how can it be safe to call preempt_disable() after call_rcu(delayed_put_task_struct)?
Back in sles9 preempt_disable was implemented as _raw_write_unlock(&tasklist_lock) and it happened _before_ release_task, and scheduling there wouldn't return because PF_DEAD was already set. If mainline can come back, it will crash for a different reason because the task struct is long gone by the time release_task+schedule() runs. Either ways, still a kernel crashing bug there is. Or is there some magic that prevents call_rcu + schedule to invoke the rcu callback?
So you may need to apply this one too (this one is needed to fix the second bug, my previous patch is needed after applying this one):
kernel/exit.c +7 -1
@@ -841,6 +841,13 @@ static void exit_notify(struct task_struwrite_unlock_irq (&tasklist_lock); +/* +* Task struct can go away at the first schedule if this was a +* self reaping task. Scheduling is forbidden until we set +* the state to TASK_DEAD. +*/ +preempt_disable (); +/* If the process is dead, release it - nobody will wait for it */ if (state ==EXIT_DEAD )release_task (tsk);
@@ -1042,7 +1049,6 @@ fastcall NORET_TYPE void do_exit(long coif (tsk->splice_pipe)__free_pipe_info (tsk->splice_pipe); -preempt_disable ();/* causes final put_task_struct in finish_task_switch(). */ tsk->state =TASK_DEAD ;
Quoted 3 lines
What are the implications of not running
shrink_dcache_parent()on the exit path sometimes? We'll leave procfs stuff behind? Will they be reaped by memory pressure later on?
Yes.
Quoted 24 lines · Andrew Morton
Quoted 21 lines · On Tue, 27 Nov 2007 14:20:22 +0100 Andrea Arcangeli <[email protected]> wrote:
Hi,
this patch fixes a sles9 system hang in
start_this_handlefrom a customer with some heavy workload where all tasks are waiting on kjournald to commit the transaction, but kjournald waits ont_updatesto go down to zero (it never does). This was reported as a lowmem shortage deadlock but when checking the debug data I noticed the VM wasn't under pressure at all (well it was really under vm pressure, because lots of tasks hanged in the VMprune_dcachemethods trying to flush dirty inodes, but no task was hanging inGFP_NOFSmode, the holder of the journal handle should have if this was a vm issue in the first place). No task was apparently holding the leftover handle in the committing transaction, so I deducedt_updateswas stuck to 1 because ajournal_stopwas never run by some path (this turned out to be correct). With a debug patch adding proper reverse links and stack trace logging in ext3 deployed in production, I foundjournal_stopis never run becausemark_inode_dirty_syncis called insiderelease_taskcalled bydo_exit. (that was quite fun because I would have never thought about this subtleness, I thought a regular path in ext3 had a bug and it forgot to calljournal_stop)
Quoted 5 lines
Quoted 2 lines
do_exit->release_task->mark_inode_dirty_sync->schedule()(will never come back to runjournal_stop)I don't see why the
schedule()will not return? Because the task hasPF_EXITINGset? Doesn'tTASK_DEADdo that?
Yes, why do we not come back from schedule?
If we are not allowed to schedule after setting PF_EXITING before we set TASK_DEAD that entire code path sounds brittle and error prone.
Quoted 3 lines
What are the implications of not running
shrink_dcache_parent()on the exit path sometimes? We'll leave procfs stuff behind? Will they be reaped by memory pressure later on?
It should. I think the reaping is just an optimization. Because we know we will never need those dentries again, and we can pin them by open directories or opening files. What I don't know off the top of my head is if there is a d_drop equivalent going on that might be a problem if we don't address it.
Eric
Quoted 25 lines · Andrea Arcangeli
Quoted 2 lines · Andrew Morton
I don't see why the
schedule()will not return? Because the task hasPF_EXITINGset? Doesn'tTASK_DEADdo that?Ouch, I assumed you couldn't sleep safely anymore in
release_taskgiven it's the function that will free the task structure itself and there was no preempt related action anywhere close to it!delayed_put_task_structcan be called if a quiescent point is reached and any scheduling would exactly allow it to run (it requires quite a bit of a race, with local irq triggering a reschedule and the timer irq invoking the tasklet to run to free the task struct beforedo_exitfinishes and all other cpus in quiescent state too).So a corollary question is how can it be safe to call
preempt_disable()aftercall_rcu(delayed_put_task_struct)?Back in sles9
preempt_disablewas implemented as_raw_write_unlock(&tasklist_lock)and it happened _before_release_task, and scheduling there wouldn't return becausePF_DEADwas already set. If mainline can come back, it will crash for a different reason because the task struct is long gone by the time release_task+schedule() runs. Either ways, still a kernel crashing bug there is. Or is there some magic that preventscall_rcu+ schedule to invoke the rcu callback?
Quoted 2 lines
So you may need to apply this one too (this one is needed to fix the second bug, my previous patch is needed after applying this one):
thinking what happened once already, I think this would be more debuggable but maybe not... dunno.
kernel/exit.c +8 -3
@@ -841,6 +841,14 @@ static void exit_notify(struct task_struwrite_unlock_irq (&tasklist_lock); +/* +* Task struct can go away at the first schedule if this was a +* self reaping task after calling release_task. Scheduling is +* forbidden until do_exit finishes. +*/ +preempt_disable (); + tsk->state =TASK_DEAD ; +/* If the process is dead, release it - nobody will wait for it */ if (state ==EXIT_DEAD )release_task (tsk);
@@ -1042,10 +1050,7 @@ fastcall NORET_TYPE void do_exit(long coif (tsk->splice_pipe)__free_pipe_info (tsk->splice_pipe); -preempt_disable ();/* causes final put_task_struct in finish_task_switch(). */ - tsk->state =TASK_DEAD ; -schedule ();BUG ();/* Avoid "noreturn function does return". */
Quoted 17 lines · Andrea Arcangeli
Quoted 2 lines · Andrew Morton
I don't see why the
schedule()will not return? Because the task hasPF_EXITINGset? Doesn'tTASK_DEADdo that?Ouch, I assumed you couldn't sleep safely anymore in
release_taskgiven it's the function that will free the task structure itself and there was no preempt related action anywhere close to it!delayed_put_task_structcan be called if a quiescent point is reached and any scheduling would exactly allow it to run (it requires quite a bit of a race, with local irq triggering a reschedule and the timer irq invoking the tasklet to run to free the task struct beforedo_exitfinishes and all other cpus in quiescent state too).So a corollary question is how can it be safe to call
preempt_disable()aftercall_rcu(delayed_put_task_struct)?
Quoted 8 lines
Back in sles9
preempt_disablewas implemented as_raw_write_unlock(&tasklist_lock)and it happened _before_release_task, and scheduling there wouldn't return becausePF_DEADwas already set. If mainline can come back, it will crash for a different reason because the task struct is long gone by the time release_task+schedule() runs. Either ways, still a kernel crashing bug there is. Or is there some magic that preventscall_rcu+ schedule to invoke the rcu callback?
I don't quite see where it comes from but there is another reference on the task struct held by the scheduler That we don't drop until finish_task_switch with exit_state == TASK_DEAD.
So since we have one additional reference the task_struct won't get freed.
We don't set TASK_DEAD until much later.
Quoted 2 lines
So you may need to apply this one too (this one is needed to fix the second bug, my previous patch is needed after applying this one):
No. We should be fine.
In fact it looks like this is just a sles9 issue and mainline is fine, as we can safely schedule until just before the end of do_exit.
Eric
Quoted 6 lines · Andrea Arcangeli
Quoted 3 lines
So you may need to apply this one too (this one is needed to fix the second bug, my previous patch is needed after applying this one):
thinking what happened once already, I think this would be more debuggable but maybe not... dunno.
Please god no. The inability to schedule is a bug that apparently
was fixed after sles9 forked off.
In mainline it is safe to sleep until the very end of do_exit and your suggest patch breaks that.
None of your patches are necessary for mainline.
I believe the commit below is the one that really fixed this issue, although it may have been something earlier. But it may have been earlier, and there have been some significant clean ups since then.
Quoted 20 lines
commit
00732b345b940de766fefaf1297ce26a67bdcea9Author: mingo <mingo> Date: Tue Oct 19 06:12:06 2004 +0000[PATCH] fix & clean up zombie/dead task handling & preemption
This patch fixes all the preempt-after-task->state-is-TASK_DEAD problems we had. Right now, the moment procfs does a
down()that sleeps inproc_pid_flush()[it could] ourTASK_DEADstate is zapped and we might be back toTASK_RUNNINGto and we trigger this assert:
schedule();
BUG();
/* Avoid "noreturn function does return". */ for (;;) ;I have split out
TASK_ZOMBIEandTASK_DEADinto a separatep->exit_statefield, to allow the detaching of exit-signal/parent/wait-handling from descheduling a dead task. Dead-task freeing is done viaPF_DEAD.
Quoted 9 lines
Tested the patch on x86 SMP and UP, but all architectures should work fine.
Signed-off-by Ingo Molnar <[email protected]>Signed-off-by Andrew Morton <[email protected]>Signed-off-by Linus Torvalds <[email protected]>BKrev: 4174b036IkpU2-XeLFkjqH681u4Pyg
Eric
Quoted 2 lines · Andrew Morton
I don't see why the
schedule()will not return? Because the task hasPF_EXITINGset? Doesn'tTASK_DEADdo that?
This appears to be a work around for an old bug only present in sles9.
It looks like it has been safe to schedule in release task for years in mainline, and that is simply much more robust.
Eric
Quoted 9 lines · Eric W. Biederman
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 27 Nov 2007 14:20:22 +0100 Andrea Arcangeli <[email protected]> wrote:
do_exit->release_task->mark_inode_dirty_sync->schedule()(will never come back to runjournal_stop)
Quoted 8 lines
Quoted 2 lines
I don't see why the
schedule()will not return? Because the task hasPF_EXITINGset? Doesn'tTASK_DEADdo that?Yes, why do we not come back from schedule?
If we are not allowed to schedule after setting
PF_EXITINGbefore we setTASK_DEADthat entire code path sounds brittle and error prone.
Yes, it is fine to schedule after release_task(). As Eric pointed out, we don't race with call_rcu(delayed_put_task_struct), scheduler has another reference
dup_task_struct: /* One for us, one for whoever does the "release_task()" (usually parent) */ atomic_set(&tsk->usage,2);
However, with CONFIG_HOTPLUG_CPU we do have the problem here, but this is off-topic. Preemption is fine, but deactivate_task() is not. We can't migrate the deactivated released task from the dead CPU.
migrate_live_tasks() can't find the task after __unhash_process()
migrate_dead_tasks() doesn't see it after deactivate_task().
And afaics try_to_wake_up() doesn't necessary change task_cpu() if it is offline.
No? But again, this is offtopic even if I am right.
Oleg.