jbd commit code deadloop when installing Linux
PATCH 1/1 jbd commit code deadloop when installing Linux
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.
That is because anaconda is invoked from linuxrc in initrd when system_state is still SYSTEM_BOOTING.
Thus the cond_resched checks in journal_commit_transaction will always return 1 without actually schedule, then the system fall into deadloop.
The patch will fix the issue.
Zou Nan hai
fs/jbd/commit.c +22 -4
@@ -625,8 +625,17 @@ wait_for_iobuf:wait_on_buffer (bh); goto wait_for_iobuf; } -if (cond_resched ()) - goto wait_for_iobuf; +if (cond_resched ()) { +if (commit_transaction->t_iobuf_list !=NULL ) { + jh = commit_transaction->t_iobuf_list->b_tprev; + bh =jh2bh (jh); +if (buffer_locked (bh)) { +wait_on_buffer (bh); + goto wait_for_iobuf; + } + }else +break ; + }if (unlikely (!buffer_uptodate (bh))) err = -EIO ;
@@ -681,8 +690,17 @@ wait_for_iobuf:wait_on_buffer (bh); goto wait_for_ctlbuf; } -if (cond_resched ()) - goto wait_for_ctlbuf; +if (cond_resched ()) { +if (commit_transaction->t_log_list !=NULL ) { + jh = commit_transaction->t_log_list->b_tprev; + bh =jh2bh (jh); +if (buffer_locked (bh)) { +wait_on_buffer (bh); + goto wait_for_ctlbuf; + } + }else +break ; + }if (unlikely (!buffer_uptodate (bh))) err = -EIO ;
Quoted 10 lines · On 28 Jun 2006 12:48:43 +0800 Zou Nan hai <[email protected]> wrote:
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.
That's a bug in cond_resched().
Something like this..
kernel/sched.c renamed from kernel/sched.c~cond_resched-fix +6 -7
@@ -4454,7 +4454,7 @@ asmlinkage long sys_sched_yield(void)return 0 ; } -static inline void __cond_resched (void ) +static inline int __cond_resched (void ) {#ifdef CONFIG_DEBUG_SPINLOCK_SLEEP __might_sleep (__FILE__, __LINE__);
@@ -4465,22 +4465,21 @@ static inline void __cond_resched(void)* cond_resched() call. */ if (unlikely (preempt_count ())) -return ; +return 0 ;if (unlikely (system_state !=SYSTEM_RUNNING )) -return ; +return 0 ;do {add_preempt_count (PREEMPT_ACTIVE );schedule ();sub_preempt_count (PREEMPT_ACTIVE ); }while (need_resched ()); +return 1 ; } int __sched cond_resched(void) { -if (need_resched ()) { -__cond_resched (); -return 1 ; - } +if (need_resched ()) +return __cond_resched ();return 0 ; }EXPORT_SYMBOL (cond_resched);
_
Ingo Molnar
Acked-byQuoted 6 lines · Andrew Morton
Quoted 6 lines
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.
[ argh ...! ]
Quoted 7 lines
Quoted 3 lines
Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.That's a bug in
cond_resched().Something like this..
Ingo
Quoted 11 lines · On Wed, 28 Jun 2006 08:38:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 6 lines
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.[ argh ...! ]
That's what I thought ;)
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.That's a bug in
cond_resched().Something like this..
Acked-by Ingo Molnar <[email protected]>
Thanks. Zou, it'd be great if you could test this in your setup, please. I've tagged it as 2.6.17.x material.
Quoted 23 lines · Andrew Morton
Quoted 11 lines · On Wed, 28 Jun 2006 08:38:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 6 lines
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.[ argh ...! ]
That's what I thought ;)
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.That's a bug in
cond_resched().
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Something like this..
Acked-by Ingo Molnar <[email protected]>Thanks. Zou, it'd be great if you could test this in your setup, please. I've tagged it as 2.6.17.x material.
Andrew,
I am building the env to test.
The patch was my original idea, but I was afraid of breaking any code
that rely on the OLD wrong cond_sched semantic. However later I did a grep found that there is very few code that checks the return value of cond_resched. So the patch should be safe.
However I think cond_resched_lock and cond_resched_softirq also need fix to make the semantic consistent.
Please check the following patch.
Zou Nan hai
kernel/sched.c +10 -13
@@ -4044,7 +4044,7 @@ asmlinkage long sys_sched_yield(void)return 0 ; } -static inline void __cond_resched (void ) +static inline int __cond_resched (void ) {/* * The BKS might be reacquired before we have dropped
@@ -4052,22 +4052,21 @@ static inline void __cond_resched(void)* cond_resched() call. */ if (unlikely (preempt_count ())) -return ; +return 0 ;if (unlikely (system_state !=SYSTEM_RUNNING )) -return ; +return 0 ;do {add_preempt_count (PREEMPT_ACTIVE );schedule ();sub_preempt_count (PREEMPT_ACTIVE ); }while (need_resched ()); +return 1 ; } int __sched cond_resched(void) { -if (need_resched ()) { -__cond_resched (); -return 1 ; - } +if (need_resched ()) +return __cond_resched ();return 0 ; }
@@ -4094,8 +4093,7 @@ int cond_resched_lock(spinlock_t *lock)if (need_resched ()) {_raw_spin_unlock (lock);preempt_enable_no_resched (); -__cond_resched (); - ret =1 ; + ret |=__cond_resched ();spin_lock (lock); }return ret;
@@ -4106,14 +4104,13 @@ EXPORT_SYMBOL(cond_resched_lock); int __sched cond_resched_softirq(void) {BUG_ON (!in_softirq ()); - +int ret =0 ;if (need_resched ()) {__local_bh_enable (); -__cond_resched (); + ret =__cond_resched ();local_bh_disable (); -return 1 ; } -return 0 ; +return ret; }EXPORT_SYMBOL (cond_resched_softirq);
Quoted 29 lines · On 28 Jun 2006 13:46:22 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 28 lines · Andrew Morton
Quoted 11 lines · On Wed, 28 Jun 2006 08:38:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 6 lines
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.[ argh ...! ]
That's what I thought ;)
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.That's a bug in
cond_resched().Something like this..
Acked-by Ingo Molnar <[email protected]>
Quoted 7 lines
Quoted 2 lines
Thanks. Zou, it'd be great if you could test this in your setup, please. I've tagged it as 2.6.17.x material.
Andrew,
I am building the env to test.
The patch was my original idea, but I was afraid of breaking any code
that rely on the OLD wrongcond_schedsemantic.
We prefer the "right" fix, however painful or risky that might be.
Quoted 3 lines
However later I did a
grep found that there is very few code that checks the return value ofcond_resched. So the patch should be safe.
Hope so.
Quoted 5 lines
However I think
cond_resched_lockandcond_resched_softirqalso need fix to make the semantic consistent.Please check the following patch.
Ah. I think the return value from these functions should mean "something disruptive happened", if you like.
See, the callers of cond_resched_lock() aren't interested in whether cond_resched_lock() actually called schedule(). They want to know whether cond_resched_lock() dropped the lock. Because if the lock was dropped, the caller needs to take some special action, regardless of whether schedule() was finally called.
So I think the patch I queued is OK, agree?
Quoted 57 lines · Andrew Morton
Quoted 29 lines · On 28 Jun 2006 13:46:22 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 28 lines · Andrew Morton
Quoted 11 lines · On Wed, 28 Jun 2006 08:38:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 6 lines
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.[ argh ...! ]
That's what I thought ;)
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.That's a bug in
cond_resched().Something like this..
Acked-by Ingo Molnar <[email protected]>Quoted 7 lines
Quoted 2 lines
Thanks. Zou, it'd be great if you could test this in your setup, please. I've tagged it as 2.6.17.x material.
Andrew,
I am building the env to test.
The patch was my original idea, but I was afraid of breaking any code
that rely on the OLD wrongcond_schedsemantic.We prefer the "right" fix, however painful or risky that might be.
Quoted 3 lines
However later I did a
grep found that there is very few code that checks the return value ofcond_resched. So the patch should be safe.Hope so.
Quoted 5 lines
However I think
cond_resched_lockandcond_resched_softirqalso need fix to make the semantic consistent.Please check the following patch.
Ah. I think the return value from these functions should mean "something disruptive happened", if you like.
Quoted 7 lines
See, the callers of
cond_resched_lock()aren't interested in whethercond_resched_lock()actually calledschedule(). They want to know whethercond_resched_lock()dropped the lock. Because if the lock was dropped, the caller needs to take some special action, regardless of whetherschedule()was finally called.So I think the patch I queued is OK, agree?
I am afraid the code like cond_resched_lock check infs/jbd/checkpoint.c log_do_checkpoint may fall into endless retry in some condition, will it?
Though I have not encountered that.
Zou Nan hai
Quoted 15 lines · On 28 Jun 2006 14:02:57 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 15 lines
Quoted 5 lines
However I think
cond_resched_lockandcond_resched_softirqalso need fix to make the semantic consistent.Please check the following patch.
Ah. I think the return value from these functions should mean "something disruptive happened", if you like.
See, the callers of
cond_resched_lock()aren't interested in whethercond_resched_lock()actually calledschedule(). They want to know whethercond_resched_lock()dropped the lock. Because if the lock was dropped, the caller needs to take some special action, regardless of whetherschedule()was finally called.
Quoted 5 lines
Quoted 1 line
So I think the patch I queued is OK, agree?
I am afraid the code like
cond_resched_lockcheck infs/jbd/checkpoint.clog_do_checkpointmay fall into endless retry in some condition, will it?
Oh crap, yes. If need_resched() and system_state==SYSTEM_BOOTING then cond_resched_lock() will drop the lock but won't schedule. So it'll return true but won't clear need_resched() and the caller will lock up.
So if cond_resched_foo() ends up dropping the lock it _must_ call schedule() to clear need_resched().
So, how about this (it needs some code comments!)
kernel/sched.c renamed from kernel/sched.c~cond_resched-fix +12 -11
@@ -4386,6 +4386,15 @@ asmlinkage long sys_sched_yield(void)return 0 ; } +static inline int __resched_legal (void ) +{ +if (unlikely (preempt_count ())) +return 0 ; +if (unlikely (system_state !=SYSTEM_RUNNING )) +return 0 ; +return 1 ; +} +static inline void __cond_resched (void ) {#ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
@@ -4396,10 +4405,6 @@ static inline void __cond_resched(void)* PREEMPT_ACTIVE, which could trigger a second * cond_resched() call. */ -if (unlikely (preempt_count ())) -return ; -if (unlikely (system_state !=SYSTEM_RUNNING )) -return ; do { add_preempt_count(PREEMPT_ACTIVE); schedule();
@@ -4409,13 +4414,12 @@ static inline void __cond_resched(void) int __sched cond_resched(void) { -if (need_resched ()) { +if (need_resched () &&__resched_legal ()) {__cond_resched ();return 1 ; }return 0 ; } -EXPORT_SYMBOL (cond_resched);/*
@@ -4436,7 +4440,7 @@ int cond_resched_lock(spinlock_t *lock) ret =1 ;spin_lock (lock); } -if (need_resched ()) { +if (need_resched () &&__resched_legal ()) {_raw_spin_unlock (lock);preempt_enable_no_resched ();__cond_resched ();
@@ -4445,14 +4449,13 @@ int cond_resched_lock(spinlock_t *lock) }return ret; } -EXPORT_SYMBOL (cond_resched_lock); int __sched cond_resched_softirq(void) {BUG_ON (!in_softirq ()); -if (need_resched ()) { +if (need_resched () &&__resched_legal ()) {__local_bh_enable ();__cond_resched ();local_bh_disable ();
@@ -4460,10 +4463,8 @@ int __sched cond_resched_softirq(void) }return 0 ; } -EXPORT_SYMBOL (cond_resched_softirq); -/** * yield - yield the current processor to other threads. *
_
Quoted 24 lines · Andrew Morton
Quoted 15 lines · On 28 Jun 2006 14:02:57 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 15 lines
Quoted 5 lines
However I think
cond_resched_lockandcond_resched_softirqalso need fix to make the semantic consistent.Please check the following patch.
Ah. I think the return value from these functions should mean "something disruptive happened", if you like.
See, the callers of
cond_resched_lock()aren't interested in whethercond_resched_lock()actually calledschedule(). They want to know whethercond_resched_lock()dropped the lock. Because if the lock was dropped, the caller needs to take some special action, regardless of whetherschedule()was finally called.Quoted 5 lines
Quoted 1 line
So I think the patch I queued is OK, agree?
I am afraid the code like
cond_resched_lockcheck infs/jbd/checkpoint.clog_do_checkpointmay fall into endless retry in some condition, will it?
Quoted 10 lines
Oh crap, yes. If
need_resched()and system_state==SYSTEM_BOOTING thencond_resched_lock()will drop the lock but won't schedule. So it'll return true but won't clearneed_resched()and the caller will lock up.So if
cond_resched_foo()ends up dropping the lock it _must_ callschedule()to clearneed_resched().So, how about this (it needs some code comments!)
The patch works for the install test env.
However I still have some concern on cond_resched_lock(), on an UP
kernel it will return 1 if schedule happen, but actually it does not drop any lock, that semantic seems to be different to SMP kernel.
Though the only code I found that checks the return value of cond_resched_lock is in checkpoint.c...
Zou Nan hai
Quoted 29 lines · On 28 Jun 2006 14:50:29 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 28 lines · Andrew Morton
Quoted 15 lines · On 28 Jun 2006 14:02:57 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 15 lines
Quoted 5 lines
However I think
cond_resched_lockandcond_resched_softirqalso need fix to make the semantic consistent.Please check the following patch.
Ah. I think the return value from these functions should mean "something disruptive happened", if you like.
See, the callers of
cond_resched_lock()aren't interested in whethercond_resched_lock()actually calledschedule(). They want to know whethercond_resched_lock()dropped the lock. Because if the lock was dropped, the caller needs to take some special action, regardless of whetherschedule()was finally called.Quoted 5 lines
Quoted 1 line
So I think the patch I queued is OK, agree?
I am afraid the code like
cond_resched_lockcheck infs/jbd/checkpoint.clog_do_checkpointmay fall into endless retry in some condition, will it?Oh crap, yes. If
need_resched()and system_state==SYSTEM_BOOTING thencond_resched_lock()will drop the lock but won't schedule. So it'll return true but won't clearneed_resched()and the caller will lock up.
Quoted 8 lines
Quoted 6 lines
So if
cond_resched_foo()ends up dropping the lock it _must_ callschedule()to clearneed_resched().So, how about this (it needs some code comments!)
The patch works for the install test env.
Thanks.
Quoted 3 lines
However I still have some concern on
cond_resched_lock(), on an UP
kernel it will return 1 if schedule happen, but actually it does not drop any lock, that semantic seems to be different to SMP kernel.
That's OK (I think - I don't have a good track record in this thread).
If the kernel is non-preemptible and UP, we want to return true from cond_resched_foo() if we called schedule(). Because schedule() might allow a different thread into the kernel which might modify the locked data.
And if the kernel is preemptible and UP, we want to return true from cond_resched_foo() if we dropped the lock, because that internally does a preempt_enable().
And the patch (hopefully) satisfies those requirements. Does that all sound solid?
Quoted 53 lines · Andrew Morton
Quoted 29 lines · On 28 Jun 2006 14:50:29 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 28 lines · Andrew Morton
Quoted 15 lines · On 28 Jun 2006 14:02:57 +0800 Zou Nan hai <[email protected]> wrote:
Quoted 15 lines
Quoted 5 lines
However I think
cond_resched_lockandcond_resched_softirqalso need fix to make the semantic consistent.Please check the following patch.
Ah. I think the return value from these functions should mean "something disruptive happened", if you like.
See, the callers of
cond_resched_lock()aren't interested in whethercond_resched_lock()actually calledschedule(). They want to know whethercond_resched_lock()dropped the lock. Because if the lock was dropped, the caller needs to take some special action, regardless of whetherschedule()was finally called.Quoted 5 lines
Quoted 1 line
So I think the patch I queued is OK, agree?
I am afraid the code like
cond_resched_lockcheck infs/jbd/checkpoint.clog_do_checkpointmay fall into endless retry in some condition, will it?Oh crap, yes. If
need_resched()and system_state==SYSTEM_BOOTING thencond_resched_lock()will drop the lock but won't schedule. So it'll return true but won't clearneed_resched()and the caller will lock up.Quoted 8 lines
Quoted 6 lines
So if
cond_resched_foo()ends up dropping the lock it _must_ callschedule()to clearneed_resched().So, how about this (it needs some code comments!)
The patch works for the install test env.
Thanks.
Quoted 3 lines
However I still have some concern on
cond_resched_lock(), on an UP
kernel it will return 1 if schedule happen, but actually it does not drop any lock, that semantic seems to be different to SMP kernel.That's OK (I think - I don't have a good track record in this thread).
If the kernel is non-preemptible and UP, we want to return true from
cond_resched_foo()if we calledschedule(). Becauseschedule()might allow a different thread into the kernel which might modify the locked data.
Quoted 6 lines
And if the kernel is preemptible and UP, we want to return true from
cond_resched_foo()if we dropped the lock, because that internally does apreempt_enable().And the patch (hopefully) satisfies those requirements. Does that all sound solid?
Ah yes, I think the logic is solid.
cond_sched_xxx will return 1 only if any thing disruptive really
happen, either dropping a lock or enabling preempt or bh or schedule.
The patch satisfied those requirements.
Thanks
Zou Nan hai
Quoted 11 lines · Andrew Morton
Quoted 3 lines
However I still have some concern on
cond_resched_lock(), on an UP
kernel it will return 1 if schedule happen, but actually it does not drop any lock, that semantic seems to be different to SMP kernel.That's OK (I think - I don't have a good track record in this thread).
If the kernel is non-preemptible and UP, we want to return true from
cond_resched_foo()if we calledschedule(). Becauseschedule()might allow a different thread into the kernel which might modify the locked data.
Quoted 6 lines
And if the kernel is preemptible and UP, we want to return true from
cond_resched_foo()if we dropped the lock, because that internally does apreempt_enable().And the patch (hopefully) satisfies those requirements. Does that all sound solid?
it looks solid to me ... but this is subtle stuff and i dont seem to have a good track record for these details today. I guess if it's inadequate Zou ought to see them in practice?
Ingo
Quoted 6 lines · Andrew Morton
Quoted 5 lines
However I think
cond_resched_lockandcond_resched_softirqalso need fix to make the semantic consistent.Please check the following patch.
Quoted 8 lines
Ah. I think the return value from these functions should mean "something disruptive happened", if you like.
See, the callers of
cond_resched_lock()aren't interested in whethercond_resched_lock()actually calledschedule(). They want to know whethercond_resched_lock()dropped the lock. Because if the lock was dropped, the caller needs to take some special action, regardless of whetherschedule()was finally called.
indeed ...!
Quoted 1 line
So I think the patch I queued is OK, agree?
yeah.
i think the really-right-fix would be to get rid of that SYSTEM_BOOTING ugliness though ... I'm quite a bit uneasy about us doing different things for an initrd app than for fully booted apps.
Ingo
Ingo Molnar
Acked-bykernel/sched.c EXPORT_SYMBOL(cond_resched_lock); · line 4105
Quoted 10 lines · Zou Nan hai
kernel/sched.c +2
-3@@ -4094,8 +4093,7 @@ int cond_resched_lock(spinlock_t *lock) -__cond_resched (); - ret =1 ; + ret |=__cond_resched ();@@ -4106,14 +4104,13 @@ EXPORT_SYMBOL(cond_resched_lock); -__cond_resched (); + ret =__cond_resched ();
Ingo
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On 28 Jun 2006 12:48:43 +0800 Zou Nan hai <[email protected]> wrote:
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.
Quoted 8 lines
Quoted 6 lines
That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.That's a bug in
cond_resched().
hummm while this fix is needed... something makes me want to think that something as complex as an OS installer really shouldn't be running with SYSTEM_BOOTING state anymore....
shouldn't we start considering running the initrd (esp now with klibc) as SYSTEM_RUNNING ?
Quoted 15 lines · Arjan van de Ven
Quoted 14 lines · Andrew Morton
Quoted 10 lines · On 28 Jun 2006 12:48:43 +0800 Zou Nan hai <[email protected]> wrote:
We see system hang in ext3 jbd code
when Linux install program anaconda copying
packages.That is because anaconda is invoked from linuxrc in initrd when
system_stateis stillSYSTEM_BOOTING.Thus the
cond_reschedchecks injournal_commit_transactionwill always return 1 without actually schedule, then the system fall into deadloop.
Quoted 7 lines
Quoted 1 line
That's a bug in
cond_resched().hummm while this fix is needed... something makes me want to think that something as complex as an OS installer really shouldn't be running with
SYSTEM_BOOTINGstate anymore....
shouldn't we start considering running the initrd (esp now with klibc) asSYSTEM_RUNNING?
Is there any historical reason that we put system_state of app runs
from initrd into SYSTEM_BOOTING instead of SYSTEM_RUNNING?
Zou Nan hai