sched: Remove SYSTEM_RUNNING checks from cond_resched*()
Subsystems (3)
Using early netconsole and gianfar driver this error pops up:
netconsole: timeout waiting for carrier
It appears that net/core/netpoll.c:netpoll_setup() is using cond_resched() in a loop waiting for a carrier.
The thing is that cond_resched() is a no-op when system_state != SYSTEM_RUNNING, and so drivers/net/phy/phy.c's state_queue is never scheduled, therefore link detection doesn't work.
This patch fixes the issue by removing SYSTEM_RUNNING checks, which might be dangerous to do, but looking at 'git blame' I can see that the first SYSTEM_RUNNING check was introduced in this commit:
commit 8ba7b0a14b2ec19583bedbcdbea7f1c5008fc922
Author: Linus Torvalds <[email protected]>
Date: Mon Mar 6 17:38:49 2006 -0800
Add early-boot-safety check to cond_resched()
Just to be safe, we should not trigger a conditional reschedule during the early boot sequence. We've historically done some questionable early on, and the safety warnings in __might_sleep() are generally turned off during that period, so there might be problems lurking.
This affects CONFIG_PREEMPT_VOLUNTARY, which takes over might_sleep() to cause a voluntary conditional reschedule.
The commit doesn't mention that it fixes some particular problem, though it surely breaks netconsole for drivers that are using phylib.
I tested the patch with CONFIG_PREEMPT_VOLUNTARY (on PowerPC), and it didn't cause any issues.
Surely, the other way to fix the netconsole is to call schedule() or msleep() directly, but I'm curious if removing the checks makes sense nowadays.
--- kernel/sched.c | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-)
kernel/sched.c +3 -4
@@ -6560,8 +6560,7 @@ static void __cond_resched(void) int __sched _cond_resched(void) { -if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE ) && - system_state ==SYSTEM_RUNNING ) { +if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE )) {__cond_resched ();return 1 ; }
@@ -6579,7 +6578,7 @@ EXPORT_SYMBOL(_cond_resched);*/ int cond_resched_lock (spinlock_t *lock) { -int resched =need_resched () && system_state ==SYSTEM_RUNNING ; +int resched =need_resched ();int ret =0 ;if (spin_needbreak (lock) || resched) {
@@ -6599,7 +6598,7 @@ int __sched cond_resched_softirq(void) {BUG_ON (!in_softirq ()); -if (need_resched () && system_state ==SYSTEM_RUNNING ) { +if (need_resched ()) {local_bh_enable ();__cond_resched ();local_bh_disable ();
Signature · 1 line
1.6.3.3
Quoted 3 lines · Anton Vorontsov
but I'm curious if removing the checks makes sense nowadays.
...
kernel/sched.c static void __cond_resched(void) · line 6566
Quoted 12 lines
kernel/sched.c +1
-2@@ -6560,8 +6560,7 @@ static void __cond_resched(void) int __sched _cond_resched(void) { -if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE ) && - system_state ==SYSTEM_RUNNING ) { +if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE )) {__cond_resched ();return 1 ; }
and, with CONFIG_PREEMPT preempt_schedule() does not check system_state, so it looks really strange cond_resched() does check SYSTEM_RUNNING.
debug_smp_processor_id() looks strange too:
/* * It is valid to assume CPU-locality during early bootup: */ if (system_state != SYSTEM_RUNNING) goto out;
this doesn't look right, smp_init() is called before we set SYSTEM_RUNNING.
Hmm, and
/* * Kernel threads bound to a single CPU can safely use * smp_processor_id(): */ if (cpumask_equal(¤t->cpus_allowed, cpumask_of(this_cpu))) goto out;
perhaps this should use PF_THREAD_BOUND ?
Oleg.
Quoted 8 lines · Oleg Nesterov
/* * It is valid to assume CPU-locality during early bootup: */ if (system_state != SYSTEM_RUNNING) goto out;this doesn't look right,
smp_init()is called before we setSYSTEM_RUNNING.
The thing is, there's also ton's of code that might end up calling cond_resched() and co before the scheduler is fully initialized. Doing so would indeed mess things up.
Also, by definition we'd have to call smp_init() before SYSTEM_RUNNING, because you simply cannot declare a system up and running when your core functionality isn't initialized.
So I'd really rather preserve these checks -- I can even remember running into some of these things a while back, but memory isn't providing specific cases.
Quoted 10 lines
Hmm, and
/* * Kernel threads bound to a single CPU can safely use * smp_processor_id(): */ if (cpumask_equal(¤t->cpus_allowed, cpumask_of(this_cpu))) goto out;perhaps this should use
PF_THREAD_BOUND?
That might predate PF_THREAD_BOUND, also I think this is more generic, and I think we used it for that set_affinity dance we did Rusty 'fixed' a while back.
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Oleg Nesterov
/* * It is valid to assume CPU-locality during early bootup: */ if (system_state != SYSTEM_RUNNING) goto out;
Quoted 5 lines
Quoted 2 lines
this doesn't look right,
smp_init()is called before we setSYSTEM_RUNNING.The thing is, there's also ton's of code that might end up calling
cond_resched()and co before the scheduler is fully initialized.
Hm. Speaking of cond_resched*() only, then it should be pretty safe to convert the SYSTEM_RUNNING checks to scheduler_running, no? scheduler_running is set after sched_init().
Something like this
kernel/sched.c +3 -3
@@ -6561,7 +6561,7 @@ static void __cond_resched(void) int __sched _cond_resched(void) {if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE ) && - system_state ==SYSTEM_RUNNING ) { + scheduler_running) {__cond_resched ();return 1 ; }
@@ -6579,7 +6579,7 @@ EXPORT_SYMBOL(_cond_resched);*/ int cond_resched_lock (spinlock_t *lock) { -int resched =need_resched () && system_state ==SYSTEM_RUNNING ; +int resched =need_resched () && scheduler_running;int ret =0 ;if (spin_needbreak (lock) || resched) {
@@ -6599,7 +6599,7 @@ int __sched cond_resched_softirq(void) {BUG_ON (!in_softirq ()); -if (need_resched () && system_state ==SYSTEM_RUNNING ) { +if (need_resched () && scheduler_running) {local_bh_enable ();__cond_resched ();local_bh_disable ();
Quoted 12 lines · Anton Vorontsov
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Oleg Nesterov
/* * It is valid to assume CPU-locality during early bootup: */ if (system_state != SYSTEM_RUNNING) goto out;this doesn't look right,
smp_init()is called before we setSYSTEM_RUNNING.
Quoted 6 lines
Quoted 2 lines
The thing is, there's also ton's of code that might end up calling
cond_resched()and co before the scheduler is fully initialized.Hm. Speaking of cond_resched*() only, then it should be pretty safe to convert the
SYSTEM_RUNNINGchecks toscheduler_running, no?scheduler_runningis set aftersched_init().
Hmm, that might work, I'd have to audit sched_init_smp() as it seems to do way too much...
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Anton Vorontsov
Quoted 13 lines · Peter Zijlstra
Quoted 8 lines · Oleg Nesterov
/* * It is valid to assume CPU-locality during early bootup: */ if (system_state != SYSTEM_RUNNING) goto out;this doesn't look right,
smp_init()is called before we setSYSTEM_RUNNING.The thing is, there's also ton's of code that might end up calling
cond_resched()and co before the scheduler is fully initialized.
Quoted 6 lines
Quoted 3 lines
Hm. Speaking of cond_resched*() only, then it should be pretty safe to convert the
SYSTEM_RUNNINGchecks toscheduler_running, no?scheduler_runningis set aftersched_init().Hmm, that might work, I'd have to audit
sched_init_smp()as it seems to do way too much...
sched_init_smp() is called from the kernel_thread(), so if the scheduler is not functional prior to kernel_thread(), you're in trouble anyway, no? The point is that a lot of code is calling schedule() prior to sched_init_smp() (e.g. msleep(), mutexes), and there are no issues. So should be no issues with cond_resched()?
Signature · 3 lines
Anton Vorontsov email: [email protected] irc://irc.freenode.net/bd2
Quoted 24 lines · Anton Vorontsov
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Anton Vorontsov
Quoted 13 lines · Peter Zijlstra
Quoted 8 lines · Oleg Nesterov
/* * It is valid to assume CPU-locality during early bootup: */ if (system_state != SYSTEM_RUNNING) goto out;this doesn't look right,
smp_init()is called before we setSYSTEM_RUNNING.The thing is, there's also ton's of code that might end up calling
cond_resched()and co before the scheduler is fully initialized.Quoted 6 lines
Quoted 3 lines
Hm. Speaking of cond_resched*() only, then it should be pretty safe to convert the
SYSTEM_RUNNINGchecks toscheduler_running, no?scheduler_runningis set aftersched_init().Hmm, that might work, I'd have to audit
sched_init_smp()as it seems to do way too much...
Quoted 6 lines
sched_init_smp()is called from thekernel_thread(), so if the scheduler is not functional prior tokernel_thread(), you're in trouble anyway, no? The point is that a lot of code is callingschedule()prior tosched_init_smp()(e.g.msleep(), mutexes), and there are no issues. So should be no issues withcond_resched()?
Yeah, it should be good, I just got paranoid looking at sched_init_smp().
PATCH 1/1 sched: Make cond_resched*() available earlier
Using early netconsole and gianfar driver this error pops up:
netconsole: timeout waiting for carrier
It appears that net/core/netpoll.c:netpoll_setup() is using cond_resched() in a loop waiting for a carrier.
The thing is that cond_resched() is a no-op when system_state != SYSTEM_RUNNING, and so drivers/net/phy/phy.c's state_queue is never scheduled, therefore link detection doesn't work.
The system_state check exists to avoid very early calls to the scheduler. Though, using cond_resched() should be safe after scheduler is initialized, so instead of checking the system_state, we can test that the scheduler is running, therefore making cond_resched() and friends available much earlier (but not too much).
---
On Wed, Jul 08, 2009 at 02:58:46PM +0200, Peter Zijlstra wrote:
[...]
Quoted 14 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Hm. Speaking of cond_resched*() only, then it should be pretty safe to convert the
SYSTEM_RUNNINGchecks toscheduler_running, no?scheduler_runningis set aftersched_init().Hmm, that might work, I'd have to audit
sched_init_smp()as it seems to do way too much...Quoted 6 lines
sched_init_smp()is called from thekernel_thread(), so if the scheduler is not functional prior tokernel_thread(), you're in trouble anyway, no? The point is that a lot of code is callingschedule()prior tosched_init_smp()(e.g.msleep(), mutexes), and there are no issues. So should be no issues withcond_resched()?
Quoted 2 lines
Yeah, it should be good, I just got paranoid looking at
sched_init_smp().
OK, thanks everyone for the reviews.
Here is an updated patch. As usual, tested on UP PowerPC, with and without SMP, PREEMPT and PREEMPT_VOLUNTARY.
kernel/sched.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
kernel/sched.c +3 -3
@@ -6561,7 +6561,7 @@ static void __cond_resched(void) int __sched _cond_resched(void) {if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE ) && - system_state ==SYSTEM_RUNNING ) { + scheduler_running) {__cond_resched ();return 1 ; }
@@ -6579,7 +6579,7 @@ EXPORT_SYMBOL(_cond_resched);*/ int cond_resched_lock (spinlock_t *lock) { -int resched =need_resched () && system_state ==SYSTEM_RUNNING ; +int resched =need_resched () && scheduler_running;int ret =0 ;if (spin_needbreak (lock) || resched) {
@@ -6599,7 +6599,7 @@ int __sched cond_resched_softirq(void) {BUG_ON (!in_softirq ()); -if (need_resched () && system_state ==SYSTEM_RUNNING ) { +if (need_resched () && scheduler_running) {local_bh_enable ();__cond_resched ();local_bh_disable ();
Signature · 1 line
1.6.3.3
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Oleg Nesterov
/* * It is valid to assume CPU-locality during early bootup: */ if (system_state != SYSTEM_RUNNING) goto out;
Quoted 6 lines
Quoted 2 lines
this doesn't look right,
smp_init()is called before we setSYSTEM_RUNNING.The thing is, there's also ton's of code that might end up calling
cond_resched()and co before the scheduler is fully initialized. Doing so would indeed mess things up.
I forget what triggered me to add some of them, but we definitely had bugs with the scheduler being called early, and having some really nasty oopses happen early in the boot (and that early, they don't get saved, so the reporting percentage goes down to something very low).
So I think that the patch Anton posted is probably the RightThing(tm) to do, and in that sense I like it - but they make me very nervous. There's a lot of "cond_resched()"s sprinkled about in helper routines, and if they are ever called early, you're basically screwed.
Quoted 3 lines
Also, by definition we'd have to call
smp_init()beforeSYSTEM_RUNNING, because you simply cannot declare a system up and running when your core functionality isn't initialized.
Now, that part I'm not sure about.
We can consider a UP system to be running, and brining up the other CPU's to be a matter of CPU hotplug.
It's not what we do _now_, of course. We don't force hotplug support on people just because they want SMP, and we don't want to go through the whole "rewrite locks" things twice etc.
Quoted 3 lines
So I'd really rather preserve these checks -- I can even remember running into some of these things a while back, but memory isn't providing specific cases.
Yes. I get nervous too about the patch.
That said, I do agree that maybe SYSTEM_RUNNING isn't the right check. Testing that the scheduler is initialized may be the more correct one. I think the SYSTEM_RUNNING one just comes from that being used for other debug issues.
Linus
Quoted 5 lines · On Wed, 8 Jul 2009 09:12:30 -0700 (PDT) Linus Torvalds…
That said, I do agree that maybe
SYSTEM_RUNNINGisn't the right check. Testing that the scheduler is initialized may be the more correct one. I think theSYSTEM_RUNNINGone just comes from that being used for other debug issues.
Agreed. system_state is too general.
If we specifically want to know whether it is safe to call schedule() then let's create a global boolean it_is_safe_to_call_schedule and test that, rather than testing something which indirectly and unreliably implies "it is safe to call schedule". If that boolean already exists then no-brainer.
All that being said, I wonder if the netconsole code should be using msleep(1) instead. Spinning on cond_resched() is a bit rude. But one would have to verify that it is safe to call schedule() at this time, and for the netconsole caller, this is dubious.
Quoted 13 lines · Andrew Morton
Quoted 5 lines · On Wed, 8 Jul 2009 09:12:30 -0700 (PDT) Linus Torvalds…
That said, I do agree that maybe
SYSTEM_RUNNINGisn't the right check. Testing that the scheduler is initialized may be the more correct one. I think theSYSTEM_RUNNINGone just comes from that being used for other debug issues.Agreed.
system_stateis too general.If we specifically want to know whether it is safe to call
schedule()then let's create a global booleanit_is_safe_to_call_scheduleand test that, rather than testing something which indirectly and unreliably implies "it is safe to call schedule". If that boolean already exists then no-brainer.
Quoted 4 lines
All that being said, I wonder if the netconsole code should be using msleep(1) instead. Spinning on
cond_resched()is a bit rude. But one would have to verify that it is safe to callschedule()at this time, and for the netconsole caller, this is dubious.
What do you mean by "verify that it is safe"? If it works, can I assume that it's safe? ;-) It works, fwiw.
Thanks,
Signature · 3 lines
Anton Vorontsov email: [email protected] irc://irc.freenode.net/bd2
(belatedly cc'ing netdev)
Original diagnosis:
: Using early netconsole and gianfar driver this error pops up:
:
: netconsole: timeout waiting for carrier
:
: It appears that net/core/netpoll.c:netpoll_setup() is using : cond_resched() in a loop waiting for a carrier.
:
: The thing is that cond_resched() is a no-op when system_state != : SYSTEM_RUNNING, and so drivers/net/phy/phy.c's state_queue is never : scheduled, therefore link detection doesn't work
Quoted 15 lines · On Thu, 9 Jul 2009 01:33:31 +0400 Anton Vorontsov <[email protected]>…
Quoted 13 lines · Andrew Morton
Quoted 5 lines · On Wed, 8 Jul 2009 09:12:30 -0700 (PDT) Linus Torvalds…
That said, I do agree that maybe
SYSTEM_RUNNINGisn't the right check. Testing that the scheduler is initialized may be the more correct one. I think theSYSTEM_RUNNINGone just comes from that being used for other debug issues.Agreed.
system_stateis too general.If we specifically want to know whether it is safe to call
schedule()then let's create a global booleanit_is_safe_to_call_scheduleand test that, rather than testing something which indirectly and unreliably implies "it is safe to call schedule". If that boolean already exists then no-brainer.
Quoted 8 lines
Quoted 4 lines
All that being said, I wonder if the netconsole code should be using msleep(1) instead. Spinning on
cond_resched()is a bit rude. But one would have to verify that it is safe to callschedule()at this time, and for the netconsole caller, this is dubious.What do you mean by "verify that it is safe"? If it works, can I assume that it's safe? ;-) It works, fwiw.
netconsole is supposed to be available as early as possible in boot for obvious reasons. I'd say there's a decent risk now and in the future that netconsole will be initialised prior to the scheduler being available.
In fact, if "netconsole: timeout waiting for carrier" newly added to netpoll_setup() a depedency on the scheduler being available then perhaps that was an incorrect change.
PATCH 1/1 netpoll: Fix carrier detection for drivers that are using phylib
Using early netconsole and gianfar driver this error pops up:
netconsole: timeout waiting for carrier
It appears that net/core/netpoll.c:netpoll_setup() is using cond_resched() in a loop waiting for a carrier.
The thing is that cond_resched() is a no-op when system_state != SYSTEM_RUNNING, and so drivers/net/phy/phy.c's state_queue is never scheduled, therefore link detection doesn't work.
I belive that the main problem is in cond_resched()[1], but despite how the cond_resched() story ends, it might be a good idea to call msleep(1) instead of cond_resched(), as suggested by Andrew Morton.
[1] http://lkml.org/lkml/2009/7/7/463
---
Quoted 39 lines · Andrew Morton
(belatedly cc'ing netdev)
Original diagnosis:
: Using early netconsole and gianfar driver this error pops up:
:
: netconsole: timeout waiting for carrier
:
: It appears that net/core/netpoll.c:netpoll_setup() is using :cond_resched()in a loop waiting for a carrier.
:
: The thing is thatcond_resched()is a no-op whensystem_state!= :SYSTEM_RUNNING, and so drivers/net/phy/phy.c'sstate_queueis never : scheduled, therefore link detection doesn't workQuoted 15 lines · On Thu, 9 Jul 2009 01:33:31 +0400 Anton Vorontsov <[email protected]>…
Quoted 13 lines · Andrew Morton
Quoted 5 lines · On Wed, 8 Jul 2009 09:12:30 -0700 (PDT) Linus Torvalds…
That said, I do agree that maybe
SYSTEM_RUNNINGisn't the right check. Testing that the scheduler is initialized may be the more correct one. I think theSYSTEM_RUNNINGone just comes from that being used for other debug issues.Agreed.
system_stateis too general.If we specifically want to know whether it is safe to call
schedule()then let's create a global booleanit_is_safe_to_call_scheduleand test that, rather than testing something which indirectly and unreliably implies "it is safe to call schedule". If that boolean already exists then no-brainer.Quoted 8 lines
Quoted 4 lines
All that being said, I wonder if the netconsole code should be using msleep(1) instead. Spinning on
cond_resched()is a bit rude. But one would have to verify that it is safe to callschedule()at this time, and for the netconsole caller, this is dubious.What do you mean by "verify that it is safe"? If it works, can I assume that it's safe? ;-) It works, fwiw.
Quoted 7 lines
netconsole is supposed to be available as early as possible in boot for obvious reasons. I'd say there's a decent risk now and in the future that netconsole will be initialised prior to the scheduler being available.
In fact, if "netconsole: timeout waiting for carrier" newly added to
netpoll_setup()a depedency on the scheduler being available then perhaps that was an incorrect change.
'git blame' says that carrier detection code didn't change since 2.6.12 (where git history starts), PHYLIB is using workqueue since its submission (2.6.13). And SYSTEM_RUNNING check was added in 2.6.16. So it's not a new dependency.
The netpoll code is using msleep() just a few lines below cond_resched(), so we won't make things worse. ;-)
Thanks!
net/core/netpoll.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
net/core/netpoll.c +1 -1
@@ -740,7 +740,7 @@ int netpoll_setup(struct netpoll *np) np->name);break ; } -cond_resched (); +msleep (1 ); }/* If carrier appears to come up instantly, we don't
Signature · 1 line
1.6.3.3
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 3 lines · Anton Vorontsov
The netpoll code is using
msleep()just a few lines belowcond_resched(), so we won't make things worse. ;-)
Yeah. That function is definitely sleeping. It does things like kmalloc(GFP_KERNEL), rtnl_lock() and synchronize_rcu() etc too, so an added msleep() is the least of our problems.
Afaik, it's called from a bog-standard "module_init()", which happens late enough that everything works.
In fact, I wonder if we should set SYSTEM_RUNNING much earlier - _before_ doing the whole "do_initcalls()". By then we've set up all the core stuff and enabled interrupts, so we really _are_ running. We just don't necessarily have drivers, filesystems etc loaded yet. But anything that happens late enough to be an initcall should be largely considered to be during "normal code".
(The "early_initcall" cases are special - those really do happen pretty early).
So ACK on Anton's patch, but I wonder if we _also_ should do the following?
Looking at the people looking at SYSTEM_RUNNING, I do note some odd cases. Why the heck does kernel/perf_counter.c do it, for example?
Linus
--- init/main.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
init/main.c +1 -1
@@ -788,6 +788,7 @@ static void __init do_initcalls(void) {initcall_t *call; + system_state =SYSTEM_RUNNING ;for (call = __early_initcall_end; call < __initcall_end; call++)do_one_initcall (*call);
@@ -839,7 +840,6 @@ static noinline int init_post(void)free_initmem ();unlock_kernel ();mark_rodata_ro (); - system_state =SYSTEM_RUNNING ;numa_default_policy ();if (sys_open ((const char __user *)"/dev/console" ,O_RDWR ,0 ) <0 )
David Miller
maintainerRe: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
From: Linus Torvalds <[email protected]>
Date: Wed, 8 Jul 2009 17:01:14 -0700 (PDT)
Quoted 1 line
So ACK on Anton's patch,
I'll integrate it into my tree, thanks.
Quoted 1 line
but I wonder if we _also_ should do the following?
I think we should set SYSTEM_RUNNING earlier, makes sense to me.
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
In reply to Linus TorvaldsQuoted 2 lines · Linus Torvalds
Looking at the people looking at
SYSTEM_RUNNING, I do note some odd cases. Why the heck doeskernel/perf_counter.cdo it, for example?
Ah, those are the swcounter and other probe entry points. I've had several cases where we called into the perf counter code from those points before it was initialized, getting in kernel segfaults due to dereferencing uninitialized data etc..
I could keep a variable that tracked the perf_counter_init() state, and use that instead if you prefer?
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 11 lines · Peter Zijlstra
Quoted 2 lines · Linus Torvalds
Looking at the people looking at
SYSTEM_RUNNING, I do note some odd cases. Why the heck doeskernel/perf_counter.cdo it, for example?Ah, those are the swcounter and other probe entry points. I've had several cases where we called into the perf counter code from those points before it was initialized, getting in kernel segfaults due to dereferencing uninitialized data etc..
I could keep a variable that tracked the
perf_counter_init()state, and use that instead if you prefer?
Looks like that'd be more accurate. Linus's proposed patch might break your current assumptions.
Signature · 1 line
http://selenic.com : development and support for Mercurial and Linux
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
In reply to Linus TorvaldsQuoted 10 lines · Linus Torvalds
Quoted 3 lines · Anton Vorontsov
The netpoll code is using
msleep()just a few lines belowcond_resched(), so we won't make things worse. ;-)Yeah. That function is definitely sleeping. It does things like
kmalloc(GFP_KERNEL),rtnl_lock()andsynchronize_rcu()etc too, so an addedmsleep()is the least of our problems.
Quoted 5 lines
Afaik, it's called from a bog-standard "
module_init()", which happens late enough that everything works.In fact, I wonder if we should set
SYSTEM_RUNNINGmuch earlier - _before_ doing the whole "do_initcalls()".
Well there are two ways of consistently defining SYSTEM_RUNNING:
a) define it with reference to the well-understood notion of booting vs running and don't switch it until handing off to init
b) define it with reference to its usage by an arbitrary user like cond_resched()
In the latter case, we obviously need to move it to the earliest point that scheduling is possible. But there are a number of things like
http://lxr.linux.no/linux+v2.6.30/kernel/printk.c#L228
that assume the definition is actually (a). We're currently within a couple lines of a strict definition of (a) already, so I actually think cond_resched() is just wrong (and we already know it broke a previously-working user). It should perhaps be using another private flag that gets set as soon as scheduling is up and running.
But I'd actually go further and say that it's unfortunate to be checking extra flags in such an important inline, especially since the check is false for all but the first couple seconds of run time. Seems like we could avoid adding an extra check by artificially elevating the preempt count in early boot (or at compile time) then dropping it when scheduling becomes available.
Signature · 1 line
http://selenic.com : development and support for Mercurial and Linux
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 14 lines · Matt Mackall
Quoted 13 lines · Linus Torvalds
Quoted 3 lines · Anton Vorontsov
The netpoll code is using
msleep()just a few lines belowcond_resched(), so we won't make things worse. ;-)Yeah. That function is definitely sleeping. It does things like
kmalloc(GFP_KERNEL),rtnl_lock()andsynchronize_rcu()etc too, so an addedmsleep()is the least of our problems.Afaik, it's called from a bog-standard "
module_init()", which happens late enough that everything works.
Quoted 7 lines
Quoted 2 lines
In fact, I wonder if we should set
SYSTEM_RUNNINGmuch earlier - _before_ doing the whole "do_initcalls()".Well there are two ways of consistently defining
SYSTEM_RUNNING:a) define it with reference to the well-understood notion of booting vs running and don't switch it until handing off to init
This makes the most sense IMHO.
Quoted 6 lines
b) define it with reference to its usage by an arbitrary user like
cond_resched()In the latter case, we obviously need to move it to the earliest point that scheduling is possible. But there are a number of things like
Quoted 7 lines
http://lxr.linux.no/linux+v2.6.30/kernel/printk.c#L228
that assume the definition is actually (a). We're currently within a couple lines of a strict definition of (a) already, so I actually think
cond_resched()is just wrong (and we already know it broke a previously-working user). It should perhaps be using another private flag that gets set as soon as scheduling is up and running.
Right as mentioned before in this thread, we grew scheduler_running a while back which could be used for this.
Quoted 6 lines
But I'd actually go further and say that it's unfortunate to be checking extra flags in such an important inline, especially since the check is false for all but the first couple seconds of run time. Seems like we could avoid adding an extra check by artificially elevating the preempt count in early boot (or at compile time) then dropping it when scheduling becomes available.
Calling cond_resched() and co when !preemptable is an error so this wouldn't actually work.
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 49 lines · Peter Zijlstra
Quoted 14 lines · Matt Mackall
Quoted 13 lines · Linus Torvalds
Quoted 3 lines · Anton Vorontsov
The netpoll code is using
msleep()just a few lines belowcond_resched(), so we won't make things worse. ;-)Yeah. That function is definitely sleeping. It does things like
kmalloc(GFP_KERNEL),rtnl_lock()andsynchronize_rcu()etc too, so an addedmsleep()is the least of our problems.Afaik, it's called from a bog-standard "
module_init()", which happens late enough that everything works.Quoted 7 lines
Quoted 2 lines
In fact, I wonder if we should set
SYSTEM_RUNNINGmuch earlier - _before_ doing the whole "do_initcalls()".Well there are two ways of consistently defining
SYSTEM_RUNNING:a) define it with reference to the well-understood notion of booting vs running and don't switch it until handing off to init
This makes the most sense IMHO.
Quoted 6 lines
b) define it with reference to its usage by an arbitrary user like
cond_resched()In the latter case, we obviously need to move it to the earliest point that scheduling is possible. But there are a number of things like
Quoted 7 lines
http://lxr.linux.no/linux+v2.6.30/kernel/printk.c#L228
that assume the definition is actually (a). We're currently within a couple lines of a strict definition of (a) already, so I actually think
cond_resched()is just wrong (and we already know it broke a previously-working user). It should perhaps be using another private flag that gets set as soon as scheduling is up and running.Right as mentioned before in this thread, we grew
scheduler_runninga while back which could be used for this.Quoted 6 lines
But I'd actually go further and say that it's unfortunate to be checking extra flags in such an important inline, especially since the check is false for all but the first couple seconds of run time. Seems like we could avoid adding an extra check by artificially elevating the preempt count in early boot (or at compile time) then dropping it when scheduling becomes available.
Quoted 2 lines
Calling
cond_resched()and co when !preemptable is an error so this wouldn't actually work.
Sorry if I was unclear. I'm suggesting setting the count so the existing PREEMPT_ACTIVE test here fires:
int __sched _cond_resched(void)
{
if (need_resched() && !(preempt_count() & PREEMPT_ACTIVE) &&
system_state == SYSTEM_RUNNING) {
__cond_resched();
return 1;
}
return 0;
}That should be kosher.
Signature · 1 line
http://selenic.com : development and support for Mercurial and Linux
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 5 lines · Matt Mackall
Sorry if I was unclear. I'm suggesting setting the count so the existing
PREEMPT_ACTIVEtest here fires:int
__sched_cond_resched(void)
Quoted 8 lines
{
if (need_resched() && !(preempt_count() & PREEMPT_ACTIVE) &&
system_state == SYSTEM_RUNNING) {
__cond_resched();
return 1;
}
return 0;
}Right, /me read preempt and thought a simple preempt inc but didn't read the code. Shame on me.
So something like (utterly untested and such)
---
arch/x86/include/asm/thread_info.h | 2 +- kernel/sched.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/thread_info.h +1 -1
@@ -49,7 +49,7 @@ struct thread_info { .exec_domain = &default_exec_domain, \ .flags = 0, \ .cpu = 0, \ - .preempt_count = 1, \ + .preempt_count = 1 + PREEMPT_ACTIVE, \ .addr_limit = KERNEL_DS, \ .restart_block = { \ .fn = do_no_restart_syscall, \
kernel/sched.c +2 -2
@@ -6599,8 +6599,7 @@ static void __cond_resched(void) int __sched _cond_resched(void) { -if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE ) && - system_state ==SYSTEM_RUNNING ) { +if (need_resched () && !(preempt_count () &PREEMPT_ACTIVE )) {__cond_resched ();return 1 ; }
@@ -9422,6 +9421,7 @@ void __init sched_init(void)perf_counter_init (); scheduler_running =1 ; +sub_preempt_count (PREEMPT_ACTIVE ); }#ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Matt Mackall
Sorry if I was unclear. I'm suggesting setting the count so the existing
PREEMPT_ACTIVEtest here fires:int
__sched_cond_resched(void)Quoted 8 lines
{ if (need_resched() && !(preempt_count() & PREEMPT_ACTIVE) && system_state == SYSTEM_RUNNING) { __cond_resched(); return 1; } return 0; }
Quoted 4 lines
Right, /me read preempt and thought a simple preempt inc but didn't read the code. Shame on me.
So something like (utterly untested and such)
Yeah, that's what I had in mind. Probably throw in a define:
/* for disabling scheduling in early boot */
#define PREEMPT_EARLY (1 + PREEMPT_ACTIVE)
and slap a comment on the sub_preempt_count().
Does anything actually use scheduler_running yet? Perhaps my tree is old.
Also, might_sleep's use of system_state probably bears revisiting.
Signature · 1 line
http://selenic.com : development and support for Mercurial and Linux
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 6 lines · Matt Mackall
Yeah, that's what I had in mind. Probably throw in a define:
/* for disabling scheduling in early boot */
#definePREEMPT_EARLY(1 +PREEMPT_ACTIVE)and slap a comment on the
sub_preempt_count().
Right, and visit all the other arch init code ;-)
I'll wait to see if Ingo has anything to say about it and then complete this thing.
Quoted 2 lines
Does anything actually use
scheduler_runningyet? Perhaps my tree is old.
# git grep scheduler_running kernel/sched.c:static __read_mostly int scheduler_running; kernel/sched.c: scheduler_running = 1; kernel/sched_rt.c: if (unlikely(!scheduler_running)) kernel/sched_rt.c: if (unlikely(!scheduler_running))
If memory serves there used to be more, but I think that migrated into kernel/sched_clock.c, which has sched_clock_running.
Quoted 1 line
Also, might_sleep's use of
system_stateprobably bears revisiting.
Yeah, all that code is from long before we had scheduler_running (which was introduced around CFS/.23).
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
Quoted 11 lines · Peter Zijlstra
Quoted 2 lines
Does anything actually use
scheduler_runningyet? Perhaps my tree is old.# git grep scheduler_running kernel/sched.c:static __read_mostly int scheduler_running; kernel/sched.c: scheduler_running = 1; kernel/sched_rt.c: if (unlikely(!scheduler_running)) kernel/sched_rt.c: if (unlikely(!scheduler_running))If memory serves there used to be more, but I think that migrated into
kernel/sched_clock.c, which hassched_clock_running.
The static in the first line makes that a bit of a head-scratcher? Oh, I
see: it's living in sin. Going to avert my eyes for now.
Signature · 1 line
http://selenic.com : development and support for Mercurial and Linux
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
In reply to Peter ZijlstraQuoted 2 lines · Peter Zijlstra
So something like (utterly untested and such)
This looks like a good patch. Please make it so - who knows what other uses of cond_resched() we have in module init routines that might have deadlocks without it. The netpoll case got fixed, but please just do this.
I'd like to do my system_state movement too (the thing is, when you load drivers as modules you _will_ have "system_state == SYSTEM_RUNNING", so any initcall that depends on it being "early boot" is already broken), but there's no way that patch is appropriate for post-rc2. This one, however, looks appropriate (modulo getting some testing, of course)
Linus
Re: [PATCH] netpoll: Fix carrier detection for drivers that are using phylib
In reply to Anton VorontsovQuoted 68 lines · Anton Vorontsov
Using early netconsole and gianfar driver this error pops up:
netconsole: timeout waiting for carrier
It appears that net/core/netpoll.c:netpoll_setup() is using
cond_resched()in a loop waiting for a carrier.The thing is that
cond_resched()is a no-op whensystem_state!=SYSTEM_RUNNING, and so drivers/net/phy/phy.c'sstate_queueis never scheduled, therefore link detection doesn't work.I belive that the main problem is in cond_resched()[1], but despite how the
cond_resched()story ends, it might be a good idea to call msleep(1) instead ofcond_resched(), as suggested by Andrew Morton.[1] http://lkml.org/lkml/2009/7/7/463
Signed-off-by Anton Vorontsov <[email protected]>---
Quoted 39 lines · Andrew Morton
(belatedly cc'ing netdev)
Original diagnosis:
: Using early netconsole and gianfar driver this error pops up:
:
: netconsole: timeout waiting for carrier
:
: It appears that net/core/netpoll.c:netpoll_setup() is using :cond_resched()in a loop waiting for a carrier.
:
: The thing is thatcond_resched()is a no-op whensystem_state!= :SYSTEM_RUNNING, and so drivers/net/phy/phy.c'sstate_queueis never : scheduled, therefore link detection doesn't workQuoted 15 lines · On Thu, 9 Jul 2009 01:33:31 +0400 Anton Vorontsov <[email protected]>…
Quoted 13 lines · Andrew Morton
Quoted 5 lines · On Wed, 8 Jul 2009 09:12:30 -0700 (PDT) Linus Torvalds…
That said, I do agree that maybe
SYSTEM_RUNNINGisn't the right check. Testing that the scheduler is initialized may be the more correct one. I think theSYSTEM_RUNNINGone just comes from that being used for other debug issues.Agreed.
system_stateis too general.If we specifically want to know whether it is safe to call
schedule()then let's create a global booleanit_is_safe_to_call_scheduleand test that, rather than testing something which indirectly and unreliably implies "it is safe to call schedule". If that boolean already exists then no-brainer.Quoted 8 lines
Quoted 4 lines
All that being said, I wonder if the netconsole code should be using msleep(1) instead. Spinning on
cond_resched()is a bit rude. But one would have to verify that it is safe to callschedule()at this time, and for the netconsole caller, this is dubious.What do you mean by "verify that it is safe"? If it works, can I assume that it's safe? ;-) It works, fwiw.
Quoted 7 lines
netconsole is supposed to be available as early as possible in boot for obvious reasons. I'd say there's a decent risk now and in the future that netconsole will be initialised prior to the scheduler being available.
In fact, if "netconsole: timeout waiting for carrier" newly added to
netpoll_setup()a depedency on the scheduler being available then perhaps that was an incorrect change.
Quoted 7 lines
'git blame' says that carrier detection code didn't change since 2.6.12 (where git history starts), PHYLIB is using workqueue since its submission (2.6.13). And
SYSTEM_RUNNINGcheck was added in 2.6.16. So it's not a new dependency.The netpoll code is using
msleep()just a few lines belowcond_resched(), so we won't make things worse. ;-)
I think that's an improvement with or without the SYSTEM_RUNNING fix.
Quoted 18 lines
Thanks!
net/core/netpoll.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)net/core/netpoll.c +1
-1@@ -740,7 +740,7 @@ int netpoll_setup(struct netpoll *np) np->name);break ; } -cond_resched (); +msleep (1 ); }/* If carrier appears to come up instantly, we don't
Signature · 1 line
http://selenic.com : development and support for Mercurial and Linux
Hi!
Quoted 11 lines
Quoted 4 lines
That said, I do agree that maybe
SYSTEM_RUNNINGisn't the right check. Testing that the scheduler is initialized may be the more correct one. I think theSYSTEM_RUNNINGone just comes from that being used for other debug issues.Agreed.
system_stateis too general.If we specifically want to know whether it is safe to call
schedule()then let's create a global booleanit_is_safe_to_call_scheduleand test that, rather than testing something which indirectly and unreliably implies "it is safe to call schedule". If that boolean already exists then no-brainer.
or maybe we could embed that check into schedule(), just returning when scheduler is not ready?
And I always wondered... system_state is not protected by any kind of lock and is not atomic_t... Does it all work by mistake?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html