Module loading/unloading and "The Stop Machine"
Hi Rusty,
I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on. One of the patches makes stop machine ignore the isolated CPUs. People of course had questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)
As far as I can tell the stop machine is basically a safety net in case some locking and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.
The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.
It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.
So. Here is what I tried today on my Core2 Duo laptop
Quoted 7 lines
kernel/stop_machine.c +1
-0@@ -204,11 +204,14 @@ int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu)/* No CPUs can come up or down during this. */ lock_cpu_hotplug (); +/*
kernel/stop_machine.c int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu) · line 217
Quoted 10 lines
kernel/stop_machine.c +2
-0p = __stop_machine_run(fn, data, cpu); if (!IS_ERR(p)) ret = kthread_stop(p); else ret = PTR_ERR(p); +*/ + ret =fn (data);unlock_cpu_hotplug ();return ret;
ie Completely disabled stop machine. It just loads/unloads modules without full halt. I then ran three scripts:
while true; do
/sbin/modprobe -r uhci_hcd
/sbin/modprobe uhci_hcd
sleep 10
done
while true; do
/sbin/modprobe -r tg3
/sbin/modprobe tg3
sleep 2
done
while true; do
/usr/sbin/tcpdump -i eth0
done
The machine has a bunch of USB devices connected to it. The two most interesting are a Bluetooth dongle and a USB mouse. By loading/unloading UHCI driver we're touching Sysfs, USB stack, Bluetooth stack, HID layer, Input layer. The X is running and is using that USB mouse. The Bluetooth services are running too.
By loading/unloading TG3 driver we're touching sysfs, network stack (a bunch of layers). The machine is running NetworkManager and tcpdumping on the eth0 which is registered by TG3.
This is a pretty good stress test in general let alone the disabled stop machine.
I left all that running for the whole day while doing normal day to day things. Compiling a bunch of things, emails, office apps, etc. That's where I'm writing this email from :). It's still running all that :)
So the question is do we still need stop machine ? I must be missing something obvious. But things seem to be working pretty well without it. I certainly feel much better about at least ignoring isolated CPUs during stop machine execution. Which btw I've doing for a couple of years now on a wide range of the machines where people are inserting modules left and right.
What do you think ?
Thanx
Max
Quoted 66 lines · Max Krasnyansky
Hi Rusty,
I was hopping you could answer a couple of questions about module loading/unloading
and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on.
One of the patches makes stop machine ignore the isolated CPUs. People of course had
questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)As far as I can tell the stop machine is basically a safety net in case some locking
and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually
handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while
module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.
So. Here is what I tried today on my Core2 Duo laptop
Quoted 7 lines
kernel/stop_machine.c +1
-0@@ -204,11 +204,14 @@ int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu)/* No CPUs can come up or down during this. */ lock_cpu_hotplug (); +/* kernel/stop_machine.c int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu) · line 217
Quoted 10 lines
kernel/stop_machine.c +2
-0p = __stop_machine_run(fn, data, cpu); if (!IS_ERR(p)) ret = kthread_stop(p); else ret = PTR_ERR(p); +*/ + ret =fn (data);unlock_cpu_hotplug ();return ret;ie Completely disabled stop machine. It just loads/unloads modules without full halt. I then ran three scripts:
while true; do
/sbin/modprobe-ruhci_hcd
/sbin/modprobeuhci_hcd
sleep 10
donewhile true; do
/sbin/modprobe-r tg3
/sbin/modprobetg3
sleep 2
donewhile true; do
/usr/sbin/tcpdump-i eth0
doneThe machine has a bunch of USB devices connected to it. The two most interesting are a Bluetooth dongle and a USB mouse. By loading/unloading UHCI driver we're touching Sysfs, USB stack, Bluetooth stack, HID layer, Input layer. The X is running and is using that USB mouse. The Bluetooth services are running too. By loading/unloading TG3 driver we're touching sysfs, network stack (a bunch of layers).
The machine is running NetworkManager and tcpdumping on the eth0 which is registered by TG3.
This is a pretty good stress test in general let alone the disabled stop machine.I left all that running for the whole day while doing normal day to day things. Compiling a bunch of things, emails, office apps, etc. That's where I'm writing this email from :). It's still running all that :)
Quoted 8 lines
So the question is do we still need stop machine ? I must be missing something obvious.
But things seem to be working pretty well without it. I certainly feel much better about at least ignoring isolated CPUs during stop machine execution. Which btw I've doing for a couple of years now on a wide range of the machines where people are inserting modules left and right.What do you think ?
Thanx
Max
Quick update on this.
I've also ran
while true; do
sudo mount -o loop loopfs loopmnt && dd if=/dev/zero of=loopmnt/dummy bs=1M
sudo umount loopmnt
sleep 2
done
and
while true; do
/sbin/modprobe -r loop
/sbin/modprobe loop
sleep 1
done
in parallel on the Core2 Quad box for about 6 hours now. Same thing. No signs of problems whatsoever, with the "stop machine" completely disabled. Everything is working as expected. Here we're exercising sysfs, block and fs layers.
So I'm now even more eager to see your response :).
btw Does anyone else have a module load/unload scenario that definitely requires stop machine ?
Max
Hello, Max.
Quoted 7 lines · Max Krasnyansky
I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on. One of the patches makes stop machine ignore the isolated CPUs. People of course had questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)
Quoted 5 lines
As far as I can tell the stop machine is basically a safety net in case some locking and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.
Nope, it's integral part of module reference counting. When using refcnt for object lifetime management, the last put should be atomic against initial get of the object. This is usually achieved by acquiring the lock used for object lookup before putting or using atomic_dec_and_lock().
For module reference counts, this means that try_module_get() and try_stop_module() should be atomic. Note that modules don't use simple refcnt so the latter part isn't module_put() but the analogy still works. There are two ways to synchronize try_module_get() against try_stop_module() - the traditional is to grab lock in try_module_get() and use atomic_dec_and_lock() in try_stop_module(), which works but performance-wise bad because try_module_get() is used way much more than try_stop_module() is. For example, an IO command can go through several try_module_get()'s.
So, all the burden of synchronization is put onto try_stop_module(). Because all of the cpus on the machine are stopped and none of them has been stopped in the middle of non-preemptible code, __try_stop_module() is synchronized from try_module_get() even though all the synchronization try_module_get() does is get_cpu().
Quoted 8 lines
The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.
Quoted 5 lines
It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.
Without the stop_machine call, there's no synchronization between initial get and final put. Things will break.
Thanks.
Signature · 1 line
tejun
Hi Tejun,
Quoted 28 lines
Quoted 10 lines · Max Krasnyansky
I was hopping you could answer a couple of questions about module loading/unloading
and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on.
One of the patches makes stop machine ignore the isolated CPUs. People of course had
questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)As far as I can tell the stop machine is basically a safety net in case some locking
and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually
handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.Nope, it's integral part of module reference counting. When using refcnt for object lifetime management, the last put should be atomic against initial get of the object. This is usually achieved by acquiring the lock used for object lookup before putting or using
atomic_dec_and_lock().For module reference counts, this means that
try_module_get()andtry_stop_module()should be atomic. Note that modules don't use simple refcnt so the latter part isn'tmodule_put()but the analogy still works. There are two ways to synchronizetry_module_get()againsttry_stop_module()- the traditional is to grab lock intry_module_get()and useatomic_dec_and_lock()intry_stop_module(), which works but performance-wise bad becausetry_module_get()is used way much more thantry_stop_module()is. For example, an IO command can go through several try_module_get()'s.
Quoted 5 lines
So, all the burden of synchronization is put onto
try_stop_module(). Because all of the cpus on the machine are stopped and none of them has been stopped in the middle of non-preemptible code,__try_stop_module()is synchronized fromtry_module_get()even though all the synchronizationtry_module_get()does isget_cpu().
Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal. And it's plain broken for the use case that I mentioned during CPU isolation discussions. ie When user-space thread(s) prevent stopmachine kthread from running, in which
case machine simply hangs until those user-space threads exit.
Initially I assumed that it had to do with subsystems registration/unregistration being potentially unsafe if it's only for module ref counting there is gotta be a less expensive way. I'll think some more about it.
Quoted 8 lines
Quoted 5 lines
The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while
module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.
Without the
stop_machinecall, there's no synchronization between initial get and final put. Things will break.
Got it.
Thanks again for the explanation. I'll stare at the module code some more with what you said in mind.
Max
Quoted 6 lines · Max Krasnyanskiy
Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.
Module loading doesn't involve stop_machine last time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't stand stop_machine, simply don't unload a module.
Quoted 4 lines
And it's plain broken for the use case that I mentioned during CPU isolation discussions. ie When user-space thread(s) prevent stopmachine kthread from running, in which
case machine simply hangs until those user-space threads exit.
This I don't know nothing about. :-)
Signature · 1 line
tejun
Quoted 8 lines · Tejun Heo
Quoted 6 lines · Max Krasnyanskiy
Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.
Quoted 4 lines
Module loading doesn't involve
stop_machinelast time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't standstop_machine, simply don't unload a module.
static struct module *load_module(void __user *umod,
unsigned long len, const char __user *uargs)
{
...
/* Now sew it into the lists so we can get lockdep and oops
* info during argument parsing. Noone should access us, since
* strong_try_module_get() will fail. */
stop_machine_run(__link_module, mod, NR_CPUS);
...
}I actually rarely unload modules. The way I notice the problem in first place is when things started hanging when tun driver was autoloaded or when fs automounts triggered some auto loading. These days it's kind hard to have a semi-general purpose machine without module loading :).
Max
Quoted 17 lines · Max Krasnyanskiy
Quoted 8 lines · Tejun Heo
Quoted 6 lines · Max Krasnyanskiy
Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.
Quoted 4 lines
Module loading doesn't involve
stop_machinelast time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't standstop_machine, simply don't unload a module.static struct module *load_module(void
__user*umod,
unsigned long len, const char__user*uargs)
Quoted 9 lines
{
...
/* Now sew it into the lists so we can get lockdep and oops
* info during argument parsing. Noone should access us, since
* strong_try_module_get() will fail. */
stop_machine_run(__link_module, mod, NR_CPUS);
...
}Ah... right. That part doesn't have anything to do with module reference counting as the comment suggests and can probably be removed by updating how kallsyms synchronize against module load/unload.
Quoted 5 lines
I actually rarely unload modules. The way I notice the problem in first place is when things started hanging when tun driver was autoloaded or when fs automounts triggered some auto loading.
These days it's kind hard to have a semi-general purpose machine without module loading :).
Yeap, agreed.
Signature · 1 line
tejun
Quoted 26 lines · Tejun Heo
Quoted 15 lines · Max Krasnyanskiy
Quoted 11 lines · Tejun Heo
Quoted 6 lines · Max Krasnyanskiy
Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.
Module loading doesn't involve
stop_machinelast time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't standstop_machine, simply don't unload a module.static struct module *load_module(void
__user*umod,
unsigned long len, const char__user*uargs)Quoted 9 lines
{ ... /* Now sew it into the lists so we can get lockdep and oops * info during argument parsing. Noone should access us, since * strong_try_module_get() will fail. */ stop_machine_run(__link_module, mod, NR_CPUS); ... }
Quoted 3 lines
Ah... right. That part doesn't have anything to do with module reference counting as the comment suggests and can probably be removed by updating how kallsyms synchronize against module load/unload.
That list (updated by __link_module) is accessed in couple of other places. ie outside symbol lookup stuff used for kallsyms.
Max
Quoted 4 lines · Max Krasnyanskiy
static struct module *load_module(void
__user*umod,
unsigned long len, const char__user*uargs)
Quoted 9 lines
{
...
/* Now sew it into the lists so we can get lockdep and oops
* info during argument parsing. Noone should access us, since
* strong_try_module_get() will fail. */
stop_machine_run(__link_module, mod, NR_CPUS);
...
}Wow you found some really bad code. I bet it wouldn't be that difficult to fix the code to allow oops safe list insertion without using the big stop machine overkill hammer.
-Andi
Hi Andi,
Quoted 14 lines
Quoted 12 lines · Max Krasnyanskiy
static struct module *load_module(void
__user*umod,
unsigned long len, const char__user*uargs){ ... /* Now sew it into the lists so we can get lockdep and oops * info during argument parsing. Noone should access us, since * strong_try_module_get() will fail. */ stop_machine_run(__link_module, mod, NR_CPUS); ... }
Quoted 3 lines
Wow you found some really bad code. I bet it wouldn't be that difficult to fix the code to allow oops safe list insertion without using the big stop machine overkill hammer.
Let me know if you have something in mind. When I get a chance I'll stare some more at that code and try to come up with an alternative solution.
Thanx
Max
Quoted 14 lines · On Friday 22 February 2008 22:53:50 Andi Kleen wrote:
Quoted 12 lines · Max Krasnyanskiy
static struct module *load_module(void
__user*umod,
unsigned long len, const char__user*uargs){ ... /* Now sew it into the lists so we can get lockdep and oops * info during argument parsing. Noone should access us, since * strong_try_module_get() will fail. */ stop_machine_run(__link_module, mod, NR_CPUS); ... }
Quoted 3 lines
Wow you found some really bad code. I bet it wouldn't be that difficult to fix the code to allow oops safe list insertion without using the big stop machine overkill hammer.
Yes it's overkill and it's becoming more noticable with larger machines. (Also, stop_machine is heavier than it should be, Kathy is working on fixing that though).
Most obvious is to add __ "lockless, but I'm in trouble anyway" accessors, but last I looked they'd have to be plumbed through a fair bit of code.
Quoted 1 line
From a quick reading of the code, a lockless add is possible, but it's fragile
if one of the readers does something more complex in future.
Thanks,
Rusty.
Quoted 4 lines · Max Krasnyansky
Hi Rusty,
I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.
I'm curious to know why it is called `stop machine', which is a queer name without any relationship with its function.
Regards,
Jike
Quoted 8 lines · Jike Song
Quoted 4 lines · Max Krasnyansky
Hi Rusty,
I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.
I'm curious to know why it is called `stop machine', which is a queer name without any relationship with its function.
I guess it's "stop the rest of the machine" and run this. Maybe it's misnamed but stop_machine is kind of cool. :-)
Signature · 1 line
tejun
Quoted 8 lines · On Thursday 14 February 2008 16:02:06 Jike Song wrote:
Quoted 4 lines · Max Krasnyansky
Hi Rusty,
I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.
I'm curious to know why it is called `stop machine', which is a queer name without any relationship with its function.
Hi Jike,
I'm surprised.. stunned even, to hear you say this. stop_machine stops the
machine: every CPU is frozen with interrupts disabled. I am having trouble thinking of a better name.
"schedule_a_thread_on_every_cpu_and_then_disable_interrupts" is a little
long, and describes the implementation not the effect.
"run_and_make_sure_no_cpus_have_preempt_disabled" also fails the "will Linus
flame me?" test, I think.
"freeze_machine" has connotations of the freezer, which is completely
different from this. "do_machine" is shorter, but sounds vaguely naughty.
Quoted 2 lines
Regards,
Jike
Perhaps you were angling for "jike_machine"? It's kind of catchy, but I don't think that patch will be accepted even if you pushed it as a git tree.
Cheers,
Rusty.