Skip to contentSkip to conversation

Module loading/unloading and "The Stop Machine"

DISCUSSION Max Krasnyansky 5 participants 14 messages Feb 8, 2008 – Mar 4, 2008

lore
Share & tools
lore
#0 lore raw

Hi Rusty,

I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on. One of the patches makes stop machine ignore the isolated CPUs. People of course had questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)

As far as I can tell the stop machine is basically a safety net in case some locking and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.

The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.

It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.

So. Here is what I tried today on my Core2 Duo laptop

Quoted 7 lines
kernel/stop_machine.c +1 -0
@@ -204,11 +204,14 @@ int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu)
 
        /* No CPUs can come up or down during this. */
        lock_cpu_hotplug();
+/*

kernel/stop_machine.c int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu) · line 217

Quoted 10 lines
kernel/stop_machine.c +2 -0
        p = __stop_machine_run(fn, data, cpu);
        if (!IS_ERR(p))
                ret = kthread_stop(p);
        else
                ret = PTR_ERR(p);
+*/
+       ret = fn(data);
        unlock_cpu_hotplug();
 
        return ret;

ie Completely disabled stop machine. It just loads/unloads modules without full halt. I then ran three scripts:

while true; do
/sbin/modprobe -r uhci_hcd
/sbin/modprobe uhci_hcd
sleep 10
done

while true; do
/sbin/modprobe -r tg3
/sbin/modprobe tg3
sleep 2
done

while true; do
/usr/sbin/tcpdump -i eth0
done

The machine has a bunch of USB devices connected to it. The two most interesting are a Bluetooth dongle and a USB mouse. By loading/unloading UHCI driver we're touching Sysfs, USB stack, Bluetooth stack, HID layer, Input layer. The X is running and is using that USB mouse. The Bluetooth services are running too.
By loading/unloading TG3 driver we're touching sysfs, network stack (a bunch of layers). The machine is running NetworkManager and tcpdumping on the eth0 which is registered by TG3.
This is a pretty good stress test in general let alone the disabled stop machine.

I left all that running for the whole day while doing normal day to day things. Compiling a bunch of things, emails, office apps, etc. That's where I'm writing this email from :). It's still running all that :)

So the question is do we still need stop machine ? I must be missing something obvious. But things seem to be working pretty well without it. I certainly feel much better about at least ignoring isolated CPUs during stop machine execution. Which btw I've doing for a couple of years now on a wide range of the machines where people are inserting modules left and right.

What do you think ?

Thanx
Max

#1 lore raw
Quoted 66 lines · Max Krasnyansky

Hi Rusty,

I was hopping you could answer a couple of questions about module loading/unloading
and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on.
One of the patches makes stop machine ignore the isolated CPUs. People of course had
questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)

As far as I can tell the stop machine is basically a safety net in case some locking
and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually
handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.

The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while
module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.

It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.

So. Here is what I tried today on my Core2 Duo laptop

Quoted 7 lines
kernel/stop_machine.c +1 -0
@@ -204,11 +204,14 @@ int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu)
 
        /* No CPUs can come up or down during this. */
        lock_cpu_hotplug();
+/*

kernel/stop_machine.c int stop_machine_run(int (*fn)(void *), void *data, unsigned int cpu) · line 217

Quoted 10 lines
kernel/stop_machine.c +2 -0
        p = __stop_machine_run(fn, data, cpu);
        if (!IS_ERR(p))
                ret = kthread_stop(p);
        else
                ret = PTR_ERR(p);
+*/
+       ret = fn(data);
        unlock_cpu_hotplug();
 
        return ret;

ie Completely disabled stop machine. It just loads/unloads modules without full halt. I then ran three scripts:

while true; do
/sbin/modprobe -r uhci_hcd
/sbin/modprobe uhci_hcd
sleep 10
done

while true; do
/sbin/modprobe -r tg3
/sbin/modprobe tg3
sleep 2
done

while true; do
/usr/sbin/tcpdump -i eth0
done

The machine has a bunch of USB devices connected to it. The two most interesting are a Bluetooth dongle and a USB mouse. By loading/unloading UHCI driver we're touching Sysfs, USB stack, Bluetooth stack, HID layer, Input layer. The X is running and is using that USB mouse. The Bluetooth services are running too. By loading/unloading TG3 driver we're touching sysfs, network stack (a bunch of layers).
The machine is running NetworkManager and tcpdumping on the eth0 which is registered by TG3.
This is a pretty good stress test in general let alone the disabled stop machine.

I left all that running for the whole day while doing normal day to day things. Compiling a bunch of things, emails, office apps, etc. That's where I'm writing this email from :). It's still running all that :)

Quoted 8 lines

So the question is do we still need stop machine ? I must be missing something obvious.
But things seem to be working pretty well without it. I certainly feel much better about at least ignoring isolated CPUs during stop machine execution. Which btw I've doing for a couple of years now on a wide range of the machines where people are inserting modules left and right.

What do you think ?

Thanx
Max

Quick update on this.
I've also ran
while true; do
sudo mount -o loop loopfs loopmnt && dd if=/dev/zero of=loopmnt/dummy bs=1M
sudo umount loopmnt
sleep 2
done
and
while true; do
/sbin/modprobe -r loop
/sbin/modprobe loop
sleep 1
done
in parallel on the Core2 Quad box for about 6 hours now. Same thing. No signs of problems whatsoever, with the "stop machine" completely disabled. Everything is working as expected. Here we're exercising sysfs, block and fs layers.
So I'm now even more eager to see your response :).

btw Does anyone else have a module load/unload scenario that definitely requires stop machine ?

Max

#2 lore raw
In reply to Max Krasnyansky

Hello, Max.

Quoted 7 lines · Max Krasnyansky

I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on. One of the patches makes stop machine ignore the isolated CPUs. People of course had questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)

Quoted 5 lines

As far as I can tell the stop machine is basically a safety net in case some locking and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.

Nope, it's integral part of module reference counting. When using refcnt for object lifetime management, the last put should be atomic against initial get of the object. This is usually achieved by acquiring the lock used for object lookup before putting or using atomic_dec_and_lock().

For module reference counts, this means that try_module_get() and try_stop_module() should be atomic. Note that modules don't use simple refcnt so the latter part isn't module_put() but the analogy still works. There are two ways to synchronize try_module_get() against try_stop_module() - the traditional is to grab lock in try_module_get() and use atomic_dec_and_lock() in try_stop_module(), which works but performance-wise bad because try_module_get() is used way much more than try_stop_module() is. For example, an IO command can go through several try_module_get()'s.

So, all the burden of synchronization is put onto try_stop_module(). Because all of the cpus on the machine are stopped and none of them has been stopped in the middle of non-preemptible code, __try_stop_module() is synchronized from try_module_get() even though all the synchronization try_module_get() does is get_cpu().

Quoted 8 lines

The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.

Quoted 5 lines

It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.

Without the stop_machine call, there's no synchronization between initial get and final put. Things will break.

Thanks.

Signature · 1 line
tejun
#11 lore raw
In reply to Max Krasnyansky
Quoted 4 lines · Max Krasnyansky

Hi Rusty,

I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.

I'm curious to know why it is called `stop machine', which is a queer name without any relationship with its function.

Regards,
Jike

#12 lore raw
Quoted 8 lines · Jike Song
Quoted 4 lines · Max Krasnyansky

Hi Rusty,

I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.

I'm curious to know why it is called `stop machine', which is a queer name without any relationship with its function.

I guess it's "stop the rest of the machine" and run this. Maybe it's misnamed but stop_machine is kind of cool. :-)

Signature · 1 line
tejun
#3 lore raw
In reply to Tejun Heo

Hi Tejun,

Quoted 28 lines
Quoted 10 lines · Max Krasnyansky

I was hopping you could answer a couple of questions about module loading/unloading
and the stop machine.
There was a recent discussion on LKML about CPU isolation patches I'm working on.
One of the patches makes stop machine ignore the isolated CPUs. People of course had
questions about that. So I started looking into more details and got this silly, crazy idea that maybe we do not need the stop machine any more :)

As far as I can tell the stop machine is basically a safety net in case some locking
and recounting mechanisms aren't bullet proof. In other words if a subsystem can actually
handle registration/unregistration in a robust way, module loader/unloader does not necessarily have to halt entire machine in order to load/unload a module that belongs to that subsystem. I may of course be completely wrong on that.

Nope, it's integral part of module reference counting. When using refcnt for object lifetime management, the last put should be atomic against initial get of the object. This is usually achieved by acquiring the lock used for object lookup before putting or using atomic_dec_and_lock().

For module reference counts, this means that try_module_get() and try_stop_module() should be atomic. Note that modules don't use simple refcnt so the latter part isn't module_put() but the analogy still works. There are two ways to synchronize try_module_get() against try_stop_module() - the traditional is to grab lock in try_module_get() and use atomic_dec_and_lock() in try_stop_module(), which works but performance-wise bad because try_module_get() is used way much more than try_stop_module() is. For example, an IO command can go through several try_module_get()'s.

Quoted 5 lines

So, all the burden of synchronization is put onto try_stop_module(). Because all of the cpus on the machine are stopped and none of them has been stopped in the middle of non-preemptible code, __try_stop_module() is synchronized from try_module_get() even though all the synchronization try_module_get() does is get_cpu().

Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal. And it's plain broken for the use case that I mentioned during CPU isolation discussions. ie When user-space thread(s) prevent stopmachine kthread from running, in which
case machine simply hangs until those user-space threads exit.

Initially I assumed that it had to do with subsystems registration/unregistration being potentially unsafe if it's only for module ref counting there is gotta be a less expensive way. I'll think some more about it.

Quoted 8 lines
Quoted 5 lines

The problem with the stop machine is that it's a very very big gun :). In a sense that it totally kills all the latencies and stuff since the entire machine gets halted while
module is being (un)loaded. Which is a major issue for any realtime apps. Specifically for CPU isolation the issue is that high-priority rt user-space thread prevents stop machine threads from running and entire box just hangs waiting for it. I'm kind of surprised that folks who use monster boxes with over 100 CPUs have not complained. It's must be a huge hit for those machines to halt the entire thing.

It seems that over the last few years most subsystems got much better at locking and refcounting. And I'm hopping that we can avoid halting the entire machine these days. For CPU isolation in particular the solution is simple. We can just ignore isolated CPUs. What I'm trying to figure out is how safe it is and whether we can avoid full halt altogether.

Without the stop_machine call, there's no synchronization between initial get and final put. Things will break.

Got it.
Thanks again for the explanation. I'll stare at the module code some more with what you said in mind.

Max

#4 lore raw
Quoted 6 lines · Max Krasnyanskiy

Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.

Module loading doesn't involve stop_machine last time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't stand stop_machine, simply don't unload a module.

Quoted 4 lines

And it's plain broken for the use case that I mentioned during CPU isolation discussions. ie When user-space thread(s) prevent stopmachine kthread from running, in which
case machine simply hangs until those user-space threads exit.

This I don't know nothing about. :-)

Signature · 1 line
tejun
#5 lore raw
Quoted 8 lines · Tejun Heo
Quoted 6 lines · Max Krasnyanskiy

Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.

Quoted 4 lines

Module loading doesn't involve stop_machine last time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't stand stop_machine, simply don't unload a module.

static struct module *load_module(void __user *umod,
unsigned long len, const char __user *uargs)

{
     ...

     /* Now sew it into the lists so we can get lockdep and oops
        * info during argument parsing.  Noone should access us, since
        * strong_try_module_get() will fail. */
       stop_machine_run(__link_module, mod, NR_CPUS);
     ...
}

I actually rarely unload modules. The way I notice the problem in first place is when things started hanging when tun driver was autoloaded or when fs automounts triggered some auto loading. These days it's kind hard to have a semi-general purpose machine without module loading :).

Max

#6 lore raw
Quoted 17 lines · Max Krasnyanskiy
Quoted 8 lines · Tejun Heo
Quoted 6 lines · Max Krasnyanskiy

Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.

Quoted 4 lines

Module loading doesn't involve stop_machine last time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't stand stop_machine, simply don't unload a module.

static struct module *load_module(void __user *umod,
unsigned long len, const char __user *uargs)

Quoted 9 lines
{
     ...

     /* Now sew it into the lists so we can get lockdep and oops
        * info during argument parsing.  Noone should access us, since
        * strong_try_module_get() will fail. */
       stop_machine_run(__link_module, mod, NR_CPUS);
     ...
}

Ah... right. That part doesn't have anything to do with module reference counting as the comment suggests and can probably be removed by updating how kallsyms synchronize against module load/unload.

Quoted 5 lines

I actually rarely unload modules. The way I notice the problem in first place is when things started hanging when tun driver was autoloaded or when fs automounts triggered some auto loading.
These days it's kind hard to have a semi-general purpose machine without module loading :).

Yeap, agreed.

Signature · 1 line
tejun
#7 lore raw
Quoted 26 lines · Tejun Heo
Quoted 15 lines · Max Krasnyanskiy
Quoted 11 lines · Tejun Heo
Quoted 6 lines · Max Krasnyanskiy

Thanks for the info. I guess I missed that from the code. In any case that seems like a pretty heavy refcounting mechanism. In a sense that every time something is loaded or unloaded entire machine freezes, potentially for several milliseconds. Normally it's not a big deal. But once you get more and more CPUs and/or start using realtime apps this becomes a big deal.

Module loading doesn't involve stop_machine last time I checked. It's a big deal when unloading a module but it's actually a very good trade off because it makes much hotter path (module_get/put) much cheaper. If your application can't stand stop_machine, simply don't unload a module.

static struct module *load_module(void __user *umod,
unsigned long len, const char __user *uargs)

Quoted 9 lines
{
     ...

     /* Now sew it into the lists so we can get lockdep and oops
        * info during argument parsing.  Noone should access us, since
        * strong_try_module_get() will fail. */
       stop_machine_run(__link_module, mod, NR_CPUS);
     ...
}
Quoted 3 lines

Ah... right. That part doesn't have anything to do with module reference counting as the comment suggests and can probably be removed by updating how kallsyms synchronize against module load/unload.

That list (updated by __link_module) is accessed in couple of other places. ie outside symbol lookup stuff used for kallsyms.

Max

#8 lore raw
In reply to Max Krasnyanskiy
Quoted 4 lines · Max Krasnyanskiy

static struct module *load_module(void __user *umod,
unsigned long len, const char __user *uargs)

Quoted 9 lines
{
     ...

     /* Now sew it into the lists so we can get lockdep and oops
        * info during argument parsing.  Noone should access us, since
        * strong_try_module_get() will fail. */
       stop_machine_run(__link_module, mod, NR_CPUS);
     ...
}

Wow you found some really bad code. I bet it wouldn't be that difficult to fix the code to allow oops safe list insertion without using the big stop machine overkill hammer.

-Andi

#9 lore raw

Hi Andi,

Quoted 14 lines
Quoted 12 lines · Max Krasnyanskiy

static struct module *load_module(void __user *umod,
unsigned long len, const char __user *uargs)

{
     ...

     /* Now sew it into the lists so we can get lockdep and oops
        * info during argument parsing.  Noone should access us, since
        * strong_try_module_get() will fail. */
       stop_machine_run(__link_module, mod, NR_CPUS);
     ...
}
Quoted 3 lines

Wow you found some really bad code. I bet it wouldn't be that difficult to fix the code to allow oops safe list insertion without using the big stop machine overkill hammer.

Let me know if you have something in mind. When I get a chance I'll stare some more at that code and try to come up with an alternative solution.

Thanx
Max

#10 lore raw
In reply to Andi Kleen
Quoted 14 lines · On Friday 22 February 2008 22:53:50 Andi Kleen wrote:
Quoted 12 lines · Max Krasnyanskiy

static struct module *load_module(void __user *umod,
unsigned long len, const char __user *uargs)

{
     ...

     /* Now sew it into the lists so we can get lockdep and oops
        * info during argument parsing.  Noone should access us, since
        * strong_try_module_get() will fail. */
       stop_machine_run(__link_module, mod, NR_CPUS);
     ...
}
Quoted 3 lines

Wow you found some really bad code. I bet it wouldn't be that difficult to fix the code to allow oops safe list insertion without using the big stop machine overkill hammer.

Yes it's overkill and it's becoming more noticable with larger machines. (Also, stop_machine is heavier than it should be, Kathy is working on fixing that though).

Most obvious is to add __ "lockless, but I'm in trouble anyway" accessors, but last I looked they'd have to be plumbed through a fair bit of code.

Quoted 1 line

From a quick reading of the code, a lockless add is possible, but it's fragile

if one of the readers does something more complex in future.

Thanks,
Rusty.

#13 lore raw
In reply to Jike Song
Quoted 8 lines · On Thursday 14 February 2008 16:02:06 Jike Song wrote:
Quoted 4 lines · Max Krasnyansky

Hi Rusty,

I was hopping you could answer a couple of questions about module loading/unloading and the stop machine.

I'm curious to know why it is called `stop machine', which is a queer name without any relationship with its function.

Hi Jike,

I'm surprised.. stunned even, to hear you say this. stop_machine stops the
machine: every CPU is frozen with interrupts disabled. I am having trouble thinking of a better name.

"schedule_a_thread_on_every_cpu_and_then_disable_interrupts" is a little
long, and describes the implementation not the effect.

"run_and_make_sure_no_cpus_have_preempt_disabled" also fails the "will Linus
flame me?" test, I think.

"freeze_machine" has connotations of the freezer, which is completely
different from this. "do_machine" is shorter, but sounds vaguely naughty.

Quoted 2 lines

Regards,
Jike

Perhaps you were angling for "jike_machine"? It's kind of catchy, but I don't think that patch will be accepted even if you pushed it as a git tree.

Cheers,
Rusty.