hotplug resource limitation
PATCH 2/2 hotplug resource limitation
Hi all,
this is the second patch to implement hotplug resource limitation (relative to 2.6.7-rc2-mm2).
In some cases it is preferrable to adapt the number of concurrent hotplug processes on the fly in addition to set the number statically during boot. Additionally, it might be required to disable hotplug / kmod event delivery altogether for debugging purposes (e.g. if a module loaded automatically is crashing the machine).
This patch implements a new sysctl variable kernel.khelper_max (with proc interface /proc/sys/kernel/khelper_max) which holds the max # of concurrently running khelper threads. Setting this variable to '0' disables hotplug event delivery altogether. Events will not be lost but rather queued within the kernel (i.e. threads are cloned which wait on a semaphore to become free). This allows for a replay of missed events once the variable is increased again.
Further explanation on top of the patch.
Please apply.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke [email protected] SuSE Linux AG S390 & zSeries Maxfeldstraße 5 +49 911 74053 688 90409 Nürnberg http://www.suse.de
Restrict max number of concurrent hotplug processes (Part 2)
The max number of concurrent hotplug / kmod events might need to be changed at runtime. This also allows for disabling the delivery of hotplug / kmod events altogether for debugging reasons and later enabling them again, with a complete replay of all queued events.
The runtime variable is implemented as a new sysctl variable, which resets the global variable 'khelper_max' to the new value. Care has to be taken if the variable is decreased and some processes are already queued / blocked: As it's not possible to ensure that a call to down() will not be blocked we take the other route of simply not calling up() on exit from an already running process if the variable has been changed during runtime of the process.
The patch alse needed to change request_module() to dynamically allocate the process arguments, as now theses arguments might be needed by a delayed execution of the kmod thread.
include/linux/sysctl.h +3 -0
@@ -136,6 +136,7 @@ enumKERN_UNKNOWN_NMI_PANIC =66 ,/* int: unknown nmi panic flag */ KERN_INTERACTIVE =67 ,/* interactive tasks can have cpu bursts */ KERN_COMPUTE =68 ,/* adjust timeslices for a compute server */ + KERN_KHELPER_MAX=69, /* max # of concurrent khelper processes */ };
@@ -784,6 +785,8 @@ extern int proc_doulongvec_minmax(ctl_ta void __user *, size_t *);extern int proc_doulongvec_ms_jiffies_minmax (ctl_table *table,int ,struct file *,void __user *,size_t *); +extern int proc_dointvec_khelper (ctl_table *,int ,struct file *, +void __user *,size_t *); extern int do_sysctl (int __user *name, int nlen, void __user *oldval, size_t __user *oldlenp,
kernel/kmod.c +103 -8
153 lines not on this page · raw message
kernel/sysctl.c +30 -0
@@ -64,6 +64,7 @@ extern int sysctl_lower_zone_protection;extern int min_free_kbytes;extern int printk_ratelimit_jiffies;extern int printk_ratelimit_burst; +extern int khelper_max;#if defined(CONFIG_X86_LOCAL_APIC ) && defined(__i386__)int unknown_nmi_panic;
@@ -414,6 +415,14 @@ static ctl_table kern_table[] = { .proc_handler = &proc_dostring, .strategy = &sysctl_string, }, + { + .ctl_name = KERN_KHELPER_MAX, + .procname = "khelper_max", + .data = &khelper_max, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec_khelper, + },#endif #ifdef CONFIG_CHR_DEV_SG {
@@ -1952,6 +1961,27 @@ int proc_dointvec_userhz_jiffies(ctl_tab do_proc_dointvec_userhz_jiffies_conv,NULL); } +extern int khelper_modify_number (int diff); + +int proc_dointvec_khelper (ctl_table *table,int write,struct file *filp, +void __user *buffer,size_t *lenp) +{ +int maxval = max_threads /2 ; +struct do_proc_dointvec_minmax_conv_param param = { + .min = (int *) &zero, + .max = (int *) &maxval, + }; +int oldval = khelper_max, retval; + + retval =do_proc_dointvec (table, write, filp, buffer, lenp, + do_proc_dointvec_minmax_conv, ¶m); + +if (khelper_max != oldval) +khelper_modify_number (khelper_max - oldval); +return retval; +} + +#else /* CONFIG_PROC_FS */ int proc_dostring(ctl_table *table, int write, struct file *filp,
Quoted 7 lines · Hannes Reinecke
Hi all,
this is the second patch to implement hotplug resource limitation (relative to 2.6.7-rc2-mm2).In some cases it is preferrable to adapt the number of concurrent hotplug processes on the fly in addition to set the number statically during boot.
Why? This should be "auto-tuning". We don't want to provide yet-another-knob-for-people-to-tweak-from-userspace, right?
Quoted 3 lines
Additionally, it might be required to disable hotplug /
kmod event delivery altogether for debugging purposes (e.g. if a module loaded automatically is crashing the machine).
Ugh, that's just not a good thing at all. You can do that by running:
echo /bin/true > /proc/sys/kernel/hotplug
today if you have to. I don't like the ability to stop the kernel from running properly, like this patch allows you to.
thanks,
greg k-h
Quoted 11 lines · Greg KH
Quoted 4 lines · Hannes Reinecke
Hi all,
this is the second patch to implement hotplug resource limitation (relative to 2.6.7-rc2-mm2).In some cases it is preferrable to adapt the number of concurrent hotplug processes on the fly in addition to set the number statically during boot.
Why? This should be "auto-tuning". We don't want to provide yet-another-knob-for-people-to-tweak-from-userspace, right?
In principle you're right. But as we don't really now how much resources the installed hotplug program will require I don't really see another way.
Quoted 9 lines
Quoted 2 lines
Additionally, it might be required to disable hotplug / kmod event delivery altogether for debugging purposes (e.g. if a module loaded automatically is crashing the machine).
Ugh, that's just not a good thing at all. You can do that by running:
echo/bin/true>/proc/sys/kernel/hotplug
today if you have to. I don't like the ability to stop the kernel from running properly, like this patch allows you to.
But then you'll lose all events which happen in the meantime.
The dynamic setting is mainly intended for two scenarios:
- Booting. You can disable all events on boot with the kernel commandline and re-enable them once your hotplug subsystem is up and running. This way you can handle all (ok, all devices appearing in sysfs) device with hotplug events; there is no need to regenerate / fake all events which might have been missed. Currently you need two sets of scripts, one for configuring all devices until hotplug is running and another one used by hotplug.
- Debugging. Especially laptops or legacy free machines do have the problem that hotplug scripts might misconfigure the machine, e.g. by loading the wrong module. Of course it's possible to keep this 'disable hotplug events on request' (which should event available during boot) entirely in userspace. But it would increase the size of /sbin/hotplug (and hence the running time) by quite a bit; and this would bite us on every hotplug event generated resulting in a general slowdown of the hotplug subsystem. If we can stop the hotplug event delivery from within the kernel, we can keep the main hotplug script as small as possible while retaining the functionality of temporarily disabling all hotplug events.
What we could do, though, is to implement two semaphores, one for kmod calls and another one for hotplug calls. Then we can queue or event delay all hotplug events while any kmod call would just be blocked until the semaphore is free again.
Better approach?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke [email protected] SuSE Linux AG S390 & zSeries Maxfeldstraße 5 +49 911 74053 688 90409 Nürnberg http://www.suse.de
Quoted 12 lines · Hannes Reinecke
Quoted 11 lines · Greg KH
Quoted 7 lines · Hannes Reinecke
Hi all,
this is the second patch to implement hotplug resource limitation (relative to 2.6.7-rc2-mm2).In some cases it is preferrable to adapt the number of concurrent hotplug processes on the fly in addition to set the number statically during boot.
Quoted 5 lines
Quoted 3 lines
Why? This should be "auto-tuning". We don't want to provide yet-another-knob-for-people-to-tweak-from-userspace, right?
In principle you're right. But as we don't really now how much resources the installed hotplug program will require I don't really see another way.
But who is going to know how to tweak such a knob? How will an admin know they should reduce the number of outstanding processes?
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Additionally, it might be required to disable hotplug /
kmod event delivery altogether for debugging purposes (e.g. if a module loaded automatically is crashing the machine).Ugh, that's just not a good thing at all. You can do that by running:
echo/bin/true>/proc/sys/kernel/hotplug
today if you have to. I don't like the ability to stop the kernel from running properly, like this patch allows you to.But then you'll lose all events which happen in the meantime.
Yup.
Quoted 8 lines
The dynamic setting is mainly intended for two scenarios:
- Booting. You can disable all events on boot with the kernel commandline and re-enable them once your hotplug subsystem is up and running. This way you can handle all (ok, all devices appearing in sysfs) device with hotplug events; there is no need to regenerate / fake all events which might have been missed. Currently you need two sets of scripts, one for configuring all devices until hotplug is running and another one used by hotplug.
Or just put hotplug into your initramfs to catch all of the events from the beginning of kernel time.
Actually, we currently only have 1 set of scripts that do the hotplug work. We have an additional one that makes some "fake" hotplug events at init time to catch the events that we were not able to catch. So we still only have 1 main code path.
Quoted 8 lines
- Debugging. Especially laptops or legacy free machines do have the problem that hotplug scripts might misconfigure the machine, e.g. by loading the wrong module. Of course it's possible to keep this 'disable hotplug events on request' (which should event available during boot) entirely in userspace. But it would increase the size of
/sbin/hotplug(and hence the running time) by quite a bit; and this would bite us on every hotplug event generated resulting in a general slowdown of the hotplug subsystem.
I'd be very supprised if you could actually measure the hit of adding 2 more lines of bash to those hotplug scripts. They aren't exactly the fastest thing in the world in the first place, nor do they need to be.
Quoted 3 lines
If we can stop the hotplug event delivery from within the kernel, we can keep the main hotplug script as small as possible while retaining the functionality of temporarily disabling all hotplug events.
So, add kernel complexity in order to reduce userspace complexity? I don't think that argument is going to win here :)
Quoted 6 lines
What we could do, though, is to implement two semaphores, one for kmod calls and another one for hotplug calls. Then we can queue or event delay all hotplug events while any kmod call would just be blocked until the semaphore is free again.
Better approach?
Drop the whole thing? I really don't see the need for this part of the patch (part 1 makes sense for some limited memory machines like the s390 boxes.) But I'm always open to further convincing...
thanks,
greg k-h
Quoted 17 lines · Greg KH
Quoted 15 lines · Hannes Reinecke
Quoted 12 lines · Greg KH
Quoted 4 lines · Hannes Reinecke
Hi all,
this is the second patch to implement hotplug resource limitation (relative to 2.6.7-rc2-mm2).In some cases it is preferrable to adapt the number of concurrent hotplug processes on the fly in addition to set the number statically during boot.
Why? This should be "auto-tuning". We don't want to provide yet-another-knob-for-people-to-tweak-from-userspace, right?
Quoted 6 lines
Quoted 1 line
In principle you're right. But as we don't really now how much resources the installed hotplug program will require I don't really see another way.
But who is going to know how to tweak such a knob? How will an admin know they should reduce the number of outstanding processes?
Documentation?
Quoted 16 lines
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
Additionally, it might be required to disable hotplug / kmod event delivery altogether for debugging purposes (e.g. if a module loaded automatically is crashing the machine).
Ugh, that's just not a good thing at all. You can do that by running:
echo/bin/true>/proc/sys/kernel/hotplug
today if you have to. I don't like the ability to stop the kernel from running properly, like this patch allows you to.But then you'll lose all events which happen in the meantime.
Yup.
Quoted 8 lines
Quoted 3 lines
The dynamic setting is mainly intended for two scenarios:
- Booting. You can disable all events on boot with the kernel
commandline and re-enable them once your hotplug subsystem is up and running. This way you can handle all (ok, all devices appearing in sysfs) device with hotplug events; there is no need to regenerate / fake all events which might have been missed. Currently you need two sets of scripts, one for configuring all devices until hotplug is running and another one used by hotplug.Or just put hotplug into your initramfs to catch all of the events from the beginning of kernel time.
Unfortunately this does not quite work as of today.
Open issues:
- udp sockets are not available during early stages, making it impossible to use udevsend/udevd
- No initial hook is given, which would allow for a setup of the initramfs (e.g. mounting of /dev on ramfs). Relying to catch event with SEQNUM 0 is dodgy (as it might well be that we never get this one); checking on every event ditto (as one might want to use 'udev' as hotplug program).
- Synchronisation of configuration files is hard to solve; either you have to update the initramfs every time after a device has been reconfigured or you do some sort of auto-detection, in which case you have to pack in every module etc. into the initramfs.
Quoted 5 lines
Actually, we currently only have 1 set of scripts that do the hotplug work. We have an additional one that makes some "fake" hotplug events at init time to catch the events that we were not able to catch. So we still only have 1 main code path.
But IMHO the main advantage of having initramfs is that we can get rid of this 'coldplug' / fake hotplug event stuff as we're able to catch all events. Otherwise I don't really see the point in initramfs.
Quoted 9 lines
Quoted 3 lines
- Debugging. Especially laptops or legacy free machines do have the
problem that hotplug scripts might misconfigure the machine, e.g. by loading the wrong module. Of course it's possible to keep this 'disable hotplug events on request' (which should event available during boot) entirely in userspace. But it would increase the size of/sbin/hotplug(and hence the running time) by quite a bit; and this would bite us on every hotplug event generated resulting in a general slowdown of the hotplug subsystem.I'd be very supprised if you could actually measure the hit of adding 2 more lines of bash to those hotplug scripts. They aren't exactly the fastest thing in the world in the first place, nor do they need to be.
But we want to keep the initial 'hotplug' script as small/fast as possible, as this script is called by _every_ event.
Any subsequent script (e.g. agents) do not really matter, as they are called much more infrequently (here on my local desktop machine I have a ration of 10:1).
Quoted 9 lines
Quoted 4 lines
If we can stop the hotplug event delivery from within the kernel, we can keep the main hotplug script as small as possible while retaining the functionality of temporarily disabling all hotplug events.
So, add kernel complexity in order to reduce userspace complexity? I don't think that argument is going to win here :)
Unfair argument :-).
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke [email protected] SuSE Linux AG S390 & zSeries Maxfeldstraße 5 +49 911 74053 688 90409 Nürnberg http://www.suse.de
Quoted 9 lines · Hannes Reinecke
Quoted 8 lines · Greg KH
Quoted 3 lines · Hannes Reinecke
Quoted 1 line · Greg KH
On Mon, Aug 09, 2004 at 03:07:15PM +0200, Hannes Reinecke wrote:
I'd be very supprised if you could actually measure the hit of adding 2 more lines of bash to those hotplug scripts. They aren't exactly the fastest thing in the world in the first place, nor do they need to be.
On several architectures (most notably s390, but also IPF and PPC) the load generated by hotplug during boot can in actual practice become quite high. The bus scans can generate events of the order of 100 to 1000.
These machines are more I/O-focused and handle many devices in parallel. So having a quick and lean hotplug becomes very desirable.
Since these machines also tend to be used in HA situations, problem recovery time and therefore reboot time becomes an essential factor to decrease periods of unavailability.
Quoted 5 lines
But we want to keep the initial 'hotplug' script as small/fast as possible, as this script is called by _every_ event.
Any subsequent script (e.g. agents) do not really matter, as they are called much more infrequently (here on my local desktop machine I have a ration of 10:1).
Signature · 1 line
Olaf Dabrunz (od/odabrunz), SUSE Linux AG, Nürnberg