Patch/resubmit linux-2.5.63-bk4 try_module_get simplification
Quoted 10 lines · Roman Zippel
Quoted 4 lines · Adam J. Richter
The following patch shrinks changes the implementation of
try_module_get()and friends to eliminate the special stopping of all CPU's when a module is unloaded. Instead, it uses a read/write semaphore in a perhaps slightly non-intuitive way.Hmm, I was waiting a bit for Rusty's comment, but there isn't any... Anyway the patch below does the same, but it gets the module ref speculative and calls
module_get_sync()if there is a problem.
That is a clever implemenation!
I do have a few questions and comments though.
Is there enough traffic on the module reference counts to make
this trade-off worthwhile? On x86, the module_ref array is 512 bytes per module (SMP_CACHE_BYTES=16 x NR_CPUS=32). For example, my gateway machine has 49 modules loaded right now, so that would be 24kB. Even in iptables, I would think that module reference counts should only be modified when a rule is added or removed (because you still need to maintain a separate usage count for each rule to know whether you can remove it, even if it's not from a loadable module).
If it's worthwhile to trade off that amount of memory usage
for that amount of reduction in cross-cpu bus traffic, then you probably should move unload_lock into each struct module rather than having it be a single statically variable, as it is not protecting any statically allocated data.
I also see a bigger corrolary of that trade-off. If there is
enough traffic to warrant a per-cpu approach for module reference counts, surely there should be other rw_semaphore users that experience more traffic on a smaller number of instances than module references. So, perhaps your code should be generalized to "struct big_fast_rw_sem". In particular, I think such a facility might be useful for the semaphore guarding name lists, such as network device names or filesystem type names (for example, file_systems_lock in fs/filesystems.c).
I posted a patch some time ago for a module_get() that never
failed but which could only be called when a one of these semaphore was held with at least a read lock, and required registration of the relevant semaphores during the module's initialization routine. ~90% of users of try_module_get users could use this interface and thereby avoid rarely used potentially buggy error branches; the remaining users would continue to try_module_get. It is precisely these cases where big_fast_rw_sem might be useful.
One common characteristic of all of the big_fast_rw_sem uses
that I have in mind, including module reference counts, is that the counter is statically allocated. This means that once per-cpu variables are supported in modules, it will make sense to use DEFINE_PER_CPU et al instead of declaring an array of NR_CPUS. This has the advantage that it may use less memory if the platform is able to determine a smaller maximum number of cpu's at run time, and can potentially produce faster code if the platform implements per-cpu using different memory mappings per cpu.
Adam J. Richter __ ______________ 575 Oroville Road [email protected] \ / Milpitas, California 95035 +1 408 309-6081 | g g d r a s i l United States of America
"Free Software For The Rest Of Us."
Hi,
Quoted 2 lines · Adam J. Richter
Is there enough traffic on the module reference counts to make
this trade-off worthwhile?
I don't know, you have to ask that Rusty.
BTW the same trick is also possible with the old module count:
int try_inc_mod_count(struct module *mod)
{
int res;
if (mod) {
__MOD_INC_USE_COUNT(mod);
smp_mb__after_atomic_inc()
if (unlikely(mod->flags & MOD_DELETED))
goto check;
}
return 1;
check:
res = 1;
spin_lock(&unload_lock);
if (mod->flags & MOD_DELETED) {
__MOD_DEC_USE_COUNT(mod);
res = 0;
}
spin_unlock(&unload_lock);
return res;
}(and a similiar change to sys_delete_module.)
bye, Roman
In message <[email protected]> you write:
Quoted 3 lines
Is there enough traffic on the module reference counts to make
this trade-off worthwhile? On x86, themodule_refarray is 512 bytes per module (SMP_CACHE_BYTES=16 x NR_CPUS=32). For example, my gateway
Good question. NR_CPUS=32 is a bad default for x86, to begin with. I have also been working on a better kmalloc_percpu routine, so modules (and, of course, anything else which can benefit) can switch across to that (if DECLARE_PER_CPU is to work in modules, we *need* something like this, which was my motivation).
I did write a "bigref" implementation a while back (attached below, probably bitrotted), which uses a fairly simple "go to single refcount when someone is watching" approach. It uses synchronize_kernel() to ensure that noone is currently doing the "if (!slow_mode) local_inc()", but I think a cleverer implementation might use something like Roman's speculative approach maybe?
Cheers,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Name: Bigrefs Implementation
Author: Rusty Russell
Status: Tested on 2.5.38
D: This is an implementation of cache-friendly reference counts. The
D: refcounters work in two modes: the normal mode just incs and decs a
D: cache-aligned per-cpu counter. When someone is waiting for the
D: reference count to hit 0, a flag is set and a shared reference
D: counter is decremented (which is slower).
D:
D: This uses a simple non-intrusive synchronize_kernel() primitive.
include/linux/bigref.h +81 -0
@@ -0,0 +1,81 @@ +#ifndef _LINUX_BIGREF_H +#define _LINUX_BIGREF_H +#include <linux/cache.h> +#include <linux/smp.h> +#include <linux/compiler.h> +#include <linux/thread_info.h> +#include <linux/preempt.h> +#include <asm/atomic.h> + +/* Big reference counts for Linux. +* (C) Copyright 2002 Paul Russell, IBM Corporation. +*/ + +struct bigref_percpu +{ +atomic_t counter; +int slow_mode; +}____cacheline_aligned_in_smp ; + +struct bigref +{ +struct bigref_percpu ref[NR_CPUS ]; + +atomic_t slow_count; +struct task_struct *waiter; +}; + +/* Initialize the bigref in slow mode */ +extern void bigref_init_slow (struct bigref *ref,int value); + +/* Move it to fast mode */ +extern void bigref_start_fast (struct bigref *ref); + +/* Get the approximate value */ +extern int bigref_val (struct bigref *ref); + +/* Wait for it refcount to hit zero (sleeps) */ +extern int bigref_wait_for_zero (struct bigref *ref,long task_state); + +/* Is it currently in fast mode? */ +static inline int bigref_is_fast (struct bigref *ref) +{ +/* Could use any cpu's, but this is cache-friendly if we're +about to frob the counter */ +return !ref->ref[smp_processor_id ()].slow_mode; +} + +extern void __bigref_inc (struct bigref *ref); +extern void __bigref_dec (struct bigref *ref); + +/* We only need protection against local interrupts. */ +#ifndef __HAVE_LOCAL_INC +#define local_inc (x) atomic_inc(x) +#define local_dec (x) atomic_dec(x) +#endif + +static inline void bigref_inc (struct bigref *ref) +{ +struct bigref_percpu *cpu; + + cpu = &ref->ref[get_cpu ()]; +if (likely (!cpu->slow_mode)) +local_inc (&cpu->counter); +else +__bigref_inc (ref); +put_cpu (); +} + +/* Return true if we were the last one to decrement it */ +static inline void bigref_dec (struct bigref *ref) +{ +struct bigref_percpu *cpu; + + cpu = &ref->ref[get_cpu ()]; +if (likely (!cpu->slow_mode)) +local_dec (&cpu->counter); +else +__bigref_dec (ref); +put_cpu (); +} +#endif /* _LINUX_BIGREF_H */
include/linux/sched.h +2 -0
@@ -449,8 +449,10 @@ do { if (atomic_dec_and_test(&(tsk)->usa#if CONFIG_SMP extern void set_cpus_allowed (task_t *p,unsigned long new_mask); +extern void synchronize_kernel (void );#else # define set_cpus_allowed (p, new_mask) do { } while (0) +# define synchronize_kernel () do { } while (0)#endif extern void set_user_nice (task_t *p,long nice);
kernel/Makefile +2 -2
@@ -3,12 +3,12 @@ # export-objs = signal.o sys.o kmod.o workqueue.o ksyms.o pm.o exec_domain.o \ - printk.o platform.o suspend.o dma.o module.o cpufreq.o + printk.o platform.o suspend.o dma.o module.o cpufreq.o bigref.o obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ module.o exit.o itimer.o time.o softirq.o resource.o \ sysctl.o capability.o ptrace.o timer.o user.o \ - signal.o sys.o kmod.o workqueue.o futex.o platform.o pid.o + signal.o sys.o kmod.o workqueue.o futex.o platform.o pid.o bigref.o obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o obj-$(CONFIG_SMP) += cpu.o
kernel/bigref.c +122 -0
122 lines not on this page · raw message
kernel/sched.c +44 -0
50 lines not on this page · raw message