Skip to contentSkip to conversation

Patch/resubmit linux-2.5.63-bk4 try_module_get simplification

DISCUSSION Adam J. Richter 3 participants 3 messages Mar 2, 2003 – Mar 7, 2003 Scheduler

lore
Share & tools
lore
#0 lore raw
Quoted 10 lines · Roman Zippel
Quoted 4 lines · Adam J. Richter

The following patch shrinks changes the implementation of
try_module_get() and friends to eliminate the special stopping of all CPU's when a module is unloaded. Instead, it uses a read/write semaphore in a perhaps slightly non-intuitive way.

Hmm, I was waiting a bit for Rusty's comment, but there isn't any... Anyway the patch below does the same, but it gets the module ref speculative and calls module_get_sync() if there is a problem.

That is a clever implemenation!

I do have a few questions and comments though.

Is there enough traffic on the module reference counts to make
this trade-off worthwhile? On x86, the module_ref array is 512 bytes per module (SMP_CACHE_BYTES=16 x NR_CPUS=32). For example, my gateway machine has 49 modules loaded right now, so that would be 24kB. Even in iptables, I would think that module reference counts should only be modified when a rule is added or removed (because you still need to maintain a separate usage count for each rule to know whether you can remove it, even if it's not from a loadable module).

If it's worthwhile to trade off that amount of memory usage
for that amount of reduction in cross-cpu bus traffic, then you probably should move unload_lock into each struct module rather than having it be a single statically variable, as it is not protecting any statically allocated data.

I also see a bigger corrolary of that trade-off. If there is
enough traffic to warrant a per-cpu approach for module reference counts, surely there should be other rw_semaphore users that experience more traffic on a smaller number of instances than module references. So, perhaps your code should be generalized to "struct big_fast_rw_sem". In particular, I think such a facility might be useful for the semaphore guarding name lists, such as network device names or filesystem type names (for example, file_systems_lock in fs/filesystems.c).

I posted a patch some time ago for a module_get() that never
failed but which could only be called when a one of these semaphore was held with at least a read lock, and required registration of the relevant semaphores during the module's initialization routine. ~90% of users of try_module_get users could use this interface and thereby avoid rarely used potentially buggy error branches; the remaining users would continue to try_module_get. It is precisely these cases where big_fast_rw_sem might be useful.

One common characteristic of all of the big_fast_rw_sem uses
that I have in mind, including module reference counts, is that the counter is statically allocated. This means that once per-cpu variables are supported in modules, it will make sense to use DEFINE_PER_CPU et al instead of declaring an array of NR_CPUS. This has the advantage that it may use less memory if the platform is able to determine a smaller maximum number of cpu's at run time, and can potentially produce faster code if the platform implements per-cpu using different memory mappings per cpu.

Adam J. Richter     __     ______________   575 Oroville Road
[email protected]     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America

"Free Software For The Rest Of Us."

#1 lore raw

Hi,

Quoted 2 lines · Adam J. Richter

Is there enough traffic on the module reference counts to make
this trade-off worthwhile?

I don't know, you have to ask that Rusty.
BTW the same trick is also possible with the old module count:

int try_inc_mod_count(struct module *mod)

{
	int res;

	if (mod) {
		__MOD_INC_USE_COUNT(mod);
		smp_mb__after_atomic_inc()
		if (unlikely(mod->flags & MOD_DELETED))
			goto check;
	}
	return 1;
check:
	res = 1;
	spin_lock(&unload_lock);
	if (mod->flags & MOD_DELETED) {
		__MOD_DEC_USE_COUNT(mod);
		res = 0;
	}
	spin_unlock(&unload_lock);
	return res;
}

(and a similiar change to sys_delete_module.)

bye, Roman

#2 lore raw
In reply to Adam J. Richter

In message <[email protected]> you write:

Quoted 3 lines

Is there enough traffic on the module reference counts to make
this trade-off worthwhile? On x86, the module_ref array is 512 bytes per module (SMP_CACHE_BYTES=16 x NR_CPUS=32). For example, my gateway

Good question. NR_CPUS=32 is a bad default for x86, to begin with. I have also been working on a better kmalloc_percpu routine, so modules (and, of course, anything else which can benefit) can switch across to that (if DECLARE_PER_CPU is to work in modules, we *need* something like this, which was my motivation).

I did write a "bigref" implementation a while back (attached below, probably bitrotted), which uses a fairly simple "go to single refcount when someone is watching" approach. It uses synchronize_kernel() to ensure that noone is currently doing the "if (!slow_mode) local_inc()", but I think a cleverer implementation might use something like Roman's speculative approach maybe?

Cheers,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Name: Bigrefs Implementation
Author: Rusty Russell
Status: Tested on 2.5.38

D: This is an implementation of cache-friendly reference counts. The
D: refcounters work in two modes: the normal mode just incs and decs a
D: cache-aligned per-cpu counter. When someone is waiting for the
D: reference count to hit 0, a flag is set and a shared reference
D: counter is decremented (which is slower).
D:
D: This uses a simple non-intrusive synchronize_kernel() primitive.

include/linux/bigref.h +81 -0
diff -urNp --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.5.42/include/linux/bigref.h working-2.5.42-bigrefs/include/linux/bigref.h
@@ -0,0 +1,81 @@
+#ifndef _LINUX_BIGREF_H
+#define _LINUX_BIGREF_H
+#include <linux/cache.h>
+#include <linux/smp.h>
+#include <linux/compiler.h>
+#include <linux/thread_info.h>
+#include <linux/preempt.h>
+#include <asm/atomic.h>
+
+/* Big reference counts for Linux.
+ *   (C) Copyright 2002 Paul Russell, IBM Corporation.
+ */
+
+struct bigref_percpu
+{
+	atomic_t counter;
+	int slow_mode;
+} ____cacheline_aligned_in_smp;
+
+struct bigref
+{
+	struct bigref_percpu ref[NR_CPUS];
+
+	atomic_t slow_count;
+	struct task_struct *waiter;
+};
+
+/* Initialize the bigref in slow mode */
+extern void bigref_init_slow(struct bigref *ref, int value);
+
+/* Move it to fast mode */
+extern void bigref_start_fast(struct bigref *ref);
+
+/* Get the approximate value */
+extern int bigref_val(struct bigref *ref);
+
+/* Wait for it refcount to hit zero (sleeps) */
+extern int bigref_wait_for_zero(struct bigref *ref, long task_state);
+
+/* Is it currently in fast mode? */
+static inline int bigref_is_fast(struct bigref *ref)
+{
+	/* Could use any cpu's, but this is cache-friendly if we're
+	   about to frob the counter */
+	return !ref->ref[smp_processor_id()].slow_mode;
+}
+
+extern void __bigref_inc(struct bigref *ref);
+extern void __bigref_dec(struct bigref *ref);
+
+/* We only need protection against local interrupts. */
+#ifndef __HAVE_LOCAL_INC
+#define local_inc(x) atomic_inc(x)
+#define local_dec(x) atomic_dec(x)
+#endif
+
+static inline void bigref_inc(struct bigref *ref)
+{
+	struct bigref_percpu *cpu;
+
+	cpu = &ref->ref[get_cpu()];
+	if (likely(!cpu->slow_mode))
+		local_inc(&cpu->counter);
+	else
+		__bigref_inc(ref);
+	put_cpu();
+}
+
+/* Return true if we were the last one to decrement it */
+static inline void bigref_dec(struct bigref *ref)
+{
+	struct bigref_percpu *cpu;
+
+	cpu = &ref->ref[get_cpu()];
+	if (likely(!cpu->slow_mode))
+		local_dec(&cpu->counter);
+	else
+		__bigref_dec(ref);
+	put_cpu();
+}
+#endif /* _LINUX_BIGREF_H */ 
include/linux/sched.h +2 -0
diff -urNp --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.5.42/include/linux/sched.h working-2.5.42-bigrefs/include/linux/sched.h
@@ -449,8 +449,10 @@ do { if (atomic_dec_and_test(&(tsk)->usa
 
 #if CONFIG_SMP
 extern void set_cpus_allowed(task_t *p, unsigned long new_mask);
+extern void synchronize_kernel(void);
 #else
 # define set_cpus_allowed(p, new_mask) do { } while (0)
+# define synchronize_kernel() do { } while (0)
 #endif
 
 extern void set_user_nice(task_t *p, long nice);
kernel/Makefile +2 -2
diff -urNp --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.5.42/kernel/Makefile working-2.5.42-bigrefs/kernel/Makefile
@@ -3,12 +3,12 @@
 #
 
 export-objs = signal.o sys.o kmod.o workqueue.o ksyms.o pm.o exec_domain.o \
-	      printk.o platform.o suspend.o dma.o module.o cpufreq.o
+	      printk.o platform.o suspend.o dma.o module.o cpufreq.o bigref.o
 
 obj-y     = sched.o fork.o exec_domain.o panic.o printk.o \
 	    module.o exit.o itimer.o time.o softirq.o resource.o \
 	    sysctl.o capability.o ptrace.o timer.o user.o \
-	    signal.o sys.o kmod.o workqueue.o futex.o platform.o pid.o
+	    signal.o sys.o kmod.o workqueue.o futex.o platform.o pid.o bigref.o
 
 obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o
 obj-$(CONFIG_SMP) += cpu.o
kernel/bigref.c +122 -0

122 lines not on this page · raw message

kernel/sched.c +44 -0

50 lines not on this page · raw message