Skip to contentSkip to conversation

Optimize the cpu hotplug locking

PATCH In review Peter Zijlstra 4 participants 46 messages Oct 2, 2013 – Oct 8, 2013

Subsystems (3)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover Optimize the cpu hotplug locking

The current cpu hotplug lock is a single global lock; therefore excluding hotplug is a very expensive proposition even though it is rare occurence under normal operation.

There is a desire for a more light weight implementation of {get,put}_online_cpus() from both the NUMA scheduling as well as the -RT side.

The current hotplug lock is a full reader preference lock -- and thus supports reader recursion. However since we're making the read side lock much cheaper it is the expectation that it will also be used far more. Which in turn would lead to writer starvation.

Therefore the new lock proposed is completely fair; albeit somewhat expensive on the write side. This in turn means that we need a per-task nesting count to support reader recursion.

This patch-set is in 3 parts;

1) The new hotplug lock implementation, very fast on the read side, very expensive on the write side.

2) Some new rcu_sync primitive by Oleg

3) Employment of the rcy_sync thingy to optimize the write-side of the new hotplug lock.

author maintainer
#1 lore raw

PATCH 1/3 hotplug: Optimize {get,put}_online_cpus()

The current implementation of get_online_cpus() is global of nature and thus not suited for any kind of common usage.

Re-implement the current recursive r/w cpu hotplug lock such that the read side locks are as light as possible.

The current cpu hotplug lock is entirely reader biased; but since readers are expensive there aren't a lot of them about and writer starvation isn't a particular problem.

However by making the reader side more usable there is a fair chance it will get used more and thus the starvation issue becomes a real possibility.

Therefore this new implementation is fair, alternating readers and writers; this however requires per-task state to allow the reader recursion -- this new task_struct member is placed in a 4 byte hole on 64bit builds.

Many comments are contributed by Paul McKenney, and many previous attempts were shown to be inadequate by both Paul and Oleg; many thanks to them for persisting to poke holes in my attempts.

---
 include/linux/cpu.h   |   67 ++++++++++++++
 include/linux/sched.h |    3 
 kernel/cpu.c          |  226 ++++++++++++++++++++++++++++++++++++--------------
 kernel/sched/core.c   |    2 
 4 files changed, 235 insertions(+), 63 deletions(-)
include/linux/cpu.h +65 -2
@@ -16,6 +16,8 @@
 #include <linux/node.h>
 #include <linux/compiler.h>
 #include <linux/cpumask.h>
+#include <linux/percpu.h>
+#include <linux/sched.h>
 
 struct device;
 
@@ -173,10 +175,69 @@ extern struct bus_type cpu_subsys;
 #ifdef CONFIG_HOTPLUG_CPU
 /* Stop CPUs going up and down. */
 
+extern void cpu_hotplug_init_task(struct task_struct *p);
+
 extern void cpu_hotplug_begin(void);
 extern void cpu_hotplug_done(void);
-extern void get_online_cpus(void);
-extern void put_online_cpus(void);
+
+extern int __cpuhp_state;
+DECLARE_PER_CPU(unsigned int, __cpuhp_refcount);
+
+extern void __get_online_cpus(void);
+
+static inline void get_online_cpus(void)
+{
+	might_sleep();
+
+	/* Support reader recursion */
+	/* The value was >= 1 and remains so, reordering causes no harm. */
+	if (current->cpuhp_ref++)
+		return;
+
+	preempt_disable();
+	/*
+	 * We are in an RCU-sched read-side critical section, so the writer
+	 * cannot both change __cpuhp_state from readers_fast and start
+	 * checking counters while we are here. So if we see !__cpuhp_state,
+	 * we know that the writer won't be checking until we past the
+	 * preempt_enable() and that once the synchronize_sched() is done, the
+	 * writer will see anything we did within this RCU-sched read-side
+	 * critical section.
+	 */
+	if (likely(!__cpuhp_state))
+		__this_cpu_inc(__cpuhp_refcount);
+	else
+		__get_online_cpus(); /* Unconditional memory barrier. */
+	preempt_enable();
+	/*
+	 * The barrier() from preempt_enable() prevents the compiler from
+	 * bleeding the critical section out.
+	 */
+}
+
+extern void __put_online_cpus(void);
+
+static inline void put_online_cpus(void)
+{
+	/* The value was >= 1 and remains so, reordering causes no harm. */
+	if (--current->cpuhp_ref)
+		return;
+
+	/*
+	 * The barrier() in preempt_disable() prevents the compiler from
+	 * bleeding the critical section out.
+	 */
+	preempt_disable();
+	/*
+	 * Same as in get_online_cpus().
+	 */
+	if (likely(!__cpuhp_state))
+		__this_cpu_dec(__cpuhp_refcount);
+	else
+		__put_online_cpus(); /* Unconditional memory barrier. */
+	preempt_enable();
+}
+
 extern void cpu_hotplug_disable(void);
 extern void cpu_hotplug_enable(void);
 #define hotcpu_notifier(fn, pri)	cpu_notifier(fn, pri)
@@ -200,6 +261,8 @@ static inline void cpu_hotplug_driver_un
 
 #else		/* CONFIG_HOTPLUG_CPU */
 
+static inline void cpu_hotplug_init_task(struct task_struct *p) {}
+
 static inline void cpu_hotplug_begin(void) {}
 static inline void cpu_hotplug_done(void) {}
 #define get_online_cpus()	do { } while (0)
include/linux/sched.h +3 -0
@@ -1039,6 +1039,9 @@ struct task_struct {
 #ifdef CONFIG_SMP
 	struct llist_node wake_entry;
 	int on_cpu;
+#ifdef CONFIG_HOTPLUG_CPU
+	int cpuhp_ref;
+#endif
 	struct task_struct *last_wakee;
 	unsigned long wakee_flips;
 	unsigned long wakee_flip_decay_ts;
kernel/cpu.c +165 -61

253 lines not on this page · raw message

kernel/sched/core.c +2 -0
@@ -1635,6 +1635,8 @@ static void __sched_fork(struct task_str
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
 	p->numa_work.next = &p->numa_work;
 #endif /* CONFIG_NUMA_BALANCING */
+
+	cpu_hotplug_init_task(p);
 }
 
 #ifdef CONFIG_NUMA_BALANCING
author maintainer
#2 lore raw

Re: [PATCH 1/3] hotplug: Optimize {get,put}_online_cpus()

Quoted 6 lines · Peter Zijlstra
kernel/cpu.c +4 -0
+	if (atomic_dec_and_test(&cpuhp_waitcount))
+		wake_up(&cpuhp_writer);
+
+	goto again:
 }

kernel/cpu.c static int cpu_hotplug_disabled; · line 149

Quoted 10 lines
kernel/cpu.c +9 -0
 
+
+#define per_cpu_sum(var)						\
+({ 									\
+ 	typeof(var) __sum = 0;						\
+ 	int cpu;							\
+ 	for_each_possible_cpu(cpu)					\
+ 		__sum += per_cpu(var, cpu);				\
+ 	__sum;								\
+)}

I just noticed I forgot a refresh..

+++ b/kernel/cpu.c
@@ -120,7 +120,7 @@ void __get_online_cpus(void)

        if (atomic_dec_and_test(&cpuhp_waitcount))
                wake_up(&cpuhp_writer);

- goto again:
+ goto again;
}
EXPORT_SYMBOL_GPL(__get_online_cpus);

@@ -146,7 +146,7 @@ EXPORT_SYMBOL_GPL(__put_online_cpus);
        for_each_possible_cpu(cpu)                                      \
                __sum += per_cpu(var, cpu);                             \
        __sum;                                                          \

-)}
+})

/*
* See srcu_readers_active_idx_check() for a rather more detailed explanation.

#3 lore raw

Re: [PATCH 1/3] hotplug: Optimize {get,put}_online_cpus()

Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
kernel/cpu.c +4 -0
+	if (atomic_dec_and_test(&cpuhp_waitcount))
+		wake_up(&cpuhp_writer);
+
+	goto again:
 }
Quoted 12 lines

kernel/cpu.c static int cpu_hotplug_disabled; · line 149

Quoted 10 lines
kernel/cpu.c +9 -0
+
+#define per_cpu_sum(var)						\
+({ 									\
+ 	typeof(var) __sum = 0;						\
+ 	int cpu;							\
+ 	for_each_possible_cpu(cpu)					\
+ 		__sum += per_cpu(var, cpu);				\
+ 	__sum;								\
+)}

I just noticed I forgot a refresh..

And including these changes as well. ;-)

Thanx, Paul

Quoted 20 lines

+++ b/kernel/cpu.c

(quoted diff) +2 -2
@@ -120,7 +120,7 @@ void __get_online_cpus(void)
        if (atomic_dec_and_test(&cpuhp_waitcount))
                wake_up(&cpuhp_writer);
 
-       goto again:
+       goto again;
 }
 EXPORT_SYMBOL_GPL(__get_online_cpus);
 
@@ -146,7 +146,7 @@ EXPORT_SYMBOL_GPL(__put_online_cpus);
        for_each_possible_cpu(cpu)                                      \
                __sum += per_cpu(var, cpu);                             \
        __sum;                                                          \
-)}
+})
 
 /*
  * See srcu_readers_active_idx_check() for a rather more detailed explanation.
Reviewed-by on v1 1/3
#4 lore raw

Re: [PATCH 1/3] hotplug: Optimize {get,put}_online_cpus()

In reply to Peter Zijlstra
Quoted 19 lines · Peter Zijlstra

The current implementation of get_online_cpus() is global of nature and thus not suited for any kind of common usage.

Re-implement the current recursive r/w cpu hotplug lock such that the read side locks are as light as possible.

The current cpu hotplug lock is entirely reader biased; but since readers are expensive there aren't a lot of them about and writer starvation isn't a particular problem.

However by making the reader side more usable there is a fair chance it will get used more and thus the starvation issue becomes a real possibility.

Therefore this new implementation is fair, alternating readers and writers; this however requires per-task state to allow the reader recursion -- this new task_struct member is placed in a 4 byte hole on 64bit builds.

Quoted 5 lines

Many comments are contributed by Paul McKenney, and many previous attempts were shown to be inadequate by both Paul and Oleg; many thanks to them for persisting to poke holes in my attempts.

One change to a now-obsolete comment called out below, with that change:

Quoted 235 lines
---
 include/linux/cpu.h   |   67 ++++++++++++++
 include/linux/sched.h |    3 
 kernel/cpu.c          |  226 ++++++++++++++++++++++++++++++++++++--------------
 kernel/sched/core.c   |    2 
 4 files changed, 235 insertions(+), 63 deletions(-)
include/linux/cpu.h +65 -2
@@ -16,6 +16,8 @@
 #include <linux/node.h>
 #include <linux/compiler.h>
 #include <linux/cpumask.h>
+#include <linux/percpu.h>
+#include <linux/sched.h>
 
 struct device;
 
@@ -173,10 +175,69 @@ extern struct bus_type cpu_subsys;
 #ifdef CONFIG_HOTPLUG_CPU
 /* Stop CPUs going up and down. */
 
+extern void cpu_hotplug_init_task(struct task_struct *p);
+
 extern void cpu_hotplug_begin(void);
 extern void cpu_hotplug_done(void);
-extern void get_online_cpus(void);
-extern void put_online_cpus(void);
+
+extern int __cpuhp_state;
+DECLARE_PER_CPU(unsigned int, __cpuhp_refcount);
+
+extern void __get_online_cpus(void);
+
+static inline void get_online_cpus(void)
+{
+	might_sleep();
+
+	/* Support reader recursion */
+	/* The value was >= 1 and remains so, reordering causes no harm. */
+	if (current->cpuhp_ref++)
+		return;
+
+	preempt_disable();
+	/*
+	 * We are in an RCU-sched read-side critical section, so the writer
+	 * cannot both change __cpuhp_state from readers_fast and start
+	 * checking counters while we are here. So if we see !__cpuhp_state,
+	 * we know that the writer won't be checking until we past the
+	 * preempt_enable() and that once the synchronize_sched() is done, the
+	 * writer will see anything we did within this RCU-sched read-side
+	 * critical section.
+	 */
+	if (likely(!__cpuhp_state))
+		__this_cpu_inc(__cpuhp_refcount);
+	else
+		__get_online_cpus(); /* Unconditional memory barrier. */
+	preempt_enable();
+	/*
+	 * The barrier() from preempt_enable() prevents the compiler from
+	 * bleeding the critical section out.
+	 */
+}
+
+extern void __put_online_cpus(void);
+
+static inline void put_online_cpus(void)
+{
+	/* The value was >= 1 and remains so, reordering causes no harm. */
+	if (--current->cpuhp_ref)
+		return;
+
+	/*
+	 * The barrier() in preempt_disable() prevents the compiler from
+	 * bleeding the critical section out.
+	 */
+	preempt_disable();
+	/*
+	 * Same as in get_online_cpus().
+	 */
+	if (likely(!__cpuhp_state))
+		__this_cpu_dec(__cpuhp_refcount);
+	else
+		__put_online_cpus(); /* Unconditional memory barrier. */
+	preempt_enable();
+}
+
 extern void cpu_hotplug_disable(void);
 extern void cpu_hotplug_enable(void);
 #define hotcpu_notifier(fn, pri)	cpu_notifier(fn, pri)
@@ -200,6 +261,8 @@ static inline void cpu_hotplug_driver_un
 
 #else		/* CONFIG_HOTPLUG_CPU */
 
+static inline void cpu_hotplug_init_task(struct task_struct *p) {}
+
 static inline void cpu_hotplug_begin(void) {}
 static inline void cpu_hotplug_done(void) {}
 #define get_online_cpus()	do { } while (0)
include/linux/sched.h +3 -0
@@ -1039,6 +1039,9 @@ struct task_struct {
 #ifdef CONFIG_SMP
 	struct llist_node wake_entry;
 	int on_cpu;
+#ifdef CONFIG_HOTPLUG_CPU
+	int cpuhp_ref;
+#endif
 	struct task_struct *last_wakee;
 	unsigned long wakee_flips;
 	unsigned long wakee_flip_decay_ts;
kernel/cpu.c +87 -24
@@ -49,88 +49,192 @@ static int cpu_hotplug_disabled;
 
 #ifdef CONFIG_HOTPLUG_CPU
 
-static struct {
-	struct task_struct *active_writer;
-	struct mutex lock; /* Synchronizes accesses to refcount, */
-	/*
-	 * Also blocks the new readers during
-	 * an ongoing cpu hotplug operation.
-	 */
-	int refcount;
-} cpu_hotplug = {
-	.active_writer = NULL,
-	.lock = __MUTEX_INITIALIZER(cpu_hotplug.lock),
-	.refcount = 0,
-};
+enum { readers_fast = 0, readers_slow, readers_block };
 
-void get_online_cpus(void)
+int __cpuhp_state;
+EXPORT_SYMBOL_GPL(__cpuhp_state);
+
+DEFINE_PER_CPU(unsigned int, __cpuhp_refcount);
+EXPORT_PER_CPU_SYMBOL_GPL(__cpuhp_refcount);
+
+static atomic_t cpuhp_waitcount;
+static DECLARE_WAIT_QUEUE_HEAD(cpuhp_readers);
+static DECLARE_WAIT_QUEUE_HEAD(cpuhp_writer);
+
+void cpu_hotplug_init_task(struct task_struct *p)
 {
-	might_sleep();
-	if (cpu_hotplug.active_writer == current)
+	p->cpuhp_ref = 0;
+}
+
+void __get_online_cpus(void)
+{
+again:
+	__this_cpu_inc(__cpuhp_refcount);
+
+	/*
+	 * Due to having preemption disabled the decrement happens on
+	 * the same CPU as the increment, avoiding the
+	 * increment-on-one-CPU-and-decrement-on-another problem.
+	 *
+	 * And yes, if the reader misses the writer's assignment of
+	 * readers_block to __cpuhp_state, then the writer is
+	 * guaranteed to see the reader's increment.  Conversely, any
+	 * readers that increment their __cpuhp_refcount after the
+	 * writer looks are guaranteed to see the readers_block value,
+	 * which in turn means that they are guaranteed to immediately
+	 * decrement their __cpuhp_refcount, so that it doesn't matter
+	 * that the writer missed them.
+	 */
+
+	smp_mb(); /* A matches D */
+
+	if (likely(__cpuhp_state != readers_block))
 		return;
-	mutex_lock(&cpu_hotplug.lock);
-	cpu_hotplug.refcount++;
-	mutex_unlock(&cpu_hotplug.lock);
 
+	/*
+	 * Make sure an outgoing writer sees the waitcount to ensure we
+	 * make progress.
+	 */
+	atomic_inc(&cpuhp_waitcount);
+
+	/*
+	 * Per the above comment; we still have preemption disabled and
+	 * will thus decrement on the same CPU as we incremented.
+	 */
+	__put_online_cpus();
+
+	/*
+	 * We either call schedule() in the wait, or we'll fall through
+	 * and reschedule on the preempt_enable() in get_online_cpus().
+	 */
+	preempt_enable_no_resched();
+	__wait_event(cpuhp_readers, __cpuhp_state != readers_block);
+	preempt_disable();
+
+	/*
+	 * Given we've still got preempt_disabled and new cpu_hotplug_begin()
+	 * must do a synchronize_sched() we're guaranteed a successfull
+	 * acquisition this time -- even if we wake the current
+	 * cpu_hotplug_end() now.
+	 */
+	if (atomic_dec_and_test(&cpuhp_waitcount))
+		wake_up(&cpuhp_writer);
+
+	goto again:
 }
-EXPORT_SYMBOL_GPL(get_online_cpus);
+EXPORT_SYMBOL_GPL(__get_online_cpus);
 
-void put_online_cpus(void)
+void __put_online_cpus(void)
 {
-	if (cpu_hotplug.active_writer == current)
-		return;
-	mutex_lock(&cpu_hotplug.lock);
+	smp_mb(); /* B matches C */
+	/*
+	 * In other words, if they see our decrement (presumably to aggregate
+	 * zero, as that is the only time it matters) they will also see our
+	 * critical section.
+	 */
+	this_cpu_dec(__cpuhp_refcount);
+
+	/* Prod writer to recheck readers_active */
+	wake_up(&cpuhp_writer);
+}
+EXPORT_SYMBOL_GPL(__put_online_cpus);
+
+#define per_cpu_sum(var)						\
+({ 									\
+ 	typeof(var) __sum = 0;						\
+ 	int cpu;							\

kernel/cpu.c static int cpu_hotplug_disabled; · line 153

Quoted 10 lines
kernel/cpu.c +7 -2
+ 	for_each_possible_cpu(cpu)					\
+ 		__sum += per_cpu(var, cpu);				\
+ 	__sum;								\
+)}
 
-	if (WARN_ON(!cpu_hotplug.refcount))
-		cpu_hotplug.refcount++; /* try to fix things up */
+/*
+ * See srcu_readers_active_idx_check() for a rather more detailed explanation.
+ */

The above comment is now obsolete, suggest something like:

/*
* Return true if the modular sum of the __cpuhp_refcount per-CPU
* variables is zero. If this sum is zero, then it is stable
* due to the fact that if any newly arriving readers increment
* a given counter, they will immediately decrement that same
* counter.
*/

Quoted 135 lines
kernel/cpu.c +71 -35
+static bool cpuhp_readers_active_check(void)
+{
+	if (per_cpu_sum(__cpuhp_refcount) != 0)
+		return false;
 
-	if (!--cpu_hotplug.refcount && unlikely(cpu_hotplug.active_writer))
-		wake_up_process(cpu_hotplug.active_writer);
-	mutex_unlock(&cpu_hotplug.lock);
+	/*
+	 * If we observed the decrement; ensure we see the entire critical
+	 * section.
+	 */
+
+	smp_mb(); /* C matches B */
 
+	return true;
 }
-EXPORT_SYMBOL_GPL(put_online_cpus);
 
 /*
- * This ensures that the hotplug operation can begin only when the
- * refcount goes to zero.
- *
- * Note that during a cpu-hotplug operation, the new readers, if any,
- * will be blocked by the cpu_hotplug.lock
- *
- * Since cpu_hotplug_begin() is always called after invoking
- * cpu_maps_update_begin(), we can be sure that only one writer is active.
- *
- * Note that theoretically, there is a possibility of a livelock:
- * - Refcount goes to zero, last reader wakes up the sleeping
- *   writer.
- * - Last reader unlocks the cpu_hotplug.lock.
- * - A new reader arrives at this moment, bumps up the refcount.
- * - The writer acquires the cpu_hotplug.lock finds the refcount
- *   non zero and goes to sleep again.
- *
- * However, this is very difficult to achieve in practice since
- * get_online_cpus() not an api which is called all that often.
- *
+ * This will notify new readers to block and wait for all active readers to
+ * complete.
  */
 void cpu_hotplug_begin(void)
 {
-	cpu_hotplug.active_writer = current;
+	/*
+	 * Since cpu_hotplug_begin() is always called after invoking
+	 * cpu_maps_update_begin(), we can be sure that only one writer is
+	 * active.
+	 */
+	lockdep_assert_held(&cpu_add_remove_lock);
+
+	/* Allow reader-in-writer recursion. */
+	current->cpuhp_ref++;
+
+	/* Notify readers to take the slow path. */
+	__cpuhp_state = readers_slow;
+
+	/* See percpu_down_write(); guarantees all readers take the slow path */
+	synchronize_sched();
+
+	/*
+	 * Notify new readers to block; up until now, and thus throughout the
+	 * longish synchronize_sched() above, new readers could still come in.
+	 */
+	__cpuhp_state = readers_block;
 
-	for (;;) {
-		mutex_lock(&cpu_hotplug.lock);
-		if (likely(!cpu_hotplug.refcount))
-			break;
-		__set_current_state(TASK_UNINTERRUPTIBLE);
-		mutex_unlock(&cpu_hotplug.lock);
-		schedule();
-	}
+	smp_mb(); /* D matches A */
+
+	/*
+	 * If they don't see our writer of readers_block to __cpuhp_state,
+	 * then we are guaranteed to see their __cpuhp_refcount increment, and
+	 * therefore will wait for them.
+	 */
+
+	/* Wait for all now active readers to complete. */
+	wait_event(cpuhp_writer, cpuhp_readers_active_check());
 }
 
 void cpu_hotplug_done(void)
 {
-	cpu_hotplug.active_writer = NULL;
-	mutex_unlock(&cpu_hotplug.lock);
+	/*
+	 * Signal the writer is done, no fast path yet.
+	 *
+	 * One reason that we cannot just immediately flip to readers_fast is
+	 * that new readers might fail to see the results of this writer's
+	 * critical section.
+	 */
+	__cpuhp_state = readers_slow;
+	wake_up_all(&cpuhp_readers);
+
+	/*
+	 * The wait_event()/wake_up_all() prevents the race where the readers
+	 * are delayed between fetching __cpuhp_state and blocking.
+	 */
+
+	/* See percpu_up_write(); readers will no longer attempt to block. */
+	synchronize_sched();
+
+	/* Let 'em rip */
+	__cpuhp_state = readers_fast;
+	current->cpuhp_ref--;
+
+	/*
+	 * Wait for any pending readers to be running. This ensures readers
+	 * after writer and avoids writers starving readers.
+	 */
+	wait_event(cpuhp_writer, !atomic_read(&cpuhp_waitcount));
 }
 
 /*
kernel/sched/core.c +2 -0
@@ -1635,6 +1635,8 @@ static void __sched_fork(struct task_str
 	p->numa_scan_period = sysctl_numa_balancing_scan_delay;
 	p->numa_work.next = &p->numa_work;
 #endif /* CONFIG_NUMA_BALANCING */
+
+	cpu_hotplug_init_task(p);
 }
 
 #ifdef CONFIG_NUMA_BALANCING
author maintainer
#5 lore raw

PATCH 2/3 rcu: Create rcu_sync infrastructure

In reply to Peter Zijlstra

From: Oleg Nesterov <[email protected]>

It is functionally equivalent to

        struct xxx_struct {
                atomic_t counter;
        };

        static inline bool xxx_is_idle(struct xxx_struct *xxx)
        {
                return atomic_read(&xxx->counter) == 0;
        }

        static inline void xxx_enter(struct xxx_struct *xxx)
        {
                atomic_inc(&xxx->counter);
                synchronize_sched();
        }

        static inline void xxx_enter(struct xxx_struct *xxx)
        {
                synchronize_sched();
                atomic_dec(&xxx->counter);
        }

except: it records the state and synchronize_sched() is only called by xxx_enter() and only if necessary.

---
 include/linux/rcusync.h |   64 ++++++++++++++++++++++++++++
 kernel/Makefile         |    3 -
 kernel/rcusync.c        |  108 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 174 insertions(+), 1 deletion(-)
include/linux/rcusync.h new file +64 -0
@@ -0,0 +1,64 @@
+#ifndef _LINUX_RCUSYNC_H_
+#define _LINUX_RCUSYNC_H_
+
+#include <linux/wait.h>
+#include <linux/rcupdate.h>
+
+struct rcu_sync_struct {
+	int			gp_state;
+	int			gp_count;
+	wait_queue_head_t	gp_wait;
+
+	int			cb_state;
+	struct rcu_head		cb_head;
+
+	void (*sync)(void);
+	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+};
+
+#define ___RCU_SYNC_INIT(name)						\
+	.gp_state = 0,							\
+	.gp_count = 0,							\
+	.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),		\
+	.cb_state = 0
+
+#define __RCU_SCHED_SYNC_INIT(name) {					\
+	___RCU_SYNC_INIT(name),						\
+	.sync = synchronize_sched,					\
+	.call = call_rcu_sched,						\
+}
+
+#define __RCU_BH_SYNC_INIT(name) {					\
+	___RCU_SYNC_INIT(name),						\
+	.sync = synchronize_rcu_bh,					\
+	.call = call_rcu_bh,						\
+}
+
+#define __RCU_SYNC_INIT(name) {						\
+	___RCU_SYNC_INIT(name),						\
+	.sync = synchronize_rcu,					\
+	.call = call_rcu,						\
+}
+
+#define DEFINE_RCU_SCHED_SYNC(name)					\
+	struct rcu_sync_struct name = __RCU_SCHED_SYNC_INIT(name)
+
+#define DEFINE_RCU_BH_SYNC(name)					\
+	struct rcu_sync_struct name = __RCU_BH_SYNC_INIT(name)
+
+#define DEFINE_RCU_SYNC(name)						\
+	struct rcu_sync_struct name = __RCU_SYNC_INIT(name)
+
+static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
+{
+	return !rss->gp_state; /* GP_IDLE */
+}
+
+enum rcu_sync_type { RCU_SYNC, RCU_SCHED_SYNC, RCU_BH_SYNC };
+
+extern void rcu_sync_init(struct rcu_sync_struct *, enum rcu_sync_type);
+extern void rcu_sync_enter(struct rcu_sync_struct *);
+extern void rcu_sync_exit(struct rcu_sync_struct *);
+
+#endif /* _LINUX_RCUSYNC_H_ */
+
kernel/Makefile +2 -1
@@ -10,7 +10,8 @@ obj-y     = fork.o exec_domain.o panic.o
 	    kthread.o wait.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o cred.o reboot.o \
-	    async.o range.o groups.o lglock.o smpboot.o
+	    async.o range.o groups.o lglock.o smpboot.o \
+	    rcusync.o
 
 ifdef CONFIG_FUNCTION_TRACER
 # Do not trace debug files and internal ftrace files
kernel/rcusync.c new file +108 -0

108 lines not on this page · raw message

#6 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 2 lines · Peter Zijlstra

From: Oleg Nesterov <[email protected]>

Thanks! I was writing the patch, and I chose almost the same naming ;)

Quoted 1 line

In fact I'd like to add my sob to 1/3 and 3/3 as well.

Paul, to remind, this is only the first step. I am going to send the following improvements:

1. Add rcu_sync->exlusive. The change is simple, just we need s/wait_queue_head_t/completion/ in rcu_sync_struct and a couple of "if (rss->exclusive)" checks in enter/exit.

2. rcu_sync_enter() should return !!need_sync. This can help in exclusive mode.

3. rcu_sync_struct needs more function pointers (perhaps we should add a single rcu_sync_struct->ops pointer but this is minor). See below.

But let me repeat just in case, we should do this later. And once this series is applied, I'll change percpu_rw_semaphore.

include/linux/rcusync.h line 16

Quoted 10 lines
include/linux/rcusync.h new file +10 -0
+struct rcu_sync_struct {
+	int			gp_state;
+	int			gp_count;
+	wait_queue_head_t	gp_wait;
+
+	int			cb_state;
+	struct rcu_head		cb_head;
+
+	void (*sync)(void);
+	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));

Yes, and we also need rcu_sync_struct->barrier(). From the patch I was working on:

	void rcu_sync_wait_for_callback(struct rcu_sync *sync)
	{
		int cb_state;

		BUG_ON(sync->gp_count);

		spin_lock_irq(&sync->state_lock);
		if (sync->cb_state == CB_REPLAY)
			sync->cb_state = CB_PENDING;
		cb_state = sync->cb_state;
		spin_unlock_irq(&sync->state_lock);

		if (cb_state != CB_IDLE) {
			rcu_barrier_sched();
			BUG_ON(sync->cb_state != CB_IDLE);
		}
	}

It should be called if you are going to kfree the object.

Perhaps another rcu_sync_struct->state_change(new_state) callback (set by the user) makes sense too, this can help (for example) to implement the array of semaphores with a single rcu_sync_struct (freeze_super).

Thanks.

Oleg.

#7 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 23 lines · Oleg Nesterov
Quoted 2 lines · Peter Zijlstra

From: Oleg Nesterov <[email protected]>

Thanks! I was writing the patch, and I chose almost the same naming ;)

Quoted 1 line

In fact I'd like to add my sob to 1/3 and 3/3 as well.

Paul, to remind, this is only the first step. I am going to send the following improvements:

1. Add rcu_sync->exlusive. The change is simple, just we need s/wait_queue_head_t/completion/ in rcu_sync_struct and a couple of "if (rss->exclusive)" checks in enter/exit.

2. rcu_sync_enter() should return !!need_sync. This can help in exclusive mode.

Quoted 6 lines

3. rcu_sync_struct needs more function pointers (perhaps we should add a single rcu_sync_struct->ops pointer but this is minor). See below.

But let me repeat just in case, we should do this later. And once this series is applied, I'll change percpu_rw_semaphore.

I took this into account in my review, the upgrades would be good! ;-)

Thanx, Paul

Quoted 42 lines

include/linux/rcusync.h line 16

Quoted 10 lines
include/linux/rcusync.h new file +10 -0
+struct rcu_sync_struct {
+	int			gp_state;
+	int			gp_count;
+	wait_queue_head_t	gp_wait;
+
+	int			cb_state;
+	struct rcu_head		cb_head;
+
+	void (*sync)(void);
+	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));

Yes, and we also need rcu_sync_struct->barrier(). From the patch I was working on:

	void rcu_sync_wait_for_callback(struct rcu_sync *sync)
	{
		int cb_state;

		BUG_ON(sync->gp_count);

		spin_lock_irq(&sync->state_lock);
		if (sync->cb_state == CB_REPLAY)
			sync->cb_state = CB_PENDING;
		cb_state = sync->cb_state;
		spin_unlock_irq(&sync->state_lock);

		if (cb_state != CB_IDLE) {
			rcu_barrier_sched();
			BUG_ON(sync->cb_state != CB_IDLE);
		}
	}

It should be called if you are going to kfree the object.

Perhaps another rcu_sync_struct->state_change(new_state) callback (set by the user) makes sense too, this can help (for example) to implement the array of semaphores with a single rcu_sync_struct (freeze_super).

Thanks.

Oleg.

author maintainer
#8 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Oleg Nesterov
Quoted 11 lines · Oleg Nesterov
Quoted 2 lines · Peter Zijlstra

From: Oleg Nesterov <[email protected]>

Thanks! I was writing the patch, and I chose almost the same naming ;)

Quoted 1 line

In fact I'd like to add my sob to 1/3 and 3/3 as well.

I think the rules for SOB are that you either have to write or pass the patch along in an aggregate work; you did neither -- although I much value your contributions.

Can I make that a reviewed-by for both instead?

#9 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Peter Zijlstra
Quoted 25 lines · Peter Zijlstra

From: Oleg Nesterov <[email protected]>

It is functionally equivalent to

        struct xxx_struct {
                atomic_t counter;
        };

        static inline bool xxx_is_idle(struct xxx_struct *xxx)
        {
                return atomic_read(&xxx->counter) == 0;
        }

        static inline void xxx_enter(struct xxx_struct *xxx)
        {
                atomic_inc(&xxx->counter);
                synchronize_sched();
        }

        static inline void xxx_enter(struct xxx_struct *xxx)
        {
                synchronize_sched();
                atomic_dec(&xxx->counter);
        }
Quoted 2 lines

except: it records the state and synchronize_sched() is only called by xxx_enter() and only if necessary.

Could you please make the "xxx_" above match the prefix below?

One comment below for the code's self-defense.

Thanx, Paul

Quoted 54 lines
---
 include/linux/rcusync.h |   64 ++++++++++++++++++++++++++++
 kernel/Makefile         |    3 -
 kernel/rcusync.c        |  108 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 174 insertions(+), 1 deletion(-)
include/linux/rcusync.h new file +43 -0
@@ -0,0 +1,64 @@
+#ifndef _LINUX_RCUSYNC_H_
+#define _LINUX_RCUSYNC_H_
+
+#include <linux/wait.h>
+#include <linux/rcupdate.h>
+
+struct rcu_sync_struct {
+	int			gp_state;
+	int			gp_count;
+	wait_queue_head_t	gp_wait;
+
+	int			cb_state;
+	struct rcu_head		cb_head;
+
+	void (*sync)(void);
+	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+};
+
+#define ___RCU_SYNC_INIT(name)						\
+	.gp_state = 0,							\
+	.gp_count = 0,							\
+	.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),		\
+	.cb_state = 0
+
+#define __RCU_SCHED_SYNC_INIT(name) {					\
+	___RCU_SYNC_INIT(name),						\
+	.sync = synchronize_sched,					\
+	.call = call_rcu_sched,						\
+}
+
+#define __RCU_BH_SYNC_INIT(name) {					\
+	___RCU_SYNC_INIT(name),						\
+	.sync = synchronize_rcu_bh,					\
+	.call = call_rcu_bh,						\
+}
+
+#define __RCU_SYNC_INIT(name) {						\
+	___RCU_SYNC_INIT(name),						\
+	.sync = synchronize_rcu,					\
+	.call = call_rcu,						\
+}
+
+#define DEFINE_RCU_SCHED_SYNC(name)					\

include/linux/rcusync.h line 53

Quoted 10 lines
include/linux/rcusync.h new file +10 -0
+	struct rcu_sync_struct name = __RCU_SCHED_SYNC_INIT(name)
+
+#define DEFINE_RCU_BH_SYNC(name)					\
+	struct rcu_sync_struct name = __RCU_BH_SYNC_INIT(name)
+
+#define DEFINE_RCU_SYNC(name)						\
+	struct rcu_sync_struct name = __RCU_SYNC_INIT(name)
+
+static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
+{

It would be nice to validate that we are in the corresponding type of RCU read-side critical section, but I would hate to pay a function call through a pointer for this if !PROVE_RCU. But even more, I would hate rare bugs due to someone forgetting an rcu_read_lock().

How about the something like the following, where ->read_side_check() gets rcu_read_lock_held(), rcu_read_lock_bh_held(), or rcu_read_lock_sched_held(), as appropriate?

#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

	rcu_sync_is_idle_check(rss);
Quoted 136 lines
include/linux/rcusync.h new file +11 -0
+	return !rss->gp_state; /* GP_IDLE */
+}
+
+enum rcu_sync_type { RCU_SYNC, RCU_SCHED_SYNC, RCU_BH_SYNC };
+
+extern void rcu_sync_init(struct rcu_sync_struct *, enum rcu_sync_type);
+extern void rcu_sync_enter(struct rcu_sync_struct *);
+extern void rcu_sync_exit(struct rcu_sync_struct *);
+
+#endif /* _LINUX_RCUSYNC_H_ */
+
kernel/Makefile +2 -1
@@ -10,7 +10,8 @@ obj-y     = fork.o exec_domain.o panic.o
 	    kthread.o wait.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o cred.o reboot.o \
-	    async.o range.o groups.o lglock.o smpboot.o
+	    async.o range.o groups.o lglock.o smpboot.o \
+	    rcusync.o
 
 ifdef CONFIG_FUNCTION_TRACER
 # Do not trace debug files and internal ftrace files
kernel/rcusync.c new file +108 -0
@@ -0,0 +1,108 @@
+
+#include <linux/rcusync.h>
+#include <linux/sched.h>
+
+enum { GP_IDLE = 0, GP_PENDING, GP_PASSED };
+enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY };
+
+#define	rss_lock	gp_wait.lock
+
+void rcu_sync_init(struct rcu_sync_struct *rss, enum rcu_sync_type type)
+{
+	memset(rss, 0, sizeof(*rss));
+	init_waitqueue_head(&rss->gp_wait);
+
+	switch (type) {
+	case RCU_SYNC:
+		rss->sync = synchronize_rcu;
+		rss->call = call_rcu;
+		break;
+
+	case RCU_SCHED_SYNC:
+		rss->sync = synchronize_sched;
+		rss->call = call_rcu_sched;
+		break;
+
+	case RCU_BH_SYNC:
+		rss->sync = synchronize_rcu_bh;
+		rss->call = call_rcu_bh;
+		break;
+	}
+}
+
+void rcu_sync_enter(struct rcu_sync_struct *rss)
+{
+	bool need_wait, need_sync;
+
+	spin_lock_irq(&rss->rss_lock);
+	need_wait = rss->gp_count++;
+	need_sync = rss->gp_state == GP_IDLE;
+	if (need_sync)
+		rss->gp_state = GP_PENDING;
+	spin_unlock_irq(&rss->rss_lock);
+
+	BUG_ON(need_wait && need_sync);
+
+	if (need_sync) {
+		rss->sync();
+		rss->gp_state = GP_PASSED;
+		wake_up_all(&rss->gp_wait);
+	} else if (need_wait) {
+		wait_event(rss->gp_wait, rss->gp_state == GP_PASSED);
+	} else {
+		/*
+		 * Possible when there's a pending CB from a rcu_sync_exit().
+		 * Nobody has yet been allowed the 'fast' path and thus we can
+		 * avoid doing any sync(). The callback will get 'dropped'.
+		 */
+		BUG_ON(rss->gp_state != GP_PASSED);
+	}
+}
+
+static void rcu_sync_func(struct rcu_head *rcu)
+{
+	struct rcu_sync_struct *rss =
+		container_of(rcu, struct rcu_sync_struct, cb_head);
+	unsigned long flags;
+
+
+	BUG_ON(rss->gp_state != GP_PASSED);
+	BUG_ON(rss->cb_state == CB_IDLE);
+
+	spin_lock_irqsave(&rss->rss_lock, flags);
+	if (rss->gp_count) {
+		/*
+		 * A new rcu_sync_begin() has happened; drop the callback.
+		 */
+		rss->cb_state = CB_IDLE;
+	} else if (rss->cb_state == CB_REPLAY) {
+		/*
+		 * A new rcu_sync_exit() has happened; requeue the callback
+		 * to catch a later GP.
+		 */
+		rss->cb_state = CB_PENDING;
+		rss->call(&rss->cb_head, rcu_sync_func);
+	} else {
+		/*
+		 * We're at least a GP after rcu_sync_exit(); eveybody will now
+		 * have observed the write side critical section. Let 'em rip!.
+		 */
+		rss->cb_state = CB_IDLE;
+		rss->gp_state = GP_IDLE;
+	}
+	spin_unlock_irqrestore(&rss->rss_lock, flags);
+}
+
+void rcu_sync_exit(struct rcu_sync_struct *rss)
+{
+	spin_lock_irq(&rss->rss_lock);
+	if (!--rss->gp_count) {
+		if (rss->cb_state == CB_IDLE) {
+			rss->cb_state = CB_PENDING;
+			rss->call(&rss->cb_head, rcu_sync_func);
+		} else if (rss->cb_state == CB_PENDING) {
+			rss->cb_state = CB_REPLAY;
+		}
+	}
+	spin_unlock_irq(&rss->rss_lock);
+}
#10 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 5 lines · Paul E. McKenney

How about the something like the following, where ->read_side_check() gets rcu_read_lock_held(), rcu_read_lock_bh_held(), or rcu_read_lock_sched_held(), as appropriate?

Quoted 7 lines

#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

	rcu_sync_is_idle_check(rss);

Agreed!

but can't we do this in a separate patch? (I will be happy to do this trivial exercise ;)

This change is trivial, but perhaps it would be better to keep the initial patch as simple as possible. And discuss the potential "cosmetic" issues (like naming) separately. Say, rcu_lockdep_assert. We can't use it directly, we need the new helper.

Oleg.

Reviewed-by
#11 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 12 lines · Oleg Nesterov
Quoted 11 lines · Paul E. McKenney

How about the something like the following, where ->read_side_check() gets rcu_read_lock_held(), rcu_read_lock_bh_held(), or rcu_read_lock_sched_held(), as appropriate?

#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

Quoted 6 lines
Quoted 1 line
	rcu_sync_is_idle_check(rss);

Agreed!

but can't we do this in a separate patch? (I will be happy to do this trivial exercise ;)

I am good with that.

Quoted 4 lines

This change is trivial, but perhaps it would be better to keep the initial patch as simple as possible. And discuss the potential "cosmetic" issues (like naming) separately. Say, rcu_lockdep_assert. We can't use it directly, we need the new helper.

OK, with s/xxx_/rcu_sync_/ in the comit log:

But someone else needs to lock all the candidate names in a room overnight and tend to the resulting wounds. ;-)

Thanx, Paul

author maintainer
#12 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Paul E. McKenney
Quoted 8 lines · Paul E. McKenney

On Wed, Oct 02, 2013 at 04:56:57PM +0200, Peter Zijlstra wrote:
#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

	rcu_sync_is_idle_check(rss);

The below actually seems to build, although I didn't test the config permutations, only defconfig.

---

include/linux/rcusync.h +7 -0
@@ -14,6 +14,7 @@ struct rcu_sync_struct {
 
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+	int  (*held)(void);
 };
 
 #define ___RCU_SYNC_INIT(name)						\
@@ -26,18 +27,21 @@ struct rcu_sync_struct {
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_sched,					\
 	.call = call_rcu_sched,						\
+	.held = rcu_read_lock_sched_held,				\
 }
 
 #define __RCU_BH_SYNC_INIT(name) {					\
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_rcu_bh,					\
 	.call = call_rcu_bh,						\
+	.held = rcu_read_lock_bh_held,					\
 }
 
 #define __RCU_SYNC_INIT(name) {						\
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_rcu,					\
 	.call = call_rcu,						\
+	.held = rcu_read_lock_held,					\
 }
 
 #define DEFINE_RCU_SCHED_SYNC(name)					\
@@ -51,6 +55,9 @@ struct rcu_sync_struct {
 
 static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 {
+#ifdef CONFIG_PROVE_RCU
+	BUG_ON(!rss->held());
+#endif
 	return !rss->gp_state; /* GP_IDLE */
 }
 
#13 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

On Wed, Oct 02, 2013 at 04:56:57PM +0200, Peter Zijlstra wrote:
#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

Quoted 4 lines
Quoted 1 line
	rcu_sync_is_idle_check(rss);

The below actually seems to build, although I didn't test the config permutations, only defconfig.

Looks good to me!

Thanx, Paul

Quoted 44 lines

---

include/linux/rcusync.h +7 -0
@@ -14,6 +14,7 @@ struct rcu_sync_struct {
 
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+	int  (*held)(void);
 };
 
 #define ___RCU_SYNC_INIT(name)						\
@@ -26,18 +27,21 @@ struct rcu_sync_struct {
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_sched,					\
 	.call = call_rcu_sched,						\
+	.held = rcu_read_lock_sched_held,				\
 }
 
 #define __RCU_BH_SYNC_INIT(name) {					\
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_rcu_bh,					\
 	.call = call_rcu_bh,						\
+	.held = rcu_read_lock_bh_held,					\
 }
 
 #define __RCU_SYNC_INIT(name) {						\
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_rcu,					\
 	.call = call_rcu,						\
+	.held = rcu_read_lock_held,					\
 }
 
 #define DEFINE_RCU_SCHED_SYNC(name)					\
@@ -51,6 +55,9 @@ struct rcu_sync_struct {
 
 static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 {
+#ifdef CONFIG_PROVE_RCU
+	BUG_ON(!rss->held());
+#endif
 	return !rss->gp_state; /* GP_IDLE */
 }
 
#14 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

On Wed, Oct 02, 2013 at 04:56:57PM +0200, Peter Zijlstra wrote:
#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

Quoted 4 lines
Quoted 1 line
	rcu_sync_is_idle_check(rss);

The below actually seems to build, although I didn't test the config permutations, only defconfig.

Yes, but...

I think it would be better to start with the patch below, this way it would be easier a) to add the new ops (we need more anyway) and b) use CONFIG_PROVE_RCU to avoid the unused members even if this is really minor.

Oleg.

rcusync: introdude struct rcu_sync_ops

CHANGELOG

---
 include/linux/rcusync.h |   63 +++++++++++++++++++++--------------------------
 kernel/rcusync.c        |   40 ++++++++++++++---------------
 2 files changed, 47 insertions(+), 56 deletions(-)
include/linux/rcusync.h +28 -35
index 7858491..30c6037 100644
@@ -4,6 +4,11 @@
 #include <linux/wait.h>
 #include <linux/rcupdate.h>
 
+struct rcu_sync_ops {
+	void (*sync)(void);
+	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+};
+
 struct rcu_sync_struct {
 	int			gp_state;
 	int			gp_count;
@@ -12,43 +17,9 @@ struct rcu_sync_struct {
 	int			cb_state;
 	struct rcu_head		cb_head;
 
-	void (*sync)(void);
-	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+	struct rcu_sync_ops	*ops;
 };
 
-#define ___RCU_SYNC_INIT(name)						\
-	.gp_state = 0,							\
-	.gp_count = 0,							\
-	.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),		\
-	.cb_state = 0
-
-#define __RCU_SCHED_SYNC_INIT(name) {					\
-	___RCU_SYNC_INIT(name),						\
-	.sync = synchronize_sched,					\
-	.call = call_rcu_sched,						\
-}
-
-#define __RCU_BH_SYNC_INIT(name) {					\
-	___RCU_SYNC_INIT(name),						\
-	.sync = synchronize_rcu_bh,					\
-	.call = call_rcu_bh,						\
-}
-
-#define __RCU_SYNC_INIT(name) {						\
-	___RCU_SYNC_INIT(name),						\
-	.sync = synchronize_rcu,					\
-	.call = call_rcu,						\
-}
-
-#define DEFINE_RCU_SCHED_SYNC(name)					\
-	struct rcu_sync_struct name = __RCU_SCHED_SYNC_INIT(name)
-
-#define DEFINE_RCU_BH_SYNC(name)					\
-	struct rcu_sync_struct name = __RCU_BH_SYNC_INIT(name)
-
-#define DEFINE_RCU_SYNC(name)						\
-	struct rcu_sync_struct name = __RCU_SYNC_INIT(name)
-
 static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 {
 	return !rss->gp_state; /* GP_IDLE */
@@ -60,5 +31,27 @@ extern void rcu_sync_init(struct rcu_sync_struct *, enum rcu_sync_type);
 extern void rcu_sync_enter(struct rcu_sync_struct *);
 extern void rcu_sync_exit(struct rcu_sync_struct *);
 
+extern struct rcu_sync_ops rcu_sync_ops_array[];
+
+#define __RCU_SYNC_INITIALIZER(name, type) {				\
+		.gp_state = 0,						\
+		.gp_count = 0,						\
+		.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),	\
+		.cb_state = 0,						\
+		.ops = rcu_sync_ops_array + (type),			\
+	}
+
+#define	__DEFINE_RCU_SYNC(name, type)	\
+	struct rcu_sync_struct name = __RCU_SYNC_INITIALIZER(name, type)
+
+#define DEFINE_RCU_SYNC(name)		\
+	__DEFINE_RCU_SYNC(name, RCU_SYNC)
+
+#define DEFINE_RCU_SCHED_SYNC(name)	\
+	__DEFINE_RCU_SYNC(name, RCU_SCHED_SYNC)
+
+#define DEFINE_RCU_BH_SYNC(name)	\
+	__DEFINE_RCU_SYNC(name, RCU_BH_SYNC)
+
 #endif /* _LINUX_RCUSYNC_H_ */
 
kernel/rcusync.c +19 -21
index f84176a..1cefb83 100644
@@ -1,4 +1,3 @@
-
 #include <linux/rcusync.h>
 #include <linux/sched.h>
 
@@ -7,27 +6,26 @@ enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY };
 
 #define	rss_lock	gp_wait.lock
 
+struct rcu_sync_ops rcu_sync_ops_array[] = {
+	[RCU_SYNC] = {
+		.sync = synchronize_rcu,
+		.call = call_rcu,
+	},
+	[RCU_SCHED_SYNC] = {
+		.sync = synchronize_sched,
+		.call = call_rcu_sched,
+	},
+	[RCU_BH_SYNC] = {
+		.sync = synchronize_rcu_bh,
+		.call = call_rcu_bh,
+	},
+};
+
 void rcu_sync_init(struct rcu_sync_struct *rss, enum rcu_sync_type type)
 {
 	memset(rss, 0, sizeof(*rss));
 	init_waitqueue_head(&rss->gp_wait);
-
-	switch (type) {
-	case RCU_SYNC:
-		rss->sync = synchronize_rcu;
-		rss->call = call_rcu;
-		break;
-
-	case RCU_SCHED_SYNC:
-		rss->sync = synchronize_sched;
-		rss->call = call_rcu_sched;
-		break;
-
-	case RCU_BH_SYNC:
-		rss->sync = synchronize_rcu_bh;
-		rss->call = call_rcu_bh;
-		break;
-	}
+	rss->ops = rcu_sync_ops_array + type;
 }
 
 void rcu_sync_enter(struct rcu_sync_struct *rss)
@@ -44,7 +42,7 @@ void rcu_sync_enter(struct rcu_sync_struct *rss)
 	BUG_ON(need_wait && need_sync);
 
 	if (need_sync) {
-		rss->sync();
+		rss->ops->sync();
 		rss->gp_state = GP_PASSED;
 		wake_up_all(&rss->gp_wait);
 	} else if (need_wait) {
@@ -81,7 +79,7 @@ static void rcu_sync_func(struct rcu_head *rcu)
 		 * to catch a later GP.
 		 */
 		rss->cb_state = CB_PENDING;
-		rss->call(&rss->cb_head, rcu_sync_func);
+		rss->ops->call(&rss->cb_head, rcu_sync_func);
 	} else {
 		/*
 		 * We're at least a GP after rcu_sync_exit(); eveybody will now
@@ -99,7 +97,7 @@ void rcu_sync_exit(struct rcu_sync_struct *rss)
 	if (!--rss->gp_count) {
 		if (rss->cb_state == CB_IDLE) {
 			rss->cb_state = CB_PENDING;
-			rss->call(&rss->cb_head, rcu_sync_func);
+			rss->ops->call(&rss->cb_head, rcu_sync_func);
 		} else if (rss->cb_state == CB_PENDING) {
 			rss->cb_state = CB_REPLAY;
 		}
#15 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 15 lines · Oleg Nesterov
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

On Wed, Oct 02, 2013 at 04:56:57PM +0200, Peter Zijlstra wrote:
#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

Quoted 4 lines
Quoted 1 line
	rcu_sync_is_idle_check(rss);

The below actually seems to build, although I didn't test the config permutations, only defconfig.

Quoted 6 lines

Yes, but...

I think it would be better to start with the patch below, this way it would be easier a) to add the new ops (we need more anyway) and b) use CONFIG_PROVE_RCU to avoid the unused members even if this is really minor.

I am fine with this as well, but I don't see the code that checks for the required RCU read-side critical section.

Thanx, Paul

Quoted 181 lines

Oleg.

rcusync: introdude struct rcu_sync_ops

CHANGELOG

---
 include/linux/rcusync.h |   63 +++++++++++++++++++++--------------------------
 kernel/rcusync.c        |   40 ++++++++++++++---------------
 2 files changed, 47 insertions(+), 56 deletions(-)
include/linux/rcusync.h +28 -35
index 7858491..30c6037 100644
@@ -4,6 +4,11 @@
 #include <linux/wait.h>
 #include <linux/rcupdate.h>
 
+struct rcu_sync_ops {
+	void (*sync)(void);
+	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+};
+
 struct rcu_sync_struct {
 	int			gp_state;
 	int			gp_count;
@@ -12,43 +17,9 @@ struct rcu_sync_struct {
 	int			cb_state;
 	struct rcu_head		cb_head;
 
-	void (*sync)(void);
-	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+	struct rcu_sync_ops	*ops;
 };
 
-#define ___RCU_SYNC_INIT(name)						\
-	.gp_state = 0,							\
-	.gp_count = 0,							\
-	.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),		\
-	.cb_state = 0
-
-#define __RCU_SCHED_SYNC_INIT(name) {					\
-	___RCU_SYNC_INIT(name),						\
-	.sync = synchronize_sched,					\
-	.call = call_rcu_sched,						\
-}
-
-#define __RCU_BH_SYNC_INIT(name) {					\
-	___RCU_SYNC_INIT(name),						\
-	.sync = synchronize_rcu_bh,					\
-	.call = call_rcu_bh,						\
-}
-
-#define __RCU_SYNC_INIT(name) {						\
-	___RCU_SYNC_INIT(name),						\
-	.sync = synchronize_rcu,					\
-	.call = call_rcu,						\
-}
-
-#define DEFINE_RCU_SCHED_SYNC(name)					\
-	struct rcu_sync_struct name = __RCU_SCHED_SYNC_INIT(name)
-
-#define DEFINE_RCU_BH_SYNC(name)					\
-	struct rcu_sync_struct name = __RCU_BH_SYNC_INIT(name)
-
-#define DEFINE_RCU_SYNC(name)						\
-	struct rcu_sync_struct name = __RCU_SYNC_INIT(name)
-
 static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 {
 	return !rss->gp_state; /* GP_IDLE */
@@ -60,5 +31,27 @@ extern void rcu_sync_init(struct rcu_sync_struct *, enum rcu_sync_type);
 extern void rcu_sync_enter(struct rcu_sync_struct *);
 extern void rcu_sync_exit(struct rcu_sync_struct *);
 
+extern struct rcu_sync_ops rcu_sync_ops_array[];
+
+#define __RCU_SYNC_INITIALIZER(name, type) {				\
+		.gp_state = 0,						\
+		.gp_count = 0,						\
+		.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),	\
+		.cb_state = 0,						\
+		.ops = rcu_sync_ops_array + (type),			\
+	}
+
+#define	__DEFINE_RCU_SYNC(name, type)	\
+	struct rcu_sync_struct name = __RCU_SYNC_INITIALIZER(name, type)
+
+#define DEFINE_RCU_SYNC(name)		\
+	__DEFINE_RCU_SYNC(name, RCU_SYNC)
+
+#define DEFINE_RCU_SCHED_SYNC(name)	\
+	__DEFINE_RCU_SYNC(name, RCU_SCHED_SYNC)
+
+#define DEFINE_RCU_BH_SYNC(name)	\
+	__DEFINE_RCU_SYNC(name, RCU_BH_SYNC)
+
 #endif /* _LINUX_RCUSYNC_H_ */
 
kernel/rcusync.c +19 -21
index f84176a..1cefb83 100644
@@ -1,4 +1,3 @@
-
 #include <linux/rcusync.h>
 #include <linux/sched.h>
 
@@ -7,27 +6,26 @@ enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY };
 
 #define	rss_lock	gp_wait.lock
 
+struct rcu_sync_ops rcu_sync_ops_array[] = {
+	[RCU_SYNC] = {
+		.sync = synchronize_rcu,
+		.call = call_rcu,
+	},
+	[RCU_SCHED_SYNC] = {
+		.sync = synchronize_sched,
+		.call = call_rcu_sched,
+	},
+	[RCU_BH_SYNC] = {
+		.sync = synchronize_rcu_bh,
+		.call = call_rcu_bh,
+	},
+};
+
 void rcu_sync_init(struct rcu_sync_struct *rss, enum rcu_sync_type type)
 {
 	memset(rss, 0, sizeof(*rss));
 	init_waitqueue_head(&rss->gp_wait);
-
-	switch (type) {
-	case RCU_SYNC:
-		rss->sync = synchronize_rcu;
-		rss->call = call_rcu;
-		break;
-
-	case RCU_SCHED_SYNC:
-		rss->sync = synchronize_sched;
-		rss->call = call_rcu_sched;
-		break;
-
-	case RCU_BH_SYNC:
-		rss->sync = synchronize_rcu_bh;
-		rss->call = call_rcu_bh;
-		break;
-	}
+	rss->ops = rcu_sync_ops_array + type;
 }
 
 void rcu_sync_enter(struct rcu_sync_struct *rss)
@@ -44,7 +42,7 @@ void rcu_sync_enter(struct rcu_sync_struct *rss)
 	BUG_ON(need_wait && need_sync);
 
 	if (need_sync) {
-		rss->sync();
+		rss->ops->sync();
 		rss->gp_state = GP_PASSED;
 		wake_up_all(&rss->gp_wait);
 	} else if (need_wait) {
@@ -81,7 +79,7 @@ static void rcu_sync_func(struct rcu_head *rcu)
 		 * to catch a later GP.
 		 */
 		rss->cb_state = CB_PENDING;
-		rss->call(&rss->cb_head, rcu_sync_func);
+		rss->ops->call(&rss->cb_head, rcu_sync_func);
 	} else {
 		/*
 		 * We're at least a GP after rcu_sync_exit(); eveybody will now
@@ -99,7 +97,7 @@ void rcu_sync_exit(struct rcu_sync_struct *rss)
 	if (!--rss->gp_count) {
 		if (rss->cb_state == CB_IDLE) {
 			rss->cb_state = CB_PENDING;
-			rss->call(&rss->cb_head, rcu_sync_func);
+			rss->ops->call(&rss->cb_head, rcu_sync_func);
 		} else if (rss->cb_state == CB_PENDING) {
 			rss->cb_state = CB_REPLAY;
 		}
#16 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 5 lines · Paul E. McKenney
Quoted 3 lines · Oleg Nesterov

Yes, but...

Quoted 7 lines
Quoted 4 lines

I think it would be better to start with the patch below, this way it would be easier a) to add the new ops (we need more anyway) and b) use CONFIG_PROVE_RCU to avoid the unused members even if this is really minor.

I am fine with this as well, but I don't see the code that checks for the required RCU read-side critical section.

Because I believe this needs another patch ;) see below, didn't test it yet.

(and yes, until I saw the patch from Peter I didn't realize we already
have all necessary helpers as functions, somehow I tought that they are macros).

Oleg.

rcusync: add the CONFIG_PROVE_RCU checks

CHANGELOG
---

include/linux/rcusync.h +6 -0
index 30c6037..aa7e9e0 100644
@@ -7,6 +7,9 @@
 struct rcu_sync_ops {
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+#ifdef CONFIG_PROVE_RCU
+	bool (*held)(void);
+#endif
 };
 
 struct rcu_sync_struct {
@@ -22,6 +25,9 @@ struct rcu_sync_struct {
 
 static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 {
+#ifdef CONFIG_PROVE_RCU
+	WARN_ON(!rss->ops->held());
+#endif
 	return !rss->gp_state; /* GP_IDLE */
 }
 
kernel/rcusync.c +9 -0
index 1cefb83..21f191f 100644
@@ -6,18 +6,27 @@ enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY };
 
 #define	rss_lock	gp_wait.lock
 
+#ifdef CONFIG_PROVE_RCU
+#define __INIT_HELD(func)	.held = func,
+#else
+#define	__INIT_HELD(func)
+#endif
+
 struct rcu_sync_ops rcu_sync_ops_array[] = {
 	[RCU_SYNC] = {
 		.sync = synchronize_rcu,
 		.call = call_rcu,
+		__INIT_HELD(rcu_read_lock_held)
 	},
 	[RCU_SCHED_SYNC] = {
 		.sync = synchronize_sched,
 		.call = call_rcu_sched,
+		__INIT_HELD(rcu_read_lock_sched_held)
 	},
 	[RCU_BH_SYNC] = {
 		.sync = synchronize_rcu_bh,
 		.call = call_rcu_bh,
+		__INIT_HELD(rcu_read_lock_bh_held)
 	},
 };
 
#17 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

include/linux/rcusync.h line 11

Quoted 9 lines · Oleg Nesterov

Because I believe this needs another patch ;) see below, didn't test it yet.
...
struct rcu_sync_ops {

include/linux/rcusync.h +2 -0
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+#ifdef CONFIG_PROVE_RCU
+	bool (*held)(void);
        ^^^^

OK, it has to return "int".

Oleg.

#18 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 11 lines · Oleg Nesterov

include/linux/rcusync.h line 11

Quoted 9 lines · Oleg Nesterov

Because I believe this needs another patch ;) see below, didn't test it yet.
...
struct rcu_sync_ops {

include/linux/rcusync.h +2 -0
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+#ifdef CONFIG_PROVE_RCU
+	bool (*held)(void);
        ^^^^
Quoted 1 line

OK, it has to return "int".

I missed this, but the rest looked good. ;-)

Thanx, Paul

#19 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 15 lines · Paul E. McKenney
Quoted 11 lines · Oleg Nesterov

include/linux/rcusync.h line 11

Quoted 9 lines · Oleg Nesterov

Because I believe this needs another patch ;) see below, didn't test it yet.
...
struct rcu_sync_ops {

include/linux/rcusync.h +2 -0
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+#ifdef CONFIG_PROVE_RCU
+	bool (*held)(void);
        ^^^^
Quoted 1 line

OK, it has to return "int".

Quoted 1 line

I missed this, but the rest looked good. ;-)

OK thanks ;)

So unless Peter objects I'll write the changelogs (always nontrivial task), test, and send these 2 patches + "add ops->barr() / rcu_sync_wait_for_cb" tomorrow.

Oleg.

#20 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 4 lines · Oleg Nesterov

So unless Peter objects I'll write the changelogs (always nontrivial task), test, and send these 2 patches + "add ops->barr() / rcu_sync_wait_for_cb" tomorrow.

And, can't resist, probably another patch below (incomplete, needs the obvious change in cpu.c and the new trivial __complete_locked() helper).

Hmm. But when I look at wait_for_completion() I think it is buggy. Will try to send the fix tomorrow.

Oleg.

rcusync: introduce rcu_sync_struct->exclusive mode

CHANGELOG.
---

include/linux/rcusync.h +16 -13
index ab787c1..265875c 100644
@@ -1,8 +1,8 @@
 #ifndef _LINUX_RCUSYNC_H_
 #define _LINUX_RCUSYNC_H_
 
-#include <linux/wait.h>
 #include <linux/rcupdate.h>
+#include <linux/completion.h>
 
 struct rcu_sync_ops {
 	void (*sync)(void);
@@ -15,11 +15,12 @@ struct rcu_sync_ops {
 struct rcu_sync_struct {
 	int			gp_state;
 	int			gp_count;
-	wait_queue_head_t	gp_wait;
+	struct completion	gp_comp;
 
 	int			cb_state;
 	struct rcu_head		cb_head;
 
+	bool			exclusive;
 	struct rcu_sync_ops	*ops;
 };
 
@@ -33,31 +34,33 @@ static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 
 enum rcu_sync_type { RCU_SYNC, RCU_SCHED_SYNC, RCU_BH_SYNC };
 
-extern void rcu_sync_init(struct rcu_sync_struct *, enum rcu_sync_type);
+extern void rcu_sync_init(struct rcu_sync_struct *,
+				enum rcu_sync_type, bool excl);
 extern void rcu_sync_enter(struct rcu_sync_struct *);
 extern void rcu_sync_exit(struct rcu_sync_struct *);
 
 extern struct rcu_sync_ops rcu_sync_ops_array[];
 
-#define __RCU_SYNC_INITIALIZER(name, type) {				\
+#define __RCU_SYNC_INITIALIZER(name, type, excl) {			\
 		.gp_state = 0,						\
 		.gp_count = 0,						\
-		.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),	\
+		.gp_comp = COMPLETION_INITIALIZER(name.gp_comp),	\
 		.cb_state = 0,						\
+		.exclusive = excl,					\
 		.ops = rcu_sync_ops_array + (type),			\
 	}
 
-#define	__DEFINE_RCU_SYNC(name, type)	\
-	struct rcu_sync_struct name = __RCU_SYNC_INITIALIZER(name, type)
+#define	__DEFINE_RCU_SYNC(name, type, excl)	\
+	struct rcu_sync_struct name = __RCU_SYNC_INITIALIZER(name, type, excl)
 
-#define DEFINE_RCU_SYNC(name)		\
-	__DEFINE_RCU_SYNC(name, RCU_SYNC)
+#define DEFINE_RCU_SYNC(name, excl)		\
+	__DEFINE_RCU_SYNC(name, RCU_SYNC, excl)
 
-#define DEFINE_RCU_SCHED_SYNC(name)	\
-	__DEFINE_RCU_SYNC(name, RCU_SCHED_SYNC)
+#define DEFINE_RCU_SCHED_SYNC(name, excl)	\
+	__DEFINE_RCU_SYNC(name, RCU_SCHED_SYNC, excl)
 
-#define DEFINE_RCU_BH_SYNC(name)	\
-	__DEFINE_RCU_SYNC(name, RCU_BH_SYNC)
+#define DEFINE_RCU_BH_SYNC(name, excl)		\
+	__DEFINE_RCU_SYNC(name, RCU_BH_SYNC, excl)
 
 #endif /* _LINUX_RCUSYNC_H_ */
 
kernel/rcusync.c +13 -5
index 21cde9b..d8068df 100644
@@ -4,7 +4,7 @@
 enum { GP_IDLE = 0, GP_PENDING, GP_PASSED };
 enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY };
 
-#define	rss_lock	gp_wait.lock
+#define	rss_lock	gp_comp.wait.lock
 
 #ifdef CONFIG_PROVE_RCU
 #define __INIT_HELD(func)	.held = func,
@@ -30,11 +30,13 @@ struct rcu_sync_ops rcu_sync_ops_array[] = {
 	},
 };
 
-void rcu_sync_init(struct rcu_sync_struct *rss, enum rcu_sync_type type)
+void rcu_sync_init(struct rcu_sync_struct *rss,
+			enum rcu_sync_type type, bool excl)
 {
 	memset(rss, 0, sizeof(*rss));
-	init_waitqueue_head(&rss->gp_wait);
+	init_completion(&rss->gp_comp);
 	rss->ops = rcu_sync_ops_array + type;
+	rss->exclusive = excl;
 }
 
 void rcu_sync_enter(struct rcu_sync_struct *rss)
@@ -53,9 +55,13 @@ void rcu_sync_enter(struct rcu_sync_struct *rss)
 	if (need_sync) {
 		rss->ops->sync();
 		rss->gp_state = GP_PASSED;
-		wake_up_all(&rss->gp_wait);
+		if (!rss->exclusive)
+			wake_up_all(&rss->gp_comp.wait);
 	} else if (need_wait) {
-		wait_event(rss->gp_wait, rss->gp_state == GP_PASSED);
+		if (!rss->exclusive)
+			wait_event(rss->gp_comp.wait, rss->gp_state == GP_PASSED);
+		else
+			wait_for_completion(&rss->gp_comp);
 	} else {
 		/*
 		 * Possible when there's a pending CB from a rcu_sync_exit().
@@ -110,6 +116,8 @@ void rcu_sync_exit(struct rcu_sync_struct *rss)
 		} else if (rss->cb_state == CB_PENDING) {
 			rss->cb_state = CB_REPLAY;
 		}
+	} else if (rss->exclusive) {
+		__complete_locked(&rss->gp_comp);
 	}
 	spin_unlock_irq(&rss->rss_lock);
 }
#21 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 9 lines · Oleg Nesterov
Quoted 4 lines · Oleg Nesterov

So unless Peter objects I'll write the changelogs (always nontrivial task), test, and send these 2 patches + "add ops->barr() / rcu_sync_wait_for_cb" tomorrow.

And, can't resist, probably another patch below (incomplete, needs the obvious change in cpu.c and the new trivial __complete_locked() helper).

Quoted 8 lines

Hmm. But when I look at wait_for_completion() I think it is buggy. Will try to send the fix tomorrow.

Oleg.

rcusync: introduce rcu_sync_struct->exclusive mode

CHANGELOG.

Should the changelog really be in all caps? (Sorry, couldn't resist...)

One confusion below (mine), otherwise cannot see any obvious problems with it. Not that this should count for much, getting a bit punch-drunk reviewing this sort of code. ;-)

Thanx, Paul

Quoted 105 lines

---

include/linux/rcusync.h +16 -13
index ab787c1..265875c 100644
@@ -1,8 +1,8 @@
 #ifndef _LINUX_RCUSYNC_H_
 #define _LINUX_RCUSYNC_H_
 
-#include <linux/wait.h>
 #include <linux/rcupdate.h>
+#include <linux/completion.h>
 
 struct rcu_sync_ops {
 	void (*sync)(void);
@@ -15,11 +15,12 @@ struct rcu_sync_ops {
 struct rcu_sync_struct {
 	int			gp_state;
 	int			gp_count;
-	wait_queue_head_t	gp_wait;
+	struct completion	gp_comp;
 
 	int			cb_state;
 	struct rcu_head		cb_head;
 
+	bool			exclusive;
 	struct rcu_sync_ops	*ops;
 };
 
@@ -33,31 +34,33 @@ static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 
 enum rcu_sync_type { RCU_SYNC, RCU_SCHED_SYNC, RCU_BH_SYNC };
 
-extern void rcu_sync_init(struct rcu_sync_struct *, enum rcu_sync_type);
+extern void rcu_sync_init(struct rcu_sync_struct *,
+				enum rcu_sync_type, bool excl);
 extern void rcu_sync_enter(struct rcu_sync_struct *);
 extern void rcu_sync_exit(struct rcu_sync_struct *);
 
 extern struct rcu_sync_ops rcu_sync_ops_array[];
 
-#define __RCU_SYNC_INITIALIZER(name, type) {				\
+#define __RCU_SYNC_INITIALIZER(name, type, excl) {			\
 		.gp_state = 0,						\
 		.gp_count = 0,						\
-		.gp_wait = __WAIT_QUEUE_HEAD_INITIALIZER(name.gp_wait),	\
+		.gp_comp = COMPLETION_INITIALIZER(name.gp_comp),	\
 		.cb_state = 0,						\
+		.exclusive = excl,					\
 		.ops = rcu_sync_ops_array + (type),			\
 	}
 
-#define	__DEFINE_RCU_SYNC(name, type)	\
-	struct rcu_sync_struct name = __RCU_SYNC_INITIALIZER(name, type)
+#define	__DEFINE_RCU_SYNC(name, type, excl)	\
+	struct rcu_sync_struct name = __RCU_SYNC_INITIALIZER(name, type, excl)
 
-#define DEFINE_RCU_SYNC(name)		\
-	__DEFINE_RCU_SYNC(name, RCU_SYNC)
+#define DEFINE_RCU_SYNC(name, excl)		\
+	__DEFINE_RCU_SYNC(name, RCU_SYNC, excl)
 
-#define DEFINE_RCU_SCHED_SYNC(name)	\
-	__DEFINE_RCU_SYNC(name, RCU_SCHED_SYNC)
+#define DEFINE_RCU_SCHED_SYNC(name, excl)	\
+	__DEFINE_RCU_SYNC(name, RCU_SCHED_SYNC, excl)
 
-#define DEFINE_RCU_BH_SYNC(name)	\
-	__DEFINE_RCU_SYNC(name, RCU_BH_SYNC)
+#define DEFINE_RCU_BH_SYNC(name, excl)		\
+	__DEFINE_RCU_SYNC(name, RCU_BH_SYNC, excl)
 
 #endif /* _LINUX_RCUSYNC_H_ */
 
kernel/rcusync.c +5 -3
index 21cde9b..d8068df 100644
@@ -4,7 +4,7 @@
 enum { GP_IDLE = 0, GP_PENDING, GP_PASSED };
 enum { CB_IDLE = 0, CB_PENDING, CB_REPLAY };
 
-#define	rss_lock	gp_wait.lock
+#define	rss_lock	gp_comp.wait.lock
 
 #ifdef CONFIG_PROVE_RCU
 #define __INIT_HELD(func)	.held = func,
@@ -30,11 +30,13 @@ struct rcu_sync_ops rcu_sync_ops_array[] = {
 	},
 };
 
-void rcu_sync_init(struct rcu_sync_struct *rss, enum rcu_sync_type type)
+void rcu_sync_init(struct rcu_sync_struct *rss,
+			enum rcu_sync_type type, bool excl)
 {
 	memset(rss, 0, sizeof(*rss));
-	init_waitqueue_head(&rss->gp_wait);
+	init_completion(&rss->gp_comp);
 	rss->ops = rcu_sync_ops_array + type;
+	rss->exclusive = excl;
 }
 
 void rcu_sync_enter(struct rcu_sync_struct *rss)

kernel/rcusync.c void rcu_sync_enter(struct rcu_sync_struct *rss) · line 59

Quoted 6 lines
kernel/rcusync.c +2 -1
@@ -53,9 +55,13 @@ void rcu_sync_enter(struct rcu_sync_struct *rss)
 	if (need_sync) {
 		rss->ops->sync();
 		rss->gp_state = GP_PASSED;
-		wake_up_all(&rss->gp_wait);
+		if (!rss->exclusive)
+			wake_up_all(&rss->gp_comp.wait);

Not sure about the wake_up_all() on a completion, but if we are exclusive, don't we need to complete() the completion here?

Oh, I guess gp_comp.wait is exactly a wait_queue_head_t, so I guess you can do wake_up_all() on it... And the thing we would wake up is ourselves, and we are already awake.

Never mind!!!

Quoted 25 lines
kernel/rcusync.c +6 -2
 	} else if (need_wait) {
-		wait_event(rss->gp_wait, rss->gp_state == GP_PASSED);
+		if (!rss->exclusive)
+			wait_event(rss->gp_comp.wait, rss->gp_state == GP_PASSED);
+		else
+			wait_for_completion(&rss->gp_comp);
 	} else {
 		/*
 		 * Possible when there's a pending CB from a rcu_sync_exit().
@@ -110,6 +116,8 @@ void rcu_sync_exit(struct rcu_sync_struct *rss)
 		} else if (rss->cb_state == CB_PENDING) {
 			rss->cb_state = CB_REPLAY;
 		}
+	} else if (rss->exclusive) {
+		__complete_locked(&rss->gp_comp);
 	}
 	spin_unlock_irq(&rss->rss_lock);
 }
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#22 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 8 lines · Paul E. McKenney
Quoted 4 lines · Oleg Nesterov

rcusync: introduce rcu_sync_struct->exclusive mode

CHANGELOG.

Should the changelog really be in all caps? (Sorry, couldn't resist...)

Apparently you do not realize it is going to be an EXCELLENT changelog!

Quoted 9 lines

kernel/rcusync.c void rcu_sync_enter(struct rcu_sync_struct *rss) · line 59

Quoted 7 lines
kernel/rcusync.c +2 -1
@@ -53,9 +55,13 @@ void rcu_sync_enter(struct rcu_sync_struct *rss)
 	if (need_sync) {
 		rss->ops->sync();
 		rss->gp_state = GP_PASSED;
-		wake_up_all(&rss->gp_wait);
+		if (!rss->exclusive)
+			wake_up_all(&rss->gp_comp.wait);

Not sure about the wake_up_all() on a completion,

Yes, we reuse completion->wait in the !exclusive case. Like we reuse its spinlock as rss_lock.

We can add a completion/complete union, but this will complicate the code a bit and imo doesn't make sense.

Quoted 2 lines

but if we are exclusive,
don't we need to complete() the completion here?

No, if we are exclusive we should delay the "wake up the next writer" till rcu_sync_exit().

Quoted 2 lines

Oh, I guess gp_comp.wait is exactly a wait_queue_head_t, so I guess you can do wake_up_all() on it...

Yes, and we never "mix" completion/wait_queue_head_t operations/members. IOW, we always use ->gp_comp if "exclusive", and only ->gp_comp.wait is used otherwise.

Quoted 1 line

Never mind!!!

Agreed ;)

Oleg.

#23 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 11 lines · Oleg Nesterov
Quoted 8 lines · Paul E. McKenney
Quoted 4 lines · Oleg Nesterov

rcusync: introduce rcu_sync_struct->exclusive mode

CHANGELOG.

Should the changelog really be in all caps? (Sorry, couldn't resist...)

Apparently you do not realize it is going to be an EXCELLENT changelog!

;-)

Quoted 10 lines
Quoted 9 lines

kernel/rcusync.c void rcu_sync_enter(struct rcu_sync_struct *rss) · line 59

Quoted 7 lines
kernel/rcusync.c +2 -1
@@ -53,9 +55,13 @@ void rcu_sync_enter(struct rcu_sync_struct *rss)
 	if (need_sync) {
 		rss->ops->sync();
 		rss->gp_state = GP_PASSED;
-		wake_up_all(&rss->gp_wait);
+		if (!rss->exclusive)
+			wake_up_all(&rss->gp_comp.wait);

Not sure about the wake_up_all() on a completion,

Quoted 5 lines

Yes, we reuse completion->wait in the !exclusive case. Like we reuse its spinlock as rss_lock.

We can add a completion/complete union, but this will complicate the code a bit and imo doesn't make sense.

Fair enough!

Thanx, Paul

Quoted 19 lines
Quoted 2 lines

but if we are exclusive,
don't we need to complete() the completion here?

No, if we are exclusive we should delay the "wake up the next writer" till rcu_sync_exit().

Quoted 2 lines

Oh, I guess gp_comp.wait is exactly a wait_queue_head_t, so I guess you can do wake_up_all() on it...

Yes, and we never "mix" completion/wait_queue_head_t operations/members. IOW, we always use ->gp_comp if "exclusive", and only ->gp_comp.wait is used otherwise.

Quoted 1 line

Never mind!!!

Agreed ;)

Oleg.

author maintainer
#24 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Oleg Nesterov
Quoted 9 lines · Oleg Nesterov
Quoted 4 lines · Oleg Nesterov

So unless Peter objects I'll write the changelogs (always nontrivial task), test, and send these 2 patches + "add ops->barr() / rcu_sync_wait_for_cb" tomorrow.

And, can't resist, probably another patch below (incomplete, needs the obvious change in cpu.c and the new trivial __complete_locked() helper).

Quoted 6 lines

Hmm. But when I look at wait_for_completion() I think it is buggy. Will try to send the fix tomorrow.

Oleg.

rcusync: introduce rcu_sync_struct->exclusive mode

What's exclusive to mean? One writer at a time? Why don't use use the waitqueue in exclusive mode and use a single wake_up instead of wake_up_all()?

#25 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Oleg Nesterov
Quoted 1 line · Oleg Nesterov
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

So unless Peter objects I'll write the changelogs (always nontrivial task), test, and send these 2 patches + "add ops->barr() / rcu_sync_wait_for_cb" tomorrow.

And, can't resist, probably another patch below (incomplete, needs the obvious change in cpu.c and the new trivial __complete_locked() helper).

Hmm. But when I look at wait_for_completion() I think it is buggy.

No, sorry for noise, it is fine.

Quoted 3 lines
Quoted 1 line

rcusync: introduce rcu_sync_struct->exclusive mode

What's exclusive to mean? One writer at a time?

Yes,

Quoted 3 lines

Why don't use use the
waitqueue in exclusive mode and use a single wake_up instead of wake_up_all()?

But this won't work, wait_event_exclusive(wq, CONDITION)-like code obviously can't guarantee that only a single thread sees CONDITION, even if "unlock" does __wake_up(nr_exclusive => 1).

Of course we can fix this, but wait_for_completion/complete already does the necessary work: x->done acts as a resource counter which is always checked/incremented/decremented under the same lock.

Oleg.

author maintainer
#26 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 3 lines · Oleg Nesterov

Of course we can fix this, but wait_for_completion/complete already does the necessary work: x->done acts as a resource counter which is always checked/incremented/decremented under the same lock.

Urgh, so now you're not using a semaphore as completion but using a completion as semaphore.

#27 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Oleg Nesterov

Of course we can fix this, but wait_for_completion/complete already does the necessary work: x->done acts as a resource counter which is always checked/incremented/decremented under the same lock.

Urgh, so now you're not using a semaphore as completion but using a completion as semaphore.

Yes. And we can use it as semaphore only because we already have gp_count.

Of course, we could add ->gp_mutex and change enter/exit to lock/unlock it if exclusive. But I do not really like it even if this can makes the code a bit (only a little bit) simpler. This will increase sizeof() and for no reason, I think, because to some degree this will also duplicate the synchronization logic. And while this doesn't really matter, this will penalize the contending writers a bit: it is not easy (but of course possible) to take this mutex after rss->gp_count++, and we do want to increment this counter "asap".

Oleg.

author maintainer
#28 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Oleg Nesterov
Quoted 3 lines · Oleg Nesterov
Quoted 1 line

What's exclusive to mean? One writer at a time?

Yes,

I'm not entirely sure what the advantage is of having that logic in this primitive. Shouldn't that be something the user of this rcu_sync stuff does (or not) depending on its needs.

#29 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Oleg Nesterov
Quoted 1 line

What's exclusive to mean? One writer at a time?

Yes,

I'm not entirely sure what the advantage is of having that logic in this primitive. Shouldn't that be something the user of this rcu_sync stuff does (or not) depending on its needs.

Yes, the user can do the locking itself. But I think this option can help. If nothing else it can help to avoid another mutex/whatever and unnecessary wakeup/scheule's, even if this is minor.

And. rcu_sync_enter() should be "bool", it should return "need_sync". IOW, rcu_sync_enter() == T means that this thread has done the FAST -> SLOW transition, this is particularly useful in "exclusive" mode.

Consider percpu_down_write(). It takes rw_sem for writing (and this blocks the readers) before clear_fast_ctr(), but we only need to do this this after sync_sched(), so it could do

	if (rcu_sync_enter(&brw->rcu_sync))
		atomic_add(clear_fast_ctr(brw), &brw->slow_read_ctr);
	else
		; /* the above was already done */

	/* exclude readers */
	down_write(&brw->rw_sem);

and now ->rw_sem is only needed to serialize readers/writer.

Sure, this all is minor (and we will probably copy the "pending writer" logic from cpu_hotplug_begin/get_online_cpus).

But we can get this feature almost for free, so I think it makes sense.

Oleg.

author maintainer
#30 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 31 lines · Oleg Nesterov
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Oleg Nesterov
Quoted 1 line

What's exclusive to mean? One writer at a time?

Yes,

I'm not entirely sure what the advantage is of having that logic in this primitive. Shouldn't that be something the user of this rcu_sync stuff does (or not) depending on its needs.

Yes, the user can do the locking itself. But I think this option can help. If nothing else it can help to avoid another mutex/whatever and unnecessary wakeup/scheule's, even if this is minor.

And. rcu_sync_enter() should be "bool", it should return "need_sync". IOW, rcu_sync_enter() == T means that this thread has done the FAST -> SLOW transition, this is particularly useful in "exclusive" mode.

Consider percpu_down_write(). It takes rw_sem for writing (and this blocks the readers) before clear_fast_ctr(), but we only need to do this this after sync_sched(), so it could do

	if (rcu_sync_enter(&brw->rcu_sync))
		atomic_add(clear_fast_ctr(brw), &brw->slow_read_ctr);
	else
		; /* the above was already done */

	/* exclude readers */
	down_write(&brw->rw_sem);
Quoted 6 lines

and now ->rw_sem is only needed to serialize readers/writer.

Sure, this all is minor (and we will probably copy the "pending writer" logic from cpu_hotplug_begin/get_online_cpus).

But we can get this feature almost for free, so I think it makes sense.

Well, the whole reason I asked is because adding that completion in there didn't at all smell like free to me; not to mention that I hadn't at all realized you're using it as a semaphore.

Also; what would be the use once you convert the per-cpu rwsem over to the scheme I used with hotplug?

I'm really starting to think we shouldn't do this in rcu_sync at all.

#31 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 30 lines · Peter Zijlstra
Quoted 28 lines · Oleg Nesterov
Quoted 4 lines · Peter Zijlstra

I'm not entirely sure what the advantage is of having that logic in this primitive. Shouldn't that be something the user of this rcu_sync stuff does (or not) depending on its needs.

Yes, the user can do the locking itself. But I think this option can help. If nothing else it can help to avoid another mutex/whatever and unnecessary wakeup/scheule's, even if this is minor.

And. rcu_sync_enter() should be "bool", it should return "need_sync". IOW, rcu_sync_enter() == T means that this thread has done the FAST -> SLOW transition, this is particularly useful in "exclusive" mode.

Consider percpu_down_write(). It takes rw_sem for writing (and this blocks the readers) before clear_fast_ctr(), but we only need to do this this after sync_sched(), so it could do

	if (rcu_sync_enter(&brw->rcu_sync))
		atomic_add(clear_fast_ctr(brw), &brw->slow_read_ctr);
	else
		; /* the above was already done */

	/* exclude readers */
	down_write(&brw->rw_sem);

and now ->rw_sem is only needed to serialize readers/writer.

Quoted 7 lines
Quoted 4 lines

Sure, this all is minor (and we will probably copy the "pending writer" logic from cpu_hotplug_begin/get_online_cpus).

But we can get this feature almost for free, so I think it makes sense.

Well, the whole reason I asked is because adding that completion in there didn't at all smell like free to me;

Why? this only adds sizeof(long).

If you dislike the idea to add the new __complete_locked() one-liner, this is not strictly necessary, just a bit simpler/understandable.

Quoted 2 lines

not to mention that I hadn't
at all realized you're using it as a semaphore.

And the logic is trivial.

Quoted 2 lines

Also; what would be the use once you convert the per-cpu rwsem over to the scheme I used with hotplug?

It will still use the exclusive more to block other writers? This avoids another mutex simplifies the code.

Quoted 1 line

I'm really starting to think we shouldn't do this in rcu_sync at all.

I do not really understand why you insist that rcu_sync() should not try to help to the users which need the exclusive mode.

rcu_sync_enter/exit have to do some work to serialize with each other anyway, we already have ->nr_writers, so why we can't add 3 simple "if exclusive" checks?

Oleg.

author maintainer
#32 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 4 lines · Oleg Nesterov
Quoted 1 line

I'm really starting to think we shouldn't do this in rcu_sync at all.

I do not really understand why you insist that rcu_sync() should not try to help to the users which need the exclusive mode.

But what other users; you yourself said that percpu-rwsem isn't one for long if ever.

#33 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Oleg Nesterov
Quoted 1 line

I'm really starting to think we shouldn't do this in rcu_sync at all.

I do not really understand why you insist that rcu_sync() should not try to help to the users which need the exclusive mode.

But what other users; you yourself said that percpu-rwsem isn't one for long if ever.

Not sure I understand...

percpu-rwsem will use exclusive mode, with or without the possible improvements we can copy from cpuhp. sb_writers can probably use it too (along with other helpers from percpu-rwsem we do not have yet).

If you mean that the "raw" rcu_sync infrastructure will never have more users, will I am not sure, it looks "natural". But I can not know for sure.

OK, I am not going to argue too much, let me send other patches you seem to agree with. I am also going to send this one for the record/review (without __complete_locked), please ack-or-nack it explicitly.

I am not trying saying this feature is "must have", of course it is not. The only problem, I am a bit puzzled why you dislike it that much.

Oleg.

author maintainer
#34 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 5 lines · Oleg Nesterov

Not sure I understand...

percpu-rwsem will use exclusive mode, with or without the possible improvements we can copy from cpuhp. sb_writers can probably use it too (along with other helpers from percpu-rwsem we do not have yet).

Oh; I thought to have understood it would go away when you switched to the reader scheme from the hotplug bits.

But thinking a little more; yes you'd still need something to serialize writers.

Quoted 3 lines

If you mean that the "raw" rcu_sync infrastructure will never have more users, will I am not sure, it looks "natural". But I can not know for sure.

Right, so this would be something I'd forgo if there were no immediate users. We could always reconsider if there was one; but apparently percpu-rwsem is one..

Quoted 3 lines

I am not trying saying this feature is "must have", of course it is not. The only problem, I am a bit puzzled why you dislike it that much.

The reason I dislike it is because I feel we're now mixing two objects into one; one object doing mutual exclusion and one object being terribly smart with sync_rcu.

#35 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Oleg Nesterov

I am not trying saying this feature is "must have", of course it is not. The only problem, I am a bit puzzled why you dislike it that much.

The reason I dislike it is because I feel we're now mixing two objects into one; one object doing mutual exclusion and one object being terribly smart with sync_rcu.

OK, I see your point.

But rcu_sync_struct has to serialize the writers anyway. The only question is how many other writers the thread doing ->sync() should wakeup and when.

And otoh. Currently nobody needs the non-exclusive mode (cpu-hotplug doesn't care because it is always exclusive itself). And in fact you initially argued with wake_up_all ;) "exclusive" is more natural, it is like rw_semaphore.

However, yes-yes-yes, I do think that we need the non-exclusive mode too, at least for percpu_down_write_nonexclusive() which I think we need as well.

Oleg.

author maintainer
#36 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 3 lines · Oleg Nesterov

However, yes-yes-yes, I do think that we need the non-exclusive mode too, at least for percpu_down_write_nonexclusive() which I think we need as well.

I just need to disagree with the write_nonexclusive() name; the construct I quite understand and could even agree with.

How about something like:

State excluding 'writers', but not itself:
percpu_read_lock()
percpu_read_unlock()

State excluding readers, but not itself:

	percpu_non_read_lock();
	percpu_non_read_unlock();

Full exclusive state:

	percpu_write_lock();
	percpu_write_unlock();

At which point I start to have doubts about the percpu prefix.. ;-)

#37 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Oleg Nesterov

However, yes-yes-yes, I do think that we need the non-exclusive mode too, at least for percpu_down_write_nonexclusive() which I think we need as well.

I just need to disagree with the write_nonexclusive() name; the construct I quite understand and could even agree with.

How about something like:

I am fine either way ;)

Quoted 1 line

At which point I start to have doubts about the percpu prefix.. ;-)

Yes... and "percpu" just refers to implementation details, this is pointless.

Oleg.

author maintainer
#38 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Oleg Nesterov
Quoted 16 lines · Oleg Nesterov
Quoted 15 lines · Paul E. McKenney
Quoted 12 lines · Oleg Nesterov

include/linux/rcusync.h line 11

Quoted 9 lines · Oleg Nesterov

Because I believe this needs another patch ;) see below, didn't test it yet.
...
struct rcu_sync_ops {

include/linux/rcusync.h +2 -0
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+#ifdef CONFIG_PROVE_RCU
+	bool (*held)(void);
        ^^^^

OK, it has to return "int".

Quoted 7 lines
Quoted 1 line

I missed this, but the rest looked good. ;-)

OK thanks ;)

So unless Peter objects I'll write the changelogs (always nontrivial task), test, and send these 2 patches + "add ops->barr() / rcu_sync_wait_for_cb" tomorrow.

No I'm fine with that.. its sad that the ops thing requires a double deref and an extra cachemiss but I don't suppose that's a real issue since we're calling synchronize_*() things which take for ever anyway :-)

#39 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

In reply to Peter Zijlstra

Il 03/10/2013 20:40, Peter Zijlstra ha scritto:

Quoted 48 lines · Paul E. McKenney
Quoted 8 lines · Peter Zijlstra

#ifdef CONFIG_PROVE_RCU
#define rcu_sync_is_idle_check(rss) BUG_ON(!rss->read_side_check()) #else
#define rcu_sync_is_idle_check(rss) do { } while (0) #endif

	rcu_sync_is_idle_check(rss);

The below actually seems to build, although I didn't test the config permutations, only defconfig.

---

include/linux/rcusync.h +4 -0
@@ -14,6 +14,7 @@ struct rcu_sync_struct {
 
 	void (*sync)(void);
 	void (*call)(struct rcu_head *, void (*)(struct rcu_head *));
+	int  (*held)(void);
 };
 
 #define ___RCU_SYNC_INIT(name)						\
@@ -26,18 +27,21 @@ struct rcu_sync_struct {
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_sched,					\
 	.call = call_rcu_sched,						\
+	.held = rcu_read_lock_sched_held,				\
 }
 
 #define __RCU_BH_SYNC_INIT(name) {					\
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_rcu_bh,					\
 	.call = call_rcu_bh,						\
+	.held = rcu_read_lock_bh_held,					\
 }
 
 #define __RCU_SYNC_INIT(name) {						\
 	___RCU_SYNC_INIT(name),						\
 	.sync = synchronize_rcu,					\
 	.call = call_rcu,						\
+	.held = rcu_read_lock_held,					\
 }
 
 #define DEFINE_RCU_SCHED_SYNC(name)					\

include/linux/rcusync.h struct rcu_sync_struct { · line 63

Quoted 9 lines
include/linux/rcusync.h +3 -0
@@ -51,6 +55,9 @@ struct rcu_sync_struct {
 
 static inline bool rcu_sync_is_idle(struct rcu_sync_struct *rss)
 {
+#ifdef CONFIG_PROVE_RCU
+	BUG_ON(!rss->held());
+#endif
 	return !rss->gp_state; /* GP_IDLE */
 }
 

Do you need to do the same in rcu_sync_init?

Paolo

author maintainer
#40 lore raw

Re: [PATCH 2/3] rcu: Create rcu_sync infrastructure

Quoted 1 line · Paolo Bonzini

Do you need to do the same in rcu_sync_init?

Yes.. I actually did but only send the diff for the header file :/

author maintainer
#41 lore raw

PATCH 3/3 hotplug: Optimize cpu_hotplug_{begin,done}() using rcu_sync

In reply to Peter Zijlstra

Use the fancy new rcu_sync bits from Oleg to optimize the fancy new hotplug lock implementation.

---
 include/linux/cpu.h |    7 ++++---
 kernel/cpu.c        |   52 +++++++++++++++++++++++-----------------------------
 2 files changed, 27 insertions(+), 32 deletions(-)
include/linux/cpu.h +4 -3
@@ -18,6 +18,7 @@
 #include <linux/cpumask.h>
 #include <linux/percpu.h>
 #include <linux/sched.h>
+#include <linux/rcusync.h>
 
 struct device;
 
@@ -180,7 +181,7 @@ extern void cpu_hotplug_init_task(struct
 extern void cpu_hotplug_begin(void);
 extern void cpu_hotplug_done(void);
 
-extern int __cpuhp_state;
+extern struct rcu_sync_struct __cpuhp_rss;
 DECLARE_PER_CPU(unsigned int, __cpuhp_refcount);
 
 extern void __get_online_cpus(void);
@@ -204,7 +205,7 @@ static inline void get_online_cpus(void)
 	 * writer will see anything we did within this RCU-sched read-side
 	 * critical section.
 	 */
-	if (likely(!__cpuhp_state))
+	if (likely(rcu_sync_is_idle(&__cpuhp_rss)))
 		__this_cpu_inc(__cpuhp_refcount);
 	else
 		__get_online_cpus(); /* Unconditional memory barrier. */
@@ -231,7 +232,7 @@ static inline void put_online_cpus(void)
 	/*
 	 * Same as in get_online_cpus().
 	 */
-	if (likely(!__cpuhp_state))
+	if (likely(rcu_sync_is_idle(&__cpuhp_rss)))
 		__this_cpu_dec(__cpuhp_refcount);
 	else
 		__put_online_cpus(); /* Unconditional memory barrier. */
kernel/cpu.c +23 -29

124 lines not on this page · raw message

#42 lore raw

Re: [PATCH 3/3] hotplug: Optimize cpu_hotplug_{begin,done}() using rcu_sync

Quoted 4 lines · Peter Zijlstra

Use the fancy new rcu_sync bits from Oleg to optimize the fancy new hotplug lock implementation.

Looks good, just a couple of comments on comments below...

Thanx, Paul

Quoted 45 lines
---
 include/linux/cpu.h |    7 ++++---
 kernel/cpu.c        |   52 +++++++++++++++++++++++-----------------------------
 2 files changed, 27 insertions(+), 32 deletions(-)
include/linux/cpu.h +4 -3
@@ -18,6 +18,7 @@
 #include <linux/cpumask.h>
 #include <linux/percpu.h>
 #include <linux/sched.h>
+#include <linux/rcusync.h>
 
 struct device;
 
@@ -180,7 +181,7 @@ extern void cpu_hotplug_init_task(struct
 extern void cpu_hotplug_begin(void);
 extern void cpu_hotplug_done(void);
 
-extern int __cpuhp_state;
+extern struct rcu_sync_struct __cpuhp_rss;
 DECLARE_PER_CPU(unsigned int, __cpuhp_refcount);
 
 extern void __get_online_cpus(void);
@@ -204,7 +205,7 @@ static inline void get_online_cpus(void)
 	 * writer will see anything we did within this RCU-sched read-side
 	 * critical section.
 	 */
-	if (likely(!__cpuhp_state))
+	if (likely(rcu_sync_is_idle(&__cpuhp_rss)))
 		__this_cpu_inc(__cpuhp_refcount);
 	else
 		__get_online_cpus(); /* Unconditional memory barrier. */
@@ -231,7 +232,7 @@ static inline void put_online_cpus(void)
 	/*
 	 * Same as in get_online_cpus().
 	 */
-	if (likely(!__cpuhp_state))
+	if (likely(rcu_sync_is_idle(&__cpuhp_rss)))
 		__this_cpu_dec(__cpuhp_refcount);
 	else
 		__put_online_cpus(); /* Unconditional memory barrier. */

kernel/cpu.c static int cpu_hotplug_disabled; · line 52

Quoted 5 lines
kernel/cpu.c +1 -1
@@ -49,14 +49,15 @@ static int cpu_hotplug_disabled;
 
 #ifdef CONFIG_HOTPLUG_CPU
 
-enum { readers_fast = 0, readers_slow, readers_block };
+enum { readers_slow, readers_block };

It took me a bit to realize that readers_fast is obsoleted by the rcu_sync_is_idle() above. ;-)

kernel/cpu.c static int cpu_hotplug_disabled; · line 54

Quoted 4 lines
kernel/cpu.c +1 -2
-int __cpuhp_state;
-EXPORT_SYMBOL_GPL(__cpuhp_state);
+DEFINE_RCU_SCHED_SYNC(__cpuhp_rss);

OK, matched RCU flavors. ;-)

Quoted 66 lines
kernel/cpu.c +12 -11
+EXPORT_SYMBOL_GPL(__cpuhp_rss);
 
 DEFINE_PER_CPU(unsigned int, __cpuhp_refcount);
 EXPORT_PER_CPU_SYMBOL_GPL(__cpuhp_refcount);
 
+static int cpuhp_state = readers_slow;
 static atomic_t cpuhp_waitcount;
 static DECLARE_WAIT_QUEUE_HEAD(cpuhp_readers);
 static DECLARE_WAIT_QUEUE_HEAD(cpuhp_writer);
@@ -68,7 +69,6 @@ void cpu_hotplug_init_task(struct task_s
 
 void __get_online_cpus(void)
 {
-again:
 	__this_cpu_inc(__cpuhp_refcount);
 
 	/*
@@ -77,7 +77,7 @@ void __get_online_cpus(void)
 	 * increment-on-one-CPU-and-decrement-on-another problem.
 	 *
 	 * And yes, if the reader misses the writer's assignment of
-	 * readers_block to __cpuhp_state, then the writer is
+	 * readers_block to cpuhp_state, then the writer is
 	 * guaranteed to see the reader's increment.  Conversely, any
 	 * readers that increment their __cpuhp_refcount after the
 	 * writer looks are guaranteed to see the readers_block value,
@@ -88,7 +88,7 @@ void __get_online_cpus(void)
 
 	smp_mb(); /* A matches D */
 
-	if (likely(__cpuhp_state != readers_block))
+	if (likely(cpuhp_state != readers_block))
 		return;
 
 	/*
@@ -108,19 +108,19 @@ void __get_online_cpus(void)
 	 * and reschedule on the preempt_enable() in get_online_cpus().
 	 */
 	preempt_enable_no_resched();
-	__wait_event(cpuhp_readers, __cpuhp_state != readers_block);
+	__wait_event(cpuhp_readers, cpuhp_state != readers_block);
 	preempt_disable();
 
+	__this_cpu_inc(__cpuhp_refcount);
+
 	/*
-	 * Given we've still got preempt_disabled and new cpu_hotplug_begin()
-	 * must do a synchronize_sched() we're guaranteed a successfull
-	 * acquisition this time -- even if we wake the current
-	 * cpu_hotplug_end() now.
+	 * cpu_hotplug_done() waits until all pending readers are gone;
+	 * this means that a new cpu_hotplug_begin() must observe our
+	 * refcount increment and wait for it to go away.
 	 */
-	if (atomic_dec_and_test(&cpuhp_waitcount))
-		wake_up(&cpuhp_writer);
 
-	goto again;
+	if (atomic_dec_and_test(&cpuhp_waitcount)) /* A */
+		wake_up(&cpuhp_writer);
 }
 EXPORT_SYMBOL_GPL(__get_online_cpus);
 
@@ -183,21 +183,18 @@ void cpu_hotplug_begin(void)
 	current->cpuhp_ref++;
 

kernel/cpu.c void cpu_hotplug_begin(void) · line 190

Quoted 10 lines
kernel/cpu.c +1 -4
 	/* Notify readers to take the slow path. */
-	__cpuhp_state = readers_slow;
-
-	/* See percpu_down_write(); guarantees all readers take the slow path */
-	synchronize_sched();
+	rcu_sync_enter(&__cpuhp_rss);
 
 	/*
 	 * Notify new readers to block; up until now, and thus throughout the
 	 * longish synchronize_sched() above, new readers could still come in.

s/synchronize_sched/rcu_sync_enter/

Quoted 47 lines
kernel/cpu.c +8 -11
 	 */
-	__cpuhp_state = readers_block;
+	cpuhp_state = readers_block;
 
 	smp_mb(); /* D matches A */
 
 	/*
-	 * If they don't see our writer of readers_block to __cpuhp_state,
+	 * If they don't see our writer of readers_block to cpuhp_state,
 	 * then we are guaranteed to see their __cpuhp_refcount increment, and
 	 * therefore will wait for them.
 	 */
@@ -215,26 +212,23 @@ void cpu_hotplug_done(void)
 	 * that new readers might fail to see the results of this writer's
 	 * critical section.
 	 */
-	__cpuhp_state = readers_slow;
+	cpuhp_state = readers_slow;
 	wake_up_all(&cpuhp_readers);
 
 	/*
 	 * The wait_event()/wake_up_all() prevents the race where the readers
-	 * are delayed between fetching __cpuhp_state and blocking.
+	 * are delayed between fetching cpuhp_state and blocking.
 	 */
 
-	/* See percpu_up_write(); readers will no longer attempt to block. */
-	synchronize_sched();
-
-	/* Let 'em rip */
-	__cpuhp_state = readers_fast;
 	current->cpuhp_ref--;
 
 	/*
-	 * Wait for any pending readers to be running. This ensures readers
-	 * after writer and avoids writers starving readers.
+	 * Wait for any pending readers to be running. This avoids writers
+	 * starving readers.
 	 */
 	wait_event(cpuhp_writer, !atomic_read(&cpuhp_waitcount));
+
+	rcu_sync_exit(&__cpuhp_rss);
 }
 
 /*
author maintainer
#43 lore raw

Re: [PATCH 3/3] hotplug: Optimize cpu_hotplug_{begin,done}() using rcu_sync

Quoted 5 lines · Paul E. McKenney

kernel/cpu.c static int cpu_hotplug_disabled; · line 52

Quoted 2 lines
kernel/cpu.c +1 -1
-enum { readers_fast = 0, readers_slow, readers_block };
+enum { readers_slow, readers_block };

It took me a bit to realize that readers_fast is obsoleted by the rcu_sync_is_idle() above. ;-)

Yeah.. I pondered changing/adding to the rcu_sync interface to allow using gp_count like status to avoid the extra variable, but decided against it for now.

#44 lore raw

Re: [PATCH 3/3] hotplug: Optimize cpu_hotplug_{begin,done}() using rcu_sync

Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Paul E. McKenney

kernel/cpu.c static int cpu_hotplug_disabled; · line 52

Quoted 2 lines
kernel/cpu.c +1 -1
-enum { readers_fast = 0, readers_slow, readers_block };
+enum { readers_slow, readers_block };

It took me a bit to realize that readers_fast is obsoleted by the rcu_sync_is_idle() above. ;-)

Yeah.. I pondered changing/adding to the rcu_sync interface to allow using gp_count like status to avoid the extra variable, but decided against it for now.

Should be OK as is -- people who haven't reviewed the old version with readers_fast are unlikely to be confused by the current setup.

Thanx, Paul

#45 lore raw

Re: [PATCH 3/3] hotplug: Optimize cpu_hotplug_{begin,done}() using rcu_sync

In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Paul E. McKenney

kernel/cpu.c static int cpu_hotplug_disabled; · line 52

Quoted 2 lines
kernel/cpu.c +1 -1
-enum { readers_fast = 0, readers_slow, readers_block };
+enum { readers_slow, readers_block };

It took me a bit to realize that readers_fast is obsoleted by the rcu_sync_is_idle() above. ;-)

Yeah.. I pondered changing/adding to the rcu_sync interface to allow using gp_count like status to avoid the extra variable, but decided against it for now.

Agreed, it looks simple enough even if get/put has to read ->gp_state or/and cpuhp_state.

But, just is case, this is one example of why it probably makes sense to rcu_sync_struct->state_changed(new_state, void *data) callback. In this case it could simply do

	static void cpuhp_rcu_sync_state_cb(state, ...)
	{
		switch (state) {
		case GP_IDLE:
			cpuhp_state = readers_fast;
			break;
		case GP_PENDING:
			cpuhp_state = readers_slow;
			break;
		}
	}

Doesn't make sense in this particular case, but perhaps in general.

Oleg.