Per-CPU-variable updates

PATCH In review Paul E. McKenney 2 participants 13 messages Oct 28, 2014 – Oct 29, 2014

Subsystems (5)

Scheduler RCU High-resolution timers, timer wheel, clockevents CPU hotplug +1 more subsystems

lore

Share & tools

lore

Paul E. McKenney

author

2014-10-28 21:55 UTC #0 lore raw

cover Per-CPU-variable updates

Hello!

This series updates RCU's handling of per-CPU variables, mostly moving from "cpu" function arguments to various "this_"-style accessors for per-CPU variables:

1. Drop the rdtp argument from RCU's dyntick-idle and sysidle
functions, substituting this_cpu_ptr(), courtesy of Christoph Lameter.

2. Use DEFINE_PER_CPU_SHARED_ALIGNED for rcu_data structures in
order to avoid false sharing with other per-CPU variables.

3-9. Remove "cpu" arguments from a number of RCU functions that are
only ever invoked on that CPU, and use appropriate "this_"-style accesssors for the per-CPU variables.

Thanx, Paul

------------------------------------------------------------------------

 b/include/linux/rcupdate.h |    2 +-
 b/include/linux/rcutiny.h  |    2 +-
 b/include/linux/rcutree.h  |    4 ++--
 b/kernel/cpu.c             |   19 +++++++++++++------
 b/kernel/rcu/tiny.c        |    2 +-
 b/kernel/rcu/tree.c        |   25 +++++++++++++------------
 b/kernel/rcu/tree.h        |    4 ++--
 b/kernel/rcu/tree_plugin.h |   11 +++++++----
 b/kernel/sched/core.c      |    2 +-
 b/kernel/softirq.c         |    2 +-
 b/kernel/time/tick-sched.c |    2 +-
 b/kernel/time/timer.c      |    3 +--
 include/linux/rcupdate.h   |    2 +-
 include/linux/rcutree.h    |    2 +-
 kernel/rcu/tree.c          |   30 +++++++++++++++---------------
 kernel/rcu/tree.h          |    8 ++++----
 kernel/rcu/tree_plugin.h   |   42 +++++++++++++++++++++---------------------
 17 files changed, 86 insertions(+), 76 deletions(-)

Paul E. McKenney

author

2014-10-28 21:57 UTC #4 lore raw

PATCH 4/10 rcu: Remove "cpu" argument to rcu_pending()

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

Because rcu_pending()'s argument is guaranteed to always be the current CPU, drop the argument and replace per_cpu_ptr() with this_cpu_ptr().

Signed-off-by Paul E. McKenney <[email protected]>

---
 kernel/rcu/tree.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

kernel/rcu/tree.c +4 -4

index 3107811bba69..0670ba3a65d3 100644
@@ -325,7 +325,7 @@ static void force_qs_rnp(struct rcu_state *rsp,
 				  unsigned long *maxj),
 			 bool *isidle, unsigned long *maxj);
 static void force_quiescent_state(struct rcu_state *rsp);
-static int rcu_pending(int cpu);
+static int rcu_pending(void);
 
 /*
  * Return the number of RCU-sched batches processed thus far for debug & stats.

@@ -2421,7 +2421,7 @@ void rcu_check_callbacks(int user)
 		rcu_bh_qs();
 	}
 	rcu_preempt_check_callbacks(smp_processor_id());
-	if (rcu_pending(smp_processor_id()))
+	if (rcu_pending())
 		invoke_rcu_core();
 	if (user)
 		rcu_note_voluntary_context_switch(current);

@@ -3144,12 +3144,12 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
  * by the current CPU, returning 1 if so.  This function is part of the
  * RCU implementation; it is -not- an exported member of the RCU API.
  */
-static int rcu_pending(int cpu)
+static int rcu_pending(void)
 {
 	struct rcu_state *rsp;
 
 	for_each_rcu_flavor(rsp)
-		if (__rcu_pending(rsp, per_cpu_ptr(rsp->rda, cpu)))
+		if (__rcu_pending(rsp, this_cpu_ptr(rsp->rda)))
 			return 1;
 	return 0;
 }

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 21:57 UTC #2 lore raw

PATCH 2/10 rcu: Use DEFINE_PER_CPU_SHARED_ALIGNED for rcu_data

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

The rcu_data per-CPU variable has a number of fields that are atomically manipulated, potentially by any CPU. This situation can result in false sharing with per-CPU variables that have the misfortune of being allocated adjacent to rcu_data in memory. This commit therefore changes the DEFINE_PER_CPU() to DEFINE_PER_CPU_SHARED_ALIGNED() in order to avoid this false sharing.

Reported-by Christoph Lameter <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Reviewed-by Christoph Lameter <[email protected]>

---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/rcu/tree.c +1 -1

index c0673c56fb1a..ab6fcfb4fe11 100644
@@ -105,7 +105,7 @@ struct rcu_state sname##_state = { \
 	.name = RCU_STATE_NAME(sname), \
 	.abbr = sabbr, \
 }; \
-DEFINE_PER_CPU(struct rcu_data, sname##_data)
+DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, sname##_data)
 
 RCU_STATE_INITIALIZER(rcu_sched, 's', call_rcu_sched);
 RCU_STATE_INITIALIZER(rcu_bh, 'b', call_rcu_bh);

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 21:57 UTC #1 lore raw

PATCH 1/10 rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks)

In reply to Paul E. McKenney

From: Christoph Lameter <[email protected]>

For some functions in kernel/rcu/tree* the rdtp parameter is always this_cpu_ptr(rdtp). Remove the parameter if constant and calculate the pointer in function.

This will have the advantage that it is obvious that the address are all per cpu offsets and thus it will enable the use of this_cpu_ops in the future.

Signed-off-by Christoph Lameter <[email protected]>

[ paulmck: Forward-ported to rcu/dev, whitespace adjustment. ]

Signed-off-by Paul E. McKenney <[email protected]>

---
 kernel/rcu/tree.c        | 25 +++++++++++++------------
 kernel/rcu/tree.h        |  4 ++--
 kernel/rcu/tree_plugin.h | 11 +++++++----
 3 files changed, 22 insertions(+), 18 deletions(-)

kernel/rcu/tree.c +13 ~~-12~~

index 9815447d22e0..c0673c56fb1a 100644
@@ -510,11 +510,11 @@ cpu_needs_another_gp(struct rcu_state *rsp, struct rcu_data *rdp)
  * we really have entered idle, and must do the appropriate accounting.
  * The caller must have disabled interrupts.
  */
-static void rcu_eqs_enter_common(struct rcu_dynticks *rdtp, long long oldval,
-				bool user)
+static void rcu_eqs_enter_common(long long oldval, bool user)
 {
 	struct rcu_state *rsp;
 	struct rcu_data *rdp;
+	struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
 
 	trace_rcu_dyntick(TPS("Start"), oldval, rdtp->dynticks_nesting);
 	if (!user && !is_idle_task(current)) {

@@ -565,7 +565,7 @@ static void rcu_eqs_enter(bool user)
 	WARN_ON_ONCE((oldval & DYNTICK_TASK_NEST_MASK) == 0);
 	if ((oldval & DYNTICK_TASK_NEST_MASK) == DYNTICK_TASK_NEST_VALUE) {
 		rdtp->dynticks_nesting = 0;
-		rcu_eqs_enter_common(rdtp, oldval, user);
+		rcu_eqs_enter_common(oldval, user);
 	} else {
 		rdtp->dynticks_nesting -= DYNTICK_TASK_NEST_VALUE;
 	}

@@ -589,7 +589,7 @@ void rcu_idle_enter(void)
 
 	local_irq_save(flags);
 	rcu_eqs_enter(false);
-	rcu_sysidle_enter(this_cpu_ptr(&rcu_dynticks), 0);
+	rcu_sysidle_enter(0);
 	local_irq_restore(flags);
 }
 EXPORT_SYMBOL_GPL(rcu_idle_enter);

@@ -639,8 +639,8 @@ void rcu_irq_exit(void)
 	if (rdtp->dynticks_nesting)
 		trace_rcu_dyntick(TPS("--="), oldval, rdtp->dynticks_nesting);
 	else
-		rcu_eqs_enter_common(rdtp, oldval, true);
-	rcu_sysidle_enter(rdtp, 1);
+		rcu_eqs_enter_common(oldval, true);
+	rcu_sysidle_enter(1);
 	local_irq_restore(flags);
 }

@@ -651,9 +651,10 @@ void rcu_irq_exit(void)
  * we really have exited idle, and must do the appropriate accounting.
  * The caller must have disabled interrupts.
  */
-static void rcu_eqs_exit_common(struct rcu_dynticks *rdtp, long long oldval,
-			       int user)
+static void rcu_eqs_exit_common(long long oldval, int user)
 {
+	struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
+
 	rcu_dynticks_task_exit();
 	smp_mb__before_atomic();  /* Force ordering w/previous sojourn. */
 	atomic_inc(&rdtp->dynticks);

@@ -691,7 +692,7 @@ static void rcu_eqs_exit(bool user)
 		rdtp->dynticks_nesting += DYNTICK_TASK_NEST_VALUE;
 	} else {
 		rdtp->dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
-		rcu_eqs_exit_common(rdtp, oldval, user);
+		rcu_eqs_exit_common(oldval, user);
 	}
 }

@@ -712,7 +713,7 @@ void rcu_idle_exit(void)
 
 	local_irq_save(flags);
 	rcu_eqs_exit(false);
-	rcu_sysidle_exit(this_cpu_ptr(&rcu_dynticks), 0);
+	rcu_sysidle_exit(0);
 	local_irq_restore(flags);
 }
 EXPORT_SYMBOL_GPL(rcu_idle_exit);

@@ -763,8 +764,8 @@ void rcu_irq_enter(void)
 	if (oldval)
 		trace_rcu_dyntick(TPS("++="), oldval, rdtp->dynticks_nesting);
 	else
-		rcu_eqs_exit_common(rdtp, oldval, true);
-	rcu_sysidle_exit(rdtp, 1);
+		rcu_eqs_exit_common(oldval, true);
+	rcu_sysidle_exit(1);
 	local_irq_restore(flags);
 }

kernel/rcu/tree.h +2 -2

index bbdc45d8d74f..8e90562010ec 100644
@@ -606,8 +606,8 @@ static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp);
 #endif /* #ifdef CONFIG_RCU_NOCB_CPU */
 static void __maybe_unused rcu_kick_nohz_cpu(int cpu);
 static bool init_nocb_callback_list(struct rcu_data *rdp);
-static void rcu_sysidle_enter(struct rcu_dynticks *rdtp, int irq);
-static void rcu_sysidle_exit(struct rcu_dynticks *rdtp, int irq);
+static void rcu_sysidle_enter(int irq);
+static void rcu_sysidle_exit(int irq);
 static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,
 				  unsigned long *maxj);
 static bool is_sysidle_rcu_state(struct rcu_state *rsp);

kernel/rcu/tree_plugin.h +7 -4

index c1d7f27bd38f..09547143628a 100644
@@ -2761,9 +2761,10 @@ static int full_sysidle_state;		/* Current system-idle state. */
  * to detect full-system idle states, not RCU quiescent states and grace
  * periods.  The caller must have disabled interrupts.
  */
-static void rcu_sysidle_enter(struct rcu_dynticks *rdtp, int irq)
+static void rcu_sysidle_enter(int irq)
 {
 	unsigned long j;
+	struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
 
 	/* If there are no nohz_full= CPUs, no need to track this. */
 	if (!tick_nohz_full_enabled())

@@ -2832,8 +2833,10 @@ void rcu_sysidle_force_exit(void)
  * usermode execution does -not- count as idle here!  The caller must
  * have disabled interrupts.
  */
-static void rcu_sysidle_exit(struct rcu_dynticks *rdtp, int irq)
+static void rcu_sysidle_exit(int irq)
 {
+	struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
+
 	/* If there are no nohz_full= CPUs, no need to track this. */
 	if (!tick_nohz_full_enabled())
 		return;

@@ -3127,11 +3130,11 @@ static void rcu_sysidle_init_percpu_data(struct rcu_dynticks *rdtp)
 
 #else /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
 
-static void rcu_sysidle_enter(struct rcu_dynticks *rdtp, int irq)
+static void rcu_sysidle_enter(int irq)
 {
 }
 
-static void rcu_sysidle_exit(struct rcu_dynticks *rdtp, int irq)
+static void rcu_sysidle_exit(int irq)
 {
 }

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 21:58 UTC #3 lore raw

PATCH 3/10 rcu: Remove "cpu" argument to rcu_check_callbacks()

From: "Paul E. McKenney" <[email protected]>

The "cpu" argument was kept around on the off-chance that RCU might offload scheduler-clock interrupts. However, this offload approach has been replaced by NO_HZ_FULL, which offloads -all- RCU processing from qualifying CPUs. It is therefore time to remove the "cpu" argument to rcu_check_callbacks(), which this commit does.

Signed-off-by Paul E. McKenney <[email protected]>

---
 include/linux/rcupdate.h | 2 +-
 kernel/rcu/tiny.c        | 2 +-
 kernel/rcu/tree.c        | 6 +++---
 kernel/time/timer.c      | 3 +--
 4 files changed, 6 insertions(+), 7 deletions(-)

include/linux/rcupdate.h +1 -1

index a4a819ffb2d1..b4ad275d90d0 100644
@@ -260,7 +260,7 @@ static inline int rcu_preempt_depth(void)
 void rcu_init(void);
 void rcu_sched_qs(void);
 void rcu_bh_qs(void);
-void rcu_check_callbacks(int cpu, int user);
+void rcu_check_callbacks(int user);
 struct notifier_block;
 void rcu_idle_enter(void);
 void rcu_idle_exit(void);

kernel/rcu/tiny.c +1 -1

index c0623fc47125..01570c68d237 100644
@@ -247,7 +247,7 @@ void rcu_bh_qs(void)
  * be called from hardirq context.  It is normally called from the
  * scheduling-clock interrupt.
  */
-void rcu_check_callbacks(int cpu, int user)
+void rcu_check_callbacks(int user)
 {
 	RCU_TRACE(check_cpu_stalls());
 	if (user || rcu_is_cpu_rrupt_from_idle())

kernel/rcu/tree.c +3 -3

index ab6fcfb4fe11..3107811bba69 100644
@@ -2388,7 +2388,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)
  * invoked from the scheduling-clock interrupt.  If rcu_pending returns
  * false, there is no point in invoking rcu_check_callbacks().
  */
-void rcu_check_callbacks(int cpu, int user)
+void rcu_check_callbacks(int user)
 {
 	trace_rcu_utilization(TPS("Start scheduler-tick"));
 	increment_cpu_stall_ticks();

@@ -2420,8 +2420,8 @@ void rcu_check_callbacks(int cpu, int user)
 
 		rcu_bh_qs();
 	}
-	rcu_preempt_check_callbacks(cpu);
-	if (rcu_pending(cpu))
+	rcu_preempt_check_callbacks(smp_processor_id());
+	if (rcu_pending(smp_processor_id()))
 		invoke_rcu_core();
 	if (user)
 		rcu_note_voluntary_context_switch(current);

kernel/time/timer.c +1 -2

index 3260ffdb368f..2d3f5c504939 100644
@@ -1377,12 +1377,11 @@ unsigned long get_next_timer_interrupt(unsigned long now)
 void update_process_times(int user_tick)
 {
 	struct task_struct *p = current;
-	int cpu = smp_processor_id();
 
 	/* Note: this timer irq context must be accounted for as well. */
 	account_process_tick(p, user_tick);
 	run_local_timers();
-	rcu_check_callbacks(cpu, user_tick);
+	rcu_check_callbacks(user_tick);
 #ifdef CONFIG_IRQ_WORK
 	if (in_irq())
 		irq_work_tick();

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 21:58 UTC #8 lore raw

PATCH 8/10 rcu: Remove "cpu" argument to rcu_prepare_for_idle()

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

The "cpu" argument to rcu_prepare_for_idle() is always the current CPU, so drop it. This in turn allows two of the uses of "cpu" in this function to be replaced with a this_cpu_ptr() and the third by smp_processor_id(), replacing that of the call to rcu_prepare_for_idle(). Again, the anticipated cross-CPU uses of these functions has been replaced by NO_HZ_FULL.

Signed-off-by Paul E. McKenney <[email protected]>

---
 kernel/rcu/tree.c        |  2 +-
 kernel/rcu/tree.h        |  2 +-
 kernel/rcu/tree_plugin.h | 10 +++++-----
 3 files changed, 7 insertions(+), 7 deletions(-)

kernel/rcu/tree.c +1 -1

index d678a98caf1d..8251938fb4ae 100644
@@ -531,7 +531,7 @@ static void rcu_eqs_enter_common(long long oldval, bool user)
 		rdp = this_cpu_ptr(rsp->rda);
 		do_nocb_deferred_wakeup(rdp);
 	}
-	rcu_prepare_for_idle(smp_processor_id());
+	rcu_prepare_for_idle();
 	/* CPUs seeing atomic_inc() must see prior RCU read-side crit sects */
 	smp_mb__before_atomic();  /* See above. */
 	atomic_inc(&rdtp->dynticks);

kernel/rcu/tree.h +1 -1

index 238ac39053fc..8bc3cacbe14a 100644
@@ -580,7 +580,7 @@ static int rcu_spawn_one_boost_kthread(struct rcu_state *rsp,
 static void __init rcu_spawn_boost_kthreads(void);
 static void rcu_prepare_kthreads(int cpu);
 static void rcu_cleanup_after_idle(int cpu);
-static void rcu_prepare_for_idle(int cpu);
+static void rcu_prepare_for_idle(void);
 static void rcu_idle_count_callbacks_posted(void);
 static void print_cpu_stall_info_begin(void);
 static void print_cpu_stall_info(struct rcu_state *rsp, int cpu);

kernel/rcu/tree_plugin.h +5 -5

index 1797b76cb3ff..5e68ab5fe317 100644
@@ -1531,7 +1531,7 @@ static void rcu_cleanup_after_idle(int cpu)
  * Do the idle-entry grace-period work, which, because CONFIG_RCU_FAST_NO_HZ=n,
  * is nothing.
  */
-static void rcu_prepare_for_idle(int cpu)
+static void rcu_prepare_for_idle(void)
 {
 }

@@ -1666,12 +1666,12 @@ int rcu_needs_cpu(unsigned long *dj)
  *
  * The caller must have disabled interrupts.
  */
-static void rcu_prepare_for_idle(int cpu)
+static void rcu_prepare_for_idle(void)
 {
 #ifndef CONFIG_RCU_NOCB_CPU_ALL
 	bool needwake;
 	struct rcu_data *rdp;
-	struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
+	struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
 	struct rcu_node *rnp;
 	struct rcu_state *rsp;
 	int tne;

@@ -1688,7 +1688,7 @@ static void rcu_prepare_for_idle(int cpu)
 		return;
 
 	/* If this is a no-CBs CPU, no callbacks, just return. */
-	if (rcu_is_nocb_cpu(cpu))
+	if (rcu_is_nocb_cpu(smp_processor_id()))
 		return;
 
 	/*

@@ -1712,7 +1712,7 @@ static void rcu_prepare_for_idle(int cpu)
 		return;
 	rdtp->last_accelerate = jiffies;
 	for_each_rcu_flavor(rsp) {
-		rdp = per_cpu_ptr(rsp->rda, cpu);
+		rdp = this_cpu_ptr(rsp->rda);
 		if (!*rdp->nxttail[RCU_DONE_TAIL])
 			continue;
 		rnp = rdp->mynode;

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 21:58 UTC #7 lore raw

PATCH 7/10 rcu: Remove "cpu" argument to rcu_needs_cpu()

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

The "cpu" argument to rcu_needs_cpu() is always the current CPU, so drop it. This in turn allows the "cpu" argument to rcu_cpu_has_callbacks() to be removed, which allows the uses of "cpu" in both functions to be replaced with a this_cpu_ptr(). Again, the anticipated cross-CPU uses of these functions has been replaced by NO_HZ_FULL.

Signed-off-by Paul E. McKenney <[email protected]>

---
 include/linux/rcupdate.h |  2 +-
 include/linux/rcutree.h  |  2 +-
 kernel/rcu/tree.c        |  4 ++--
 kernel/rcu/tree_plugin.h | 12 ++++++------
 kernel/time/tick-sched.c |  2 +-
 5 files changed, 11 insertions(+), 11 deletions(-)

include/linux/rcupdate.h +1 -1

index b4ad275d90d0..4eb810832b19 100644
@@ -1103,7 +1103,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
 	__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
 
 #if defined(CONFIG_TINY_RCU) || defined(CONFIG_RCU_NOCB_CPU_ALL)
-static inline int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
+static inline int rcu_needs_cpu(unsigned long *delta_jiffies)
 {
 	*delta_jiffies = ULONG_MAX;
 	return 0;

include/linux/rcutree.h +1 -1

index 7b5484db1857..52953790dcca 100644
@@ -32,7 +32,7 @@
 
 void rcu_note_context_switch(void);
 #ifndef CONFIG_RCU_NOCB_CPU_ALL
-int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies);
+int rcu_needs_cpu(unsigned long *delta_jiffies);
 #endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */
 void rcu_cpu_stall_reset(void);

kernel/rcu/tree.c +2 -2

index b591f1459240..d678a98caf1d 100644
@@ -3159,7 +3159,7 @@ static int rcu_pending(void)
  * non-NULL, store an indication of whether all callbacks are lazy.
  * (If there are no callbacks, all of them are deemed to be lazy.)
  */
-static int __maybe_unused rcu_cpu_has_callbacks(int cpu, bool *all_lazy)
+static int __maybe_unused rcu_cpu_has_callbacks(bool *all_lazy)
 {
 	bool al = true;
 	bool hc = false;

@@ -3167,7 +3167,7 @@ static int __maybe_unused rcu_cpu_has_callbacks(int cpu, bool *all_lazy)
 	struct rcu_state *rsp;
 
 	for_each_rcu_flavor(rsp) {
-		rdp = per_cpu_ptr(rsp->rda, cpu);
+		rdp = this_cpu_ptr(rsp->rda);
 		if (!rdp->nxtlist)
 			continue;
 		hc = true;

kernel/rcu/tree_plugin.h +6 -6

index 495d4cce47a7..1797b76cb3ff 100644
@@ -1512,10 +1512,10 @@ static void rcu_prepare_kthreads(int cpu)
  * any flavor of RCU.
  */
 #ifndef CONFIG_RCU_NOCB_CPU_ALL
-int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
+int rcu_needs_cpu(unsigned long *delta_jiffies)
 {
 	*delta_jiffies = ULONG_MAX;
-	return rcu_cpu_has_callbacks(cpu, NULL);
+	return rcu_cpu_has_callbacks(NULL);
 }
 #endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */

@@ -1624,15 +1624,15 @@ static bool __maybe_unused rcu_try_advance_all_cbs(void)
  * The caller must have disabled interrupts.
  */
 #ifndef CONFIG_RCU_NOCB_CPU_ALL
-int rcu_needs_cpu(int cpu, unsigned long *dj)
+int rcu_needs_cpu(unsigned long *dj)
 {
-	struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
+	struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
 
 	/* Snapshot to detect later posting of non-lazy callback. */
 	rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted;
 
 	/* If no callbacks, RCU doesn't need the CPU. */
-	if (!rcu_cpu_has_callbacks(cpu, &rdtp->all_lazy)) {
+	if (!rcu_cpu_has_callbacks(&rdtp->all_lazy)) {
 		*dj = ULONG_MAX;
 		return 0;
 	}

@@ -1679,7 +1679,7 @@ static void rcu_prepare_for_idle(int cpu)
 	/* Handle nohz enablement switches conservatively. */
 	tne = ACCESS_ONCE(tick_nohz_active);
 	if (tne != rdtp->tick_nohz_enabled_snap) {
-		if (rcu_cpu_has_callbacks(cpu, NULL))
+		if (rcu_cpu_has_callbacks(NULL))
 			invoke_rcu_core(); /* force nohz to see update. */
 		rdtp->tick_nohz_enabled_snap = tne;
 		return;

kernel/time/tick-sched.c +1 -1

index 7b5741fc4110..1f4356037a7d 100644
@@ -585,7 +585,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
 		last_jiffies = jiffies;
 	} while (read_seqretry(&jiffies_lock, seq));
 
-	if (rcu_needs_cpu(cpu, &rcu_delta_jiffies) ||
+	if (rcu_needs_cpu(&rcu_delta_jiffies) ||
 	    arch_needs_cpu() || irq_work_needs_cpu()) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 21:58 UTC #9 lore raw

PATCH 9/10 rcu: Remove "cpu" argument to rcu_cleanup_after_idle()

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

The "cpu" argument to rcu_cleanup_after_idle() is always the current CPU, so drop it. This moves the smp_processor_id() from the caller to rcu_cleanup_after_idle(), saving argument-passing overhead. Again, the anticipated cross-CPU uses of these functions has been replaced by NO_HZ_FULL.

Signed-off-by Paul E. McKenney <[email protected]>

---
 kernel/rcu/tree.c        | 2 +-
 kernel/rcu/tree.h        | 2 +-
 kernel/rcu/tree_plugin.h | 6 +++---
 3 files changed, 5 insertions(+), 5 deletions(-)

kernel/rcu/tree.c +1 -1

index 8251938fb4ae..fa76fd3c219c 100644
@@ -661,7 +661,7 @@ static void rcu_eqs_exit_common(long long oldval, int user)
 	/* CPUs seeing atomic_inc() must see later RCU read-side crit sects */
 	smp_mb__after_atomic();  /* See above. */
 	WARN_ON_ONCE(!(atomic_read(&rdtp->dynticks) & 0x1));
-	rcu_cleanup_after_idle(smp_processor_id());
+	rcu_cleanup_after_idle();
 	trace_rcu_dyntick(TPS("End"), oldval, rdtp->dynticks_nesting);
 	if (!user && !is_idle_task(current)) {
 		struct task_struct *idle __maybe_unused =

kernel/rcu/tree.h +1 -1

index 8bc3cacbe14a..36c30390e4e9 100644
@@ -579,7 +579,7 @@ static int rcu_spawn_one_boost_kthread(struct rcu_state *rsp,
 #endif /* #ifdef CONFIG_RCU_BOOST */
 static void __init rcu_spawn_boost_kthreads(void);
 static void rcu_prepare_kthreads(int cpu);
-static void rcu_cleanup_after_idle(int cpu);
+static void rcu_cleanup_after_idle(void);
 static void rcu_prepare_for_idle(void);
 static void rcu_idle_count_callbacks_posted(void);
 static void print_cpu_stall_info_begin(void);

kernel/rcu/tree_plugin.h +3 -3

index 5e68ab5fe317..2443282737ba 100644
@@ -1523,7 +1523,7 @@ int rcu_needs_cpu(unsigned long *delta_jiffies)
  * Because we do not have RCU_FAST_NO_HZ, don't bother cleaning up
  * after it.
  */
-static void rcu_cleanup_after_idle(int cpu)
+static void rcu_cleanup_after_idle(void)
 {
 }

@@ -1731,10 +1731,10 @@ static void rcu_prepare_for_idle(void)
  * any grace periods that elapsed while the CPU was idle, and if any
  * callbacks are now ready to invoke, initiate invocation.
  */
-static void rcu_cleanup_after_idle(int cpu)
+static void rcu_cleanup_after_idle(void)
 {
 #ifndef CONFIG_RCU_NOCB_CPU_ALL
-	if (rcu_is_nocb_cpu(cpu))
+	if (rcu_is_nocb_cpu(smp_processor_id()))
 		return;
 	if (rcu_try_advance_all_cbs())
 		invoke_rcu_core();

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 21:59 UTC #6 lore raw

PATCH 6/10 rcu: Remove "cpu" argument to rcu_note_context_switch()

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

The "cpu" argument to rcu_note_context_switch() is always the current CPU, so drop it. This in turn allows the "cpu" argument to rcu_preempt_note_context_switch() to be removed, which allows the sole use of "cpu" in both functions to be replaced with a this_cpu_ptr(). Again, the anticipated cross-CPU uses of these functions has been replaced by NO_HZ_FULL.

Signed-off-by Paul E. McKenney <[email protected]>

---
 include/linux/rcutiny.h  | 2 +-
 include/linux/rcutree.h  | 4 ++--
 kernel/rcu/tree.c        | 4 ++--
 kernel/rcu/tree.h        | 2 +-
 kernel/rcu/tree_plugin.h | 6 +++---
 kernel/sched/core.c      | 2 +-
 kernel/softirq.c         | 2 +-
 7 files changed, 11 insertions(+), 11 deletions(-)

include/linux/rcutiny.h +1 -1

index 38cc5b1e252d..0e5366200154 100644
@@ -78,7 +78,7 @@ static inline void kfree_call_rcu(struct rcu_head *head,
 	call_rcu(head, func);
 }
 
-static inline void rcu_note_context_switch(int cpu)
+static inline void rcu_note_context_switch(void)
 {
 	rcu_sched_qs();
 }

include/linux/rcutree.h +2 -2

index 3e2f5d432743..7b5484db1857 100644
@@ -30,7 +30,7 @@
 #ifndef __LINUX_RCUTREE_H
 #define __LINUX_RCUTREE_H
 
-void rcu_note_context_switch(int cpu);
+void rcu_note_context_switch(void);
 #ifndef CONFIG_RCU_NOCB_CPU_ALL
 int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies);
 #endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */

@@ -43,7 +43,7 @@ void rcu_cpu_stall_reset(void);
  */
 static inline void rcu_virt_note_context_switch(int cpu)
 {
-	rcu_note_context_switch(cpu);
+	rcu_note_context_switch();
 }
 
 void synchronize_rcu_bh(void);

kernel/rcu/tree.c +2 -2

index 1af5e2cdcbeb..b591f1459240 100644
@@ -286,11 +286,11 @@ static void rcu_momentary_dyntick_idle(void)
  * and requires special handling for preemptible RCU.
  * The caller must have disabled preemption.
  */
-void rcu_note_context_switch(int cpu)
+void rcu_note_context_switch(void)
 {
 	trace_rcu_utilization(TPS("Start context switch"));
 	rcu_sched_qs();
-	rcu_preempt_note_context_switch(cpu);
+	rcu_preempt_note_context_switch();
 	if (unlikely(raw_cpu_read(rcu_sched_qs_mask)))
 		rcu_momentary_dyntick_idle();
 	trace_rcu_utilization(TPS("End context switch"));

kernel/rcu/tree.h +1 -1

index 94a26e330c1b..238ac39053fc 100644
@@ -547,7 +547,7 @@ DECLARE_PER_CPU(char, rcu_cpu_has_work);
 /* Forward declarations for rcutree_plugin.h */
 static void rcu_bootup_announce(void);
 long rcu_batches_completed(void);
-static void rcu_preempt_note_context_switch(int cpu);
+static void rcu_preempt_note_context_switch(void);
 static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp);
 #ifdef CONFIG_HOTPLUG_CPU
 static void rcu_report_unblock_qs_rnp(struct rcu_node *rnp,

kernel/rcu/tree_plugin.h +3 -3

index 7158814b7b4a..495d4cce47a7 100644
@@ -156,7 +156,7 @@ static void rcu_preempt_qs(void)
  *
  * Caller must disable preemption.
  */
-static void rcu_preempt_note_context_switch(int cpu)
+static void rcu_preempt_note_context_switch(void)
 {
 	struct task_struct *t = current;
 	unsigned long flags;

@@ -167,7 +167,7 @@ static void rcu_preempt_note_context_switch(int cpu)
 	    !t->rcu_read_unlock_special.b.blocked) {
 
 		/* Possibly blocking in an RCU read-side critical section. */
-		rdp = per_cpu_ptr(rcu_preempt_state.rda, cpu);
+		rdp = this_cpu_ptr(rcu_preempt_state.rda);
 		rnp = rdp->mynode;
 		raw_spin_lock_irqsave(&rnp->lock, flags);
 		smp_mb__after_unlock_lock();

@@ -945,7 +945,7 @@ EXPORT_SYMBOL_GPL(rcu_batches_completed);
  * Because preemptible RCU does not exist, we never have to check for
  * CPUs being in quiescent states.
  */
-static void rcu_preempt_note_context_switch(int cpu)
+static void rcu_preempt_note_context_switch(void)
 {
 }

kernel/sched/core.c +1 -1

index 44999505e1bf..cc186945296d 100644
@@ -2802,7 +2802,7 @@ need_resched:
 	preempt_disable();
 	cpu = smp_processor_id();
 	rq = cpu_rq(cpu);
-	rcu_note_context_switch(cpu);
+	rcu_note_context_switch();
 	prev = rq->curr;
 
 	schedule_debug(prev);

kernel/softirq.c +1 -1

index 0699add19164..501baa9ac1be 100644
@@ -656,7 +656,7 @@ static void run_ksoftirqd(unsigned int cpu)
 		 * in the task stack here.
 		 */
 		__do_softirq();
-		rcu_note_context_switch(cpu);
+		rcu_note_context_switch();
 		local_irq_enable();
 		cond_resched();
 		return;

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 22:00 UTC #5 lore raw

PATCH 5/10 rcu: Remove "cpu" argument to rcu_preempt_check_callbacks()

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

Because rcu_preempt_check_callbacks()'s argument is guaranteed to always be the current CPU, drop the argument and replace per_cpu() with __this_cpu_read().

Signed-off-by Paul E. McKenney <[email protected]>

---
 kernel/rcu/tree.c        | 2 +-
 kernel/rcu/tree.h        | 2 +-
 kernel/rcu/tree_plugin.h | 8 ++++----
 3 files changed, 6 insertions(+), 6 deletions(-)

kernel/rcu/tree.c +1 -1

index 0670ba3a65d3..1af5e2cdcbeb 100644
@@ -2420,7 +2420,7 @@ void rcu_check_callbacks(int user)
 
 		rcu_bh_qs();
 	}
-	rcu_preempt_check_callbacks(smp_processor_id());
+	rcu_preempt_check_callbacks();
 	if (rcu_pending())
 		invoke_rcu_core();
 	if (user)

kernel/rcu/tree.h +1 -1

index 8e90562010ec..94a26e330c1b 100644
@@ -561,7 +561,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
 				     struct rcu_node *rnp,
 				     struct rcu_data *rdp);
 #endif /* #ifdef CONFIG_HOTPLUG_CPU */
-static void rcu_preempt_check_callbacks(int cpu);
+static void rcu_preempt_check_callbacks(void);
 void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu));
 #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_TREE_PREEMPT_RCU)
 static void rcu_report_exp_rnp(struct rcu_state *rsp, struct rcu_node *rnp,

kernel/rcu/tree_plugin.h +4 -4

index 09547143628a..7158814b7b4a 100644
@@ -621,7 +621,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
  *
  * Caller must disable hard irqs.
  */
-static void rcu_preempt_check_callbacks(int cpu)
+static void rcu_preempt_check_callbacks(void)
 {
 	struct task_struct *t = current;

@@ -630,8 +630,8 @@ static void rcu_preempt_check_callbacks(int cpu)
 		return;
 	}
 	if (t->rcu_read_lock_nesting > 0 &&
-	    per_cpu(rcu_preempt_data, cpu).qs_pending &&
-	    !per_cpu(rcu_preempt_data, cpu).passed_quiesce)
+	    __this_cpu_read(rcu_preempt_data.qs_pending) &&
+	    !__this_cpu_read(rcu_preempt_data.passed_quiesce))
 		t->rcu_read_unlock_special.b.need_qs = true;
 }

@@ -1017,7 +1017,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
  * Because preemptible RCU does not exist, it never has any callbacks
  * to check.
  */
-static void rcu_preempt_check_callbacks(int cpu)
+static void rcu_preempt_check_callbacks(void)
 {
 }

Signature · 1 line

1.8.1.5

Paul E. McKenney

author

2014-10-28 22:00 UTC #10 lore raw

PATCH 10/10 cpu: Avoid puts_pending overflow

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

A long string of get_online_cpus() with each followed by a put_online_cpu() that fails to acquire cpu_hotplug.lock can result in overflow of the cpu_hotplug.puts_pending counter. Although this is perhaps improbably, a system with absolutely no CPU-hotplug operations will have an arbitrarily long time in which this overflow could occur. This commit therefore adds overflow checks to get_online_cpus() and try_get_online_cpus().

Signed-off-by Paul E. McKenney <[email protected]>

---
 kernel/cpu.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

kernel/cpu.c +13 -6

index 90a3d017b90c..5d220234b3ca 100644
@@ -86,6 +86,16 @@ static struct {
 #define cpuhp_lock_acquire()      lock_map_acquire(&cpu_hotplug.dep_map)
 #define cpuhp_lock_release()      lock_map_release(&cpu_hotplug.dep_map)
 
+static void apply_puts_pending(int max)
+{
+	int delta;
+
+	if (atomic_read(&cpu_hotplug.puts_pending) >= max) {
+		delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
+		cpu_hotplug.refcount -= delta;
+	}
+}
+
 void get_online_cpus(void)
 {
 	might_sleep();

@@ -93,6 +103,7 @@ void get_online_cpus(void)
 		return;
 	cpuhp_lock_acquire_read();
 	mutex_lock(&cpu_hotplug.lock);
+	apply_puts_pending(65536);
 	cpu_hotplug.refcount++;
 	mutex_unlock(&cpu_hotplug.lock);
 }

@@ -105,6 +116,7 @@ bool try_get_online_cpus(void)
 	if (!mutex_trylock(&cpu_hotplug.lock))
 		return false;
 	cpuhp_lock_acquire_tryread();
+	apply_puts_pending(65536);
 	cpu_hotplug.refcount++;
 	mutex_unlock(&cpu_hotplug.lock);
 	return true;

@@ -161,12 +173,7 @@ void cpu_hotplug_begin(void)
 	cpuhp_lock_acquire();
 	for (;;) {
 		mutex_lock(&cpu_hotplug.lock);
-		if (atomic_read(&cpu_hotplug.puts_pending)) {
-			int delta;
-
-			delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
-			cpu_hotplug.refcount -= delta;
-		}
+		apply_puts_pending(1);
 		if (likely(!cpu_hotplug.refcount))
 			break;
 		__set_current_state(TASK_UNINTERRUPTIBLE);

Signature · 1 line

1.8.1.5

Pranith Kumar

2014-10-29 17:53 UTC #12 lore raw

In reply to Paul E. McKenney

Quoted 10 lines · Paul E. McKenney

Hello!
This series updates RCU's handling of per-CPU variables, mostly moving from "cpu" function arguments to various "this_"-style accessors for per-CPU variables:
1. Drop the rdtp argument from RCU's dyntick-idle and sysidle
functions, substituting this_cpu_ptr(), courtesy of Christoph Lameter.

Quoted 7 lines

2. Use DEFINE_PER_CPU_SHARED_ALIGNED for rcu_data structures in
order to avoid false sharing with other per-CPU variables.
3-9. Remove "cpu" arguments from a number of RCU functions that are
only ever invoked on that CPU, and use appropriate "this_"-style accesssors for the per-CPU variables.

Reviewd-by Pranith Kumar <[email protected]>

Quoted 23 lines

Thanx, Paul

------------------------------------------------------------------------

 b/include/linux/rcupdate.h |    2 +-
 b/include/linux/rcutiny.h  |    2 +-
 b/include/linux/rcutree.h  |    4 ++--
 b/kernel/cpu.c             |   19 +++++++++++++------
 b/kernel/rcu/tiny.c        |    2 +-
 b/kernel/rcu/tree.c        |   25 +++++++++++++------------
 b/kernel/rcu/tree.h        |    4 ++--
 b/kernel/rcu/tree_plugin.h |   11 +++++++----
 b/kernel/sched/core.c      |    2 +-
 b/kernel/softirq.c         |    2 +-
 b/kernel/time/tick-sched.c |    2 +-
 b/kernel/time/timer.c      |    3 +--
 include/linux/rcupdate.h   |    2 +-
 include/linux/rcutree.h    |    2 +-
 kernel/rcu/tree.c          |   30 +++++++++++++++---------------
 kernel/rcu/tree.h          |    8 ++++----
 kernel/rcu/tree_plugin.h   |   42 +++++++++++++++++++++---------------------
 17 files changed, 86 insertions(+), 76 deletions(-)

Signature · 1 line

Pranith

Pranith Kumar

Reviewed-by on v1 10/10

2014-10-29 17:56 UTC #11 lore raw

Re: [PATCH tip/core/rcu 10/10] cpu: Avoid puts_pending overflow

In reply to Paul E. McKenney

Quoted 11 lines · Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>
A long string of get_online_cpus() with each followed by a put_online_cpu() that fails to acquire cpu_hotplug.lock can result in overflow of the cpu_hotplug.puts_pending counter. Although this is perhaps improbably, a system with absolutely no CPU-hotplug operations will have an arbitrarily long time in which this overflow could occur. This commit therefore adds overflow checks to get_online_cpus() and try_get_online_cpus().
Signed-off-by Paul E. McKenney <[email protected]>

This patch seems to be missing in the cover letter.

Reviewed-by Pranith Kumar <[email protected]>

Quoted 58 lines

---
 kernel/cpu.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

kernel/cpu.c +13 -7

index 90a3d017b90c..5d220234b3ca 100644
@@ -86,6 +86,16 @@ static struct {
 #define cpuhp_lock_acquire()      lock_map_acquire(&cpu_hotplug.dep_map)
 #define cpuhp_lock_release()      lock_map_release(&cpu_hotplug.dep_map)
 
+static void apply_puts_pending(int max)
+{
+       int delta;
+
+       if (atomic_read(&cpu_hotplug.puts_pending) >= max) {
+               delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
+               cpu_hotplug.refcount -= delta;
+       }
+}
+
 void get_online_cpus(void)
 {
        might_sleep();

@@ -93,6 +103,7 @@ void get_online_cpus(void)
                return;
        cpuhp_lock_acquire_read();
        mutex_lock(&cpu_hotplug.lock);
+       apply_puts_pending(65536);
        cpu_hotplug.refcount++;
        mutex_unlock(&cpu_hotplug.lock);
 }

@@ -105,6 +116,7 @@ bool try_get_online_cpus(void)
        if (!mutex_trylock(&cpu_hotplug.lock))
                return false;
        cpuhp_lock_acquire_tryread();
+       apply_puts_pending(65536);
        cpu_hotplug.refcount++;
        mutex_unlock(&cpu_hotplug.lock);
        return true;

@@ -161,12 +173,7 @@ void cpu_hotplug_begin(void)
        cpuhp_lock_acquire();
        for (;;) {
                mutex_lock(&cpu_hotplug.lock);
-               if (atomic_read(&cpu_hotplug.puts_pending)) {
-                       int delta;
-
-                       delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
-                       cpu_hotplug.refcount -= delta;
-               }
+               apply_puts_pending(1);
                if (likely(!cpu_hotplug.refcount))
                        break;
                __set_current_state(TASK_UNINTERRUPTIBLE);
--

1.8.1.5

Signature · 1 line

Pranith