Stop using ACCESS_ONCE (for v4.2)
Subsystems (3)
cover Stop using ACCESS_ONCE (for v4.2)
Hello!
This patch series converts from ACCESS_ONCE() to either READ_ONCE() or WRITE_ONCE(), as appropriate. This permits better code generation and error checking, and is also more compatible with C11 directions. Each file is updated separately to simplify handling of git conflicts.
Thanx, Paul
------------------------------------------------------------------------
include/linux/rculist.h | 6 - include/linux/rcupdate.h | 16 ++-- kernel/rcu/rcutorture.c | 2 kernel/rcu/srcu.c | 10 +- kernel/rcu/tiny_plugin.h | 12 +-- kernel/rcu/tree.c | 184 +++++++++++++++++++++++------------------------ kernel/rcu/tree_plugin.h | 93 +++++++++++------------ kernel/rcu/tree_trace.c | 6 - kernel/rcu/update.c | 30 +++---- 9 files changed, 180 insertions(+), 179 deletions(-)
PATCH v2 1/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in rculist.h
From: "Paul E. McKenney" <[email protected]>
This commit moves include/linux/rculist.h from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- include/linux/rculist.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/rculist.h +3 -3
@@ -29,8 +29,8 @@*/ static inline void INIT_LIST_HEAD_RCU (struct list_head *list) { -ACCESS_ONCE (list->next) = list; -ACCESS_ONCE (list->prev) = list; +WRITE_ONCE (list->next, list); +WRITE_ONCE (list->prev, list); }/*
@@ -288,7 +288,7 @@ static inline void list_splice_init_rcu(struct list_head *list,#define list_first_or_null_rcu (ptr, type, member) \ ({ \ struct list_head *__ptr = (ptr); \ - struct list_head *__next = ACCESS_ONCE(__ptr->next); \ + struct list_head *__next = READ_ONCE(__ptr->next); \ likely(__ptr != __next) ? list_entry_rcu(__next, type, member) : NULL; \ })
Signature · 1 line
1.8.1.5
PATCH v2 2/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in rcupdate.h
From: "Paul E. McKenney" <[email protected]>
This commit moves include/linux/rcupdate.h from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- include/linux/rcupdate.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
include/linux/rcupdate.h +8 -8
@@ -364,8 +364,8 @@ extern struct srcu_struct tasks_rcu_exit_srcu;#define rcu_note_voluntary_context_switch (t) \ do { \ rcu_all_qs(); \ - if (ACCESS_ONCE((t)->rcu_tasks_holdout)) \ - ACCESS_ONCE((t)->rcu_tasks_holdout) = false; \ + if (READ_ONCE((t)->rcu_tasks_holdout)) \ + WRITE_ONCE((t)->rcu_tasks_holdout, false); \ } while (0)#else /* #ifdef CONFIG_TASKS_RCU */ #define TASKS_RCU (x) do { } while (0)
@@ -609,7 +609,7 @@ static inline void rcu_preempt_sleep_check(void)#define __rcu_access_pointer (p, space) \ ({ \ - typeof(*p) *_________p1 = (typeof(*p) *__force)ACCESS_ONCE(p); \ + typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \ rcu_dereference_sparse(p, space); \ ((typeof(*p) __force __kernel *)(_________p1)); \ })
@@ -630,7 +630,7 @@ static inline void rcu_preempt_sleep_check(void)#define __rcu_access_index (p, space) \ ({ \ - typeof(p) _________p1 = ACCESS_ONCE(p); \ + typeof(p) _________p1 = READ_ONCE(p); \ rcu_dereference_sparse(p, space); \ (_________p1); \ })
@@ -659,7 +659,7 @@ static inline void rcu_preempt_sleep_check(void)*/ #define lockless_dereference (p) \ ({ \ - typeof(p) _________p1 = ACCESS_ONCE(p); \ + typeof(p) _________p1 = READ_ONCE(p); \ smp_read_barrier_depends();/* Dependency order vs. p above. */ \ (_________p1); \ })
@@ -702,7 +702,7 @@ static inline void rcu_preempt_sleep_check(void)* @p: The pointer to read * * Return the value of the specified RCU-protected pointer, but omit the -* smp_read_barrier_depends() and keep the ACCESS_ONCE(). This is useful +* smp_read_barrier_depends() and keep the READ_ONCE(). This is useful * when the value of this pointer is accessed, but the pointer is not * dereferenced, for example, when testing an RCU-protected pointer against * NULL. Although rcu_access_pointer() may also be used in cases where
@@ -791,7 +791,7 @@ static inline void rcu_preempt_sleep_check(void)* @p: The index to read * * Return the value of the specified RCU-protected index, but omit the -* smp_read_barrier_depends() and keep the ACCESS_ONCE(). This is useful +* smp_read_barrier_depends() and keep the READ_ONCE(). This is useful * when the value of this index is accessed, but the index is not * dereferenced, for example, when testing an RCU-protected index against * -1. Although rcu_access_index() may also be used in cases where
@@ -827,7 +827,7 @@ static inline void rcu_preempt_sleep_check(void)* @c: The conditions under which the dereference will take place * * Return the value of the specified RCU-protected pointer, but omit -* both the smp_read_barrier_depends() and the ACCESS_ONCE(). This +* both the smp_read_barrier_depends() and the READ_ONCE(). This * is useful in cases where update-side locks prevent the value of the * pointer from changing. Please note that this primitive does -not- * prevent the compiler from repeating this reference or combining it
Signature · 1 line
1.8.1.5
PATCH v2 3/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in rcutorture.c
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves kernel/rcu/rcutorture.c from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- kernel/rcu/rcutorture.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/rcutorture.c +1 -1
@@ -1413,7 +1413,7 @@ static int rcu_torture_barrier_cbs(void *arg) do { wait_event(barrier_cbs_wq[myid], (newphase = - ACCESS_ONCE(barrier_phase)) != lastphase || + READ_ONCE(barrier_phase)) != lastphase || torture_must_stop()); lastphase = newphase; smp_mb(); /* ensure barrier_phase load before ->call(). */
Signature · 1 line
1.8.1.5
PATCH v2 4/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in srcu.c
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves kernel/rcu/srcu.c from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- kernel/rcu/srcu.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/rcu/srcu.c +5 -5
@@ -151,7 +151,7 @@ static unsigned long srcu_readers_seq_idx(struct srcu_struct *sp, int idx)unsigned long t;for_each_possible_cpu (cpu) { - t =ACCESS_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->seq[idx]); + t =READ_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->seq[idx]); sum += t; }return sum;
@@ -168,7 +168,7 @@ static unsigned long srcu_readers_active_idx(struct srcu_struct *sp, int idx)unsigned long t;for_each_possible_cpu (cpu) { - t =ACCESS_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->c[idx]); + t =READ_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->c[idx]); sum += t; }return sum;
@@ -265,8 +265,8 @@ static int srcu_readers_active(struct srcu_struct *sp)unsigned long sum =0 ;for_each_possible_cpu (cpu) { - sum +=ACCESS_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->c[0 ]); - sum +=ACCESS_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->c[1 ]); + sum +=READ_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->c[0 ]); + sum +=READ_ONCE (per_cpu_ptr (sp->per_cpu_ref, cpu)->c[1 ]); }return sum; }
@@ -296,7 +296,7 @@ int __srcu_read_lock(struct srcu_struct *sp) {int idx; - idx =ACCESS_ONCE (sp->completed) &0x1 ; + idx =READ_ONCE (sp->completed) &0x1 ;preempt_disable ();__this_cpu_inc (sp->per_cpu_ref->c[idx]);smp_mb ();/* B */ /* Avoid leaking the critical section. */
Signature · 1 line
1.8.1.5
PATCH v2 5/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in tiny_plugin.h
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves kernel/rcu/tiny_plugin.h from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- kernel/rcu/tiny_plugin.h | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
kernel/rcu/tiny_plugin.h +7 -5
@@ -144,16 +144,17 @@ static void check_cpu_stall(struct rcu_ctrlblk *rcp)return ; rcp->ticks_this_gp++; j = jiffies; - js =ACCESS_ONCE (rcp->jiffies_stall); + js =READ_ONCE (rcp->jiffies_stall);if (rcp->rcucblist &&ULONG_CMP_GE (j, js)) {pr_err ("INFO: %s stall on CPU (%lu ticks this GP) idle=%llx (t=%lu jiffies q=%ld)\n" , rcp->name, rcp->ticks_this_gp,DYNTICK_TASK_EXIT_IDLE , jiffies - rcp->gp_start, rcp->qlen);dump_stack (); -ACCESS_ONCE (rcp->jiffies_stall) = jiffies + -3 *rcu_jiffies_till_stall_check () +3 ; +WRITE_ONCE (rcp->jiffies_stall, + jiffies +3 *rcu_jiffies_till_stall_check () +3 ); }else if (ULONG_CMP_GE (j, js)) { -ACCESS_ONCE (rcp->jiffies_stall) = jiffies +rcu_jiffies_till_stall_check (); +WRITE_ONCE (rcp->jiffies_stall, + jiffies +rcu_jiffies_till_stall_check ()); } }
@@ -161,7 +162,8 @@ static void reset_cpu_stall_ticks(struct rcu_ctrlblk *rcp) { rcp->ticks_this_gp =0 ; rcp->gp_start = jiffies; -ACCESS_ONCE (rcp->jiffies_stall) = jiffies +rcu_jiffies_till_stall_check (); +WRITE_ONCE (rcp->jiffies_stall, + jiffies +rcu_jiffies_till_stall_check ()); }static void check_cpu_stalls (void )
Signature · 1 line
1.8.1.5
PATCH v2 6/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in tree.c
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves kernel/rcu/tree.c from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- kernel/rcu/tree.c | 184 +++++++++++++++++++++++++++--------------------------- 1 file changed, 92 insertions(+), 92 deletions(-)
kernel/rcu/tree.c +92 -92
@@ -188,17 +188,17 @@ unsigned long rcutorture_vernum;*/ unsigned long rcu_rnp_online_cpus (struct rcu_node *rnp) { -return ACCESS_ONCE (rnp->qsmaskinitnext); +return READ_ONCE (rnp->qsmaskinitnext); }/* -* Return true if an RCU grace period is in progress. The ACCESS_ONCE()s +* Return true if an RCU grace period is in progress. The READ_ONCE()s * permit this function to be invoked without holding the root rcu_node * structure's ->lock, but of course results can be subject to change. */ static int rcu_gp_in_progress (struct rcu_state *rsp) { -return ACCESS_ONCE (rsp->completed) !=ACCESS_ONCE (rsp->gpnum); +return READ_ONCE (rsp->completed) !=READ_ONCE (rsp->gpnum); }/*
@@ -275,8 +275,8 @@ static void rcu_momentary_dyntick_idle(void)if (!(resched_mask & rsp->flavor_mask))continue ;smp_mb ();/* rcu_sched_qs_mask before cond_resched_completed. */ -if (ACCESS_ONCE (rdp->mynode->completed) != -ACCESS_ONCE (rdp->cond_resched_completed)) +if (READ_ONCE (rdp->mynode->completed) != +READ_ONCE (rdp->cond_resched_completed))continue ;/*
@@ -488,9 +488,9 @@ void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,break ; }if (rsp !=NULL ) { - *flags =ACCESS_ONCE (rsp->gp_flags); - *gpnum =ACCESS_ONCE (rsp->gpnum); - *completed =ACCESS_ONCE (rsp->completed); + *flags =READ_ONCE (rsp->gp_flags); + *gpnum =READ_ONCE (rsp->gpnum); + *completed =READ_ONCE (rsp->completed);return ; } *flags =0 ;
@@ -536,10 +536,10 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp)static int rcu_future_needs_gp (struct rcu_state *rsp) {struct rcu_node *rnp =rcu_get_root (rsp); -int idx = (ACCESS_ONCE (rnp->completed) +1 ) &0x1 ; +int idx = (READ_ONCE (rnp->completed) +1 ) &0x1 ;int *fp = &rnp->need_future_gp[idx]; -return ACCESS_ONCE (*fp); +return READ_ONCE (*fp); }/*
@@ -562,7 +562,7 @@ cpu_needs_another_gp(struct rcu_state *rsp, struct rcu_data *rdp)return 1 ;/* Yes, this CPU has newly registered callbacks. */ for (i =RCU_WAIT_TAIL ; i <RCU_NEXT_TAIL ; i++)if (rdp->nxttail[i -1 ] != rdp->nxttail[i] && -ULONG_CMP_LT (ACCESS_ONCE (rsp->completed), +ULONG_CMP_LT (READ_ONCE (rsp->completed), rdp->nxtcompleted[i]))return 1 ;/* Yes, CBs for future grace period. */ return 0 ;/* No grace period needed. */
@@ -1008,9 +1008,9 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp,trace_rcu_fqs (rdp->rsp->name, rdp->gpnum, rdp->cpu,TPS ("dti" ));return 1 ; }else { -if (ULONG_CMP_LT (ACCESS_ONCE (rdp->gpnum) +ULONG_MAX /4 , +if (ULONG_CMP_LT (READ_ONCE (rdp->gpnum) +ULONG_MAX /4 , rdp->mynode->gpnum)) -ACCESS_ONCE (rdp->gpwrap) = true; +WRITE_ONCE (rdp->gpwrap, true);return 0 ; } }
@@ -1090,12 +1090,12 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,if (ULONG_CMP_GE (jiffies, rdp->rsp->gp_start + jiffies_till_sched_qs) ||ULONG_CMP_GE (jiffies, rdp->rsp->jiffies_resched)) { -if (!(ACCESS_ONCE (*rcrmp) & rdp->rsp->flavor_mask)) { -ACCESS_ONCE (rdp->cond_resched_completed) = -ACCESS_ONCE (rdp->mynode->completed); +if (!(READ_ONCE (*rcrmp) & rdp->rsp->flavor_mask)) { +WRITE_ONCE (rdp->cond_resched_completed, +READ_ONCE (rdp->mynode->completed));smp_mb ();/* ->cond_resched_completed before *rcrmp. */ -ACCESS_ONCE (*rcrmp) = -ACCESS_ONCE (*rcrmp) + rdp->rsp->flavor_mask; +WRITE_ONCE (*rcrmp, +READ_ONCE (*rcrmp) + rdp->rsp->flavor_mask);resched_cpu (rdp->cpu);/* Force CPU into scheduler. */ rdp->rsp->jiffies_resched +=5 ;/* Enable beating. */ }else if (ULONG_CMP_GE (jiffies, rdp->rsp->jiffies_resched)) {
@@ -1116,9 +1116,9 @@ static void record_gp_stall_check_time(struct rcu_state *rsp) rsp->gp_start = j;smp_wmb ();/* Record start time before stall time. */ j1 =rcu_jiffies_till_stall_check (); -ACCESS_ONCE (rsp->jiffies_stall) = j + j1; +WRITE_ONCE (rsp->jiffies_stall, j + j1); rsp->jiffies_resched = j + j1 /2 ; - rsp->n_force_qs_gpstart =ACCESS_ONCE (rsp->n_force_qs); + rsp->n_force_qs_gpstart =READ_ONCE (rsp->n_force_qs); }/*
@@ -1130,7 +1130,7 @@ static void rcu_check_gp_kthread_starvation(struct rcu_state *rsp)unsigned long j; j = jiffies; - gpa =ACCESS_ONCE (rsp->gp_activity); + gpa =READ_ONCE (rsp->gp_activity);if (j - gpa >2 *HZ )pr_err ("%s kthread starved for %ld jiffies!\n" , rsp->name, j - gpa);
@@ -1170,12 +1170,13 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)/* Only let one CPU complain about others per time interval. */ raw_spin_lock_irqsave (&rnp->lock, flags); - delta = jiffies -ACCESS_ONCE (rsp->jiffies_stall); + delta = jiffies -READ_ONCE (rsp->jiffies_stall);if (delta <RCU_STALL_RAT_DELAY || !rcu_gp_in_progress (rsp)) {raw_spin_unlock_irqrestore (&rnp->lock, flags);return ; } -ACCESS_ONCE (rsp->jiffies_stall) = jiffies +3 *rcu_jiffies_till_stall_check () +3 ; +WRITE_ONCE (rsp->jiffies_stall, + jiffies +3 *rcu_jiffies_till_stall_check () +3 );raw_spin_unlock_irqrestore (&rnp->lock, flags);/*
@@ -1209,12 +1210,12 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)if (ndetected) {rcu_dump_cpu_stacks (rsp); }else { -if (ACCESS_ONCE (rsp->gpnum) != gpnum || -ACCESS_ONCE (rsp->completed) == gpnum) { +if (READ_ONCE (rsp->gpnum) != gpnum || +READ_ONCE (rsp->completed) == gpnum) {pr_err ("INFO: Stall ended before state dump start\n" ); }else { j = jiffies; - gpa =ACCESS_ONCE (rsp->gp_activity); + gpa =READ_ONCE (rsp->gp_activity); pr_err("All QSes seen, last %s kthread activity %ld (%ld-%ld), jiffies_till_next_fqs=%ld, root ->qsmask %#lx\n", rsp->name, j - gpa, j, gpa, jiffies_till_next_fqs,
@@ -1259,9 +1260,9 @@ static void print_cpu_stall(struct rcu_state *rsp)rcu_dump_cpu_stacks (rsp);raw_spin_lock_irqsave (&rnp->lock, flags); -if (ULONG_CMP_GE (jiffies,ACCESS_ONCE (rsp->jiffies_stall))) -ACCESS_ONCE (rsp->jiffies_stall) = jiffies + -3 *rcu_jiffies_till_stall_check () +3 ; +if (ULONG_CMP_GE (jiffies,READ_ONCE (rsp->jiffies_stall))) +WRITE_ONCE (rsp->jiffies_stall, + jiffies +3 *rcu_jiffies_till_stall_check () +3 );raw_spin_unlock_irqrestore (&rnp->lock, flags);/*
@@ -1304,20 +1305,20 @@ static void check_cpu_stall(struct rcu_state *rsp, struct rcu_data *rdp)* Given this check, comparisons of jiffies, rsp->jiffies_stall, * and rsp->gp_start suffice to forestall false positives. */ - gpnum =ACCESS_ONCE (rsp->gpnum); + gpnum =READ_ONCE (rsp->gpnum);smp_rmb ();/* Pick up ->gpnum first... */ - js =ACCESS_ONCE (rsp->jiffies_stall); + js =READ_ONCE (rsp->jiffies_stall);smp_rmb ();/* ...then ->jiffies_stall before the rest... */ - gps =ACCESS_ONCE (rsp->gp_start); + gps =READ_ONCE (rsp->gp_start);smp_rmb ();/* ...and finally ->gp_start before ->completed. */ - completed =ACCESS_ONCE (rsp->completed); + completed =READ_ONCE (rsp->completed);if (ULONG_CMP_GE (completed, gpnum) ||ULONG_CMP_LT (j, js) ||ULONG_CMP_GE (gps, js))return ;/* No stall or GP completed since entering function. */ rnp = rdp->mynode;if (rcu_gp_in_progress (rsp) && - (ACCESS_ONCE (rnp->qsmask) & rdp->grpmask)) { + (READ_ONCE (rnp->qsmask) & rdp->grpmask)) {/* We haven't checked in, so go dump stack. */ print_cpu_stall (rsp);
@@ -1344,7 +1345,7 @@ void rcu_cpu_stall_reset(void)struct rcu_state *rsp;for_each_rcu_flavor (rsp) -ACCESS_ONCE (rsp->jiffies_stall) = jiffies +ULONG_MAX /2 ; +WRITE_ONCE (rsp->jiffies_stall, jiffies +ULONG_MAX /2 ); }/*
@@ -1454,7 +1455,7 @@ rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp,* doing some extra useless work. */ if (rnp->gpnum != rnp->completed || -ACCESS_ONCE (rnp_root->gpnum) !=ACCESS_ONCE (rnp_root->completed)) { +READ_ONCE (rnp_root->gpnum) !=READ_ONCE (rnp_root->completed)) { rnp->need_future_gp[c &0x1 ]++;trace_rcu_future_gp (rnp, rdp, c,TPS ("Startedleaf" )); goto out;
@@ -1539,7 +1540,7 @@ static int rcu_future_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)static void rcu_gp_kthread_wake (struct rcu_state *rsp) {if (current == rsp->gp_kthread || - !ACCESS_ONCE (rsp->gp_flags) || + !READ_ONCE (rsp->gp_flags) || !rsp->gp_kthread)return ;wake_up (&rsp->gp_wq);
@@ -1674,7 +1675,7 @@ static bool __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp,/* Handle the ends of any preceding grace periods first. */ if (rdp->completed == rnp->completed && - !unlikely (ACCESS_ONCE (rdp->gpwrap))) { + !unlikely (READ_ONCE (rdp->gpwrap))) {/* No grace period end, so just accelerate recent callbacks. */ ret =rcu_accelerate_cbs (rsp, rnp, rdp);
@@ -1689,7 +1690,7 @@ static bool __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp,trace_rcu_grace_period (rsp->name, rdp->gpnum,TPS ("cpuend" )); } -if (rdp->gpnum != rnp->gpnum ||unlikely (ACCESS_ONCE (rdp->gpwrap))) { +if (rdp->gpnum != rnp->gpnum ||unlikely (READ_ONCE (rdp->gpwrap))) {/* * If the current grace period is waiting for this CPU, * set up to detect a quiescent state, otherwise don't
@@ -1701,7 +1702,7 @@ static bool __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp, rdp->rcu_qs_ctr_snap =__this_cpu_read (rcu_qs_ctr); rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);zero_cpu_stall_ticks (rdp); -ACCESS_ONCE (rdp->gpwrap) = false; +WRITE_ONCE (rdp->gpwrap, false); }return ret; }
@@ -1714,9 +1715,9 @@ static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp)local_irq_save (flags); rnp = rdp->mynode; -if ((rdp->gpnum ==ACCESS_ONCE (rnp->gpnum) && - rdp->completed ==ACCESS_ONCE (rnp->completed) && - !unlikely (ACCESS_ONCE (rdp->gpwrap))) ||/* w/out lock. */ +if ((rdp->gpnum ==READ_ONCE (rnp->gpnum) && + rdp->completed ==READ_ONCE (rnp->completed) && + !unlikely (READ_ONCE (rdp->gpwrap))) ||/* w/out lock. */ !raw_spin_trylock (&rnp->lock)) {/* irqs already off, so later. */ local_irq_restore (flags);return ;
@@ -1737,15 +1738,15 @@ static int rcu_gp_init(struct rcu_state *rsp)struct rcu_data *rdp;struct rcu_node *rnp =rcu_get_root (rsp); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies);raw_spin_lock_irq (&rnp->lock);smp_mb__after_unlock_lock (); -if (!ACCESS_ONCE (rsp->gp_flags)) { +if (!READ_ONCE (rsp->gp_flags)) {/* Spurious wakeup, tell caller to go back to sleep. */ raw_spin_unlock_irq (&rnp->lock);return 0 ; } -ACCESS_ONCE (rsp->gp_flags) =0 ;/* Clear all flags: New grace period. */ +WRITE_ONCE (rsp->gp_flags,0 );/* Clear all flags: New grace period. */ if (WARN_ON_ONCE (rcu_gp_in_progress (rsp))) {/*
@@ -1831,9 +1832,9 @@ static int rcu_gp_init(struct rcu_state *rsp) rdp =this_cpu_ptr (rsp->rda);rcu_preempt_check_blocked_tasks (rnp); rnp->qsmask = rnp->qsmaskinit; -ACCESS_ONCE (rnp->gpnum) = rsp->gpnum; +WRITE_ONCE (rnp->gpnum, rsp->gpnum);if (WARN_ON_ONCE (rnp->completed != rsp->completed)) -ACCESS_ONCE (rnp->completed) = rsp->completed; +WRITE_ONCE (rnp->completed, rsp->completed);if (rnp == rdp->mynode) (void )__note_gp_changes (rsp, rnp, rdp);rcu_preempt_boost_start_gp (rnp);
@@ -1842,7 +1843,7 @@ static int rcu_gp_init(struct rcu_state *rsp) rnp->grphi, rnp->qsmask);raw_spin_unlock_irq (&rnp->lock);cond_resched_rcu_qs (); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies);if (IS_ENABLED (CONFIG_RCU_TORTURE_TEST_SLOW_INIT ) && gp_init_delay >0 && !(rsp->gpnum % (rcu_num_nodes *3 * gp_init_delay)))
@@ -1862,7 +1863,7 @@ static int rcu_gp_fqs(struct rcu_state *rsp, int fqs_state_in)unsigned long maxj;struct rcu_node *rnp =rcu_get_root (rsp); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies); rsp->n_force_qs++;if (fqs_state ==RCU_SAVE_DYNTICK ) {/* Collect dyntick-idle snapshots. */
@@ -1880,11 +1881,11 @@ static int rcu_gp_fqs(struct rcu_state *rsp, int fqs_state_in)force_qs_rnp (rsp, rcu_implicit_dynticks_qs, &isidle, &maxj); }/* Clear flag to prevent immediate re-entry. */ -if (ACCESS_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) { +if (READ_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) {raw_spin_lock_irq (&rnp->lock);smp_mb__after_unlock_lock (); -ACCESS_ONCE (rsp->gp_flags) = -ACCESS_ONCE (rsp->gp_flags) & ~RCU_GP_FLAG_FQS ; +WRITE_ONCE (rsp->gp_flags, +READ_ONCE (rsp->gp_flags) & ~RCU_GP_FLAG_FQS );raw_spin_unlock_irq (&rnp->lock); }return fqs_state;
@@ -1901,7 +1902,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)struct rcu_data *rdp;struct rcu_node *rnp =rcu_get_root (rsp); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies);raw_spin_lock_irq (&rnp->lock);smp_mb__after_unlock_lock (); gp_duration = jiffies - rsp->gp_start;
@@ -1932,7 +1933,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)smp_mb__after_unlock_lock ();WARN_ON_ONCE (rcu_preempt_blocked_readers_cgp (rnp));WARN_ON_ONCE (rnp->qsmask); -ACCESS_ONCE (rnp->completed) = rsp->gpnum; +WRITE_ONCE (rnp->completed, rsp->gpnum); rdp =this_cpu_ptr (rsp->rda);if (rnp == rdp->mynode) needgp =__note_gp_changes (rsp, rnp, rdp) || needgp;
@@ -1940,7 +1941,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp) nocb +=rcu_future_gp_cleanup (rsp, rnp);raw_spin_unlock_irq (&rnp->lock);cond_resched_rcu_qs (); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies); } rnp =rcu_get_root (rsp);raw_spin_lock_irq (&rnp->lock);
@@ -1948,16 +1949,16 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)rcu_nocb_gp_set (rnp, nocb);/* Declare grace period done. */ -ACCESS_ONCE (rsp->completed) = rsp->gpnum; +WRITE_ONCE (rsp->completed, rsp->gpnum);trace_rcu_grace_period (rsp->name, rsp->completed,TPS ("end" )); rsp->fqs_state =RCU_GP_IDLE ; rdp =this_cpu_ptr (rsp->rda);/* Advance CBs to reduce false positives below. */ needgp =rcu_advance_cbs (rsp, rnp, rdp) || needgp;if (needgp ||cpu_needs_another_gp (rsp, rdp)) { -ACCESS_ONCE (rsp->gp_flags) =RCU_GP_FLAG_INIT ; +WRITE_ONCE (rsp->gp_flags,RCU_GP_FLAG_INIT );trace_rcu_grace_period (rsp->name, -ACCESS_ONCE (rsp->gpnum), +READ_ONCE (rsp->gpnum),TPS ("newreq" )); }raw_spin_unlock_irq (&rnp->lock);
@@ -1981,20 +1982,20 @@ static int __noreturn rcu_gp_kthread(void *arg)/* Handle grace-period start. */ for (;;) {trace_rcu_grace_period (rsp->name, -ACCESS_ONCE (rsp->gpnum), +READ_ONCE (rsp->gpnum),TPS ("reqwait" )); rsp->gp_state =RCU_GP_WAIT_GPS ;wait_event_interruptible (rsp->gp_wq, -ACCESS_ONCE (rsp->gp_flags) & +READ_ONCE (rsp->gp_flags) &RCU_GP_FLAG_INIT );/* Locking provides needed memory barrier. */ if (rcu_gp_init (rsp))break ;cond_resched_rcu_qs (); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies);WARN_ON (signal_pending (current));trace_rcu_grace_period (rsp->name, -ACCESS_ONCE (rsp->gpnum), +READ_ONCE (rsp->gpnum),TPS ("reqwaitsig" )); }
@@ -2010,39 +2011,39 @@ static int __noreturn rcu_gp_kthread(void *arg)if (!ret) rsp->jiffies_force_qs = jiffies + j;trace_rcu_grace_period (rsp->name, -ACCESS_ONCE (rsp->gpnum), +READ_ONCE (rsp->gpnum),TPS ("fqswait" )); rsp->gp_state =RCU_GP_WAIT_FQS ; ret =wait_event_interruptible_timeout (rsp->gp_wq, - ((gf =ACCESS_ONCE (rsp->gp_flags)) & + ((gf =READ_ONCE (rsp->gp_flags)) &RCU_GP_FLAG_FQS ) || - (!ACCESS_ONCE (rnp->qsmask) && + (!READ_ONCE (rnp->qsmask) && !rcu_preempt_blocked_readers_cgp (rnp)), j);/* Locking provides needed memory barriers. */ /* If grace period done, leave loop. */ -if (!ACCESS_ONCE (rnp->qsmask) && +if (!READ_ONCE (rnp->qsmask) && !rcu_preempt_blocked_readers_cgp (rnp))break ;/* If time for quiescent-state forcing, do it. */ if (ULONG_CMP_GE (jiffies, rsp->jiffies_force_qs) || (gf &RCU_GP_FLAG_FQS )) {trace_rcu_grace_period (rsp->name, -ACCESS_ONCE (rsp->gpnum), +READ_ONCE (rsp->gpnum),TPS ("fqsstart" )); fqs_state =rcu_gp_fqs (rsp, fqs_state);trace_rcu_grace_period (rsp->name, -ACCESS_ONCE (rsp->gpnum), +READ_ONCE (rsp->gpnum),TPS ("fqsend" ));cond_resched_rcu_qs (); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies); }else {/* Deal with stray signal. */ cond_resched_rcu_qs (); -ACCESS_ONCE (rsp->gp_activity) = jiffies; +WRITE_ONCE (rsp->gp_activity, jiffies);WARN_ON (signal_pending (current));trace_rcu_grace_period (rsp->name, -ACCESS_ONCE (rsp->gpnum), +READ_ONCE (rsp->gpnum),TPS ("fqswaitsig" )); } j = jiffies_till_next_fqs;
@@ -2084,8 +2085,8 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,*/ return false; } -ACCESS_ONCE (rsp->gp_flags) =RCU_GP_FLAG_INIT ; -trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum), +WRITE_ONCE (rsp->gp_flags,RCU_GP_FLAG_INIT ); +trace_rcu_grace_period (rsp->name,READ_ONCE (rsp->gpnum),TPS ("newreq" ));/*
@@ -2355,7 +2356,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp, rsp->qlen += rdp->qlen; rdp->n_cbs_orphaned += rdp->qlen; rdp->qlen_lazy =0 ; -ACCESS_ONCE (rdp->qlen) =0 ; +WRITE_ONCE (rdp->qlen,0 ); }/*
@@ -2567,7 +2568,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)/* If no callbacks are ready, just return. */ if (!cpu_has_callbacks_ready_to_invoke (rdp)) {trace_rcu_batch_start (rsp->name, rdp->qlen_lazy, rdp->qlen,0 ); -trace_rcu_batch_end (rsp->name,0 , !!ACCESS_ONCE (rdp->nxtlist), +trace_rcu_batch_end (rsp->name,0 , !!READ_ONCE (rdp->nxtlist),need_resched (),is_idle_task (current),rcu_is_callbacks_kthread ());return ;
@@ -2623,7 +2624,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) }smp_mb ();/* List handling before counting for rcu_barrier(). */ rdp->qlen_lazy -= count_lazy; -ACCESS_ONCE (rdp->qlen) = rdp->qlen - count; +WRITE_ONCE (rdp->qlen, rdp->qlen - count); rdp->n_cbs_invoked += count;/* Reinstate batch limit if we have worked down the excess. */
@@ -2780,7 +2781,7 @@ static void force_quiescent_state(struct rcu_state *rsp)/* Funnel through hierarchy to reduce memory contention. */ rnp =__this_cpu_read (rsp->rda->mynode);for (; rnp !=NULL ; rnp = rnp->parent) { - ret = (ACCESS_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) || + ret = (READ_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) || !raw_spin_trylock (&rnp->fqslock);if (rnp_old !=NULL )raw_spin_unlock (&rnp_old->fqslock);
@@ -2796,13 +2797,12 @@ static void force_quiescent_state(struct rcu_state *rsp)raw_spin_lock_irqsave (&rnp_old->lock, flags);smp_mb__after_unlock_lock ();raw_spin_unlock (&rnp_old->fqslock); -if (ACCESS_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) { +if (READ_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) { rsp->n_force_qs_lh++;raw_spin_unlock_irqrestore (&rnp_old->lock, flags);return ;/* Someone beat us to it. */ } -ACCESS_ONCE (rsp->gp_flags) = -ACCESS_ONCE (rsp->gp_flags) |RCU_GP_FLAG_FQS ; +WRITE_ONCE (rsp->gp_flags,READ_ONCE (rsp->gp_flags) |RCU_GP_FLAG_FQS );raw_spin_unlock_irqrestore (&rnp_old->lock, flags);rcu_gp_kthread_wake (rsp); }
@@ -2868,7 +2868,7 @@ static void rcu_process_callbacks(struct softirq_action *unused)*/ static void invoke_rcu_callbacks (struct rcu_state *rsp,struct rcu_data *rdp) { -if (unlikely (!ACCESS_ONCE (rcu_scheduler_fully_active))) +if (unlikely (!READ_ONCE (rcu_scheduler_fully_active)))return ;if (likely (!rsp->boost)) {rcu_do_batch (rsp, rdp);
@@ -2959,7 +2959,7 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),WARN_ON_ONCE ((unsigned long )head &0x1 );/* Misaligned rcu_head! */ if (debug_rcu_head_queue (head)) {/* Probable double call_rcu(), so leak the callback. */ -ACCESS_ONCE (head->func) = rcu_leak_callback; +WRITE_ONCE (head->func, rcu_leak_callback);WARN_ONCE (1 ,"__call_rcu(): Leaked duplicate callback\n" );return ; }
@@ -2998,7 +2998,7 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),if (!likely (rdp->nxtlist))init_default_callback_list (rdp); } -ACCESS_ONCE (rdp->qlen) = rdp->qlen +1 ; +WRITE_ONCE (rdp->qlen, rdp->qlen +1 );if (lazy) rdp->qlen_lazy++;else
@@ -3437,14 +3437,14 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp) }/* Has another RCU grace period completed? */ -if (ACCESS_ONCE (rnp->completed) != rdp->completed) {/* outside lock */ +if (READ_ONCE (rnp->completed) != rdp->completed) {/* outside lock */ rdp->n_rp_gp_completed++;return 1 ; }/* Has a new RCU grace period started? */ -if (ACCESS_ONCE (rnp->gpnum) != rdp->gpnum || -unlikely (ACCESS_ONCE (rdp->gpwrap))) {/* outside lock */ +if (READ_ONCE (rnp->gpnum) != rdp->gpnum || +unlikely (READ_ONCE (rdp->gpwrap))) {/* outside lock */ rdp->n_rp_gp_started++;return 1 ; }
@@ -3551,7 +3551,7 @@ static void _rcu_barrier(struct rcu_state *rsp) {int cpu;struct rcu_data *rdp; -unsigned long snap =ACCESS_ONCE (rsp->n_barrier_done); +unsigned long snap =READ_ONCE (rsp->n_barrier_done);unsigned long snap_done;_rcu_barrier_trace (rsp,"Begin" ,-1 , snap);
@@ -3593,10 +3593,10 @@ static void _rcu_barrier(struct rcu_state *rsp)/* * Increment ->n_barrier_done to avoid duplicate work. Use -* ACCESS_ONCE() to prevent the compiler from speculating +* WRITE_ONCE() to prevent the compiler from speculating * the increment to precede the early-exit check. */ -ACCESS_ONCE (rsp->n_barrier_done) = rsp->n_barrier_done +1 ; +WRITE_ONCE (rsp->n_barrier_done, rsp->n_barrier_done +1 );WARN_ON_ONCE ((rsp->n_barrier_done &0x1 ) !=1 );_rcu_barrier_trace (rsp,"Inc1" ,-1 , rsp->n_barrier_done);smp_mb ();/* Order ->n_barrier_done increment with below mechanism. */
@@ -3632,7 +3632,7 @@ static void _rcu_barrier(struct rcu_state *rsp)__call_rcu (&rdp->barrier_head, rcu_barrier_callback, rsp, cpu,0 ); } - }else if (ACCESS_ONCE (rdp->qlen)) { + }else if (READ_ONCE (rdp->qlen)) {_rcu_barrier_trace (rsp,"OnlineQ" , cpu, rsp->n_barrier_done);smp_call_function_single (cpu, rcu_barrier_func, rsp,1 );
@@ -3652,7 +3652,7 @@ static void _rcu_barrier(struct rcu_state *rsp)/* Increment ->n_barrier_done to prevent duplicate work. */ smp_mb ();/* Keep increment after above mechanism. */ -ACCESS_ONCE (rsp->n_barrier_done) = rsp->n_barrier_done +1 ; +WRITE_ONCE (rsp->n_barrier_done, rsp->n_barrier_done +1 );WARN_ON_ONCE ((rsp->n_barrier_done &0x1 ) !=0 );_rcu_barrier_trace (rsp,"Inc2" ,-1 , rsp->n_barrier_done);smp_mb ();/* Keep increment before caller's subsequent code. */
Signature · 1 line
1.8.1.5
PATCH v2 7/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in tree_plugin.h
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves kernel/rcu/tree_plugin.h from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- kernel/rcu/tree_plugin.h | 93 ++++++++++++++++++++++++------------------------ 1 file changed, 46 insertions(+), 47 deletions(-)
kernel/rcu/tree_plugin.h +46 -47
@@ -574,7 +574,7 @@ static int rcu_preempted_readers_exp(struct rcu_node *rnp)static int sync_rcu_preempt_exp_done (struct rcu_node *rnp) {return !rcu_preempted_readers_exp (rnp) && -ACCESS_ONCE (rnp->expmask) ==0 ; +READ_ONCE (rnp->expmask) ==0 ; }/*
@@ -720,7 +720,7 @@ void synchronize_rcu_expedited(void)int trycount =0 ;smp_mb ();/* Caller's modifications seen first by other CPUs. */ - snap =ACCESS_ONCE (sync_rcu_preempt_exp_count) +1 ; + snap =READ_ONCE (sync_rcu_preempt_exp_count) +1 ;smp_mb ();/* Above access cannot bleed into critical section. */ /*
@@ -744,7 +744,7 @@ void synchronize_rcu_expedited(void)*/ while (!mutex_trylock (&sync_rcu_preempt_exp_mutex)) {if (ULONG_CMP_LT (snap, -ACCESS_ONCE (sync_rcu_preempt_exp_count))) { +READ_ONCE (sync_rcu_preempt_exp_count))) {put_online_cpus (); goto mb_ret;/* Others did our work for us. */ }
@@ -756,7 +756,7 @@ void synchronize_rcu_expedited(void)return ; } } -if (ULONG_CMP_LT (snap,ACCESS_ONCE (sync_rcu_preempt_exp_count))) { +if (ULONG_CMP_LT (snap,READ_ONCE (sync_rcu_preempt_exp_count))) {put_online_cpus (); goto unlock_mb_ret;/* Others did our work for us. */ }
@@ -784,8 +784,7 @@ void synchronize_rcu_expedited(void)/* Clean up and exit. */ smp_mb ();/* ensure expedited GP seen before counter increment. */ -ACCESS_ONCE (sync_rcu_preempt_exp_count) = - sync_rcu_preempt_exp_count +1 ; +WRITE_ONCE (sync_rcu_preempt_exp_count, sync_rcu_preempt_exp_count +1 ); unlock_mb_ret:mutex_unlock (&sync_rcu_preempt_exp_mutex); mb_ret:
@@ -998,8 +997,8 @@ static int rcu_boost(struct rcu_node *rnp)struct task_struct *t;struct list_head *tb; -if (ACCESS_ONCE (rnp->exp_tasks) ==NULL && -ACCESS_ONCE (rnp->boost_tasks) ==NULL ) +if (READ_ONCE (rnp->exp_tasks) ==NULL && +READ_ONCE (rnp->boost_tasks) ==NULL )return 0 ;/* Nothing left to boost. */ raw_spin_lock_irqsave (&rnp->lock, flags);
@@ -1052,8 +1051,8 @@ static int rcu_boost(struct rcu_node *rnp)rt_mutex_lock (&rnp->boost_mtx);rt_mutex_unlock (&rnp->boost_mtx);/* Then keep lockdep happy. */ -return ACCESS_ONCE (rnp->exp_tasks) !=NULL || -ACCESS_ONCE (rnp->boost_tasks) !=NULL ; +return READ_ONCE (rnp->exp_tasks) !=NULL || +READ_ONCE (rnp->boost_tasks) !=NULL ; }/*
@@ -1466,7 +1465,7 @@ static bool __maybe_unused rcu_try_advance_all_cbs(void)* callbacks not yet ready to invoke. */ if ((rdp->completed != rnp->completed || -unlikely (ACCESS_ONCE (rdp->gpwrap))) && +unlikely (READ_ONCE (rdp->gpwrap))) && rdp->nxttail[RCU_DONE_TAIL ] != rdp->nxttail[RCU_NEXT_TAIL ])note_gp_changes (rsp, rdp);
@@ -1538,7 +1537,7 @@ static void rcu_prepare_for_idle(void)int tne;/* Handle nohz enablement switches conservatively. */ - tne =ACCESS_ONCE (tick_nohz_active); + tne =READ_ONCE (tick_nohz_active);if (tne != rdtp->tick_nohz_enabled_snap) {if (rcu_cpu_has_callbacks (NULL ))invoke_rcu_core ();/* force nohz to see update. */
@@ -1764,7 +1763,7 @@ static void print_cpu_stall_info(struct rcu_state *rsp, int cpu)atomic_read (&rdtp->dynticks) &0xfff , rdtp->dynticks_nesting, rdtp->dynticks_nmi_nesting, rdp->softirq_snap,kstat_softirqs_cpu (RCU_SOFTIRQ , cpu), -ACCESS_ONCE (rsp->n_force_qs) - rsp->n_force_qs_gpstart, +READ_ONCE (rsp->n_force_qs) - rsp->n_force_qs_gpstart, fast_no_hz); }
@@ -1902,11 +1901,11 @@ static void wake_nocb_leader(struct rcu_data *rdp, bool force) {struct rcu_data *rdp_leader = rdp->nocb_leader; -if (!ACCESS_ONCE (rdp_leader->nocb_kthread)) +if (!READ_ONCE (rdp_leader->nocb_kthread))return ; -if (ACCESS_ONCE (rdp_leader->nocb_leader_sleep) || force) { +if (READ_ONCE (rdp_leader->nocb_leader_sleep) || force) {/* Prior smp_mb__after_atomic() orders against prior enqueue. */ -ACCESS_ONCE (rdp_leader->nocb_leader_sleep) = false; +WRITE_ONCE (rdp_leader->nocb_leader_sleep, false);wake_up (&rdp_leader->nocb_wq); } }
@@ -1938,14 +1937,14 @@ static bool rcu_nocb_cpu_needs_barrier(struct rcu_state *rsp, int cpu) ret =atomic_long_read (&rdp->nocb_q_count);#ifdef CONFIG_PROVE_RCU - rhp =ACCESS_ONCE (rdp->nocb_head); + rhp =READ_ONCE (rdp->nocb_head);if (!rhp) - rhp =ACCESS_ONCE (rdp->nocb_gp_head); + rhp =READ_ONCE (rdp->nocb_gp_head);if (!rhp) - rhp =ACCESS_ONCE (rdp->nocb_follower_head); + rhp =READ_ONCE (rdp->nocb_follower_head);/* Having no rcuo kthread but CBs after scheduler starts is bad! */ -if (!ACCESS_ONCE (rdp->nocb_kthread) && rhp && +if (!READ_ONCE (rdp->nocb_kthread) && rhp && rcu_scheduler_fully_active) {/* RCU callback enqueued before CPU first came online??? */ pr_err("RCU: Never-onlined no-CBs CPU %d has CB %p\n",
@@ -1979,12 +1978,12 @@ static void __call_rcu_nocb_enqueue(struct rcu_data *rdp,atomic_long_add (rhcount, &rdp->nocb_q_count);/* rcu_barrier() relies on ->nocb_q_count add before xchg. */ old_rhpp =xchg (&rdp->nocb_tail, rhtp); -ACCESS_ONCE (*old_rhpp) = rhp; +WRITE_ONCE (*old_rhpp, rhp);atomic_long_add (rhcount_lazy, &rdp->nocb_q_count_lazy);smp_mb__after_atomic ();/* Store *old_rhpp before _wake test. */ /* If we are not being polled and there is a kthread, awaken it ... */ - t =ACCESS_ONCE (rdp->nocb_kthread); + t =READ_ONCE (rdp->nocb_kthread);if (rcu_nocb_poll || !t) {trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,TPS ("WakeNotPoll" ));
@@ -2122,7 +2121,7 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)for (;;) {wait_event_interruptible ( rnp->nocb_gp_wq[c &0x1 ], - (d =ULONG_CMP_GE (ACCESS_ONCE (rnp->completed), c))); + (d =ULONG_CMP_GE (READ_ONCE (rnp->completed), c)));if (likely (d))break ;WARN_ON (signal_pending (current));
@@ -2149,7 +2148,7 @@ wait_again:if (!rcu_nocb_poll) {trace_rcu_nocb_wake (my_rdp->rsp->name, my_rdp->cpu,"Sleep" );wait_event_interruptible (my_rdp->nocb_wq, - !ACCESS_ONCE (my_rdp->nocb_leader_sleep)); + !READ_ONCE (my_rdp->nocb_leader_sleep));/* Memory barrier handled by smp_mb() calls below and repoll. */ }else if (firsttime) { firsttime = false;/* Don't drown trace log with "Poll"! */
@@ -2163,12 +2162,12 @@ wait_again:*/ gotcbs = false;for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_follower) { - rdp->nocb_gp_head =ACCESS_ONCE (rdp->nocb_head); + rdp->nocb_gp_head =READ_ONCE (rdp->nocb_head);if (!rdp->nocb_gp_head)continue ;/* No CBs here, try next follower. */ /* Move callbacks to wait-for-GP list, which is empty. */ -ACCESS_ONCE (rdp->nocb_head) =NULL ; +WRITE_ONCE (rdp->nocb_head,NULL ); rdp->nocb_gp_tail =xchg (&rdp->nocb_tail, &rdp->nocb_head); gotcbs = true; }
@@ -2188,7 +2187,7 @@ wait_again: my_rdp->nocb_leader_sleep = true;smp_mb ();/* Ensure _sleep true before scan. */ for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_follower) -if (ACCESS_ONCE (rdp->nocb_head)) { +if (READ_ONCE (rdp->nocb_head)) {/* Found CB, so short-circuit next wait. */ my_rdp->nocb_leader_sleep = false;break ;
@@ -2209,7 +2208,7 @@ wait_again:/* Each pass through the following loop wakes a follower, if needed. */ for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_follower) { -if (ACCESS_ONCE (rdp->nocb_head)) +if (READ_ONCE (rdp->nocb_head)) my_rdp->nocb_leader_sleep = false;/* No need to sleep.*/ if (!rdp->nocb_gp_head)continue ;/* No CBs, so no need to wake follower. */
@@ -2245,7 +2244,7 @@ static void nocb_follower_wait(struct rcu_data *rdp)trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,"FollowerSleep" );wait_event_interruptible (rdp->nocb_wq, -ACCESS_ONCE (rdp->nocb_follower_head)); +READ_ONCE (rdp->nocb_follower_head)); }else if (firsttime) {/* Don't drown trace log with "Poll"! */ firsttime = false;
@@ -2286,10 +2285,10 @@ static int rcu_nocb_kthread(void *arg)nocb_follower_wait (rdp);/* Pull the ready-to-invoke callbacks onto local list. */ - list =ACCESS_ONCE (rdp->nocb_follower_head); + list =READ_ONCE (rdp->nocb_follower_head);BUG_ON (!list);trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,"WokeNonEmpty" ); -ACCESS_ONCE (rdp->nocb_follower_head) =NULL ; +WRITE_ONCE (rdp->nocb_follower_head,NULL ); tail =xchg (&rdp->nocb_follower_tail, &rdp->nocb_follower_head);/* Each pass through the following loop invokes a callback. */
@@ -2328,7 +2327,7 @@ static int rcu_nocb_kthread(void *arg)/* Is a deferred wakeup of rcu_nocb_kthread() required? */ static int rcu_nocb_need_deferred_wakeup (struct rcu_data *rdp) { -return ACCESS_ONCE (rdp->nocb_defer_wakeup); +return READ_ONCE (rdp->nocb_defer_wakeup); }/* Do a deferred wakeup of rcu_nocb_kthread(). */
@@ -2338,8 +2337,8 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)if (!rcu_nocb_need_deferred_wakeup (rdp))return ; - ndw =ACCESS_ONCE (rdp->nocb_defer_wakeup); -ACCESS_ONCE (rdp->nocb_defer_wakeup) =RCU_NOGP_WAKE_NOT ; + ndw =READ_ONCE (rdp->nocb_defer_wakeup); +WRITE_ONCE (rdp->nocb_defer_wakeup,RCU_NOGP_WAKE_NOT );wake_nocb_leader (rdp, ndw ==RCU_NOGP_WAKE_FORCE );trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,TPS ("DeferredWake" )); }
@@ -2452,7 +2451,7 @@ static void rcu_spawn_one_nocb_kthread(struct rcu_state *rsp, int cpu) t =kthread_run (rcu_nocb_kthread, rdp_spawn,"rcuo%c/%d" , rsp->abbr, cpu);BUG_ON (IS_ERR (t)); -ACCESS_ONCE (rdp_spawn->nocb_kthread) = t; +WRITE_ONCE (rdp_spawn->nocb_kthread, t); }/*
@@ -2667,7 +2666,7 @@ static void rcu_sysidle_enter(int irq)/* Record start of fully idle period. */ j = jiffies; -ACCESS_ONCE (rdtp->dynticks_idle_jiffies) = j; +WRITE_ONCE (rdtp->dynticks_idle_jiffies, j);smp_mb__before_atomic ();atomic_inc (&rdtp->dynticks_idle);smp_mb__after_atomic ();
@@ -2685,7 +2684,7 @@ static void rcu_sysidle_enter(int irq)*/ void rcu_sysidle_force_exit (void ) { -int oldstate =ACCESS_ONCE (full_sysidle_state); +int oldstate =READ_ONCE (full_sysidle_state);int newoldstate;/*
@@ -2798,7 +2797,7 @@ static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,smp_mb ();/* Read counters before timestamps. */ /* Pick up timestamps. */ - j =ACCESS_ONCE (rdtp->dynticks_idle_jiffies); + j =READ_ONCE (rdtp->dynticks_idle_jiffies);/* If this CPU entered idle more recently, update maxj timestamp. */ if (ULONG_CMP_LT (*maxj, j)) *maxj = j;
@@ -2835,11 +2834,11 @@ static unsigned long rcu_sysidle_delay(void)static void rcu_sysidle (unsigned long j) {/* Check the current state. */ -switch (ACCESS_ONCE (full_sysidle_state)) { +switch (READ_ONCE (full_sysidle_state)) {case RCU_SYSIDLE_NOT :/* First time all are idle, so note a short idle period. */ -ACCESS_ONCE (full_sysidle_state) =RCU_SYSIDLE_SHORT ; +WRITE_ONCE (full_sysidle_state,RCU_SYSIDLE_SHORT );break ;case RCU_SYSIDLE_SHORT :
@@ -2877,7 +2876,7 @@ static void rcu_sysidle_cancel(void) {smp_mb ();if (full_sysidle_state >RCU_SYSIDLE_SHORT ) -ACCESS_ONCE (full_sysidle_state) =RCU_SYSIDLE_NOT ; +WRITE_ONCE (full_sysidle_state,RCU_SYSIDLE_NOT ); }/*
@@ -2929,7 +2928,7 @@ static void rcu_sysidle_cb(struct rcu_head *rhp)smp_mb ();/* grace period precedes setting inuse. */ rshp = container_of(rhp, struct rcu_sysidle_head, rh); -ACCESS_ONCE (rshp->inuse) =0 ; +WRITE_ONCE (rshp->inuse,0 ); }/*
@@ -2940,7 +2939,7 @@ static void rcu_sysidle_cb(struct rcu_head *rhp)bool rcu_sys_is_idle (void ) {static struct rcu_sysidle_head rsh; -int rss =ACCESS_ONCE (full_sysidle_state); +int rss =READ_ONCE (full_sysidle_state);if (WARN_ON_ONCE (smp_processor_id () != tick_do_timer_cpu))return false;
@@ -2968,7 +2967,7 @@ bool rcu_sys_is_idle(void) }rcu_sysidle_report (rcu_state_p, isidle, maxj, false); oldrss = rss; - rss =ACCESS_ONCE (full_sysidle_state); + rss =READ_ONCE (full_sysidle_state); } }
@@ -3052,7 +3051,7 @@ static bool rcu_nohz_full_cpu(struct rcu_state *rsp)#ifdef CONFIG_NO_HZ_FULL if (tick_nohz_full_cpu (smp_processor_id ()) && (!rcu_gp_in_progress (rsp) || -ULONG_CMP_LT (jiffies,ACCESS_ONCE (rsp->gp_start) +HZ ))) +ULONG_CMP_LT (jiffies,READ_ONCE (rsp->gp_start) +HZ )))return 1 ;#endif /* #ifdef CONFIG_NO_HZ_FULL */ return 0 ;
@@ -3081,7 +3080,7 @@ static void rcu_bind_gp_kthread(void)static void rcu_dynticks_task_enter (void ) {#if defined(CONFIG_TASKS_RCU ) && defined(CONFIG_NO_HZ_FULL ) -ACCESS_ONCE (current->rcu_tasks_idle_cpu) =smp_processor_id (); +WRITE_ONCE (current->rcu_tasks_idle_cpu,smp_processor_id ());#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ }
@@ -3089,6 +3088,6 @@ static void rcu_dynticks_task_enter(void)static void rcu_dynticks_task_exit (void ) {#if defined(CONFIG_TASKS_RCU ) && defined(CONFIG_NO_HZ_FULL ) -ACCESS_ONCE (current->rcu_tasks_idle_cpu) =-1 ; +WRITE_ONCE (current->rcu_tasks_idle_cpu,-1 );#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ }
Signature · 1 line
1.8.1.5
PATCH v2 8/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in tree_trace.c
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves kernel/rcu/tree_trace.c from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- kernel/rcu/tree_trace.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/rcu/tree_trace.c +3 -3
@@ -277,7 +277,7 @@ static void print_one_rcu_state(struct seq_file *m, struct rcu_state *rsp)seq_printf (m,"nfqs=%lu/nfqsng=%lu(%lu) fqlh=%lu oqlen=%ld/%ld\n" , rsp->n_force_qs, rsp->n_force_qs_ngp, rsp->n_force_qs - rsp->n_force_qs_ngp, -ACCESS_ONCE (rsp->n_force_qs_lh), rsp->qlen_lazy, rsp->qlen); +READ_ONCE (rsp->n_force_qs_lh), rsp->qlen_lazy, rsp->qlen);for (rnp = &rsp->node[0 ]; rnp - &rsp->node[0 ] < rcu_num_nodes; rnp++) {if (rnp->level != level) {seq_puts (m,"\n" );
@@ -323,8 +323,8 @@ static void show_one_rcugp(struct seq_file *m, struct rcu_state *rsp)struct rcu_node *rnp = &rsp->node[0 ];raw_spin_lock_irqsave (&rnp->lock, flags); - completed =ACCESS_ONCE (rsp->completed); - gpnum =ACCESS_ONCE (rsp->gpnum); + completed =READ_ONCE (rsp->completed); + gpnum =READ_ONCE (rsp->gpnum);if (completed == gpnum) gpage =0 ;else
Signature · 1 line
1.8.1.5
PATCH v2 9/9 Convert ACCESS_ONCE() to READ_ONCE() and WRITE_ONCE() in update.c
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves kernel/rcu/update.c from the old ACCESS_ONCE() API to the new READ_ONCE() and WRITE_ONCE() APIs.
--- kernel/rcu/update.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
kernel/rcu/update.c +15 -15
@@ -150,14 +150,14 @@ void __rcu_read_unlock(void)barrier ();/* critical section before exit code. */ t->rcu_read_lock_nesting =INT_MIN ;barrier ();/* assign before ->rcu_read_unlock_special load */ -if (unlikely (ACCESS_ONCE (t->rcu_read_unlock_special.s))) +if (unlikely (READ_ONCE (t->rcu_read_unlock_special.s)))rcu_read_unlock_special (t);barrier ();/* ->rcu_read_unlock_special load before assign */ t->rcu_read_lock_nesting =0 ; }#ifdef CONFIG_PROVE_LOCKING { -int rrln =ACCESS_ONCE (t->rcu_read_lock_nesting); +int rrln =READ_ONCE (t->rcu_read_lock_nesting);WARN_ON_ONCE (rrln <0 && rrln >INT_MIN /2 ); }
@@ -389,17 +389,17 @@ module_param(rcu_cpu_stall_timeout, int, 0644);int rcu_jiffies_till_stall_check (void ) { -int till_stall_check =ACCESS_ONCE (rcu_cpu_stall_timeout); +int till_stall_check =READ_ONCE (rcu_cpu_stall_timeout);/* * Limit check must be consistent with the Kconfig limits * for CONFIG_RCU_CPU_STALL_TIMEOUT. */ if (till_stall_check <3 ) { -ACCESS_ONCE (rcu_cpu_stall_timeout) =3 ; +WRITE_ONCE (rcu_cpu_stall_timeout,3 ); till_stall_check =3 ; }else if (till_stall_check >300 ) { -ACCESS_ONCE (rcu_cpu_stall_timeout) =300 ; +WRITE_ONCE (rcu_cpu_stall_timeout,300 ); till_stall_check =300 ; }return till_stall_check *HZ +RCU_STALL_DELAY_DELTA ;
@@ -550,12 +550,12 @@ static void check_holdout_task(struct task_struct *t, {int cpu; -if (!ACCESS_ONCE (t->rcu_tasks_holdout) || - t->rcu_tasks_nvcsw !=ACCESS_ONCE (t->nvcsw) || - !ACCESS_ONCE (t->on_rq) || +if (!READ_ONCE (t->rcu_tasks_holdout) || + t->rcu_tasks_nvcsw !=READ_ONCE (t->nvcsw) || + !READ_ONCE (t->on_rq) || (IS_ENABLED (CONFIG_NO_HZ_FULL ) && !is_idle_task (t) && t->rcu_tasks_idle_cpu >=0 )) { -ACCESS_ONCE (t->rcu_tasks_holdout) = false; +WRITE_ONCE (t->rcu_tasks_holdout, false);list_del_init (&t->rcu_tasks_holdout_list);put_task_struct (t);return ;
@@ -639,11 +639,11 @@ static int __noreturn rcu_tasks_kthread(void *arg)*/ rcu_read_lock ();for_each_process_thread (g, t) { -if (t != current &&ACCESS_ONCE (t->on_rq) && +if (t != current &&READ_ONCE (t->on_rq) && !is_idle_task (t)) {get_task_struct (t); - t->rcu_tasks_nvcsw =ACCESS_ONCE (t->nvcsw); -ACCESS_ONCE (t->rcu_tasks_holdout) = true; + t->rcu_tasks_nvcsw =READ_ONCE (t->nvcsw); +WRITE_ONCE (t->rcu_tasks_holdout, true);list_add (&t->rcu_tasks_holdout_list, &rcu_tasks_holdouts); }
@@ -672,7 +672,7 @@ static int __noreturn rcu_tasks_kthread(void *arg)struct task_struct *t1;schedule_timeout_interruptible (HZ ); - rtst =ACCESS_ONCE (rcu_task_stall_timeout); + rtst =READ_ONCE (rcu_task_stall_timeout); needreport = rtst >0 &&time_after (jiffies, lastreport + rtst);if (needreport)
@@ -728,7 +728,7 @@ static void rcu_spawn_tasks_kthread(void)static struct task_struct *rcu_tasks_kthread_ptr;struct task_struct *t; -if (ACCESS_ONCE (rcu_tasks_kthread_ptr)) { +if (READ_ONCE (rcu_tasks_kthread_ptr)) {smp_mb ();/* Ensure caller sees full kthread. */ return ; }
@@ -740,7 +740,7 @@ static void rcu_spawn_tasks_kthread(void) t =kthread_run (rcu_tasks_kthread,NULL ,"rcu_tasks_kthread" );BUG_ON (IS_ERR (t));smp_mb ();/* Ensure others see full kthread. */ -ACCESS_ONCE (rcu_tasks_kthread_ptr) = t; +WRITE_ONCE (rcu_tasks_kthread_ptr, t);mutex_unlock (&rcu_tasks_kthread_mutex); }
Signature · 1 line
1.8.1.5
Steven Rostedt
reviewerQuoted 11 lines · On Mon, 16 Mar 2015 11:41:55 -0700 "Paul E. McKenney"…
Hello!
This patch series converts from
ACCESS_ONCE()to eitherREAD_ONCE()orWRITE_ONCE(), as appropriate. This permits better code generation and error checking, and is also more compatible with C11 directions. Each file is updated separately to simplify handling of git conflicts.Thanx, Paul
------------------------------------------------------------------------
Quoted 10 lines
include/linux/rculist.h | 6 - include/linux/rcupdate.h | 16 ++-- kernel/rcu/rcutorture.c | 2 kernel/rcu/srcu.c | 10 +- kernel/rcu/tiny_plugin.h | 12 +-- kernel/rcu/tree.c | 184 +++++++++++++++++++++++------------------------ kernel/rcu/tree_plugin.h | 93 +++++++++++------------ kernel/rcu/tree_trace.c | 6 - kernel/rcu/update.c | 30 +++---- 9 files changed, 180 insertions(+), 179 deletions(-)
Not sure why this was 9 patches. I think this could have easily been a single patch.
-- Steve
Quoted 25 lines · Steven Rostedt
Quoted 11 lines · On Mon, 16 Mar 2015 11:41:55 -0700 "Paul E. McKenney"…
Hello!
This patch series converts from
ACCESS_ONCE()to eitherREAD_ONCE()orWRITE_ONCE(), as appropriate. This permits better code generation and error checking, and is also more compatible with C11 directions. Each file is updated separately to simplify handling of git conflicts.Thanx, Paul
------------------------------------------------------------------------
Quoted 10 lines
include/linux/rculist.h | 6 - include/linux/rcupdate.h | 16 ++-- kernel/rcu/rcutorture.c | 2 kernel/rcu/srcu.c | 10 +- kernel/rcu/tiny_plugin.h | 12 +-- kernel/rcu/tree.c | 184 +++++++++++++++++++++++------------------------ kernel/rcu/tree_plugin.h | 93 +++++++++++------------ kernel/rcu/tree_trace.c | 6 - kernel/rcu/update.c | 30 +++---- 9 files changed, 180 insertions(+), 179 deletions(-)
Quoted 2 lines
Not sure why this was 9 patches. I think this could have easily been a single patch.
In order to make it easier to deal with conflicts and to move patches across this series as needed. I could consolidate them later on, just before they are merged, if that would help.
Thanx, Paul
Steven Rostedt
reviewerQuoted 6 lines · On Mon, 23 Mar 2015 08:25:07 -0700 "Paul E. McKenney"…
Quoted 2 lines
Not sure why this was 9 patches. I think this could have easily been a single patch.
In order to make it easier to deal with conflicts and to move patches across this series as needed. I could consolidate them later on, just before they are merged, if that would help.
No, just seemed odd to me. I usually do conversions in one big swoop if it stays in a single subsystem and architecture. Splitting a conversion by arch or subsystem makes sense. But these were all confined to generic RCU.
But if you are sprinkling these within another series, then I guess it does make sense.
-- Steve