Apply new rest conversion patches to /dev branch
Subsystems (6)
cover Apply new rest conversion patches to /dev branch
This series fixes the rcu/dev branch so it can apply the new ReST conversion patches.
Patches based on "00ec8f46465e rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs"
The easiest was to do this is to revert the patches that conflict and then applying the doc patches, and then applying them again. But in the re-application, we convert the documentation
No manual fix ups were done in this process, other than to documentation.
thanks,
- Joel
And in the process I learnt about get_user() and compiler barriers ;-)
Joel Fernandes (Google) (8):
Revert "rcu: Restore barrier() to rcu_read_lock() and rcu_read_unlock()"
Revert "rcu: Add support for consolidated-RCU reader checking" Revert "treewide: Rename rcu_dereference_raw_notrace() to _check()"
docs: rcu: Correct links referring to titles
docs: rcu: Increase toctree to 3
Revert "Revert "treewide: Rename rcu_dereference_raw_notrace() to _check()""
Revert "Revert "rcu: Add support for consolidated-RCU reader checking""
Revert "Revert "rcu: Restore barrier() to rcu_read_lock() and rcu_read_unlock()""
Mauro Carvalho Chehab (1):
docs: rcu: convert some articles from html to ReST
.../Data-Structures/Data-Structures.html | 1391 ------- .../Data-Structures/Data-Structures.rst | 1163 ++++++ .../Expedited-Grace-Periods.html | 668 ---- .../Expedited-Grace-Periods.rst | 521 +++ .../Memory-Ordering/Tree-RCU-Diagram.html | 9 - .../Tree-RCU-Memory-Ordering.html | 704 ---- .../Tree-RCU-Memory-Ordering.rst | 624 +++ .../RCU/Design/Requirements/Requirements.html | 3401 ----------------- .../RCU/Design/Requirements/Requirements.rst | 2704 +++++++++++++ Documentation/RCU/index.rst | 7 +- Documentation/RCU/whatisRCU.txt | 4 +- 11 files changed, 5020 insertions(+), 6176 deletions(-)
delete mode 100644 Documentation/RCU/Design/Data-Structures/Data-Structures.html create mode 100644 Documentation/RCU/Design/Data-Structures/Data-Structures.rst delete mode 100644 Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.html create mode 100644 Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst delete mode 100644 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Diagram.html delete mode 100644 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.html create mode 100644 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst delete mode 100644 Documentation/RCU/Design/Requirements/Requirements.html create mode 100644 Documentation/RCU/Design/Requirements/Requirements.rst
--
2.22.0.770.g0f2c4a37fd-goog
PATCH 1/9 Revert "rcu: Restore barrier() to rcu_read_lock() and rcu_read_unlock()"
This reverts commit d6b9cd7dc8e041ee83cb1362fce59a3cdb1f2709.
--- .../RCU/Design/Requirements/Requirements.html | 71 ------------------- kernel/rcu/tree_plugin.h | 11 +++ 2 files changed, 11 insertions(+), 71 deletions(-)
Documentation/RCU/Design/Requirements/Requirements.html +0 -71
@@ -2129,8 +2129,6 @@ Some of the relevant points of interest are as follows: <li> <a href="#Hotplug CPU">Hotplug CPU</a>. <li> <a href="#Scheduler and RCU">Scheduler and RCU</a>. <li> <a href="#Tracing and RCU">Tracing and RCU</a>. -<li> <a href="#Accesses to User Memory and RCU"> -Accesses to User Memory and RCU</a>. <li> <a href="#Energy Efficiency">Energy Efficiency</a>. <li> <a href="#Scheduling-Clock Interrupts and RCU"> Scheduling-Clock Interrupts and RCU</a>.
@@ -2523,75 +2521,6 @@ cannot be used. The tracing folks both located the requirement and provided the needed fix, so this surprise requirement was relatively painless. -<h3><a name="Accesses to User Memory and RCU"> -Accesses to User Memory and RCU</a></h3> - -<p> -The kernel needs to access user-space memory, for example, to access -data referenced by system-call parameters. -The <tt>get_user()</tt> macro does this job. - -<p> -However, user-space memory might well be paged out, which means -that <tt>get_user()</tt> might well page-fault and thus block while -waiting for the resulting I/O to complete. -It would be a very bad thing for the compiler to reorder -a <tt>get_user()</tt> invocation into an RCU read-side critical -section. -For example, suppose that the source code looked like this: - -<blockquote> -<pre> - 1 rcu_read_lock(); - 2 p = rcu_dereference(gp); - 3 v = p->value; - 4 rcu_read_unlock(); - 5 get_user(user_v, user_p); - 6 do_something_with(v, user_v); -</pre> -</blockquote> - -<p> -The compiler must not be permitted to transform this source code into -the following: - -<blockquote> -<pre> - 1 rcu_read_lock(); - 2 p = rcu_dereference(gp); - 3 get_user(user_v, user_p); // BUG: POSSIBLE PAGE FAULT!!! - 4 v = p->value; - 5 rcu_read_unlock(); - 6 do_something_with(v, user_v); -</pre> -</blockquote> - -<p> -If the compiler did make this transformation in a -<tt>CONFIG_PREEMPT=n</tt> kernel build, and if <tt>get_user()</tt> did -page fault, the result would be a quiescent state in the middle -of an RCU read-side critical section. -This misplaced quiescent state could result in line 4 being -a use-after-free access, which could be bad for your kernel's -actuarial statistics. -Similar examples can be constructed with the call to <tt>get_user()</tt> -preceding the <tt>rcu_read_lock()</tt>. - -<p> -Unfortunately, <tt>get_user()</tt> doesn't have any particular -ordering properties, and in some architectures the underlying <tt>asm</tt> -isn't even marked <tt>volatile</tt>. -And even if it was marked <tt>volatile</tt>, the above access to -<tt>p->value</tt> is not volatile, so the compiler would not have any -reason to keep those two accesses in order. - -<p> -Therefore, the Linux-kernel definitions of <tt>rcu_read_lock()</tt> -and <tt>rcu_read_unlock()</tt> must act as compiler barriers, -at least for outermost instances of <tt>rcu_read_lock()</tt> and -<tt>rcu_read_unlock()</tt> within a nested set of RCU read-side critical -sections. - <h3><a name="Energy Efficiency">Energy Efficiency</a></h3> <p>
kernel/rcu/tree_plugin.h +11 -0
@@ -288,6 +288,7 @@ void rcu_note_context_switch(bool preempt)struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp; +barrier ();/* Avoid RCU read-side critical sections leaking down. */ trace_rcu_utilization (TPS ("Start context switch" ));lockdep_assert_irqs_disabled ();WARN_ON_ONCE (!preempt && t->rcu_read_lock_nesting >0 );
@@ -330,6 +331,7 @@ void rcu_note_context_switch(bool preempt)if (rdp->exp_deferred_qs)rcu_report_exp_rdp (rdp);trace_rcu_utilization (TPS ("End context switch" )); +barrier ();/* Avoid RCU read-side critical sections leaking up. */ }EXPORT_SYMBOL_GPL (rcu_note_context_switch);
@@ -813,6 +815,11 @@ static void rcu_qs(void)* dyntick-idle quiescent state visible to other CPUs, which will in * some cases serve for expedited as well as normal grace periods. * Either way, register a lightweight quiescent state. +* +* The barrier() calls are redundant in the common case when this is +* called externally, but just in case this is called from within this +* file. +* */ void rcu_all_qs (void ) {
@@ -827,12 +834,14 @@ void rcu_all_qs(void)return ; }this_cpu_write (rcu_data.rcu_urgent_qs, false); +barrier ();/* Avoid RCU read-side critical sections leaking down. */ if (unlikely (raw_cpu_read (rcu_data.rcu_need_heavy_qs))) {local_irq_save (flags);rcu_momentary_dyntick_idle ();local_irq_restore (flags); }rcu_qs (); +barrier ();/* Avoid RCU read-side critical sections leaking up. */ preempt_enable (); }EXPORT_SYMBOL_GPL (rcu_all_qs);
@@ -842,6 +851,7 @@ EXPORT_SYMBOL_GPL(rcu_all_qs);*/ void rcu_note_context_switch (bool preempt) { +barrier ();/* Avoid RCU read-side critical sections leaking down. */ trace_rcu_utilization (TPS ("Start context switch" ));rcu_qs ();/* Load rcu_urgent_qs before other flags. */
@@ -854,6 +864,7 @@ void rcu_note_context_switch(bool preempt)rcu_tasks_qs (current); out:trace_rcu_utilization (TPS ("End context switch" )); +barrier ();/* Avoid RCU read-side critical sections leaking up. */ }EXPORT_SYMBOL_GPL (rcu_note_context_switch);
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
PATCH 2/9 Revert "rcu: Add support for consolidated-RCU reader checking"
In reply to Joel Fernandes (Google)This reverts commit 50ad3f1f9b13c8a6f2ae79df4cecb2c21da1c7c8.
--- include/linux/rculist.h | 32 ++++---------------- include/linux/rcupdate.h | 7 ----- kernel/rcu/Kconfig.debug | 11 ------- kernel/rcu/update.c | 65 ++++++++++++++-------------------------- 4 files changed, 27 insertions(+), 88 deletions(-)
include/linux/rculist.h +5 -27
@@ -40,24 +40,6 @@ static inline void INIT_LIST_HEAD_RCU(struct list_head *list)*/ #define list_next_rcu (list) (*((struct list_head __rcu **)(&(list)->next))) -/* -* Check during list traversal that we are within an RCU reader -*/ - -#define check_arg_count_one (dummy) - -#ifdef CONFIG_PROVE_RCU_LIST -#define __list_check_rcu(dummy, cond, extra...) \ - ({ \ - check_arg_count_one(extra); \ - RCU_LOCKDEP_WARN(!cond && !rcu_read_lock_any_held(), \ - "RCU-list traversed in non-reader section!"); \ - }) -#else -#define __list_check_rcu(dummy, cond, extra...) \ - ({ check_arg_count_one(extra); }) -#endif -/* * Insert a new entry between two known consecutive entries. *
@@ -361,16 +343,14 @@ static inline void list_splice_tail_init_rcu(struct list_head *list,* @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the list_head within the struct. -* @cond: optional lockdep expression if called from non-RCU protection. * * This list-traversal primitive may safely run concurrently with * the _rcu list-mutation primitives such as list_add_rcu() * as long as the traversal is guarded by rcu_read_lock(). */ -#define list_for_each_entry_rcu(pos, head, member, cond...) \ - for (__list_check_rcu(dummy, ## cond, 0), \ - pos = list_entry_rcu((head)->next, typeof(*pos), member); \ - &pos->member != (head); \ +#define list_for_each_entry_rcu (pos, head, member) \ + for (pos = list_entry_rcu((head)->next, typeof(*pos), member); \ + &pos->member != (head); \ pos = list_entry_rcu(pos->member.next, typeof(*pos), member))/**
@@ -636,15 +616,13 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the hlist_node within the struct. -* @cond: optional lockdep expression if called from non-RCU protection. * * This list-traversal primitive may safely run concurrently with * the _rcu list-mutation primitives such as hlist_add_head_rcu() * as long as the traversal is guarded by rcu_read_lock(). */ -#define hlist_for_each_entry_rcu(pos, head, member, cond...) \ - for (__list_check_rcu(dummy, ## cond, 0), \ - pos = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)),\ +#define hlist_for_each_entry_rcu (pos, head, member) \ + for (pos = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)),\ typeof(*(pos)), member); \ pos; \ pos = hlist_entry_safe(rcu_dereference_raw(hlist_next_rcu(\
include/linux/rcupdate.h +0 -7
@@ -221,7 +221,6 @@ int debug_lockdep_rcu_enabled(void);int rcu_read_lock_held (void );int rcu_read_lock_bh_held (void );int rcu_read_lock_sched_held (void ); -int rcu_read_lock_any_held (void );#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
@@ -242,12 +241,6 @@ static inline int rcu_read_lock_sched_held(void) {return !preemptible (); } - -static inline int rcu_read_lock_any_held (void ) -{ -return !preemptible (); -} -#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ #ifdef CONFIG_PROVE_RCU
kernel/rcu/Kconfig.debug +0 -11
@@ -8,17 +8,6 @@ menu "RCU Debugging" config PROVE_RCU def_bool PROVE_LOCKING -config PROVE_RCU_LIST - bool "RCU list lockdep debugging" - depends on PROVE_RCU && RCU_EXPERT - default n - help - Enable RCU lockdep checking for list usages. By default it is - turned off since there are several list RCU users that still - need to be converted to pass a lockdep expression. To prevent - false-positive splats, we keep it default disabled but once all - users are converted, we can remove this config option. - config TORTURE_TEST tristate default n
kernel/rcu/update.c +22 -43
90 lines not on this page · raw message
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
PATCH 3/9 Revert "treewide: Rename rcu_dereference_raw_notrace() to _check()"
In reply to Joel Fernandes (Google)This reverts commit 355e9972da81e803bbb825b76106ae9b358caf8e.
--- Documentation/RCU/Design/Requirements/Requirements.html | 2 +- arch/powerpc/include/asm/kvm_book3s_64.h | 2 +- include/linux/rculist.h | 6 +++--- include/linux/rcupdate.h | 2 +- kernel/trace/ftrace_internal.h | 8 ++++---- kernel/trace/trace.c | 4 ++-- 6 files changed, 12 insertions(+), 12 deletions(-)
Documentation/RCU/Design/Requirements/Requirements.html +1 -1
@@ -2512,7 +2512,7 @@ disabled across the entire RCU read-side critical section. <p> It is possible to use tracing on RCU code, but tracing itself uses RCU. -For this reason, <tt>rcu_dereference_raw_check()</tt> +For this reason, <tt>rcu_dereference_raw_notrace()</tt> is provided for use by tracing, which avoids the destructive recursion that could otherwise ensue. This API is also used by virtualization in some architectures,
arch/powerpc/include/asm/kvm_book3s_64.h +1 -1
@@ -535,7 +535,7 @@ static inline void note_hpte_modification(struct kvm *kvm,*/ static inline struct kvm_memslots *kvm_memslots_raw (struct kvm *kvm) { -return rcu_dereference_raw_check (kvm->memslots[0 ]); +return rcu_dereference_raw_notrace (kvm->memslots[0 ]); }extern void kvmppc_mmu_debugfs_init (struct kvm *kvm);
include/linux/rculist.h +3 -3
@@ -622,7 +622,7 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* as long as the traversal is guarded by rcu_read_lock(). */ #define hlist_for_each_entry_rcu (pos, head, member) \ - for (pos = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)),\ + for (pos = hlist_entry_safe (rcu_dereference_raw(hlist_first_rcu(head)),\ typeof(*(pos)), member); \ pos; \ pos = hlist_entry_safe(rcu_dereference_raw(hlist_next_rcu(\
@@ -642,10 +642,10 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* not do any RCU debugging or tracing. */ #define hlist_for_each_entry_rcu_notrace (pos, head, member) \ - for (pos = hlist_entry_safe(rcu_dereference_raw_check(hlist_first_rcu(head)),\ + for (pos = hlist_entry_safe (rcu_dereference_raw_notrace(hlist_first_rcu(head)),\ typeof(*(pos)), member); \ pos; \ - pos = hlist_entry_safe(rcu_dereference_raw_check(hlist_next_rcu(\ + pos = hlist_entry_safe(rcu_dereference_raw_notrace(hlist_next_rcu(\ &(pos)->member)), typeof(*(pos)), member))/**
include/linux/rcupdate.h +1 -1
@@ -476,7 +476,7 @@ do { \* The no-tracing version of rcu_dereference_raw() must not call * rcu_read_lock_held(). */ -#define rcu_dereference_raw_check (p) __rcu_dereference_check((p), 1, __rcu) +#define rcu_dereference_raw_notrace (p) __rcu_dereference_check((p), 1, __rcu)/** * rcu_dereference_protected() - fetch RCU pointer when updates prevented
kernel/trace/ftrace_internal.h +4 -4
@@ -6,22 +6,22 @@/* * Traverse the ftrace_global_list, invoking all entries. The reason that we -* can use rcu_dereference_raw_check() is that elements removed from this list +* can use rcu_dereference_raw_notrace() is that elements removed from this list * are simply leaked, so there is no need to interact with a grace-period -* mechanism. The rcu_dereference_raw_check() calls are needed to handle +* mechanism. The rcu_dereference_raw_notrace() calls are needed to handle * concurrent insertions into the ftrace_global_list. * * Silly Alpha and silly pointer-speculation compiler optimizations! */ #define do_for_each_ftrace_op (op, list) \ - op = rcu_dereference_raw_check(list); \ + op = rcu_dereference_raw_notrace(list); \ do/* * Optimized for just a single item in the list (as that is the normal case). */ #define while_for_each_ftrace_op (op) \ - while (likely(op = rcu_dereference_raw_check((op)->next)) && \ + while (likely(op = rcu_dereference_raw_notrace((op)->next)) && \ unlikely((op) != &ftrace_list_end))extern struct ftrace_ops __rcu *ftrace_ops_list;
kernel/trace/trace.c +2 -2
@@ -2642,10 +2642,10 @@ static void ftrace_exports(struct ring_buffer_event *event)preempt_disable_notrace (); - export =rcu_dereference_raw_check (ftrace_exports_list); + export =rcu_dereference_raw_notrace (ftrace_exports_list);while (export) {trace_process_export (export, event); - export =rcu_dereference_raw_check (export->next); + export =rcu_dereference_raw_notrace (export->next); }preempt_enable_notrace ();
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
PATCH 5/9 docs: rcu: Correct links referring to titles
In reply to Joel Fernandes (Google)Mauro's auto conversion broken these links, fix them.
--- .../Tree-RCU-Memory-Ordering.rst | 17 ++-- .../RCU/Design/Requirements/Requirements.rst | 90 ++++++++----------- 2 files changed, 47 insertions(+), 60 deletions(-)
Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst +8 -9
@@ -230,15 +230,14 @@ Tree RCU Grace Period Memory Ordering Components Tree RCU's grace-period memory-ordering guarantee is provided by a number of RCU components: -#. `Callback Registry <#Callback%20Registry>`__ -#. `Grace-Period Initialization <#Grace-Period%20Initialization>`__ -#. `Self-Reported Quiescent - States <#Self-Reported%20Quiescent%20States>`__ -#. `Dynamic Tick Interface <#Dynamic%20Tick%20Interface>`__ -#. `CPU-Hotplug Interface <#CPU-Hotplug%20Interface>`__ -#. `Forcing Quiescent States <Forcing%20Quiescent%20States>`__ -#. `Grace-Period Cleanup <Grace-Period%20Cleanup>`__ -#. `Callback Invocation <Callback%20Invocation>`__ +#. `Callback Registry`_ +#. `Grace-Period Initialization`_ +#. `Self-Reported Quiescent States`_ +#. `Dynamic Tick Interface`_ +#. `CPU-Hotplug Interface`_ +#. `Forcing Quiescent States`_ +#. `Grace-Period Cleanup`_ +#. `Callback Invocation`_ Each of the following section looks at the corresponding component in detail.
Documentation/RCU/Design/Requirements/Requirements.rst +39 -51
126 lines not on this page · raw message
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
PATCH 6/9 docs: rcu: Increase toctree to 3
In reply to Joel Fernandes (Google)These documents are long and have various sections. Provide a good toc nesting level.
--- Documentation/RCU/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/RCU/index.rst +1 -1
@@ -5,7 +5,7 @@ RCU concepts ============ .. toctree:: - :maxdepth: 1 + :maxdepth: 3 rcu listRCU
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
PATCH 7/9 Revert "Revert "treewide: Rename rcu_dereference_raw_notrace() to _check()""
In reply to Joel Fernandes (Google)This reverts commit 61d814760f1d2dffdc8db636f70bbef07c30acd5.
--- Documentation/RCU/Design/Requirements/Requirements.rst | 2 +- arch/powerpc/include/asm/kvm_book3s_64.h | 2 +- include/linux/rculist.h | 6 +++--- include/linux/rcupdate.h | 2 +- kernel/trace/ftrace_internal.h | 8 ++++---- kernel/trace/trace.c | 4 ++-- 6 files changed, 12 insertions(+), 12 deletions(-)
Documentation/RCU/Design/Requirements/Requirements.rst +1 -1
@@ -1997,7 +1997,7 @@ Tracing and RCU ~~~~~~~~~~~~~~~ It is possible to use tracing on RCU code, but tracing itself uses RCU. -For this reason, ``rcu_dereference_raw_notrace()`` is provided for use +For this reason, ``rcu_dereference_raw_check()`` is provided for use by tracing, which avoids the destructive recursion that could otherwise ensue. This API is also used by virtualization in some architectures, where RCU readers execute in environments in which tracing cannot be
arch/powerpc/include/asm/kvm_book3s_64.h +1 -1
@@ -535,7 +535,7 @@ static inline void note_hpte_modification(struct kvm *kvm,*/ static inline struct kvm_memslots *kvm_memslots_raw (struct kvm *kvm) { -return rcu_dereference_raw_notrace (kvm->memslots[0 ]); +return rcu_dereference_raw_check (kvm->memslots[0 ]); }extern void kvmppc_mmu_debugfs_init (struct kvm *kvm);
include/linux/rculist.h +3 -3
@@ -622,7 +622,7 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* as long as the traversal is guarded by rcu_read_lock(). */ #define hlist_for_each_entry_rcu (pos, head, member) \ - for (pos = hlist_entry_safe (rcu_dereference_raw(hlist_first_rcu(head)),\ + for (pos = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)),\ typeof(*(pos)), member); \ pos; \ pos = hlist_entry_safe(rcu_dereference_raw(hlist_next_rcu(\
@@ -642,10 +642,10 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* not do any RCU debugging or tracing. */ #define hlist_for_each_entry_rcu_notrace (pos, head, member) \ - for (pos = hlist_entry_safe (rcu_dereference_raw_notrace(hlist_first_rcu(head)),\ + for (pos = hlist_entry_safe(rcu_dereference_raw_check(hlist_first_rcu(head)),\ typeof(*(pos)), member); \ pos; \ - pos = hlist_entry_safe(rcu_dereference_raw_notrace(hlist_next_rcu(\ + pos = hlist_entry_safe(rcu_dereference_raw_check(hlist_next_rcu(\ &(pos)->member)), typeof(*(pos)), member))/**
include/linux/rcupdate.h +1 -1
@@ -476,7 +476,7 @@ do { \* The no-tracing version of rcu_dereference_raw() must not call * rcu_read_lock_held(). */ -#define rcu_dereference_raw_notrace (p) __rcu_dereference_check((p), 1, __rcu) +#define rcu_dereference_raw_check (p) __rcu_dereference_check((p), 1, __rcu)/** * rcu_dereference_protected() - fetch RCU pointer when updates prevented
kernel/trace/ftrace_internal.h +4 -4
@@ -6,22 +6,22 @@/* * Traverse the ftrace_global_list, invoking all entries. The reason that we -* can use rcu_dereference_raw_notrace() is that elements removed from this list +* can use rcu_dereference_raw_check() is that elements removed from this list * are simply leaked, so there is no need to interact with a grace-period -* mechanism. The rcu_dereference_raw_notrace() calls are needed to handle +* mechanism. The rcu_dereference_raw_check() calls are needed to handle * concurrent insertions into the ftrace_global_list. * * Silly Alpha and silly pointer-speculation compiler optimizations! */ #define do_for_each_ftrace_op (op, list) \ - op = rcu_dereference_raw_notrace(list); \ + op = rcu_dereference_raw_check(list); \ do/* * Optimized for just a single item in the list (as that is the normal case). */ #define while_for_each_ftrace_op (op) \ - while (likely(op = rcu_dereference_raw_notrace((op)->next)) && \ + while (likely(op = rcu_dereference_raw_check((op)->next)) && \ unlikely((op) != &ftrace_list_end))extern struct ftrace_ops __rcu *ftrace_ops_list;
kernel/trace/trace.c +2 -2
@@ -2642,10 +2642,10 @@ static void ftrace_exports(struct ring_buffer_event *event)preempt_disable_notrace (); - export =rcu_dereference_raw_notrace (ftrace_exports_list); + export =rcu_dereference_raw_check (ftrace_exports_list);while (export) {trace_process_export (export, event); - export =rcu_dereference_raw_notrace (export->next); + export =rcu_dereference_raw_check (export->next); }preempt_enable_notrace ();
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
PATCH 8/9 Revert "Revert "rcu: Add support for consolidated-RCU reader checking""
In reply to Joel Fernandes (Google)This reverts commit 24be1727c524b5874d5dc7828cd392cf86c3341e.
--- include/linux/rculist.h | 32 ++++++++++++++++---- include/linux/rcupdate.h | 7 +++++ kernel/rcu/Kconfig.debug | 11 +++++++ kernel/rcu/update.c | 65 ++++++++++++++++++++++++++-------------- 4 files changed, 88 insertions(+), 27 deletions(-)
include/linux/rculist.h +27 -5
@@ -40,6 +40,24 @@ static inline void INIT_LIST_HEAD_RCU(struct list_head *list)*/ #define list_next_rcu (list) (*((struct list_head __rcu **)(&(list)->next))) +/* +* Check during list traversal that we are within an RCU reader +*/ + +#define check_arg_count_one (dummy) + +#ifdef CONFIG_PROVE_RCU_LIST +#define __list_check_rcu(dummy, cond, extra...) \ + ({ \ + check_arg_count_one(extra); \ + RCU_LOCKDEP_WARN(!cond && !rcu_read_lock_any_held(), \ + "RCU-list traversed in non-reader section!"); \ + }) +#else +#define __list_check_rcu(dummy, cond, extra...) \ + ({ check_arg_count_one(extra); }) +#endif +/* * Insert a new entry between two known consecutive entries. *
@@ -343,14 +361,16 @@ static inline void list_splice_tail_init_rcu(struct list_head *list,* @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the list_head within the struct. +* @cond: optional lockdep expression if called from non-RCU protection. * * This list-traversal primitive may safely run concurrently with * the _rcu list-mutation primitives such as list_add_rcu() * as long as the traversal is guarded by rcu_read_lock(). */ -#define list_for_each_entry_rcu (pos, head, member) \ - for (pos = list_entry_rcu((head)->next, typeof(*pos), member); \ - &pos->member != (head); \ +#define list_for_each_entry_rcu(pos, head, member, cond...) \ + for (__list_check_rcu(dummy, ## cond, 0), \ + pos = list_entry_rcu((head)->next, typeof(*pos), member); \ + &pos->member != (head); \ pos = list_entry_rcu(pos->member.next, typeof(*pos), member))/**
@@ -616,13 +636,15 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n,* @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the hlist_node within the struct. +* @cond: optional lockdep expression if called from non-RCU protection. * * This list-traversal primitive may safely run concurrently with * the _rcu list-mutation primitives such as hlist_add_head_rcu() * as long as the traversal is guarded by rcu_read_lock(). */ -#define hlist_for_each_entry_rcu (pos, head, member) \ - for (pos = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)),\ +#define hlist_for_each_entry_rcu(pos, head, member, cond...) \ + for (__list_check_rcu(dummy, ## cond, 0), \ + pos = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(head)),\ typeof(*(pos)), member); \ pos; \ pos = hlist_entry_safe(rcu_dereference_raw(hlist_next_rcu(\
include/linux/rcupdate.h +7 -0
@@ -221,6 +221,7 @@ int debug_lockdep_rcu_enabled(void);int rcu_read_lock_held (void );int rcu_read_lock_bh_held (void );int rcu_read_lock_sched_held (void ); +int rcu_read_lock_any_held (void );#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
@@ -241,6 +242,12 @@ static inline int rcu_read_lock_sched_held(void) {return !preemptible (); } + +static inline int rcu_read_lock_any_held (void ) +{ +return !preemptible (); +} +#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ #ifdef CONFIG_PROVE_RCU
kernel/rcu/Kconfig.debug +11 -0
@@ -8,6 +8,17 @@ menu "RCU Debugging" config PROVE_RCU def_bool PROVE_LOCKING +config PROVE_RCU_LIST + bool "RCU list lockdep debugging" + depends on PROVE_RCU && RCU_EXPERT + default n + help + Enable RCU lockdep checking for list usages. By default it is + turned off since there are several list RCU users that still + need to be converted to pass a lockdep expression. To prevent + false-positive splats, we keep it default disabled but once all + users are converted, we can remove this config option. + config TORTURE_TEST tristate default n
kernel/rcu/update.c +43 -22
90 lines not on this page · raw message
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
PATCH 9/9 Revert "Revert "rcu: Restore barrier() to rcu_read_lock() and rcu_read_unlock()""
In reply to Joel Fernandes (Google)This reverts commit 43ddb98ebe7171ff1c6e11c1616fd03726d8e9bf while adding the documentation that the original commit added but in ReST format.
--- .../RCU/Design/Requirements/Requirements.rst | 54 +++++++++++++++++++ kernel/rcu/tree_plugin.h | 11 ---- 2 files changed, 54 insertions(+), 11 deletions(-)
Documentation/RCU/Design/Requirements/Requirements.rst +54 -0
@@ -1691,6 +1691,7 @@ follows: #. `Hotplug CPU`_ #. `Scheduler and RCU`_ #. `Tracing and RCU`_ +#. `Accesses to User Memory and RCU`_ #. `Energy Efficiency`_ #. `Scheduling-Clock Interrupts and RCU`_ #. `Memory Efficiency`_
@@ -2004,6 +2005,59 @@ where RCU readers execute in environments in which tracing cannot be used. The tracing folks both located the requirement and provided the needed fix, so this surprise requirement was relatively painless. +Accesses to User Memory and RCU +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The kernel needs to access user-space memory, for example, to access data +referenced by system-call parameters. The ``get_user()`` macro does this job. + +However, user-space memory might well be paged out, which means that +``get_user()`` might well page-fault and thus block while waiting for the +resulting I/O to complete. It would be a very bad thing for the compiler to +reorder a ``get_user()`` invocation into an RCU read-side critical section. + +For example, suppose that the source code looked like this: + + :: + + 1 rcu_read_lock(); + 2 p = rcu_dereference(gp); + 3 v = p->value; + 4 rcu_read_unlock(); + 5 get_user(user_v, user_p); + 6 do_something_with(v, user_v); + +The compiler must not be permitted to transform this source code into +the following: + + :: + + 1 rcu_read_lock(); + 2 p = rcu_dereference(gp); + 3 get_user(user_v, user_p); // BUG: POSSIBLE PAGE FAULT!!! + 4 v = p->value; + 5 rcu_read_unlock(); + 6 do_something_with(v, user_v); + +If the compiler did make this transformation in a ``CONFIG_PREEMPT=n`` kernel +build, and if ``get_user()`` did page fault, the result would be a quiescent +state in the middle of an RCU read-side critical section. This misplaced +quiescent state could result in line 4 being a use-after-free access, +which could be bad for your kernel's actuarial statistics. Similar examples +can be constructed with the call to ``get_user()`` preceding the +``rcu_read_lock()``. + +Unfortunately, ``get_user()`` doesn't have any particular ordering properties, +and in some architectures the underlying ``asm`` isn't even marked +``volatile``. And even if it was marked ``volatile``, the above access to +``p->value`` is not volatile, so the compiler would not have any reason to keep +those two accesses in order. + +Therefore, the Linux-kernel definitions of ``rcu_read_lock()`` and +``rcu_read_unlock()`` must act as compiler barriers, at least for outermost +instances of ``rcu_read_lock()`` and ``rcu_read_unlock()`` within a nested set +of RCU read-side critical sections. + Energy Efficiency ~~~~~~~~~~~~~~~~~
kernel/rcu/tree_plugin.h +0 -11
@@ -288,7 +288,6 @@ void rcu_note_context_switch(bool preempt)struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp; -barrier ();/* Avoid RCU read-side critical sections leaking down. */ trace_rcu_utilization (TPS ("Start context switch" ));lockdep_assert_irqs_disabled ();WARN_ON_ONCE (!preempt && t->rcu_read_lock_nesting >0 );
@@ -331,7 +330,6 @@ void rcu_note_context_switch(bool preempt)if (rdp->exp_deferred_qs)rcu_report_exp_rdp (rdp);trace_rcu_utilization (TPS ("End context switch" )); -barrier ();/* Avoid RCU read-side critical sections leaking up. */ }EXPORT_SYMBOL_GPL (rcu_note_context_switch);
@@ -815,11 +813,6 @@ static void rcu_qs(void)* dyntick-idle quiescent state visible to other CPUs, which will in * some cases serve for expedited as well as normal grace periods. * Either way, register a lightweight quiescent state. -* -* The barrier() calls are redundant in the common case when this is -* called externally, but just in case this is called from within this -* file. -* */ void rcu_all_qs (void ) {
@@ -834,14 +827,12 @@ void rcu_all_qs(void)return ; }this_cpu_write (rcu_data.rcu_urgent_qs, false); -barrier ();/* Avoid RCU read-side critical sections leaking down. */ if (unlikely (raw_cpu_read (rcu_data.rcu_need_heavy_qs))) {local_irq_save (flags);rcu_momentary_dyntick_idle ();local_irq_restore (flags); }rcu_qs (); -barrier ();/* Avoid RCU read-side critical sections leaking up. */ preempt_enable (); }EXPORT_SYMBOL_GPL (rcu_all_qs);
@@ -851,7 +842,6 @@ EXPORT_SYMBOL_GPL(rcu_all_qs);*/ void rcu_note_context_switch (bool preempt) { -barrier ();/* Avoid RCU read-side critical sections leaking down. */ trace_rcu_utilization (TPS ("Start context switch" ));rcu_qs ();/* Load rcu_urgent_qs before other flags. */
@@ -864,7 +854,6 @@ void rcu_note_context_switch(bool preempt)rcu_tasks_qs (current); out:trace_rcu_utilization (TPS ("End context switch" )); -barrier ();/* Avoid RCU read-side critical sections leaking up. */ }EXPORT_SYMBOL_GPL (rcu_note_context_switch);
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
Quoted 2 lines · Joel Fernandes (Google)
This series fixes the rcu/dev branch so it can apply the new ReST conversion patches.
Weirdly enough, Patch 4/9 did not get received by the archive, but patchwork did pick it up:
https://lore.kernel.org/patchwork/patch/1109574/
Quoted 10 lines · Joel Fernandes (Google)
This series fixes the rcu/dev branch so it can apply the new ReST conversion patches.
Patches based on "
00ec8f46465ercu/nohz: Makemulti_cpu_stop()enable tick on all online CPUs"The easiest was to do this is to revert the patches that conflict and then applying the doc patches, and then applying them again. But in the re-application, we convert the documentation
No manual fix ups were done in this process, other than to documentation.
Ah, I was expecting that you would forward-port the conversion, but yes, that could be painful and error prone.
But given that there are some dependencies on these patches, could you please use the following alternative procedure for the patches that touch both code and documentation?
o Revert only the documentation portion of each commit. I will
then merge the partial reverts with the original commits.
o Apply the documentation conversion.
o Reapply the documentation portions on top of the conversion.
Thanx, Paul
Quoted 48 lines
thanks,
- Joel
And in the process I learnt about
get_user()and compiler barriers ;-)Joel Fernandes (Google) (8):
Revert "rcu: Restorebarrier()torcu_read_lock()andrcu_read_unlock()"
Revert "rcu: Add support for consolidated-RCU reader checking" Revert "treewide: Renamercu_dereference_raw_notrace()to_check()"
docs: rcu: Correct links referring to titles
docs: rcu: Increase toctree to 3
Revert "Revert "treewide: Renamercu_dereference_raw_notrace()to_check()""
Revert "Revert "rcu: Add support for consolidated-RCU reader checking""
Revert "Revert "rcu: Restorebarrier()torcu_read_lock()andrcu_read_unlock()""Mauro Carvalho Chehab (1):
docs: rcu: convert some articles from html to ReST.../Data-Structures/Data-Structures.html | 1391 ------- .../Data-Structures/Data-Structures.rst | 1163 ++++++ .../Expedited-Grace-Periods.html | 668 ---- .../Expedited-Grace-Periods.rst | 521 +++ .../Memory-Ordering/Tree-RCU-Diagram.html | 9 - .../Tree-RCU-Memory-Ordering.html | 704 ---- .../Tree-RCU-Memory-Ordering.rst | 624 +++ .../RCU/Design/Requirements/Requirements.html | 3401 ----------------- .../RCU/Design/Requirements/Requirements.rst | 2704 +++++++++++++ Documentation/RCU/index.rst | 7 +- Documentation/RCU/whatisRCU.txt | 4 +- 11 files changed, 5020 insertions(+), 6176 deletions(-)delete mode 100644
Documentation/RCU/Design/Data-Structures/Data-Structures.htmlcreate mode 100644Documentation/RCU/Design/Data-Structures/Data-Structures.rstdelete mode 100644Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.htmlcreate mode 100644Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rstdelete mode 100644Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Diagram.htmldelete mode 100644Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.htmlcreate mode 100644Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rstdelete mode 100644Documentation/RCU/Design/Requirements/Requirements.htmlcreate mode 100644Documentation/RCU/Design/Requirements/Requirements.rst--
2.22.0.770.g0f2c4a37fd-goog
Quoted 20 lines · Paul E. McKenney
Quoted 10 lines · Joel Fernandes (Google)
This series fixes the rcu/dev branch so it can apply the new ReST conversion patches.
Patches based on "
00ec8f46465ercu/nohz: Makemulti_cpu_stop()enable tick on all online CPUs"The easiest was to do this is to revert the patches that conflict and then applying the doc patches, and then applying them again. But in the re-application, we convert the documentation
No manual fix ups were done in this process, other than to documentation.
Ah, I was expecting that you would forward-port the conversion, but yes, that could be painful and error prone.
But given that there are some dependencies on these patches, could you please use the following alternative procedure for the patches that touch both code and documentation?
Quoted 6 lines
o Revert only the documentation portion of each commit. I will
then merge the partial reverts with the original commits.o Apply the documentation conversion.
o Reapply the documentation portions on top of the conversion.
Sure, this would be better. I will do this in the evening and send it to you. Thanks,
- Joel
Quoted 24 lines · Joel Fernandes
Quoted 23 lines · Paul E. McKenney
Quoted 10 lines · Joel Fernandes (Google)
This series fixes the rcu/dev branch so it can apply the new ReST conversion patches.
Patches based on "
00ec8f46465ercu/nohz: Makemulti_cpu_stop()enable tick on all online CPUs"The easiest was to do this is to revert the patches that conflict and then applying the doc patches, and then applying them again. But in the re-application, we convert the documentation
No manual fix ups were done in this process, other than to documentation.
Ah, I was expecting that you would forward-port the conversion, but yes, that could be painful and error prone.
But given that there are some dependencies on these patches, could you please use the following alternative procedure for the patches that touch both code and documentation?
o Revert only the documentation portion of each commit. I will
then merge the partial reverts with the original commits.
Quoted 6 lines
Quoted 3 lines
o Apply the documentation conversion.
o Reapply the documentation portions on top of the conversion.
Sure, this would be better. I will do this in the evening and send it to you. Thanks,
Very good, looking forward to seeing it!
The smoke tests of the rebase onto v5.3-rc2 passed, so on to create the usual branches. Yes, this will complicate merging of the partial reverts, but there will be the need to apply review comments and the like anyway, so not really a net increase in complexity. ;-)
Thanx, Paul