Track RCU dereferences in RCU read-side critical sections
Subsystems (7)
cover Track RCU dereferences in RCU read-side critical sections
As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an rcu_dereference() might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.
In this series, we are introducing LOCKED_ACCESS framework and based on which, we implement the RCU_LOCKED_ACCESS functionality to give us a clear hint: which rcu_dereference() happens in which RCU read-side critical section.
The basic idea of LOCKED_ACCESS is to maintain a chain of locks we have acquired already, and when there happens a data access, correlate the data access with the chain.
Lockdep already has lock chains, but we introduce a new but similar one
concept: acqchain, an acqchain is similar to a lock chain, except that the key of an acqchain is the hash sum of the acquire (instruction) positions of the locks in the chain, whereas the key of a lock chain is the hash sum of the class keys of the locks in the chain.
Acqchains are introduced because we want to correlate data accesses with critical sections and critical sections are better represented by the acquire positions rather than lock classes.
The acqchain key of a task is maintained in the same way as lock chain keys in lockdep.
Similar as lockdep, LOCKED_ACCESS also classify locks and data accesses by groups, locked access class is introduced for this reason. A locked access class also contains the data for allocation and lookup of acqchains and accesses, and the address of a locked access class is used as its key. By tagging locks and data accesses with the keys, we could describe which locks and data accesses are related.
The entry point of LOCKED_ACCESS is locked_access_point(). Calling locked_access_point() indicates that a data access happens, and after it called the data access will be correlated with the current acqchain.
We also provide a /proc filesystem interface to show the information we've collected, for each locked access class with the name <name> there will be a file at /proc/locked_access/<name> showing all the relationships collected so far for this locked access classes.
Based on LOCKED_ACCESS, we implement RCU_LOCKED_ACCESS, that tracks rcu_dereference()s inside RCU read-side critical sections.
This patchset is based on v4.5-rc2 and consists of 6 patches(in which patch 2-5 are the implementation of LOCKED_ACCESS):
1. Introduce some functions of irq_context.
2. Introduce locked access class and acqchain.
3. Maintain the keys of acqchains.
4. Introduce the entry point of LOCKED_ACCESS.
5. Add proc interface for locked access class
6. Enables LOCKED_ACCESS for RCU.
Tested by 0day and I also did a simple test on x86: build and boot a kernel with RCU_LOCKED_ACCESS=y and CONFIG_PROVE_LOCKING=y and ran several workloads(kernel building, git cloning, dbench), no problem has been observed, and /proc/locked_access/rcu was able to collect the relationships between ~300 RCU read-critical sections and ~500 rcu_dereference*().
Snippets of /proc/locked_access/rcu are as follow:
...(this rcu_dereference() happens after one rcu_read_lock())
...
ACQCHAIN 0xfdbf0c6aeea, 1 locks, irq_context 0:
LOCK at [<ffffffff812b1115>] get_proc_task_net+0x5/0x140
ACCESS TYPE 1 at kernel/pid.c:441
...
...(this rcu_dereference() happens after three rcu_read_lock())
...
ACQCHAIN 0xfe042af3bbfb2605, 3 locks, irq_context 0:
LOCK at [<ffffffff81094b47>] SyS_kill+0x97/0x2a0
LOCK at [<ffffffff8109286f>] kill_pid_info+0x1f/0x140
LOCK at [<ffffffff81092605>] group_send_sig_info+0x5/0x130
ACCESS TYPE 1 at kernel/signal.c:695...
Looking forwards to any suggestion, comment and question ;-)
Regards,
Boqun
RFC 1/6 lockdep: Add helper functions of irq_context
Add two helper functions of irq_context:
task_irq_context(): return the encoded irq_context of the task, the return value is encoded in the same as ->irq_context of held_lock. Always return 0 if !(CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING)
is_same_irq_context(): compare whether two irq_context are the same. Always return true if !(CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING)
These two functions are added for the future use of irq_context by LOCKED_ACCESS, because LOCKED_ACCESS needs to get the irq_context value of a task when a data access happens rather than when a lock is acquired, so it needs another way than getting the value from held_lock::irq_context
--- kernel/locking/lockdep.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
kernel/locking/lockdep.c +21 -2
@@ -2839,6 +2839,16 @@ static int mark_irqflags(struct task_struct *curr, struct held_lock *hlock)return 1 ; } +static unsigned int task_irq_context (struct task_struct *task) +{ +return 2 * (task->hardirq_context ?1 :0 ) + task->softirq_context; +} + +static bool is_same_irq_context (unsigned int ctx1,unsigned int ctx2) +{ +return ctx1 == ctx2; +} +static int separate_irq_context (struct task_struct *curr,struct held_lock *hlock) {
@@ -2847,8 +2857,7 @@ static int separate_irq_context(struct task_struct *curr,/* * Keep track of points where we cross into an interrupt context: */ - hlock->irq_context =2 *(curr->hardirq_context ?1 :0 ) + - curr->softirq_context; + hlock->irq_context =task_irq_context (curr);if (depth) {struct held_lock *prev_hlock;
@@ -2880,6 +2889,16 @@ static inline int mark_irqflags(struct task_struct *curr,return 1 ; } +static unsigned int task_irq_context (struct task_struct *task) +{ +return 0 ; +} + +static bool is_same_irq_context (unsigned int ctx1,unsigned int ctx2) +{ +return true; +} +static inline int separate_irq_context (struct task_struct *curr,struct held_lock *hlock) {
Signature · 1 line
2.7.0
RFC 2/6 lockdep: LOCKED_ACCESS: Introduce locked access class and acqchain
In reply to Boqun FengAs a similar concept as a lock class, a locked access class is a group of locks and related data accesses of those locks. A locked access class also contains the structures for allocation and lookup of acqchains and accesses.
The address of a locked access class is used as its key, we tag a group of locks with a locked access class by setting the content of the group's lock_class_key to be the key(i.e. address) of locked access class, and we can do this trick because, AFAIK, lockdep only uses the address of a lock_class_key. However, although this doesn't change the size of a lock_class_key, this does change the alignment requirement.
And tagging a lock_class_key with a locked access class is the step #1 to use LOCKED_ACCESS for a locking mechanism.
As a similar concept as a lock class chain, an acqchain is short for acquire chain, which is a chain of lock acquisitions. The difference between a lock class chain and an acqchain is that a lock class chain is keyed by the hash sum of the class keys, whereas an acqchain is keyed by the hash sum of the acquire ips. We introduce acqchains here because in LOCKED_ACCESS, we care more about the locations of critical sections.
This patch defines the structures for those concepts, along with the manipulation operations. Besides, config option CONFIG_LOCKED_ACCESS is introduced in this patch, too.
--- include/linux/lockdep.h | 17 ++++ kernel/locking/lockdep.c | 156 +++++++++++++++++++++++++++++++++++++ kernel/locking/lockdep_internals.h | 74 ++++++++++++++++++ lib/Kconfig.debug | 12 +++ 4 files changed, 259 insertions(+)
include/linux/lockdep.h +17 -0
@@ -51,9 +51,26 @@ struct lockdep_subclass_key {char __one_byte; } __attribute__ ((__packed__)); +#ifdef CONFIG_LOCKED_ACCESS +struct locked_access_class ; + +struct lock_class_key { +union { +struct lockdep_subclass_key subkeys[MAX_LOCKDEP_SUBCLASSES ]; +/* +* Use the content of the lock_class_key to store locked access +* class, as lockdep only use the address of a lock_class_key. +* However, please note by doing so, we will change the align +* requirement of lock_class_key +*/ +struct locked_access_class *laclass; + }; +}; +#else struct lock_class_key {struct lockdep_subclass_key subkeys[MAX_LOCKDEP_SUBCLASSES ]; }; +#endif extern struct lock_class_key __lockdep_no_validate__;
kernel/locking/lockdep.c +156 -0
159 lines not on this page · raw message
kernel/locking/lockdep_internals.h +74 -0
77 lines not on this page · raw message
lib/Kconfig.debug +12 -0
@@ -1065,6 +1065,18 @@ config LOCK_STAT CONFIG_LOCK_STAT defines "contended" and "acquired" lock events. (CONFIG_LOCKDEP defines "acquire" and "release" events.) +config LOCKED_ACCESS + bool + depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + select DEBUG_LOCK_ALLOC + default n + help + Provide a way to correlate data accesses with critical sections, + LOCKED_ACCESS is designed for the users of locking mechanisms to see + which data access happens inside which lock critical section. This + could help the users of a lock mechanism to gain more detailed + information about their lock usage. + config DEBUG_LOCKDEP bool "Lock dependency engine debugging" depends on DEBUG_KERNEL && LOCKDEP
Signature · 1 line
2.7.0
RFC 3/6 lockdep: LOCKED_ACCESS: Maintain the keys of acqchains
In reply to Boqun FengAdd held_lock::prev_acqchain_key and task_struct::curr_acqchain_key to maintain the keys of acqchains as the same as what lockdep does for keys of lock classes.
--- include/linux/lockdep.h | 3 +++ include/linux/sched.h | 3 +++ kernel/fork.c | 3 +++ kernel/locking/lockdep.c | 31 +++++++++++++++++++++++++++++++ 4 files changed, 40 insertions(+)
include/linux/lockdep.h +3 -0
@@ -244,6 +244,9 @@ struct held_lock {* with zero), here we store the previous hash value: */ u64 prev_chain_key; +#ifdef CONFIG_LOCKED_ACCESS +u64 prev_acqchain_key; +#endif unsigned long acquire_ip;struct lockdep_map *instance;struct lockdep_map *nest_lock;
include/linux/sched.h +3 -0
@@ -1643,6 +1643,9 @@ struct task_struct {unsigned int lockdep_recursion;struct held_lock held_locks[MAX_LOCK_DEPTH ];gfp_t lockdep_reclaim_gfp; +#ifdef CONFIG_LOCKED_ACCESS +u64 curr_acqchain_key; +#endif #endif #ifdef CONFIG_UBSAN unsigned int in_ubsan;
kernel/fork.c +3 -0
@@ -1404,6 +1404,9 @@ static struct task_struct *copy_process(unsigned long clone_flags, p->lockdep_depth =0 ;/* no locks held yet */ p->curr_chain_key =0 ; p->lockdep_recursion =0 ; +# ifdef CONFIG_LOCKED_ACCESS + p->curr_acqchain_key =0 ; +# endif #endif #ifdef CONFIG_DEBUG_MUTEXES
kernel/locking/lockdep.c +31 -0
@@ -3096,6 +3096,9 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,int chain_head =0 ;int class_idx;u64 chain_key; +#ifdef CONFIG_LOCKED_ACCESS +u64 acqchain_key; +#endif if (unlikely (!debug_locks))return 0 ;
@@ -3203,6 +3206,9 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,return 0 ; chain_key = curr->curr_chain_key; +#ifdef CONFIG_LOCKED_ACCESS + acqchain_key = curr->curr_acqchain_key; +#endif if (!depth) {/* * How can we have a chain hash when we ain't got no keys?!
@@ -3213,12 +3219,23 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, } hlock->prev_chain_key = chain_key; +#ifdef CONFIG_LOCKED_ACCESS + hlock->prev_acqchain_key = acqchain_key; +#endif if (separate_irq_context (curr, hlock)) { chain_key =0 ; + +#ifdef CONFIG_LOCKED_ACCESS + acqchain_key =0 ; +#endif + chain_head =1 ; } chain_key =iterate_chain_key (chain_key, id); +#ifdef CONFIG_LOCKED_ACCESS + acqchain_key =iterate_acqchain_key (acqchain_key, ip); +#endif if (nest_lock && !__lock_is_held (nest_lock))return print_lock_nested_lock_not_held (curr, hlock, ip);
@@ -3226,6 +3243,9 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,return 0 ; curr->curr_chain_key = chain_key; +#ifdef CONFIG_LOCKED_ACCESS + curr->curr_acqchain_key = acqchain_key; +#endif curr->lockdep_depth++;check_chain_key (curr);#ifdef CONFIG_DEBUG_LOCKDEP
@@ -3355,6 +3375,9 @@ found_it: curr->lockdep_depth = i; curr->curr_chain_key = hlock->prev_chain_key; +#ifdef CONFIG_LOCKED_ACCESS + curr->curr_acqchain_key = hlock->prev_acqchain_key; +#endif for (; i < depth; i++) { hlock = curr->held_locks + i;
@@ -3445,6 +3468,9 @@ found_it: curr->lockdep_depth = i; curr->curr_chain_key = hlock->prev_chain_key; +#ifdef CONFIG_LOCKED_ACCESS + curr->curr_acqchain_key = hlock->prev_acqchain_key; +#endif for (i++; i < depth; i++) { hlock = curr->held_locks + i;
@@ -3886,6 +3912,11 @@ void lockdep_reset(void)raw_local_irq_save (flags); current->curr_chain_key =0 ; + +#ifdef CONFIG_LOCKED_ACCESS + current->curr_acqchain_key =0 ; +#endif + current->lockdep_depth =0 ; current->lockdep_recursion =0 ;memset (current->held_locks,0 ,MAX_LOCK_DEPTH *sizeof (struct held_lock ));
Signature · 1 line
2.7.0
RFC 4/6 lockdep: LOCKED_ACCESS: Introduce locked_access_point()
In reply to Boqun Fenglocked_access_point() is the entry point of the whole LOCKED_ACCESS framework, every time a locked_access_point() is called, LOCKE_ACCESS will correlate the data access location with the current acqchain.
So putting locked_access_point() at the data accesses you care about is the step #2 to use LOCKED_ACCESS.
--- include/linux/lockdep.h | 29 ++++++++++++ kernel/locking/lockdep.c | 121 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+)
include/linux/lockdep.h +29 -0
@@ -570,4 +570,33 @@ lockdep_rcu_suspicious(const char *file, const int line, const char *s) }#endif +#ifdef CONFIG_LOCKED_ACCESS +struct locked_access_location { +/* Filename of the access */ +const char *filename; +/* Line number of the access */ +long lineno; +}; + +#define LOCKED_ACCESS_TYPE_READ 1/* read */ + +extern void locked_access (struct locked_access_class *laclass, +struct locked_access_location *access, +int type); + +/* +* Entry point of LOCKED_ACCESS, should be called at every place the data +* accesses of the laclass happen. +* +* @_type must be one of the LOCKED_ACCESS_TYPE_* +*/ +#define locked_access_point (_laclass, _type) \ +({ \ + static struct locked_access_location a = { \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + }; \ + locked_access(_laclass, &a, _type); \ +}) +#endif /* CONFIG_LOCKED_ACCESS */ #endif /* __LINUX_LOCKDEP_H */
kernel/locking/lockdep.c +121 -0
125 lines not on this page · raw message
Signature · 1 line
2.7.0
RFC 5/6 lockdep: LOCKED_ACCESS: Add proc interface for locked access class
In reply to Boqun FengProvide the proc filesystem interface for LOCKED_ACCESS, for a locked access class whose name is <name>, there will be a file at /proc/locked_access/<name> containing all the information the LOCKED_ACCESS has collected so far.
Also add a macro to define a locked access class.
--- kernel/locking/lockdep_internals.h | 17 +++++ kernel/locking/lockdep_proc.c | 127 +++++++++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+)
kernel/locking/lockdep_internals.h +17 -0
@@ -241,4 +241,21 @@ struct locked_access_class { .nr_acqchain_hlocks = 0,\ .nr_access_structs = 0, \ } + +extern int create_laclass_proc (const char *name, +struct locked_access_class *laclass); + +#define DEFINE_CREATE_LACLASS_PROC (name) \ +static int __init ___create_##name##_laclass_proc(void) \ +{ \ + return create_laclass_proc(#name, &name##_laclass); \ +} \ +late_initcall(___create_##name##_laclass_proc) + +/* Define a Locked Access Class and create its proc file */ +#define DEFINE_LACLASS (name) \ + struct locked_access_class name##_laclass = \ + INIT_LOCKED_ACCESS_DATA(name); \ + EXPORT_SYMBOL(name##_laclass); \ + DEFINE_CREATE_LACLASS_PROC(name)#endif /* CONFIG_LOCKED_ACCESS */
kernel/locking/lockdep_proc.c +127 -0
139 lines not on this page · raw message
Signature · 1 line
2.7.0
RFC 6/6 RCU: Track rcu_dereference() in RCU read-side critical section
In reply to Boqun FengThe variables protected by an RCU read-side critical section are sometimes hard to figure out, especially when the critical section is long or has some function calls in it. However, figuring out which variable a RCU read-side critical section protects could save us a lot of time for code reviewing, bug fixing or performance tuning.
This patch therefore uses the LOCKED_ACCESS to collect the information of relationship between rcu_dereference*() and rcu_read_lock*() by doing:
Step 0: define a locked_access_class for RCU.
Step 1: set the content of rcu_*_lock_key and __srcu_key to the address of the locked_access_class for RCU.
Step 2: add locked_access_point() in __rcu_dereference_check()
After that we can figure out not only in which RCU read-side critical section but also after which rcu_read_lock*() called an rcu_dereference*() is called.
This feature is controlled by a config option RCU_LOCKED_ACCESS.
Also clean up the initialization code of lockdep_maps for different flavors of RCU a little bit.
--- include/linux/rcupdate.h | 8 ++++++++ include/linux/srcu.h | 8 +++++++- kernel/locking/lockdep.c | 3 +++ kernel/rcu/update.c | 31 +++++++++++++++++-------------- lib/Kconfig.debug | 12 ++++++++++++ 5 files changed, 47 insertions(+), 15 deletions(-)
include/linux/rcupdate.h +8 -0
@@ -610,6 +610,13 @@ static inline void rcu_preempt_sleep_check(void)#define rcu_dereference_sparse (p, space)#endif /* #else #ifdef __CHECKER__ */ +#ifdef CONFIG_RCU_LOCKED_ACCESS +extern struct locked_access_class rcu_laclass; +#define rcu_dereference_access () \ + locked_access_point(&rcu_laclass, LOCKED_ACCESS_TYPE_READ) +#else /* #ifdef CONFIG_LOCKED_ACCESS */ +#define rcu_dereference_access () +#endif /* #else #ifdef CONFIG_LOCKED_ACCESS */ #define __rcu_access_pointer (p, space) \ ({ \ typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \
@@ -622,6 +629,7 @@ static inline void rcu_preempt_sleep_check(void) typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \RCU_LOCKDEP_WARN (!(c),"suspicious rcu_dereference_check() usage" ); \rcu_dereference_sparse (p, space); \ +rcu_dereference_access (); \ ((typeof(*p) __force __kernel *)(________p1)); \ })#define __rcu_dereference_protected (p, c, space) \
include/linux/srcu.h +7 -1
@@ -64,12 +64,18 @@ struct srcu_struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else +# define RCU_KEY { 0 } +# endif +int __init_srcu_struct (struct srcu_struct *sp,const char *name,struct lock_class_key *key);#define init_srcu_struct (sp) \ ({ \ - static struct lock_class_key __srcu_key; \ + static struct lock_class_key __srcu_key = RCU_KEY; \ \ __init_srcu_struct((sp), #sp, &__srcu_key); \ })
kernel/locking/lockdep.c +3 -0
@@ -4623,5 +4623,8 @@ void locked_access(struct locked_access_class *laclass, correlate_locked_access(laclass, acqchain, loc, type); }EXPORT_SYMBOL (locked_access); +#ifdef CONFIG_RCU_LOCKED_ACCESS +DEFINE_LACLASS (rcu); +#endif #endif /* CONFIG_LOCKED_ACCESS */
kernel/rcu/update.c +17 -14
@@ -235,20 +235,23 @@ EXPORT_SYMBOL_GPL(__rcu_read_unlock);#endif /* #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_DEBUG_LOCK_ALLOC -static struct lock_class_key rcu_lock_key; -struct lockdep_map rcu_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock" , &rcu_lock_key); -EXPORT_SYMBOL_GPL (rcu_lock_map); - -static struct lock_class_key rcu_bh_lock_key; -struct lockdep_map rcu_bh_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_bh" , &rcu_bh_lock_key); -EXPORT_SYMBOL_GPL (rcu_bh_lock_map); - -static struct lock_class_key rcu_sched_lock_key; -struct lockdep_map rcu_sched_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_sched" , &rcu_sched_lock_key); -EXPORT_SYMBOL_GPL (rcu_sched_lock_map); + +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else +# define RCU_KEY { 0 } +# endif + +#define DEFINE_RCU_LOCKDEP_MAP (flavor) \ + static struct lock_class_key rcu##flavor##_lock_key = RCU_KEY; \ + struct lockdep_map rcu##flavor##_lock_map = \ + STATIC_LOCKDEP_MAP_INIT("rcu" #flavor "_read_lock", \ + &rcu##flavor##_lock_key); \ + EXPORT_SYMBOL_GPL(rcu##flavor##_lock_map) + +DEFINE_RCU_LOCKDEP_MAP (); +DEFINE_RCU_LOCKDEP_MAP (_bh); +DEFINE_RCU_LOCKDEP_MAP (_sched);static struct lock_class_key rcu_callback_key; struct lockdep_map rcu_callback_map =
lib/Kconfig.debug +12 -0
@@ -1410,6 +1410,18 @@ config RCU_EQS_DEBUG Say N here if you need ultimate kernel/user switch latencies Say Y if you are unsure +config RCU_LOCKED_ACCESS + bool "Track data access in RCU read-side critical sections" + depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + select LOCKED_ACCESS + default n + help + Track data acces in RCU read-side critical sections, + by doing so, one can examine which rcu_dereference() and its + friends are called in which RCU read-side critical sections, + and even more detailed, after which rcu_read_lock() and + its friends are called. + endmenu # "RCU Debugging" config DEBUG_BLOCK_EXT_DEVT
Signature · 1 line
2.7.0
Re: [RFC 6/6] RCU: Track rcu_dereference() in RCU read-side critical section
Quoted 72 lines · Boqun Feng
The variables protected by an RCU read-side critical section are sometimes hard to figure out, especially when the critical section is long or has some function calls in it. However, figuring out which variable a RCU read-side critical section protects could save us a lot of time for code reviewing, bug fixing or performance tuning.
This patch therefore uses the
LOCKED_ACCESSto collect the information of relationship between rcu_dereference*() and rcu_read_lock*() by doing:Step 0: define a
locked_access_classfor RCU.Step 1: set the content of rcu_*_lock_key and
__srcu_keyto the address of thelocked_access_classfor RCU.Step 2: add
locked_access_point()in__rcu_dereference_check()After that we can figure out not only in which RCU read-side critical section but also after which rcu_read_lock*() called an rcu_dereference*() is called.
This feature is controlled by a config option
RCU_LOCKED_ACCESS.Also clean up the initialization code of
lockdep_mapsfor different flavors of RCU a little bit.Signed-off-by Boqun Feng <[email protected]>--- include/linux/rcupdate.h | 8 ++++++++ include/linux/srcu.h | 8 +++++++- kernel/locking/lockdep.c | 3 +++ kernel/rcu/update.c | 31 +++++++++++++++++-------------- lib/Kconfig.debug | 12 ++++++++++++ 5 files changed, 47 insertions(+), 15 deletions(-)include/linux/rcupdate.h +8
-0@@ -610,6 +610,13 @@ static inline void rcu_preempt_sleep_check(void)#define rcu_dereference_sparse (p, space)#endif /* #else #ifdef __CHECKER__ */ +#ifdef CONFIG_RCU_LOCKED_ACCESS +extern struct locked_access_class rcu_laclass; +#define rcu_dereference_access () \ + locked_access_point(&rcu_laclass, LOCKED_ACCESS_TYPE_READ) +#else /* #ifdef CONFIG_LOCKED_ACCESS */ +#define rcu_dereference_access () +#endif /* #else #ifdef CONFIG_LOCKED_ACCESS */ #define __rcu_access_pointer (p, space) \ ({ \ typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \@@ -622,6 +629,7 @@ static inline void rcu_preempt_sleep_check(void) typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \RCU_LOCKDEP_WARN (!(c),"suspicious rcu_dereference_check() usage" ); \rcu_dereference_sparse (p, space); \ +rcu_dereference_access (); \ ((typeof(*p) __force __kernel *)(________p1)); \ })#define __rcu_dereference_protected (p, c, space) \include/linux/srcu.h +3
-0@@ -64,12 +64,18 @@ struct srcu_struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else
include/linux/srcu.h struct srcu_struct { · line 78
Quoted 10 lines
include/linux/srcu.h +4
-1+# define RCU_KEY { 0 } +# endif +int __init_srcu_struct (struct srcu_struct *sp,const char *name,struct lock_class_key *key);#define init_srcu_struct (sp) \ ({ \ - static struct lock_class_key __srcu_key; \ + static struct lock_class_key __srcu_key = RCU_KEY; \
This gets me the following for the TASKS01, TINY02, TREE02, TREE05, TREE06, and TREE08 configurations:
CC mm/mmap.o
In file included from /home/paulmck/public_git/linux-rcu/include/linux/notifier.h:15:0,
from /home/paulmck/public_git/linux-rcu/arch/x86/include/asm/kdebug.h:4,
from /home/paulmck/public_git/linux-rcu/include/linux/kdebug.h:4,
from /home/paulmck/public_git/linux-rcu/kernel/notifier.c:1:/home/paulmck/public_git/linux-rcu/kernel/notifier.c: In function ‘srcu_init_notifier_head’:/home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: missing braces around initializer [-Wmissing-braces]
static struct lock_class_key __srcu_key = RCU_KEY; \
^
/home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’
if (init_srcu_struct(&nh->srcu) < 0)
^
/home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: (near initialization for ‘__srcu_key.subkeys’) [-Wmissing-braces]
static struct lock_class_key __srcu_key = RCU_KEY; \
^
/home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’
if (init_srcu_struct(&nh->srcu) < 0)
^These have the following in their .config files:
CONFIG_DEBUG_LOCK_ALLOC=y
However, none of your new Kconfig options are set, which needs to be tested because this will be the common case. Several of them have CONFIG_PROVE_LOCKING=y, but others do not.
I have dequeued these for the moment. Please send an updated patch series when you have this fixed.
Thanx, Paul
Quoted 84 lines
include/linux/srcu.h +0
-0\ __init_srcu_struct((sp), #sp, &__srcu_key); \ })kernel/locking/lockdep.c +3
-0@@ -4623,5 +4623,8 @@ void locked_access(struct locked_access_class *laclass, correlate_locked_access(laclass, acqchain, loc, type); }EXPORT_SYMBOL (locked_access); +#ifdef CONFIG_RCU_LOCKED_ACCESS +DEFINE_LACLASS (rcu); +#endif #endif /* CONFIG_LOCKED_ACCESS */ kernel/rcu/update.c +17
-14@@ -235,20 +235,23 @@ EXPORT_SYMBOL_GPL(__rcu_read_unlock);#endif /* #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_DEBUG_LOCK_ALLOC -static struct lock_class_key rcu_lock_key; -struct lockdep_map rcu_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock" , &rcu_lock_key); -EXPORT_SYMBOL_GPL (rcu_lock_map); - -static struct lock_class_key rcu_bh_lock_key; -struct lockdep_map rcu_bh_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_bh" , &rcu_bh_lock_key); -EXPORT_SYMBOL_GPL (rcu_bh_lock_map); - -static struct lock_class_key rcu_sched_lock_key; -struct lockdep_map rcu_sched_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_sched" , &rcu_sched_lock_key); -EXPORT_SYMBOL_GPL (rcu_sched_lock_map); + +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else +# define RCU_KEY { 0 } +# endif + +#define DEFINE_RCU_LOCKDEP_MAP (flavor) \ + static struct lock_class_key rcu##flavor##_lock_key = RCU_KEY; \ + struct lockdep_map rcu##flavor##_lock_map = \ + STATIC_LOCKDEP_MAP_INIT("rcu" #flavor "_read_lock", \ + &rcu##flavor##_lock_key); \ + EXPORT_SYMBOL_GPL(rcu##flavor##_lock_map) + +DEFINE_RCU_LOCKDEP_MAP (); +DEFINE_RCU_LOCKDEP_MAP (_bh); +DEFINE_RCU_LOCKDEP_MAP (_sched);static struct lock_class_key rcu_callback_key; struct lockdep_map rcu_callback_map =lib/Kconfig.debug +12
-0@@ -1410,6 +1410,18 @@ config RCU_EQS_DEBUG Say N here if you need ultimate kernel/user switch latencies Say Y if you are unsure +config RCU_LOCKED_ACCESS + bool "Track data access in RCU read-side critical sections" + depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + select LOCKED_ACCESS + default n + help + Track data acces in RCU read-side critical sections, + by doing so, one can examine which rcu_dereference() and its + friends are called in which RCU read-side critical sections, + and even more detailed, after which rcu_read_lock() and + its friends are called. + endmenu # "RCU Debugging" config DEBUG_BLOCK_EXT_DEVTSignature · 1 line
2.7.0
Re: [RFC 6/6] RCU: Track rcu_dereference() in RCU read-side critical section
Quoted 106 lines · Paul E. McKenney
Quoted 72 lines · Boqun Feng
The variables protected by an RCU read-side critical section are sometimes hard to figure out, especially when the critical section is long or has some function calls in it. However, figuring out which variable a RCU read-side critical section protects could save us a lot of time for code reviewing, bug fixing or performance tuning.
This patch therefore uses the
LOCKED_ACCESSto collect the information of relationship between rcu_dereference*() and rcu_read_lock*() by doing:Step 0: define a
locked_access_classfor RCU.Step 1: set the content of rcu_*_lock_key and
__srcu_keyto the address of thelocked_access_classfor RCU.Step 2: add
locked_access_point()in__rcu_dereference_check()After that we can figure out not only in which RCU read-side critical section but also after which rcu_read_lock*() called an rcu_dereference*() is called.
This feature is controlled by a config option
RCU_LOCKED_ACCESS.Also clean up the initialization code of
lockdep_mapsfor different flavors of RCU a little bit.Signed-off-by Boqun Feng <[email protected]>--- include/linux/rcupdate.h | 8 ++++++++ include/linux/srcu.h | 8 +++++++- kernel/locking/lockdep.c | 3 +++ kernel/rcu/update.c | 31 +++++++++++++++++-------------- lib/Kconfig.debug | 12 ++++++++++++ 5 files changed, 47 insertions(+), 15 deletions(-)include/linux/rcupdate.h +8
-0@@ -610,6 +610,13 @@ static inline void rcu_preempt_sleep_check(void)#define rcu_dereference_sparse (p, space)#endif /* #else #ifdef __CHECKER__ */ +#ifdef CONFIG_RCU_LOCKED_ACCESS +extern struct locked_access_class rcu_laclass; +#define rcu_dereference_access () \ + locked_access_point(&rcu_laclass, LOCKED_ACCESS_TYPE_READ) +#else /* #ifdef CONFIG_LOCKED_ACCESS */ +#define rcu_dereference_access () +#endif /* #else #ifdef CONFIG_LOCKED_ACCESS */ #define __rcu_access_pointer (p, space) \ ({ \ typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \@@ -622,6 +629,7 @@ static inline void rcu_preempt_sleep_check(void) typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \RCU_LOCKDEP_WARN (!(c),"suspicious rcu_dereference_check() usage" ); \rcu_dereference_sparse (p, space); \ +rcu_dereference_access (); \ ((typeof(*p) __force __kernel *)(________p1)); \ })#define __rcu_dereference_protected (p, c, space) \include/linux/srcu.h +3
-0@@ -64,12 +64,18 @@ struct srcu_struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else include/linux/srcu.h struct srcu_struct { · line 78
Quoted 10 lines
include/linux/srcu.h +4
-1+# define RCU_KEY { 0 } +# endif +int __init_srcu_struct (struct srcu_struct *sp,const char *name,struct lock_class_key *key);#define init_srcu_struct (sp) \ ({ \ - static struct lock_class_key __srcu_key; \ + static struct lock_class_key __srcu_key = RCU_KEY; \This gets me the following for the TASKS01, TINY02, TREE02, TREE05, TREE06, and TREE08 configurations:
CC
mm/mmap.o
In file included from/home/paulmck/public_git/linux-rcu/include/linux/notifier.h:15:0,
from/home/paulmck/public_git/linux-rcu/arch/x86/include/asm/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/include/linux/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/kernel/notifier.c:1:/home/paulmck/public_git/linux-rcu/kernel/notifier.c: In function ‘srcu_init_notifier_head’:/home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: missing braces around initializer [-Wmissing-braces]static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^ /home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: (near initialization for ‘__srcu_key.subkeys’) [-Wmissing-braces] static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^
Quoted 8 lines
These have the following in their .config files:
CONFIG_DEBUG_LOCK_ALLOC=yHowever, none of your new
Kconfigoptions are set, which needs to be tested because this will be the common case. Several of them haveCONFIG_PROVE_LOCKING=y, but others do not.
Oh.. this is embarrassing ;-(
Hmm.. when CONFIG_RCU_LOCKED_ACCESS=n and CONFIG_DEBUG_LOCK_ALLOC=y, this line becomes:
static struct lock_class_key __srcu_key = { 0 };IIUC, "={ 0 }" is the unverisal zero initializer in C, could be used for zero initialising any structure or array, so it's OK here, right?
And may I ask your compiler's version? Because looks to me this may be a compiler bug according to:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64709
Also I can't reproduce this with CONFIG_RCU_LOCKED_ACCESS=n and CONFIG_DEBUG_LOCK_ALLOC=y on my machine, whose gcc version is 5.3.0.
Of course I can use a different way here and do not need a "={ 0 }" here, because __srcu_key is static, and I just want to make sure we know what's going on here before we use a workaround, or I just want to know if I'm a bad C programmer ;-)
Regards,
Boqun
Quoted 90 lines
I have dequeued these for the moment. Please send an updated patch series when you have this fixed.
Thanx, Paul
Quoted 84 lines
include/linux/srcu.h +0
-0\ __init_srcu_struct((sp), #sp, &__srcu_key); \ })kernel/locking/lockdep.c +3
-0@@ -4623,5 +4623,8 @@ void locked_access(struct locked_access_class *laclass, correlate_locked_access(laclass, acqchain, loc, type); }EXPORT_SYMBOL (locked_access); +#ifdef CONFIG_RCU_LOCKED_ACCESS +DEFINE_LACLASS (rcu); +#endif #endif /* CONFIG_LOCKED_ACCESS */ kernel/rcu/update.c +17
-14@@ -235,20 +235,23 @@ EXPORT_SYMBOL_GPL(__rcu_read_unlock);#endif /* #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_DEBUG_LOCK_ALLOC -static struct lock_class_key rcu_lock_key; -struct lockdep_map rcu_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock" , &rcu_lock_key); -EXPORT_SYMBOL_GPL (rcu_lock_map); - -static struct lock_class_key rcu_bh_lock_key; -struct lockdep_map rcu_bh_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_bh" , &rcu_bh_lock_key); -EXPORT_SYMBOL_GPL (rcu_bh_lock_map); - -static struct lock_class_key rcu_sched_lock_key; -struct lockdep_map rcu_sched_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_sched" , &rcu_sched_lock_key); -EXPORT_SYMBOL_GPL (rcu_sched_lock_map); + +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else +# define RCU_KEY { 0 } +# endif + +#define DEFINE_RCU_LOCKDEP_MAP (flavor) \ + static struct lock_class_key rcu##flavor##_lock_key = RCU_KEY; \ + struct lockdep_map rcu##flavor##_lock_map = \ + STATIC_LOCKDEP_MAP_INIT("rcu" #flavor "_read_lock", \ + &rcu##flavor##_lock_key); \ + EXPORT_SYMBOL_GPL(rcu##flavor##_lock_map) + +DEFINE_RCU_LOCKDEP_MAP (); +DEFINE_RCU_LOCKDEP_MAP (_bh); +DEFINE_RCU_LOCKDEP_MAP (_sched);static struct lock_class_key rcu_callback_key; struct lockdep_map rcu_callback_map =lib/Kconfig.debug +12
-0@@ -1410,6 +1410,18 @@ config RCU_EQS_DEBUG Say N here if you need ultimate kernel/user switch latencies Say Y if you are unsure +config RCU_LOCKED_ACCESS + bool "Track data access in RCU read-side critical sections" + depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + select LOCKED_ACCESS + default n + help + Track data acces in RCU read-side critical sections, + by doing so, one can examine which rcu_dereference() and its + friends are called in which RCU read-side critical sections, + and even more detailed, after which rcu_read_lock() and + its friends are called. + endmenu # "RCU Debugging" config DEBUG_BLOCK_EXT_DEVTSignature · 1 line
2.7.0
Re: [RFC 6/6] RCU: Track rcu_dereference() in RCU read-side critical section
Quoted 123 lines · Boqun Feng
Quoted 106 lines · Paul E. McKenney
Quoted 72 lines · Boqun Feng
The variables protected by an RCU read-side critical section are sometimes hard to figure out, especially when the critical section is long or has some function calls in it. However, figuring out which variable a RCU read-side critical section protects could save us a lot of time for code reviewing, bug fixing or performance tuning.
This patch therefore uses the
LOCKED_ACCESSto collect the information of relationship between rcu_dereference*() and rcu_read_lock*() by doing:Step 0: define a
locked_access_classfor RCU.Step 1: set the content of rcu_*_lock_key and
__srcu_keyto the address of thelocked_access_classfor RCU.Step 2: add
locked_access_point()in__rcu_dereference_check()After that we can figure out not only in which RCU read-side critical section but also after which rcu_read_lock*() called an rcu_dereference*() is called.
This feature is controlled by a config option
RCU_LOCKED_ACCESS.Also clean up the initialization code of
lockdep_mapsfor different flavors of RCU a little bit.Signed-off-by Boqun Feng <[email protected]>--- include/linux/rcupdate.h | 8 ++++++++ include/linux/srcu.h | 8 +++++++- kernel/locking/lockdep.c | 3 +++ kernel/rcu/update.c | 31 +++++++++++++++++-------------- lib/Kconfig.debug | 12 ++++++++++++ 5 files changed, 47 insertions(+), 15 deletions(-)include/linux/rcupdate.h +8
-0@@ -610,6 +610,13 @@ static inline void rcu_preempt_sleep_check(void)#define rcu_dereference_sparse (p, space)#endif /* #else #ifdef __CHECKER__ */ +#ifdef CONFIG_RCU_LOCKED_ACCESS +extern struct locked_access_class rcu_laclass; +#define rcu_dereference_access () \ + locked_access_point(&rcu_laclass, LOCKED_ACCESS_TYPE_READ) +#else /* #ifdef CONFIG_LOCKED_ACCESS */ +#define rcu_dereference_access () +#endif /* #else #ifdef CONFIG_LOCKED_ACCESS */ #define __rcu_access_pointer (p, space) \ ({ \ typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \@@ -622,6 +629,7 @@ static inline void rcu_preempt_sleep_check(void) typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \RCU_LOCKDEP_WARN (!(c),"suspicious rcu_dereference_check() usage" ); \rcu_dereference_sparse (p, space); \ +rcu_dereference_access (); \ ((typeof(*p) __force __kernel *)(________p1)); \ })#define __rcu_dereference_protected (p, c, space) \include/linux/srcu.h +3
-0@@ -64,12 +64,18 @@ struct srcu_struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else include/linux/srcu.h struct srcu_struct { · line 78
Quoted 10 lines
include/linux/srcu.h +4
-1+# define RCU_KEY { 0 } +# endif +int __init_srcu_struct (struct srcu_struct *sp,const char *name,struct lock_class_key *key);#define init_srcu_struct (sp) \ ({ \ - static struct lock_class_key __srcu_key; \ + static struct lock_class_key __srcu_key = RCU_KEY; \This gets me the following for the TASKS01, TINY02, TREE02, TREE05, TREE06, and TREE08 configurations:
CC
mm/mmap.o
In file included from/home/paulmck/public_git/linux-rcu/include/linux/notifier.h:15:0,
from/home/paulmck/public_git/linux-rcu/arch/x86/include/asm/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/include/linux/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/kernel/notifier.c:1:/home/paulmck/public_git/linux-rcu/kernel/notifier.c: In function ‘srcu_init_notifier_head’:/home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: missing braces around initializer [-Wmissing-braces]static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^ /home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: (near initialization for ‘__srcu_key.subkeys’) [-Wmissing-braces] static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^Quoted 8 lines
These have the following in their .config files:
CONFIG_DEBUG_LOCK_ALLOC=yHowever, none of your new
Kconfigoptions are set, which needs to be tested because this will be the common case. Several of them haveCONFIG_PROVE_LOCKING=y, but others do not.Oh.. this is embarrassing ;-(
Hmm.. when
CONFIG_RCU_LOCKED_ACCESS=nandCONFIG_DEBUG_LOCK_ALLOC=y, this line becomes:static struct lock_class_key __srcu_key = { 0 };
Quoted 8 lines
IIUC, "={ 0 }" is the unverisal zero initializer in C, could be used for zero initialising any structure or array, so it's OK here, right?
And may I ask your compiler's version? Because looks to me this may be a compiler bug according to:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64709
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04) (KSIC)
Quoted 7 lines
Also I can't reproduce this with
CONFIG_RCU_LOCKED_ACCESS=nandCONFIG_DEBUG_LOCK_ALLOC=yon my machine, whose gcc version is 5.3.0.Of course I can use a different way here and do not need a "={ 0 }" here, because
__srcu_keyis static, and I just want to make sure we know what's going on here before we use a workaround, or I just want to know if I'm a bad C programmer ;-)
Well, it is quite possible that your code is correct in theory, but we do need to make it so that compilers can deal with it. It would not be the first compiler-bug workaround in RCU...
Thanx, Paul
Quoted 93 lines
Regards,
BoqunQuoted 90 lines
I have dequeued these for the moment. Please send an updated patch series when you have this fixed.
Thanx, Paul
Quoted 84 lines
include/linux/srcu.h +0
-0\ __init_srcu_struct((sp), #sp, &__srcu_key); \ })kernel/locking/lockdep.c +3
-0@@ -4623,5 +4623,8 @@ void locked_access(struct locked_access_class *laclass, correlate_locked_access(laclass, acqchain, loc, type); }EXPORT_SYMBOL (locked_access); +#ifdef CONFIG_RCU_LOCKED_ACCESS +DEFINE_LACLASS (rcu); +#endif #endif /* CONFIG_LOCKED_ACCESS */ kernel/rcu/update.c +17
-14@@ -235,20 +235,23 @@ EXPORT_SYMBOL_GPL(__rcu_read_unlock);#endif /* #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_DEBUG_LOCK_ALLOC -static struct lock_class_key rcu_lock_key; -struct lockdep_map rcu_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock" , &rcu_lock_key); -EXPORT_SYMBOL_GPL (rcu_lock_map); - -static struct lock_class_key rcu_bh_lock_key; -struct lockdep_map rcu_bh_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_bh" , &rcu_bh_lock_key); -EXPORT_SYMBOL_GPL (rcu_bh_lock_map); - -static struct lock_class_key rcu_sched_lock_key; -struct lockdep_map rcu_sched_lock_map = -STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_sched" , &rcu_sched_lock_key); -EXPORT_SYMBOL_GPL (rcu_sched_lock_map); + +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else +# define RCU_KEY { 0 } +# endif + +#define DEFINE_RCU_LOCKDEP_MAP (flavor) \ + static struct lock_class_key rcu##flavor##_lock_key = RCU_KEY; \ + struct lockdep_map rcu##flavor##_lock_map = \ + STATIC_LOCKDEP_MAP_INIT("rcu" #flavor "_read_lock", \ + &rcu##flavor##_lock_key); \ + EXPORT_SYMBOL_GPL(rcu##flavor##_lock_map) + +DEFINE_RCU_LOCKDEP_MAP (); +DEFINE_RCU_LOCKDEP_MAP (_bh); +DEFINE_RCU_LOCKDEP_MAP (_sched);static struct lock_class_key rcu_callback_key; struct lockdep_map rcu_callback_map =lib/Kconfig.debug +12
-0@@ -1410,6 +1410,18 @@ config RCU_EQS_DEBUG Say N here if you need ultimate kernel/user switch latencies Say Y if you are unsure +config RCU_LOCKED_ACCESS + bool "Track data access in RCU read-side critical sections" + depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + select LOCKED_ACCESS + default n + help + Track data acces in RCU read-side critical sections, + by doing so, one can examine which rcu_dereference() and its + friends are called in which RCU read-side critical sections, + and even more detailed, after which rcu_read_lock() and + its friends are called. + endmenu # "RCU Debugging" config DEBUG_BLOCK_EXT_DEVTSignature · 1 line
2.7.0
Re: [RFC 6/6] RCU: Track rcu_dereference() in RCU read-side critical section
Quoted 138 lines · Paul E. McKenney
Quoted 123 lines · Boqun Feng
Quoted 106 lines · Paul E. McKenney
Quoted 72 lines · Boqun Feng
The variables protected by an RCU read-side critical section are sometimes hard to figure out, especially when the critical section is long or has some function calls in it. However, figuring out which variable a RCU read-side critical section protects could save us a lot of time for code reviewing, bug fixing or performance tuning.
This patch therefore uses the
LOCKED_ACCESSto collect the information of relationship between rcu_dereference*() and rcu_read_lock*() by doing:Step 0: define a
locked_access_classfor RCU.Step 1: set the content of rcu_*_lock_key and
__srcu_keyto the address of thelocked_access_classfor RCU.Step 2: add
locked_access_point()in__rcu_dereference_check()After that we can figure out not only in which RCU read-side critical section but also after which rcu_read_lock*() called an rcu_dereference*() is called.
This feature is controlled by a config option
RCU_LOCKED_ACCESS.Also clean up the initialization code of
lockdep_mapsfor different flavors of RCU a little bit.Signed-off-by Boqun Feng <[email protected]>--- include/linux/rcupdate.h | 8 ++++++++ include/linux/srcu.h | 8 +++++++- kernel/locking/lockdep.c | 3 +++ kernel/rcu/update.c | 31 +++++++++++++++++-------------- lib/Kconfig.debug | 12 ++++++++++++ 5 files changed, 47 insertions(+), 15 deletions(-)include/linux/rcupdate.h +8
-0@@ -610,6 +610,13 @@ static inline void rcu_preempt_sleep_check(void)#define rcu_dereference_sparse (p, space)#endif /* #else #ifdef __CHECKER__ */ +#ifdef CONFIG_RCU_LOCKED_ACCESS +extern struct locked_access_class rcu_laclass; +#define rcu_dereference_access () \ + locked_access_point(&rcu_laclass, LOCKED_ACCESS_TYPE_READ) +#else /* #ifdef CONFIG_LOCKED_ACCESS */ +#define rcu_dereference_access () +#endif /* #else #ifdef CONFIG_LOCKED_ACCESS */ #define __rcu_access_pointer (p, space) \ ({ \ typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \@@ -622,6 +629,7 @@ static inline void rcu_preempt_sleep_check(void) typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \RCU_LOCKDEP_WARN (!(c),"suspicious rcu_dereference_check() usage" ); \rcu_dereference_sparse (p, space); \ +rcu_dereference_access (); \ ((typeof(*p) __force __kernel *)(________p1)); \ })#define __rcu_dereference_protected (p, c, space) \include/linux/srcu.h +3
-0@@ -64,12 +64,18 @@ struct srcu_struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else include/linux/srcu.h struct srcu_struct { · line 78
Quoted 10 lines
include/linux/srcu.h +4
-1+# define RCU_KEY { 0 } +# endif +int __init_srcu_struct (struct srcu_struct *sp,const char *name,struct lock_class_key *key);#define init_srcu_struct (sp) \ ({ \ - static struct lock_class_key __srcu_key; \ + static struct lock_class_key __srcu_key = RCU_KEY; \This gets me the following for the TASKS01, TINY02, TREE02, TREE05, TREE06, and TREE08 configurations:
CC
mm/mmap.o
In file included from/home/paulmck/public_git/linux-rcu/include/linux/notifier.h:15:0,
from/home/paulmck/public_git/linux-rcu/arch/x86/include/asm/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/include/linux/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/kernel/notifier.c:1:/home/paulmck/public_git/linux-rcu/kernel/notifier.c: In function ‘srcu_init_notifier_head’:/home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: missing braces around initializer [-Wmissing-braces]static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^ /home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: (near initialization for ‘__srcu_key.subkeys’) [-Wmissing-braces] static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^Quoted 8 lines
These have the following in their .config files:
CONFIG_DEBUG_LOCK_ALLOC=yHowever, none of your new
Kconfigoptions are set, which needs to be tested because this will be the common case. Several of them haveCONFIG_PROVE_LOCKING=y, but others do not.Oh.. this is embarrassing ;-(
Hmm.. when
CONFIG_RCU_LOCKED_ACCESS=nandCONFIG_DEBUG_LOCK_ALLOC=y, this line becomes:static struct lock_class_key __srcu_key = { 0 };Quoted 8 lines
IIUC, "={ 0 }" is the unverisal zero initializer in C, could be used for zero initialising any structure or array, so it's OK here, right?
And may I ask your compiler's version? Because looks to me this may be a compiler bug according to:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64709gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04) (KSIC)
Quoted 3 lines
Also I can't reproduce this with
CONFIG_RCU_LOCKED_ACCESS=nandCONFIG_DEBUG_LOCK_ALLOC=yon my machine, whose gcc version is 5.3.0.
Quoted 9 lines
Quoted 4 lines
Of course I can use a different way here and do not need a "={ 0 }" here, because
__srcu_keyis static, and I just want to make sure we know what's going on here before we use a workaround, or I just want to know if I'm a bad C programmer ;-)Well, it is quite possible that your code is correct in theory, but we do need to make it so that compilers can deal with it. It would not be the first compiler-bug workaround in RCU...
Fair enough!
Quoted 7 lines
Thanx, Paul
Quoted 5 lines
Regards,
BoqunQuoted 2 lines
I have dequeued these for the moment. Please send an updated patch series when you have this fixed.
Moreover, I hit some "defined but not used" warnings with CONFIG_DEBUG_LOCK_ALLOC=y and CONFIG_RCU_LOCKED_ACCESS=n, will fix that too and send out a new series.
Regards,
Boqun
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Thanx, Paul
Josh Triplett
maintainerRe: [RFC 6/6] RCU: Track rcu_dereference() in RCU read-side critical section
In reply to Boqun FengQuoted 118 lines · Boqun Feng
Quoted 106 lines · Paul E. McKenney
Quoted 72 lines · Boqun Feng
The variables protected by an RCU read-side critical section are sometimes hard to figure out, especially when the critical section is long or has some function calls in it. However, figuring out which variable a RCU read-side critical section protects could save us a lot of time for code reviewing, bug fixing or performance tuning.
This patch therefore uses the
LOCKED_ACCESSto collect the information of relationship between rcu_dereference*() and rcu_read_lock*() by doing:Step 0: define a
locked_access_classfor RCU.Step 1: set the content of rcu_*_lock_key and
__srcu_keyto the address of thelocked_access_classfor RCU.Step 2: add
locked_access_point()in__rcu_dereference_check()After that we can figure out not only in which RCU read-side critical section but also after which rcu_read_lock*() called an rcu_dereference*() is called.
This feature is controlled by a config option
RCU_LOCKED_ACCESS.Also clean up the initialization code of
lockdep_mapsfor different flavors of RCU a little bit.Signed-off-by Boqun Feng <[email protected]>--- include/linux/rcupdate.h | 8 ++++++++ include/linux/srcu.h | 8 +++++++- kernel/locking/lockdep.c | 3 +++ kernel/rcu/update.c | 31 +++++++++++++++++-------------- lib/Kconfig.debug | 12 ++++++++++++ 5 files changed, 47 insertions(+), 15 deletions(-)include/linux/rcupdate.h +8
-0@@ -610,6 +610,13 @@ static inline void rcu_preempt_sleep_check(void)#define rcu_dereference_sparse (p, space)#endif /* #else #ifdef __CHECKER__ */ +#ifdef CONFIG_RCU_LOCKED_ACCESS +extern struct locked_access_class rcu_laclass; +#define rcu_dereference_access () \ + locked_access_point(&rcu_laclass, LOCKED_ACCESS_TYPE_READ) +#else /* #ifdef CONFIG_LOCKED_ACCESS */ +#define rcu_dereference_access () +#endif /* #else #ifdef CONFIG_LOCKED_ACCESS */ #define __rcu_access_pointer (p, space) \ ({ \ typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \@@ -622,6 +629,7 @@ static inline void rcu_preempt_sleep_check(void) typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \RCU_LOCKDEP_WARN (!(c),"suspicious rcu_dereference_check() usage" ); \rcu_dereference_sparse (p, space); \ +rcu_dereference_access (); \ ((typeof(*p) __force __kernel *)(________p1)); \ })#define __rcu_dereference_protected (p, c, space) \include/linux/srcu.h +3
-0@@ -64,12 +64,18 @@ struct srcu_struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC +# ifdef CONFIG_RCU_LOCKED_ACCESS +# define RCU_KEY { .laclass = &rcu_laclass } +# else include/linux/srcu.h struct srcu_struct { · line 78
Quoted 10 lines
include/linux/srcu.h +4
-1+# define RCU_KEY { 0 } +# endif +int __init_srcu_struct (struct srcu_struct *sp,const char *name,struct lock_class_key *key);#define init_srcu_struct (sp) \ ({ \ - static struct lock_class_key __srcu_key; \ + static struct lock_class_key __srcu_key = RCU_KEY; \This gets me the following for the TASKS01, TINY02, TREE02, TREE05, TREE06, and TREE08 configurations:
CC
mm/mmap.o
In file included from/home/paulmck/public_git/linux-rcu/include/linux/notifier.h:15:0,
from/home/paulmck/public_git/linux-rcu/arch/x86/include/asm/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/include/linux/kdebug.h:4,
from/home/paulmck/public_git/linux-rcu/kernel/notifier.c:1:/home/paulmck/public_git/linux-rcu/kernel/notifier.c: In function ‘srcu_init_notifier_head’:/home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: missing braces around initializer [-Wmissing-braces]static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^ /home/paulmck/public_git/linux-rcu/include/linux/srcu.h:78:16: warning: (near initialization for ‘__srcu_key.subkeys’) [-Wmissing-braces] static struct lock_class_key __srcu_key = RCU_KEY; \ ^ /home/paulmck/public_git/linux-rcu/kernel/notifier.c:526:6: note: in expansion of macro ‘init_srcu_struct’ if (init_srcu_struct(&nh->srcu) < 0) ^Quoted 8 lines
These have the following in their .config files:
CONFIG_DEBUG_LOCK_ALLOC=yHowever, none of your new
Kconfigoptions are set, which needs to be tested because this will be the common case. Several of them haveCONFIG_PROVE_LOCKING=y, but others do not.Oh.. this is embarrassing ;-(
Quoted 7 lines
Hmm.. when
CONFIG_RCU_LOCKED_ACCESS=nandCONFIG_DEBUG_LOCK_ALLOC=y, this line becomes:static struct lock_class_key __srcu_key = { 0 };IIUC, "={ 0 }" is the unverisal zero initializer in C, could be used for zero initialising any structure or array, so it's OK here, right?
Compilers seem touchy about that. Sometimes { 0 } works, and sometimes {} works (that really should *always* work, but it doesn't).
However, for a static struct, you can always just leave off the initializer.
- Josh Triplett
Quoted 7 lines · Boqun Feng
As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an
rcu_dereference()might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.
Seeing no objections, I am queuing this series for review and testing. The information it provides would have been extremely helpful to me several times in past!
Thanx, Paul
Quoted 85 lines
In this series, we are introducing
LOCKED_ACCESSframework and based on which, we implement theRCU_LOCKED_ACCESSfunctionality to give us a clear hint: whichrcu_dereference()happens in which RCU read-side critical section.The basic idea of
LOCKED_ACCESSis to maintain a chain of locks we have acquired already, and when there happens a data access, correlate the data access with the chain.Lockdep already has lock chains, but we introduce a new but similar one
concept: acqchain, an acqchain is similar to a lock chain, except that the key of an acqchain is the hash sum of the acquire (instruction) positions of the locks in the chain, whereas the key of a lock chain is the hash sum of the class keys of the locks in the chain.Acqchains are introduced because we want to correlate data accesses with critical sections and critical sections are better represented by the acquire positions rather than lock classes.
The acqchain key of a task is maintained in the same way as lock chain keys in lockdep.
Similar as lockdep,
LOCKED_ACCESSalso classify locks and data accesses by groups, locked access class is introduced for this reason. A locked access class also contains the data for allocation and lookup of acqchains and accesses, and the address of a locked access class is used as its key. By tagging locks and data accesses with the keys, we could describe which locks and data accesses are related.The entry point of
LOCKED_ACCESSislocked_access_point(). Callinglocked_access_point()indicates that a data access happens, and after it called the data access will be correlated with the current acqchain.We also provide a /proc filesystem interface to show the information we've collected, for each locked access class with the name <name> there will be a file at /proc/locked_access/<name> showing all the relationships collected so far for this locked access classes.
Based on
LOCKED_ACCESS, we implementRCU_LOCKED_ACCESS, that tracks rcu_dereference()s inside RCU read-side critical sections.This patchset is based on v4.5-rc2 and consists of 6
patches(in which patch 2-5 are the implementation of LOCKED_ACCESS):1. Introduce some functions of
irq_context.2. Introduce locked access class and acqchain.
3. Maintain the keys of acqchains.
4. Introduce the entry point of
LOCKED_ACCESS.5. Add proc interface for locked access class
6. Enables
LOCKED_ACCESSfor RCU.Tested by 0day and I also did a simple test on x86: build and boot a kernel with RCU_LOCKED_ACCESS=y and
CONFIG_PROVE_LOCKING=yand ran severalworkloads(kernel building, git cloning, dbench), no problem has been observed, and/proc/locked_access/rcuwas able to collect the relationships between ~300 RCU read-critical sections and ~500 rcu_dereference*().Snippets of
/proc/locked_access/rcuare as follow:...(this
rcu_dereference()happens after onercu_read_lock())
...
ACQCHAIN0xfdbf0c6aeea, 1 locks,irq_context0:
LOCK at [<ffffffff812b1115>]get_proc_task_net+0x5/0x140
ACCESS TYPE 1 atkernel/pid.c:441
...
...(thisrcu_dereference()happens after threercu_read_lock())
...
ACQCHAIN0xfe042af3bbfb2605, 3 locks,irq_context0:
LOCK at [<ffffffff81094b47>]SyS_kill+0x97/0x2a0LOCK at [<ffffffff8109286f>] kill_pid_info+0x1f/0x140 LOCK at [<ffffffff81092605>] group_send_sig_info+0x5/0x130 ACCESS TYPE 1 at kernel/signal.c:695...
Looking forwards to any suggestion, comment and question ;-)
Regards,
Boqun