Skip to contentSkip to conversation

Track RCU dereferences in RCU read-side critical sections

RFC In review Boqun Feng 4 participants 21 messages Feb 16, 2016 – Apr 23, 2016

Subsystems (7)
lore
Share & tools
lore

Boqun Feng

author
#0 lore raw

v2 cover Track RCU dereferences in RCU read-side critical sections

Hi all,

This is the v2 for RCU_LOCKED_ACCESS.

Link of v1: http://article.gmane.org/gmane.linux.kernel/2143674

Changes since v1:

* Define newly introduced irq_context helpers as macros when
lockdep doesn't trace irq_context, because these helpers will be quite simple then and this could avoid some "defined but not used" warnings.

* Introduce a new macro to static define a lock_class_key for RCU,
in order to work around a compiler's "bug" about embracer-enclosed initializer list. (Paul Mckenney)

As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an rcu_dereference() might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.

In this series, we are introducing LOCKED_ACCESS framework and based on which, we implement the RCU_LOCKED_ACCESS functionality to give us a clear hint: which rcu_dereference() happens in which RCU read-side critical section.

After this series applied, and if CONFIG_RCU_LOCKED_ACCESS=y, the proc file /proc/locked_access/rcu will show all relationships collected so far for rcu_read_lock() and their friends and rcu_dereference*().

Snippets of /proc/locked_access/rcu are as follow:

...(this rcu_dereference() happens after one rcu_read_lock())
...
ACQCHAIN 0xfdbf0c6aeea, 1 locks, irq_context 0:
LOCK at [<ffffffff812b1115>] get_proc_task_net+0x5/0x140
ACCESS TYPE 1 at kernel/pid.c:441
...
...(this rcu_dereference() happens after three rcu_read_lock())
...
ACQCHAIN 0xfe042af3bbfb2605, 3 locks, irq_context 0:
LOCK at [<ffffffff81094b47>] SyS_kill+0x97/0x2a0

    LOCK at [<ffffffff8109286f>] kill_pid_info+0x1f/0x140
      LOCK at [<ffffffff81092605>] group_send_sig_info+0x5/0x130
        ACCESS TYPE 1 at kernel/signal.c:695

...

This patchset is based on v4.5-rc2 and consists of 6 patches(in which patch 2-5 are the implementation of LOCKED_ACCESS):

1. Introduce some functions of irq_context.

2. Introduce locked access class and acqchain.

3. Maintain the keys of acqchains.

4. Introduce the entry point of LOCKED_ACCESS.

5. Add proc interface for locked access class

6. Enables LOCKED_ACCESS for RCU.

Tested by 0day and I also did a simple test on x86: build and boot a kernel with RCU_LOCKED_ACCESS=y and CONFIG_PROVE_LOCKING=y and ran several workloads(kernel building, git cloning, dbench), and /proc/locked_access/rcu was able to collect the relationships between ~300 RCU read-critical sections and ~500 rcu_dereference*().

Looking forwards to any suggestion, comment and question ;-)

Regards,
Boqun

author
#1 lore raw

RFC v2 1/6 lockdep: Add helper functions of irq_context

Add two helper functions of irq_context:

task_irq_context(): return the encoded irq_context of the task, the return value is encoded in the same as ->irq_context of held_lock. Always return 0 if !(CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING)

is_same_irq_context(): compare whether two irq_context are the same. Always return true if !(CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING)

These two functions are added for the future use of irq_context by LOCKED_ACCESS, because LOCKED_ACCESS needs to get the irq_context value of a task when a data access happens rather than when a lock is acquired, so it needs another way than getting the value from held_lock::irq_context

---
 kernel/locking/lockdep.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
kernel/locking/lockdep.c +14 -2
index 60ace56..f36df37 100644
@@ -2839,6 +2839,16 @@ static int mark_irqflags(struct task_struct *curr, struct held_lock *hlock)
 	return 1;
 }
 
+static unsigned int task_irq_context(struct task_struct *task)
+{
+	return 2 * (task->hardirq_context ? 1 : 0) + task->softirq_context;
+}
+
+static bool is_same_irq_context(unsigned int ctx1, unsigned int ctx2)
+{
+	return ctx1 == ctx2;
+}
+
 static int separate_irq_context(struct task_struct *curr,
 		struct held_lock *hlock)
 {
@@ -2847,8 +2857,7 @@ static int separate_irq_context(struct task_struct *curr,
 	/*
 	 * Keep track of points where we cross into an interrupt context:
 	 */
-	hlock->irq_context = 2*(curr->hardirq_context ? 1 : 0) +
-				curr->softirq_context;
+	hlock->irq_context = task_irq_context(curr);
 	if (depth) {
 		struct held_lock *prev_hlock;
 
@@ -2880,6 +2889,9 @@ static inline int mark_irqflags(struct task_struct *curr,
 	return 1;
 }
 
+#define task_irq_context(task)			0
+#define is_same_irq_context(ctx1, ctx2)		true
+
 static inline int separate_irq_context(struct task_struct *curr,
 		struct held_lock *hlock)
 {
Signature · 1 line
2.7.1
#2 lore raw

Re: [RFC v2 1/6] lockdep: Add helper functions of irq_context

Quoted 9 lines · Boqun Feng

Add two helper functions of irq_context:

task_irq_context(): return the encoded irq_context of the task, the return value is encoded in the same as ->irq_context of held_lock. Always return 0 if !(CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING)

is_same_irq_context(): compare whether two irq_context are the same. Always return true if !(CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING)

Quoted 7 lines

These two functions are added for the future use of irq_context by LOCKED_ACCESS, because LOCKED_ACCESS needs to get the irq_context value of a task when a data access happens rather than when a lock is acquired, so it needs another way than getting the value from held_lock::irq_context

Queued again for review and testing. ;-)

Thanx, Paul

Quoted 48 lines
---
 kernel/locking/lockdep.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
kernel/locking/lockdep.c +14 -2
index 60ace56..f36df37 100644
@@ -2839,6 +2839,16 @@ static int mark_irqflags(struct task_struct *curr, struct held_lock *hlock)
 	return 1;
 }
 
+static unsigned int task_irq_context(struct task_struct *task)
+{
+	return 2 * (task->hardirq_context ? 1 : 0) + task->softirq_context;
+}
+
+static bool is_same_irq_context(unsigned int ctx1, unsigned int ctx2)
+{
+	return ctx1 == ctx2;
+}
+
 static int separate_irq_context(struct task_struct *curr,
 		struct held_lock *hlock)
 {
@@ -2847,8 +2857,7 @@ static int separate_irq_context(struct task_struct *curr,
 	/*
 	 * Keep track of points where we cross into an interrupt context:
 	 */
-	hlock->irq_context = 2*(curr->hardirq_context ? 1 : 0) +
-				curr->softirq_context;
+	hlock->irq_context = task_irq_context(curr);
 	if (depth) {
 		struct held_lock *prev_hlock;
 
@@ -2880,6 +2889,9 @@ static inline int mark_irqflags(struct task_struct *curr,
 	return 1;
 }
 
+#define task_irq_context(task)			0
+#define is_same_irq_context(ctx1, ctx2)		true
+
 static inline int separate_irq_context(struct task_struct *curr,
 		struct held_lock *hlock)
 {
Signature · 1 line
2.7.1
#3 lore raw

[tip:locking/urgent] locking/lockdep: Fix ->irq_context calculation

In reply to Boqun Feng
locking/lockdep: Fix ->irq_context calculation
Commit-ID:  c24697566298df04cac9913e0601501b5ee2b3f5
Gitweb:     http://git.kernel.org/tip/c24697566298df04cac9913e0601501b5ee2b3f5
Author:     Boqun Feng <[email protected]>
AuthorDate: Tue, 16 Feb 2016 13:57:40 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 23 Apr 2016 13:53:03 +0200

locking/lockdep: Fix ->irq_context calculation

task_irq_context() returns the encoded irq_context of the task, the return value is encoded in the same as ->irq_context of held_lock.

Always return 0 if !(CONFIG_TRACE_IRQFLAGS && CONFIG_PROVE_LOCKING)

Cc Andrew Morton <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/locking/lockdep.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
kernel/locking/lockdep.c +11 -2
index ed94109..beb06f6 100644
@@ -2932,6 +2932,11 @@ static int mark_irqflags(struct task_struct *curr, struct held_lock *hlock)
 	return 1;
 }
 
+static inline unsigned int task_irq_context(struct task_struct *task)
+{
+	return 2 * !!task->hardirq_context + !!task->softirq_context;
+}
+
 static int separate_irq_context(struct task_struct *curr,
 		struct held_lock *hlock)
 {
@@ -2940,8 +2945,6 @@ static int separate_irq_context(struct task_struct *curr,
 	/*
 	 * Keep track of points where we cross into an interrupt context:
 	 */
-	hlock->irq_context = 2*(curr->hardirq_context ? 1 : 0) +
-				curr->softirq_context;
 	if (depth) {
 		struct held_lock *prev_hlock;
 
@@ -2973,6 +2976,11 @@ static inline int mark_irqflags(struct task_struct *curr,
 	return 1;
 }
 
+static inline unsigned int task_irq_context(struct task_struct *task)
+{
+	return 0;
+}
+
 static inline int separate_irq_context(struct task_struct *curr,
 		struct held_lock *hlock)
 {
@@ -3241,6 +3249,7 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
 	hlock->acquire_ip = ip;
 	hlock->instance = lock;
 	hlock->nest_lock = nest_lock;
+	hlock->irq_context = task_irq_context(curr);
 	hlock->trylock = trylock;
 	hlock->read = read;
 	hlock->check = check;
author
#4 lore raw

RFC v2 2/6 lockdep: LOCKED_ACCESS: Introduce locked access class and acqchain

In reply to Boqun Feng

As a similar concept as a lock class, a locked access class is a group of locks and related data accesses of those locks. A locked access class also contains the structures for allocation and lookup of acqchains and accesses.

The address of a locked access class is used as its key, we tag a group of locks with a locked access class by setting the content of the group's lock_class_key to be the key(i.e. address) of locked access class, and we can do this trick because, AFAIK, lockdep only uses the address of a lock_class_key. However, although this doesn't change the size of a lock_class_key, this does change the alignment requirement.

And tagging a lock_class_key with a locked access class is the step #1 to use LOCKED_ACCESS for a locking mechanism.

As a similar concept as a lock class chain, an acqchain is short for acquire chain, which is a chain of lock acquisitions. The difference between a lock class chain and an acqchain is that a lock class chain is keyed by the hash sum of the class keys, whereas an acqchain is keyed by the hash sum of the acquire ips. We introduce acqchains here because in LOCKED_ACCESS, we care more about the locations of critical sections.

This patch defines the structures for those concepts, along with the manipulation operations. Besides, config option CONFIG_LOCKED_ACCESS is introduced in this patch, too.

---
 include/linux/lockdep.h            |  17 ++++
 kernel/locking/lockdep.c           | 156 +++++++++++++++++++++++++++++++++++++
 kernel/locking/lockdep_internals.h |  74 ++++++++++++++++++
 lib/Kconfig.debug                  |  12 +++
 4 files changed, 259 insertions(+)
include/linux/lockdep.h +17 -0
index c57e424..140c1c3 100644
@@ -51,9 +51,26 @@ struct lockdep_subclass_key {
 	char __one_byte;
 } __attribute__ ((__packed__));
 
+#ifdef CONFIG_LOCKED_ACCESS
+struct locked_access_class;
+
+struct lock_class_key {
+	union {
+		struct lockdep_subclass_key subkeys[MAX_LOCKDEP_SUBCLASSES];
+		/*
+		 * Use the content of the lock_class_key to store locked access
+		 * class, as lockdep only use the address of a lock_class_key.
+		 * However, please note by doing so, we will change the align
+		 * requirement of lock_class_key
+		 */
+		struct locked_access_class *laclass;
+	};
+};
+#else
 struct lock_class_key {
 	struct lockdep_subclass_key	subkeys[MAX_LOCKDEP_SUBCLASSES];
 };
+#endif
 
 extern struct lock_class_key __lockdep_no_validate__;
 
kernel/locking/lockdep.c +156 -0

159 lines not on this page · raw message

kernel/locking/lockdep_internals.h +74 -0

77 lines not on this page · raw message

lib/Kconfig.debug +12 -0
index ecb9e75..178f288 100644
@@ -1065,6 +1065,18 @@ config LOCK_STAT
 	 CONFIG_LOCK_STAT defines "contended" and "acquired" lock events.
 	 (CONFIG_LOCKDEP defines "acquire" and "release" events.)
 
+config LOCKED_ACCESS
+	bool
+	depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	select DEBUG_LOCK_ALLOC
+	default n
+	help
+	  Provide a way to correlate data accesses with critical sections,
+	  LOCKED_ACCESS is designed for the users of locking mechanisms to see
+	  which data access happens inside which lock critical section. This
+	  could help the users of a lock mechanism to gain more detailed
+	  information about their lock usage.
+
 config DEBUG_LOCKDEP
 	bool "Lock dependency engine debugging"
 	depends on DEBUG_KERNEL && LOCKDEP
Signature · 1 line
2.7.1
author
#5 lore raw

RFC v2 3/6 lockdep: LOCKED_ACCESS: Maintain the keys of acqchains

In reply to Boqun Feng

Add held_lock::prev_acqchain_key and task_struct::curr_acqchain_key to maintain the keys of acqchains as the same as what lockdep does for keys of lock classes.

---
 include/linux/lockdep.h  |  3 +++
 include/linux/sched.h    |  3 +++
 kernel/fork.c            |  3 +++
 kernel/locking/lockdep.c | 31 +++++++++++++++++++++++++++++++
 4 files changed, 40 insertions(+)
include/linux/lockdep.h +3 -0
index 140c1c3..2ffe6c3 100644
@@ -244,6 +244,9 @@ struct held_lock {
 	 * with zero), here we store the previous hash value:
 	 */
 	u64				prev_chain_key;
+#ifdef CONFIG_LOCKED_ACCESS
+	u64				prev_acqchain_key;
+#endif
 	unsigned long			acquire_ip;
 	struct lockdep_map		*instance;
 	struct lockdep_map		*nest_lock;
include/linux/sched.h +3 -0
index a10494a..c24348e 100644
@@ -1643,6 +1643,9 @@ struct task_struct {
 	unsigned int lockdep_recursion;
 	struct held_lock held_locks[MAX_LOCK_DEPTH];
 	gfp_t lockdep_reclaim_gfp;
+#ifdef CONFIG_LOCKED_ACCESS
+	u64 curr_acqchain_key;
+#endif
 #endif
 #ifdef CONFIG_UBSAN
 	unsigned int in_ubsan;
kernel/fork.c +3 -0
index 2e391c7..d9fc51b 100644
@@ -1404,6 +1404,9 @@ static struct task_struct *copy_process(unsigned long clone_flags,
 	p->lockdep_depth = 0; /* no locks held yet */
 	p->curr_chain_key = 0;
 	p->lockdep_recursion = 0;
+# ifdef CONFIG_LOCKED_ACCESS
+	p->curr_acqchain_key = 0;
+# endif
 #endif
 
 #ifdef CONFIG_DEBUG_MUTEXES
kernel/locking/lockdep.c +31 -0
index fdb1b8c..3ebd00d 100644
@@ -3089,6 +3089,9 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
 	int chain_head = 0;
 	int class_idx;
 	u64 chain_key;
+#ifdef CONFIG_LOCKED_ACCESS
+	u64 acqchain_key;
+#endif
 
 	if (unlikely(!debug_locks))
 		return 0;
@@ -3196,6 +3199,9 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
 		return 0;
 
 	chain_key = curr->curr_chain_key;
+#ifdef CONFIG_LOCKED_ACCESS
+	acqchain_key = curr->curr_acqchain_key;
+#endif
 	if (!depth) {
 		/*
 		 * How can we have a chain hash when we ain't got no keys?!
@@ -3206,12 +3212,23 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
 	}
 
 	hlock->prev_chain_key = chain_key;
+#ifdef CONFIG_LOCKED_ACCESS
+	hlock->prev_acqchain_key = acqchain_key;
+#endif
 	if (separate_irq_context(curr, hlock)) {
 		chain_key = 0;
+
+#ifdef CONFIG_LOCKED_ACCESS
+		acqchain_key = 0;
+#endif
+
 		chain_head = 1;
 	}
 	chain_key = iterate_chain_key(chain_key, id);
 
+#ifdef CONFIG_LOCKED_ACCESS
+	acqchain_key = iterate_acqchain_key(acqchain_key, ip);
+#endif
 	if (nest_lock && !__lock_is_held(nest_lock))
 		return print_lock_nested_lock_not_held(curr, hlock, ip);
 
@@ -3219,6 +3236,9 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
 		return 0;
 
 	curr->curr_chain_key = chain_key;
+#ifdef CONFIG_LOCKED_ACCESS
+	curr->curr_acqchain_key = acqchain_key;
+#endif
 	curr->lockdep_depth++;
 	check_chain_key(curr);
 #ifdef CONFIG_DEBUG_LOCKDEP
@@ -3348,6 +3368,9 @@ found_it:
 
 	curr->lockdep_depth = i;
 	curr->curr_chain_key = hlock->prev_chain_key;
+#ifdef CONFIG_LOCKED_ACCESS
+	curr->curr_acqchain_key = hlock->prev_acqchain_key;
+#endif
 
 	for (; i < depth; i++) {
 		hlock = curr->held_locks + i;
@@ -3438,6 +3461,9 @@ found_it:
 
 	curr->lockdep_depth = i;
 	curr->curr_chain_key = hlock->prev_chain_key;
+#ifdef CONFIG_LOCKED_ACCESS
+	curr->curr_acqchain_key = hlock->prev_acqchain_key;
+#endif
 
 	for (i++; i < depth; i++) {
 		hlock = curr->held_locks + i;
@@ -3879,6 +3905,11 @@ void lockdep_reset(void)
 
 	raw_local_irq_save(flags);
 	current->curr_chain_key = 0;
+
+#ifdef CONFIG_LOCKED_ACCESS
+	current->curr_acqchain_key = 0;
+#endif
+
 	current->lockdep_depth = 0;
 	current->lockdep_recursion = 0;
 	memset(current->held_locks, 0, MAX_LOCK_DEPTH*sizeof(struct held_lock));
Signature · 1 line
2.7.1
author
#6 lore raw

RFC v2 4/6 lockdep: LOCKED_ACCESS: Introduce locked_access_point()

In reply to Boqun Feng

locked_access_point() is the entry point of the whole LOCKED_ACCESS framework, every time a locked_access_point() is called, LOCKE_ACCESS will correlate the data access location with the current acqchain.

So putting locked_access_point() at the data accesses you care about is the step #2 to use LOCKED_ACCESS.

---
 include/linux/lockdep.h  |  29 ++++++++++++
 kernel/locking/lockdep.c | 121 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 150 insertions(+)
include/linux/lockdep.h +29 -0
index 2ffe6c3..cf7b6c8 100644
@@ -570,4 +570,33 @@ lockdep_rcu_suspicious(const char *file, const int line, const char *s)
 }
 #endif
 
+#ifdef CONFIG_LOCKED_ACCESS
+struct locked_access_location {
+	/* Filename of the access */
+	const char			*filename;
+	/* Line number of the access */
+	long				lineno;
+};
+
+#define LOCKED_ACCESS_TYPE_READ		1 /* read */
+
+extern void locked_access(struct locked_access_class *laclass,
+			  struct locked_access_location *access,
+			  int type);
+
+/*
+ * Entry point of LOCKED_ACCESS, should be called at every place the data
+ * accesses of the laclass happen.
+ *
+ * @_type must be one of the LOCKED_ACCESS_TYPE_*
+ */
+#define locked_access_point(_laclass, _type) \
+({ \
+	static struct locked_access_location a = { \
+				.filename = __FILE__, \
+				.lineno = __LINE__, \
+	}; \
+	locked_access(_laclass, &a, _type); \
+})
+#endif /* CONFIG_LOCKED_ACCESS */
 #endif /* __LINUX_LOCKDEP_H */
kernel/locking/lockdep.c +121 -0

125 lines not on this page · raw message

Signature · 1 line
2.7.1
author
#7 lore raw

RFC v2 5/6 lockdep: LOCKED_ACCESS: Add proc interface for locked access class

In reply to Boqun Feng

Provide the proc filesystem interface for LOCKED_ACCESS, for a locked access class whose name is <name>, there will be a file at /proc/locked_access/<name> containing all the information the LOCKED_ACCESS has collected so far.

Also add a macro to define a locked access class.

---
 kernel/locking/lockdep_internals.h |  17 +++++
 kernel/locking/lockdep_proc.c      | 127 +++++++++++++++++++++++++++++++++++++
 2 files changed, 144 insertions(+)
kernel/locking/lockdep_internals.h +17 -0
index 5e2e133..cacf869 100644
@@ -241,4 +241,21 @@ struct locked_access_class {
 		.nr_acqchain_hlocks = 0,\
 		.nr_access_structs = 0, \
 	}
+
+extern int create_laclass_proc(const char *name,
+			   struct locked_access_class *laclass);
+
+#define DEFINE_CREATE_LACLASS_PROC(name) \
+static int __init ___create_##name##_laclass_proc(void) \
+{ \
+	return create_laclass_proc(#name, &name##_laclass); \
+} \
+late_initcall(___create_##name##_laclass_proc)
+
+/* Define a Locked Access Class and create its proc file */
+#define DEFINE_LACLASS(name) \
+	struct locked_access_class name##_laclass = \
+			INIT_LOCKED_ACCESS_DATA(name); \
+	EXPORT_SYMBOL(name##_laclass); \
+	DEFINE_CREATE_LACLASS_PROC(name)
 #endif /* CONFIG_LOCKED_ACCESS */
kernel/locking/lockdep_proc.c +127 -0

139 lines not on this page · raw message

Signature · 1 line
2.7.1
author
#8 lore raw

RFC v2 6/6 RCU: Track rcu_dereference() in RCU read-side critical section

In reply to Boqun Feng

The variables protected by an RCU read-side critical section are sometimes hard to figure out, especially when the critical section is long or has some function calls in it. However, figuring out which variable a RCU read-side critical section protects could save us a lot of time for code reviewing, bug fixing or performance tuning.

This patch therefore uses the LOCKED_ACCESS to collect the information of relationship between rcu_dereference*() and rcu_read_lock*() by doing:

Step 0: define a locked_access_class for RCU.

Step 1: set the content of rcu_*_lock_key and __srcu_key to the address of the locked_access_class for RCU.

Step 2: add locked_access_point() in __rcu_dereference_check()

After that we can figure out not only in which RCU read-side critical section but also after which rcu_read_lock*() called an rcu_dereference*() is called.

This feature is controlled by a config option RCU_LOCKED_ACCESS.

Also clean up the initialization code of lockdep_maps for different flavors of RCU a little bit.

---
 include/linux/rcupdate.h | 12 ++++++++++++
 include/linux/srcu.h     |  2 +-
 kernel/locking/lockdep.c |  3 +++
 kernel/rcu/update.c      | 25 +++++++++++--------------
 lib/Kconfig.debug        | 12 ++++++++++++
 5 files changed, 39 insertions(+), 15 deletions(-)
include/linux/rcupdate.h +12 -0
index 14e6f47..7ca22c1 100644
@@ -610,6 +610,17 @@ static inline void rcu_preempt_sleep_check(void)
 #define rcu_dereference_sparse(p, space)
 #endif /* #else #ifdef __CHECKER__ */
 
+#ifdef CONFIG_RCU_LOCKED_ACCESS
+extern struct locked_access_class rcu_laclass;
+#define STATIC_DEFINE_RCU_LOCK_CLASS_KEY(name) \
+	static struct lock_class_key name = { .laclass = &rcu_laclass }
+#define rcu_dereference_access() \
+	locked_access_point(&rcu_laclass, LOCKED_ACCESS_TYPE_READ)
+#else /* #ifdef CONFIG_LOCKED_ACCESS */
+#define rcu_dereference_access()
+#define STATIC_DEFINE_RCU_LOCK_CLASS_KEY(name) \
+	static struct lock_class_key name
+#endif /* #else #ifdef CONFIG_LOCKED_ACCESS */
 #define __rcu_access_pointer(p, space) \
 ({ \
 	typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \
@@ -622,6 +633,7 @@ static inline void rcu_preempt_sleep_check(void)
 	typeof(*p) *________p1 = (typeof(*p) *__force)lockless_dereference(p); \
 	RCU_LOCKDEP_WARN(!(c), "suspicious rcu_dereference_check() usage"); \
 	rcu_dereference_sparse(p, space); \
+	rcu_dereference_access(); \
 	((typeof(*p) __force __kernel *)(________p1)); \
 })
 #define __rcu_dereference_protected(p, c, space) \
include/linux/srcu.h +1 -1
index f5f80c5..ba4bcbe 100644
@@ -69,7 +69,7 @@ int __init_srcu_struct(struct srcu_struct *sp, const char *name,
 
 #define init_srcu_struct(sp) \
 ({ \
-	static struct lock_class_key __srcu_key; \
+	STATIC_DEFINE_RCU_LOCK_CLASS_KEY(__srcu_key); \
 	\
 	__init_srcu_struct((sp), #sp, &__srcu_key); \
 })
kernel/locking/lockdep.c +3 -0
index 4094757..c6c25c5 100644
@@ -4616,5 +4616,8 @@ void locked_access(struct locked_access_class *laclass,
 		correlate_locked_access(laclass, acqchain, loc, type);
 }
 EXPORT_SYMBOL(locked_access);
+#ifdef CONFIG_RCU_LOCKED_ACCESS
+DEFINE_LACLASS(rcu);
+#endif
 
 #endif /* CONFIG_LOCKED_ACCESS */
kernel/rcu/update.c +11 -14
index 76b94e1..ecb29aa 100644
@@ -235,20 +235,17 @@ EXPORT_SYMBOL_GPL(__rcu_read_unlock);
 #endif /* #ifdef CONFIG_PREEMPT_RCU */
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
-static struct lock_class_key rcu_lock_key;
-struct lockdep_map rcu_lock_map =
-	STATIC_LOCKDEP_MAP_INIT("rcu_read_lock", &rcu_lock_key);
-EXPORT_SYMBOL_GPL(rcu_lock_map);
-
-static struct lock_class_key rcu_bh_lock_key;
-struct lockdep_map rcu_bh_lock_map =
-	STATIC_LOCKDEP_MAP_INIT("rcu_read_lock_bh", &rcu_bh_lock_key);
-EXPORT_SYMBOL_GPL(rcu_bh_lock_map);
-
-static struct lock_class_key rcu_sched_lock_key;
-struct lockdep_map rcu_sched_lock_map =
-	STATIC_LOCKDEP_MAP_INIT("rcu_read_lock_sched", &rcu_sched_lock_key);
-EXPORT_SYMBOL_GPL(rcu_sched_lock_map);
+
+#define DEFINE_RCU_LOCKDEP_MAP(flavor)					\
+	STATIC_DEFINE_RCU_LOCK_CLASS_KEY(rcu##flavor##_lock_key);	\
+	struct lockdep_map rcu##flavor##_lock_map =			\
+		STATIC_LOCKDEP_MAP_INIT("rcu" #flavor "_read_lock",	\
+					&rcu##flavor##_lock_key);	\
+	EXPORT_SYMBOL_GPL(rcu##flavor##_lock_map)
+
+DEFINE_RCU_LOCKDEP_MAP();
+DEFINE_RCU_LOCKDEP_MAP(_bh);
+DEFINE_RCU_LOCKDEP_MAP(_sched);
 
 static struct lock_class_key rcu_callback_key;
 struct lockdep_map rcu_callback_map =
lib/Kconfig.debug +12 -0
index 178f288..b6003d4 100644
@@ -1410,6 +1410,18 @@ config RCU_EQS_DEBUG
 	  Say N here if you need ultimate kernel/user switch latencies
 	  Say Y if you are unsure
 
+config RCU_LOCKED_ACCESS
+	bool "Track data access in RCU read-side critical sections"
+	depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	select LOCKED_ACCESS
+	default n
+	help
+	  Track data acces in RCU read-side critical sections,
+	  by doing so, one can examine which rcu_dereference() and its
+	  friends are called in which RCU read-side critical sections,
+	  and even more detailed, after which rcu_read_lock() and
+	  its friends are called.
+
 endmenu # "RCU Debugging"
 
 config DEBUG_BLOCK_EXT_DEVT
Signature · 1 line
2.7.1
maintainer
#9 lore raw
In reply to Boqun Feng
Quoted 8 lines · Boqun Feng

As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an rcu_dereference() might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.

Quoted 9 lines

In this series, we are introducing LOCKED_ACCESS framework and based on which, we implement the RCU_LOCKED_ACCESS functionality to give us a clear hint: which rcu_dereference() happens in which RCU read-side critical section.

After this series applied, and if CONFIG_RCU_LOCKED_ACCESS=y, the proc file /proc/locked_access/rcu will show all relationships collected so far for rcu_read_lock() and their friends and rcu_dereference*().

But why !? What does this bring us, why do I want to even look at these patches?

#10 lore raw
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Boqun Feng

As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an rcu_dereference() might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.

In this series, we are introducing LOCKED_ACCESS framework and based on which, we implement the RCU_LOCKED_ACCESS functionality to give us a clear hint: which rcu_dereference() happens in which RCU read-side critical section.

Quoted 6 lines
Quoted 3 lines

After this series applied, and if CONFIG_RCU_LOCKED_ACCESS=y, the proc file /proc/locked_access/rcu will show all relationships collected so far for rcu_read_lock() and their friends and rcu_dereference*().

But why !? What does this bring us, why do I want to even look at these patches?

There were some complaints about the difficulty of figuring out what was being protected by a given rcu_read_lock() in cases where the corresponding rcu_dereference() is several function calls down, and especially in cases where the function calls are via pointers. These cases show up in a number of places, perhaps most prominently in networking.

Boqun's patches therefore use lockdep to make an association between each rcu_dereference() and the rcu_read_lock() protecting it.

Seem reasonable, or were the complaints just a flash in the pan?

Thanx, Paul

author
#11 lore raw
Quoted 29 lines · Paul E. McKenney
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Boqun Feng

As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an rcu_dereference() might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.

In this series, we are introducing LOCKED_ACCESS framework and based on which, we implement the RCU_LOCKED_ACCESS functionality to give us a clear hint: which rcu_dereference() happens in which RCU read-side critical section.

Quoted 6 lines
Quoted 3 lines

After this series applied, and if CONFIG_RCU_LOCKED_ACCESS=y, the proc file /proc/locked_access/rcu will show all relationships collected so far for rcu_read_lock() and their friends and rcu_dereference*().

But why !? What does this bring us, why do I want to even look at these patches?

There were some complaints about the difficulty of figuring out what was being protected by a given rcu_read_lock() in cases where the corresponding rcu_dereference() is several function calls down, and especially in cases where the function calls are via pointers. These cases show up in a number of places, perhaps most prominently in networking.

Quoted 3 lines

Boqun's patches therefore use lockdep to make an association between each rcu_dereference() and the rcu_read_lock() protecting it.

Yes, this patch is aiming to provide some information about this, actually the background of this patchset is a discussion between Ingo and Paul last year:

http://lkml.kernel.org/g/[email protected]

In that discussion, Ingo gave one example about how hard it is to figure out what a RCU read-side critical section is protecting. Ingo's proposal of solving this problem was to add an extra parameter for RCU related primitives, however, Paul seemed to have concerns about that approach, because there were many RCU users' code that would need to be modified and there were corner cases where the one extra parameter was not enough or not necessary.

Therefore I tried to figure out a way for making the association of rcu_dereference() and rcu_read_lock() automatically without the modification of code of the RCU users. That's how this patchset comes.

Regards,
Boqun

Quoted 4 lines

Seem reasonable, or were the complaints just a flash in the pan?

Thanx, Paul

maintainer
#12 lore raw
Quoted 47 lines · Boqun Feng
Quoted 22 lines · Paul E. McKenney
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Boqun Feng

As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an rcu_dereference() might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.

In this series, we are introducing LOCKED_ACCESS framework and based on which, we implement the RCU_LOCKED_ACCESS functionality to give us a clear hint: which rcu_dereference() happens in which RCU read-side critical section.

Quoted 6 lines
Quoted 3 lines

After this series applied, and if CONFIG_RCU_LOCKED_ACCESS=y, the proc file /proc/locked_access/rcu will show all relationships collected so far for rcu_read_lock() and their friends and rcu_dereference*().

But why !? What does this bring us, why do I want to even look at these patches?

Quoted 9 lines

There were some complaints about the difficulty of figuring out what was being protected by a given rcu_read_lock() in cases where the corresponding rcu_dereference() is several function calls down, and especially in cases where the function calls are via pointers. These cases show up in a number of places, perhaps most prominently in networking.
Boqun's patches therefore use lockdep to make an association between each rcu_dereference() and the rcu_read_lock() protecting it.

Yes, this patch is aiming to provide some information about this, actually the background of this patchset is a discussion between Ingo and Paul last year:

http://lkml.kernel.org/g/[email protected]

In that discussion, Ingo gave one example about how hard it is to figure out what a RCU read-side critical section is protecting. Ingo's proposal of solving this problem was to add an extra parameter for RCU related primitives, however, Paul seemed to have concerns about that approach, because there were many RCU users' code that would need to be modified and there were corner cases where the one extra parameter was not enough or not necessary.

Quoted 4 lines

Therefore I tried to figure out a way for making the association of rcu_dereference() and rcu_read_lock() automatically without the modification of code of the RCU users. That's how this patchset comes.

So the problem I have with this is that your text didn't have a problem statement, and if there's no problem, what the heck is all this code for.

Secondly, I'm not sure a random list of data is going to help anybody.

The whole point of lockdep is that it tells you if you do broken. This doesn't, and cannot, because it lacks information to verify against.

So I'm still not sure this is useful. Also, I would argue your code has problems if you cannot even find your rcu_read_lock().

And yes, networking is a special snowflake, they very often push their rcu_read_lock(),local_bh_disable(),preempt_disable() whatnot, all the way out, because cycles.

Which makes it entirely impossible to tell what all it guards. That's a very concious choice they make, but its a royal pain for a number of things. I'm not sure we can do anything about it.

So no, I'm not convinced. Again, why is this useful?

author
#13 lore raw
Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Boqun Feng
Quoted 22 lines · Paul E. McKenney
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Boqun Feng

As a characteristic of RCU, read-side critical sections have a very loose connection with rcu_dereference()s, which is you can only be sure about an rcu_dereference() might be called in some read-side critical section, but if code gets complex, you may not be sure which read-side critical section exactly, this might be also an problem for some other locking mechanisms, that is the critical sections protecting data and the data accesses protected are not clearly correlated.

In this series, we are introducing LOCKED_ACCESS framework and based on which, we implement the RCU_LOCKED_ACCESS functionality to give us a clear hint: which rcu_dereference() happens in which RCU read-side critical section.

Quoted 6 lines
Quoted 3 lines

After this series applied, and if CONFIG_RCU_LOCKED_ACCESS=y, the proc file /proc/locked_access/rcu will show all relationships collected so far for rcu_read_lock() and their friends and rcu_dereference*().

But why !? What does this bring us, why do I want to even look at these patches?

Quoted 9 lines

There were some complaints about the difficulty of figuring out what was being protected by a given rcu_read_lock() in cases where the corresponding rcu_dereference() is several function calls down, and especially in cases where the function calls are via pointers. These cases show up in a number of places, perhaps most prominently in networking.
Boqun's patches therefore use lockdep to make an association between each rcu_dereference() and the rcu_read_lock() protecting it.

Yes, this patch is aiming to provide some information about this, actually the background of this patchset is a discussion between Ingo and Paul last year:

http://lkml.kernel.org/g/[email protected]

In that discussion, Ingo gave one example about how hard it is to figure out what a RCU read-side critical section is protecting. Ingo's proposal of solving this problem was to add an extra parameter for RCU related primitives, however, Paul seemed to have concerns about that approach, because there were many RCU users' code that would need to be modified and there were corner cases where the one extra parameter was not enough or not necessary.

Quoted 9 lines
Quoted 4 lines

Therefore I tried to figure out a way for making the association of rcu_dereference() and rcu_read_lock() automatically without the modification of code of the RCU users. That's how this patchset comes.

So the problem I have with this is that your text didn't have a problem statement, and if there's no problem, what the heck is all this code for.

You are right, that's my bad. I should have done this better. Sorry about that..

Please allow me to restate the problem and what this patchset can help ;-)

The problem is:

Currently there is no way to know which rcu_dereference() and its friends a rcu_read_lock() or one of its friends is protecting. And the lack of such information is a pain for kernel developers and reviewers to understand the purpose of a certain rcu_read_lock().

And this patchset is trying to provide the information to developers.

Quoted 2 lines

Secondly, I'm not sure a random list of data is going to help anybody.

One possible usage might be:

If someone is going to refactor a piece of code which originally uses RCU for synchronization, and now he wants to switch to other synchronization mechanism for some data because of some reason(for consistency or writer side latency guarantee, etc.). Before he makes the change, he needs to know if this rcu_read_lock() also protects other data, and this is not an easy job and I don't think we have anything helpful for this. With RCU_LOCKED_ACCESS, he can build the kernel with RCU_LOCKED_ACCESS enabled, run some workloads and get the information from a /proc file. In this way, RCU_LOCKED_ACCESS is helpful to developers who want to know the information of the associations between rcu_read_lock() and rcu_dereference().

Quoted 3 lines

The whole point of lockdep is that it tells you if you do broken. This doesn't, and cannot, because it lacks information to verify against.

Yes.. RCU_LOCKED_ACCESS is not a validator, so it cannot be as useful as lockdep for finding bug.

Quoted 3 lines

So I'm still not sure this is useful. Also, I would argue your code has problems if you cannot even find your rcu_read_lock().

I think what you mean here is, for example, the case where we use preempt_disable() instead of rcu_read_lock_sched() to pair with synchronize_sched(), right?

If so, sure, this series cannot do anything about this, however, this series is to correlate rcu_read_lock() and its friends with rcu_dereference() and its friends. So it may miss some RCU usages if rcu_read_lock() and its friends are not used. But these missing usages can be covered in a similar way as this series, I can either add it in the next version of this series or provide it as a separate follow-up patchset, of course, if we all agree the problem is worth solving and this patchset is helpful.

Or I'm missing your point here?

Quoted 9 lines

And yes, networking is a special snowflake, they very often push their rcu_read_lock(),local_bh_disable(),preempt_disable() whatnot, all the way out, because cycles.

Which makes it entirely impossible to tell what all it guards. That's a very concious choice they make, but its a royal pain for a number of things. I'm not sure we can do anything about it.

So no, I'm not convinced. Again, why is this useful?

Sorry again for not providing enough text for you to review this series, hope now it's more clear.

Looking forwards to your thoughts, thank you ;-)

If you think this series deserves a reconsideration I will send out a new version with more text about what's the problem and why this is useful.

Regards,
Boqun

maintainer
#14 lore raw
Quoted 6 lines · Boqun Feng

The problem is:

Currently there is no way to know which rcu_dereference() and its friends a rcu_read_lock() or one of its friends is protecting. And the lack of such information is a pain for kernel developers and reviewers to understand the purpose of a certain rcu_read_lock().

If I have to go run a kernel and look at /proc data in order to complete a review the patch will fail review, end of story.

Quoted 5 lines
Quoted 2 lines

Secondly, I'm not sure a random list of data is going to help anybody.

One possible usage might be:

Quoted 11 lines

If someone is going to refactor a piece of code which originally uses RCU for synchronization, and now he wants to switch to other synchronization mechanism for some data because of some reason(for consistency or writer side latency guarantee, etc.). Before he makes the change, he needs to know if this rcu_read_lock() also protects other data, and this is not an easy job and I don't think we have anything helpful for this. With RCU_LOCKED_ACCESS, he can build the kernel with RCU_LOCKED_ACCESS enabled, run some workloads and get the information from a /proc file. In this way, RCU_LOCKED_ACCESS is helpful to developers who want to know the information of the associations between rcu_read_lock() and rcu_dereference().

So 'possible', but will anyone actually do this? `

Quoted 6 lines
Quoted 3 lines

The whole point of lockdep is that it tells you if you do broken. This doesn't, and cannot, because it lacks information to verify against.

Yes.. RCU_LOCKED_ACCESS is not a validator, so it cannot be as useful as lockdep for finding bug.

I would try really _really_ hard to make it a validator, those are so much more useful.

One could for example allow something like:

	rcu_read_lock();
	rcu_annotate(&var->field);

	foo();

	rcu_read_unlock();

As an alternative to the syntax suggested by Ingo. This would allow keeping the existing rcu_read_lock() signature so you don't have to force update the entire kernel at once, while also (easily) allowing multiple variables. Like:

	rcu_read_lock();
	rcu_annotate(&var->field);
	rcu_annotate(&var2->field2);

You can then have a special rule that if a particular RCU section has an annotation, any rcu_dereference() not matched will field a warning. If the annotation section is empty, nothing.

Quoted 7 lines
Quoted 3 lines

So I'm still not sure this is useful. Also, I would argue your code has problems if you cannot even find your rcu_read_lock().

I think what you mean here is, for example, the case where we use preempt_disable() instead of rcu_read_lock_sched() to pair with synchronize_sched(), right?

No, I was more like:

	rcu_read_lock();
	foo()
	  bar()
	    var->func();
	      obj->func();
	        whatever();

and you're looking at a change to whatever() and wonder where the heck the corresponding rcu_read_lock() lives and if we're having it held at all.

Quoted 1 line

If so, sure, this series cannot do anything about this, however, this

Why not? Can't you stick whatever you need into preempt_disable() ? kernel/sched/core.c:preempt_count_{add,sub}() are there for a (debug) reason.

Quoted 3 lines

If you think this series deserves a reconsideration I will send out a new version with more text about what's the problem and why this is useful.

So please make it a validator. That's so much more useful.

author
#15 lore raw
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Boqun Feng

The problem is:

Quoted 8 lines
Quoted 4 lines

Currently there is no way to know which rcu_dereference() and its friends a rcu_read_lock() or one of its friends is protecting. And the lack of such information is a pain for kernel developers and reviewers to understand the purpose of a certain rcu_read_lock().

If I have to go run a kernel and look at /proc data in order to complete a review the patch will fail review, end of story.

OK, got it.

Quoted 17 lines
Quoted 5 lines
Quoted 2 lines

Secondly, I'm not sure a random list of data is going to help anybody.

One possible usage might be:

Quoted 11 lines

If someone is going to refactor a piece of code which originally uses RCU for synchronization, and now he wants to switch to other synchronization mechanism for some data because of some reason(for consistency or writer side latency guarantee, etc.). Before he makes the change, he needs to know if this rcu_read_lock() also protects other data, and this is not an easy job and I don't think we have anything helpful for this. With RCU_LOCKED_ACCESS, he can build the kernel with RCU_LOCKED_ACCESS enabled, run some workloads and get the information from a /proc file. In this way, RCU_LOCKED_ACCESS is helpful to developers who want to know the information of the associations between rcu_read_lock() and rcu_dereference().

Quoted 2 lines

So 'possible', but will anyone actually do this? `

Not sure, but one of the purposes of sending out this RFC is telling people we have a way to do this and see if they are interested in ;-)

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

The whole point of lockdep is that it tells you if you do broken. This doesn't, and cannot, because it lacks information to verify against.

Yes.. RCU_LOCKED_ACCESS is not a validator, so it cannot be as useful as lockdep for finding bug.

I would try really _really_ hard to make it a validator, those are so much more useful.

Agreed. I would love to make it a validator too. But I think that information collection is the first step and an essential part for making a validator, right? And what RCU_LOCKED_ACCESS does are collecting information and showing that via a proc file. I really think it's better to know what information we can collect before we actually try to make a validator, besides, the proc file output could serve as a way to debug the validator if we really had a validator.

So maybe we can first discuss about and work on how to collect and show the information, and then seek a way for making a validator?

Quoted 14 lines

One could for example allow something like:

	rcu_read_lock();
	rcu_annotate(&var->field);

	foo();

	rcu_read_unlock();

As an alternative to the syntax suggested by Ingo. This would allow keeping the existing rcu_read_lock() signature so you don't have to force update the entire kernel at once, while also (easily) allowing multiple variables. Like:

Quoted 8 lines
	rcu_read_lock();
	rcu_annotate(&var->field);
	rcu_annotate(&var2->field2);

You can then have a special rule that if a particular RCU section has an annotation, any rcu_dereference() not matched will field a warning. If the annotation section is empty, nothing.

Good idea! but I don't think annotating a field in C language is easy, I will try to see what we can get. Do you have something already in your mind?

Quoted 18 lines
Quoted 8 lines
Quoted 4 lines

So I'm still not sure this is useful. Also, I would argue your code has problems if you cannot even find your rcu_read_lock().

I think what you mean here is, for example, the case where we use preempt_disable() instead of rcu_read_lock_sched() to pair with synchronize_sched(), right?

No, I was more like:

	rcu_read_lock();
	foo()
	  bar()
	    var->func();
	      obj->func();
	        whatever();
Quoted 4 lines

and you're looking at a change to whatever() and wonder where the heck the corresponding rcu_read_lock() lives and if we're having it held at all.

Confused.. RCU_LOCKED_ACCESS has such information, For example, in the piece of /proc/locked_access/rcu I put in the cover letter, which I will put in the commit logs for the next version of this series:

ACQCHAIN 0xfe042af3bbfb2605, 3 locks, irq_context 0:
LOCK at [<ffffffff81094b47>] SyS_kill+0x97/0x2a0

    LOCK at [<ffffffff8109286f>] kill_pid_info+0x1f/0x140
      LOCK at [<ffffffff81092605>] group_send_sig_info+0x5/0x130
        ACCESS TYPE 1 at kernel/signal.c:695

, which actually reflects:

	(in kernel/signal.c)
	SyS_kill()
	  kill_something_info()
	    rcu_read_lock()
	    kill_pid_info()
	      rcu_read_lock()
	      group_send_sig_info()
	        rcu_read_lock()
		check_kill_permission()
		  kill_ok_by_cred()
		    __task_cred()

and __task_cred() is an rcu_dereference(), so if anyone wants to figure out what rcu_read_lock()s are hold when this deref happens, he can dump the proc file, and search for the filename and linenumber of __task_cred() and he will get the symbol + offset of all the rcu_read_lock()s for this deref.

(Yes, I know there is an inconsistency on symbol + offset versus filename + linenumber here, the reason of this inconsistency is that the original purpose of this feature is to show some information to developers to help understand code, therefore filename + linenumber is more appropriate for this purpose. So I use filename + linenumber to indicate the locations for rcu_dereference(), however, as what I can only get from the task_struct::held_locks is acquire_ip, so I have to use the symbol + offset for rcu_read_lock(), and there comes the inconsistency. Maybe I should make them all symbol + offset, and use tools like addr2line to convert to filename + linenumber.)

Quoted 6 lines
Quoted 1 line

If so, sure, this series cannot do anything about this, however, this

Why not? Can't you stick whatever you need into preempt_disable() ? kernel/sched/core.c:preempt_count_{add,sub}() are there for a (debug) reason.

Yeah, you are right. I can do something similar to preempt_disable() by maintaining a stack of callsites of preempt_disable() for each thread, which means a little much memory cost, but it's worthwhile if we make it useful.

Quoted 5 lines
Quoted 3 lines

If you think this series deserves a reconsideration I will send out a new version with more text about what's the problem and why this is useful.

So please make it a validator. That's so much more useful.

I cerntainly have done a bad job of the explanation of this series in the cover letter and the commit logs, so I'm considering sending the next version of this series with better text and examples, or you prefer I should try to make it a validator first and send out as a whole for reviewing?

Thank you for your thoughts and comments ;-)

Regards,
Boqun

maintainer
#16 lore raw
Quoted 18 lines · Boqun Feng
Quoted 18 lines

One could for example allow something like:

	rcu_read_lock();
	rcu_annotate(&var->field);

	foo();

	rcu_read_unlock();

As an alternative to the syntax suggested by Ingo. This would allow keeping the existing rcu_read_lock() signature so you don't have to force update the entire kernel at once, while also (easily) allowing multiple variables. Like:

	rcu_read_lock();
	rcu_annotate(&var->field);
	rcu_annotate(&var2->field2);
Quoted 8 lines
Quoted 4 lines

You can then have a special rule that if a particular RCU section has an annotation, any rcu_dereference() not matched will field a warning. If the annotation section is empty, nothing.

Good idea! but I don't think annotating a field in C language is easy, I will try to see what we can get. Do you have something already in your mind?

No, didn't really think about that :-/ The most restrictive version is taking the absolute address, but that would make things like actual data structures impossible.

Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
Quoted 3 lines

So I'm still not sure this is useful. Also, I would argue your code has problems if you cannot even find your rcu_read_lock().

I think what you mean here is, for example, the case where we use preempt_disable() instead of rcu_read_lock_sched() to pair with synchronize_sched(), right?

No, I was more like:

	rcu_read_lock();
	foo()
	  bar()
	    var->func();
	      obj->func();
	        whatever();
Quoted 8 lines
Quoted 4 lines

and you're looking at a change to whatever() and wonder where the heck the corresponding rcu_read_lock() lives and if we're having it held at all.

Confused.. RCU_LOCKED_ACCESS has such information, For example, in the piece of /proc/locked_access/rcu I put in the cover letter, which I will put in the commit logs for the next version of this series:

Yes, but my point was that if it wasn't obvious from the code, your code has issues. You should not be needing a tool to figure this out.

maintainer
#17 lore raw
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Boqun Feng
Quoted 22 lines

One could for example allow something like:

	rcu_read_lock();
	rcu_annotate(&var->field);

	foo();

	rcu_read_unlock();

As an alternative to the syntax suggested by Ingo. This would allow keeping the existing rcu_read_lock() signature so you don't have to force update the entire kernel at once, while also (easily) allowing multiple variables. Like:

	rcu_read_lock();
	rcu_annotate(&var->field);
	rcu_annotate(&var2->field2);

You can then have a special rule that if a particular RCU section has an annotation, any rcu_dereference() not matched will field a warning. If the annotation section is empty, nothing.

Quoted 7 lines
Quoted 3 lines

Good idea! but I don't think annotating a field in C language is easy, I will try to see what we can get. Do you have something already in your mind?

No, didn't really think about that :-/ The most restrictive version is taking the absolute address, but that would make things like actual data structures impossible.

So the thing with locks is they get a struct lockdep_map added, in which we store all kinds of useful. But I don't think we cannot add a similar structure to each and every RCU dereferencable (is that a word?) variable.

author
#18 lore raw
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Boqun Feng
Quoted 22 lines

One could for example allow something like:

	rcu_read_lock();
	rcu_annotate(&var->field);

	foo();

	rcu_read_unlock();

As an alternative to the syntax suggested by Ingo. This would allow keeping the existing rcu_read_lock() signature so you don't have to force update the entire kernel at once, while also (easily) allowing multiple variables. Like:

	rcu_read_lock();
	rcu_annotate(&var->field);
	rcu_annotate(&var2->field2);

You can then have a special rule that if a particular RCU section has an annotation, any rcu_dereference() not matched will field a warning. If the annotation section is empty, nothing.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Good idea! but I don't think annotating a field in C language is easy, I will try to see what we can get. Do you have something already in your mind?

No, didn't really think about that :-/ The most restrictive version is taking the absolute address, but that would make things like actual data structures impossible.

Another problem of taking the absolute address is the address may change from rcu_annotate() to rcu_dereference() for example:

	rcu_read_lock();
	rcu_annotate(&var->field);
					// in another thread
					var = new_var;
	
	// the address of var->field is different now.
	rcu_dereference(var->field);
	rcu_read_unlock();
Quoted 5 lines

So the thing with locks is they get a struct lockdep_map added, in which we store all kinds of useful. But I don't think we cannot add a similar structure to each and every RCU dereferencable (is that a word?) variable.

Well, some of them have rcu_head, but not all.. so you're right.

Quoted 1 line

I come up with something you may not like ;-) , which is taking the strings of the expressions, for example:

	rcu_read_lock();
	rcu_annotate(var->field);	// "var->field" is added for 
					// the current section

	rcu_dereference(var->field);	// OK, because the expression
					// "var->field" is annotated.

	rcu_dereference(var->field2);	// Not OK, because the
					// expression "var->field2" is
					// not annotated, nor is any of
					// its suffix.

	rcu_annotate(field3);		// "field3" is added for the
					// current section
	
	rcu_dereference(var2->field3);	// OK, because the suffix
					// "field3" is annotated.
	rcu_read_unlock();

I think this is more accurate than taking the absolute address because the address changing situations exist. So.. thoughts?

Regards,
Boqun

maintainer
#19 lore raw
Quoted 2 lines · Boqun Feng

I come up with something you may not like ;-) , which is taking the strings of the expressions, for example:

Heh, cute. Better than nothing I suppose.

#20 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines · Boqun Feng

I come up with something you may not like ;-) , which is taking the strings of the expressions, for example:

Heh, cute. Better than nothing I suppose.

It would be very good to get feedback from people using RCU heavily. Some RCU read-side critical sections access a lot of fields, which might make this sort of notation a bit of a maintainability issue.

Thanx, Paul