Skip to contentSkip to conversation

liblockdep fixes for 5.9-rc1

PATCH In review Sasha Levin 2 participants 17 messages Aug 5, 2020 – Aug 7, 2020 Bitmap API

lore
Share & tools
lore
#0 lore raw

v4 cover liblockdep fixes for 5.9-rc1

Hi Linus,

Please consider applying these patches for liblockdep, or alternatively pull from:

git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux.git tags/liblockdep-fixes-040820

The patches fix up compilation and functionality of liblockdep on 5.8, they were tested using liblockdep's internal testsuite.

I was unable to get the x86 folks to pull these fixes for the past few months:

- https://lkml.org/lkml/2020/2/17/1089
- https://lkml.org/lkml/2020/4/18/817
- https://lkml.org/lkml/2020/6/22/1262

Which is why this pull request ends up going straight to you.

Sasha Levin (14):
tools headers: Add kprobes.h header
tools headers: Add rcupdate.h header
tools/kernel.h: extend with dummy RCU functions
tools bitmap: add bitmap_andnot definition tools/lib/lockdep: add definition required for IRQ flag tracing tools bitmap: add bitmap_clear definition tools/lib/lockdep: Hook up vsprintf, find_bit, hweight libraries tools/lib/lockdep: Enable building with CONFIG_TRACE_IRQFLAGS tools/lib/lockdep: New stacktrace API tools/lib/lockdep: call lockdep_init_task on init tools/lib/lockdep: switch to using lockdep_init_map_waits tools/kernel.h: hide noinstr
tools/lib/lockdep: explicitly declare lockdep_init_task() tools/kernel.h: hide task_struct.hardirq_chain_key

 tools/include/linux/bitmap.h                  | 10 ++++++
 tools/include/linux/kernel.h                  | 11 ++++++
 tools/include/linux/kprobes.h                 |  7 ++++
 tools/include/linux/lockdep.h                 | 10 ++++++
 tools/include/linux/rcupdate.h                | 12 +++++++
 tools/include/linux/stacktrace.h              |  8 +++++
 tools/lib/bitmap.c                            | 35 +++++++++++++++++++
 tools/lib/lockdep/Build                       |  2 +-
 tools/lib/lockdep/Makefile                    |  2 +-
 tools/lib/lockdep/include/liblockdep/common.h |  4 +--
 tools/lib/lockdep/include/liblockdep/mutex.h  |  2 +-
 tools/lib/lockdep/include/liblockdep/rwlock.h |  2 +-
 tools/lib/lockdep/lockdep.c                   |  4 +--
 tools/lib/lockdep/preload.c                   |  8 ++++-
 14 files changed, 108 insertions(+), 9 deletions(-)
 create mode 100644 tools/include/linux/kprobes.h
 create mode 100644 tools/include/linux/rcupdate.h
Signature · 1 line
2.25.1
#1 lore raw

PATCH v4 1/14 tools headers: Add kprobes.h header

This is now needed by liblockdep as a result of 2f43c6022d84 ("kprobes:
Prohibit probing on lockdep functions").

---
 tools/include/linux/kprobes.h | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tools/include/linux/kprobes.h
tools/include/linux/kprobes.h new file +7 -0
new file mode 100644
index 0000000000000..f665725ea4d59
@@ -0,0 +1,7 @@
+#ifndef _TOOLS_LINUX_KPROBES_H_
+#define _TOOLS_LINUX_KPROBES_H_
+
+#define NOKPROBE_SYMBOL(x)
+#define nokprobe_inline
+
+#endif
Signature · 1 line
2.25.1
#2 lore raw

PATCH v4 2/14 tools headers: Add rcupdate.h header

In reply to Sasha Levin

This is now needed by liblockdep as a result of a0b0fd53e1e6 ("locking/lockdep: Free lock classes that are no longer in use").

---
 tools/include/linux/rcupdate.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 tools/include/linux/rcupdate.h
tools/include/linux/rcupdate.h new file +12 -0
new file mode 100644
index 0000000000000..1b7abc86f4336
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LIBLOCKDEP_RCUPDATE_H_
+#define _LIBLOCKDEP_RCUPDATE_H_
+
+#define call_rcu(x, y)
+#define init_rcu_head(x)
+
+struct rcu_head {
+	char dummy;
+};
+
+#endif
Signature · 1 line
2.25.1
#3 lore raw

PATCH v4 3/14 tools/kernel.h: extend with dummy RCU functions

In reply to Sasha Levin

These calls were added by 108c14858b9e ("locking/lockdep: Add support for dynamic keys") and require no special handling in userspace, so just add empty function definitions.

---
 tools/include/linux/kernel.h | 7 +++++++
 1 file changed, 7 insertions(+)
tools/include/linux/kernel.h +7 -0
index a7e54a08fb54c..902d3b9ab4c17 100644
@@ -117,4 +117,11 @@ int scnprintf_pad(char * buf, size_t size, const char * fmt, ...);
 #define current_gfp_context(k) 0
 #define synchronize_rcu()
 
+static __maybe_unused int system_state;
+#define SYSTEM_SCHEDULING 0
+
+#define might_sleep()
+#define rcu_read_lock()
+#define rcu_read_unlock()
+
 #endif
Signature · 1 line
2.25.1
#4 lore raw

PATCH v4 4/14 tools bitmap: add bitmap_andnot definition

In reply to Sasha Levin

Add definition of bitmap_andnot() and wire tools/lib/bitmap.c into liblockdep.

This is needed as a result of de4643a77356 ("locking/lockdep: Reuse lock chains that have been freed").

---
 tools/include/linux/bitmap.h | 10 ++++++++++
 tools/lib/bitmap.c           | 15 +++++++++++++++
 tools/lib/lockdep/Build      |  2 +-
 3 files changed, 26 insertions(+), 1 deletion(-)
tools/include/linux/bitmap.h +10 -0
index 477a1cae513f2..ab5df035f8eda 100644
@@ -18,6 +18,8 @@ int __bitmap_and(unsigned long *dst, const unsigned long *bitmap1,
 int __bitmap_equal(const unsigned long *bitmap1,
 		   const unsigned long *bitmap2, unsigned int bits);
 void bitmap_clear(unsigned long *map, unsigned int start, int len);
+int __bitmap_andnot(unsigned long *dst, const unsigned long *bitmap1,
+			const unsigned long *bitmap2, unsigned int bits);
 
 #define BITMAP_FIRST_WORD_MASK(start) (~0UL << ((start) & (BITS_PER_LONG - 1)))
 
@@ -178,4 +180,12 @@ static inline int bitmap_equal(const unsigned long *src1,
 	return __bitmap_equal(src1, src2, nbits);
 }
 
+static inline int bitmap_andnot(unsigned long *dst, const unsigned long *src1,
+                        const unsigned long *src2, unsigned int nbits)
+{
+	if (small_const_nbits(nbits))
+		return (*dst = *src1 & ~(*src2) & BITMAP_LAST_WORD_MASK(nbits)) != 0;
+	return __bitmap_andnot(dst, src1, src2, nbits);
+}
+
 #endif /* _PERF_BITOPS_H */
tools/lib/bitmap.c +15 -0
index 5043747ef6c5f..b6bc037623fc1 100644
@@ -86,3 +86,18 @@ int __bitmap_equal(const unsigned long *bitmap1,
 
 	return 1;
 }
+
+int __bitmap_andnot(unsigned long *dst, const unsigned long *bitmap1,
+                                const unsigned long *bitmap2, unsigned int bits)
+{
+	unsigned int k;
+	unsigned int lim = bits/BITS_PER_LONG;
+	unsigned long result = 0;
+
+	for (k = 0; k < lim; k++)
+		result |= (dst[k] = bitmap1[k] & ~bitmap2[k]);
+	if (bits % BITS_PER_LONG)
+		result |= (dst[k] = bitmap1[k] & ~bitmap2[k] &
+			BITMAP_LAST_WORD_MASK(bits));
+	return result != 0;
+}
tools/lib/lockdep/Build +1 -1
index 6f667355b0687..219a9e2d9e0ba 100644
@@ -1 +1 @@
-liblockdep-y += common.o lockdep.o preload.o rbtree.o
+liblockdep-y += common.o lockdep.o preload.o rbtree.o ../../lib/bitmap.o
Signature · 1 line
2.25.1
#5 lore raw

PATCH v4 5/14 tools/lib/lockdep: add definition required for IRQ flag tracing

In reply to Sasha Levin

We are going to start building with CONFIG_TRACE_IRQFLAGS defined, so let's wire up a few dummy variables in our task_struct.

This isn't needed in userspace, but due to some refactoring in kernel-side lockdep it's easier to just wire it up and enable CONFIG_TRACE_IRQFLAGS.

---
 tools/include/linux/lockdep.h | 8 ++++++++
 tools/lib/lockdep/lockdep.c   | 4 ++--
 2 files changed, 10 insertions(+), 2 deletions(-)
tools/include/linux/lockdep.h +8 -0
index e56997288f2b0..3a9924d6d3ae8 100644
@@ -23,6 +23,8 @@
 
 #include "../../../include/linux/lockdep.h"
 
+static bool early_boot_irqs_disabled;
+
 struct task_struct {
 	u64 curr_chain_key;
 	int lockdep_depth;
@@ -31,6 +33,12 @@ struct task_struct {
 	gfp_t lockdep_reclaim_gfp;
 	int pid;
 	int state;
+	int softirqs_enabled, hardirqs_enabled, softirqs_disabled, hardirqs_disabled, irq_events;
+	unsigned long softirq_disable_ip, softirq_enable_ip;
+	unsigned int softirq_disable_event, softirq_enable_event;
+	unsigned long hardirq_disable_ip, hardirq_enable_ip;
+	unsigned int hardirq_disable_event, hardirq_enable_event;
+	int softirq_context, hardirq_context;
 	char comm[17];
 };
 
tools/lib/lockdep/lockdep.c +2 -2
index 348a9d0fb766a..9be12d233477a 100644
@@ -15,10 +15,10 @@ u32 prandom_u32(void)
 	abort();
 }
 
-void print_irqtrace_events(struct task_struct *curr)
+/*void print_irqtrace_events(struct task_struct *curr)
 {
 	abort();
-}
+}*/
 
 static struct new_utsname *init_utsname(void)
 {
Signature · 1 line
2.25.1
#6 lore raw

PATCH v4 6/14 tools bitmap: add bitmap_clear definition

In reply to Sasha Levin

This is needed as a result of de4643a77356 ("locking/lockdep: Reuse lock chains that have been freed").

---
 tools/lib/bitmap.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/lib/bitmap.c +20 -0
index b6bc037623fc1..873176c501e43 100644
@@ -101,3 +101,23 @@ int __bitmap_andnot(unsigned long *dst, const unsigned long *bitmap1,
 			BITMAP_LAST_WORD_MASK(bits));
 	return result != 0;
 }
+
+void bitmap_clear(unsigned long *map, unsigned int start, int len)
+{
+	unsigned long *p = map + BIT_WORD(start);
+	const unsigned int size = start + len;
+	int bits_to_clear = BITS_PER_LONG - (start % BITS_PER_LONG);
+	unsigned long mask_to_clear = BITMAP_FIRST_WORD_MASK(start);
+
+	while (len - bits_to_clear >= 0) {
+		*p &= ~mask_to_clear;
+		len -= bits_to_clear;
+		bits_to_clear = BITS_PER_LONG;
+		mask_to_clear = ~0UL;
+		p++;
+	}
+	if (len) {
+		mask_to_clear &= BITMAP_LAST_WORD_MASK(size);
+		*p &= ~mask_to_clear;
+	}
+}
Signature · 1 line
2.25.1
#7 lore raw

PATCH v4 7/14 tools/lib/lockdep: Hook up vsprintf, find_bit, hweight libraries

In reply to Sasha Levin

They already exist in tools/lib/, and are now required by liblockdep, so just add them to the build manifest.

---
 tools/lib/lockdep/Build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/lockdep/Build +1 -1
index 219a9e2d9e0ba..1e7c25690b8ea 100644
@@ -1 +1 @@
-liblockdep-y += common.o lockdep.o preload.o rbtree.o ../../lib/bitmap.o
+liblockdep-y += common.o lockdep.o preload.o rbtree.o ../../lib/bitmap.o ../../lib/vsprintf.o ../../lib/find_bit.o ../../lib/hweight.o
Signature · 1 line
2.25.1
#8 lore raw

PATCH v4 8/14 tools/lib/lockdep: Enable building with CONFIG_TRACE_IRQFLAGS

In reply to Sasha Levin

At this point it's always enabled with CONFIG_PROVE_LOCKING, so it's easier to enable it in liblockdep as well rather than try and fix up the lockdep code.

---
 tools/lib/lockdep/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/lockdep/Makefile +1 -1
index 9dafb8cb752fe..efb6b7980a009 100644
@@ -78,7 +78,7 @@ export Q VERBOSE
 INCLUDES = -I. -I./uinclude -I./include -I../../include $(CONFIG_INCLUDES)
 
 # Set compile option CFLAGS if not set elsewhere
-CFLAGS ?= -g -DCONFIG_LOCKDEP -DCONFIG_STACKTRACE -DCONFIG_PROVE_LOCKING -DBITS_PER_LONG=__WORDSIZE -DLIBLOCKDEP_VERSION='"$(LIBLOCKDEP_VERSION)"' -rdynamic -O0 -g
+CFLAGS ?= -g -DCONFIG_LOCKDEP -DCONFIG_STACKTRACE -DCONFIG_PROVE_LOCKING -DCONFIG_TRACE_IRQFLAGS -DBITS_PER_LONG=__WORDSIZE -DLIBLOCKDEP_VERSION='"$(LIBLOCKDEP_VERSION)"' -rdynamic -O0 -g
 CFLAGS += -fPIC
 CFLAGS += -Wall
 
Signature · 1 line
2.25.1
#9 lore raw

PATCH v4 9/14 tools/lib/lockdep: New stacktrace API

In reply to Sasha Levin

The kernel switched to using kernel/stacktrace.c, and the API slightly changed.

Adjust it to make stack traces work again.

---
 tools/include/linux/stacktrace.h | 8 ++++++++
 1 file changed, 8 insertions(+)
tools/include/linux/stacktrace.h +8 -0
index ae343ac35bfa7..b290b98d883aa 100644
@@ -15,10 +15,18 @@ static inline void print_stack_trace(struct stack_trace *trace, int spaces)
 	backtrace_symbols_fd((void **)trace->entries, trace->nr_entries, 1);
 }
 
+static inline void stack_trace_print(const long unsigned int *entries, unsigned int nr, int spaces)
+{
+	backtrace_symbols_fd((void **)entries, nr, 1);
+}
+
 #define save_stack_trace(trace)	\
 	((trace)->nr_entries =	\
 		backtrace((void **)(trace)->entries, (trace)->max_entries))
 
+#define stack_trace_save(e, m, x) \
+	backtrace((void **)(e), (m))
+
 static inline int dump_stack(void)
 {
 	void *array[64];
Signature · 1 line
2.25.1
#10 lore raw

PATCH v4 10/14 tools/lib/lockdep: call lockdep_init_task on init

In reply to Sasha Levin

We now have to explicitly call lockdep_init_task() when starting up.

---
 tools/lib/lockdep/preload.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/lib/lockdep/preload.c +4 -0
index 8f1adbe887b2f..578fdeda9422c 100644
@@ -9,6 +9,8 @@
 #include "include/liblockdep/mutex.h"
 #include "../../include/linux/rbtree.h"
 
+extern struct task_struct *__curr(void);
+
 /**
  * struct lock_lookup - liblockdep's view of a single unique lock
  * @orig: pointer to the original pthread lock, used for lookups
@@ -421,6 +423,8 @@ __attribute__((constructor)) static void init_preload(void)
 	if (__init_state == done)
 		return;
 
+	lockdep_init_task(__curr());
+
 #ifndef __GLIBC__
 	__init_state = prepare;
 
Signature · 1 line
2.25.1
#11 lore raw

PATCH v4 11/14 tools/lib/lockdep: switch to using lockdep_init_map_waits

In reply to Sasha Levin

As of de8f5e4f2dc1 ("lockdep: Introduce wait-type checks") lockdep exports lockdep_init_map_waits() instead of lockdep_init_map() for initialization.

---
 tools/include/linux/kernel.h                  | 2 ++
 tools/include/linux/lockdep.h                 | 1 +
 tools/lib/lockdep/include/liblockdep/common.h | 4 ++--
 tools/lib/lockdep/include/liblockdep/mutex.h  | 2 +-
 tools/lib/lockdep/include/liblockdep/rwlock.h | 2 +-
 tools/lib/lockdep/preload.c                   | 2 +-
 6 files changed, 8 insertions(+), 5 deletions(-)
tools/include/linux/kernel.h +2 -0
index 902d3b9ab4c17..5ac493eef4237 100644
@@ -124,4 +124,6 @@ static __maybe_unused int system_state;
 #define rcu_read_lock()
 #define rcu_read_unlock()
 
+#define in_nmi() 0
+
 #endif
tools/include/linux/lockdep.h +1 -0
index 3a9924d6d3ae8..4e5f31f28ca3e 100644
@@ -34,6 +34,7 @@ struct task_struct {
 	int pid;
 	int state;
 	int softirqs_enabled, hardirqs_enabled, softirqs_disabled, hardirqs_disabled, irq_events;
+	int hardirq_threaded, irq_config;
 	unsigned long softirq_disable_ip, softirq_enable_ip;
 	unsigned int softirq_disable_event, softirq_enable_event;
 	unsigned long hardirq_disable_ip, hardirq_enable_ip;
tools/lib/lockdep/include/liblockdep/common.h +2 -2
index a6d7ee5f18ba9..5f698671f45c2 100644
@@ -37,8 +37,8 @@ struct lockdep_map {
 #endif
 };
 
-void lockdep_init_map(struct lockdep_map *lock, const char *name,
-			struct lock_class_key *key, int subclass);
+void lockdep_init_map_waits(struct lockdep_map *lock, const char *name,
+	struct lock_class_key *key, int subclass, short inner, short outer);
 void lock_acquire(struct lockdep_map *lock, unsigned int subclass,
 			int trylock, int read, int check,
 			struct lockdep_map *nest_lock, unsigned long ip);
tools/lib/lockdep/include/liblockdep/mutex.h +1 -1
index bd106b82759b7..6106fc73da687 100644
@@ -24,7 +24,7 @@ static inline int __mutex_init(liblockdep_pthread_mutex_t *lock,
 				struct lock_class_key *key,
 				const pthread_mutexattr_t *__mutexattr)
 {
-	lockdep_init_map(&lock->dep_map, name, key, 0);
+	lockdep_init_map_waits(&lock->dep_map, name, key, 0, 0, 0);
 	return pthread_mutex_init(&lock->mutex, __mutexattr);
 }
 
tools/lib/lockdep/include/liblockdep/rwlock.h +1 -1
index 6d5d2932bf4d9..222748d04219d 100644
@@ -23,7 +23,7 @@ static inline int __rwlock_init(liblockdep_pthread_rwlock_t *lock,
 				struct lock_class_key *key,
 				const pthread_rwlockattr_t *attr)
 {
-	lockdep_init_map(&lock->dep_map, name, key, 0);
+	lockdep_init_map_waits(&lock->dep_map, name, key, 0, 0, 0);
 
 	return pthread_rwlock_init(&lock->rwlock, attr);
 }
tools/lib/lockdep/preload.c +1 -1
index 578fdeda9422c..5bd58c51066c4 100644
@@ -199,7 +199,7 @@ static struct lock_lookup *__get_lock(void *lock)
 	 * TODO: Get the real name of the lock using libdwarf
 	 */
 	sprintf(l->name, "%p", lock);
-	lockdep_init_map(&l->dep_map, l->name, &l->key, 0);
+//	lockdep_init_map_waits(&l->dep_map, l->name, &l->key, 0, 0, 0);
 
 	ll_pthread_rwlock_wrlock(&locks_rwlock);
 	/* This might have changed since the last time we fetched it */
Signature · 1 line
2.25.1
#12 lore raw

PATCH v4 12/14 tools/kernel.h: hide noinstr

In reply to Sasha Levin

After 655389666643 ("vmlinux.lds.h: Create section for protection against instrumentation") we need to ignore "noinstr" in userspace.

---
 tools/include/linux/kernel.h | 2 ++
 1 file changed, 2 insertions(+)
tools/include/linux/kernel.h +2 -0
index 5ac493eef4237..39fcbb144a34a 100644
@@ -126,4 +126,6 @@ static __maybe_unused int system_state;
 
 #define in_nmi() 0
 
+#define noinstr
+
 #endif
Signature · 1 line
2.25.1
#13 lore raw

PATCH v4 13/14 tools/lib/lockdep: explicitly declare lockdep_init_task()

In reply to Sasha Levin

Avoid a warning as we can't pull the decleration for lockdep's header.

---
 tools/lib/lockdep/preload.c | 2 ++
 1 file changed, 2 insertions(+)
tools/lib/lockdep/preload.c +2 -0
index 5bd58c51066c4..2928c712f87f8 100644
@@ -92,6 +92,8 @@ static int (*ll_pthread_rwlock_trywrlock)(pthread_rwlock_t *rwlock)	= __pthread_
 static int (*ll_pthread_rwlock_wrlock)(pthread_rwlock_t *rwlock)	= __pthread_rwlock_wrlock;
 static int (*ll_pthread_rwlock_unlock)(pthread_rwlock_t *rwlock)	= __pthread_rwlock_unlock;
 
+extern void lockdep_init_task(struct task_struct *task);
+
 enum { none, prepare, done, } __init_state;
 static void init_preload(void);
 static void try_init_preload(void)
Signature · 1 line
2.25.1
#14 lore raw

PATCH v4 14/14 tools/kernel.h: hide task_struct.hardirq_chain_key

In reply to Sasha Levin

As a result of c86e9b987cea ("lockdep: Prepare for noinstr sections") we need to hide hardirq_chain_key from userspace to avoid a build error.

---
 tools/include/linux/lockdep.h | 1 +
 1 file changed, 1 insertion(+)
tools/include/linux/lockdep.h +1 -0
index 4e5f31f28ca3e..b6c9e81eb45e5 100644
@@ -40,6 +40,7 @@ struct task_struct {
 	unsigned long hardirq_disable_ip, hardirq_enable_ip;
 	unsigned int hardirq_disable_event, hardirq_enable_event;
 	int softirq_context, hardirq_context;
+	unsigned long hardirq_chain_key;
 	char comm[17];
 };
 
Signature · 1 line
2.25.1
#15 lore raw
In reply to Sasha Levin
Quoted 5 lines · Sasha Levin

Hi Linus,

Please consider applying these patches for liblockdep, or alternatively pull from:

Quoted 7 lines

git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux.git tags/liblockdep-fixes-040820

The patches fix up compilation and functionality of liblockdep on 5.8, they were tested using liblockdep's internal testsuite.

I was unable to get the x86 folks to pull these fixes for the past few months:

So the primary reason I didn't pull is that liblockdep was permanently build-broken from February 2019 to around February 2020, despite me pinging you multiple times about it.

Quoted 1 line

- https://lkml.org/lkml/2020/2/17/1089

This pull request still said that if fixes "most of" liblockdep, not "all of", which is the benchmark really after such a long series of breakage.

Quoted 1 line

- https://lkml.org/lkml/2020/4/18/817

This still said "most of".

Quoted 1 line

- https://lkml.org/lkml/2020/6/22/1262

Same 'most of' verbiage.

Quoted 1 line

Which is why this pull request ends up going straight to you.

So at this point I think we need to ask whether it's worth it: are there any actual users of liblockdep, besides the testcases in liblockdep itself? I see there's a 'liblockdep-dev' package for Debian, but not propagated to Ubuntu or other popular variants AFAICS.

Also, could you please specify whether all bugs are fixed or just 'most'?

Quoted 7 lines

Sasha Levin (14):
tools headers: Add kprobes.h header
tools headers: Add rcupdate.h header
tools/kernel.h: extend with dummy RCU functions
tools bitmap: add bitmap_andnot definition tools/lib/lockdep: add definition required for IRQ flag tracing tools bitmap: add bitmap_clear definition

Quoted 8 lines

tools/lib/lockdep: Hook up vsprintf, find_bit, hweight libraries tools/lib/lockdep: Enable building with CONFIG_TRACE_IRQFLAGS tools/lib/lockdep: New stacktrace API tools/lib/lockdep: call lockdep_init_task on init tools/lib/lockdep: switch to using lockdep_init_map_waits tools/kernel.h: hide noinstr
tools/lib/lockdep: explicitly declare lockdep_init_task() tools/kernel.h: hide task_struct.hardirq_chain_key

Style nits, please use consistent titles for patches:

- First word should be capitalized consistently, instead of mismash of lower case mixed with upper case.

- First word should preferably be a verb, i.e. "Add new stacktrace API stubs", not "New stacktrace API"

Also, please always check linux-next whether there's some new upstream changes that liblockdep needs to adapt to. Right now there's a new build breakage even with all your fixes applied:

thule:~/tip/tools/lib/lockdep> make
CC common.o
In file included from ../../include/linux/lockdep.h:24,
from common.c:5:
../../include/linux/../../../include/linux/lockdep.h:13:10: fatal error: linux/lockdep_types.h: No such file or directory
13 | #include <linux/lockdep_types.h>
| ^~~~~~~~~~~~~~~~~~~~~~~

At which point we need to step back and analyze the development model:
this comparatively high rate of breakage derives from the unorthodox direct coupling of a kernel subsystem to a user-space library.

The solution for that would be to use the method how perf syncs to kernel space headers, by maintaining a 100% copy in tools/include/ and having automatic mechanism that warns about out of sync headers but doesn't break functionality.

See tools/perf/check-headers.sh for details.

I believe this same half-automated sync-on-upstream-changes model could be used for liblockdep as well, i.e. lets copy kernel/lockdep.c and lockdep*.h over to tools/lib/lockdep/, and reuse the perf header syncing method to keep it synchronized from that point on.

That would result in a far more maintainable liblockdep end result IMO?

Thanks,

Ingo

#16 lore raw
Quoted 13 lines · Ingo Molnar
Quoted 10 lines · Sasha Levin

Hi Linus,

Please consider applying these patches for liblockdep, or alternatively pull from:

git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux.git tags/liblockdep-fixes-040820

The patches fix up compilation and functionality of liblockdep on 5.8, they were tested using liblockdep's internal testsuite.

Quoted 6 lines
Quoted 2 lines

I was unable to get the x86 folks to pull these fixes for the past few months:

So the primary reason I didn't pull is that liblockdep was permanently build-broken from February 2019 to around February 2020, despite me pinging you multiple times about it.

I'm not sure how ignoring me for additional 6 months helped the state of liblockdep.

Quoted 6 lines
Quoted 1 line

- https://lkml.org/lkml/2020/2/17/1089

This pull request still said that if fixes "most of" liblockdep, not "all of", which is the benchmark really after such a long series of breakage.

Quoted 7 lines
Quoted 1 line

- https://lkml.org/lkml/2020/4/18/817

This still said "most of".

Quoted 1 line

- https://lkml.org/lkml/2020/6/22/1262

Same 'most of' verbiage.

Right, and there are still benign build warnings around unused variables which doesn't impede liblockdep's functionality.

Should they be cleaned up? sure, but let's get it working again.

Quoted 6 lines
Quoted 1 line

Which is why this pull request ends up going straight to you.

So at this point I think we need to ask whether it's worth it: are there any actual users of liblockdep, besides the testcases in liblockdep itself? I see there's a 'liblockdep-dev' package for Debian, but not propagated to Ubuntu or other popular variants AFAICS.

Right, I can't really say how many users of this there are out there. I get an occasional email with a question or comment but I'm not aware of how many of those users are out there.

I do think that if we keep liblockdep alive we should do a better job at not taking commits that break it to begin with. The model where someone is chasing lockdep to fix breakage after breakage isn't sustainable.

Quoted 2 lines

Also, could you please specify whether all bugs are fixed or just 'most'?

It passes the testsuite, there are compiler warnings that show up on some gcc versions I'll work on cleaning up.

Quoted 16 lines
Quoted 7 lines

Sasha Levin (14):
tools headers: Add kprobes.h header
tools headers: Add rcupdate.h header
tools/kernel.h: extend with dummy RCU functions
tools bitmap: add bitmap_andnot definition tools/lib/lockdep: add definition required for IRQ flag tracing tools bitmap: add bitmap_clear definition

Quoted 8 lines

tools/lib/lockdep: Hook up vsprintf, find_bit, hweight libraries tools/lib/lockdep: Enable building with CONFIG_TRACE_IRQFLAGS tools/lib/lockdep: New stacktrace API tools/lib/lockdep: call lockdep_init_task on init tools/lib/lockdep: switch to using lockdep_init_map_waits tools/kernel.h: hide noinstr
tools/lib/lockdep: explicitly declare lockdep_init_task() tools/kernel.h: hide task_struct.hardirq_chain_key

Quoted 7 lines

Style nits, please use consistent titles for patches:

- First word should be capitalized consistently, instead of mismash of lower case mixed with upper case.

- First word should preferably be a verb, i.e. "Add new stacktrace API stubs", not "New stacktrace API"

I'll address these.

Quoted 23 lines

Also, please always check linux-next whether there's some new upstream changes that liblockdep needs to adapt to. Right now there's a new build breakage even with all your fixes applied:

thule:~/tip/tools/lib/lockdep> make
CC common.o
In file included from ../../include/linux/lockdep.h:24,
from common.c:5:
../../include/linux/../../../include/linux/lockdep.h:13:10: fatal error: linux/lockdep_types.h: No such file or directory
13 | #include <linux/lockdep_types.h>
| ^~~~~~~~~~~~~~~~~~~~~~~

At which point we need to step back and analyze the development model:
this comparatively high rate of breakage derives from the unorthodox direct coupling of a kernel subsystem to a user-space library.

The solution for that would be to use the method how perf syncs to kernel space headers, by maintaining a 100% copy in tools/include/ and having automatic mechanism that warns about out of sync headers but doesn't break functionality.

See tools/perf/check-headers.sh for details.

Quoted 7 lines

I believe this same half-automated sync-on-upstream-changes model could be used for liblockdep as well, i.e. lets copy kernel/lockdep.c and lockdep*.h over to tools/lib/lockdep/, and reuse the perf header syncing method to keep it synchronized from that point on.

That would result in a far more maintainable liblockdep end result IMO?

How does perf handle doing changes on top of those headers? As an example, if you look at patch #12, it basically makes userspace ignore the new "noinstr" annotation, otherwise compiling code that uses "noinstr" in userspace breaks badly.

For liblockdep it's not enough to just sync headers, we need to do changes to those headers as well.

Signature · 2 lines
Thanks,
Sasha