Skip to contentSkip to conversation

kmemleak report format changes

PATCH In review Jim Cromie 2 participants 8 messages Apr 25, 2023 – Nov 16, 2023

Subsystems (2)
lore
Share & tools
lore

Jim Cromie

author
#0 lore raw

cover kmemleak report format changes

If format changes are not /sys/** ABI violating, heres 3 minor ones:

1st strips "age <increasing>" from output. This makes the output idempotent; unchanging until a new leak is reported.

2nd adds the backtrace.checksum to the "backtrace:" line. This lets a user see repeats without actually reading the whole backtrace. So now the backtrace line looks like this:

backtrace (ck 603070071): # also see below

Q: should ck be spelled crc ? it feels more communicative.

NB: with ck exposed, it becomes possible to do a "selective clear", something like:

echo drop 603070071 > /sys/kernel/debug/kmemleak

The 3rd patch takes __init off of kmemleak_test_init(). This fixes a bare-pointer in the 2nd line of the backtrace below, which previously looked like:

    [<00000000ef738764>] 0xffffffffc02350a2

NB: this happens still/again, after rmmod kmemleak-test.

unreferenced object 0xffff888005d9ca40 (size 32):
comm "modprobe", pid 412, jiffies 4294703300
hex dump (first 32 bytes):

    00 cd d9 05 80 88 ff ff 40 cf d9 05 80 88 ff ff  ........@.......
    14 a7 c4 f6 7d f9 87 10 00 00 00 00 00 00 00 00  ....}...........
  backtrace (ck 1354775490):
    [<000000002c474f61>] kmalloc_trace+0x26/0x90
    [<00000000b26599c1>] kmemleak_test_init+0x58/0x2d0 [kmemleak_test]
    [<0000000044d13990>] do_one_initcall+0x43/0x210
    [<00000000131bc505>] do_init_module+0x4a/0x210
    [<00000000b2902890>] __do_sys_finit_module+0x93/0xf0
    [<00000000673fdce2>] do_syscall_64+0x34/0x80
    [<00000000357a2d80>] entry_SYSCALL_64_after_hwframe+0x46/0xb0

Jim Cromie (3):
kmemleak: drop (age <increasing>) from leak record
kmemleak: add checksum to backtrace report
kmemleak-test: drop __init to get better backtrace

 mm/kmemleak.c                    | 8 +++-----
 samples/kmemleak/kmemleak-test.c | 2 +-
 2 files changed, 4 insertions(+), 6 deletions(-)
Signature · 1 line
2.40.0
author
#1 lore raw

PATCH 1/3 kmemleak: drop (age <increasing>) from leak record

Displaying age is pretty, but counter-productive; it surrenders idempotency of the output, which breaks simple hash-based cataloging of the records by the user.

The trouble: sequential reads, wo new leaks, get new results:

:#> sum /sys/kernel/debug/kmemleak
53439 74 /sys/kernel/debug/kmemleak
:#> sum /sys/kernel/debug/kmemleak
59066 74 /sys/kernel/debug/kmemleak

and age is why (nothing else changes):

:#> grep -v age /sys/kernel/debug/kmemleak | sum
58894 67
:#> grep -v age /sys/kernel/debug/kmemleak | sum
58894 67

Further, age is not an intrinsic property of the leak, its an artifact of when it was scanned, and relative age is embedded in leak order.

While userspace could work around the always-changing output, ISTM none could be relying upon age in any important way, and having idempotent output is just better.

---
 mm/kmemleak.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/kmemleak.c +2 -4
index a2d34226e3c8..f025c7bc845b 100644
@@ -355,14 +355,12 @@ static void print_unreferenced(struct seq_file *seq,
 	int i;
 	unsigned long *entries;
 	unsigned int nr_entries;
-	unsigned int msecs_age = jiffies_to_msecs(jiffies - object->jiffies);
 
 	nr_entries = stack_depot_fetch(object->trace_handle, &entries);
 	warn_or_seq_printf(seq, "unreferenced object 0x%08lx (size %zu):\n",
 			  object->pointer, object->size);
-	warn_or_seq_printf(seq, "  comm \"%s\", pid %d, jiffies %lu (age %d.%03ds)\n",
-			   object->comm, object->pid, object->jiffies,
-			   msecs_age / 1000, msecs_age % 1000);
+	warn_or_seq_printf(seq, "  comm \"%s\", pid %d, jiffies %lu\n",
+			   object->comm, object->pid, object->jiffies);
 	hex_dump_object(seq, object);
 	warn_or_seq_printf(seq, "  backtrace:\n");
 
Signature · 1 line
2.40.0
author
#2 lore raw

PATCH 2/3 kmemleak: add checksum to backtrace report

In reply to Jim Cromie

change kmemleak report format:

from: "  backtrace:"
to:   "  backtrace (ck <cksum>):"

The <cksum> allows a user to see recurring backtraces without detailed/careful reading of multiline backtraces. So after cycling kmemleak-test a few times, I know some leaks are repeating.

bash-5.2# grep backtrace /sys/kernel/debug/kmemleak | wc

     62     186    1792
  bash-5.2# grep backtrace /sys/kernel/debug/kmemleak | sort -u | wc
     37     111    1067
---
 mm/kmemleak.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/kmemleak.c +1 -1
index f025c7bc845b..2d1dfed4293d 100644
@@ -362,7 +362,7 @@ static void print_unreferenced(struct seq_file *seq,
 	warn_or_seq_printf(seq, "  comm \"%s\", pid %d, jiffies %lu\n",
 			   object->comm, object->pid, object->jiffies);
 	hex_dump_object(seq, object);
-	warn_or_seq_printf(seq, "  backtrace:\n");
+	warn_or_seq_printf(seq, "  backtrace (ck %u):\n", object->checksum);
 
 	for (i = 0; i < nr_entries; i++) {
 		void *ptr = (void *)entries[i];
Signature · 1 line
2.40.0
author
#3 lore raw

PATCH 3/3 kmemleak-test: drop __init to get better backtrace

In reply to Jim Cromie

Drop the __init on kmemleak_test_init(). With it, the storage is reclaimed, but then the symbol isn't available for "%pS" rendering, and the backtrace gets a bare pointer where the actual leak happened.

unreferenced object 0xffff88800a2b0800 (size 1024):
comm "modprobe", pid 413, jiffies 4294953430 hex dump (first 32 bytes):

    73 02 00 00 75 01 00 68 02 00 00 01 00 00 00 04  s...u..h........
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  backtrace (ck 603070071):
    [<00000000fabad728>] kmalloc_trace+0x26/0x90
    [<00000000ef738764>] 0xffffffffc02350a2
    [<00000000004e5795>] do_one_initcall+0x43/0x210
    [<00000000d768905e>] do_init_module+0x4a/0x210
    [<0000000087135ab5>] __do_sys_finit_module+0x93/0xf0
    [<000000004fcb1fa2>] do_syscall_64+0x34/0x80
    [<00000000c73c8d9d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0

with __init gone, that trace entry renders like:

    [<00000000ef738764>] kmemleak_test_init+<offset>/<size>
---
 samples/kmemleak/kmemleak-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
samples/kmemleak/kmemleak-test.c +1 -1
index 7b476eb8285f..6ced5ddd99d4 100644
@@ -32,7 +32,7 @@ static DEFINE_PER_CPU(void *, kmemleak_test_pointer);
  * Some very simple testing. This function needs to be extended for
  * proper testing.
  */
-static int __init kmemleak_test_init(void)
+static int kmemleak_test_init(void)
 {
 	struct test_node *elem;
 	int i;
Signature · 1 line
2.40.0
maintainer Acked-by on v1 3/3
#4 lore raw

Re: [PATCH 3/3] kmemleak-test: drop __init to get better backtrace

Quoted 18 lines · Jim Cromie

Drop the __init on kmemleak_test_init(). With it, the storage is reclaimed, but then the symbol isn't available for "%pS" rendering, and the backtrace gets a bare pointer where the actual leak happened.

unreferenced object 0xffff88800a2b0800 (size 1024):
comm "modprobe", pid 413, jiffies 4294953430 hex dump (first 32 bytes):

    73 02 00 00 75 01 00 68 02 00 00 01 00 00 00 04  s...u..h........
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  backtrace (ck 603070071):
    [<00000000fabad728>] kmalloc_trace+0x26/0x90
    [<00000000ef738764>] 0xffffffffc02350a2
    [<00000000004e5795>] do_one_initcall+0x43/0x210
    [<00000000d768905e>] do_init_module+0x4a/0x210
    [<0000000087135ab5>] __do_sys_finit_module+0x93/0xf0
    [<000000004fcb1fa2>] do_syscall_64+0x34/0x80
    [<00000000c73c8d9d>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
Quoted 3 lines

with __init gone, that trace entry renders like:

    [<00000000ef738764>] kmemleak_test_init+<offset>/<size>
maintainer
#5 lore raw
In reply to Jim Cromie
Quoted 5 lines · Jim Cromie

If format changes are not /sys/** ABI violating, heres 3 minor ones:

1st strips "age <increasing>" from output. This makes the output idempotent; unchanging until a new leak is reported.

Quoted 7 lines

2nd adds the backtrace.checksum to the "backtrace:" line. This lets a user see repeats without actually reading the whole backtrace. So now the backtrace line looks like this:

backtrace (ck 603070071): # also see below

Q: should ck be spelled crc ? it feels more communicative.

These all would make sense (and 'crc' sounds better) if they were done from the start. I know there are test scripts out there parsing the kmemleak sysfs file. I can't tell whether these changes would break them.

Cc'ing Dmitry, I think syzbot was regularly checking kmemleak (not sure it still does).

Quoted 26 lines

NB: with ck exposed, it becomes possible to do a "selective clear", something like:

echo drop 603070071 > /sys/kernel/debug/kmemleak

The 3rd patch takes __init off of kmemleak_test_init(). This fixes a bare-pointer in the 2nd line of the backtrace below, which previously looked like:

    [<00000000ef738764>] 0xffffffffc02350a2

NB: this happens still/again, after rmmod kmemleak-test.

unreferenced object 0xffff888005d9ca40 (size 32):
comm "modprobe", pid 412, jiffies 4294703300
hex dump (first 32 bytes):

    00 cd d9 05 80 88 ff ff 40 cf d9 05 80 88 ff ff  ........@.......
    14 a7 c4 f6 7d f9 87 10 00 00 00 00 00 00 00 00  ....}...........
  backtrace (ck 1354775490):
    [<000000002c474f61>] kmalloc_trace+0x26/0x90
    [<00000000b26599c1>] kmemleak_test_init+0x58/0x2d0 [kmemleak_test]
    [<0000000044d13990>] do_one_initcall+0x43/0x210
    [<00000000131bc505>] do_init_module+0x4a/0x210
    [<00000000b2902890>] __do_sys_finit_module+0x93/0xf0
    [<00000000673fdce2>] do_syscall_64+0x34/0x80
    [<00000000357a2d80>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
Signature · 1 line
Catalin
#6 lore raw
Quoted 15 lines · Catalin Marinas
Quoted 5 lines · Jim Cromie

If format changes are not /sys/** ABI violating, heres 3 minor ones:

1st strips "age <increasing>" from output. This makes the output idempotent; unchanging until a new leak is reported.

Quoted 7 lines

2nd adds the backtrace.checksum to the "backtrace:" line. This lets a user see repeats without actually reading the whole backtrace. So now the backtrace line looks like this:

backtrace (ck 603070071): # also see below

Q: should ck be spelled crc ? it feels more communicative.

Quoted 8 lines

These all would make sense (and 'crc' sounds better) if they were done from the start. I know there are test scripts out there parsing the kmemleak sysfs file. I can't tell whether these changes would break them.

Cc'ing Dmitry, I think syzbot was regularly checking kmemleak (not sure it still does).

I took a look at syzkaller repo, found kmemleak parsing in executor/common_linux.h
in static void check_leaks(char** frames, int nframes)

this parse just counts occurrences of "unreferenced object", it does not expect to find "age" nor would it (apparently) choke on "crc" being added.

There are also a few kmemleak refs in several github repos 3 have updates since 2020.

perf-monitor has 2 forks, both have the same minor compile warning-turned-error.
umemleak makes mention of kmemleak, but it has no code, just a readme.

QED: there are no kmemleak parsers in public github repos that would break with these changes

While there may be parsers on the dark-web,
ISTM none could be relying upon "age" in any meaningful way. and none are likely to choke because "(crc: <foo>)" was added to the stack trace display.

If these patches spend some time in purgatory (in linux-next or linux-eventually)
perhaps the remaining risks can be discounted ?

Quoted 29 lines
Quoted 16 lines

NB: with ck exposed, it becomes possible to do a "selective clear", something like:

echo drop 603070071 > /sys/kernel/debug/kmemleak

The 3rd patch takes __init off of kmemleak_test_init(). This fixes a bare-pointer in the 2nd line of the backtrace below, which previously looked like:

    [<00000000ef738764>] 0xffffffffc02350a2

NB: this happens still/again, after rmmod kmemleak-test.

unreferenced object 0xffff888005d9ca40 (size 32):
comm "modprobe", pid 412, jiffies 4294703300
hex dump (first 32 bytes):

Quoted 10 lines
    00 cd d9 05 80 88 ff ff 40 cf d9 05 80 88 ff ff  ........@.......
    14 a7 c4 f6 7d f9 87 10 00 00 00 00 00 00 00 00  ....}...........
  backtrace (ck 1354775490):
    [<000000002c474f61>] kmalloc_trace+0x26/0x90
    [<00000000b26599c1>] kmemleak_test_init+0x58/0x2d0 [kmemleak_test]
    [<0000000044d13990>] do_one_initcall+0x43/0x210
    [<00000000131bc505>] do_init_module+0x4a/0x210
    [<00000000b2902890>] __do_sys_finit_module+0x93/0xf0
    [<00000000673fdce2>] do_syscall_64+0x34/0x80
    [<00000000357a2d80>] entry_SYSCALL_64_after_hwframe+0x46/0xb0

--
Catalin

maintainer
#7 lore raw
Quoted 23 lines · On Fri, Nov 10, 2023 at 05:19:38PM -0700, [email protected] wrote:
Quoted 21 lines · Catalin Marinas
Quoted 5 lines · Jim Cromie

If format changes are not /sys/** ABI violating, heres 3 minor ones:

1st strips "age <increasing>" from output. This makes the output idempotent; unchanging until a new leak is reported.

Quoted 7 lines

2nd adds the backtrace.checksum to the "backtrace:" line. This lets a user see repeats without actually reading the whole backtrace. So now the backtrace line looks like this:

backtrace (ck 603070071): # also see below

Q: should ck be spelled crc ? it feels more communicative.

These all would make sense (and 'crc' sounds better) if they were done from the start. I know there are test scripts out there parsing the kmemleak sysfs file. I can't tell whether these changes would break them.

Cc'ing Dmitry, I think syzbot was regularly checking kmemleak (not sure it still does).

[...]

Quoted 2 lines

QED: there are no kmemleak parsers in public github repos that would break with these changes

Thanks for digging into this, I completely forgot about this series. Would you mind rebasing to the latest kernel and reposting?

Thanks.

Signature · 1 line
Catalin