Skip to contentSkip to conversation

check: add kmemleak support to blktests

PATCH Applied Nilay Shroff 5 participants 6 messages Jan 13 – Jan 22

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 check: add kmemleak support to blktests

Running blktests can also help uncover kernel memory leaks when the kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].

This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of /sys/kernel/debug/kmemleak to determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.

With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.

[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/

---
 check | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
check +47 -0
index 6d77d8e..3a6e837 100755
@@ -183,6 +183,36 @@ _check_dmesg() {
 	fi
 }
 
+_setup_kmemleak() {
+	local f="/sys/kernel/debug/kmemleak"
+
+	if [[ ! -e $f || ! -r $f ]]; then
+		return 0
+	fi
+
+	echo clear > "$f"
+}
+
+_check_kmemleak() {
+	local kmemleak
+	local f="/sys/kernel/debug/kmemleak"
+
+	if [[ ! -e $f || ! -r $f ]]; then
+		return 0
+	fi
+
+	echo scan > "$f"
+	sleep 1
+	kmemleak=$(cat "$f")
+
+	if [[ -z $kmemleak ]]; then
+		return 0
+	fi
+
+	printf '%s\n' "$kmemleak" > "${seqres}.kmemleak"
+	return 1
+}
+
 _read_last_test_run() {
 	local seqres="${RESULTS_DIR}/${TEST_NAME}"
 
@@ -377,6 +407,8 @@ _call_test() {
 	if [[ -v SKIP_REASONS ]]; then
 		TEST_RUN["status"]="not run"
 	else
+		_setup_kmemleak
+
 		if [[ -w /dev/kmsg ]]; then
 			local dmesg_marker="run blktests $TEST_NAME at ${TEST_RUN["date"]}"
 			echo "$dmesg_marker" >> /dev/kmsg
@@ -414,6 +446,9 @@ _call_test() {
 		elif ! _check_dmesg "$dmesg_marker"; then
 			TEST_RUN["status"]=fail
 			TEST_RUN["reason"]=dmesg
+		elif ! _check_kmemleak; then
+			TEST_RUN["status"]=fail
+			TEST_RUN["reason"]=kmemleak
 		else
 			TEST_RUN["status"]=pass
 		fi
@@ -451,6 +486,18 @@ _call_test() {
 				print \"    \" \$0
 			}" "${seqres}.dmesg"
 			;;
+		kmemleak)
+			echo "    kmemleak detected:"
+                        awk "
+                        {
+                                if (NR > 10) {
+                                        print \"    ...\"
+                                        print \"    (See '${seqres}.kmemleak' for the entire message)\"
+                                        exit
+                                }
+                                print \"    \" \$0
+                        }" "${seqres}.kmemleak"
+                        ;;
 		esac
 		return 1
 	else
Signature · 1 line
2.52.0
#2 lore raw
In reply to Nilay Shroff
Quoted 23 lines · Nilay Shroff

Running blktests can also help uncover kernel memory leaks when the kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].

This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of /sys/kernel/debug/kmemleak to determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.

With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.

Quoted 4 lines

[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/

Nilay, thank you very much. This idea is excellent, and the patch looks good except one nit comment below. I also did trial runs, and confirmed memory leaks can be detected and reported as test case failures. Great :)

Quoted 4 lines
check +0 -0
index 6d77d8e..3a6e837 100755

[...]

Quoted 6 lines
check +2 -0
@@ -451,6 +486,18 @@ _call_test() {
 				print \"    \" \$0
 			}" "${seqres}.dmesg"
 			;;
+		kmemleak)
+			echo "    kmemleak detected:"

check _call_test() { · line 500

Quoted 10 lines
check +10 -0
+                        awk "
+                        {
+                                if (NR > 10) {
+                                        print \"    ...\"
+                                        print \"    (See '${seqres}.kmemleak' for the entire message)\"
+                                        exit
+                                }
+                                print \"    \" \$0
+                        }" "${seqres}.kmemleak"
+                        ;;

Nit: the hunk above uses spaces for indent. Assuming there is no other review comments on this patch, I will replace the spaces with tabs when I apply it. I will wait a few more days to see someone makes comments or not. Thanks.

Reviewed-by on v1 1/1
#3 lore raw
In reply to Nilay Shroff
Quoted 18 lines · Nilay Shroff

Running blktests can also help uncover kernel memory leaks when the kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].

This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of /sys/kernel/debug/kmemleak to determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.

Quoted 9 lines

With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.

[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/

Thanks for the patch, that's really helpful to catch the kmemleak issue.

Quoted 85 lines
---
 check | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
check +47 -1
index 6d77d8e..3a6e837 100755
@@ -183,6 +183,36 @@ _check_dmesg() {
        fi
 }
 
+_setup_kmemleak() {
+       local f="/sys/kernel/debug/kmemleak"
+
+       if [[ ! -e $f || ! -r $f ]]; then
+               return 0
+       fi
+
+       echo clear > "$f"
+}
+
+_check_kmemleak() {
+       local kmemleak
+       local f="/sys/kernel/debug/kmemleak"
+
+       if [[ ! -e $f || ! -r $f ]]; then
+               return 0
+       fi
+
+       echo scan > "$f"
+       sleep 1
+       kmemleak=$(cat "$f")
+
+       if [[ -z $kmemleak ]]; then
+               return 0
+       fi
+
+       printf '%s\n' "$kmemleak" > "${seqres}.kmemleak"
+       return 1
+}
+
 _read_last_test_run() {
        local seqres="${RESULTS_DIR}/${TEST_NAME}"
 
@@ -377,6 +407,8 @@ _call_test() {
        if [[ -v SKIP_REASONS ]]; then
                TEST_RUN["status"]="not run"
        else
+               _setup_kmemleak
+
                if [[ -w /dev/kmsg ]]; then
                        local dmesg_marker="run blktests $TEST_NAME at ${TEST_RUN["date"]}"
                        echo "$dmesg_marker" >> /dev/kmsg
@@ -414,6 +446,9 @@ _call_test() {
                elif ! _check_dmesg "$dmesg_marker"; then
                        TEST_RUN["status"]=fail
                        TEST_RUN["reason"]=dmesg
+               elif ! _check_kmemleak; then
+                       TEST_RUN["status"]=fail
+                       TEST_RUN["reason"]=kmemleak
                else
                        TEST_RUN["status"]=pass
                fi
@@ -451,6 +486,18 @@ _call_test() {
                                print \"    \" \$0
                        }" "${seqres}.dmesg"
                        ;;
+               kmemleak)
+                       echo "    kmemleak detected:"
+                        awk "
+                        {
+                                if (NR > 10) {
+                                        print \"    ...\"
+                                        print \"    (See '${seqres}.kmemleak' for the entire message)\"
+                                        exit
+                                }
+                                print \"    \" \$0
+                        }" "${seqres}.kmemleak"
+                        ;;
                esac
                return 1
        else
--
Quoted 2 lines

2.52.0

--
Best Regards,
Yi Zhang

Reviewed-by on v1 1/1
#4 lore raw
In reply to Nilay Shroff
Quoted 17 lines · Nilay Shroff

Running blktests can also help uncover kernel memory leaks when the kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].

This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of /sys/kernel/debug/kmemleak to determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.

Quoted 8 lines

With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.

[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/

Nice!

#5 lore raw
In reply to Nilay Shroff
Quoted 23 lines · Nilay Shroff

Running blktests can also help uncover kernel memory leaks when the kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].

This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of /sys/kernel/debug/kmemleak to determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.

With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.

Quoted 4 lines

[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/

I applied this patch replacing the spaces with tabs. Nilay, thank you for the contribution :)

I ran whole blktests with this change and observed two kmemleak failures at nvme/061 and rnbd/002. FYI, here share the failure symptoms. Debug support the nvme/061 failure will be appreciated. I plan to post a fix patch for the rnbd/002 failure.

1) nvme/061 (tr=rdma, tr=tcp)

When nvme/061 is ran for rdma transport and tcp transport in sequence, kemeleak error below was observed at the run for tcp transport. The backtrace indicates the leak happened for rdma transport.

unreferenced object 0xffff88812b7d8cc0 (size 32):
comm "kworker/2:0H", pid 66707, jiffies 4298507259 hex dump (first 32 bytes):

    82 69 85 06 00 ea ff ff 00 00 00 00 00 10 00 00  .i..............
    00 60 5a a1 81 88 ff ff 00 10 00 00 00 00 00 00  .`Z.............
  backtrace (crc e0e63884):
    __kmalloc_noprof+0x62f/0x8b0
    sgl_alloc_order+0x74/0x330
    nvmet_req_alloc_sgls+0x283/0x4e0 [nvmet]
    0xffffffffc1d60f0d
    0xffffffffc1d67064
    __ib_process_cq+0x14f/0x3e0 [ib_core]
    ib_cq_poll_work+0x49/0x160 [ib_core]
    process_one_work+0x868/0x1490
    worker_thread+0x5ee/0xfd0
    kthread+0x3af/0x770
    ret_from_fork+0x55c/0x810
    ret_from_fork_asm+0x1a/0x30
unreferenced object 0xffff88812b7d8a00 (size 32):
  comm "kworker/2:0H", pid 66707, jiffies 4298507259
  hex dump (first 32 bytes):
    c2 65 85 06 00 ea ff ff 00 00 00 00 00 10 00 00  .e..............
    00 70 59 a1 81 88 ff ff 00 10 00 00 00 00 00 00  .pY.............
  backtrace (crc e120dc78):
    __kmalloc_noprof+0x62f/0x8b0
    sgl_alloc_order+0x74/0x330
    nvmet_req_alloc_sgls+0x283/0x4e0 [nvmet]
    0xffffffffc1d60f0d
    0xffffffffc1d67064
    __ib_process_cq+0x14f/0x3e0 [ib_core]
    ib_cq_poll_work+0x49/0x160 [ib_core]
    process_one_work+0x868/0x1490
    worker_thread+0x5ee/0xfd0
    kthread+0x3af/0x770
    ret_from_fork+0x55c/0x810
    ret_from_fork_asm+0x1a/0x30

...

2) rnbd/002

The kememleak message was as follows.

unreferenced object 0xffff88826c5f8720 (size 8):
comm "check", pid 145048, jiffies 4301507712
hex dump (first 8 bytes):

    72 6e 62 64 00 15 b3 a6                          rnbd....
  backtrace (crc f6c244b4):
    __kmalloc_node_track_caller_noprof+0x69d/0x900
    kstrdup+0x42/0xc0
    kobject_set_name_vargs+0x44/0x110
    kobject_init_and_add+0xcf/0x150
    rnbd_clt_map_device_store.cold+0x164/0x792 [rnbd_client]
    kernfs_fop_write_iter+0x3d6/0x5e0
    vfs_write+0x52c/0xf80
    ksys_write+0xfb/0x200
    do_syscall_64+0x95/0x540
    entry_SYSCALL_64_after_hwframe+0x76/0x7e

I tried out a fix patch below, and it avoided the kmemleak message.

drivers/block/rnbd/rnbd-clt.c +1 -0
index 8194a970f002..db6890121012 100644
@@ -60,6 +60,7 @@ static void rnbd_clt_put_dev(struct rnbd_clt_dev *dev)
        kfree(dev->pathname);
        rnbd_clt_put_sess(dev->sess);
        mutex_destroy(&dev->lock);
+       kobject_put(&dev->kobj);
        kfree(dev);
 }