check: add kmemleak support to blktests
Subsystems (2)
PATCH 1/1 check: add kmemleak support to blktests
Running blktests can also help uncover kernel memory leaks when the kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].
This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of /sys/kernel/debug/kmemleak to determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.
With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.
[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
--- check | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
check +47 -0
@@ -183,6 +183,36 @@ _check_dmesg() { fi } +_setup_kmemleak() { + local f="/sys/kernel/debug/kmemleak" + + if [[ ! -e $f || ! -r $f ]]; then + return 0 + fi + + echo clear > "$f" +} + +_check_kmemleak() { + local kmemleak + local f="/sys/kernel/debug/kmemleak" + + if [[ ! -e $f || ! -r $f ]]; then + return 0 + fi + + echo scan > "$f" + sleep 1 + kmemleak=$(cat "$f") + + if [[ -z $kmemleak ]]; then + return 0 + fi + + printf '%s\n' "$kmemleak" > "${seqres}.kmemleak" + return 1 +} + _read_last_test_run() { local seqres="${RESULTS_DIR}/${TEST_NAME}"
@@ -377,6 +407,8 @@ _call_test() { if [[ -v SKIP_REASONS ]]; then TEST_RUN["status"]="not run" else + _setup_kmemleak + if [[ -w /dev/kmsg ]]; then local dmesg_marker="run blktests $TEST_NAME at ${TEST_RUN["date"]}" echo "$dmesg_marker" >> /dev/kmsg
@@ -414,6 +446,9 @@ _call_test() { elif ! _check_dmesg "$dmesg_marker"; then TEST_RUN["status"]=fail TEST_RUN["reason"]=dmesg + elif ! _check_kmemleak; then + TEST_RUN["status"]=fail + TEST_RUN["reason"]=kmemleak else TEST_RUN["status"]=pass fi
@@ -451,6 +486,18 @@ _call_test() { print \" \" \$0 }" "${seqres}.dmesg" ;; + kmemleak) + echo " kmemleak detected:" + awk " + { + if (NR > 10) { + print \" ...\" + print \" (See '${seqres}.kmemleak' for the entire message)\" + exit + } + print \" \" \$0 + }" "${seqres}.kmemleak" + ;; esac return 1 else
Signature · 1 line
2.52.0
Looks good to me,
Quoted 23 lines · Nilay Shroff
Running blktests can also help uncover kernel memory leaks when the kernel is built with
CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of
/sys/kernel/debug/kmemleakto determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with
CONFIG_DEBUG_KMEMLEAKsupport) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.
Quoted 4 lines
[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/Signed-off-by Nilay Shroff <[email protected]>
Nilay, thank you very much. This idea is excellent, and the patch looks good except one nit comment below. I also did trial runs, and confirmed memory leaks can be detected and reported as test case failures. Great :)
Quoted 4 lines
check +0
-0
[...]
Quoted 6 lines
check +2
-0@@ -451,6 +486,18 @@ _call_test() { print \" \" \$0 }" "${seqres}.dmesg" ;; + kmemleak) + echo " kmemleak detected:"
check _call_test() { · line 500
Quoted 10 lines
check +10
-0+ awk " + { + if (NR > 10) { + print \" ...\" + print \" (See '${seqres}.kmemleak' for the entire message)\" + exit + } + print \" \" \$0 + }" "${seqres}.kmemleak" + ;;
Nit: the hunk above uses spaces for indent. Assuming there is no other review comments on this patch, I will replace the spaces with tabs when I apply it. I will wait a few more days to see someone makes comments or not. Thanks.
Quoted 18 lines · Nilay Shroff
Running blktests can also help uncover kernel memory leaks when the kernel is built with
CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of
/sys/kernel/debug/kmemleakto determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.
Quoted 9 lines
With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with
CONFIG_DEBUG_KMEMLEAKsupport) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
Thanks for the patch, that's really helpful to catch the kmemleak issue.
Quoted 85 lines
Signed-off-by Nilay Shroff <[email protected]>--- check | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)check +47
-1@@ -183,6 +183,36 @@ _check_dmesg() { fi } +_setup_kmemleak() { + local f="/sys/kernel/debug/kmemleak" + + if [[ ! -e $f || ! -r $f ]]; then + return 0 + fi + + echo clear > "$f" +} + +_check_kmemleak() { + local kmemleak + local f="/sys/kernel/debug/kmemleak" + + if [[ ! -e $f || ! -r $f ]]; then + return 0 + fi + + echo scan > "$f" + sleep 1 + kmemleak=$(cat "$f") + + if [[ -z $kmemleak ]]; then + return 0 + fi + + printf '%s\n' "$kmemleak" > "${seqres}.kmemleak" + return 1 +} + _read_last_test_run() { local seqres="${RESULTS_DIR}/${TEST_NAME}"@@ -377,6 +407,8 @@ _call_test() { if [[ -v SKIP_REASONS ]]; then TEST_RUN["status"]="not run" else + _setup_kmemleak + if [[ -w /dev/kmsg ]]; then local dmesg_marker="run blktests $TEST_NAME at ${TEST_RUN["date"]}" echo "$dmesg_marker" >> /dev/kmsg@@ -414,6 +446,9 @@ _call_test() { elif ! _check_dmesg "$dmesg_marker"; then TEST_RUN["status"]=fail TEST_RUN["reason"]=dmesg + elif ! _check_kmemleak; then + TEST_RUN["status"]=fail + TEST_RUN["reason"]=kmemleak else TEST_RUN["status"]=pass fi@@ -451,6 +486,18 @@ _call_test() { print \" \" \$0 }" "${seqres}.dmesg" ;; + kmemleak) + echo " kmemleak detected:" + awk " + { + if (NR > 10) { + print \" ...\" + print \" (See '${seqres}.kmemleak' for the entire message)\" + exit + } + print \" \" \$0 + }" "${seqres}.kmemleak" + ;; esac return 1 else --
Quoted 2 lines
2.52.0
--
Best Regards,
Yi Zhang
Quoted 17 lines · Nilay Shroff
Running blktests can also help uncover kernel memory leaks when the kernel is built with
CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of
/sys/kernel/debug/kmemleakto determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.
Quoted 8 lines
With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with
CONFIG_DEBUG_KMEMLEAKsupport) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
Nice!
Shinichiro Kawasaki
AppliedQuoted 23 lines · Nilay Shroff
Running blktests can also help uncover kernel memory leaks when the kernel is built with
CONFIG_DEBUG_KMEMLEAK. However, until now the blktests framework had no way to automatically detect or report such leaks. Users typically had to manually setup kmemleak and trigger scans after running tests[1][2].This change integrates kmemleak support directly into the blktests framework. Before running each test, the framework checks for the presence of
/sys/kernel/debug/kmemleakto determine whether kmemleak is enabled for the running kernel. If available, before running a test, any existing kmemleak reports are cleared to avoid false positives from previous tests. After the test completes, the framework explicitly triggers a kmemleak scan. If memory leaks are detected, they are written to a per-test file at, "results/.../.../<test>.kmemleak" and the corresponding test is marked as FAIL. Users can then inspect the <test>.kmemleak file to analyze the reported leaks.With this enhancement, blktests can automatically detect kernel memory leaks (if kerel is configured with
CONFIG_DEBUG_KMEMLEAKsupport) on a per-test basis, removing the need for manual kmemleak setup and scans. This should make it easier and faster to identify memory leaks introduced by individual tests.
Quoted 4 lines
[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/Signed-off-by Nilay Shroff <[email protected]>
I applied this patch replacing the spaces with tabs. Nilay, thank you for the contribution :)
I ran whole blktests with this change and observed two kmemleak failures at nvme/061 and rnbd/002. FYI, here share the failure symptoms. Debug support the nvme/061 failure will be appreciated. I plan to post a fix patch for the rnbd/002 failure.
1) nvme/061 (tr=rdma, tr=tcp)
When nvme/061 is ran for rdma transport and tcp transport in sequence, kemeleak error below was observed at the run for tcp transport. The backtrace indicates the leak happened for rdma transport.
unreferenced object 0xffff88812b7d8cc0 (size 32):
comm "kworker/2:0H", pid 66707, jiffies 4298507259 hex dump (first 32 bytes):
82 69 85 06 00 ea ff ff 00 00 00 00 00 10 00 00 .i..............
00 60 5a a1 81 88 ff ff 00 10 00 00 00 00 00 00 .`Z.............
backtrace (crc e0e63884):
__kmalloc_noprof+0x62f/0x8b0
sgl_alloc_order+0x74/0x330
nvmet_req_alloc_sgls+0x283/0x4e0 [nvmet]
0xffffffffc1d60f0d
0xffffffffc1d67064
__ib_process_cq+0x14f/0x3e0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x868/0x1490
worker_thread+0x5ee/0xfd0
kthread+0x3af/0x770
ret_from_fork+0x55c/0x810
ret_from_fork_asm+0x1a/0x30
unreferenced object 0xffff88812b7d8a00 (size 32):
comm "kworker/2:0H", pid 66707, jiffies 4298507259
hex dump (first 32 bytes):
c2 65 85 06 00 ea ff ff 00 00 00 00 00 10 00 00 .e..............
00 70 59 a1 81 88 ff ff 00 10 00 00 00 00 00 00 .pY.............
backtrace (crc e120dc78):
__kmalloc_noprof+0x62f/0x8b0
sgl_alloc_order+0x74/0x330
nvmet_req_alloc_sgls+0x283/0x4e0 [nvmet]
0xffffffffc1d60f0d
0xffffffffc1d67064
__ib_process_cq+0x14f/0x3e0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x868/0x1490
worker_thread+0x5ee/0xfd0
kthread+0x3af/0x770
ret_from_fork+0x55c/0x810
ret_from_fork_asm+0x1a/0x30...
2) rnbd/002
The kememleak message was as follows.
unreferenced object 0xffff88826c5f8720 (size 8):
comm "check", pid 145048, jiffies 4301507712
hex dump (first 8 bytes):
72 6e 62 64 00 15 b3 a6 rnbd....
backtrace (crc f6c244b4):
__kmalloc_node_track_caller_noprof+0x69d/0x900
kstrdup+0x42/0xc0
kobject_set_name_vargs+0x44/0x110
kobject_init_and_add+0xcf/0x150
rnbd_clt_map_device_store.cold+0x164/0x792 [rnbd_client]
kernfs_fop_write_iter+0x3d6/0x5e0
vfs_write+0x52c/0xf80
ksys_write+0xfb/0x200
do_syscall_64+0x95/0x540
entry_SYSCALL_64_after_hwframe+0x76/0x7eI tried out a fix patch below, and it avoided the kmemleak message.
drivers/block/rnbd/rnbd-clt.c +1 -0
@@ -60,6 +60,7 @@ static void rnbd_clt_put_dev(struct rnbd_clt_dev *dev)kfree (dev->pathname);rnbd_clt_put_sess (dev->sess);mutex_destroy (&dev->lock); +kobject_put (&dev->kobj);kfree (dev); }