kmemleak in rdma_core observed during blktests nvme/rdma use siw
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
unreferenced object 0xffff88855da53400 (size 192):
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes):
37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7...............
10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.]....
backtrace (crc 47f66721):
[<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0
[<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core]
[<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
[<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
[<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core]
[<ffffffffc2a3d389>] 0xffffffffc2a3d389
[<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
[<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]
[<ffffffffc264648c>]
rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[<ffffffff9270e7b5>] netlink_unicast+0x445/0x710
[<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40
[<ffffffff9249db29>] __sys_sendto+0x3a9/0x420
[<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0
[<ffffffff92db0ad3>] do_syscall_64+0x93/0x180
[<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Signature · 2 lines
Best Regards, Yi Zhang
Leon Romanovsky
maintainerQuoted 21 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes):
Quoted 11 lines
37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7...............
10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.]....
backtrace (crc 47f66721):
[<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0
[<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core]
[<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
[<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
[<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core]
[<ffffffffc2a3d389>] 0xffffffffc2a3d389
[<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core]My guess is that your test didn't call too nldev_dellink() and left device registered.
Thanks
Quoted 15 lines
[<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Signature · 2 lines
Best Regards, Yi Zhang
Leon Romanovsky
maintainerQuoted 34 lines · Leon Romanovsky
Quoted 21 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes):Quoted 11 lines
37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
Quoted 2 lines
My guess is that your test didn't call too
nldev_dellink()and left device registered.
Sorry, I was wrong, it looks like GID entry leak.
Thanks
Quoted 19 lines
Thanks
Quoted 15 lines
[<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Signature · 2 lines
Best Regards, Yi Zhang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit 586b5dfb51b) with the below.
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):
Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922
hex dump (first 32 bytes):
37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7...............
10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.]....
backtrace (crc 47f66721):
[<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0
[<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core]
[<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess add_modify_gid is called from config_non_roce_gid_cache, not sure why we don't check the return value of it here.
Looks put_gid_entry is called in case add_modify_gid returns failure, it would trigger schedule_free_gid -> queue_work(ib_wq, &entry->del_work), then free_gid_work -> free_gid_entry_locked would free storage asynchronously by put_gid_ndev and also entry.
Quoted 6 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
[<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
[<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core]
[<ffffffffc2a3d389>] 0xffffffffc2a3d389
[<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
[<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]Quoted 8 lines
[<ffffffffc264648c>]
rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[<ffffffff9270e7b5>] netlink_unicast+0x445/0x710
[<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40
[<ffffffff9249db29>] __sys_sendto+0x3a9/0x420
[<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0
[<ffffffff92db0ad3>] do_syscall_64+0x93/0x180
[<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79After ib_cache_setup_one failed, maybe ib_cache_cleanup_one is needed which flush ib_wq to ensure storage is freed. Could you try with the change?
drivers/infiniband/core/device.c +0 -0
@@ -1388,7 +1388,7 @@ int ib_register_device(struct ib_device *device, const char *name,
if (ret) {
dev_warn(&device->dev,
"Couldn't set up InfiniBand P_Key/GID cache\n"); - return ret;
+ goto cache_cleanup;
}
Thanks,
Guoqing
Quoted 11 lines · Guoqing Jiang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.
It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object 0xffff8883a87e8800 (size 192):
comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes):
32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2...............
10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~.....
backtrace (crc 4db191c4):
[<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0
[<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core]
[<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core]
[<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
[<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
[<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core]
[<ffffffffc24ac389>] 0xffffffffc24ac389
[<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
[<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]
[<ffffffffc208448c>]
rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[<ffffffff8e30e715>] netlink_unicast+0x445/0x710
[<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40
[<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420
[<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0
[<ffffffff8e9afad3>] do_syscall_64+0x93/0x180
[<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)0x2eff7 is in alloc_gid_entry (./include/linux/slab.h:628). 623
624 if (size > KMALLOC_MAX_CACHE_SIZE)
625 return kmalloc_large(size, flags);
626
627 index = kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return __kmalloc(size, flags);
(gdb) l *(add_modify_gid+0x166)0x30206 is in add_modify_gid (drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry = alloc_gid_entry(attr);
448 if (!entry)
449 return -ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {
Quoted 50 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 6 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]Quoted 8 lines
[<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?
Will try it later.
Quoted 12 lines
drivers/infiniband/core/device.c +1
-1@@ -1388,7 +1388,7 @@ int ib_register_device(struct ib_device *device, const char *name, if (ret) {dev_warn (&device->dev,"Couldn't set up InfiniBand P_Key/GID cache\n" ); -return ret; + goto cache_cleanup; }
Quoted 3 lines
Thanks,
Guoqing
--
Best Regards,
Yi Zhang
Quoted 11 lines · Yi Zhang
Quoted 9 lines · Guoqing Jiang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.
Quoted 6 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
Quoted 104 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):
comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes):32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 50 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79
Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object 0xffff8881f89fde00 (size 192):
comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes):
02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................
10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................
backtrace (crc 888c494b):
[<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0
[<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core]
[<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core]
[<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
[<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
[<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core]
[<ffffffffc22ee389>]
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520 [siw]
[<ffffffffc1f46cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
[<ffffffffc1f03fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]
[<ffffffffc1f0448c>]
rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[<ffffffffa930e715>] netlink_unicast+0x445/0x710
[<ffffffffa930f151>] netlink_sendmsg+0x761/0xc40
[<ffffffffa909da89>] __sys_sendto+0x3a9/0x420
[<ffffffffa909dbec>] __x64_sys_sendto+0xdc/0x1b0
[<ffffffffa99afad3>] do_syscall_64+0x93/0x180
[<ffffffffa9a00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 20 lines
Quoted 12 lines
drivers/infiniband/core/device.c +1
-1@@ -1388,7 +1388,7 @@ int ib_register_device(struct ib_device *device, const char *name, if (ret) {dev_warn (&device->dev,"Couldn't set up InfiniBand P_Key/GID cache\n" ); -return ret; + goto cache_cleanup; }Quoted 3 lines
Thanks,
Guoqing--
Best Regards,
Yi Zhang
Signature · 2 lines
Best Regards, Yi Zhang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Yi Zhang
Quoted 7 lines · Guoqing Jiang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed] runtime 52.763s ... 392.027s device: nvme0
Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]
Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]
Is it possible to run the test with rxe instead of siw? In case it is only happened with siw, I'd suggest to revert 0b988c1bee28 to check if it causes the issue. But I don't understand why siw_qp_state_to_ib_qp_state was appeared in the middle of above trace.
Quoted 10 lines
[<ffffffffc1f46cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
[<ffffffffc1f03fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]
[<ffffffffc1f0448c>]
rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[<ffffffffa930e715>] netlink_unicast+0x445/0x710
[<ffffffffa930f151>] netlink_sendmsg+0x761/0xc40
[<ffffffffa909da89>] __sys_sendto+0x3a9/0x420
[<ffffffffa909dbec>] __x64_sys_sendto+0xdc/0x1b0
[<ffffffffa99afad3>] do_syscall_64+0x93/0x180
[<ffffffffa9a00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Thanks,
Guoqing
Quoted 151 lines · Guoqing Jiang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Yi Zhang
Quoted 7 lines · Guoqing Jiang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]
Quoted 6 lines
Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.
Quoted 15 lines
Quoted 11 lines
[<ffffffffc1f46cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc1f03fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc1f0448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffffa930e715>] netlink_unicast+0x445/0x710 [<ffffffffa930f151>] netlink_sendmsg+0x761/0xc40 [<ffffffffa909da89>] __sys_sendto+0x3a9/0x420 [<ffffffffa909dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffffa99afad3>] do_syscall_64+0x93/0x180 [<ffffffffa9a00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Thanks,
Guoqing
Signature · 2 lines
Best Regards, Yi Zhang
Quoted 155 lines · Yi Zhang
Quoted 148 lines · Guoqing Jiang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Yi Zhang
Quoted 7 lines · Guoqing Jiang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Quoted 6 lines
Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.
Yes, pls bisect them.
Quoted 1 line
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3d RDMA/siw: Call llist_reverse_order in siw_run_sq bee024d20451 RDMA/siw: Correct wrong debug messageb056327bee09 RDMA/siw: Balance the reference of cep->kref in the error path 91f36237b4b9 RDMA/siw: Fix tx thread initialization. bad5b6e34ffb RDMA/siw: Fabricate a GID on tun and loopback devices 9191df002926 RDMA/siw: use vmalloc_array and vcalloc
Thanks,
Guoqing
Quoted 159 lines · Guoqing Jiang
Quoted 155 lines · Yi Zhang
Quoted 154 lines · Guoqing Jiang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Yi Zhang
Quoted 7 lines · Guoqing Jiang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.
Quoted 6 lines
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.Yes, pls bisect them.
Sure, will do that after I back from holiday next week.
Quoted 12 lines
Quoted 2 lines
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3dRDMA/siw: Callllist_reverse_orderinsiw_run_sqbee024d20451RDMA/siw: Correct wrong debug messageb056327bee09RDMA/siw: Balance the reference ofcep->krefin the error path91f36237b4b9RDMA/siw: Fix tx thread initialization.bad5b6e34ffbRDMA/siw: Fabricate a GID on tun and loopback devices9191df002926RDMA/siw: usevmalloc_arrayand vcallocThanks,
Guoqing
Signature · 2 lines
Best Regards, Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
commit f8ef1be816bf9a0c406c696368c2264a9597a994
Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400
RDMA/cma: Avoid GID lookups on iWARP devices
Quoted 187 lines · Yi Zhang
Quoted 159 lines · Guoqing Jiang
Quoted 155 lines · Yi Zhang
Quoted 154 lines · Guoqing Jiang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Yi Zhang
Quoted 7 lines · Guoqing Jiang
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.Quoted 6 lines
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.Yes, pls bisect them.
Sure, will do that after I back from holiday next week.
Quoted 9 lines
Quoted 2 lines
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3dRDMA/siw: Callllist_reverse_orderinsiw_run_sqbee024d20451RDMA/siw: Correct wrong debug messageb056327bee09RDMA/siw: Balance the reference ofcep->krefin the error path91f36237b4b9RDMA/siw: Fix tx thread initialization.bad5b6e34ffbRDMA/siw: Fabricate a GID on tun and loopback devices9191df002926RDMA/siw: usevmalloc_arrayand vcallocQuoted 3 lines
Thanks,
Guoqing--
Best Regards,
Yi Zhang
Signature · 2 lines
Best Regards, Yi Zhang
Quoted 8 lines · Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
commit
f8ef1be816bf9a0c406c696368c2264a9597a994Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400RDMA/cma: Avoid GID lookups on iWARP devices
Got it. Thanks a lot. I will delve into this problem.
Zhu Yanjun
Quoted 192 lines · Yi Zhang
Quoted 187 lines · Guoqing Jiang
Quoted 159 lines · Yi Zhang
Quoted 155 lines · Guoqing Jiang
Quoted 154 lines · Yi Zhang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Guoqing Jiang
Quoted 7 lines
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.Quoted 6 lines
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.Yes, pls bisect them.
Sure, will do that after I back from holiday next week.
Quoted 9 lines
Quoted 2 lines
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3dRDMA/siw: Callllist_reverse_orderinsiw_run_sqbee024d20451RDMA/siw: Correct wrong debug messageb056327bee09RDMA/siw: Balance the reference ofcep->krefin the error path91f36237b4b9RDMA/siw: Fix tx thread initialization.bad5b6e34ffbRDMA/siw: Fabricate a GID on tun and loopback devices9191df002926RDMA/siw: usevmalloc_arrayand vcallocQuoted 3 lines
Thanks,
Guoqing--
Best Regards,
Yi Zhang
Quoted 8 lines · Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
commit
f8ef1be816bf9a0c406c696368c2264a9597a994Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400RDMA/cma: Avoid GID lookups on iWARP devices
Not sure if the following can fix this problem or not. Please let me know the test result.
Thanks a lot.
drivers/infiniband/core/cma.c +5 -1
@@ -715,9 +715,13 @@ cma_validate_port(struct ib_device *device, u32 port,rcu_read_lock (); ndev =rcu_dereference (sgid_attr->ndev);if (!net_eq (dev_net (ndev), dev_addr->net) || - ndev->ifindex != bound_if_index) + ndev->ifindex != bound_if_index) { +rdma_put_gid_attr (sgid_attr); sgid_attr =ERR_PTR (-ENODEV ); + }rcu_read_unlock (); +if (!IS_ERR (sgid_attr)) +rdma_put_gid_attr (sgid_attr); goto out; }
Zhu Yanjun
Quoted 192 lines · Yi Zhang
Quoted 187 lines · Guoqing Jiang
Quoted 159 lines · Yi Zhang
Quoted 155 lines · Guoqing Jiang
Quoted 154 lines · Yi Zhang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Guoqing Jiang
Quoted 7 lines
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.Quoted 6 lines
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.Yes, pls bisect them.
Sure, will do that after I back from holiday next week.
Quoted 9 lines
Quoted 2 lines
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3dRDMA/siw: Callllist_reverse_orderinsiw_run_sqbee024d20451RDMA/siw: Correct wrong debug messageb056327bee09RDMA/siw: Balance the reference ofcep->krefin the error path91f36237b4b9RDMA/siw: Fix tx thread initialization.bad5b6e34ffbRDMA/siw: Fabricate a GID on tun and loopback devices9191df002926RDMA/siw: usevmalloc_arrayand vcallocQuoted 3 lines
Thanks,
Guoqing--
Best Regards,
Yi Zhang
Jason Gunthorpe
maintainerQuoted 22 lines · Zhu Yanjun
Quoted 8 lines · Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
commit
f8ef1be816bf9a0c406c696368c2264a9597a994Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400RDMA/cma: Avoid GID lookups on iWARP devices
Not sure if the following can fix this problem or not. Please let me know the test result.
Thanks a lot.drivers/infiniband/core/cma.c +0
-0@@ -715,9 +715,13 @@ cma_validate_port(struct ib_device *device, u32 port,rcu_read_lock (); ndev =rcu_dereference (sgid_attr->ndev); if (!net_eq(dev_net(ndev), dev_addr->net) ||
drivers/infiniband/core/cma.c cma_validate_port(struct ib_device *device, u32 port, · line 726
Quoted 10 lines
drivers/infiniband/core/cma.c +5
-1- ndev->ifindex != bound_if_index) + ndev->ifindex != bound_if_index) { +rdma_put_gid_attr (sgid_attr); sgid_attr =ERR_PTR (-ENODEV ); + }rcu_read_unlock (); +if (!IS_ERR (sgid_attr)) +rdma_put_gid_attr (sgid_attr); goto out; }
That does look needed regardless!
Chuck?
Jason
Chuck Lever
Reviewed-byQuoted 36 lines · Jason Gunthorpe
Quoted 22 lines · Zhu Yanjun
Quoted 8 lines · Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
commit
f8ef1be816bf9a0c406c696368c2264a9597a994Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400RDMA/cma: Avoid GID lookups on iWARP devices
Not sure if the following can fix this problem or not. Please let me know the test result.
Thanks a lot.drivers/infiniband/core/cma.c +0
-0@@ -715,9 +715,13 @@ cma_validate_port(struct ib_device *device, u32 port,rcu_read_lock (); ndev =rcu_dereference (sgid_attr->ndev); if (!net_eq(dev_net(ndev), dev_addr->net) ||drivers/infiniband/core/cma.c cma_validate_port(struct ib_device *device, u32 port, · line 726
Quoted 10 lines
drivers/infiniband/core/cma.c +5
-1- ndev->ifindex != bound_if_index) + ndev->ifindex != bound_if_index) { +rdma_put_gid_attr (sgid_attr); sgid_attr =ERR_PTR (-ENODEV ); + }rcu_read_unlock (); +if (!IS_ERR (sgid_attr)) +rdma_put_gid_attr (sgid_attr); goto out; }That does look needed regardless!
Quoted 1 line
Chuck?
Releasing the sgid_attr looks sensible.
Signature · 1 line
Chuck Lever
Quoted 9 lines · Zhu Yanjun
Quoted 7 lines · Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
commit
f8ef1be816bf9a0c406c696368c2264a9597a994Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400
Quoted 5 lines
Quoted 1 line
RDMA/cma: Avoid GID lookups on iWARP devices
Not sure if the following can fix this problem or not. Please let me know the test result.
Thanks a lot.
Hi Yanjun
Seems the change introduced new issue, here is the log:
[ 3017.171697] run blktests nvme/003 at 2024-05-09 21:35:41
[ 3032.344539] ==================================================================
[ 3032.351780] BUG: KASAN: slab-use-after-free in
rdma_put_gid_attr+0x23/0xa0 [ib_core]
[ 3032.359659] Write of size 4 at addr ffff88813c1c3f00 by task kworker/27:1/370
[ 3032.366791]
[ 3032.368293] CPU: 27 PID: 370 Comm: kworker/27:1 Not tainted
6.9.0-rc2.rdma.fix+ #1
[ 3032.375859] Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS
2.20.1 09/13/2023
[ 3032.383425] Workqueue: nvmet-wq nvmet_rdma_release_queue_work [nvmet_rdma]
[ 3032.390304] Call Trace:
[ 3032.392757] <TASK>
[ 3032.394864] dump_stack_lvl+0x84/0xd0
[ 3032.398537] ? rdma_put_gid_attr+0x23/0xa0 [ib_core]
[ 3032.403561] print_report+0x19d/0x52e
[ 3032.407228] ? rdma_put_gid_attr+0x23/0xa0 [ib_core]
[ 3032.412256] ? __virt_addr_valid+0x228/0x420
[ 3032.416537] ? rdma_put_gid_attr+0x23/0xa0 [ib_core]
[ 3032.421563] kasan_report+0xab/0x180
[ 3032.425142] ? rdma_put_gid_attr+0x23/0xa0 [ib_core]
[ 3032.430173] kasan_check_range+0x104/0x1b0
[ 3032.434275] rdma_put_gid_attr+0x23/0xa0 [ib_core]
[ 3032.439128] ? cma_dev_put+0x1f/0x60 [rdma_cm]
[ 3032.443591] cma_release_dev+0x1b2/0x270 [rdma_cm]
[ 3032.448401] _destroy_id+0x35f/0xc80 [rdma_cm]
[ 3032.452867] nvmet_rdma_free_queue+0x7a/0x380 [nvmet_rdma]
[ 3032.458360] nvmet_rdma_release_queue_work+0x42/0x90 [nvmet_rdma]
[ 3032.464460] process_one_work+0x85d/0x13e0
[ 3032.468573] ? worker_thread+0xcc/0x1130
[ 3032.472501] ? __pfx_process_one_work+0x10/0x10
[ 3032.477038] ? assign_work+0x16c/0x240
[ 3032.480797] worker_thread+0x6da/0x1130
[ 3032.484648] ? __pfx_worker_thread+0x10/0x10
[ 3032.488923] kthread+0x2ed/0x3c0
[ 3032.492155] ? _raw_spin_unlock_irq+0x28/0x60
[ 3032.496514] ? __pfx_kthread+0x10/0x10
[ 3032.500267] ret_from_fork+0x31/0x70
[ 3032.503854] ? __pfx_kthread+0x10/0x10
[ 3032.507607] ret_from_fork_asm+0x1a/0x30
[ 3032.511539] </TASK>
[ 3032.513734]
[ 3032.515234] Allocated by task 1997:
[ 3032.518725] kasan_save_stack+0x30/0x50
[ 3032.522562] kasan_save_track+0x14/0x30
[ 3032.526402] __kasan_kmalloc+0x8f/0xa0
[ 3032.530155] add_modify_gid+0x18e/0xb80 [ib_core]
[ 3032.534922] ib_cache_update.part.0+0x6fc/0x8e0 [ib_core]
[ 3032.540380] ib_cache_setup_one+0x3ff/0x5f0 [ib_core]
[ 3032.545495] ib_register_device+0x5ba/0xa20 [ib_core]
[ 3032.550607] siw_newlink+0xb0d/0xe50 [siw]
[ 3032.554724] nldev_newlink+0x301/0x520 [ib_core]
[ 3032.559404] rdma_nl_rcv_msg+0x2e7/0x600 [ib_core]
[ 3032.564256] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[ 3032.570756] netlink_unicast+0x437/0x6e0
[ 3032.574679] netlink_sendmsg+0x775/0xc10
[ 3032.578607] __sys_sendto+0x3e5/0x490
[ 3032.582273] __x64_sys_sendto+0xe0/0x1c0
[ 3032.586198] do_syscall_64+0x9a/0x1a0
[ 3032.589862] entry_SYSCALL_64_after_hwframe+0x71/0x79
[ 3032.594917]
[ 3032.596416] Freed by task 339:
[ 3032.599475] kasan_save_stack+0x30/0x50
[ 3032.603312] kasan_save_track+0x14/0x30
[ 3032.607153] kasan_save_free_info+0x3b/0x60
[ 3032.611338] poison_slab_object+0x103/0x190
[ 3032.615523] __kasan_slab_free+0x14/0x30
[ 3032.619450] kfree+0x120/0x3a0
[ 3032.622508] free_gid_work+0xd4/0x120 [ib_core]
[ 3032.627100] process_one_work+0x85d/0x13e0
[ 3032.631200] worker_thread+0x6da/0x1130
[ 3032.635038] kthread+0x2ed/0x3c0
[ 3032.638271] ret_from_fork+0x31/0x70
[ 3032.641849] ret_from_fork_asm+0x1a/0x30
[ 3032.645777]
[ 3032.647277] Last potentially related work creation:
[ 3032.652153] kasan_save_stack+0x30/0x50
[ 3032.655994] __kasan_record_aux_stack+0x8e/0xa0
[ 3032.660525] insert_work+0x36/0x310
[ 3032.664019] __queue_work+0x6a4/0xcb0
[ 3032.667685] queue_work_on+0x99/0xb0
[ 3032.671263] cma_release_dev+0x1b2/0x270 [rdma_cm]
[ 3032.676072] _destroy_id+0x35f/0xc80 [rdma_cm]
[ 3032.680537] nvme_rdma_free_queue+0x4a/0x70 [nvme_rdma]
[ 3032.685768] nvme_do_delete_ctrl+0x146/0x160 [nvme_core]
[ 3032.691108] nvme_delete_ctrl_sync.cold+0x8/0xd [nvme_core]
[ 3032.696707] nvme_sysfs_delete+0x96/0xc0 [nvme_core]
[ 3032.701696] kernfs_fop_write_iter+0x3a5/0x5b0
[ 3032.706142] vfs_write+0x62e/0x1010
[ 3032.709636] ksys_write+0xfb/0x1d0
[ 3032.713041] do_syscall_64+0x9a/0x1a0
[ 3032.716707] entry_SYSCALL_64_after_hwframe+0x71/0x79
[ 3032.721760]
[ 3032.723259] The buggy address belongs to the object at ffff88813c1c3f00
[ 3032.723259] which belongs to the cache kmalloc-rnd-07-192 of size 192
[ 3032.736370] The buggy address is located 0 bytes inside of
[ 3032.736370] freed 192-byte region [ffff88813c1c3f00, ffff88813c1c3fc0)
[ 3032.748441]
[ 3032.749942] The buggy address belongs to the physical page:
[ 3032.755513] page: refcount:1 mapcount:0 mapping:0000000000000000
index:0x0 pfn:0x13c1c2
[ 3032.763511] head: order:1 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[ 3032.770212] flags:
0x17ffffe0000840(slab|head|node=0|zone=2|lastcpupid=0x3fffff)
[ 3032.777604] page_type: 0xffffffff()
[ 3032.781097] raw: 0017ffffe0000840 ffff888100053c00 dead000000000122
0000000000000000
[ 3032.788837] raw: 0000000000000000 0000000080200020 00000001ffffffff
0000000000000000
[ 3032.796574] head: 0017ffffe0000840 ffff888100053c00
dead000000000122 0000000000000000
[ 3032.804400] head: 0000000000000000 0000000080200020
00000001ffffffff 0000000000000000
[ 3032.812225] head: 0017ffffe0000001 ffffea0004f07081
ffffea0004f070c8 00000000ffffffff
[ 3032.820050] head: 0000000200000000 0000000000000000
00000000ffffffff 0000000000000000
[ 3032.827874] page dumped because: kasan: bad access detected
[ 3032.833447]
[ 3032.834944] Memory state around the buggy address:
[ 3032.839737] ffff88813c1c3e00: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 3032.846958] ffff88813c1c3e80: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 3032.854177] >ffff88813c1c3f00: fa fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 3032.861393] ^
[ 3032.864628] ffff88813c1c3f80: fb fb fb fb fb fb fb fb fc fc fc fc
fc fc fc fc
[ 3032.871845] ffff88813c1c4000: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 3032.879065] ==================================================================
[ 3032.886311] Disabling lock debugging due to kernel taint
[ 3032.891630] ------------[ cut here ]------------
[ 3032.896255] refcount_t: underflow; use-after-free.
[ 3032.901104] WARNING: CPU: 27 PID: 370 at lib/refcount.c:28
refcount_warn_saturate+0xf2/0x150
[ 3032.909552] Modules linked in: siw ib_uverbs nvmet_rdma nvmet
nvme_keyring nvme_rdma nvme_fabrics nvme_core nvme_auth rdma_cm iw_cm
ib_cm ib_core intel_rapl_msr intel_rapl_coma
[ 3032.992273] CPU: 27 PID: 370 Comm: kworker/27:1 Tainted: G B
6.9.0-rc2.rdma.fix+ #1
[ 3033.001324] Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS
2.20.1 09/13/2023
[ 3033.008899] Workqueue: nvmet-wq nvmet_rdma_release_queue_work [nvmet_rdma]
[ 3033.015787] RIP: 0010:refcount_warn_saturate+0xf2/0x150
[ 3033.021022] Code: 2f 1b 66 04 01 e8 6e f7 9c fe 0f 0b eb 91 80 3d
1e 1b 66 04 00 75 88 48 c7 c7 80 16 c5 91 c6 05 0e 1b 66 04 01 e8 4e
f7 9c fe <0f> 0b e9 6e ff ff ff 80 3d fe7
[ 3033.039775] RSP: 0018:ffffc9000e627c10 EFLAGS: 00010286
[ 3033.045019] RAX: 0000000000000000 RBX: ffff88813c1c3f00 RCX: 0000000000000000
[ 3033.052158] RDX: 0000000000000000 RSI: ffffffff91c5c760 RDI: 0000000000000001
[ 3033.059303] RBP: 0000000000000003 R08: 0000000000000001 R09: fffff52001cc4f36
[ 3033.066449] R10: ffffc9000e6279b7 R11: 0000000000000001 R12: ffff88a88f394248
[ 3033.073588] R13: ffff888135750240 R14: ffff88825385fb80 R15: dead000000000100
[ 3033.080730] FS: 0000000000000000(0000) GS:ffff88c7ad200000(0000)
knlGS:0000000000000000
[ 3033.088824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3033.094579] CR2: 00007fdfcb573c58 CR3: 0000000ea6e98004 CR4: 00000000007706f0
[ 3033.101720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3033.108859] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3033.116001] PKRU: 55555554
[ 3033.118723] Call Trace:
[ 3033.121182] <TASK>
[ 3033.123300] ? __warn+0xcc/0x170
[ 3033.126548] ? refcount_warn_saturate+0xf2/0x150
[ 3033.131176] ? report_bug+0x1fc/0x3d0
[ 3033.134860] ? handle_bug+0x3c/0x80
[ 3033.138368] ? exc_invalid_op+0x17/0x40
[ 3033.142217] ? asm_exc_invalid_op+0x1a/0x20
[ 3033.146428] ? refcount_warn_saturate+0xf2/0x150
[ 3033.151056] cma_release_dev+0x1b2/0x270 [rdma_cm]
[ 3033.155874] _destroy_id+0x35f/0xc80 [rdma_cm]
[ 3033.160348] nvmet_rdma_free_queue+0x7a/0x380 [nvmet_rdma]
[ 3033.165851] nvmet_rdma_release_queue_work+0x42/0x90 [nvmet_rdma]
[ 3033.171958] process_one_work+0x85d/0x13e0
[ 3033.176077] ? worker_thread+0xcc/0x1130
[ 3033.180017] ? __pfx_process_one_work+0x10/0x10
[ 3033.184561] ? assign_work+0x16c/0x240
[ 3033.188332] worker_thread+0x6da/0x1130
[ 3033.192187] ? __pfx_worker_thread+0x10/0x10
[ 3033.196475] kthread+0x2ed/0x3c0
[ 3033.199724] ? _raw_spin_unlock_irq+0x28/0x60
[ 3033.204099] ? __pfx_kthread+0x10/0x10
[ 3033.207860] ret_from_fork+0x31/0x70
[ 3033.211447] ? __pfx_kthread+0x10/0x10
[ 3033.215209] ret_from_fork_asm+0x1a/0x30
[ 3033.219152] </TASK>
[ 3033.221352] irq event stamp: 255979
[ 3033.224855] hardirqs last enabled at (255979):
[<ffffffff9160160a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 3033.234863] hardirqs last disabled at (255978):
[<ffffffff91433f0a>] __do_softirq+0x75a/0x967
[ 3033.243391] softirqs last enabled at (255766):
[<ffffffff8e269f56>] __irq_exit_rcu+0xc6/0x1d0
[ 3033.252015] softirqs last disabled at (255757):
[<ffffffff8e269f56>] __irq_exit_rcu+0xc6/0x1d0
[ 3033.260637] ---[ end trace 0000000000000000 ]---Quoted 214 lines
drivers/infiniband/core/cma.c +5
-1@@ -715,9 +715,13 @@ cma_validate_port(struct ib_device *device, u32 port,rcu_read_lock (); ndev =rcu_dereference (sgid_attr->ndev);if (!net_eq (dev_net (ndev), dev_addr->net) || - ndev->ifindex != bound_if_index) + ndev->ifindex != bound_if_index) { +rdma_put_gid_attr (sgid_attr); sgid_attr =ERR_PTR (-ENODEV ); + }rcu_read_unlock (); +if (!IS_ERR (sgid_attr)) +rdma_put_gid_attr (sgid_attr); goto out; }Zhu Yanjun
Quoted 192 lines · Yi Zhang
Quoted 187 lines · Guoqing Jiang
Quoted 159 lines · Yi Zhang
Quoted 155 lines · Guoqing Jiang
Quoted 154 lines · Yi Zhang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Guoqing Jiang
Quoted 7 lines
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.Quoted 6 lines
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.Yes, pls bisect them.
Sure, will do that after I back from holiday next week.
Quoted 9 lines
Quoted 2 lines
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3dRDMA/siw: Callllist_reverse_orderinsiw_run_sqbee024d20451RDMA/siw: Correct wrong debug messageb056327bee09RDMA/siw: Balance the reference ofcep->krefin the error path91f36237b4b9RDMA/siw: Fix tx thread initialization.bad5b6e34ffbRDMA/siw: Fabricate a GID on tun and loopback devices9191df002926RDMA/siw: usevmalloc_arrayand vcallocQuoted 3 lines
Thanks,
Guoqing--
Best Regards,
Yi Zhang
Signature · 2 lines
Best Regards, Yi Zhang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Zhu Yanjun
Quoted 7 lines · Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
commit
f8ef1be816bf9a0c406c696368c2264a9597a994Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
RDMA/cma: Avoid GID lookups on iWARP devices
Not sure if the following can fix this problem or not. Please let me know the test result.
Thanks a lot.Hi Yanjun
Seems the change introduced new issue, here is the log:
Yeah. I can reproduce this problem. I will delve into this problem.
Zhu Yanjun
Quoted 404 lines
[ 3017.171697] run blktests nvme/003 at 2024-05-09 21:35:41 [ 3032.344539] ================================================================== [ 3032.351780] BUG: KASAN: slab-use-after-free in rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.359659] Write of size 4 at addr ffff88813c1c3f00 by task kworker/27:1/370 [ 3032.366791] [ 3032.368293] CPU: 27 PID: 370 Comm: kworker/27:1 Not tainted 6.9.0-rc2.rdma.fix+ #1 [ 3032.375859] Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS 2.20.1 09/13/2023 [ 3032.383425] Workqueue: nvmet-wq nvmet_rdma_release_queue_work [nvmet_rdma] [ 3032.390304] Call Trace: [ 3032.392757] <TASK> [ 3032.394864] dump_stack_lvl+0x84/0xd0 [ 3032.398537] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.403561] print_report+0x19d/0x52e [ 3032.407228] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.412256] ? __virt_addr_valid+0x228/0x420 [ 3032.416537] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.421563] kasan_report+0xab/0x180 [ 3032.425142] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.430173] kasan_check_range+0x104/0x1b0 [ 3032.434275] rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.439128] ? cma_dev_put+0x1f/0x60 [rdma_cm] [ 3032.443591] cma_release_dev+0x1b2/0x270 [rdma_cm] [ 3032.448401] _destroy_id+0x35f/0xc80 [rdma_cm] [ 3032.452867] nvmet_rdma_free_queue+0x7a/0x380 [nvmet_rdma] [ 3032.458360] nvmet_rdma_release_queue_work+0x42/0x90 [nvmet_rdma] [ 3032.464460] process_one_work+0x85d/0x13e0 [ 3032.468573] ? worker_thread+0xcc/0x1130 [ 3032.472501] ? __pfx_process_one_work+0x10/0x10 [ 3032.477038] ? assign_work+0x16c/0x240 [ 3032.480797] worker_thread+0x6da/0x1130 [ 3032.484648] ? __pfx_worker_thread+0x10/0x10 [ 3032.488923] kthread+0x2ed/0x3c0 [ 3032.492155] ? _raw_spin_unlock_irq+0x28/0x60 [ 3032.496514] ? __pfx_kthread+0x10/0x10 [ 3032.500267] ret_from_fork+0x31/0x70 [ 3032.503854] ? __pfx_kthread+0x10/0x10 [ 3032.507607] ret_from_fork_asm+0x1a/0x30 [ 3032.511539] </TASK> [ 3032.513734] [ 3032.515234] Allocated by task 1997: [ 3032.518725] kasan_save_stack+0x30/0x50 [ 3032.522562] kasan_save_track+0x14/0x30 [ 3032.526402] __kasan_kmalloc+0x8f/0xa0 [ 3032.530155] add_modify_gid+0x18e/0xb80 [ib_core] [ 3032.534922] ib_cache_update.part.0+0x6fc/0x8e0 [ib_core] [ 3032.540380] ib_cache_setup_one+0x3ff/0x5f0 [ib_core] [ 3032.545495] ib_register_device+0x5ba/0xa20 [ib_core] [ 3032.550607] siw_newlink+0xb0d/0xe50 [siw] [ 3032.554724] nldev_newlink+0x301/0x520 [ib_core] [ 3032.559404] rdma_nl_rcv_msg+0x2e7/0x600 [ib_core] [ 3032.564256] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [ 3032.570756] netlink_unicast+0x437/0x6e0 [ 3032.574679] netlink_sendmsg+0x775/0xc10 [ 3032.578607] __sys_sendto+0x3e5/0x490 [ 3032.582273] __x64_sys_sendto+0xe0/0x1c0 [ 3032.586198] do_syscall_64+0x9a/0x1a0 [ 3032.589862] entry_SYSCALL_64_after_hwframe+0x71/0x79 [ 3032.594917] [ 3032.596416] Freed by task 339: [ 3032.599475] kasan_save_stack+0x30/0x50 [ 3032.603312] kasan_save_track+0x14/0x30 [ 3032.607153] kasan_save_free_info+0x3b/0x60 [ 3032.611338] poison_slab_object+0x103/0x190 [ 3032.615523] __kasan_slab_free+0x14/0x30 [ 3032.619450] kfree+0x120/0x3a0 [ 3032.622508] free_gid_work+0xd4/0x120 [ib_core] [ 3032.627100] process_one_work+0x85d/0x13e0 [ 3032.631200] worker_thread+0x6da/0x1130 [ 3032.635038] kthread+0x2ed/0x3c0 [ 3032.638271] ret_from_fork+0x31/0x70 [ 3032.641849] ret_from_fork_asm+0x1a/0x30 [ 3032.645777] [ 3032.647277] Last potentially related work creation: [ 3032.652153] kasan_save_stack+0x30/0x50 [ 3032.655994] __kasan_record_aux_stack+0x8e/0xa0 [ 3032.660525] insert_work+0x36/0x310 [ 3032.664019] __queue_work+0x6a4/0xcb0 [ 3032.667685] queue_work_on+0x99/0xb0 [ 3032.671263] cma_release_dev+0x1b2/0x270 [rdma_cm] [ 3032.676072] _destroy_id+0x35f/0xc80 [rdma_cm] [ 3032.680537] nvme_rdma_free_queue+0x4a/0x70 [nvme_rdma] [ 3032.685768] nvme_do_delete_ctrl+0x146/0x160 [nvme_core] [ 3032.691108] nvme_delete_ctrl_sync.cold+0x8/0xd [nvme_core] [ 3032.696707] nvme_sysfs_delete+0x96/0xc0 [nvme_core] [ 3032.701696] kernfs_fop_write_iter+0x3a5/0x5b0 [ 3032.706142] vfs_write+0x62e/0x1010 [ 3032.709636] ksys_write+0xfb/0x1d0 [ 3032.713041] do_syscall_64+0x9a/0x1a0 [ 3032.716707] entry_SYSCALL_64_after_hwframe+0x71/0x79 [ 3032.721760] [ 3032.723259] The buggy address belongs to the object at ffff88813c1c3f00 [ 3032.723259] which belongs to the cache kmalloc-rnd-07-192 of size 192 [ 3032.736370] The buggy address is located 0 bytes inside of [ 3032.736370] freed 192-byte region [ffff88813c1c3f00, ffff88813c1c3fc0) [ 3032.748441] [ 3032.749942] The buggy address belongs to the physical page: [ 3032.755513] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x13c1c2 [ 3032.763511] head: order:1 entire_mapcount:0 nr_pages_mapped:0 pincount:0 [ 3032.770212] flags: 0x17ffffe0000840(slab|head|node=0|zone=2|lastcpupid=0x3fffff) [ 3032.777604] page_type: 0xffffffff() [ 3032.781097] raw: 0017ffffe0000840 ffff888100053c00 dead000000000122 0000000000000000 [ 3032.788837] raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 [ 3032.796574] head: 0017ffffe0000840 ffff888100053c00 dead000000000122 0000000000000000 [ 3032.804400] head: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 [ 3032.812225] head: 0017ffffe0000001 ffffea0004f07081 ffffea0004f070c8 00000000ffffffff [ 3032.820050] head: 0000000200000000 0000000000000000 00000000ffffffff 0000000000000000 [ 3032.827874] page dumped because: kasan: bad access detected [ 3032.833447] [ 3032.834944] Memory state around the buggy address: [ 3032.839737] ffff88813c1c3e00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 3032.846958] ffff88813c1c3e80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 3032.854177] >ffff88813c1c3f00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 3032.861393] ^ [ 3032.864628] ffff88813c1c3f80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc [ 3032.871845] ffff88813c1c4000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3032.879065] ================================================================== [ 3032.886311] Disabling lock debugging due to kernel taint [ 3032.891630] ------------[ cut here ]------------ [ 3032.896255] refcount_t: underflow; use-after-free. [ 3032.901104] WARNING: CPU: 27 PID: 370 at lib/refcount.c:28 refcount_warn_saturate+0xf2/0x150 [ 3032.909552] Modules linked in: siw ib_uverbs nvmet_rdma nvmet nvme_keyring nvme_rdma nvme_fabrics nvme_core nvme_auth rdma_cm iw_cm ib_cm ib_core intel_rapl_msr intel_rapl_coma [ 3032.992273] CPU: 27 PID: 370 Comm: kworker/27:1 Tainted: G B 6.9.0-rc2.rdma.fix+ #1 [ 3033.001324] Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS 2.20.1 09/13/2023 [ 3033.008899] Workqueue: nvmet-wq nvmet_rdma_release_queue_work [nvmet_rdma] [ 3033.015787] RIP: 0010:refcount_warn_saturate+0xf2/0x150 [ 3033.021022] Code: 2f 1b 66 04 01 e8 6e f7 9c fe 0f 0b eb 91 80 3d 1e 1b 66 04 00 75 88 48 c7 c7 80 16 c5 91 c6 05 0e 1b 66 04 01 e8 4e f7 9c fe <0f> 0b e9 6e ff ff ff 80 3d fe7 [ 3033.039775] RSP: 0018:ffffc9000e627c10 EFLAGS: 00010286 [ 3033.045019] RAX: 0000000000000000 RBX: ffff88813c1c3f00 RCX: 0000000000000000 [ 3033.052158] RDX: 0000000000000000 RSI: ffffffff91c5c760 RDI: 0000000000000001 [ 3033.059303] RBP: 0000000000000003 R08: 0000000000000001 R09: fffff52001cc4f36 [ 3033.066449] R10: ffffc9000e6279b7 R11: 0000000000000001 R12: ffff88a88f394248 [ 3033.073588] R13: ffff888135750240 R14: ffff88825385fb80 R15: dead000000000100 [ 3033.080730] FS: 0000000000000000(0000) GS:ffff88c7ad200000(0000) knlGS:0000000000000000 [ 3033.088824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3033.094579] CR2: 00007fdfcb573c58 CR3: 0000000ea6e98004 CR4: 00000000007706f0 [ 3033.101720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3033.108859] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3033.116001] PKRU: 55555554 [ 3033.118723] Call Trace: [ 3033.121182] <TASK> [ 3033.123300] ? __warn+0xcc/0x170 [ 3033.126548] ? refcount_warn_saturate+0xf2/0x150 [ 3033.131176] ? report_bug+0x1fc/0x3d0 [ 3033.134860] ? handle_bug+0x3c/0x80 [ 3033.138368] ? exc_invalid_op+0x17/0x40 [ 3033.142217] ? asm_exc_invalid_op+0x1a/0x20 [ 3033.146428] ? refcount_warn_saturate+0xf2/0x150 [ 3033.151056] cma_release_dev+0x1b2/0x270 [rdma_cm] [ 3033.155874] _destroy_id+0x35f/0xc80 [rdma_cm] [ 3033.160348] nvmet_rdma_free_queue+0x7a/0x380 [nvmet_rdma] [ 3033.165851] nvmet_rdma_release_queue_work+0x42/0x90 [nvmet_rdma] [ 3033.171958] process_one_work+0x85d/0x13e0 [ 3033.176077] ? worker_thread+0xcc/0x1130 [ 3033.180017] ? __pfx_process_one_work+0x10/0x10 [ 3033.184561] ? assign_work+0x16c/0x240 [ 3033.188332] worker_thread+0x6da/0x1130 [ 3033.192187] ? __pfx_worker_thread+0x10/0x10 [ 3033.196475] kthread+0x2ed/0x3c0 [ 3033.199724] ? _raw_spin_unlock_irq+0x28/0x60 [ 3033.204099] ? __pfx_kthread+0x10/0x10 [ 3033.207860] ret_from_fork+0x31/0x70 [ 3033.211447] ? __pfx_kthread+0x10/0x10 [ 3033.215209] ret_from_fork_asm+0x1a/0x30 [ 3033.219152] </TASK> [ 3033.221352] irq event stamp: 255979 [ 3033.224855] hardirqs last enabled at (255979): [<ffffffff9160160a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3033.234863] hardirqs last disabled at (255978): [<ffffffff91433f0a>] __do_softirq+0x75a/0x967 [ 3033.243391] softirqs last enabled at (255766): [<ffffffff8e269f56>] __irq_exit_rcu+0xc6/0x1d0 [ 3033.252015] softirqs last disabled at (255757): [<ffffffff8e269f56>] __irq_exit_rcu+0xc6/0x1d0 [ 3033.260637] ---[ end trace 0000000000000000 ]---Quoted 203 lines
drivers/infiniband/core/cma.c +5
-1@@ -715,9 +715,13 @@ cma_validate_port(struct ib_device *device, u32 port,rcu_read_lock (); ndev =rcu_dereference (sgid_attr->ndev);if (!net_eq (dev_net (ndev), dev_addr->net) || - ndev->ifindex != bound_if_index) + ndev->ifindex != bound_if_index) { +rdma_put_gid_attr (sgid_attr); sgid_attr =ERR_PTR (-ENODEV ); + }rcu_read_unlock (); +if (!IS_ERR (sgid_attr)) +rdma_put_gid_attr (sgid_attr); goto out; }Zhu Yanjun
Quoted 183 lines · Yi Zhang
Quoted 180 lines · Guoqing Jiang
Quoted 157 lines · Yi Zhang
Quoted 154 lines · Guoqing Jiang
Quoted 153 lines · Yi Zhang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Guoqing Jiang
Quoted 7 lines
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 14 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):Quoted 8 lines
comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.Quoted 5 lines
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.Yes, pls bisect them.
Sure, will do that after I back from holiday next week.
Quoted 11 lines
Quoted 1 line
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3dRDMA/siw: Callllist_reverse_orderinsiw_run_sqbee024d20451RDMA/siw: Correct wrong debug messageb056327bee09RDMA/siw: Balance the reference ofcep->krefin the error path91f36237b4b9RDMA/siw: Fix tx thread initialization.bad5b6e34ffbRDMA/siw: Fabricate a GID on tun and loopback devices9191df002926RDMA/siw: usevmalloc_arrayand vcallocThanks,
Guoqing--
Best Regards,
Yi Zhang
Signature · 2 lines
Best Regards, Yanjun.Zhu
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAHj4cs9XsHiKeVGNRjTCcg87ghWrDm972VMq+5w=8jvetjFG0w@mail.gmail.com
Quoted 15 lines · Yi Zhang
Quoted 14 lines · Zhu Yanjun
I can reproduce the new problem that you mentioned in the mail. And in the latest diff, I fixed the new problem.
About the kmemleak problem, when
sgid_attris set toERR_PTR(-ENODEV),rdma_put_gid_attrshould be called.But
rdma_put_gid_attris not called in commitf8ef1be816bf9a("RDMA/cma:
Avoid GID lookups on iWARP devices").I think this is the root cause that results in kmemleak error. I made a lot of tests in my local host. This will not introduce new problem that you mentioned.
Quoted 7 lines
Quoted 4 lines
Because I can not reproduce the kmemleak error locally, I send the followings to you.
If any problem, please feel free to let me know.
The issue fixed now, feel free to add
Tested-by Yi Zhang <[email protected]>
Thanks a lot. I will send out the official patch very soon.
Your helps and efforts are much appreciated.
Zhu Yanjun
Quoted 442 lines
Quoted 441 lines
drivers/infiniband/core/cma.c +4
-1@@ -715,8 +715,11 @@ cma_validate_port(struct ib_device *device, u32 port,rcu_read_lock (); ndev =rcu_dereference (sgid_attr->ndev);if (!net_eq (dev_net (ndev), dev_addr->net) || - ndev->ifindex != bound_if_index) + ndev->ifindex != bound_if_index) { +rdma_put_gid_attr (sgid_attr); sgid_attr =ERR_PTR (-ENODEV ); + } +rcu_read_unlock (); goto out; }Zhu Yanjun
Quoted 207 lines · Yi Zhang
Quoted 4 lines · Zhu Yanjun
Quoted 3 lines · Yi Zhang
So bisect shows it was introduced with below commit, please help check and fix it, thanks.
Quoted 8 lines
Quoted 5 lines
commit
f8ef1be816bf9a0c406c696368c2264a9597a994Author: Chuck Lever <[email protected]> Date: Mon Jul 17 11:12:32 2023 -0400RDMA/cma: Avoid GID lookups on iWARP devices
Not sure if the following can fix this problem or not. Please let me know the test result.
Thanks a lot.Hi Yanjun
Seems the change introduced new issue, here is the log:
[ 3017.171697] run blktests nvme/003 at 2024-05-09 21:35:41 [ 3032.344539] ================================================================== [ 3032.351780] BUG: KASAN: slab-use-after-free in rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.359659] Write of size 4 at addr ffff88813c1c3f00 by task kworker/27:1/370 [ 3032.366791] [ 3032.368293] CPU: 27 PID: 370 Comm: kworker/27:1 Not tainted 6.9.0-rc2.rdma.fix+ #1 [ 3032.375859] Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS 2.20.1 09/13/2023 [ 3032.383425] Workqueue: nvmet-wq nvmet_rdma_release_queue_work [nvmet_rdma] [ 3032.390304] Call Trace: [ 3032.392757] <TASK> [ 3032.394864] dump_stack_lvl+0x84/0xd0 [ 3032.398537] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.403561] print_report+0x19d/0x52e [ 3032.407228] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.412256] ? __virt_addr_valid+0x228/0x420 [ 3032.416537] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.421563] kasan_report+0xab/0x180 [ 3032.425142] ? rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.430173] kasan_check_range+0x104/0x1b0 [ 3032.434275] rdma_put_gid_attr+0x23/0xa0 [ib_core] [ 3032.439128] ? cma_dev_put+0x1f/0x60 [rdma_cm] [ 3032.443591] cma_release_dev+0x1b2/0x270 [rdma_cm] [ 3032.448401] _destroy_id+0x35f/0xc80 [rdma_cm] [ 3032.452867] nvmet_rdma_free_queue+0x7a/0x380 [nvmet_rdma] [ 3032.458360] nvmet_rdma_release_queue_work+0x42/0x90 [nvmet_rdma] [ 3032.464460] process_one_work+0x85d/0x13e0 [ 3032.468573] ? worker_thread+0xcc/0x1130 [ 3032.472501] ? __pfx_process_one_work+0x10/0x10 [ 3032.477038] ? assign_work+0x16c/0x240 [ 3032.480797] worker_thread+0x6da/0x1130 [ 3032.484648] ? __pfx_worker_thread+0x10/0x10 [ 3032.488923] kthread+0x2ed/0x3c0 [ 3032.492155] ? _raw_spin_unlock_irq+0x28/0x60 [ 3032.496514] ? __pfx_kthread+0x10/0x10 [ 3032.500267] ret_from_fork+0x31/0x70 [ 3032.503854] ? __pfx_kthread+0x10/0x10 [ 3032.507607] ret_from_fork_asm+0x1a/0x30 [ 3032.511539] </TASK> [ 3032.513734] [ 3032.515234] Allocated by task 1997: [ 3032.518725] kasan_save_stack+0x30/0x50 [ 3032.522562] kasan_save_track+0x14/0x30 [ 3032.526402] __kasan_kmalloc+0x8f/0xa0 [ 3032.530155] add_modify_gid+0x18e/0xb80 [ib_core] [ 3032.534922] ib_cache_update.part.0+0x6fc/0x8e0 [ib_core] [ 3032.540380] ib_cache_setup_one+0x3ff/0x5f0 [ib_core] [ 3032.545495] ib_register_device+0x5ba/0xa20 [ib_core] [ 3032.550607] siw_newlink+0xb0d/0xe50 [siw] [ 3032.554724] nldev_newlink+0x301/0x520 [ib_core] [ 3032.559404] rdma_nl_rcv_msg+0x2e7/0x600 [ib_core] [ 3032.564256] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [ 3032.570756] netlink_unicast+0x437/0x6e0 [ 3032.574679] netlink_sendmsg+0x775/0xc10 [ 3032.578607] __sys_sendto+0x3e5/0x490 [ 3032.582273] __x64_sys_sendto+0xe0/0x1c0 [ 3032.586198] do_syscall_64+0x9a/0x1a0 [ 3032.589862] entry_SYSCALL_64_after_hwframe+0x71/0x79 [ 3032.594917] [ 3032.596416] Freed by task 339: [ 3032.599475] kasan_save_stack+0x30/0x50 [ 3032.603312] kasan_save_track+0x14/0x30 [ 3032.607153] kasan_save_free_info+0x3b/0x60 [ 3032.611338] poison_slab_object+0x103/0x190 [ 3032.615523] __kasan_slab_free+0x14/0x30 [ 3032.619450] kfree+0x120/0x3a0 [ 3032.622508] free_gid_work+0xd4/0x120 [ib_core] [ 3032.627100] process_one_work+0x85d/0x13e0 [ 3032.631200] worker_thread+0x6da/0x1130 [ 3032.635038] kthread+0x2ed/0x3c0 [ 3032.638271] ret_from_fork+0x31/0x70 [ 3032.641849] ret_from_fork_asm+0x1a/0x30 [ 3032.645777] [ 3032.647277] Last potentially related work creation: [ 3032.652153] kasan_save_stack+0x30/0x50 [ 3032.655994] __kasan_record_aux_stack+0x8e/0xa0 [ 3032.660525] insert_work+0x36/0x310 [ 3032.664019] __queue_work+0x6a4/0xcb0 [ 3032.667685] queue_work_on+0x99/0xb0 [ 3032.671263] cma_release_dev+0x1b2/0x270 [rdma_cm] [ 3032.676072] _destroy_id+0x35f/0xc80 [rdma_cm] [ 3032.680537] nvme_rdma_free_queue+0x4a/0x70 [nvme_rdma] [ 3032.685768] nvme_do_delete_ctrl+0x146/0x160 [nvme_core] [ 3032.691108] nvme_delete_ctrl_sync.cold+0x8/0xd [nvme_core] [ 3032.696707] nvme_sysfs_delete+0x96/0xc0 [nvme_core] [ 3032.701696] kernfs_fop_write_iter+0x3a5/0x5b0 [ 3032.706142] vfs_write+0x62e/0x1010 [ 3032.709636] ksys_write+0xfb/0x1d0 [ 3032.713041] do_syscall_64+0x9a/0x1a0 [ 3032.716707] entry_SYSCALL_64_after_hwframe+0x71/0x79 [ 3032.721760] [ 3032.723259] The buggy address belongs to the object at ffff88813c1c3f00 [ 3032.723259] which belongs to the cache kmalloc-rnd-07-192 of size 192 [ 3032.736370] The buggy address is located 0 bytes inside of [ 3032.736370] freed 192-byte region [ffff88813c1c3f00, ffff88813c1c3fc0) [ 3032.748441] [ 3032.749942] The buggy address belongs to the physical page: [ 3032.755513] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x13c1c2 [ 3032.763511] head: order:1 entire_mapcount:0 nr_pages_mapped:0 pincount:0 [ 3032.770212] flags: 0x17ffffe0000840(slab|head|node=0|zone=2|lastcpupid=0x3fffff) [ 3032.777604] page_type: 0xffffffff() [ 3032.781097] raw: 0017ffffe0000840 ffff888100053c00 dead000000000122 0000000000000000 [ 3032.788837] raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 [ 3032.796574] head: 0017ffffe0000840 ffff888100053c00 dead000000000122 0000000000000000 [ 3032.804400] head: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 [ 3032.812225] head: 0017ffffe0000001 ffffea0004f07081 ffffea0004f070c8 00000000ffffffff [ 3032.820050] head: 0000000200000000 0000000000000000 00000000ffffffff 0000000000000000 [ 3032.827874] page dumped because: kasan: bad access detected [ 3032.833447] [ 3032.834944] Memory state around the buggy address: [ 3032.839737] ffff88813c1c3e00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 3032.846958] ffff88813c1c3e80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 3032.854177] >ffff88813c1c3f00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 3032.861393] ^ [ 3032.864628] ffff88813c1c3f80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc [ 3032.871845] ffff88813c1c4000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3032.879065] ================================================================== [ 3032.886311] Disabling lock debugging due to kernel taint [ 3032.891630] ------------[ cut here ]------------ [ 3032.896255] refcount_t: underflow; use-after-free. [ 3032.901104] WARNING: CPU: 27 PID: 370 at lib/refcount.c:28 refcount_warn_saturate+0xf2/0x150 [ 3032.909552] Modules linked in: siw ib_uverbs nvmet_rdma nvmet nvme_keyring nvme_rdma nvme_fabrics nvme_core nvme_auth rdma_cm iw_cm ib_cm ib_core intel_rapl_msr intel_rapl_coma [ 3032.992273] CPU: 27 PID: 370 Comm: kworker/27:1 Tainted: G B 6.9.0-rc2.rdma.fix+ #1 [ 3033.001324] Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS 2.20.1 09/13/2023 [ 3033.008899] Workqueue: nvmet-wq nvmet_rdma_release_queue_work [nvmet_rdma] [ 3033.015787] RIP: 0010:refcount_warn_saturate+0xf2/0x150 [ 3033.021022] Code: 2f 1b 66 04 01 e8 6e f7 9c fe 0f 0b eb 91 80 3d 1e 1b 66 04 00 75 88 48 c7 c7 80 16 c5 91 c6 05 0e 1b 66 04 01 e8 4e f7 9c fe <0f> 0b e9 6e ff ff ff 80 3d fe7 [ 3033.039775] RSP: 0018:ffffc9000e627c10 EFLAGS: 00010286 [ 3033.045019] RAX: 0000000000000000 RBX: ffff88813c1c3f00 RCX: 0000000000000000 [ 3033.052158] RDX: 0000000000000000 RSI: ffffffff91c5c760 RDI: 0000000000000001 [ 3033.059303] RBP: 0000000000000003 R08: 0000000000000001 R09: fffff52001cc4f36 [ 3033.066449] R10: ffffc9000e6279b7 R11: 0000000000000001 R12: ffff88a88f394248 [ 3033.073588] R13: ffff888135750240 R14: ffff88825385fb80 R15: dead000000000100 [ 3033.080730] FS: 0000000000000000(0000) GS:ffff88c7ad200000(0000) knlGS:0000000000000000 [ 3033.088824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3033.094579] CR2: 00007fdfcb573c58 CR3: 0000000ea6e98004 CR4: 00000000007706f0 [ 3033.101720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3033.108859] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3033.116001] PKRU: 55555554 [ 3033.118723] Call Trace: [ 3033.121182] <TASK> [ 3033.123300] ? __warn+0xcc/0x170 [ 3033.126548] ? refcount_warn_saturate+0xf2/0x150 [ 3033.131176] ? report_bug+0x1fc/0x3d0 [ 3033.134860] ? handle_bug+0x3c/0x80 [ 3033.138368] ? exc_invalid_op+0x17/0x40 [ 3033.142217] ? asm_exc_invalid_op+0x1a/0x20 [ 3033.146428] ? refcount_warn_saturate+0xf2/0x150 [ 3033.151056] cma_release_dev+0x1b2/0x270 [rdma_cm] [ 3033.155874] _destroy_id+0x35f/0xc80 [rdma_cm] [ 3033.160348] nvmet_rdma_free_queue+0x7a/0x380 [nvmet_rdma] [ 3033.165851] nvmet_rdma_release_queue_work+0x42/0x90 [nvmet_rdma] [ 3033.171958] process_one_work+0x85d/0x13e0 [ 3033.176077] ? worker_thread+0xcc/0x1130 [ 3033.180017] ? __pfx_process_one_work+0x10/0x10 [ 3033.184561] ? assign_work+0x16c/0x240 [ 3033.188332] worker_thread+0x6da/0x1130 [ 3033.192187] ? __pfx_worker_thread+0x10/0x10 [ 3033.196475] kthread+0x2ed/0x3c0 [ 3033.199724] ? _raw_spin_unlock_irq+0x28/0x60 [ 3033.204099] ? __pfx_kthread+0x10/0x10 [ 3033.207860] ret_from_fork+0x31/0x70 [ 3033.211447] ? __pfx_kthread+0x10/0x10 [ 3033.215209] ret_from_fork_asm+0x1a/0x30 [ 3033.219152] </TASK> [ 3033.221352] irq event stamp: 255979 [ 3033.224855] hardirqs last enabled at (255979):Quoted 208 lines
[<
ffffffff9160160a>]asm_sysvec_apic_timer_interrupt+0x1a/0x20[ 3033.234863] hardirqs last disabled at (255978): [<ffffffff91433f0a>] __do_softirq+0x75a/0x967 [ 3033.243391] softirqs last enabled at (255766): [<ffffffff8e269f56>] __irq_exit_rcu+0xc6/0x1d0 [ 3033.252015] softirqs last disabled at (255757): [<ffffffff8e269f56>] __irq_exit_rcu+0xc6/0x1d0 [ 3033.260637] ---[ end trace 0000000000000000 ]---Quoted 199 lines
drivers/infiniband/core/cma.c +5
-1@@ -715,9 +715,13 @@ cma_validate_port(struct ib_device *device, u32 port,rcu_read_lock (); ndev =rcu_dereference (sgid_attr->ndev);if (!net_eq (dev_net (ndev), dev_addr->net) || - ndev->ifindex != bound_if_index) + ndev->ifindex != bound_if_index) { +rdma_put_gid_attr (sgid_attr); sgid_attr =ERR_PTR (-ENODEV ); + }rcu_read_unlock (); +if (!IS_ERR (sgid_attr)) +rdma_put_gid_attr (sgid_attr); goto out; }Zhu Yanjun
Quoted 179 lines · Yi Zhang
Quoted 177 lines · Guoqing Jiang
Quoted 155 lines · Yi Zhang
Quoted 153 lines · Guoqing Jiang
Quoted 152 lines · Yi Zhang
Quoted 9 lines · Yi Zhang
Quoted 8 lines · Guoqing Jiang
Quoted 7 lines
Hi,
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the
latest linux-rdma/for-next, please help check it and let me know if
you need any info/testing for it, thanks.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Could you share which test case caused the issue? I can't reproduce it with 6.9-rc3+ kernel (commit
586b5dfb51b) with the below.It can be reproduced by [1], you can find more info from the symbol info[2], I also attached the config file, maybe you can this config file
Just attached the config file
I tried with the config, but still unlucky.
# nvme_trtype=rdma ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns)
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]runtime 52.763s ... 392.027s device: nvme0Quoted 119 lines
Quoted 100 lines
[1] nvme_trtype=rdma ./check nvme/012
[2]
unreferenced object0xffff8883a87e8800(size 192):comm "rdma", pid 2355, jiffies 4294836069 hex dump (first 32 bytes): 32 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 2............... 10 88 7e a8 83 88 ff ff 10 88 7e a8 83 88 ff ff ..~.......~..... backtrace (crc 4db191c4): [<ffffffff8cd251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc207eff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2080206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2081468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2082e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc207749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc24ac389>] 0xffffffffc24ac389 [<ffffffffc20c6cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2083fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc208448c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff8e30e715>] netlink_unicast+0x445/0x710 [<ffffffff8e30f151>] netlink_sendmsg+0x761/0xc40 [<ffffffff8e09da89>] __sys_sendto+0x3a9/0x420 [<ffffffff8e09dbec>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff8e9afad3>] do_syscall_64+0x93/0x180 [<ffffffff8ea00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79(gdb) l *(alloc_gid_entry+0x47)
0x2eff7is inalloc_gid_entry(./include/linux/slab.h:628). 623
624 if (size >KMALLOC_MAX_CACHE_SIZE)
625 returnkmalloc_large(size, flags);
626
627 index =kmalloc_index(size);
628 return kmalloc_trace(
629 kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], 630 flags, size);
631 }
632 return__kmalloc(size, flags);(gdb) l *(add_modify_gid+0x166)
0x30206is inadd_modify_gid(drivers/infiniband/core/cache.c:447). 442 * empty table entries instead of storing them.
443 */
444 if (rdma_is_zero_gid(&attr->gid))
445 return 0;
446
447 entry =alloc_gid_entry(attr);
448 if (!entry)
449 return-ENOMEM;
450
451 if (rdma_protocol_roce(attr->device, attr->port_num)) {Quoted 47 lines
use_siw=1 nvme_trtype=rdma ./check nvme/
Quoted 22 lines
# dmesg | grep kmemleak
[ 67.130652] kmemleak: Kernel memory leak detector initialized (mem
pool available: 36041)
[ 67.130728] kmemleak: Automatic memory scanning thread started
[ 1051.771867] kmemleak: 2 new suspected memory leaks (see
/sys/kernel/debug/kmemleak)
[ 1832.796189] kmemleak: 8 new suspected memory leaks (see
/sys/kernel/debug/kmemleak)
[ 2578.189075] kmemleak: 17 new suspected memory leaks (see
/sys/kernel/debug/kmemleak)
[ 3330.710984] kmemleak: 4 new suspected memory leaks (see
/sys/kernel/debug/kmemleak)unreferenced object 0xffff88855da53400 (size 192):
comm "rdma", pid 10630, jiffies 4296575922
hex dump (first 32 bytes):
37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7...............
10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.]....
backtrace (crc 47f66721):
[<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0
[<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core]
[<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core]I guess
add_modify_gidis called fromconfig_non_roce_gid_cache, not sure why we don't check the return value of it here.Looks
put_gid_entryis called in caseadd_modify_gidreturns failure, it would
triggerschedule_free_gid->queue_work(ib_wq, &entry->del_work), thenfree_gid_work->free_gid_entry_lockedwould free storage asynchronously byput_gid_ndevand also entry.Quoted 14 lines
[<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core]
[<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core]
[<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core]
[<ffffffffc2a3d389>] 0xffffffffc2a3d389
[<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core]
[<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core]
[<ffffffffc264648c>]
rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core]
[<ffffffff9270e7b5>] netlink_unicast+0x445/0x710
[<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40
[<ffffffff9249db29>] __sys_sendto+0x3a9/0x420
[<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0
[<ffffffff92db0ad3>] do_syscall_64+0x93/0x180
[<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79Quoted 4 lines
Quoted 2 lines
After
ib_cache_setup_onefailed, maybeib_cache_cleanup_oneis needed which flushib_wqto ensure storage is freed. Could you try with the change?Will try it later.
The kmemleak still can be reproduced with this change:
unreferenced object
0xffff8881f89fde00(size 192):comm "rdma", pid 8708, jiffies 4295703453 hex dump (first 32 bytes): 02 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 ................ 10 de 9f f8 81 88 ff ff 10 de 9f f8 81 88 ff ff ................ backtrace (crc 888c494b): [<ffffffffa7d251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc1efeff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc1f00206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc1f01468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc1f02e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc1ef749e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc22ee389>]Quoted 1 line
siw_qp_state_to_ib_qp_state+0x28a9/0xfffffffffffd1520[siw]Is it possible to run the test with rxe instead of siw? In case it is only happened
with siw, I'd suggest to revert0b988c1bee28to check if it causes the issue.
But I don't understand whysiw_qp_state_to_ib_qp_statewas appeared in the middle of above trace.Quoted 5 lines
Quoted 4 lines
Hi Guoqing
This issue only can be reproduced with siw, I did more testing today and it cannot be reproduced with 6.5, seems it was introduced from 6.6-rc1, and I saw there are some siw updates from 6.6-rc1.Yes, pls bisect them.
Sure, will do that after I back from holiday next week.
Quoted 11 lines
Quoted 1 line
git log --oneline v6.5..v6.6-rc1 drivers/infiniband/sw/siw/|cat
9dfccb6d0d3dRDMA/siw: Callllist_reverse_orderinsiw_run_sqbee024d20451RDMA/siw: Correct wrong debug messageb056327bee09RDMA/siw: Balance the reference ofcep->krefin the error path91f36237b4b9RDMA/siw: Fix tx thread initialization.bad5b6e34ffbRDMA/siw: Fabricate a GID on tun and loopback devices9191df002926RDMA/siw: usevmalloc_arrayand vcallocThanks,
Guoqing--
Best Regards,
Yi Zhang--
Best Regards,
Yanjun.Zhu
Signature · 2 lines
I only represent myself. Zhu Yanjun or Yanjun.Zhu
Quoted 4 lines · Yi Zhang
Hi
I found the below kmemleak issue during blktests nvme/rdma on the latest linux-rdma/for-next, please help check it and let me know if you need any info/testing for it, thanks.
I have once made blktests nvme/rdma. It seems that this problem does not exist at that time.
So can we use "git bisec" to find the commit that cause this problem if this problem can be reproduced?
Just my 2 cent suggestions.
Zhu Yanjun
Quoted 38 lines
# dmesg | grep kmemleak [ 67.130652] kmemleak: Kernel memory leak detector initialized (mem pool available: 36041) [ 67.130728] kmemleak: Automatic memory scanning thread started [ 1051.771867] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 1832.796189] kmemleak: 8 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 2578.189075] kmemleak: 17 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [ 3330.710984] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)unreferenced object
0xffff88855da53400(size 192):comm "rdma", pid 10630, jiffies 4296575922 hex dump (first 32 bytes): 37 00 00 00 00 00 00 00 c0 ff ff ff 1f 00 00 00 7............... 10 34 a5 5d 85 88 ff ff 10 34 a5 5d 85 88 ff ff .4.].....4.].... backtrace (crc 47f66721): [<ffffffff911251bd>] kmalloc_trace+0x30d/0x3b0 [<ffffffffc2640ff7>] alloc_gid_entry+0x47/0x380 [ib_core] [<ffffffffc2642206>] add_modify_gid+0x166/0x930 [ib_core] [<ffffffffc2643468>] ib_cache_update.part.0+0x6d8/0x910 [ib_core] [<ffffffffc2644e1a>] ib_cache_setup_one+0x24a/0x350 [ib_core] [<ffffffffc263949e>] ib_register_device+0x9e/0x3a0 [ib_core] [<ffffffffc2a3d389>] 0xffffffffc2a3d389 [<ffffffffc2688cd8>] nldev_newlink+0x2b8/0x520 [ib_core] [<ffffffffc2645fe3>] rdma_nl_rcv_msg+0x2c3/0x520 [ib_core] [<ffffffffc264648c>] rdma_nl_rcv_skb.constprop.0.isra.0+0x23c/0x3a0 [ib_core] [<ffffffff9270e7b5>] netlink_unicast+0x445/0x710 [<ffffffff9270f1f1>] netlink_sendmsg+0x761/0xc40 [<ffffffff9249db29>] __sys_sendto+0x3a9/0x420 [<ffffffff9249dc8c>] __x64_sys_sendto+0xdc/0x1b0 [<ffffffff92db0ad3>] do_syscall_64+0x93/0x180 [<ffffffff92e00126>] entry_SYSCALL_64_after_hwframe+0x71/0x79