rnbd-clt: fix refcount underflow in device unmap path
Subsystems (2)
PATCH 1/1 rnbd-clt: fix refcount underflow in device unmap path
During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:
[14747.574913] ------------[ cut here ]------------ [14747.574916] refcount_t: underflow; use-after-free. [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378 [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ... [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G O N 6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary) [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client] [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90 [14747.575037] Call Trace: [14747.575038] <TASK> [14747.575038] rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client] [14747.575044] process_one_work+0x211/0x600 [14747.575052] worker_thread+0x184/0x330 [14747.575055] ? __pfx_worker_thread+0x10/0x10 [14747.575058] kthread+0x10d/0x250 [14747.575062] ? __pfx_kthread+0x10/0x10 [14747.575066] ret_from_fork+0x319/0x390 [14747.575069] ? __pfx_kthread+0x10/0x10 [14747.575072] ret_from_fork_asm+0x1a/0x30 [14747.575083] </TASK> [14747.575096] ---[ end trace 0000000000000000 ]---
Befor this patch :-
The bug is a double kobject_put() on dev->kobj during device cleanup.
Kobject Lifecycle:
kobject_init_and_add() sets kobj.kref = 1 (initialization) kobject_put() sets kobj.kref = 0 (should be called once)
* Before this patch:
rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs]
kobject_put(&dev->kobj) PUT #1 (WRONG!)
kref: 1 to 0
rnbd_dev_release()
kfree(dev) [DEVICE FREED!]
rnbd_destroy_gen_disk() [use-after-free!]
rnbd_clt_put_dev()
refcount_dec_and_test(&dev->refcount)
kobject_put(&dev->kobj) PUT #2 (UNDERFLOW!)
kref: 0 to -1 [WARNING!]The first kobject_put() in rnbd_destroy_sysfs() prematurely frees the device via rnbd_dev_release(), then the second kobject_put() in rnbd_clt_put_dev() causes refcount underflow.
* After this patch :-
Remove kobject_put() from rnbd_destroy_sysfs(). This function should only remove sysfs visibility (kobject_del), not manage object lifetime.
Call Graph (FIXED):
rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs only]
[kref unchanged: 1]
rnbd_destroy_gen_disk() [device still valid]
rnbd_clt_put_dev()
refcount_dec_and_test(&dev->refcount)
kobject_put(&dev->kobj) ONLY PUT (CORRECT!)
kref: 1 to 0 [BALANCED]
rnbd_dev_release()
kfree(dev) [CLEAN DESTRUCTION]
This follows the kernel pattern where sysfs removal (kobject_del) is separate from object destruction (kobject_put).
--- drivers/block/rnbd/rnbd-clt.c | 1 - 1 file changed, 1 deletion(-)
drivers/block/rnbd/rnbd-clt.c +0 -1
@@ -1676,7 +1676,6 @@ static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev,/* To avoid deadlock firstly remove itself */ sysfs_remove_file_self (&dev->kobj, sysfs_self);kobject_del (&dev->kobj); -kobject_put (&dev->kobj); } }
Signature · 1 line
2.40.0
Quoted 75 lines · Chaitanya Kulkarni
During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:
[14747.574913] ------------[ cut here ]------------ [14747.574916] refcount_t: underflow; use-after-free. [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378 [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ... [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G O N 6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary) [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client] [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90 [14747.575037] Call Trace: [14747.575038] <TASK> [14747.575038] rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client] [14747.575044] process_one_work+0x211/0x600 [14747.575052] worker_thread+0x184/0x330 [14747.575055] ? __pfx_worker_thread+0x10/0x10 [14747.575058] kthread+0x10d/0x250 [14747.575062] ? __pfx_kthread+0x10/0x10 [14747.575066] ret_from_fork+0x319/0x390 [14747.575069] ? __pfx_kthread+0x10/0x10 [14747.575072] ret_from_fork_asm+0x1a/0x30 [14747.575083] </TASK> [14747.575096] ---[ end trace 0000000000000000 ]---Befor this patch :-
The bug is a double
kobject_put()ondev->kobjduring device cleanup.Kobject Lifecycle:
kobject_init_and_add() sets kobj.kref = 1 (initialization) kobject_put() sets kobj.kref = 0 (should be called once)* Before this patch:
rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj)[remove from sysfs]
kobject_put(&dev->kobj)PUT #1 (WRONG!)
kref: 1 to 0
rnbd_dev_release()
kfree(dev) [DEVICE FREED!]
rnbd_destroy_gen_disk()[use-after-free!]
rnbd_clt_put_dev()refcount_dec_and_test(&dev->refcount) kobject_put(&dev->kobj) PUT #2 (UNDERFLOW!) kref: 0 to -1 [WARNING!]The first
kobject_put()inrnbd_destroy_sysfs()prematurely frees the device viarnbd_dev_release(), then the secondkobject_put()inrnbd_clt_put_dev()causes refcount underflow.* After this patch :-
Remove
kobject_put()fromrnbd_destroy_sysfs(). This function should only remove sysfs visibility (kobject_del), not manage object lifetime.Call Graph (FIXED):
rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj)[remove from sysfs only]
[kref unchanged: 1]
rnbd_destroy_gen_disk()[device still valid]
rnbd_clt_put_dev()
refcount_dec_and_test(&dev->refcount)
kobject_put(&dev->kobj)ONLY PUT (CORRECT!)
kref: 1 to 0 [BALANCED]
rnbd_dev_release()
kfree(dev) [CLEAN DESTRUCTION]
Quoted 5 lines
This follows the kernel pattern where sysfs removal (
kobject_del) is separate from object destruction (kobject_put).Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")Signed-off-by Chaitanya Kulkarni <[email protected]>
lgtm, thx for the fix.
Quoted 19 lines
--- drivers/block/rnbd/rnbd-clt.c | 1 - 1 file changed, 1 deletion(-)drivers/block/rnbd/rnbd-clt.c +0
-2@@ -1676,7 +1676,6 @@ static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev,/* To avoid deadlock firstly remove itself */ sysfs_remove_file_self (&dev->kobj, sysfs_self);kobject_del (&dev->kobj); -kobject_put (&dev->kobj); } } --2.40.0
Quoted 7 lines · Jinpu Wang
Quoted 5 lines
This follows the kernel pattern where sysfs removal (
kobject_del) is separate from object destruction (kobject_put).Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")Signed-off-by Chaitanya Kulkarni<[email protected]>lgtm, thx for the fix.
Acked-by Jack Wang<[email protected]>
If this is correct we will needs reviewed-by tag to apply this patch.
-ck
Quoted 11 lines · Chaitanya Kulkarni
Quoted 7 lines · Jinpu Wang
Quoted 5 lines
This follows the kernel pattern where sysfs removal (
kobject_del) is separate from object destruction (kobject_put).Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")Signed-off-by Chaitanya Kulkarni<[email protected]>lgtm, thx for the fix.
Acked-by Jack Wang<[email protected]>If this is correct we will needs reviewed-by tag to apply this patch.
My understanding is as module maintainer of rnbd, acked-by is enough for Jens to pickup/apply the fix.
Anyway.
Quoted 4 lines
-ck
Jens Axboe
maintainerQuoted 10 lines · Chaitanya Kulkarni
Quoted 7 lines · Jinpu Wang
Quoted 5 lines
This follows the kernel pattern where sysfs removal (
kobject_del) is separate from object destruction (kobject_put).Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")Signed-off-by Chaitanya Kulkarni<[email protected]>lgtm, thx for the fix.
Acked-by Jack Wang<[email protected]>If this is correct we will needs reviewed-by tag to apply this patch.
Eh no, that is not correct. What I care about is that it has been seen and acknowledged. Whether it's a acked-by or reviewed-by doesn't matter one single bit.
Signature · 1 line
Jens Axboe
Quoted 24 lines · Chaitanya Kulkarni
During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:
[14747.574913] ------------[ cut here ]------------ [14747.574916] refcount_t: underflow; use-after-free. [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378 [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ... [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G O N 6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary) [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client] [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90 [14747.575037] Call Trace: [14747.575038] <TASK> [14747.575038] rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client] [14747.575044] process_one_work+0x211/0x600 [14747.575052] worker_thread+0x184/0x330 [14747.575055] ? __pfx_worker_thread+0x10/0x10 [14747.575058] kthread+0x10d/0x250 [14747.575062] ? __pfx_kthread+0x10/0x10 [14747.575066] ret_from_fork+0x319/0x390 [14747.575069] ? __pfx_kthread+0x10/0x10 [14747.575072] ret_from_fork_asm+0x1a/0x30 [14747.575083] </TASK> [14747.575096] ---[ end trace 0000000000000000 ]---
Quoted 1 line
[...]
Applied, thanks!
[1/1] rnbd-clt: fix refcount underflow in device unmap path
commit: ec19ed2b3e2af8ec5380400cdee9cb6560144506
Best regards,
Signature · 1 line
Jens Axboe
Quoted 74 lines · Chaitanya Kulkarni
During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:
[14747.574913] ------------[ cut here ]------------ [14747.574916] refcount_t: underflow; use-after-free. [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378 [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ... [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G O N 6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary) [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client] [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90 [14747.575037] Call Trace: [14747.575038] <TASK> [14747.575038] rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client] [14747.575044] process_one_work+0x211/0x600 [14747.575052] worker_thread+0x184/0x330 [14747.575055] ? __pfx_worker_thread+0x10/0x10 [14747.575058] kthread+0x10d/0x250 [14747.575062] ? __pfx_kthread+0x10/0x10 [14747.575066] ret_from_fork+0x319/0x390 [14747.575069] ? __pfx_kthread+0x10/0x10 [14747.575072] ret_from_fork_asm+0x1a/0x30 [14747.575083] </TASK> [14747.575096] ---[ end trace 0000000000000000 ]---Befor this patch :-
The bug is a double
kobject_put()ondev->kobjduring device cleanup.Kobject Lifecycle:
kobject_init_and_add() sets kobj.kref = 1 (initialization) kobject_put() sets kobj.kref = 0 (should be called once)* Before this patch:
rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj)[remove from sysfs]
kobject_put(&dev->kobj)PUT #1 (WRONG!)
kref: 1 to 0
rnbd_dev_release()
kfree(dev) [DEVICE FREED!]
rnbd_destroy_gen_disk()[use-after-free!]
rnbd_clt_put_dev()refcount_dec_and_test(&dev->refcount) kobject_put(&dev->kobj) PUT #2 (UNDERFLOW!) kref: 0 to -1 [WARNING!]The first
kobject_put()inrnbd_destroy_sysfs()prematurely frees the device viarnbd_dev_release(), then the secondkobject_put()inrnbd_clt_put_dev()causes refcount underflow.* After this patch :-
Remove
kobject_put()fromrnbd_destroy_sysfs(). This function should only remove sysfs visibility (kobject_del), not manage object lifetime.Call Graph (FIXED):
rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj)[remove from sysfs only]
[kref unchanged: 1]
rnbd_destroy_gen_disk()[device still valid]
rnbd_clt_put_dev()
refcount_dec_and_test(&dev->refcount)
kobject_put(&dev->kobj)ONLY PUT (CORRECT!)
kref: 1 to 0 [BALANCED]
rnbd_dev_release()
kfree(dev) [CLEAN DESTRUCTION]
Quoted 5 lines
This follows the kernel pattern where sysfs removal (
kobject_del) is separate from object destruction (kobject_put).Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")Signed-off-by Chaitanya Kulkarni <[email protected]>
Recently, blktests introduced the kmemleak feature. With this, I observed that rnbd/001 and rnbd/002 fail on v6.19-rc6 and v6.19-rc7 kernels with the kmemleak report as follows:
unreferenced object 0xffff88826c5f8720 (size 8):
comm "check", pid 145048, jiffies 4301507712
hex dump (first 8 bytes):
72 6e 62 64 00 15 b3 a6 rnbd....
backtrace (crc f6c244b4):
__kmalloc_node_track_caller_noprof+0x69d/0x900
kstrdup+0x42/0xc0
kobject_set_name_vargs+0x44/0x110
kobject_init_and_add+0xcf/0x150
rnbd_clt_map_device_store.cold+0x164/0x792 [rnbd_client]
kernfs_fop_write_iter+0x3d6/0x5e0
vfs_write+0x52c/0xf80
ksys_write+0xfb/0x200
do_syscall_64+0x95/0x540
entry_SYSCALL_64_after_hwframe+0x76/0x7eI bisected and found this commit ec19ed2b3e2a ("rnbd-clt: fix refcount underflow in device unmap path") is the trigger. When I revert it from v6.19-rc6 tag, the failure disappeared. I also found that,
- The commit message assumes that rnbd_clt_put_dev() calls kobject_put(), but such call does not exist in the kernel v6.19-rc6.
- The commit message has the Fixes tag for the commit 581cf833cac4 ("block:
rnbd: add .release to rnbd_dev_ktype"), but that is not in the kernel v6.19-rc6. Instead, the commit 581cf833cac4 is in block/for-7.0/block branch.
- The commit 581cf833cac4 added the kobject_put() call to rnbd_clt_put_dev() that the commit ec19ed2b3e2a assumes.
So I think the commit ec19ed2b3e2a was meant for block/for-7.0/block branch.
drivers/block/rnbd/rnbd-clt.c static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev, · line 1679
Quoted 13 lines
--- drivers/block/rnbd/rnbd-clt.c | 1 - 1 file changed, 1 deletion(-)drivers/block/rnbd/rnbd-clt.c +0
-1@@ -1676,7 +1676,6 @@ static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev,/* To avoid deadlock firstly remove itself */ sysfs_remove_file_self (&dev->kobj, sysfs_self);kobject_del (&dev->kobj); -kobject_put (&dev->kobj);
As to the current Linus master branch, the hank above was applied as the commit ec19ed2b3e2a, but the commit 581cf833cac4 is missing. So one more kobject_put() is missing. It caused the memory leak and the kmemleak feature detected it.
Haris and Jens,
I suggest to pick up 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype") to v6.19-rc8 to fix the kmemleak issue on the Linus master branch. I applied 581cf833cac4 on top of v6.18-rc7 kernel, and confirmed the kmemleak failure disappears.
Quoted 6 lines · Shinichiro Kawasaki
Haris and Jens,
I suggest to pick up
581cf833cac4("block: rnbd: add .release tornbd_dev_ktype") to v6.19-rc8 to fix the kmemleak issue on the Linus master branch. I applied581cf833cac4on top of v6.18-rc7 kernel, and confirmed the kmemleak failure disappears.
Thanks you Shinichiro for detailed analysis. Please let me know if there is anything I can do. -ck
Jens Axboe
maintainerQuoted 6 lines · Shinichiro Kawasaki
Haris and Jens,
I suggest to pick up
581cf833cac4("block: rnbd: add .release tornbd_dev_ktype") to v6.19-rc8 to fix the kmemleak issue on the Linus master branch. I applied581cf833cac4on top of v6.18-rc7 kernel, and confirmed the kmemleak failure disappears.
I reverted the fix from 6.19 instead, and picked up the offending commit in for-7.0/block.
Signature · 1 line
Jens Axboe