Skip to contentSkip to conversation

rnbd-clt: fix refcount underflow in device unmap path

PATCH Applied Chaitanya Kulkarni 5 participants 9 messages Jan 12 – Jan 28

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rnbd-clt: fix refcount underflow in device unmap path

During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:

  [14747.574913] ------------[ cut here ]------------
  [14747.574916] refcount_t: underflow; use-after-free.
  [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378
  [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ...
  [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G           O     N  6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary)
  [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client]
  [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90
  [14747.575037]  Call Trace:
  [14747.575038]   <TASK>
  [14747.575038]   rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client]
  [14747.575044]   process_one_work+0x211/0x600
  [14747.575052]   worker_thread+0x184/0x330
  [14747.575055]   ? __pfx_worker_thread+0x10/0x10
  [14747.575058]   kthread+0x10d/0x250
  [14747.575062]   ? __pfx_kthread+0x10/0x10
  [14747.575066]   ret_from_fork+0x319/0x390
  [14747.575069]   ? __pfx_kthread+0x10/0x10
  [14747.575072]   ret_from_fork_asm+0x1a/0x30
  [14747.575083]   </TASK>
  [14747.575096] ---[ end trace 0000000000000000 ]---

Befor this patch :-

The bug is a double kobject_put() on dev->kobj during device cleanup.

Kobject Lifecycle:

  kobject_init_and_add()  sets kobj.kref = 1  (initialization)
  kobject_put()           sets kobj.kref = 0  (should be called once)

* Before this patch:

rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs]
kobject_put(&dev->kobj) PUT #1 (WRONG!)
kref: 1 to 0
rnbd_dev_release()
kfree(dev) [DEVICE FREED!]

rnbd_destroy_gen_disk() [use-after-free!]

rnbd_clt_put_dev()

    refcount_dec_and_test(&dev->refcount)
    kobject_put(&dev->kobj)                   PUT #2 (UNDERFLOW!)
      kref: 0 to -1                           [WARNING!]

The first kobject_put() in rnbd_destroy_sysfs() prematurely frees the device via rnbd_dev_release(), then the second kobject_put() in rnbd_clt_put_dev() causes refcount underflow.

* After this patch :-

Remove kobject_put() from rnbd_destroy_sysfs(). This function should only remove sysfs visibility (kobject_del), not manage object lifetime.

Call Graph (FIXED):

rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs only]
[kref unchanged: 1]

rnbd_destroy_gen_disk() [device still valid]

rnbd_clt_put_dev()
refcount_dec_and_test(&dev->refcount)
kobject_put(&dev->kobj) ONLY PUT (CORRECT!)
kref: 1 to 0 [BALANCED]
rnbd_dev_release()
kfree(dev) [CLEAN DESTRUCTION]

This follows the kernel pattern where sysfs removal (kobject_del) is separate from object destruction (kobject_put).

Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")
---
 drivers/block/rnbd/rnbd-clt.c | 1 -
 1 file changed, 1 deletion(-)
drivers/block/rnbd/rnbd-clt.c +0 -1
index b781e8b99569..619b10f05a80 100644
@@ -1676,7 +1676,6 @@ static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev,
 			/* To avoid deadlock firstly remove itself */
 			sysfs_remove_file_self(&dev->kobj, sysfs_self);
 		kobject_del(&dev->kobj);
-		kobject_put(&dev->kobj);
 	}
 }
 
Signature · 1 line
2.40.0
maintainer Acked-by on v1 1/1
#1 lore raw
Quoted 75 lines · Chaitanya Kulkarni

During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:

  [14747.574913] ------------[ cut here ]------------
  [14747.574916] refcount_t: underflow; use-after-free.
  [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378
  [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ...
  [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G           O     N  6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary)
  [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client]
  [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90
  [14747.575037]  Call Trace:
  [14747.575038]   <TASK>
  [14747.575038]   rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client]
  [14747.575044]   process_one_work+0x211/0x600
  [14747.575052]   worker_thread+0x184/0x330
  [14747.575055]   ? __pfx_worker_thread+0x10/0x10
  [14747.575058]   kthread+0x10d/0x250
  [14747.575062]   ? __pfx_kthread+0x10/0x10
  [14747.575066]   ret_from_fork+0x319/0x390
  [14747.575069]   ? __pfx_kthread+0x10/0x10
  [14747.575072]   ret_from_fork_asm+0x1a/0x30
  [14747.575083]   </TASK>
  [14747.575096] ---[ end trace 0000000000000000 ]---

Befor this patch :-

The bug is a double kobject_put() on dev->kobj during device cleanup.

Kobject Lifecycle:

  kobject_init_and_add()  sets kobj.kref = 1  (initialization)
  kobject_put()           sets kobj.kref = 0  (should be called once)

* Before this patch:

rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs]
kobject_put(&dev->kobj) PUT #1 (WRONG!)
kref: 1 to 0
rnbd_dev_release()
kfree(dev) [DEVICE FREED!]

rnbd_destroy_gen_disk() [use-after-free!]

rnbd_clt_put_dev()

    refcount_dec_and_test(&dev->refcount)
    kobject_put(&dev->kobj)                   PUT #2 (UNDERFLOW!)
      kref: 0 to -1                           [WARNING!]

The first kobject_put() in rnbd_destroy_sysfs() prematurely frees the device via rnbd_dev_release(), then the second kobject_put() in rnbd_clt_put_dev() causes refcount underflow.

* After this patch :-

Remove kobject_put() from rnbd_destroy_sysfs(). This function should only remove sysfs visibility (kobject_del), not manage object lifetime.

Call Graph (FIXED):

rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs only]
[kref unchanged: 1]

rnbd_destroy_gen_disk() [device still valid]

rnbd_clt_put_dev()
refcount_dec_and_test(&dev->refcount)
kobject_put(&dev->kobj) ONLY PUT (CORRECT!)
kref: 1 to 0 [BALANCED]
rnbd_dev_release()
kfree(dev) [CLEAN DESTRUCTION]

Quoted 5 lines

This follows the kernel pattern where sysfs removal (kobject_del) is separate from object destruction (kobject_put).

Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")

lgtm, thx for the fix.

Quoted 19 lines
---
 drivers/block/rnbd/rnbd-clt.c | 1 -
 1 file changed, 1 deletion(-)
drivers/block/rnbd/rnbd-clt.c +0 -2
index b781e8b99569..619b10f05a80 100644
@@ -1676,7 +1676,6 @@ static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev,
                        /* To avoid deadlock firstly remove itself */
                        sysfs_remove_file_self(&dev->kobj, sysfs_self);
                kobject_del(&dev->kobj);
-               kobject_put(&dev->kobj);
        }
 }
 
--

2.40.0

maintainer Reviewed-by
#3 lore raw
Quoted 11 lines · Chaitanya Kulkarni
Quoted 7 lines · Jinpu Wang
Quoted 5 lines

This follows the kernel pattern where sysfs removal (kobject_del) is separate from object destruction (kobject_put).

Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")

lgtm, thx for the fix.

If this is correct we will needs reviewed-by tag to apply this patch.

My understanding is as module maintainer of rnbd, acked-by is enough for Jens to pickup/apply the fix.
Anyway.

Quoted 4 lines

-ck

maintainer
#4 lore raw
In reply to Chaitanya Kulkarni
Quoted 10 lines · Chaitanya Kulkarni
Quoted 7 lines · Jinpu Wang
Quoted 5 lines

This follows the kernel pattern where sysfs removal (kobject_del) is separate from object destruction (kobject_put).

Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")

lgtm, thx for the fix.

If this is correct we will needs reviewed-by tag to apply this patch.

Eh no, that is not correct. What I care about is that it has been seen and acknowledged. Whether it's a acked-by or reviewed-by doesn't matter one single bit.

Signature · 1 line
Jens Axboe
maintainer Applied
#5 lore raw
In reply to Chaitanya Kulkarni
Quoted 24 lines · Chaitanya Kulkarni

During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:

  [14747.574913] ------------[ cut here ]------------
  [14747.574916] refcount_t: underflow; use-after-free.
  [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378
  [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ...
  [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G           O     N  6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary)
  [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client]
  [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90
  [14747.575037]  Call Trace:
  [14747.575038]   <TASK>
  [14747.575038]   rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client]
  [14747.575044]   process_one_work+0x211/0x600
  [14747.575052]   worker_thread+0x184/0x330
  [14747.575055]   ? __pfx_worker_thread+0x10/0x10
  [14747.575058]   kthread+0x10d/0x250
  [14747.575062]   ? __pfx_kthread+0x10/0x10
  [14747.575066]   ret_from_fork+0x319/0x390
  [14747.575069]   ? __pfx_kthread+0x10/0x10
  [14747.575072]   ret_from_fork_asm+0x1a/0x30
  [14747.575083]   </TASK>
  [14747.575096] ---[ end trace 0000000000000000 ]---
Quoted 1 line

[...]

Applied, thanks!

[1/1] rnbd-clt: fix refcount underflow in device unmap path

      commit: ec19ed2b3e2af8ec5380400cdee9cb6560144506

Best regards,

Signature · 1 line
Jens Axboe
#6 lore raw
In reply to Chaitanya Kulkarni
Quoted 74 lines · Chaitanya Kulkarni

During device unmapping (triggered by module unload or explicit unmap), a refcount underflow occurs causing a use-after-free warning:

  [14747.574913] ------------[ cut here ]------------
  [14747.574916] refcount_t: underflow; use-after-free.
  [14747.574917] WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x55/0x90, CPU#9: kworker/9:1/378
  [14747.574924] Modules linked in: rnbd_client(-) rtrs_client rnbd_server rtrs_server rtrs_core ...
  [14747.574998] CPU: 9 UID: 0 PID: 378 Comm: kworker/9:1 Tainted: G           O     N  6.19.0-rc3lblk-fnext+ #42 PREEMPT(voluntary)
  [14747.575005] Workqueue: rnbd_clt_wq unmap_device_work [rnbd_client]
  [14747.575010] RIP: 0010:refcount_warn_saturate+0x55/0x90
  [14747.575037]  Call Trace:
  [14747.575038]   <TASK>
  [14747.575038]   rnbd_clt_unmap_device+0x170/0x1d0 [rnbd_client]
  [14747.575044]   process_one_work+0x211/0x600
  [14747.575052]   worker_thread+0x184/0x330
  [14747.575055]   ? __pfx_worker_thread+0x10/0x10
  [14747.575058]   kthread+0x10d/0x250
  [14747.575062]   ? __pfx_kthread+0x10/0x10
  [14747.575066]   ret_from_fork+0x319/0x390
  [14747.575069]   ? __pfx_kthread+0x10/0x10
  [14747.575072]   ret_from_fork_asm+0x1a/0x30
  [14747.575083]   </TASK>
  [14747.575096] ---[ end trace 0000000000000000 ]---

Befor this patch :-

The bug is a double kobject_put() on dev->kobj during device cleanup.

Kobject Lifecycle:

  kobject_init_and_add()  sets kobj.kref = 1  (initialization)
  kobject_put()           sets kobj.kref = 0  (should be called once)

* Before this patch:

rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs]
kobject_put(&dev->kobj) PUT #1 (WRONG!)
kref: 1 to 0
rnbd_dev_release()
kfree(dev) [DEVICE FREED!]

rnbd_destroy_gen_disk() [use-after-free!]

rnbd_clt_put_dev()

    refcount_dec_and_test(&dev->refcount)
    kobject_put(&dev->kobj)                   PUT #2 (UNDERFLOW!)
      kref: 0 to -1                           [WARNING!]

The first kobject_put() in rnbd_destroy_sysfs() prematurely frees the device via rnbd_dev_release(), then the second kobject_put() in rnbd_clt_put_dev() causes refcount underflow.

* After this patch :-

Remove kobject_put() from rnbd_destroy_sysfs(). This function should only remove sysfs visibility (kobject_del), not manage object lifetime.

Call Graph (FIXED):

rnbd_clt_unmap_device()
rnbd_destroy_sysfs()
kobject_del(&dev->kobj) [remove from sysfs only]
[kref unchanged: 1]

rnbd_destroy_gen_disk() [device still valid]

rnbd_clt_put_dev()
refcount_dec_and_test(&dev->refcount)
kobject_put(&dev->kobj) ONLY PUT (CORRECT!)
kref: 1 to 0 [BALANCED]
rnbd_dev_release()
kfree(dev) [CLEAN DESTRUCTION]

Quoted 5 lines

This follows the kernel pattern where sysfs removal (kobject_del) is separate from object destruction (kobject_put).

Fixes 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype")

Recently, blktests introduced the kmemleak feature. With this, I observed that rnbd/001 and rnbd/002 fail on v6.19-rc6 and v6.19-rc7 kernels with the kmemleak report as follows:

unreferenced object 0xffff88826c5f8720 (size 8):
comm "check", pid 145048, jiffies 4301507712
hex dump (first 8 bytes):

    72 6e 62 64 00 15 b3 a6                          rnbd....
  backtrace (crc f6c244b4):
    __kmalloc_node_track_caller_noprof+0x69d/0x900
    kstrdup+0x42/0xc0
    kobject_set_name_vargs+0x44/0x110
    kobject_init_and_add+0xcf/0x150
    rnbd_clt_map_device_store.cold+0x164/0x792 [rnbd_client]
    kernfs_fop_write_iter+0x3d6/0x5e0
    vfs_write+0x52c/0xf80
    ksys_write+0xfb/0x200
    do_syscall_64+0x95/0x540
    entry_SYSCALL_64_after_hwframe+0x76/0x7e

I bisected and found this commit ec19ed2b3e2a ("rnbd-clt: fix refcount underflow in device unmap path") is the trigger. When I revert it from v6.19-rc6 tag, the failure disappeared. I also found that,

- The commit message assumes that rnbd_clt_put_dev() calls kobject_put(), but such call does not exist in the kernel v6.19-rc6.
- The commit message has the Fixes tag for the commit 581cf833cac4 ("block:
rnbd: add .release to rnbd_dev_ktype"), but that is not in the kernel v6.19-rc6. Instead, the commit 581cf833cac4 is in block/for-7.0/block branch.
- The commit 581cf833cac4 added the kobject_put() call to rnbd_clt_put_dev() that the commit ec19ed2b3e2a assumes.

So I think the commit ec19ed2b3e2a was meant for block/for-7.0/block branch.

drivers/block/rnbd/rnbd-clt.c static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev, · line 1679

Quoted 13 lines
---
 drivers/block/rnbd/rnbd-clt.c | 1 -
 1 file changed, 1 deletion(-)
drivers/block/rnbd/rnbd-clt.c +0 -1
index b781e8b99569..619b10f05a80 100644
@@ -1676,7 +1676,6 @@ static void rnbd_destroy_sysfs(struct rnbd_clt_dev *dev,
 			/* To avoid deadlock firstly remove itself */
 			sysfs_remove_file_self(&dev->kobj, sysfs_self);
 		kobject_del(&dev->kobj);
-		kobject_put(&dev->kobj);

As to the current Linus master branch, the hank above was applied as the commit ec19ed2b3e2a, but the commit 581cf833cac4 is missing. So one more kobject_put() is missing. It caused the memory leak and the kmemleak feature detected it.

Haris and Jens,

I suggest to pick up 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype") to v6.19-rc8 to fix the kmemleak issue on the Linus master branch. I applied 581cf833cac4 on top of v6.18-rc7 kernel, and confirmed the kmemleak failure disappears.

#7 lore raw
Quoted 6 lines · Shinichiro Kawasaki

Haris and Jens,

I suggest to pick up 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype") to v6.19-rc8 to fix the kmemleak issue on the Linus master branch. I applied 581cf833cac4 on top of v6.18-rc7 kernel, and confirmed the kmemleak failure disappears.

Thanks you Shinichiro for detailed analysis. Please let me know if there is anything I can do. -ck

maintainer
#8 lore raw
In reply to Shinichiro Kawasaki
Quoted 6 lines · Shinichiro Kawasaki

Haris and Jens,

I suggest to pick up 581cf833cac4 ("block: rnbd: add .release to rnbd_dev_ktype") to v6.19-rc8 to fix the kmemleak issue on the Linus master branch. I applied 581cf833cac4 on top of v6.18-rc7 kernel, and confirmed the kmemleak failure disappears.

I reverted the fix from 6.19 instead, and picked up the offending commit in for-7.0/block.

Signature · 1 line
Jens Axboe