Pre-req for hfi1 cdev rework
Subsystems (3)
cover Pre-req for hfi1 cdev rework
Kaike found a couple issues while working on the cdev stuff. Here are two fixes that should probably precede those patches (which are yet to be posted)
---
Kaike Wan (2):
IB/hfi1: Fix memory leaks in sysfs registration and unregistration IB/hfi1: Call kobject_put() when kobject_init_and_add() fails
drivers/infiniband/hw/hfi1/sysfs.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
--
-Denny
PATCH 1/2 IB/hfi1: Fix memory leaks in sysfs registration and unregistration
From: Kaike Wan <[email protected]>
When the hfi1 driver is unloaded, kmemleak will report the following issue:
unreferenced object 0xffff8888461a4c08 (size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:
[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>] kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>] kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>] 0xffffffffa0c5ecba [<000000006cfc5819>] 0xffffffffa0c866b9 [<0000000031c65580>] 0xffffffffa0c38e87 [<00000000e9739b3f>] local_pci_probe+0x41/0x80 [<000000006c69911d>] work_for_cpu_fn+0x16/0x20 [<00000000601267b5>] process_one_work+0x171/0x380 [<0000000049a0eefa>] worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>] kthread+0xf8/0x130 [<0000000058f5f874>] ret_from_fork+0x35/0x40
This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject in hfi1_unregister_sysfs().
- This will fix the memory leak.
- Calling kobject_put() to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) in hfi1_verbs_register_sysfs().
--- drivers/infiniband/hw/hfi1/sysfs.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/infiniband/hw/hfi1/sysfs.c +11 -2
@@ -853,8 +853,13 @@ int hfi1_verbs_register_sysfs(struct hfi1_devdata *dd)return 0 ; bail: -for (i =0 ; i < dd->num_sdma; i++) -kobject_del (&dd->per_sdma[i].kobj); +/* +* The function kobject_put() will call kobject_del() if the kobject +* has been added successfully. The sysfs files created under the +* kobject directory will also be removed during the process. +*/ +for (; i >=0 ; i--) +kobject_put (&dd->per_sdma[i].kobj);return ret; }
@@ -867,6 +872,10 @@ void hfi1_verbs_unregister_sysfs(struct hfi1_devdata *dd)struct hfi1_pportdata *ppd;int i; +/* Unwind operations in hfi1_verbs_register_sysfs() */ +for (i =0 ; i < dd->num_sdma; i++) +kobject_put (&dd->per_sdma[i].kobj); +for (i =0 ; i < dd->num_pports; i++) { ppd = &dd->pport[i];
Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration
Quoted 30 lines · Dennis Dalessandro
From: Kaike Wan <[email protected]>
When the hfi1 driver is unloaded, kmemleak will report the following issue:
unreferenced object
0xffff8888461a4c08(size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>] kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>] kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>] 0xffffffffa0c5ecba [<000000006cfc5819>] 0xffffffffa0c866b9 [<0000000031c65580>] 0xffffffffa0c38e87 [<00000000e9739b3f>] local_pci_probe+0x41/0x80 [<000000006c69911d>] work_for_cpu_fn+0x16/0x20 [<00000000601267b5>] process_one_work+0x171/0x380 [<0000000049a0eefa>] worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>] kthread+0xf8/0x130 [<0000000058f5f874>] ret_from_fork+0x35/0x40This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject inhfi1_unregister_sysfs().
- This will fix the memory leak.
- Callingkobject_put()to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) inhfi1_verbs_register_sysfs().
Quoted 8 lines
Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinity setup")Cc <[email protected]>Reviewed-by Mike Marciniszyn <[email protected]>Signed-off-by Kaike Wan <[email protected]>Signed-off-by Dennis Dalessandro <[email protected]>--- drivers/infiniband/hw/hfi1/sysfs.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
I'm not certain, but this seems unwise.
After hfi1_verbs_unregiser_sysfs() returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.
kobject_del() triggers synchronous delete of the sysfs, while kobject_put() potentially defers it to the future.
Will ib unregister fail if the kobject_del() has not happened yet? I am unsure.
Jason
RE: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration
Quoted 51 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 1:26 PM
To: Dalessandro, Dennis <[email protected]>
Cc: [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]; Wan, Kaike <[email protected]>
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistrationQuoted 10 lines · Dennis Dalessandro
From: Kaike Wan <[email protected]>
When the hfi1 driver is unloaded, kmemleak will report the following issue:
unreferenced object
0xffff8888461a4c08(size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:Quoted 6 lines
[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>] kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>] kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>] 0xffffffffa0c5ecba[<
000000006cfc5819>]0xffffffffa0c866b9[<0000000031c65580>]0xffffffffa0c38e87[<00000000e9739b3f>]local_pci_probe+0x41/0x80[<000000006c69911d>]work_for_cpu_fn+0x16/0x20[<
00000000601267b5>]Quoted 12 lines
process_one_work+0x171/0x380 [<0000000049a0eefa>] worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>] kthread+0xf8/0x130 [<0000000058f5f874>] ret_from_fork+0x35/0x40This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject inhfi1_unregister_sysfs().
- This will fix the memory leak.
- Callingkobject_put()to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) inhfi1_verbs_register_sysfs().Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinityQuoted 8 lines
setup")
Cc <[email protected]>Reviewed-by Mike Marciniszyn <[email protected]>Signed-off-by Kaike Wan <[email protected]>Signed-off-by Dennis Dalessandro <[email protected]>--- drivers/infiniband/hw/hfi1/sysfs.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
Quoted 7 lines
I'm not certain, but this seems unwise.
After
hfi1_verbs_unregiser_sysfs()returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.
kobject_del()triggers synchronous delete of the sysfs, whilekobject_put()potentially defers it to the future.
True. However, kobject_del() will only delete the sysfs for the object, ie, unwrap what has been done in object_add, but it will not decrement the refcount for the kobject. To unwap kobject_init_and_add(), one can call
(1) kobject_del() (optional)
(2) object_put()
The kobject cleanup function (kobject_cleanup()) will call kobject_del if kobj->state_in_sys is set. Therefore, one can call object_put() alone to get the job done.
Quoted 3 lines
Will ib unregister fail if the
kobject_del()has not happened yet? I am unsure.
I don't think so. We only observed the kmemleak complaints after unloading the driver, nothing else.
Kaike
Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration
Quoted 57 lines · Kaike
Quoted 49 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 1:26 PM
To: Dalessandro, Dennis <[email protected]>
Cc: [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]; Wan, Kaike <[email protected]>
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistrationQuoted 10 lines · Dennis Dalessandro
From: Kaike Wan <[email protected]>
When the hfi1 driver is unloaded, kmemleak will report the following issue:
unreferenced object
0xffff8888461a4c08(size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:Quoted 6 lines
[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>] kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>] kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>] 0xffffffffa0c5ecba[<
000000006cfc5819>]0xffffffffa0c866b9[<0000000031c65580>]0xffffffffa0c38e87[<00000000e9739b3f>]local_pci_probe+0x41/0x80[<000000006c69911d>]work_for_cpu_fn+0x16/0x20[<
00000000601267b5>]Quoted 11 lines
process_one_work+0x171/0x380 [<0000000049a0eefa>] worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>] kthread+0xf8/0x130 [<0000000058f5f874>] ret_from_fork+0x35/0x40This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject inhfi1_unregister_sysfs().
- This will fix the memory leak.
- Callingkobject_put()to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) inhfi1_verbs_register_sysfs().Quoted 8 lines
Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinitysetup")
Cc <[email protected]>Reviewed-by Mike Marciniszyn <[email protected]>Signed-off-by Kaike Wan <[email protected]>Signed-off-by Dennis Dalessandro <[email protected]>drivers/infiniband/hw/hfi1/sysfs.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)Quoted 7 lines
I'm not certain, but this seems unwise.
After
hfi1_verbs_unregiser_sysfs()returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.
kobject_del()triggers synchronous delete of the sysfs, whilekobject_put()potentially defers it to the future.
Quoted 6 lines
True. However,
kobject_del()will only delete the sysfs for the object, ie, unwrap what has been done inobject_add, but it will not decrement the refcount for the kobject. To unwapkobject_init_and_add(), one can call
(1)kobject_del()(optional)
(2)object_put()
Yes, you must call both, but kobject_put is not a replacement for kobject_del.
Quoted 3 lines
The kobject cleanup function (
kobject_cleanup()) will callkobject_delifkobj->state_in_sysis set. Therefore, one can callobject_put()alone to get the job done.
No, as I already explained, the moment that kobject_del happens is no longer reliable with kobject_put.
Quoted 4 lines
Quoted 1 line
Will ib unregister fail if the
kobject_del()has not happened yet? I am unsure.I don't think so. We only observed the kmemleak complaints after unloading the driver, nothing else.
Of course, hfi1 is missing the required kobject_put, so it was only a leak.
To see if there is an issue here delete the kobject_del and kobject_put entirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.
Jason
RE: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration
Quoted 59 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 3:43 PM
To: Wan, Kaike <[email protected]>
Cc: Dalessandro, Dennis <[email protected]>; [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistrationQuoted 16 lines
Quoted 16 lines
Quoted 16 lines
When the hfi1 driver is unloaded, kmemleak will report the following
issue:unreferenced object
0xffff8888461a4c08(size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>] kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>] kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>]
0xffffffffa0c5ecba[<000000006cfc5819>]0xffffffffa0c866b9[<0000000031c65580>]0xffffffffa0c38e87[<00000000e9739b3f>]local_pci_probe+0x41/0x80[<000000006c69911d>]work_for_cpu_fn+0x16/0x20Quoted 3 lines
Quoted 3 lines
[<
00000000601267b5>]Quoted 2 lines
process_one_work+0x171/0x380 [<0000000049a0eefa>] worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>]kthread+0xf8/0x130 > > > [<0000000058f5f874>] ret_from_fork+0x35/0x40Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject inhfi1_unregister_sysfs().
- This will fix the memory leak.
- Callingkobject_put()to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including thecurrent
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
one that has just failed) in
hfi1_verbs_register_sysfs().Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinitysetup")
Cc <[email protected]>Reviewed-by Mike Marciniszyn <[email protected]>Signed-off-by Kaike Wan <[email protected]>Signed-off-by Dennis Dalessandro <[email protected]>drivers/infiniband/hw/hfi1/sysfs.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)I'm not certain, but this seems unwise.
After
hfi1_verbs_unregiser_sysfs()returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.
kobject_del()triggers synchronous delete of the sysfs, whilekobject_put()potentially defers it to the future.
Quoted 8 lines
Quoted 6 lines
True. However,
kobject_del()will only delete the sysfs for the object, ie, unwrap what has been done inobject_add, but it will not decrement the refcount for the kobject. To unwapkobject_init_and_add(), one can call
(1)kobject_del()(optional)
(2)object_put()Yes, you must call both, but
kobject_putis not a replacement forkobject_del.
We can do that.
Quoted 11 lines
Quoted 4 lines
The kobject cleanup function (
kobject_cleanup()) will callkobject_delifkobj->state_in_sysis set. Therefore, one can callobject_put()alone to get the job done.No, as I already explained, the moment that
kobject_delhappens is no longer reliable withkobject_put.Quoted 1 line
Quoted 1 line
Will ib unregister fail if the
kobject_del()has not happened yet? I amunsure.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
I don't think so. We only observed the kmemleak complaints after unloading the driver, nothing else.
Of course, hfi1 is missing the required
kobject_put, so it was only a leak.To see if there is an issue here delete the
kobject_delandkobject_putentirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.
I tried a patch wherein the function hfi1_verbs_unregister_sysfs() is never called at all and when unloading the driver the ib device un-registration went through smoothly(no error, the /sys/class/infiniband/hfi1_0 directory gone). Only kmemleak complaints were observed.
I will re-spin the patches.
Thanks,
Kaike
Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration
Quoted 9 lines · Kaike
Quoted 3 lines
To see if there is an issue here delete the
kobject_delandkobject_putentirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.I tried a patch wherein the function
hfi1_verbs_unregister_sysfs()is never called at all and when unloading the driver the ib device un-registration went throughsmoothly(no error, the /sys/class/infiniband/hfi1_0 directory gone). Only kmemleak complaints were observed.
Then perhaps there is nothing to worry about and the patches are fine
Jason
RE: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration
Quoted 18 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 6:37 PM
To: Wan, Kaike <[email protected]>
Cc: Dalessandro, Dennis <[email protected]>; [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistrationQuoted 3 lines · Kaike
Quoted 3 lines
To see if there is an issue here delete the
kobject_delandkobject_putentirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.
Quoted 8 lines
Quoted 5 lines
I tried a patch wherein the function
hfi1_verbs_unregister_sysfs()is never called at all and when unloading the driver the ib device un-registration went throughsmoothly(no error, the /sys/class/infiniband/hfi1_0 directory gone). Only kmemleak complaints were observed.Then perhaps there is nothing to worry about and the patches are fine
Then I don't have to re-spin the patches?
Kaike
PATCH 2/2 IB/hfi1: Call kobject_put() when kobject_init_and_add() fails
In reply to Dennis DalessandroFrom: Kaike Wan <[email protected]>
When kobject_init_and_add() returns an error in the function hfi1_create_port_files(), the function kobject_put() is not called for the corresponding kobject, which potentially leads to memory leak.
This patch fixes the issue by calling kobject_put() even if kobject_init_and_add() fails.
--- drivers/infiniband/hw/hfi1/sysfs.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/infiniband/hw/hfi1/sysfs.c +8 -5
@@ -674,7 +674,11 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,dd_dev_err (dd,"Skipping sc2vl sysfs info, (err %d) port %u\n" , ret, port_num); - goto bail; +/* +* Based on the documentation for kobject_init_and_add(), the +* caller should call kobject_put even if this call fails. +*/ + goto bail_sc2vl; }kobject_uevent (&ppd->sc2vl_kobj,KOBJ_ADD );
@@ -684,7 +688,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,dd_dev_err (dd,"Skipping sl2sc sysfs info, (err %d) port %u\n" , ret, port_num); - goto bail_sc2vl; + goto bail_sl2sc; }kobject_uevent (&ppd->sl2sc_kobj,KOBJ_ADD );
@@ -694,7 +698,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,dd_dev_err (dd,"Skipping vl2mtu sysfs info, (err %d) port %u\n" , ret, port_num); - goto bail_sl2sc; + goto bail_vl2mtu; }kobject_uevent (&ppd->vl2mtu_kobj,KOBJ_ADD );
@@ -704,7 +708,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,dd_dev_err (dd,"Skipping Congestion Control sysfs info, (err %d) port %u\n" , ret, port_num); - goto bail_vl2mtu; + goto bail_cc; }kobject_uevent (&ppd->pport_cc_kobj,KOBJ_ADD );
@@ -742,7 +746,6 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,kobject_put (&ppd->sl2sc_kobj); bail_sc2vl:kobject_put (&ppd->sc2vl_kobj); -bail:return ret; }
Re: [PATCH for-rc 2/2] IB/hfi1: Call kobject_put() when kobject_init_and_add() fails
Quoted 6 lines · Dennis Dalessandro
From: Kaike Wan <[email protected]>
When
kobject_init_and_add()returns an error in the functionhfi1_create_port_files(), the functionkobject_put()is not called for the corresponding kobject, which potentially leads to memory leak.
Quoted 7 lines
This patch fixes the issue by calling
kobject_put()even ifkobject_init_and_add()fails.Cc <[email protected]>Reviewed-by Mike Marciniszyn <[email protected]>Signed-off-by Kaike Wan <[email protected]>Signed-off-by Dennis Dalessandro <[email protected]>
My bad, overzealous commit message clean up on my part, lost this:
-Denny
Jason Gunthorpe
AppliedQuoted 8 lines · Dennis Dalessandro
Kaike found a couple issues while working on the cdev stuff. Here are two fixes that should probably precede those patches (which are yet to be posted)
---
Kaike Wan (2):
IB/hfi1: Fix memory leaks in sysfs registration and unregistration IB/hfi1: Callkobject_put()whenkobject_init_and_add()fails
Applied to for-next, for-rc is done now.
Thanks,
Jason