Skip to contentSkip to conversation

Pre-req for hfi1 cdev rework

PATCH Applied Dennis Dalessandro 3 participants 11 messages Mar 26, 2020 – Mar 27, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Pre-req for hfi1 cdev rework

Kaike found a couple issues while working on the cdev stuff. Here are two fixes that should probably precede those patches (which are yet to be posted)

---

Kaike Wan (2):
IB/hfi1: Fix memory leaks in sysfs registration and unregistration IB/hfi1: Call kobject_put() when kobject_init_and_add() fails

 drivers/infiniband/hw/hfi1/sysfs.c |   26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

--
-Denny

#1 lore raw

PATCH 1/2 IB/hfi1: Fix memory leaks in sysfs registration and unregistration

From: Kaike Wan <[email protected]>

When the hfi1 driver is unloaded, kmemleak will report the following issue:

unreferenced object 0xffff8888461a4c08 (size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:

[<00000000311a6ef5>] kvasprintf+0x62/0xd0
[<00000000ade94d9f>] kobject_set_name_vargs+0x1c/0x90
[<0000000060657dbb>] kobject_init_and_add+0x5d/0xb0
[<00000000346fe72b>] 0xffffffffa0c5ecba
[<000000006cfc5819>] 0xffffffffa0c866b9
[<0000000031c65580>] 0xffffffffa0c38e87
[<00000000e9739b3f>] local_pci_probe+0x41/0x80
[<000000006c69911d>] work_for_cpu_fn+0x16/0x20
[<00000000601267b5>] process_one_work+0x171/0x380
[<0000000049a0eefa>] worker_thread+0x1d1/0x3f0
[<00000000909cf2b9>] kthread+0xf8/0x130
[<0000000058f5f874>] ret_from_fork+0x35/0x40

This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject in hfi1_unregister_sysfs().
- This will fix the memory leak.
- Calling kobject_put() to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) in hfi1_verbs_register_sysfs().

Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinity setup")
---
 drivers/infiniband/hw/hfi1/sysfs.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/infiniband/hw/hfi1/sysfs.c +11 -2
index 90f62c4..f1bcecf 100644
@@ -853,8 +853,13 @@ int hfi1_verbs_register_sysfs(struct hfi1_devdata *dd)
 
 	return 0;
 bail:
-	for (i = 0; i < dd->num_sdma; i++)
-		kobject_del(&dd->per_sdma[i].kobj);
+	/*
+	 * The function kobject_put() will call kobject_del() if the kobject
+	 * has been added successfully. The sysfs files created under the
+	 * kobject directory will also be removed during the process.
+	 */
+	for (; i >= 0; i--)
+		kobject_put(&dd->per_sdma[i].kobj);
 
 	return ret;
 }
@@ -867,6 +872,10 @@ void hfi1_verbs_unregister_sysfs(struct hfi1_devdata *dd)
 	struct hfi1_pportdata *ppd;
 	int i;
 
+	/* Unwind operations in hfi1_verbs_register_sysfs() */
+	for (i = 0; i < dd->num_sdma; i++)
+		kobject_put(&dd->per_sdma[i].kobj);
+
 	for (i = 0; i < dd->num_pports; i++) {
 		ppd = &dd->pport[i];
 
#2 lore raw

Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 30 lines · Dennis Dalessandro

From: Kaike Wan <[email protected]>

When the hfi1 driver is unloaded, kmemleak will report the following issue:

unreferenced object 0xffff8888461a4c08 (size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:

[<00000000311a6ef5>] kvasprintf+0x62/0xd0
[<00000000ade94d9f>] kobject_set_name_vargs+0x1c/0x90
[<0000000060657dbb>] kobject_init_and_add+0x5d/0xb0
[<00000000346fe72b>] 0xffffffffa0c5ecba
[<000000006cfc5819>] 0xffffffffa0c866b9
[<0000000031c65580>] 0xffffffffa0c38e87
[<00000000e9739b3f>] local_pci_probe+0x41/0x80
[<000000006c69911d>] work_for_cpu_fn+0x16/0x20
[<00000000601267b5>] process_one_work+0x171/0x380
[<0000000049a0eefa>] worker_thread+0x1d1/0x3f0
[<00000000909cf2b9>] kthread+0xf8/0x130
[<0000000058f5f874>] ret_from_fork+0x35/0x40

This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject in hfi1_unregister_sysfs().
- This will fix the memory leak.
- Calling kobject_put() to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) in hfi1_verbs_register_sysfs().

Quoted 8 lines
Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinity setup")
---
 drivers/infiniband/hw/hfi1/sysfs.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

I'm not certain, but this seems unwise.

After hfi1_verbs_unregiser_sysfs() returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.

kobject_del() triggers synchronous delete of the sysfs, while kobject_put() potentially defers it to the future.

Will ib unregister fail if the kobject_del() has not happened yet? I am unsure.

Jason

#3 lore raw

RE: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 51 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 1:26 PM
To: Dalessandro, Dennis <[email protected]>
Cc: [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]; Wan, Kaike <[email protected]>
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 10 lines · Dennis Dalessandro

From: Kaike Wan <[email protected]>

When the hfi1 driver is unloaded, kmemleak will report the following issue:

unreferenced object 0xffff8888461a4c08 (size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:

Quoted 6 lines
[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>]
kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>]
kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>] 0xffffffffa0c5ecba

[<000000006cfc5819>] 0xffffffffa0c866b9 [<0000000031c65580>] 0xffffffffa0c38e87 [<00000000e9739b3f>] local_pci_probe+0x41/0x80 [<000000006c69911d>] work_for_cpu_fn+0x16/0x20

[<00000000601267b5>]

Quoted 12 lines
process_one_work+0x171/0x380 [<0000000049a0eefa>]
worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>] kthread+0xf8/0x130
[<0000000058f5f874>] ret_from_fork+0x35/0x40

This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject in hfi1_unregister_sysfs().
- This will fix the memory leak.
- Calling kobject_put() to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) in hfi1_verbs_register_sysfs().

Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinity
Quoted 8 lines

setup")

---
 drivers/infiniband/hw/hfi1/sysfs.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
Quoted 7 lines

I'm not certain, but this seems unwise.

After hfi1_verbs_unregiser_sysfs() returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.

kobject_del() triggers synchronous delete of the sysfs, while kobject_put() potentially defers it to the future.

True. However, kobject_del() will only delete the sysfs for the object, ie, unwrap what has been done in object_add, but it will not decrement the refcount for the kobject. To unwap kobject_init_and_add(), one can call
(1) kobject_del() (optional)
(2) object_put()

The kobject cleanup function (kobject_cleanup()) will call kobject_del if kobj->state_in_sys is set. Therefore, one can call object_put() alone to get the job done.

Quoted 3 lines

Will ib unregister fail if the kobject_del() has not happened yet? I am unsure.

I don't think so. We only observed the kmemleak complaints after unloading the driver, nothing else.

Kaike

#4 lore raw

Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 57 lines · Kaike
Quoted 49 lines

From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 1:26 PM
To: Dalessandro, Dennis <[email protected]>
Cc: [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]; Wan, Kaike <[email protected]>
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 10 lines · Dennis Dalessandro

From: Kaike Wan <[email protected]>

When the hfi1 driver is unloaded, kmemleak will report the following issue:

unreferenced object 0xffff8888461a4c08 (size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:

Quoted 6 lines
[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>]
kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>]
kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>] 0xffffffffa0c5ecba

[<000000006cfc5819>] 0xffffffffa0c866b9 [<0000000031c65580>] 0xffffffffa0c38e87 [<00000000e9739b3f>] local_pci_probe+0x41/0x80 [<000000006c69911d>] work_for_cpu_fn+0x16/0x20

[<00000000601267b5>]

Quoted 11 lines
process_one_work+0x171/0x380 [<0000000049a0eefa>]
worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>] kthread+0xf8/0x130
[<0000000058f5f874>] ret_from_fork+0x35/0x40

This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject in hfi1_unregister_sysfs().
- This will fix the memory leak.
- Calling kobject_put() to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the current one that has just failed) in hfi1_verbs_register_sysfs().

Quoted 8 lines
Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinity

setup")

 drivers/infiniband/hw/hfi1/sysfs.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
Quoted 7 lines

I'm not certain, but this seems unwise.

After hfi1_verbs_unregiser_sysfs() returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.

kobject_del() triggers synchronous delete of the sysfs, while kobject_put() potentially defers it to the future.

Quoted 6 lines

True. However, kobject_del() will only delete the sysfs for the object, ie, unwrap what has been done in object_add, but it will not decrement the refcount for the kobject. To unwap kobject_init_and_add(), one can call
(1) kobject_del() (optional)
(2) object_put()

Yes, you must call both, but kobject_put is not a replacement for kobject_del.

Quoted 3 lines

The kobject cleanup function (kobject_cleanup()) will call kobject_del if kobj->state_in_sys is set. Therefore, one can call object_put() alone to get the job done.

No, as I already explained, the moment that kobject_del happens is no longer reliable with kobject_put.

Quoted 4 lines
Quoted 1 line

Will ib unregister fail if the kobject_del() has not happened yet? I am unsure.

I don't think so. We only observed the kmemleak complaints after unloading the driver, nothing else.

Of course, hfi1 is missing the required kobject_put, so it was only a leak.

To see if there is an issue here delete the kobject_del and kobject_put entirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.

Jason

#5 lore raw

RE: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 59 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 3:43 PM
To: Wan, Kaike <[email protected]>
Cc: Dalessandro, Dennis <[email protected]>; [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

When the hfi1 driver is unloaded, kmemleak will report the following
issue:

unreferenced object 0xffff8888461a4c08 (size 8):
comm "kworker/0:0", pid 5, jiffies 4298601264 (age 2047.134s) hex dump (first 8 bytes):
73 64 6d 61 30 00 ff ff sdma0...
backtrace:

[<00000000311a6ef5>] kvasprintf+0x62/0xd0 [<00000000ade94d9f>]
kobject_set_name_vargs+0x1c/0x90 [<0000000060657dbb>]
kobject_init_and_add+0x5d/0xb0 [<00000000346fe72b>]

0xffffffffa0c5ecba [<000000006cfc5819>] 0xffffffffa0c866b9 [<0000000031c65580>]
0xffffffffa0c38e87 [<00000000e9739b3f>] local_pci_probe+0x41/0x80 [<000000006c69911d>] work_for_cpu_fn+0x16/0x20

Quoted 3 lines
Quoted 3 lines

[<00000000601267b5>]

Quoted 2 lines
process_one_work+0x171/0x380 [<0000000049a0eefa>]
worker_thread+0x1d1/0x3f0 [<00000000909cf2b9>]
kthread+0xf8/0x130
> > > [<0000000058f5f874>] ret_from_fork+0x35/0x40
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

This patch fixes the issue by:
- Releasing dd->per_sdma[i].kobject in hfi1_unregister_sysfs().
- This will fix the memory leak.
- Calling kobject_put() to unwind operations only for those entries in dd->per_sdma[] whose operations have succeeded (including the

current

Quoted 18 lines
Quoted 18 lines
Quoted 10 lines

one that has just failed) in hfi1_verbs_register_sysfs().

Fixes 0cb2aa690c7e ("IB/hfi1: Add sysfs interface for affinity

setup")

 drivers/infiniband/hw/hfi1/sysfs.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

I'm not certain, but this seems unwise.

After hfi1_verbs_unregiser_sysfs() returns there should be no sysfs left under the ibdev as we are going to delete the ibdev sysfs next.

kobject_del() triggers synchronous delete of the sysfs, while kobject_put() potentially defers it to the future.

Quoted 8 lines
Quoted 6 lines

True. However, kobject_del() will only delete the sysfs for the object, ie, unwrap what has been done in object_add, but it will not decrement the refcount for the kobject. To unwap kobject_init_and_add(), one can call
(1) kobject_del() (optional)
(2) object_put()

Yes, you must call both, but kobject_put is not a replacement for kobject_del.

We can do that.

Quoted 11 lines
Quoted 4 lines

The kobject cleanup function (kobject_cleanup()) will call kobject_del if kobj->state_in_sys is set. Therefore, one can call object_put() alone to get the job done.

No, as I already explained, the moment that kobject_del happens is no longer reliable with kobject_put.

Quoted 1 line
Quoted 1 line

Will ib unregister fail if the kobject_del() has not happened yet? I am

unsure.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

I don't think so. We only observed the kmemleak complaints after unloading the driver, nothing else.

Of course, hfi1 is missing the required kobject_put, so it was only a leak.

To see if there is an issue here delete the kobject_del and kobject_put entirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.

I tried a patch wherein the function hfi1_verbs_unregister_sysfs() is never called at all and when unloading the driver the ib device un-registration went through smoothly(no error, the /sys/class/infiniband/hfi1_0 directory gone). Only kmemleak complaints were observed.

I will re-spin the patches.

Thanks,

Kaike

#6 lore raw

Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 9 lines · Kaike
Quoted 3 lines

To see if there is an issue here delete the kobject_del and kobject_put entirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.

I tried a patch wherein the function hfi1_verbs_unregister_sysfs() is never called at all and when unloading the driver the ib device un-registration went through smoothly(no error, the /sys/class/infiniband/hfi1_0 directory gone). Only kmemleak complaints were observed.

Then perhaps there is nothing to worry about and the patches are fine

Jason

#7 lore raw

RE: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 18 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, March 26, 2020 6:37 PM
To: Wan, Kaike <[email protected]>
Cc: Dalessandro, Dennis <[email protected]>; [email protected]; [email protected]; Marciniszyn, Mike <[email protected]>; [email protected]
Subject: Re: [PATCH for-rc 1/2] IB/hfi1: Fix memory leaks in sysfs registration and unregistration

Quoted 3 lines · Kaike
Quoted 3 lines

To see if there is an issue here delete the kobject_del and kobject_put entirely to leave a dangling sysfs during registration and see if ib device unregistration explodes.

Quoted 8 lines
Quoted 5 lines

I tried a patch wherein the function hfi1_verbs_unregister_sysfs() is never called at all and when unloading the driver the ib device un-registration went through smoothly(no error, the /sys/class/infiniband/hfi1_0 directory gone). Only kmemleak complaints were observed.

Then perhaps there is nothing to worry about and the patches are fine

Then I don't have to re-spin the patches?

Kaike

#8 lore raw

PATCH 2/2 IB/hfi1: Call kobject_put() when kobject_init_and_add() fails

In reply to Dennis Dalessandro

From: Kaike Wan <[email protected]>

When kobject_init_and_add() returns an error in the function hfi1_create_port_files(), the function kobject_put() is not called for the corresponding kobject, which potentially leads to memory leak.

This patch fixes the issue by calling kobject_put() even if kobject_init_and_add() fails.

---
 drivers/infiniband/hw/hfi1/sysfs.c |   13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/infiniband/hw/hfi1/sysfs.c +8 -5
index f1bcecf..074ec71 100644
@@ -674,7 +674,11 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,
 		dd_dev_err(dd,
 			   "Skipping sc2vl sysfs info, (err %d) port %u\n",
 			   ret, port_num);
-		goto bail;
+		/*
+		 * Based on the documentation for kobject_init_and_add(), the
+		 * caller should call kobject_put even if this call fails.
+		 */
+		goto bail_sc2vl;
 	}
 	kobject_uevent(&ppd->sc2vl_kobj, KOBJ_ADD);
 
@@ -684,7 +688,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,
 		dd_dev_err(dd,
 			   "Skipping sl2sc sysfs info, (err %d) port %u\n",
 			   ret, port_num);
-		goto bail_sc2vl;
+		goto bail_sl2sc;
 	}
 	kobject_uevent(&ppd->sl2sc_kobj, KOBJ_ADD);
 
@@ -694,7 +698,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,
 		dd_dev_err(dd,
 			   "Skipping vl2mtu sysfs info, (err %d) port %u\n",
 			   ret, port_num);
-		goto bail_sl2sc;
+		goto bail_vl2mtu;
 	}
 	kobject_uevent(&ppd->vl2mtu_kobj, KOBJ_ADD);
 
@@ -704,7 +708,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,
 		dd_dev_err(dd,
 			   "Skipping Congestion Control sysfs info, (err %d) port %u\n",
 			   ret, port_num);
-		goto bail_vl2mtu;
+		goto bail_cc;
 	}
 
 	kobject_uevent(&ppd->pport_cc_kobj, KOBJ_ADD);
@@ -742,7 +746,6 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,
 	kobject_put(&ppd->sl2sc_kobj);
 bail_sc2vl:
 	kobject_put(&ppd->sc2vl_kobj);
-bail:
 	return ret;
 }
 
#9 lore raw

Re: [PATCH for-rc 2/2] IB/hfi1: Call kobject_put() when kobject_init_and_add() fails

Quoted 6 lines · Dennis Dalessandro

From: Kaike Wan <[email protected]>

When kobject_init_and_add() returns an error in the function hfi1_create_port_files(), the function kobject_put() is not called for the corresponding kobject, which potentially leads to memory leak.

Quoted 7 lines

This patch fixes the issue by calling kobject_put() even if kobject_init_and_add() fails.

My bad, overzealous commit message clean up on my part, lost this:

Fixes 7724105686e7 ("IB/hfi1: add driver files")

-Denny

#10 lore raw
In reply to Dennis Dalessandro
Quoted 8 lines · Dennis Dalessandro

Kaike found a couple issues while working on the cdev stuff. Here are two fixes that should probably precede those patches (which are yet to be posted)

---

Kaike Wan (2):
IB/hfi1: Fix memory leaks in sysfs registration and unregistration IB/hfi1: Call kobject_put() when kobject_init_and_add() fails

Applied to for-next, for-rc is done now.

Thanks,
Jason