RDMA/core: Fix WARNING in gid_table_release_one
PATCH v2 1/1 RDMA/core: Fix WARNING in gid_table_release_one
GID entry ref leak for dev syz1 index 2 ref=615
...
Call Trace: <TASK> ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509 device_release+0x99/0x1c0 drivers/base/core.c:-1 kobject_cleanup lib/kobject.c:689 [inline] kobject_release lib/kobject.c:720 [inline] kref_put include/linux/kref.h:65 [inline] kobject_put+0x228/0x480 lib/kobject.c:737 process_one_work kernel/workqueue.c:3263 [inline] process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346 worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x47c/0x820 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 </TASK>
When the state of a GID is GID_TABLE_ENTRY_PENDING_DEL, it indicates that the GID is about to be released soon. Therefore, it does not appear to be a leak.
---V1->V2: Use flush_workqueue instead of while loop
--- drivers/infiniband/core/cache.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
drivers/infiniband/core/cache.c +13 -3
@@ -799,16 +799,26 @@ static void release_gid_table(struct ib_device *device,if (!table)return ; +mutex_lock (&table->lock);for (i =0 ; i < table->sz; i++) {if (is_gid_entry_free (table->data_vec[i]))continue ; -WARN_ONCE (true, -"GID entry ref leak for dev %s index %d ref=%u\n" , +WARN_ONCE (table->data_vec[i]->state !=GID_TABLE_ENTRY_PENDING_DEL , +"GID entry ref leak for dev %s index %d ref=%u, state: %d\n" ,dev_name (&device->dev), i, -kref_read (&table->data_vec[i]->kref)); +kref_read (&table->data_vec[i]->kref), table->data_vec[i]->state); +/* +* The entry may be sitting in the WQ waiting for +* free_gid_work(), flush it to try to clean it. +*/ +mutex_unlock (&table->lock); +flush_workqueue (ib_wq); +mutex_lock (&table->lock); } +mutex_unlock (&table->lock); +mutex_destroy (&table->lock);kfree (table->data_vec);kfree (table);
Signature · 1 line
2.51.2
Quoted 49 lines · Zhu Yanjun
GID entry ref leak for dev syz1 index 2 ref=615
...Call Trace: <TASK> ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509 device_release+0x99/0x1c0 drivers/base/core.c:-1 kobject_cleanup lib/kobject.c:689 [inline] kobject_release lib/kobject.c:720 [inline] kref_put include/linux/kref.h:65 [inline] kobject_put+0x228/0x480 lib/kobject.c:737 process_one_work kernel/workqueue.c:3263 [inline] process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346 worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x47c/0x820 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 </TASK>When the state of a GID is
GID_TABLE_ENTRY_PENDING_DEL, it indicates that the GID is about to be released soon. Therefore, it does not appear to be a leak.Fixes b150c3862d21 ("IB/core: Introduce GID entry reference counts")Reported-by [email protected]Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Useflush_workqueueinstead of while loop--- drivers/infiniband/core/cache.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)drivers/infiniband/core/cache.c +3
-2@@ -799,16 +799,26 @@ static void release_gid_table(struct ib_device *device,if (!table)return ; +mutex_lock (&table->lock);for (i =0 ; i < table->sz; i++) {if (is_gid_entry_free (table->data_vec[i]))continue ; - WARN_ONCE(true, - "GID entry ref leak for dev %s index %d ref=%u\n", + WARN_ONCE(table->data_vec[i]->state != GID_TABLE_ENTRY_PENDING_DEL, + "GID entry ref leak for dev %s index %d ref=%u, state: %d\n",
drivers/infiniband/core/cache.c static void release_gid_table(struct ib_device *device, · line 817
Quoted 10 lines
drivers/infiniband/core/cache.c +8
-1dev_name (&device->dev), i, - kref_read(&table->data_vec[i]->kref)); + kref_read(&table->data_vec[i]->kref), table->data_vec[i]->state); +/* +* The entry may be sitting in the WQ waiting for +* free_gid_work(), flush it to try to clean it. +*/ +mutex_unlock (&table->lock); +flush_workqueue (ib_wq); +mutex_lock (&table->lock);
I can't agree with idea that flush_workqueue() is called in the loop.
Thanks
Quoted 10 lines
drivers/infiniband/core/cache.c +2
-0} +mutex_unlock (&table->lock); +mutex_destroy (&table->lock);kfree (table->data_vec);kfree (table);Signature · 1 line
2.51.2
Quoted 46 lines · Leon Romanovsky
Quoted 45 lines · Zhu Yanjun
GID entry ref leak for dev syz1 index 2 ref=615
...Call Trace: <TASK> ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509 device_release+0x99/0x1c0 drivers/base/core.c:-1 kobject_cleanup lib/kobject.c:689 [inline] kobject_release lib/kobject.c:720 [inline] kref_put include/linux/kref.h:65 [inline] kobject_put+0x228/0x480 lib/kobject.c:737 process_one_work kernel/workqueue.c:3263 [inline] process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346 worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x47c/0x820 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 </TASK>When the state of a GID is
GID_TABLE_ENTRY_PENDING_DEL, it indicates that the GID is about to be released soon. Therefore, it does not appear to be a leak.Fixes b150c3862d21 ("IB/core: Introduce GID entry reference counts")Reported-by [email protected]Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Useflush_workqueueinstead of while loop--- drivers/infiniband/core/cache.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)drivers/infiniband/core/cache.c +1
-0@@ -799,16 +799,26 @@ static void release_gid_table(struct ib_device *device,if (!table)return ; +mutex_lock (&table->lock);for (i =0 ; i < table->sz; i++) {if (is_gid_entry_free (table->data_vec[i]))continue ;
Quoted 16 lines
Quoted 4 lines
drivers/infiniband/core/cache.c +2
-2- WARN_ONCE(true, - "GID entry ref leak for dev %s index %d ref=%u\n", + WARN_ONCE(table->data_vec[i]->state != GID_TABLE_ENTRY_PENDING_DEL, + "GID entry ref leak for dev %s index %d ref=%u, state: %d\n",drivers/infiniband/core/cache.c static void release_gid_table(struct ib_device *device, · line 817
Quoted 10 lines
drivers/infiniband/core/cache.c +8
-1dev_name (&device->dev), i, - kref_read(&table->data_vec[i]->kref)); + kref_read(&table->data_vec[i]->kref), table->data_vec[i]->state); +/* +* The entry may be sitting in the WQ waiting for +* free_gid_work(), flush it to try to clean it. +*/ +mutex_unlock (&table->lock); +flush_workqueue (ib_wq); +mutex_lock (&table->lock);I can't agree with idea that
flush_workqueue()is called in the loop.
Since we almost never see these WARN_ON's it isn't really called in a loop, but sure you could put a conditional around it to do it only once.
The WARN on is in the wrong order, it is not a kernel bug if the workqueue is still pending. flush the queue and then check again, and then do the warn.
@@ -791,22 +791,31 @@ static struct ib_gid_table *alloc_gid_table(int sz)
return NULL;
}
-static void release_gid_table(struct ib_device *device,
- struct ib_gid_table *table)
+static bool is_gid_table_clean(struct ib_gid_table *table)
{
int i;+ guard(mutex)(&table->lock);
+ for (i = 0; i < table->sz; i++)
+ if (!is_gid_entry_free(table->data_vec[i]))
+ return false;
+ return true;
+}
+
+static void release_gid_table(struct ib_device *device,
+ struct ib_gid_table *table)
+{
if (!table)
return;- for (i = 0; i < table->sz; i++) {
- if (is_gid_entry_free(table->data_vec[i]))
- continue;
-
- WARN_ONCE(true,
- "GID entry ref leak for dev %s index %d ref=%u\n",
- dev_name(&device->dev), i,
- kref_read(&table->data_vec[i]->kref));
+ if (!is_gid_table_clean(table)) {
+ /*
+ * The entry may be sitting in the WQ waiting for
+ * free_gid_work(), flush it to try to clean it.
+ */
+ flush_workqueue(ib_wq);
+ if (!is_gid_table_clean(table))
+ WARN_ONCE(true, "GID entry has leaked");
}
mutex_destroy(&table->lock);Jason
Quoted 64 lines · Jason Gunthorpe
Quoted 46 lines · Leon Romanovsky
Quoted 45 lines · Zhu Yanjun
GID entry ref leak for dev syz1 index 2 ref=615
...Call Trace: <TASK> ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509 device_release+0x99/0x1c0 drivers/base/core.c:-1 kobject_cleanup lib/kobject.c:689 [inline] kobject_release lib/kobject.c:720 [inline] kref_put include/linux/kref.h:65 [inline] kobject_put+0x228/0x480 lib/kobject.c:737 process_one_work kernel/workqueue.c:3263 [inline] process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346 worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x47c/0x820 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 </TASK>When the state of a GID is
GID_TABLE_ENTRY_PENDING_DEL, it indicates that the GID is about to be released soon. Therefore, it does not appear to be a leak.Fixes b150c3862d21 ("IB/core: Introduce GID entry reference counts")Reported-by [email protected]Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Useflush_workqueueinstead of while loop--- drivers/infiniband/core/cache.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)drivers/infiniband/core/cache.c +1
-0@@ -799,16 +799,26 @@ static void release_gid_table(struct ib_device *device,if (!table)return ; +mutex_lock (&table->lock);for (i =0 ; i < table->sz; i++) {if (is_gid_entry_free (table->data_vec[i]))continue ;Quoted 16 lines
Quoted 4 lines
drivers/infiniband/core/cache.c +2
-2- WARN_ONCE(true, - "GID entry ref leak for dev %s index %d ref=%u\n", + WARN_ONCE(table->data_vec[i]->state != GID_TABLE_ENTRY_PENDING_DEL, + "GID entry ref leak for dev %s index %d ref=%u, state: %d\n",drivers/infiniband/core/cache.c static void release_gid_table(struct ib_device *device, · line 817
Quoted 10 lines
drivers/infiniband/core/cache.c +8
-1dev_name (&device->dev), i, - kref_read(&table->data_vec[i]->kref)); + kref_read(&table->data_vec[i]->kref), table->data_vec[i]->state); +/* +* The entry may be sitting in the WQ waiting for +* free_gid_work(), flush it to try to clean it. +*/ +mutex_unlock (&table->lock); +flush_workqueue (ib_wq); +mutex_lock (&table->lock);I can't agree with idea that
flush_workqueue()is called in the loop.
Quoted 3 lines
Since we almost never see these WARN_ON's it isn't really called in a loop, but sure you could put a conditional around it to do it only once.
We have WARN_ONCE(), this is why you don't see many WARN_ON's.
Quoted 49 lines
The WARN on is in the wrong order, it is not a kernel bug if the workqueue is still pending. flush the queue and then check again, and then do the warn.
(quoted diff) +19
-10@@ -791,22 +791,31 @@ static struct ib_gid_table *alloc_gid_table(int sz) return NULL; } -static void release_gid_table(struct ib_device *device, - struct ib_gid_table *table) +static bool is_gid_table_clean(struct ib_gid_table *table) { int i; + guard(mutex)(&table->lock); + for (i = 0; i < table->sz; i++) + if (!is_gid_entry_free(table->data_vec[i])) + return false; + return true; +} + +static void release_gid_table(struct ib_device *device, + struct ib_gid_table *table) +{ if (!table) return; - for (i = 0; i < table->sz; i++) { - if (is_gid_entry_free(table->data_vec[i])) - continue; - - WARN_ONCE(true, - "GID entry ref leak for dev %s index %d ref=%u\n", - dev_name(&device->dev), i, - kref_read(&table->data_vec[i]->kref)); + if (!is_gid_table_clean(table)) { + /* + * The entry may be sitting in the WQ waiting for + * free_gid_work(), flush it to try to clean it. + */ + flush_workqueue(ib_wq); + if (!is_gid_table_clean(table)) + WARN_ONCE(true, "GID entry has leaked"); } mutex_destroy(&table->lock);Jason
Quoted 99 lines · Jason Gunthorpe
Quoted 46 lines · Leon Romanovsky
Quoted 45 lines · Zhu Yanjun
GID entry ref leak for dev syz1 index 2 ref=615
...Call Trace: <TASK> ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509 device_release+0x99/0x1c0 drivers/base/core.c:-1 kobject_cleanup lib/kobject.c:689 [inline] kobject_release lib/kobject.c:720 [inline] kref_put include/linux/kref.h:65 [inline] kobject_put+0x228/0x480 lib/kobject.c:737 process_one_work kernel/workqueue.c:3263 [inline] process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346 worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x47c/0x820 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 </TASK>When the state of a GID is
GID_TABLE_ENTRY_PENDING_DEL, it indicates that the GID is about to be released soon. Therefore, it does not appear to be a leak.Fixes b150c3862d21 ("IB/core: Introduce GID entry reference counts")Reported-by [email protected]Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Useflush_workqueueinstead of while loop--- drivers/infiniband/core/cache.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)drivers/infiniband/core/cache.c +1
-0@@ -799,16 +799,26 @@ static void release_gid_table(struct ib_device *device,if (!table)return ; +mutex_lock (&table->lock);for (i =0 ; i < table->sz; i++) {if (is_gid_entry_free (table->data_vec[i]))continue ;Quoted 15 lines
Quoted 4 lines
drivers/infiniband/core/cache.c +2
-2- WARN_ONCE(true, - "GID entry ref leak for dev %s index %d ref=%u\n", + WARN_ONCE(table->data_vec[i]->state != GID_TABLE_ENTRY_PENDING_DEL, + "GID entry ref leak for dev %s index %d ref=%u, state: %d\n",drivers/infiniband/core/cache.c static void release_gid_table(struct ib_device *device, · line 817
Quoted 10 lines
drivers/infiniband/core/cache.c +8
-1dev_name (&device->dev), i, - kref_read(&table->data_vec[i]->kref)); + kref_read(&table->data_vec[i]->kref), table->data_vec[i]->state); +/* +* The entry may be sitting in the WQ waiting for +* free_gid_work(), flush it to try to clean it. +*/ +mutex_unlock (&table->lock); +flush_workqueue (ib_wq); +mutex_lock (&table->lock);I can't agree with idea that
flush_workqueue()is called in the loop.Since we almost never see these WARN_ON's it isn't really called in a loop, but sure you could put a conditional around it to do it only once.
The WARN on is in the wrong order, it is not a kernel bug if the workqueue is still pending. flush the queue and then check again, and then do the warn.
(quoted diff) +11
-8@@ -791,22 +791,31 @@ static struct ib_gid_table *alloc_gid_table(int sz) return NULL; } -static void release_gid_table(struct ib_device *device, - struct ib_gid_table *table) +static bool is_gid_table_clean(struct ib_gid_table *table) { int i; + guard(mutex)(&table->lock); + for (i = 0; i < table->sz; i++) + if (!is_gid_entry_free(table->data_vec[i])) + return false; + return true; +} + +static void release_gid_table(struct ib_device *device, + struct ib_gid_table *table) +{ if (!table) return; - for (i = 0; i < table->sz; i++) { - if (is_gid_entry_free(table->data_vec[i])) - continue; - - WARN_ONCE(true, - "GID entry ref leak for dev %s index %d ref=%u\n",
Review static struct ib_gid_table *alloc_gid_table(int sz) · line 818
Quoted 10 lines
(quoted diff) +8
-2- dev_name(&device->dev), i, - kref_read(&table->data_vec[i]->kref)); + if (!is_gid_table_clean(table)) { + /* + * The entry may be sitting in the WQ waiting for + * free_gid_work(), flush it to try to clean it. + */ + flush_workqueue(ib_wq); + if (!is_gid_table_clean(table)) + WARN_ONCE(true, "GID entry has leaked");
Thanks a lot. IMO, if a leak occurs, more information should be helpful. Thus, the following should be better?
WARN_ONCE(table->data_vec[i]->state != GID_TABLE_ENTRY_PENDING_DEL, "GID entry ref leak for dev %s index %d ref=%u, state: %d\n", dev_name(&device->dev), index, kref_read(&table->data_vec[index]->kref), table->data_vec[index]->state);
I will make tests with syz and send the latest patch very soon.
Zhu Yanjun
Quoted 5 lines
drivers/infiniband/core/cache.c +0
-0}mutex_destroy (&table->lock);Jason
Signature · 2 lines
Best Regards, Yanjun.Zhu