Skip to contentSkip to conversation

xdp: obtain the mem_id mutex before trying to remove an entry.

PATCH Applied Jonathan Lemon 5 participants 7 messages Dec 3, 2019 – Dec 5, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xdp: obtain the mem_id mutex before trying to remove an entry.

A lockdep splat was observed when trying to remove an xdp memory model from the table since the mutex was obtained when trying to remove the entry, but not before the table walk started:

Fix the splat by obtaining the lock before starting the table walk.

Fixes c3f812cea0d7 ("page_pool: do not release pool until inflight == 0.")
---
 net/core/xdp.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
net/core/xdp.c +4 -4
index e334fad0a6b8..7c8390ad4dc6 100644
@@ -80,12 +80,8 @@ static void mem_xa_remove(struct xdp_mem_allocator *xa)
 {
 	trace_mem_disconnect(xa);
 
-	mutex_lock(&mem_id_lock);
-
 	if (!rhashtable_remove_fast(mem_id_ht, &xa->node, mem_id_rht_params))
 		call_rcu(&xa->rcu, __xdp_mem_allocator_rcu_free);
-
-	mutex_unlock(&mem_id_lock);
 }
 
 static void mem_allocator_disconnect(void *allocator)
@@ -93,6 +89,8 @@ static void mem_allocator_disconnect(void *allocator)
 	struct xdp_mem_allocator *xa;
 	struct rhashtable_iter iter;
 
+	mutex_lock(&mem_id_lock);
+
 	rhashtable_walk_enter(mem_id_ht, &iter);
 	do {
 		rhashtable_walk_start(&iter);
@@ -106,6 +104,8 @@ static void mem_allocator_disconnect(void *allocator)
 
 	} while (xa == ERR_PTR(-EAGAIN));
 	rhashtable_walk_exit(&iter);
+
+	mutex_unlock(&mem_id_lock);
 }
 
 static void mem_id_disconnect(int id)
Signature · 1 line
2.17.1
#1 lore raw
Quoted 9 lines · On Tue, 3 Dec 2019 14:01:14 -0800 Jonathan Lemon <[email protected]>…

A lockdep splat was observed when trying to remove an xdp memory model from the table since the mutex was obtained when trying to remove the entry, but not before the table walk started:

Fix the splat by obtaining the lock before starting the table walk.

Fixes c3f812cea0d7 ("page_pool: do not release pool until inflight == 0.")

Have you tested if this patch fix the problem reported by Grygorii?

Grygorii can you test this?

Quoted 31 lines
---
 net/core/xdp.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
net/core/xdp.c +2 -4
index e334fad0a6b8..7c8390ad4dc6 100644
@@ -80,12 +80,8 @@ static void mem_xa_remove(struct xdp_mem_allocator *xa)
 {
 	trace_mem_disconnect(xa);
 
-	mutex_lock(&mem_id_lock);
-
 	if (!rhashtable_remove_fast(mem_id_ht, &xa->node, mem_id_rht_params))
 		call_rcu(&xa->rcu, __xdp_mem_allocator_rcu_free);
-
-	mutex_unlock(&mem_id_lock);
 }
 
 static void mem_allocator_disconnect(void *allocator)
@@ -93,6 +89,8 @@ static void mem_allocator_disconnect(void *allocator)
 	struct xdp_mem_allocator *xa;
 	struct rhashtable_iter iter;
 
+	mutex_lock(&mem_id_lock);
+
 	rhashtable_walk_enter(mem_id_ht, &iter);
 	do {
 		rhashtable_walk_start(&iter);

net/core/xdp.c static void mem_allocator_disconnect(void *allocator) · line 111

Quoted 8 lines
net/core/xdp.c +2 -0
@@ -106,6 +104,8 @@ static void mem_allocator_disconnect(void *allocator)
 
 	} while (xa == ERR_PTR(-EAGAIN));
 	rhashtable_walk_exit(&iter);
+
+	mutex_unlock(&mem_id_lock);
 }
 
 static void mem_id_disconnect(int id)

Moving the mutex-lock might be a good idea, but I'm not sure that fixes the issue. I'm more suspect about the usage of rcu_read_lock() in xdp_rxq_info_unreg_mem_model(), listed below.

void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq)

{
	struct xdp_mem_allocator *xa;
	int id = xdp_rxq->mem.id;

	if (xdp_rxq->reg_state != REG_STATE_REGISTERED) {
		WARN(1, "Missing register, driver bug");
		return;
	}

	if (id == 0)
		return;

	if (xdp_rxq->mem.type == MEM_TYPE_ZERO_COPY)
		return mem_id_disconnect(id);

	if (xdp_rxq->mem.type == MEM_TYPE_PAGE_POOL) {
		rcu_read_lock();
		xa = rhashtable_lookup(mem_id_ht, &id, mem_id_rht_params);
		page_pool_destroy(xa->page_pool);
		rcu_read_unlock();
	}
}

EXPORT_SYMBOL_GPL(xdp_rxq_info_unreg_mem_model);

Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#2 lore raw
Quoted 13 lines · Jesper Dangaard Brouer
Quoted 9 lines · On Tue, 3 Dec 2019 14:01:14 -0800 Jonathan Lemon <[email protected]>…

A lockdep splat was observed when trying to remove an xdp memory model from the table since the mutex was obtained when trying to remove the entry, but not before the table walk started:

Fix the splat by obtaining the lock before starting the table walk.

Fixes c3f812cea0d7 ("page_pool: do not release pool until inflight == 0.")
Quoted 5 lines

Have you tested if this patch fix the problem reported by Grygorii?

Grygorii can you test this?

Thanks.
I do not see this trace any more and networking is working after if down/up

Signature · 2 lines
Best regards,
grygorii
#3 lore raw
Quoted 16 lines · On Wed, 4 Dec 2019 12:07:22 +0200 Grygorii Strashko <[email protected]>…
Quoted 15 lines · Jesper Dangaard Brouer
Quoted 9 lines · On Tue, 3 Dec 2019 14:01:14 -0800 Jonathan Lemon <[email protected]>…

A lockdep splat was observed when trying to remove an xdp memory model from the table since the mutex was obtained when trying to remove the entry, but not before the table walk started:

Fix the splat by obtaining the lock before starting the table walk.

Fixes c3f812cea0d7 ("page_pool: do not release pool until inflight == 0.")

Have you tested if this patch fix the problem reported by Grygorii?

Quoted 9 lines
Quoted 3 lines

Grygorii can you test this?

Thanks.
I do not see this trace any more and networking is working after if down/up

Well if it fixes you issue, then I guess its okay.

I just though it was related to the rcu_read_lock() around the page_pool_destroy() call. Guess, I was wrong.

Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#4 lore raw
Quoted 11 lines · Jesper Dangaard Brouer
Quoted 8 lines · On Wed, 4 Dec 2019 12:07:22 +0200 Grygorii Strashko <[email protected]>…
Quoted 7 lines · Jesper Dangaard Brouer
Quoted 4 lines · On Tue, 3 Dec 2019 14:01:14 -0800 Jonathan Lemon <[email protected]>…

A lockdep splat was observed when trying to remove an xdp memory model from the table since the mutex was obtained when trying to remove the entry, but not before the table walk started:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Fix the splat by obtaining the lock before starting the table walk.

Fixes c3f812cea0d7 ("page_pool: do not release pool until inflight == 0.")

Have you tested if this patch fix the problem reported by Grygorii?

Yes, I reproduced the problem locally, and confirmed that the patch resolves the issue.
--
Jonathan

Quoted 23 lines
Quoted 10 lines
Quoted 4 lines

Grygorii can you test this?

Thanks.
I do not see this trace any more and networking is working after if down/up

Well if it fixes you issue, then I guess its okay.

I just though it was related to the rcu_read_lock() around the page_pool_destroy() call. Guess, I was wrong.

Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#5 lore raw
In reply to Grygorii Strashko
Quoted 18 lines · Grygorii Strashko
Quoted 15 lines · Jesper Dangaard Brouer
Quoted 9 lines · On Tue, 3 Dec 2019 14:01:14 -0800 Jonathan Lemon <[email protected]>…

A lockdep splat was observed when trying to remove an xdp memory model from the table since the mutex was obtained when trying to remove the entry, but not before the table walk started:

Fix the splat by obtaining the lock before starting the table walk.

Fixes c3f812cea0d7 ("page_pool: do not release pool until inflight == 0.")

Have you tested if this patch fix the problem reported by Grygorii?

Quoted 13 lines
Quoted 3 lines

Grygorii can you test this?

Thanks.
I do not see this trace any more and networking is working after if down/up

Signature · 2 lines
Best regards,
grygorii
maintainer Applied
#6 lore raw
In reply to Jonathan Lemon

From: Jonathan Lemon <[email protected]>
Date: Tue, 3 Dec 2019 14:01:14 -0800

Quoted 9 lines

A lockdep splat was observed when trying to remove an xdp memory model from the table since the mutex was obtained when trying to remove the entry, but not before the table walk started:

Fix the splat by obtaining the lock before starting the table walk.

Fixes c3f812cea0d7 ("page_pool: do not release pool until inflight == 0.")

Applied, and since this fixes a bug fix that went back to v5.3 in -stable I will queue it there as well.

Thanks.