Skip to contentSkip to conversation

Change page_pool timeout handling

PATCH Superseded Jonathan Lemon 3 participants 7 messages Nov 14, 2019

Subsystems (8)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Change page_pool timeout handling

It isn't safe to remove the page pool until all in-flight pages are returned. If the pages are handed up the stack, it might be a while before they are returned.

The page pool can also be used independently of xdp, so it shouldn't be depending on timeout handling from the xdp memory model. Change things around so the pool handles its own timeout.

v2:
- fix wording in commit log
- fix compile error for !PAGE_POOL case
- revise tracepoint handling

Jonathan Lemon (2):
page_pool: do not release pool until inflight == 0.
page_pool: remove hold/release count from tracepoints

 .../net/ethernet/stmicro/stmmac/stmmac_main.c |   4 +-
 include/net/page_pool.h                       |  54 +++-----
 include/net/xdp_priv.h                        |   4 -
 include/trace/events/page_pool.h              |  24 ++--
 include/trace/events/xdp.h                    |  19 +--
 net/core/page_pool.c                          | 123 ++++++++++-------
 net/core/xdp.c                                | 125 +++++++-----------
 7 files changed, 152 insertions(+), 201 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/2 page_pool: do not release pool until inflight == 0.

The page pool keeps track of the number of pages in flight, and it isn't safe to remove the pool until all pages are returned.

Disallow removing the pool until all pages are back, so the pool is always available for page producers.

Make the page pool responsible for its own delayed destruction instead of relying on XDP, so the page pool can be used without the xdp memory model.

When all pages are returned, free the pool and notify xdp if the pool is registered with the xdp memory system. Have the callback perform a table walk since some drivers (cpsw) may share the pool among multiple xdp_rxq_info.

Note that the increment of pages_state_release_cnt may result in inflight == 0, releasing the pool.

Fixes d956a048cd3f ("xdp: force mem allocator removal and periodic warning")
---
 .../net/ethernet/stmicro/stmmac/stmmac_main.c |   4 +-
 include/net/page_pool.h                       |  54 +++-----
 include/net/xdp_priv.h                        |   4 -
 include/trace/events/xdp.h                    |  19 +--
 net/core/page_pool.c                          | 115 ++++++++++------
 net/core/xdp.c                                | 125 +++++++-----------
 6 files changed, 137 insertions(+), 184 deletions(-)
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +1 -3
index 199c4f938bb2..cd0d8a4bbeb3 100644
@@ -1502,10 +1502,8 @@ static void free_dma_rx_desc_resources(struct stmmac_priv *priv)
 					  rx_q->dma_erx, rx_q->dma_rx_phy);
 
 		kfree(rx_q->buf_pool);
-		if (rx_q->page_pool) {
-			page_pool_request_shutdown(rx_q->page_pool);
+		if (rx_q->page_pool)
 			page_pool_destroy(rx_q->page_pool);
-		}
 	}
 }
 
include/net/page_pool.h +15 -39

83 lines not on this page · raw message

include/net/xdp_priv.h +0 -4
index 6a8cba6ea79a..a9d5b7603b89 100644
@@ -12,12 +12,8 @@ struct xdp_mem_allocator {
 		struct page_pool *page_pool;
 		struct zero_copy_allocator *zc_alloc;
 	};
-	int disconnect_cnt;
-	unsigned long defer_start;
 	struct rhash_head node;
 	struct rcu_head rcu;
-	struct delayed_work defer_wq;
-	unsigned long defer_warn;
 };
 
 #endif /* __LINUX_NET_XDP_PRIV_H__ */
include/trace/events/xdp.h +4 -15
index c7e3c9c5bad3..a7378bcd9928 100644
@@ -317,19 +317,15 @@ __MEM_TYPE_MAP(__MEM_TYPE_TP_FN)
 
 TRACE_EVENT(mem_disconnect,
 
-	TP_PROTO(const struct xdp_mem_allocator *xa,
-		 bool safe_to_remove, bool force),
+	TP_PROTO(const struct xdp_mem_allocator *xa),
 
-	TP_ARGS(xa, safe_to_remove, force),
+	TP_ARGS(xa),
 
 	TP_STRUCT__entry(
 		__field(const struct xdp_mem_allocator *,	xa)
 		__field(u32,		mem_id)
 		__field(u32,		mem_type)
 		__field(const void *,	allocator)
-		__field(bool,		safe_to_remove)
-		__field(bool,		force)
-		__field(int,		disconnect_cnt)
 	),
 
 	TP_fast_assign(
@@ -337,19 +333,12 @@ TRACE_EVENT(mem_disconnect,
 		__entry->mem_id		= xa->mem.id;
 		__entry->mem_type	= xa->mem.type;
 		__entry->allocator	= xa->allocator;
-		__entry->safe_to_remove	= safe_to_remove;
-		__entry->force		= force;
-		__entry->disconnect_cnt	= xa->disconnect_cnt;
 	),
 
-	TP_printk("mem_id=%d mem_type=%s allocator=%p"
-		  " safe_to_remove=%s force=%s disconnect_cnt=%d",
+	TP_printk("mem_id=%d mem_type=%s allocator=%p",
 		  __entry->mem_id,
 		  __print_symbolic(__entry->mem_type, __MEM_TYPE_SYM_TAB),
-		  __entry->allocator,
-		  __entry->safe_to_remove ? "true" : "false",
-		  __entry->force ? "true" : "false",
-		  __entry->disconnect_cnt
+		  __entry->allocator
 	)
 );
 
net/core/page_pool.c +71 -44

149 lines not on this page · raw message

net/core/xdp.c +46 -79

177 lines not on this page · raw message

Signature · 1 line
2.17.1
#3 lore raw

PATCH v2 2/2 page_pool: remove hold/release count from tracepoints

In reply to Jonathan Lemon

When the last page is released from the page pool, it is possible that the delayed removal thread sees inflight == 0, and frees the pool. While the freed pointer is only copied by the tracepoint and not dereferenced, it really isn't correct. Avoid this case by reporting the page release before releasing the page.

This also removes a second atomic operation from the release path.

---
 include/trace/events/page_pool.h | 24 ++++++++++--------------
 net/core/page_pool.c             |  8 +++++---
 2 files changed, 15 insertions(+), 17 deletions(-)
include/trace/events/page_pool.h +10 -14
index 47b5ee880aa9..0adf9aed9f5b 100644
@@ -35,50 +35,46 @@ TRACE_EVENT(page_pool_inflight,
 	  __entry->pool, __entry->inflight, __entry->hold, __entry->release)
 );
 
-TRACE_EVENT(page_pool_state_release,
+TRACE_EVENT(page_pool_page_release,
 
 	TP_PROTO(const struct page_pool *pool,
-		 const struct page *page, u32 release),
+		 const struct page *page)
 
-	TP_ARGS(pool, page, release),
+	TP_ARGS(pool, page),
 
 	TP_STRUCT__entry(
 		__field(const struct page_pool *,	pool)
 		__field(const struct page *,		page)
-		__field(u32,				release)
 	),
 
 	TP_fast_assign(
 		__entry->pool		= pool;
 		__entry->page		= page;
-		__entry->release	= release;
 	),
 
-	TP_printk("page_pool=%p page=%p release=%u",
-		  __entry->pool, __entry->page, __entry->release)
+	TP_printk("page_pool=%p page=%p",
+		  __entry->pool, __entry->page)
 );
 
-TRACE_EVENT(page_pool_state_hold,
+TRACE_EVENT(page_pool_page_hold,
 
 	TP_PROTO(const struct page_pool *pool,
-		 const struct page *page, u32 hold),
+		 const struct page *page),
 
-	TP_ARGS(pool, page, hold),
+	TP_ARGS(pool, page),
 
 	TP_STRUCT__entry(
 		__field(const struct page_pool *,	pool)
 		__field(const struct page *,		page)
-		__field(u32,				hold)
 	),
 
 	TP_fast_assign(
 		__entry->pool	= pool;
 		__entry->page	= page;
-		__entry->hold	= hold;
 	),
 
-	TP_printk("page_pool=%p page=%p hold=%u",
-		  __entry->pool, __entry->page, __entry->hold)
+	TP_printk("page_pool=%p page=%p",
+		  __entry->pool, __entry->page)
 );
 
 #endif /* _TRACE_PAGE_POOL_H */
net/core/page_pool.c +5 -3
index bfe96326335d..1e66341fdac8 100644
@@ -163,7 +163,7 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
 	/* Track how many pages are held 'in-flight' */
 	pool->pages_state_hold_cnt++;
 
-	trace_page_pool_state_hold(pool, page, pool->pages_state_hold_cnt);
+	trace_page_pool_page_hold(pool, page);
 
 	/* When page just alloc'ed is should/must have refcnt 1. */
 	return page;
@@ -222,9 +222,11 @@ static void __page_pool_clean_page(struct page_pool *pool,
 			     DMA_ATTR_SKIP_CPU_SYNC);
 	page->dma_addr = 0;
 skip_dma_unmap:
+	trace_page_pool_page_release(pool, page);
+	/* This may be the last page returned, releasing the pool, so
+	 * it is not safe to reference pool afterwards.
+	 */
 	atomic_inc(&pool->pages_state_release_cnt);
-	trace_page_pool_state_release(pool, page,
-			      atomic_read(&pool->pages_state_release_cnt));
 }
 
 /* unmap the page and clean our state */
Signature · 1 line
2.17.1
#4 lore raw

Re: [net-next PATCH v2 2/2] page_pool: remove hold/release count from tracepoints

I love your patch! Yet something to improve:

Hi Jonathan,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]
[also build test ERROR on v5.4-rc7 next-20191114]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Jonathan-Lemon/page_pool-do-not-release-pool-until-inflight-0/20191115-024705
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 75a1ccfe6c726ba33a2f9859d39deb2eba620583
config: x86_64-randconfig-s1-20191115 (attached as .config)
compiler: gcc-7 (Debian 7.4.0-14) 7.4.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

In file included from net//core/page_pool.c:19:0:
include/trace/events/page_pool.h:57:1: error: macro "TRACE_EVENT" requires 6 arguments, but only 5 given

    );
    ^
   include/trace/events/page_pool.h:38:1: warning: data definition has no type or storage class
    TRACE_EVENT(page_pool_page_release,
    ^~~~~~~~~~~
   include/trace/events/page_pool.h:38:1: error: type defaults to 'int' in declaration of 'TRACE_EVENT' [-Werror=implicit-int]
   net//core/page_pool.c: In function '__page_pool_clean_page':
>> net//core/page_pool.c:225:2: error: implicit declaration of function 'trace_page_pool_page_release'; did you mean 'trace_page_pool_page_hold'? [-Werror=implicit-function-declaration]
     trace_page_pool_page_release(pool, page);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
     trace_page_pool_page_hold

cc1: some warnings being treated as errors

vim +225 net//core/page_pool.c

208
209 /* Cleanup page_pool state from page */
210 static void __page_pool_clean_page(struct page_pool *pool,
211 struct page *page)
212 {
213 dma_addr_t dma;
214
215 if (!(pool->p.flags & PP_FLAG_DMA_MAP))
216 goto skip_dma_unmap;
217
218 dma = page->dma_addr;
219 /* DMA unmap */
220 dma_unmap_page_attrs(pool->p.dev, dma,
221 PAGE_SIZE << pool->p.order, pool->p.dma_dir,
222 DMA_ATTR_SKIP_CPU_SYNC);
223 page->dma_addr = 0;
224 skip_dma_unmap:

 > 225		trace_page_pool_page_release(pool, page);

226 /* This may be the last page returned, releasing the pool, so
227 * it is not safe to reference pool afterwards.
228 */
229 atomic_inc(&pool->pages_state_release_cnt);
230 }
231

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

#5 lore raw

Re: [net-next PATCH v2 2/2] page_pool: remove hold/release count from tracepoints

In reply to Jonathan Lemon
Quoted 5 lines · On Thu, 14 Nov 2019 08:37:15 -0800 Jonathan Lemon <[email protected]>…

When the last page is released from the page pool, it is possible that the delayed removal thread sees inflight == 0, and frees the pool. While the freed pointer is only copied by the tracepoint and not dereferenced, it really isn't correct. Avoid this case by reporting the page release before releasing the page.

I don't like this patch!

I'm actually using these counters, in my current version of my bpftrace leak detector for page_pool:

https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_leaks01.bt

Quoted 7 lines

This also removes a second atomic operation from the release path.

---
 include/trace/events/page_pool.h | 24 ++++++++++--------------
 net/core/page_pool.c             |  8 +++++---
 2 files changed, 15 insertions(+), 17 deletions(-)

[...]

net/core/page_pool.c static void __page_pool_clean_page(struct page_pool *pool, · line 230

Quoted 12 lines
net/core/page_pool.c +4 -2
@@ -222,9 +222,11 @@ static void __page_pool_clean_page(struct page_pool *pool,
 			     DMA_ATTR_SKIP_CPU_SYNC);
 	page->dma_addr = 0;
 skip_dma_unmap:
+	trace_page_pool_page_release(pool, page);
+	/* This may be the last page returned, releasing the pool, so
+	 * it is not safe to reference pool afterwards.
+	 */
 	atomic_inc(&pool->pages_state_release_cnt);
-	trace_page_pool_state_release(pool, page,
-			      atomic_read(&pool->pages_state_release_cnt));
 }

I will prefer that you do an atomic_inc_return, and send the cnt to the existing tracepoint. I'm not dereferencing the pool in my tracepoint use-case, and as Alexei wrote, this would still be 'safe' (as in not crashing) for a tracepoint if someone do.

Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#2 lore raw

Re: [net-next PATCH v2 1/2] page_pool: do not release pool until inflight == 0.

In reply to Jonathan Lemon
Quoted 5 lines · On Thu, 14 Nov 2019 08:37:14 -0800 Jonathan Lemon <[email protected]>…

The page pool keeps track of the number of pages in flight, and it isn't safe to remove the pool until all pages are returned.

Disallow removing the pool until all pages are back, so the pool is always available for page producers.

I like this patch.

Quoted 11 lines

Make the page pool responsible for its own delayed destruction instead of relying on XDP, so the page pool can be used without the xdp memory model.

When all pages are returned, free the pool and notify xdp if the pool is registered with the xdp memory system. Have the callback perform a table walk since some drivers (cpsw) may share the pool among multiple xdp_rxq_info.

Note that the increment of pages_state_release_cnt may result in inflight == 0, releasing the pool.

Maybe we can just do the atomic_inc_return trick, and this patch can itself can go in.

Alternative is to release the pool via RCU, like the xa structure is freed via RCU (which have a pointer to the pool).

Quoted 3 lines
Fixes d956a048cd3f ("xdp: force mem allocator removal and periodic warning")

---

Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#6 lore raw

Re: [net-next PATCH v2 2/2] page_pool: remove hold/release count from tracepoints

In reply to Jesper Dangaard Brouer
Quoted 4 lines · Jesper Dangaard Brouer

I will prefer that you do an atomic_inc_return, and send the cnt to the existing tracepoint. I'm not dereferencing the pool in my tracepoint use-case, and as Alexei wrote, this would still be 'safe' (as in not crashing) for a tracepoint if someone do.

Okay, will make that change, and send out a revision.

Signature · 1 line
Jonathan