Change page_pool timeout handling
Subsystems (8)
v2 cover Change page_pool timeout handling
It isn't safe to remove the page pool until all in-flight pages are returned. If the pages are handed up the stack, it might be a while before they are returned.
The page pool can also be used independently of xdp, so it shouldn't be depending on timeout handling from the xdp memory model. Change things around so the pool handles its own timeout.
v2:
- fix wording in commit log
- fix compile error for !PAGE_POOL case
- revise tracepoint handling
Jonathan Lemon (2):
page_pool: do not release pool until inflight == 0.
page_pool: remove hold/release count from tracepoints
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 4 +- include/net/page_pool.h | 54 +++----- include/net/xdp_priv.h | 4 - include/trace/events/page_pool.h | 24 ++-- include/trace/events/xdp.h | 19 +-- net/core/page_pool.c | 123 ++++++++++------- net/core/xdp.c | 125 +++++++----------- 7 files changed, 152 insertions(+), 201 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/2 page_pool: do not release pool until inflight == 0.
The page pool keeps track of the number of pages in flight, and it isn't safe to remove the pool until all pages are returned.
Disallow removing the pool until all pages are back, so the pool is always available for page producers.
Make the page pool responsible for its own delayed destruction instead of relying on XDP, so the page pool can be used without the xdp memory model.
When all pages are returned, free the pool and notify xdp if the pool is registered with the xdp memory system. Have the callback perform a table walk since some drivers (cpsw) may share the pool among multiple xdp_rxq_info.
Note that the increment of pages_state_release_cnt may result in inflight == 0, releasing the pool.
--- .../net/ethernet/stmicro/stmmac/stmmac_main.c | 4 +- include/net/page_pool.h | 54 +++----- include/net/xdp_priv.h | 4 - include/trace/events/xdp.h | 19 +-- net/core/page_pool.c | 115 ++++++++++------ net/core/xdp.c | 125 +++++++----------- 6 files changed, 137 insertions(+), 184 deletions(-)
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +1 -3
@@ -1502,10 +1502,8 @@ static void free_dma_rx_desc_resources(struct stmmac_priv *priv) rx_q->dma_erx, rx_q->dma_rx_phy);kfree (rx_q->buf_pool); -if (rx_q->page_pool) { -page_pool_request_shutdown (rx_q->page_pool); +if (rx_q->page_pool)page_pool_destroy (rx_q->page_pool); - } } }
include/net/page_pool.h +15 -39
83 lines not on this page · raw message
include/net/xdp_priv.h +0 -4
@@ -12,12 +12,8 @@ struct xdp_mem_allocator {struct page_pool *page_pool; struct zero_copy_allocator *zc_alloc; }; -int disconnect_cnt; -unsigned long defer_start;struct rhash_head node;struct rcu_head rcu; -struct delayed_work defer_wq; -unsigned long defer_warn; };#endif /* __LINUX_NET_XDP_PRIV_H__ */
include/trace/events/xdp.h +4 -15
@@ -317,19 +317,15 @@ __MEM_TYPE_MAP(__MEM_TYPE_TP_FN) TRACE_EVENT(mem_disconnect, - TP_PROTO(const struct xdp_mem_allocator *xa, - bool safe_to_remove, bool force), + TP_PROTO(const struct xdp_mem_allocator *xa), - TP_ARGS(xa, safe_to_remove, force), + TP_ARGS(xa), TP_STRUCT__entry( __field(const struct xdp_mem_allocator *, xa) __field(u32, mem_id) __field(u32, mem_type) __field(const void *, allocator) - __field(bool, safe_to_remove) - __field(bool, force) - __field(int, disconnect_cnt) ), TP_fast_assign(
@@ -337,19 +333,12 @@ TRACE_EVENT(mem_disconnect, __entry->mem_id = xa->mem.id; __entry->mem_type = xa->mem.type; __entry->allocator = xa->allocator; - __entry->safe_to_remove = safe_to_remove; - __entry->force = force; - __entry->disconnect_cnt = xa->disconnect_cnt; ), -TP_printk ("mem_id=%d mem_type=%s allocator=%p" -" safe_to_remove=%s force=%s disconnect_cnt=%d" , +TP_printk ("mem_id=%d mem_type=%s allocator=%p" , __entry->mem_id,__print_symbolic (__entry->mem_type, __MEM_TYPE_SYM_TAB), - __entry->allocator, - __entry->safe_to_remove ?"true" :"false" , - __entry->force ?"true" :"false" , - __entry->disconnect_cnt + __entry->allocator ) );
net/core/page_pool.c +71 -44
149 lines not on this page · raw message
net/core/xdp.c +46 -79
177 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [net-next PATCH v2 1/2] page_pool: do not release pool until inflight == 0.
Quoted 5 lines · On Thu, 14 Nov 2019 08:37:14 -0800 Jonathan Lemon <[email protected]>…
The page pool keeps track of the number of pages in flight, and it isn't safe to remove the pool until all pages are returned.
Disallow removing the pool until all pages are back, so the pool is always available for page producers.
I like this patch.
Quoted 11 lines
Make the page pool responsible for its own delayed destruction instead of relying on XDP, so the page pool can be used without the xdp memory model.
When all pages are returned, free the pool and notify xdp if the pool is registered with the xdp memory system. Have the callback perform a table walk since some drivers (cpsw) may share the pool among multiple
xdp_rxq_info.Note that the increment of
pages_state_release_cntmay result in inflight == 0, releasing the pool.
Maybe we can just do the atomic_inc_return trick, and this patch can itself can go in.
Alternative is to release the pool via RCU, like the xa structure is freed via RCU (which have a pointer to the pool).
Quoted 3 lines
Fixes d956a048cd3f ("xdp: force mem allocator removal and periodic warning")Signed-off-by Jonathan Lemon <[email protected]>---
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
PATCH v2 2/2 page_pool: remove hold/release count from tracepoints
In reply to Jonathan LemonWhen the last page is released from the page pool, it is possible that the delayed removal thread sees inflight == 0, and frees the pool. While the freed pointer is only copied by the tracepoint and not dereferenced, it really isn't correct. Avoid this case by reporting the page release before releasing the page.
This also removes a second atomic operation from the release path.
--- include/trace/events/page_pool.h | 24 ++++++++++-------------- net/core/page_pool.c | 8 +++++--- 2 files changed, 15 insertions(+), 17 deletions(-)
include/trace/events/page_pool.h +10 -14
@@ -35,50 +35,46 @@ TRACE_EVENT(page_pool_inflight, __entry->pool, __entry->inflight, __entry->hold, __entry->release) ); -TRACE_EVENT (page_pool_state_release, +TRACE_EVENT (page_pool_page_release, TP_PROTO(const struct page_pool *pool, - const struct page *page, u32 release), + const struct page *page) -TP_ARGS (pool, page, release), +TP_ARGS (pool, page),TP_STRUCT__entry ( __field(const struct page_pool *, pool) __field(const struct page *, page) -__field (u32, release) ),TP_fast_assign ( __entry->pool = pool; __entry->page = page; - __entry->release = release; ), -TP_printk ("page_pool=%p page=%p release=%u" , - __entry->pool, __entry->page, __entry->release) +TP_printk ("page_pool=%p page=%p" , + __entry->pool, __entry->page) ); -TRACE_EVENT (page_pool_state_hold, +TRACE_EVENT (page_pool_page_hold, TP_PROTO(const struct page_pool *pool, - const struct page *page, u32 hold), + const struct page *page), -TP_ARGS (pool, page, hold), +TP_ARGS (pool, page),TP_STRUCT__entry ( __field(const struct page_pool *, pool) __field(const struct page *, page) -__field (u32, hold) ),TP_fast_assign ( __entry->pool = pool; __entry->page = page; - __entry->hold = hold; ), -TP_printk ("page_pool=%p page=%p hold=%u" , - __entry->pool, __entry->page, __entry->hold) +TP_printk ("page_pool=%p page=%p" , + __entry->pool, __entry->page) );#endif /* _TRACE_PAGE_POOL_H */
net/core/page_pool.c +5 -3
@@ -163,7 +163,7 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,/* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; -trace_page_pool_state_hold (pool, page, pool->pages_state_hold_cnt); +trace_page_pool_page_hold (pool, page);/* When page just alloc'ed is should/must have refcnt 1. */ return page;
@@ -222,9 +222,11 @@ static void __page_pool_clean_page(struct page_pool *pool, DMA_ATTR_SKIP_CPU_SYNC); page->dma_addr =0 ; skip_dma_unmap: +trace_page_pool_page_release (pool, page); +/* This may be the last page returned, releasing the pool, so +* it is not safe to reference pool afterwards. +*/ atomic_inc (&pool->pages_state_release_cnt); -trace_page_pool_state_release (pool, page, -atomic_read (&pool->pages_state_release_cnt)); }/* unmap the page and clean our state */
Signature · 1 line
2.17.1
kbuild test robot
build botRe: [net-next PATCH v2 2/2] page_pool: remove hold/release count from tracepoints
I love your patch! Yet something to improve:
Hi Jonathan,
I love your patch! Yet something to improve:
[auto build test ERROR on net-next/master]
[also build test ERROR on v5.4-rc7 next-20191114]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Jonathan-Lemon/page_pool-do-not-release-pool-until-inflight-0/20191115-024705 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 75a1ccfe6c726ba33a2f9859d39deb2eba620583 config: x86_64-randconfig-s1-20191115 (attached as .config) compiler: gcc-7 (Debian 7.4.0-14) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
In file included from net//core/page_pool.c:19:0:
include/trace/events/page_pool.h:57:1: error: macro "TRACE_EVENT" requires 6 arguments, but only 5 given
);
^
include/trace/events/page_pool.h:38:1: warning: data definition has no type or storage class
TRACE_EVENT(page_pool_page_release,
^~~~~~~~~~~
include/trace/events/page_pool.h:38:1: error: type defaults to 'int' in declaration of 'TRACE_EVENT' [-Werror=implicit-int]
net//core/page_pool.c: In function '__page_pool_clean_page':
>> net//core/page_pool.c:225:2: error: implicit declaration of function 'trace_page_pool_page_release'; did you mean 'trace_page_pool_page_hold'? [-Werror=implicit-function-declaration]
trace_page_pool_page_release(pool, page);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
trace_page_pool_page_holdcc1: some warnings being treated as errors
vim +225 net//core/page_pool.c
208
209 /* Cleanup page_pool state from page */
210 static void __page_pool_clean_page(struct page_pool *pool,
211 struct page *page)
212 {
213 dma_addr_t dma;
214
215 if (!(pool->p.flags & PP_FLAG_DMA_MAP))
216 goto skip_dma_unmap;
217
218 dma = page->dma_addr;
219 /* DMA unmap */
220 dma_unmap_page_attrs(pool->p.dev, dma,
221 PAGE_SIZE << pool->p.order, pool->p.dma_dir,
222 DMA_ATTR_SKIP_CPU_SYNC);
223 page->dma_addr = 0;
224 skip_dma_unmap:
> 225 trace_page_pool_page_release(pool, page);
226 /* This may be the last page returned, releasing the pool, so
227 * it is not safe to reference pool afterwards.
228 */
229 atomic_inc(&pool->pages_state_release_cnt);
230 }
231
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation
Re: [net-next PATCH v2 2/2] page_pool: remove hold/release count from tracepoints
In reply to Jonathan LemonQuoted 5 lines · On Thu, 14 Nov 2019 08:37:15 -0800 Jonathan Lemon <[email protected]>…
When the last page is released from the page pool, it is possible that the delayed removal thread sees inflight == 0, and frees the pool. While the freed pointer is only copied by the tracepoint and not dereferenced, it really isn't correct. Avoid this case by reporting the page release before releasing the page.
I don't like this patch!
I'm actually using these counters, in my current version of my bpftrace leak detector for page_pool:
https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_leaks01.bt
Quoted 7 lines
This also removes a second atomic operation from the release path.
Signed-off-by Jonathan Lemon <[email protected]>--- include/trace/events/page_pool.h | 24 ++++++++++-------------- net/core/page_pool.c | 8 +++++--- 2 files changed, 15 insertions(+), 17 deletions(-)
[...]
net/core/page_pool.c static void __page_pool_clean_page(struct page_pool *pool, · line 230
Quoted 12 lines
net/core/page_pool.c +4
-2@@ -222,9 +222,11 @@ static void __page_pool_clean_page(struct page_pool *pool, DMA_ATTR_SKIP_CPU_SYNC); page->dma_addr =0 ; skip_dma_unmap: +trace_page_pool_page_release (pool, page); +/* This may be the last page returned, releasing the pool, so +* it is not safe to reference pool afterwards. +*/ atomic_inc (&pool->pages_state_release_cnt); -trace_page_pool_state_release (pool, page, -atomic_read (&pool->pages_state_release_cnt)); }
I will prefer that you do an atomic_inc_return, and send the cnt to the existing tracepoint. I'm not dereferencing the pool in my tracepoint use-case, and as Alexei wrote, this would still be 'safe' (as in not crashing) for a tracepoint if someone do.
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Re: [net-next PATCH v2 2/2] page_pool: remove hold/release count from tracepoints
Quoted 4 lines · Jesper Dangaard Brouer
I will prefer that you do an
atomic_inc_return, and send the cnt to the existing tracepoint. I'm not dereferencing the pool in my tracepoint use-case, and as Alexei wrote, this would still be 'safe' (as in not crashing) for a tracepoint if someone do.
Okay, will make that change, and send out a revision.
Signature · 1 line
Jonathan