page_pool: NUMA node handling fixes
Subsystems (3)
v6 cover page_pool: NUMA node handling fixes
The recently added NUMA changes (merged for v5.5) to page_pool, it both contains a bug in handling NUMA_NO_NODE condition, and added code to the fast-path.
This patchset fixes the bug and moves code out of fast-path. The first patch contains a fix that should be considered for 5.5. The second patch reduce code size and overhead in case CONFIG_NUMA is disabled.
Currently the NUMA_NO_NODE setting bug only affects driver 'ti_cpsw' (drivers/net/ethernet/ti/), but after this patchset, we plan to move other drivers (netsec and mvneta) to use NUMA_NO_NODE setting.
---
Jesper Dangaard Brouer (2):
page_pool: handle page recycle for NUMA_NO_NODE condition
page_pool: help compiler remove code in case CONFIG_NUMA=n
net/core/page_pool.c | 89 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 70 insertions(+), 19 deletions(-)
--
PATCH v6 1/2 page_pool: handle page recycle for NUMA_NO_NODE condition
The check in pool_page_reusable (page_to_nid(page) == pool->p.nid) is not valid if page_pool was configured with pool->p.nid = NUMA_NO_NODE.
The goal of the NUMA changes in commit d5394610b1ba ("page_pool: Don't recycle non-reusable pages"), were to have RX-pages that belongs to the same NUMA node as the CPU processing RX-packet during softirq/NAPI. As illustrated by the performance measurements.
This patch moves the NAPI checks out of fast-path, and at the same time solves the NUMA_NO_NODE issue.
First realize that alloc_pages_node() with pool->p.nid = NUMA_NO_NODE will lookup current CPU nid (Numa ID) via numa_mem_id(), which is used as the the preferred nid. It is only in rare situations, where e.g. NUMA zone runs dry, that page gets doesn't get allocated from preferred nid. The page_pool API allows drivers to control the nid themselves via controlling pool->p.nid.
This patch moves the NAPI check to when alloc cache is refilled, via dequeuing/consuming pages from the ptr_ring. Thus, we can allow placing pages from remote NUMA into the ptr_ring, as the dequeue/consume step will check the NUMA node. All current drivers using page_pool will alloc/refill RX-ring from same CPU running softirq/NAPI process.
Drivers that control the nid explicitly, also use page_pool_update_nid when changing nid runtime. To speed up transision to new nid the alloc cache is now flushed on nid changes. This force pages to come from ptr_ring, which does the appropate nid check.
For the NUMA_NO_NODE case, when a NIC IRQ is moved to another NUMA node, we accept that transitioning the alloc cache doesn't happen immediately. The preferred nid change runtime via consulting numa_mem_id() based on the CPU processing RX-packets.
Notice, to avoid stressing the page buddy allocator and avoid doing too much work under softirq with preempt disabled, the NUMA check at ptr_ring dequeue will break the refill cycle, when detecting a NUMA mismatch. This will cause a slower transition, but its done on purpose.
--- net/core/page_pool.c | 80 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 61 insertions(+), 19 deletions(-)
net/core/page_pool.c +61 -19
@@ -96,10 +96,60 @@ struct page_pool *page_pool_create(const struct page_pool_params *params) }EXPORT_SYMBOL (page_pool_create); +static void __page_pool_return_page (struct page_pool *pool,struct page *page); + +noinline +static struct page *page_pool_refill_alloc_cache(struct page_pool *pool, +bool refill) +{ +struct ptr_ring *r = &pool->ring; +struct page *page; +int pref_nid;/* preferred NUMA node */ + +/* Quicker fallback, avoid locks when ring is empty */ +if (__ptr_ring_empty (r)) +return NULL ; + +/* Softirq guarantee CPU and thus NUMA node is stable. This, +* assumes CPU refilling driver RX-ring will also run RX-NAPI. +*/ + pref_nid = (pool->p.nid ==NUMA_NO_NODE ) ?numa_mem_id () : pool->p.nid; + +/* Slower-path: Get pages from locked ring queue */ +spin_lock (&r->consumer_lock); + +/* Refill alloc array, but only if NUMA match */ +do { + page =__ptr_ring_consume (r); +if (unlikely (!page)) +break ; + +if (likely (page_to_nid (page) == pref_nid)) { + pool->alloc.cache[pool->alloc.count++] = page; + }else { +/* NUMA mismatch; +* (1) release 1 page to page-allocator and +* (2) break out to fallthough to alloc_pages_node. +* This limit stress on page buddy alloactor. +*/ +__page_pool_return_page (pool, page); + page =NULL ; +break ; + } + }while (pool->alloc.count <PP_ALLOC_CACHE_REFILL && + refill); + +/* Return last page */ +if (likely (pool->alloc.count >0 )) + page = pool->alloc.cache[--pool->alloc.count]; + +spin_unlock (&r->consumer_lock); +return page; +} +/* fast path */ static struct page *__page_pool_get_cached (struct page_pool *pool) { -struct ptr_ring *r = &pool->ring;bool refill = false;struct page *page;
@@ -113,20 +163,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) refill = true; } -/* Quicker fallback, avoid locks when ring is empty */ -if (__ptr_ring_empty (r)) -return NULL ; - -/* Slow-path: Get page from locked ring queue, -* refill alloc array if requested. -*/ -spin_lock (&r->consumer_lock); - page =__ptr_ring_consume (r); -if (refill) - pool->alloc.count =__ptr_ring_consume_batched (r, - pool->alloc.cache, -PP_ALLOC_CACHE_REFILL ); -spin_unlock (&r->consumer_lock); + page =page_pool_refill_alloc_cache (pool, refill);return page; }
@@ -311,13 +348,10 @@ static bool __page_pool_recycle_direct(struct page *page,/* page is NOT reusable when: * 1) allocated when system is under some pressure. (page_is_pfmemalloc) -* 2) belongs to a different NUMA node than pool->p.nid. -* -* To update pool->p.nid users must call page_pool_update_nid. */ static bool pool_page_reusable (struct page_pool *pool,struct page *page) { -return !page_is_pfmemalloc (page) &&page_to_nid (page) == pool->p.nid; +return !page_is_pfmemalloc (page); } void __page_pool_put_page(struct page_pool *pool, struct page *page,
@@ -484,7 +518,15 @@ EXPORT_SYMBOL(page_pool_destroy);/* Caller must provide appropriate safe context, e.g. NAPI. */ void page_pool_update_nid (struct page_pool *pool,int new_nid) { +struct page *page; +trace_page_pool_update_nid (pool, new_nid); pool->p.nid = new_nid; + +/* Flush pool alloc cache, as refill will check NUMA node */ +while (pool->alloc.count) { + page = pool->alloc.cache[--pool->alloc.count]; +__page_pool_return_page (pool, page); + } }EXPORT_SYMBOL (page_pool_update_nid);
Re: [net-next v6 PATCH 1/2] page_pool: handle page recycle for NUMA_NO_NODE condition
Quoted 10 lines · Jesper Dangaard Brouer
The check in
pool_page_reusable(page_to_nid(page)==pool->p.nid) is not valid ifpage_poolwas configured withpool->p.nid=NUMA_NO_NODE.The goal of the NUMA changes in commit
d5394610b1ba("page_pool: Don't recycle non-reusable pages"), were to have RX-pages that belongs to the same NUMA node as the CPU processing RX-packet during softirq/NAPI. As illustrated by the performance measurements.This patch moves the NAPI checks out of fast-path, and at the same time solves the
NUMA_NO_NODEissue.
There seems to be a minor NUMA_NO_NODE case that has not been handled by this patch yet:
1. When the page is always recycled to pool->alloc.cache.
2. And page_pool_alloc_pages always return pages from pool->alloc.cache.
Then non-local page will be reused when the rx NAPI affinity changes.
Of coure we can handle above by calling page_pool_update_nid(), which may require very user to call page_pool_update_nid() explicitly even the user has specify the pool->p.nid as NUMA_NO_NODE.
Any consideration for above case?
Quoted 69 lines
First realize that
alloc_pages_node()withpool->p.nid=NUMA_NO_NODEwill lookup current CPU nid (Numa ID) vianuma_mem_id(), which is used as the the preferred nid. It is only in rare situations, where e.g. NUMA zone runs dry, that page gets doesn't get allocated from preferred nid. Thepage_poolAPI allows drivers to control the nid themselves via controllingpool->p.nid.This patch moves the NAPI check to when alloc cache is refilled, via dequeuing/consuming pages from the
ptr_ring. Thus, we can allow placing pages from remote NUMA into theptr_ring, as the dequeue/consume step will check the NUMA node. All current drivers usingpage_poolwill alloc/refill RX-ring from same CPU running softirq/NAPI process.Drivers that control the nid explicitly, also use
page_pool_update_nidwhen changing nid runtime. To speed up transision to new nid the alloc cache is now flushed on nid changes. This force pages to come fromptr_ring, which does the appropate nid check.For the
NUMA_NO_NODEcase, when a NIC IRQ is moved to another NUMA node, we accept that transitioning the alloc cache doesn't happen immediately. The preferred nid change runtime via consultingnuma_mem_id()based on the CPU processing RX-packets.Notice, to avoid stressing the page buddy allocator and avoid doing too much work under softirq with preempt disabled, the NUMA check at
ptr_ringdequeue will break the refill cycle, when detecting a NUMA mismatch. This will cause a slower transition, but its done on purpose.Fixes d5394610b1ba ("page_pool: Don't recycle non-reusable pages")Reported-by Li RongQing <[email protected]>Reported-by Yunsheng Lin <[email protected]>Signed-off-by Jesper Dangaard Brouer <[email protected]>--- net/core/page_pool.c | 80 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 61 insertions(+), 19 deletions(-)net/core/page_pool.c +24
-0@@ -96,10 +96,60 @@ struct page_pool *page_pool_create(const struct page_pool_params *params) }EXPORT_SYMBOL (page_pool_create); +static void __page_pool_return_page (struct page_pool *pool,struct page *page); + +noinline +static struct page *page_pool_refill_alloc_cache(struct page_pool *pool, +bool refill) +{ +struct ptr_ring *r = &pool->ring; +struct page *page; +int pref_nid;/* preferred NUMA node */ + +/* Quicker fallback, avoid locks when ring is empty */ +if (__ptr_ring_empty (r)) +return NULL ; + +/* Softirq guarantee CPU and thus NUMA node is stable. This, +* assumes CPU refilling driver RX-ring will also run RX-NAPI. +*/ + pref_nid = (pool->p.nid ==NUMA_NO_NODE ) ?numa_mem_id () : pool->p.nid; + +/* Slower-path: Get pages from locked ring queue */ +spin_lock (&r->consumer_lock); + +/* Refill alloc array, but only if NUMA match */ + do {
net/core/page_pool.c struct page_pool *page_pool_create(const struct page_pool_params *params) · line 132
Quoted 10 lines
net/core/page_pool.c +10
-0+ page =__ptr_ring_consume (r); +if (unlikely (!page)) +break ; + +if (likely (page_to_nid (page) == pref_nid)) { + pool->alloc.cache[pool->alloc.count++] = page; + }else { +/* NUMA mismatch; +* (1) release 1 page to page-allocator and +* (2) break out to fallthough to alloc_pages_node.
fallthough -> fallthrough ?
Quoted 82 lines
net/core/page_pool.c +27
-19+* This limit stress on page buddy alloactor. +*/ +__page_pool_return_page (pool, page); + page =NULL ; +break ; + } + }while (pool->alloc.count <PP_ALLOC_CACHE_REFILL && + refill); + +/* Return last page */ +if (likely (pool->alloc.count >0 )) + page = pool->alloc.cache[--pool->alloc.count]; + +spin_unlock (&r->consumer_lock); +return page; +} +/* fast path */ static struct page *__page_pool_get_cached (struct page_pool *pool) { -struct ptr_ring *r = &pool->ring;bool refill = false;struct page *page;@@ -113,20 +163,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) refill = true; } -/* Quicker fallback, avoid locks when ring is empty */ -if (__ptr_ring_empty (r)) -return NULL ; - -/* Slow-path: Get page from locked ring queue, -* refill alloc array if requested. -*/ -spin_lock (&r->consumer_lock); - page =__ptr_ring_consume (r); -if (refill) - pool->alloc.count =__ptr_ring_consume_batched (r, - pool->alloc.cache, -PP_ALLOC_CACHE_REFILL ); -spin_unlock (&r->consumer_lock); + page =page_pool_refill_alloc_cache (pool, refill);return page; }@@ -311,13 +348,10 @@ static bool __page_pool_recycle_direct(struct page *page,/* page is NOT reusable when: * 1) allocated when system is under some pressure. (page_is_pfmemalloc) -* 2) belongs to a different NUMA node than pool->p.nid. -* -* To update pool->p.nid users must call page_pool_update_nid. */ static bool pool_page_reusable (struct page_pool *pool,struct page *page) { -return !page_is_pfmemalloc (page) &&page_to_nid (page) == pool->p.nid; +return !page_is_pfmemalloc (page); } void __page_pool_put_page(struct page_pool *pool, struct page *page,@@ -484,7 +518,15 @@ EXPORT_SYMBOL(page_pool_destroy);/* Caller must provide appropriate safe context, e.g. NAPI. */ void page_pool_update_nid (struct page_pool *pool,int new_nid) { +struct page *page; +trace_page_pool_update_nid (pool, new_nid); pool->p.nid = new_nid; + +/* Flush pool alloc cache, as refill will check NUMA node */ +while (pool->alloc.count) { + page = pool->alloc.cache[--pool->alloc.count]; +__page_pool_return_page (pool, page); + } }EXPORT_SYMBOL (page_pool_update_nid);.
Re: [net-next v6 PATCH 1/2] page_pool: handle page recycle for NUMA_NO_NODE condition
Quoted 18 lines · On Mon, 30 Dec 2019 09:59:23 +0800 Yunsheng Lin <[email protected]> wrote:
Quoted 10 lines · Jesper Dangaard Brouer
The check in
pool_page_reusable(page_to_nid(page)==pool->p.nid) is not valid ifpage_poolwas configured withpool->p.nid=NUMA_NO_NODE.The goal of the NUMA changes in commit
d5394610b1ba("page_pool: Don't recycle non-reusable pages"), were to have RX-pages that belongs to the same NUMA node as the CPU processing RX-packet during softirq/NAPI. As illustrated by the performance measurements.This patch moves the NAPI checks out of fast-path, and at the same time solves the
NUMA_NO_NODEissue.There seems to be a minor
NUMA_NO_NODEcase that has not been handled by this patch yet:1. When the page is always recycled to
pool->alloc.cache.
2. Andpage_pool_alloc_pagesalways return pages frompool->alloc.cache.
Quoted 7 lines
Then non-local page will be reused when the rx NAPI affinity changes.
Of coure we can handle above by calling
page_pool_update_nid(), which may require very user to callpage_pool_update_nid()explicitly even the user has specify thepool->p.nidasNUMA_NO_NODE.Any consideration for above case?
Yes, I tried to explain below, that "we accept that transitioning the alloc cache doesn't happen immediately.". Thus, I've not forgotten about this case, and I still consider this patch correct.
You have to consider/understand how this page_pool cache is used. The alloc cache can only get "recycled" pages in rare situations. The case basically only happen for XDP_DROP (which have extreme performance needs). Sure, in a 100% XDP_DROP case, then your described case can happen, but that case is unlikely/useless in real-life production. Some packets must be let through, which will have to travel through the ptr_ring, where the NUMA check is done. Thus, pages will eventually transition to the new NAPI node of the new IRQ-CPU.
Quoted 19 lines
Quoted 19 lines
First realize that
alloc_pages_node()withpool->p.nid=NUMA_NO_NODEwill lookup current CPU nid (Numa ID) vianuma_mem_id(), which is used as the the preferred nid. It is only in rare situations, where e.g. NUMA zone runs dry, that page gets doesn't get allocated from preferred nid. Thepage_poolAPI allows drivers to control the nid themselves via controllingpool->p.nid.This patch moves the NAPI check to when alloc cache is refilled, via dequeuing/consuming pages from the
ptr_ring. Thus, we can allow placing pages from remote NUMA into theptr_ring, as the dequeue/consume step will check the NUMA node. All current drivers usingpage_poolwill alloc/refill RX-ring from same CPU running softirq/NAPI process.Drivers that control the nid explicitly, also use
page_pool_update_nidwhen changing nid runtime. To speed up transision to new nid the alloc cache is now flushed on nid changes. This force pages to come fromptr_ring, which does the appropate nid check.
Quoted 4 lines
Quoted 4 lines
For the
NUMA_NO_NODEcase, when a NIC IRQ is moved to another NUMA node, we accept that transitioning the alloc cache doesn't happen immediately. The preferred nid change runtime via consultingnuma_mem_id()based on the CPU processing RX-packets.
This is the section, where I say: "we accept that transitioning the alloc cache doesn't happen immediately. "
Quoted 9 lines
Quoted 9 lines
Notice, to avoid stressing the page buddy allocator and avoid doing too much work under softirq with preempt disabled, the NUMA check at
ptr_ringdequeue will break the refill cycle, when detecting a NUMA mismatch. This will cause a slower transition, but its done on purpose.Fixes d5394610b1ba ("page_pool: Don't recycle non-reusable pages")Reported-by Li RongQing <[email protected]>Reported-by Yunsheng Lin <[email protected]>Signed-off-by Jesper Dangaard Brouer <[email protected]>
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
PATCH v6 2/2 page_pool: help compiler remove code in case CONFIG_NUMA=n
In reply to Jesper Dangaard BrouerWhen kernel is compiled without NUMA support, then page_pool NUMA config setting (pool->p.nid) doesn't make any practical sense. The compiler cannot see that it can remove the code paths.
This patch avoids reading pool->p.nid setting in case of !CONFIG_NUMA, in allocation and numa check code, which helps compiler to see the optimisation potential. It leaves update code intact to keep API the same.
$ ./scripts/bloat-o-meter net/core/page_pool.o-numa-enabled \
net/core/page_pool.o-numa-disabled
add/remove: 0/0 grow/shrink: 0/3 up/down: 0/-113 (-113)
Function old new delta
page_pool_create 401 398 -3
__page_pool_alloc_pages_slow 439 426 -13
page_pool_refill_alloc_cache 425 328 -97
Total: Before=3611, After=3498, chg -3.13%--- net/core/page_pool.c | 9 +++++++++ 1 file changed, 9 insertions(+)
net/core/page_pool.c +9 -0
@@ -113,7 +113,12 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool,/* Softirq guarantee CPU and thus NUMA node is stable. This, * assumes CPU refilling driver RX-ring will also run RX-NAPI. */ +#ifdef CONFIG_NUMA pref_nid = (pool->p.nid ==NUMA_NO_NODE ) ?numa_mem_id () : pool->p.nid; +#else +/* Ignore pool->p.nid setting if !CONFIG_NUMA, helps compiler */ + pref_nid =numa_mem_id ();/* will be zero like page_to_nid() */ +#endif /* Slower-path: Get pages from locked ring queue */ spin_lock (&r->consumer_lock);
@@ -200,7 +205,11 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,*/ /* Cache was empty, do real allocation */ +#ifdef CONFIG_NUMA page =alloc_pages_node (pool->p.nid, gfp, pool->p.order); +#else + page =alloc_pages (gfp, pool->p.order); +#endif if (!page)return NULL ;
David Miller
maintainerFrom: Jesper Dangaard Brouer <[email protected]>
Date: Fri, 27 Dec 2019 18:13:13 +0100
Quoted 11 lines
The recently added NUMA changes (merged for v5.5) to
page_pool, it both contains a bug in handlingNUMA_NO_NODEcondition, and added code to the fast-path.This patchset fixes the bug and moves code out of fast-path. The first patch contains a fix that should be considered for 5.5. The second patch reduce code size and overhead in case
CONFIG_NUMAis disabled.Currently the
NUMA_NO_NODEsetting bug only affects driver 'ti_cpsw' (drivers/net/ethernet/ti/), but after this patchset, we plan to move other drivers (netsec and mvneta) to useNUMA_NO_NODEsetting.
Series applied to net-next with the "fallthrough" misspelling fixed in patch #1.
Thank you.
Thanks David
Quoted 11 lines · David Miller
From: Jesper Dangaard Brouer <[email protected]>
Date: Fri, 27 Dec 2019 18:13:13 +0100Quoted 8 lines
The recently added NUMA changes (merged for v5.5) to
page_pool, it both contains a bug in handlingNUMA_NO_NODEcondition, and added code to the fast-path.This patchset fixes the bug and moves code out of fast-path. The first patch contains a fix that should be considered for 5.5. The second patch reduce code size and overhead in case
CONFIG_NUMAis disabled.
Quoted 8 lines
Quoted 3 lines
Currently the
NUMA_NO_NODEsetting bug only affects driver 'ti_cpsw' (drivers/net/ethernet/ti/), but after this patchset, we plan to move other drivers (netsec and mvneta) to useNUMA_NO_NODEsetting.Series applied to net-next with the "fallthrough" misspelling fixed in patch #1.
Thank you.
I did review the patch and everything seemed fine, i was waiting Saeed to test it
in any case you can add my reviewed by if it's not too late