page_pool: fix logic in __page_pool_get_cached
Subsystems (3)
PATCH 1/1 page_pool: fix logic in __page_pool_get_cached
__page_pool_get_cached() will return NULL when the ring is empty, even if there are pages present in the lookaside cache.
It is also possible to refill the cache, and then return a NULL page.
Restructure the logic so eliminate both cases.
--- net/core/page_pool.c | 39 ++++++++++++++++----------------------- 1 file changed, 16 insertions(+), 23 deletions(-)
net/core/page_pool.c +16 -23
@@ -82,12 +82,9 @@ EXPORT_SYMBOL(page_pool_create);static struct page *__page_pool_get_cached (struct page_pool *pool) {struct ptr_ring *r = &pool->ring; +bool refill = false;struct page *page; -/* Quicker fallback, avoid locks when ring is empty */ -if (__ptr_ring_empty (r)) -return NULL ; -/* Test for safe-context, caller should provide this guarantee */ if (likely (in_serving_softirq ())) {if (likely (pool->alloc.count)) {
@@ -95,27 +92,23 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) page = pool->alloc.cache[--pool->alloc.count];return page; } -/* Slower-path: Alloc array empty, time to refill -* -* Open-coded bulk ptr_ring consumer. -* -* Discussion: the ring consumer lock is not really -* needed due to the softirq/NAPI protection, but -* later need the ability to reclaim pages on the -* ring. Thus, keeping the locks. -*/ -spin_lock (&r->consumer_lock); -while ((page =__ptr_ring_consume (r))) { -if (pool->alloc.count ==PP_ALLOC_CACHE_REFILL ) -break ; - pool->alloc.cache[pool->alloc.count++] = page; - } -spin_unlock (&r->consumer_lock); -return page; + refill = true; } -/* Slow-path: Get page from locked ring queue */ - page =ptr_ring_consume (&pool->ring); +/* Quicker fallback, avoid locks when ring is empty */ +if (__ptr_ring_empty (r)) +return NULL ; + +/* Slow-path: Get page from locked ring queue, +* refill alloc array if requested. +*/ +spin_lock (&r->consumer_lock); + page =__ptr_ring_consume (r); +if (refill) + pool->alloc.count =__ptr_ring_consume_batched (r, + pool->alloc.cache, +PP_ALLOC_CACHE_REFILL ); +spin_unlock (&r->consumer_lock);return page; }
Signature · 1 line
2.17.1
Quoted 9 lines · On Tue, 13 Aug 2019 10:45:09 -0700 Jonathan Lemon <[email protected]>…
__page_pool_get_cached()will return NULL when the ring is empty, even if there are pages present in the lookaside cache.It is also possible to refill the cache, and then return a NULL page.
Restructure the logic so eliminate both cases.
Signed-off-by Jonathan Lemon <[email protected]>
Thanks for catching and improving this!
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Hi Jonathan,
Thanks!
Quoted 7 lines · Jonathan Lemon
__page_pool_get_cached()will return NULL when the ring is empty, even if there are pages present in the lookaside cache.It is also possible to refill the cache, and then return a NULL page.
Restructure the logic so eliminate both cases.
Quoted 70 lines
Signed-off-by Jonathan Lemon <[email protected]>--- net/core/page_pool.c | 39 ++++++++++++++++----------------------- 1 file changed, 16 insertions(+), 23 deletions(-)net/core/page_pool.c +16
-23@@ -82,12 +82,9 @@ EXPORT_SYMBOL(page_pool_create);static struct page *__page_pool_get_cached (struct page_pool *pool) {struct ptr_ring *r = &pool->ring; +bool refill = false;struct page *page; -/* Quicker fallback, avoid locks when ring is empty */ -if (__ptr_ring_empty (r)) -return NULL ; -/* Test for safe-context, caller should provide this guarantee */ if (likely (in_serving_softirq ())) {if (likely (pool->alloc.count)) {@@ -95,27 +92,23 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) page = pool->alloc.cache[--pool->alloc.count];return page; } -/* Slower-path: Alloc array empty, time to refill -* -* Open-coded bulk ptr_ring consumer. -* -* Discussion: the ring consumer lock is not really -* needed due to the softirq/NAPI protection, but -* later need the ability to reclaim pages on the -* ring. Thus, keeping the locks. -*/ -spin_lock (&r->consumer_lock); -while ((page =__ptr_ring_consume (r))) { -if (pool->alloc.count ==PP_ALLOC_CACHE_REFILL ) -break ; - pool->alloc.cache[pool->alloc.count++] = page; - } -spin_unlock (&r->consumer_lock); -return page; + refill = true; } -/* Slow-path: Get page from locked ring queue */ - page =ptr_ring_consume (&pool->ring); +/* Quicker fallback, avoid locks when ring is empty */ +if (__ptr_ring_empty (r)) +return NULL ; + +/* Slow-path: Get page from locked ring queue, +* refill alloc array if requested. +*/ +spin_lock (&r->consumer_lock); + page =__ptr_ring_consume (r); +if (refill) + pool->alloc.count =__ptr_ring_consume_batched (r, + pool->alloc.cache, +PP_ALLOC_CACHE_REFILL ); +spin_unlock (&r->consumer_lock);return page; }Signature · 1 line
2.17.1
From: Jonathan Lemon <[email protected]>
Date: Tue, 13 Aug 2019 10:45:09 -0700
Quoted 9 lines
__page_pool_get_cached()will return NULL when the ring is empty, even if there are pages present in the lookaside cache.It is also possible to refill the cache, and then return a NULL page.
Restructure the logic so eliminate both cases.
Signed-off-by Jonathan Lemon <[email protected]>
Applied.