Page fragment updates
Subsystems (7)
v3 cover Page fragment updates
This patch series takes care of a few cleanups for the page fragments API.
First we do some renames so that things are much more consistent. First we move the page_frag_ portion of the name to the front of the functions names. Secondly we split out the cache specific functions from the other page fragment functions by adding the word "cache" to the name.
Finally I added a bit of documentation that will hopefully help to explain some of this. I plan to revisit this later as we get things more ironed out in the near future with the changes planned for the DMA setup to support eXpress Data Path.
---
v2: Fixed a comparison between a void* and 0 due to copy/paste from free_pages
v3: Updated first rename patch so that it is just a rename and doesn't impact
the actual functionality to avoid performance regression.
I'm submitting this to Intel Wired Lan and Jeff Kirsher's "next-queue" for acceptance as I have a series of other patches for igb that are blocked by by these patches since I had to rename the functionality fo draining extra references.
This series was going to be accepted for mmotm back when it was v1, however since then I found a few minor issues that needed to be fixed.
I am hoping to get an Acked-by from Andrew Morton for these patches and then have them submitted to David Miller as he has said he will accept them if I get the Acked-by. In the meantime if these can be applied to next-queue while waiting on that Acked-by then I can submit the other patches for igb and ixgbe for testing.
Alexander Duyck (3):
mm: Rename __alloc_page_frag to page_frag_alloc and __free_page_frag to page_frag_free
mm: Rename __page_frag functions to __page_frag_cache, drop order from drain
mm: Add documentation for page fragment APIs
Documentation/vm/page_frags | 42 +++++++++++++++++++++++++++++ drivers/net/ethernet/intel/igb/igb_main.c | 6 ++-- include/linux/gfp.h | 9 +++--- include/linux/skbuff.h | 2 + mm/page_alloc.c | 23 ++++++++-------- net/core/skbuff.c | 8 +++--- 6 files changed, 66 insertions(+), 24 deletions(-) create mode 100644 Documentation/vm/page_frags
--
PATCH v3 1/3 mm: Rename __alloc_page_frag to page_frag_alloc and __free_page_frag to page_frag_free
From: Alexander Duyck <[email protected]>
This patch renames the page frag functions to be more consistent with other APIs. Specifically we place the name page_frag first in the name and then have either an alloc or free call name that we append as the suffix. This makes it a bit clearer in terms of naming.
In addition we drop the leading double underscores since we are technically no longer a backing interface and instead the front end that is called from the networking APIs.
---
v2: Fixed a comparison between a void* and 0 due to copy/paste from free_pages
v3: Dropped changes to function and make this a rename patch only.
This fixes a small performance regression I saw in some tests.
include/linux/gfp.h | 2 +- include/linux/skbuff.h | 2 +- mm/page_alloc.c | 10 +++++----- net/core/skbuff.c | 8 ++++---- 4 files changed, 11 insertions(+), 11 deletions(-)
include/linux/gfp.h +1 -1
@@ -510,7 +510,7 @@ extern void __page_frag_drain(struct page *page, unsigned int order, unsigned int count);extern void *__alloc_page_frag (struct page_frag_cache *nc,unsigned int fragsz,gfp_t gfp_mask); -extern void __free_page_frag (void *addr); +extern void page_frag_free (void *addr);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0)
include/linux/skbuff.h +1 -1
@@ -2480,7 +2480,7 @@ static inline struct sk_buff *netdev_alloc_skb_ip_align(struct net_device *dev,static inline void skb_free_frag (void *addr) { -__free_page_frag (addr); +page_frag_free (addr); }void *napi_alloc_frag (unsigned int fragsz);
mm/page_alloc.c +5 -5
@@ -3939,8 +3939,8 @@ void __page_frag_drain(struct page *page, unsigned int order, }EXPORT_SYMBOL (__page_frag_drain); -void *__alloc_page_frag (struct page_frag_cache *nc, -unsigned int fragsz,gfp_t gfp_mask) +void *page_frag_alloc (struct page_frag_cache *nc, +unsigned int fragsz,gfp_t gfp_mask) {unsigned int size =PAGE_SIZE ;struct page *page;
@@ -3991,19 +3991,19 @@ void *__alloc_page_frag(struct page_frag_cache *nc,return nc->va + offset; } -EXPORT_SYMBOL (__alloc_page_frag); +EXPORT_SYMBOL (page_frag_alloc);/* * Frees a page fragment allocated out of either a compound or order 0 page. */ -void __free_page_frag (void *addr) +void page_frag_free (void *addr) {struct page *page =virt_to_head_page (addr);if (unlikely (put_page_testzero (page)))__free_pages_ok (page,compound_order (page)); } -EXPORT_SYMBOL (__free_page_frag); +EXPORT_SYMBOL (page_frag_free);static void *make_alloc_exact (unsigned long addr,unsigned int order,size_t size)
net/core/skbuff.c +4 -4
@@ -369,7 +369,7 @@ static void *__netdev_alloc_frag(unsigned int fragsz, gfp_t gfp_mask)local_irq_save (flags); nc =this_cpu_ptr (&netdev_alloc_cache); - data =__alloc_page_frag (nc, fragsz, gfp_mask); + data =page_frag_alloc (nc, fragsz, gfp_mask);local_irq_restore (flags);return data; }
@@ -391,7 +391,7 @@ static void *__napi_alloc_frag(unsigned int fragsz, gfp_t gfp_mask) {struct napi_alloc_cache *nc =this_cpu_ptr (&napi_alloc_cache); -return __alloc_page_frag (&nc->page, fragsz, gfp_mask); +return page_frag_alloc (&nc->page, fragsz, gfp_mask); }void *napi_alloc_frag (unsigned int fragsz)
@@ -441,7 +441,7 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len,local_irq_save (flags); nc =this_cpu_ptr (&netdev_alloc_cache); - data =__alloc_page_frag (nc, len, gfp_mask); + data =page_frag_alloc (nc, len, gfp_mask); pfmemalloc = nc->pfmemalloc;local_irq_restore (flags);
@@ -505,7 +505,7 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len,if (sk_memalloc_socks ()) gfp_mask |= __GFP_MEMALLOC; - data =__alloc_page_frag (&nc->page, len, gfp_mask); + data =page_frag_alloc (&nc->page, len, gfp_mask);if (unlikely (!data))return NULL ;
kbuild test robot
build botRe: [Intel-wired-lan] [next PATCH v3 1/3] mm: Rename __alloc_page_frag to page_frag_alloc and __free_page_frag to page_frag_free
reproduce:
Hi Alexander,
[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10-rc2 next-20170103]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Alexander-Duyck/Page-fragment-updates/20170104-080239 base: git://git.cmpxchg.org/linux-mmotm.git master config: i386-randconfig-x003-201701 (attached as .config) compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 reproduce: # save the attached .config to linux build tree make ARCH=i386
Note: the linux-review/Alexander-Duyck/Page-fragment-updates/20170104-080239 HEAD 81fa8f8a69c1b7187ae789eb536cdfcb3263d366 builds fine.
It only hurts bisectibility.
All error/warnings (new ones prefixed by >>):
net/core/skbuff.c: In function '__netdev_alloc_frag':
>> net/core/skbuff.c:372:9: error: implicit declaration of function 'page_frag_alloc' [-Werror=implicit-function-declaration]
data = page_frag_alloc(nc, fragsz, gfp_mask);
^~~~~~~~~~~~~~~
>> net/core/skbuff.c:372:7: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
data = page_frag_alloc(nc, fragsz, gfp_mask);
^
net/core/skbuff.c: In function '__napi_alloc_frag':
>> net/core/skbuff.c:394:9: warning: return makes pointer from integer without a cast [-Wint-conversion]
return page_frag_alloc(&nc->page, fragsz, gfp_mask);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
net/core/skbuff.c: In function '__netdev_alloc_skb':
net/core/skbuff.c:444:7: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
data = page_frag_alloc(nc, len, gfp_mask);
^
net/core/skbuff.c: In function '__napi_alloc_skb':
net/core/skbuff.c:508:7: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
data = page_frag_alloc(&nc->page, len, gfp_mask);
^cc1: some warnings being treated as errors
vim +/page_frag_alloc +372 net/core/skbuff.c
366 struct page_frag_cache *nc;
367 unsigned long flags;
368 void *data;
369
370 local_irq_save(flags);
371 nc = this_cpu_ptr(&netdev_alloc_cache);
> 372 data = page_frag_alloc(nc, fragsz, gfp_mask);
373 local_irq_restore(flags);
374 return data;
375 }
376
377 /**
378 * netdev_alloc_frag - allocate a page fragment
379 * @fragsz: fragment size
380 *
381 * Allocates a frag from a page for receive buffer.
382 * Uses GFP_ATOMIC allocations.
383 */
384 void *netdev_alloc_frag(unsigned int fragsz)
385 {
386 return __netdev_alloc_frag(fragsz, GFP_ATOMIC | __GFP_COLD);
387 }
388 EXPORT_SYMBOL(netdev_alloc_frag);
389
390 static void *__napi_alloc_frag(unsigned int fragsz, gfp_t gfp_mask)
391 {
392 struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache);
393
> 394 return page_frag_alloc(&nc->page, fragsz, gfp_mask); 395 }
396
397 void *napi_alloc_frag(unsigned int fragsz)
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 25575 bytes
Desc: not available
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20170104/905cc418/attachment-0001.bin>
PATCH v3 2/3 mm: Rename __page_frag functions to __page_frag_cache, drop order from drain
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This patch does two things.
First it goes through and renames the __page_frag prefixed functions to __page_frag_cache so that we can be clear that we are draining or refilling the cache, not the frags themselves.
Second we drop the order parameter from __page_frag_cache_drain since we don't actually need to pass it since all fragments are either order 0 or must be a compound page.
---
v2,v3: No change
drivers/net/ethernet/intel/igb/igb_main.c | 6 +++--- include/linux/gfp.h | 7 +++---- mm/page_alloc.c | 13 +++++++------ 3 files changed, 13 insertions(+), 13 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +3 -3
@@ -3964,8 +3964,8 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring)PAGE_SIZE ,DMA_FROM_DEVICE , DMA_ATTR_SKIP_CPU_SYNC); -__page_frag_drain (buffer_info->page,0 , - buffer_info->pagecnt_bias); +__page_frag_cache_drain (buffer_info->page, + buffer_info->pagecnt_bias); buffer_info->page =NULL ; }
@@ -6993,7 +6993,7 @@ static struct sk_buff *igb_fetch_rx_buffer(struct igb_ring *rx_ring,dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma,PAGE_SIZE ,DMA_FROM_DEVICE ,DMA_ATTR_SKIP_CPU_SYNC ); -__page_frag_drain (page,0 , rx_buffer->pagecnt_bias); +__page_frag_cache_drain (page, rx_buffer->pagecnt_bias); }/* clear contents of rx_buffer */
include/linux/gfp.h +3 -4
@@ -506,10 +506,9 @@ extern struct page *alloc_pages_vma(gfp_t gfp_mask, int order,extern void free_hot_cold_page_list (struct list_head *list,bool cold);struct page_frag_cache ; -extern void __page_frag_drain (struct page *page,unsigned int order, -unsigned int count); -extern void *__alloc_page_frag (struct page_frag_cache *nc, -unsigned int fragsz,gfp_t gfp_mask); +extern void __page_frag_cache_drain (struct page *page,unsigned int count); +extern void *page_frag_alloc (struct page_frag_cache *nc, +unsigned int fragsz,gfp_t gfp_mask);extern void page_frag_free (void *addr);#define __free_page (page) __free_pages((page), 0)
mm/page_alloc.c +7 -6
@@ -3904,8 +3904,8 @@ void free_pages(unsigned long addr, unsigned int order)* drivers to provide a backing region of memory for use as either an * sk_buff->head, or to be used in the "frags" portion of skb_shared_info. */ -static struct page *__page_frag_refill (struct page_frag_cache *nc, -gfp_t gfp_mask) +static struct page *__page_frag_cache_refill (struct page_frag_cache *nc, +gfp_t gfp_mask) {struct page *page =NULL ;gfp_t gfp = gfp_mask;
@@ -3925,19 +3925,20 @@ static struct page *__page_frag_refill(struct page_frag_cache *nc,return page; } -void __page_frag_drain (struct page *page,unsigned int order, -unsigned int count) +void __page_frag_cache_drain (struct page *page,unsigned int count) {VM_BUG_ON_PAGE (page_ref_count (page) ==0 , page);if (page_ref_sub_and_test (page, count)) { +unsigned int order =compound_order (page); +if (order ==0 )free_hot_cold_page (page, false);else __free_pages_ok (page, order); } } -EXPORT_SYMBOL (__page_frag_drain); +EXPORT_SYMBOL (__page_frag_cache_drain);void *page_frag_alloc (struct page_frag_cache *nc,unsigned int fragsz,gfp_t gfp_mask)
@@ -3948,7 +3949,7 @@ void *page_frag_alloc(struct page_frag_cache *nc,if (unlikely (!nc->va)) { refill: - page =__page_frag_refill (nc, gfp_mask); + page =__page_frag_cache_refill (nc, gfp_mask);if (!page)return NULL ;
PATCH v3 3/3 mm: Add documentation for page fragment APIs
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This is a first pass at trying to add documentation for the page_frag APIs. They may still change over time but for now I thought I would try to get these documented so that as more network drivers and stack calls make use of them we have one central spot to document how they are meant to be used.
---
v2,v3: No change
Documentation/vm/page_frags | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 Documentation/vm/page_frags
Documentation/vm/page_frags new file +42 -0
@@ -0,0 +1,42 @@ +Page fragments +-------------- + +A page fragment is an arbitrary-length arbitrary-offset area of memory +which resides within a 0 or higher order compound page. Multiple +fragments within that page are individually refcounted, in the page's +reference counter. + +The page_frag functions, page_frag_alloc and page_frag_free, provide a +simple allocation framework for page fragments. This is used by the +network stack and network device drivers to provide a backing region of +memory for use as either an sk_buff->head, or to be used in the "frags" +portion of skb_shared_info. + +In order to make use of the page fragment APIs a backing page fragment +cache is needed. This provides a central point for the fragment allocation +and tracks allows multiple calls to make use of a cached page. The +advantage to doing this is that multiple calls to get_page can be avoided +which can be expensive at allocation time. However due to the nature of +this caching it is required that any calls to the cache be protected by +either a per-cpu limitation, or a per-cpu limitation and forcing interrupts +to be disabled when executing the fragment allocation. + +The network stack uses two separate caches per CPU to handle fragment +allocation. The netdev_alloc_cache is used by callers making use of the +__netdev_alloc_frag and __netdev_alloc_skb calls. The napi_alloc_cache is +used by callers of the __napi_alloc_frag and __napi_alloc_skb calls. The +main difference between these two calls is the context in which they may be +called. The "netdev" prefixed functions are usable in any context as these +functions will disable interrupts, while the "napi" prefixed functions are +only usable within the softirq context. + +Many network device drivers use a similar methodology for allocating page +fragments, but the page fragments are cached at the ring or descriptor +level. In order to enable these cases it is necessary to provide a generic +way of tearing down a page cache. For this reason __page_frag_cache_drain +was implemented. It allows for freeing multiple references from a single +page via a single call. The advantage to doing this is that it allows for +cleaning up the multiple references that were added to a page in order to +avoid calling get_page per allocation. + +Alexander Duyck, Nov 29, 2016.