Add support for writable pages and build_skb
Subsystems (2)
v2 cover Add support for writable pages and build_skb
This patch set enables support for using the recent changes that allow for unmapping pages without invalidating their contents via DMA_ATTR_SKIP_CPU_SYNC. With this change DMA pages should be writable and as a result we should be able to make use of build_skb which can be used to drop the skb->head memory allocation, header parsing, and memcpy from the receive path which can greatly help to improve performance.
My main concern at this point is that there might be an architecture where I didn't get DMA_ATTR_SKIP_CPU_SYNC implemented that might still need it. For that reason I have also added a ethtool private flag called out as "legacy-rx". If a platform encounters an issue where the Rx can possibly corrupt data it can be enbled by running:
ethtool --set-priv-flags DEVNAME legacy-rx on
The testing matrix for all of these patches is going to be pretty extensive. Basically we want to test these patches on as many platforms and architectures as possible with as many features being toggled as possible including RSC, FCoE, SR-IOV, and Jumbo Frames all while receiving traffic.
Within the patches there is also some intialization changes. Specifically I have updated the code paths to defer clearing the rings until we are about to initialize them and give them to hardware. By doing this we are able to avoid having to dirty memory we don't need to which should help to improve suspend/resume times for when we start looking at possibly using the suspend/resume approach for migration of interface in VMs.
v2: Fixes for build issues on non-x86 architectures
Fixes for issues ethtool issues found during testing
---
Alexander Duyck (11):
ixgbe: Add function for checking to see if we can reuse page
ixgbe: Only DMA sync frame length
ixgbe: Update driver to make use of DMA attributes in Rx path
ixgbe: Update code to better handle incrementing page count
ixgbe: Make use of order 1 pages and 3K buffers independent of FCoE
ixgbe: Use length to determine if descriptor is done
ixgbe: Break out Rx buffer page management
ixgbe: Add support for padding packet
ixgbe: Add private flag to control buffer mode
ixgbe: Add support for build_skb
ixgbe: Don't bother clearing buffer memory for descriptor rings
drivers/net/ethernet/intel/ixgbe/ixgbe.h | 45 +- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 60 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 585 ++++++++++++++-------- 3 files changed, 467 insertions(+), 223 deletions(-)
--
PATCH v2 1/11 ixgbe: Add function for checking to see if we can reuse page
From: Alexander Duyck <[email protected]>
This patch consolidates the code for the ixgbe driver so that it is more inline with what is already in igb. The general idea is to just consolidate functions that represent logical steps in the Rx process so we can later update them more easily.
---
v2: Fix missing rx_ring parameter for PAGE_SIZE > 4K
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 71 +++++++++++++++---------- 1 file changed, 42 insertions(+), 29 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +42 -29
@@ -1947,6 +1947,42 @@ static inline bool ixgbe_page_is_reserved(struct page *page)return (page_to_nid (page) !=numa_mem_id ()) ||page_is_pfmemalloc (page); } +static bool ixgbe_can_reuse_rx_page (struct ixgbe_ring *rx_ring, +struct ixgbe_rx_buffer *rx_buffer, +struct page *page, +const unsigned int truesize) +{ +#if (PAGE_SIZE >=8192 ) +unsigned int last_offset =ixgbe_rx_pg_size (rx_ring) - +ixgbe_rx_bufsz (rx_ring); +#endif +/* avoid re-using remote pages */ +if (unlikely (ixgbe_page_is_reserved (page))) +return false; + +#if (PAGE_SIZE <8192 ) +/* if we are only owner of page we can reuse it */ +if (unlikely (page_count (page) !=1 )) +return false; + +/* flip page offset to other buffer */ + rx_buffer->page_offset ^= truesize; +#else +/* move offset up to the next cache line */ + rx_buffer->page_offset += truesize; + +if (rx_buffer->page_offset > last_offset) +return false; +#endif + +/* Even if we own the page, we are not allowed to use atomic_set() +* This would break get_page_unless_zero() users. +*/ +page_ref_inc (page); + +return true; +} +/** * ixgbe_add_rx_frag - Add contents of Rx buffer to sk_buff * @rx_ring: rx descriptor ring to transact packets on
@@ -1968,18 +2004,18 @@ static bool ixgbe_add_rx_frag(struct ixgbe_ring *rx_ring, struct sk_buff *skb) {struct page *page = rx_buffer->page; +unsigned char *va =page_address (page) + rx_buffer->page_offset;unsigned int size =le16_to_cpu (rx_desc->wb.upper.length);#if (PAGE_SIZE <8192 )unsigned int truesize =ixgbe_rx_bufsz (rx_ring);#else unsigned int truesize =ALIGN (size,L1_CACHE_BYTES ); -unsigned int last_offset =ixgbe_rx_pg_size (rx_ring) - -ixgbe_rx_bufsz (rx_ring);#endif -if ((size <=IXGBE_RX_HDR_SIZE ) && !skb_is_nonlinear (skb)) { -unsigned char *va =page_address (page) + rx_buffer->page_offset; +if (unlikely (skb_is_nonlinear (skb))) + goto add_tail_frag; +if (size <=IXGBE_RX_HDR_SIZE ) {memcpy (__skb_put (skb, size), va,ALIGN (size,sizeof (long )));/* page is not reserved, we can reuse buffer as-is */
@@ -1991,34 +2027,11 @@ static bool ixgbe_add_rx_frag(struct ixgbe_ring *rx_ring,return false; } +add_tail_frag:skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, page, rx_buffer->page_offset, size, truesize); -/* avoid re-using remote pages */ -if (unlikely (ixgbe_page_is_reserved (page))) -return false; - -#if (PAGE_SIZE <8192 ) -/* if we are only owner of page we can reuse it */ -if (unlikely (page_count (page) !=1 )) -return false; - -/* flip page offset to other buffer */ - rx_buffer->page_offset ^= truesize; -#else -/* move offset up to the next cache line */ - rx_buffer->page_offset += truesize; - -if (rx_buffer->page_offset > last_offset) -return false; -#endif - -/* Even if we own the page, we are not allowed to use atomic_set() -* This would break get_page_unless_zero() users. -*/ -page_ref_inc (page); - -return true; +return ixgbe_can_reuse_rx_page (rx_ring, rx_buffer, page, truesize); } static struct sk_buff *ixgbe_fetch_rx_buffer(struct ixgbe_ring *rx_ring,
Bowers
PATCH v2 1/11 ixgbe: Add function for checking to see if we can reuse page
Quoted 16 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:36 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 01/11] ixgbe: Add function for checking to see if we can reuse pageFrom: Alexander Duyck <[email protected]>
This patch consolidates the code for the ixgbe driver so that it is more inline with what is already in igb. The general idea is to just consolidate functions that represent logical steps in the Rx process so we can later update them more easily.
Quoted 8 lines
Signed-off-by Alexander Duyck <[email protected]>---
v2: Fix missing
rx_ringparameter forPAGE_SIZE> 4Kdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 71 +++++++++++++++---- ------ 1 file changed, 42 insertions(+), 29 deletions(-)
PATCH v2 2/11 ixgbe: Only DMA sync frame length
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
On some platforms, syncing a buffer for DMA is expensive. Rather than sync the whole 2K receive buffer, only synchronise the length of the frame, which will typically be the MTU, or a much smaller TCP ACK.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +5 -5
@@ -1858,7 +1858,7 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,dma_sync_single_range_for_cpu (rx_ring->dev,IXGBE_CB (skb)->dma, frag->page_offset, -ixgbe_rx_bufsz (rx_ring), +skb_frag_size (frag),DMA_FROM_DEVICE ); }IXGBE_CB (skb)->dma =0 ;
@@ -2000,12 +2000,11 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,**/ static bool ixgbe_add_rx_frag (struct ixgbe_ring *rx_ring,struct ixgbe_rx_buffer *rx_buffer, -union ixgbe_adv_rx_desc *rx_desc, +unsigned int size,struct sk_buff *skb) {struct page *page = rx_buffer->page;unsigned char *va =page_address (page) + rx_buffer->page_offset; -unsigned int size =le16_to_cpu (rx_desc->wb.upper.length);#if (PAGE_SIZE <8192 )unsigned int truesize =ixgbe_rx_bufsz (rx_ring);#else
@@ -2037,6 +2036,7 @@ static bool ixgbe_add_rx_frag(struct ixgbe_ring *rx_ring,static struct sk_buff *ixgbe_fetch_rx_buffer (struct ixgbe_ring *rx_ring,union ixgbe_adv_rx_desc *rx_desc) { +unsigned int size =le16_to_cpu (rx_desc->wb.upper.length);struct ixgbe_rx_buffer *rx_buffer;struct sk_buff *skb;struct page *page;
@@ -2091,14 +2091,14 @@ static struct sk_buff *ixgbe_fetch_rx_buffer(struct ixgbe_ring *rx_ring,dma_sync_single_range_for_cpu (rx_ring->dev, rx_buffer->dma, rx_buffer->page_offset, -ixgbe_rx_bufsz (rx_ring), + size,DMA_FROM_DEVICE ); rx_buffer->skb =NULL ; }/* pull page into skb */ -if (ixgbe_add_rx_frag (rx_ring, rx_buffer, rx_desc, skb)) { +if (ixgbe_add_rx_frag (rx_ring, rx_buffer, size, skb)) {/* hand second half of page back to the ring */ ixgbe_reuse_rx_page (rx_ring, rx_buffer); }else if (IXGBE_CB (skb)->dma == rx_buffer->dma) {
Bowers
PATCH v2 2/11 ixgbe: Only DMA sync frame length
Quoted 11 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:36 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 02/11] ixgbe: Only DMA sync frame lengthFrom: Alexander Duyck <[email protected]>
Quoted 8 lines
On some platforms, syncing a buffer for DMA is expensive. Rather than sync the whole 2K receive buffer, only synchronise the length of the frame, which will typically be the MTU, or a much smaller TCP ACK.
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
PATCH v2 3/11 ixgbe: Update driver to make use of DMA attributes in Rx path
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This patch adds support for DMA_ATTR_SKIP_CPU_SYNC and DMA_ATTR_WEAK_ORDERING. By enabling both of these for the Rx path we are able to see performance improvements on architectures that implement either one due to the fact that page mapping and unmapping only has to sync what is actually being used instead of the entire buffer. In addition by enabling the weak ordering attribute enables a performance improvement for architectures that can associate a memory ordering with a DMA buffer such as Sparc.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 3 + drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 56 +++++++++++++++++-------- 2 files changed, 40 insertions(+), 19 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe.h +3 -0
@@ -107,6 +107,9 @@/* How many Rx Buffers do we bundle into one write to the hardware ? */ #define IXGBE_RX_BUFFER_WRITE 16/* Must be power of 2 */ +#define IXGBE_RX_DMA_ATTR \ + (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING) +enum ixgbe_tx_flags {/* cmd_type flags */ IXGBE_TX_FLAGS_HW_VLAN =0x01 ,
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +37 -19
98 lines not on this page · raw message
Bowers
PATCH v2 3/11 ixgbe: Update driver to make use of DMA attributes in Rx path
Quoted 20 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:36 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 03/11] ixgbe: Update driver to make use of DMA attributes in Rx pathFrom: Alexander Duyck <[email protected]>
This patch adds support for
DMA_ATTR_SKIP_CPU_SYNCandDMA_ATTR_WEAK_ORDERING. By enabling both of these for the Rx path we are able to see performance improvements on architectures that implement either one due to the fact that page mapping and unmapping only has to sync what is actually being used instead of the entire buffer. In addition by enabling the weak ordering attribute enables a performance improvement for architectures that can associate a memory ordering with a DMA buffer such as Sparc.
Quoted 6 lines
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 3 + drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 56 +++++++++++++++++- ------- 2 files changed, 40 insertions(+), 19 deletions(-)
PATCH v2 4/11 ixgbe: Update code to better handle incrementing page count
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
Batch the page count updates instead of doing them one at a time. By doing this we can improve the overall performance as the atomic increment operations can be expensive due to the fact that on x86 they are locked operations which can cause stalls. By doing bulk updates we can consolidate the stall which should help to improve the overall receive performance.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 7 ++++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 39 ++++++++++++++++--------- 2 files changed, 31 insertions(+), 15 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe.h +6 -1
@@ -198,7 +198,12 @@ struct ixgbe_rx_buffer {struct sk_buff *skb;dma_addr_t dma;struct page *page; -unsigned int page_offset; +#if (BITS_PER_LONG >32 ) || (PAGE_SIZE >=65536 ) +__u32 page_offset; +#else +__u16 page_offset; +#endif +__u16 pagecnt_bias; };struct ixgbe_queue_stats {
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +25 -14
@@ -1602,6 +1602,7 @@ static bool ixgbe_alloc_mapped_page(struct ixgbe_ring *rx_ring, bi->dma = dma; bi->page = page; bi->page_offset =0 ; + bi->pagecnt_bias =1 ;return true; }
@@ -1960,13 +1961,15 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,unsigned int last_offset =ixgbe_rx_pg_size (rx_ring) -ixgbe_rx_bufsz (rx_ring);#endif +unsigned int pagecnt_bias = rx_buffer->pagecnt_bias--; +/* avoid re-using remote pages */ if (unlikely (ixgbe_page_is_reserved (page)))return false;#if (PAGE_SIZE <8192 )/* if we are only owner of page we can reuse it */ -if (unlikely (page_count (page) !=1 )) +if (unlikely (page_count (page) != pagecnt_bias))return false;/* flip page offset to other buffer */
@@ -1979,10 +1982,14 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,return false;#endif -/* Even if we own the page, we are not allowed to use atomic_set() -* This would break get_page_unless_zero() users. +/* If we have drained the page fragment pool we need to update +* the pagecnt_bias and page count so that we fully restock the +* number of references the driver holds. */ -page_ref_inc (page); +if (unlikely (pagecnt_bias ==1 )) { +page_ref_add (page,USHRT_MAX ); + rx_buffer->pagecnt_bias =USHRT_MAX ; + }return true; }
@@ -2026,7 +2033,6 @@ static bool ixgbe_add_rx_frag(struct ixgbe_ring *rx_ring,return true;/* this page cannot be reused so discard it */ -__free_pages (page,ixgbe_rx_pg_order (rx_ring));return false; }
@@ -2105,15 +2111,19 @@ static struct sk_buff *ixgbe_fetch_rx_buffer(struct ixgbe_ring *rx_ring,if (ixgbe_add_rx_frag (rx_ring, rx_buffer, size, skb)) {/* hand second half of page back to the ring */ ixgbe_reuse_rx_page (rx_ring, rx_buffer); - }else if (IXGBE_CB (skb)->dma == rx_buffer->dma) { -/* the page has been released from the ring */ -IXGBE_CB (skb)->page_released = true; }else { -/* we are not reusing the buffer so unmap it */ -dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, -ixgbe_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IXGBE_RX_DMA_ATTR ); +if (IXGBE_CB (skb)->dma == rx_buffer->dma) { +/* the page has been released from the ring */ +IXGBE_CB (skb)->page_released = true; + }else { +/* we are not reusing the buffer so unmap it */ +dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, +ixgbe_rx_pg_size (rx_ring), +DMA_FROM_DEVICE , +IXGBE_RX_DMA_ATTR ); + } +__page_frag_cache_drain (page, + rx_buffer->pagecnt_bias); }/* clear contents of buffer_info */
@@ -4973,7 +4983,8 @@ static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring)ixgbe_rx_pg_size (rx_ring),DMA_FROM_DEVICE , IXGBE_RX_DMA_ATTR); -__free_pages (rx_buffer->page,ixgbe_rx_pg_order (rx_ring)); +__page_frag_cache_drain (rx_buffer->page, + rx_buffer->pagecnt_bias); rx_buffer->page =NULL ; }
Bowers
PATCH v2 4/11 ixgbe: Update code to better handle incrementing page count
Quoted 17 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:36 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 04/11] ixgbe: Update code to better handle incrementing page countFrom: Alexander Duyck <[email protected]>
Batch the page count updates instead of doing them one at a time. By doing this we can improve the overall performance as the atomic increment operations can be expensive due to the fact that on x86 they are locked operations which can cause stalls. By doing bulk updates we can consolidate the stall which should help to improve the overall receive performance.
Quoted 6 lines
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 7 ++++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 39 ++++++++++++++++-- ------- 2 files changed, 31 insertions(+), 15 deletions(-)
PATCH v2 4/11 ixgbe: Update code to better handle incrementing page count
In reply to Alexander DuyckQuoted 9 lines · Alexander Duyck
From: Alexander Duyck <[email protected]>
Batch the page count updates instead of doing them one at a time. By doing this we can improve the overall performance as the atomic increment operations can be expensive due to the fact that on x86 they are locked operations which can cause stalls. By doing bulk updates we can consolidate the stall which should help to improve the overall receive performance.
[...]
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static bool ixgbe_alloc_mapped_page(struct ixgbe_ring *rx_ring, · line 1605
Quoted 10 lines
struct
ixgbe_queue_stats{drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +1
-0@@ -1602,6 +1602,7 @@ static bool ixgbe_alloc_mapped_page(struct ixgbe_ring *rx_ring, bi->dma = dma; bi->page = page; bi->page_offset =0 ; + bi->pagecnt_bias =1 ;
I'm curious why set pagecnt_bias to '1' here but in the ixgbe_can_reuse_rx_page path set it to USHRT_MAX? Any reason not to just always use the USHRT_MAX scheme.
Quoted 28 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +5
-3return true; }@@ -1960,13 +1961,15 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,unsigned int last_offset =ixgbe_rx_pg_size (rx_ring) -ixgbe_rx_bufsz (rx_ring);#endif +unsigned int pagecnt_bias = rx_buffer->pagecnt_bias--; +/* avoid re-using remote pages */ if (unlikely (ixgbe_page_is_reserved (page)))return false;#if (PAGE_SIZE <8192 )/* if we are only owner of page we can reuse it */ -if (unlikely (page_count (page) !=1 )) +if (unlikely (page_count (page) != pagecnt_bias))return false;/* flip page offset to other buffer */ @@ -1979,10 +1982,14 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,return false;#endif -/* Even if we own the page, we are not allowed to use atomic_set() -* This would break get_page_unless_zero() users. +/* If we have drained the page fragment pool we need to update +* the pagecnt_bias and page count so that we fully restock the
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring, · line 1995
Quoted 10 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +5
-1+* number of references the driver holds. */ -page_ref_inc (page); +if (unlikely (pagecnt_bias ==1 )) { +page_ref_add (page,USHRT_MAX ); + rx_buffer->pagecnt_bias =USHRT_MAX ; + }return true; }
Otherwise patch looks good to me. Although it took me a moment to catch the ref_add/pagecnt_bias usage here.
PATCH v2 4/11 ixgbe: Update code to better handle incrementing page count
Quoted 13 lines · John Fastabend
Quoted 9 lines · Alexander Duyck
From: Alexander Duyck <[email protected]>
Batch the page count updates instead of doing them one at a time. By doing this we can improve the overall performance as the atomic increment operations can be expensive due to the fact that on x86 they are locked operations which can cause stalls. By doing bulk updates we can consolidate the stall which should help to improve the overall receive performance.
[...]
Quoted 14 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static bool ixgbe_alloc_mapped_page(struct ixgbe_ring *rx_ring, · line 1605
Quoted 10 lines
struct
ixgbe_queue_stats{drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +1
-0@@ -1602,6 +1602,7 @@ static bool ixgbe_alloc_mapped_page(struct ixgbe_ring *rx_ring, bi->dma = dma; bi->page = page; bi->page_offset =0 ; + bi->pagecnt_bias =1 ;I'm curious why set
pagecnt_biasto '1' here but in theixgbe_can_reuse_rx_pagepath set it toUSHRT_MAX? Any reason not to just always use theUSHRT_MAXscheme.
Because I wanted to defer having to use page_ref_add until I know I am going to reuse the page. The allocation can be from a memory pool that is reserved for emergency allocations or could be from the wrong node. In such a case we would take a penalty if we are having to perform an atomic add and subtract from the page count. By setting it to 1 we reflect that for now we hold the only reference to the page and we are waiting to see if it is reusable.
Quoted 36 lines
Quoted 36 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +10
-4return true; }@@ -1960,13 +1961,15 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,unsigned int last_offset =ixgbe_rx_pg_size (rx_ring) -ixgbe_rx_bufsz (rx_ring);#endif +unsigned int pagecnt_bias = rx_buffer->pagecnt_bias--; +/* avoid re-using remote pages */ if (unlikely (ixgbe_page_is_reserved (page)))return false;#if (PAGE_SIZE <8192 )/* if we are only owner of page we can reuse it */ -if (unlikely (page_count (page) !=1 )) +if (unlikely (page_count (page) != pagecnt_bias))return false;/* flip page offset to other buffer */ @@ -1979,10 +1982,14 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,return false;#endif -/* Even if we own the page, we are not allowed to use atomic_set() -* This would break get_page_unless_zero() users. +/* If we have drained the page fragment pool we need to update +* the pagecnt_bias and page count so that we fully restock the +* number of references the driver holds. */ -page_ref_inc (page); +if (unlikely (pagecnt_bias ==1 )) { +page_ref_add (page,USHRT_MAX ); + rx_buffer->pagecnt_bias =USHRT_MAX ; + }
Quoted 5 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring, · line 1995
Quoted 2 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +0
-0return true; }Otherwise patch looks good to me. Although it took me a moment to catch the ref_add/pagecnt_bias usage here.
Right. What we are doing here is dealing with the first real priming of the pagecnt_bias. We had set it to one when the page count was one, now we are forking things a bit and giving the stack one instance of the reference count and filling the pagecnt_bias to the maximum value supported assuming a u32 system. The value for pagecnt_bias will always shrink from this point. It might bounce by 1 if we are doing a copy-break but it should never exceed USHRT_MAX.
The advantage of doing it here is that this is after the page_is_reserved check so we are are essentially just taking this hit in place of having to do another page allocation versus adding it onto a page allocation if we had done this earlier.
Quoted 2 lines
Acked-by John Fastabend <[email protected]>
PATCH v2 5/11 ixgbe: Make use of order 1 pages and 3K buffers independent of FCoE
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
In order to support build_skb with jumbo frames it will be necessary to use 3K buffers for the Rx path with 8K pages backing them. This is needed on architectures that implement 4K pages because we can't support 2K buffers plus padding in a 4K page.
In the case of systems that support page sizes larger than 4K the 3K attribute will only be applied to FCoE as we can fall back to using just 2K buffers and adding the padding.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 20 +++++++++----------- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 19 +++++++++++++------ 2 files changed, 22 insertions(+), 17 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe.h +9 -11
@@ -234,13 +234,14 @@ struct ixgbe_rx_queue_stats {#define IXGBE_TS_HDR_LEN 8enum ixgbe_ring_state_t { + __IXGBE_RX_3K_BUFFER, + __IXGBE_RX_RSC_ENABLED, + __IXGBE_RX_CSUM_UDP_ZERO_ERR, + __IXGBE_RX_FCOE, __IXGBE_TX_FDIR_INIT_DONE, __IXGBE_TX_XPS_INIT_DONE, __IXGBE_TX_DETECT_HANG, __IXGBE_HANG_CHECK_ARMED, - __IXGBE_RX_RSC_ENABLED, - __IXGBE_RX_CSUM_UDP_ZERO_ERR, - __IXGBE_RX_FCOE, };struct ixgbe_fwd_adapter {
@@ -352,19 +353,16 @@ struct ixgbe_ring_feature {*/ static inline unsigned int ixgbe_rx_bufsz (struct ixgbe_ring *ring) { -#ifdef IXGBE_FCOE -if (test_bit (__IXGBE_RX_FCOE, &ring->state)) -return (PAGE_SIZE <8192 ) ?IXGBE_RXBUFFER_4K : -IXGBE_RXBUFFER_3K ; -#endif +if (test_bit (__IXGBE_RX_3K_BUFFER, &ring->state)) +return IXGBE_RXBUFFER_3K ;return IXGBE_RXBUFFER_2K ; }static inline unsigned int ixgbe_rx_pg_order (struct ixgbe_ring *ring) { -#ifdef IXGBE_FCOE -if (test_bit (__IXGBE_RX_FCOE, &ring->state)) -return (PAGE_SIZE <8192 ) ?1 :0 ; +#if (PAGE_SIZE <8192 ) +if (test_bit (__IXGBE_RX_3K_BUFFER, &ring->state)) +return 1 ;#endif return 0 ; }
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +13 -6
@@ -1617,6 +1617,7 @@ void ixgbe_alloc_rx_buffers(struct ixgbe_ring *rx_ring, u16 cleaned_count)union ixgbe_adv_rx_desc *rx_desc;struct ixgbe_rx_buffer *bi;u16 i = rx_ring->next_to_use; +u16 bufsz;/* nothing to do */ if (!cleaned_count)
@@ -1626,14 +1627,15 @@ void ixgbe_alloc_rx_buffers(struct ixgbe_ring *rx_ring, u16 cleaned_count) bi = &rx_ring->rx_buffer_info[i]; i -= rx_ring->count; + bufsz =ixgbe_rx_bufsz (rx_ring); + do { if (!ixgbe_alloc_mapped_page(rx_ring, bi)) break; /* sync the buffer for use by the device */ dma_sync_single_range_for_device(rx_ring->dev, bi->dma, - bi->page_offset, - ixgbe_rx_bufsz(rx_ring), + bi->page_offset, bufsz, DMA_FROM_DEVICE); /*
@@ -2017,9 +2019,9 @@ static bool ixgbe_add_rx_frag(struct ixgbe_ring *rx_ring,struct page *page = rx_buffer->page;unsigned char *va =page_address (page) + rx_buffer->page_offset;#if (PAGE_SIZE <8192 ) -unsigned int truesize =ixgbe_rx_bufsz (rx_ring); +unsigned int truesize =ixgbe_rx_pg_size (rx_ring) /2 ;#else -unsigned int truesize =ALIGN (size,L1_CACHE_BYTES ); +unsigned int truesize =SKB_DATA_ALIGN (size);#endif if (unlikely (skb_is_nonlinear (skb)))
@@ -3918,10 +3920,15 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter)*/ for (i =0 ; i < adapter->num_rx_queues; i++) { rx_ring = adapter->rx_ring[i]; + +clear_ring_rsc_enabled (rx_ring); +clear_bit (__IXGBE_RX_3K_BUFFER, &rx_ring->state); +if (adapter->flags2 &IXGBE_FLAG2_RSC_ENABLED )set_ring_rsc_enabled (rx_ring); -else -clear_ring_rsc_enabled (rx_ring); + +if (test_bit (__IXGBE_RX_FCOE, &rx_ring->state)) +set_bit (__IXGBE_RX_3K_BUFFER, &rx_ring->state); } }
Bowers
PATCH v2 5/11 ixgbe: Make use of order 1 pages and 3K buffers independent of FCoE
Quoted 20 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:36 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 05/11] ixgbe: Make use of order 1 pages and 3K buffers independent of FCoEFrom: Alexander Duyck <[email protected]>
In order to support
build_skbwith jumbo frames it will be necessary to use 3K buffers for the Rx path with 8K pages backing them. This is needed on architectures that implement 4K pages because we can't support 2K buffers plus padding in a 4K page.In the case of systems that support page sizes larger than 4K the 3K attribute will only be applied to FCoE as we can fall back to using just 2K buffers and adding the padding.
Quoted 5 lines
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 20 +++++++++----------- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 19 +++++++++++++------ 2 files changed, 22 insertions(+), 17 deletions(-)
PATCH v2 6/11 ixgbe: Use length to determine if descriptor is done
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This change makes it so that we use the length of the packet instead of the DD status bit to determine if a new descriptor is ready to be processed. The obvious advantage is that it cuts down on reads as we don't really even need the DD bit if going from a 0 to a non-zero value on size is enough to inform us that the packet has been completed.
---
v2: Update ethtool loopback test to use length check instead of DD check
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 2 +- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +1 -1
@@ -1909,7 +1909,7 @@ static u16 ixgbe_clean_test_rings(struct ixgbe_ring *rx_ring, tx_ntc = tx_ring->next_to_clean; rx_desc =IXGBE_RX_DESC (rx_ring, rx_ntc); -while (ixgbe_test_staterr (rx_desc,IXGBE_RXD_STAT_DD )) { +while (rx_desc->wb.upper.length) {/* check Rx buffer */ rx_buffer = &rx_ring->rx_buffer_info[rx_ntc];
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +8 -6
@@ -1653,8 +1653,8 @@ void ixgbe_alloc_rx_buffers(struct ixgbe_ring *rx_ring, u16 cleaned_count) i -= rx_ring->count; } -/* clear the status bits for the next_to_use descriptor */ - rx_desc->wb.upper.status_error =0 ; +/* clear the length for the next_to_use descriptor */ + rx_desc->wb.upper.length =0 ; cleaned_count--; }while (cleaned_count);
@@ -2171,7 +2171,7 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector, rx_desc =IXGBE_RX_DESC (rx_ring, rx_ring->next_to_clean); -if (!rx_desc->wb.upper.status_error) +if (!rx_desc->wb.upper.length)break ;/* This memory barrier is needed to keep us from reading
@@ -3750,6 +3750,7 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, struct ixgbe_ring *ring) {struct ixgbe_hw *hw = &adapter->hw; +union ixgbe_adv_rx_desc *rx_desc;u64 rdba = ring->dma;u32 rxdctl;u8 reg_idx = ring->reg_idx;
@@ -3784,6 +3785,10 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter, rxdctl |=0x080420 ; } +/* initialize Rx descriptor 0 */ + rx_desc =IXGBE_RX_DESC (ring,0 ); + rx_desc->wb.upper.length =0 ; +/* enable receive descriptor ring */ rxdctl |=IXGBE_RXDCTL_ENABLE ;IXGBE_WRITE_REG (hw,IXGBE_RXDCTL (reg_idx), rxdctl);
@@ -4999,9 +5004,6 @@ static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring) size =sizeof (struct ixgbe_rx_buffer ) * rx_ring->count;memset (rx_ring->rx_buffer_info,0 , size); -/* Zero out the descriptor ring */ -memset (rx_ring->desc,0 , rx_ring->size); - rx_ring->next_to_alloc =0 ; rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ;
Bowers
PATCH v2 6/11 ixgbe: Use length to determine if descriptor is done
Quoted 17 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:36 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 06/11] ixgbe: Use length to determine if descriptor is doneFrom: Alexander Duyck <[email protected]>
This change makes it so that we use the length of the packet instead of the DD status bit to determine if a new descriptor is ready to be processed. The obvious advantage is that it cuts down on reads as we don't really even need the DD bit if going from a 0 to a non-zero value on size is enough to inform us that the packet has been completed.
Quoted 8 lines
Signed-off-by Alexander Duyck <[email protected]>---
v2: Update ethtool loopback test to use length check instead of DD check
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 2 +- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-)
PATCH v2 7/11 ixgbe: Break out Rx buffer page management
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
We are going to be expanding the number of Rx paths in the driver. Instead of duplicating all that code I am pulling it apart into separate functions so that we don't have so much code duplication.
---
v2: Update patch to incorporate addition of ring to can_reuse_page function
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 231 +++++++++++++------------ 1 file changed, 122 insertions(+), 109 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +122 -109
@@ -1864,7 +1864,6 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,ixgbe_rx_pg_size (rx_ring),DMA_FROM_DEVICE , IXGBE_RX_DMA_ATTR); -IXGBE_CB (skb)->page_released = false; }else {struct skb_frag_struct *frag = &skb_shinfo (skb)->frags[0 ];
@@ -1874,7 +1873,6 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,skb_frag_size (frag), DMA_FROM_DEVICE); } -IXGBE_CB (skb)->dma =0 ; }/**
@@ -1945,8 +1943,14 @@ static void ixgbe_reuse_rx_page(struct ixgbe_ring *rx_ring, nta++; rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ; -/* transfer page from old buffer to new buffer */ - *new_buff = *old_buff; +/* Transfer page from old buffer to new buffer. +* Move each member individually to avoid possible store +* forwarding stalls and unnecessary copy of skb. +*/ + new_buff->dma = old_buff->dma; + new_buff->page = old_buff->page; + new_buff->page_offset = old_buff->page_offset; + new_buff->pagecnt_bias = old_buff->pagecnt_bias; }static inline bool ixgbe_page_is_reserved (struct page *page)
@@ -1954,16 +1958,10 @@ static inline bool ixgbe_page_is_reserved(struct page *page)return (page_to_nid (page) !=numa_mem_id ()) ||page_is_pfmemalloc (page); } -static bool ixgbe_can_reuse_rx_page (struct ixgbe_ring *rx_ring, -struct ixgbe_rx_buffer *rx_buffer, -struct page *page, -const unsigned int truesize) +static bool ixgbe_can_reuse_rx_page (struct ixgbe_rx_buffer *rx_buffer) { -#if (PAGE_SIZE >=8192 ) -unsigned int last_offset =ixgbe_rx_pg_size (rx_ring) - -ixgbe_rx_bufsz (rx_ring); -#endif -unsigned int pagecnt_bias = rx_buffer->pagecnt_bias--; +unsigned int pagecnt_bias = rx_buffer->pagecnt_bias; +struct page *page = rx_buffer->page;/* avoid re-using remote pages */ if (unlikely (ixgbe_page_is_reserved (page)))
@@ -1971,16 +1969,17 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,#if (PAGE_SIZE <8192 )/* if we are only owner of page we can reuse it */ -if (unlikely (page_count (page) != pagecnt_bias)) +if (unlikely ((page_ref_count (page) - pagecnt_bias) >1 ))return false; - -/* flip page offset to other buffer */ - rx_buffer->page_offset ^= truesize;#else -/* move offset up to the next cache line */ - rx_buffer->page_offset += truesize; - -if (rx_buffer->page_offset > last_offset) +/* The last offset is a bit aggressive in that we assume the +* worst case of FCoE being enabled and using a 3K buffer. +* However this should have minimal impact as the 1K extra is +* still less than one buffer in size. +*/ +#define IXGBE_LAST_OFFSET \ + (SKB_WITH_OVERHEAD(PAGE_SIZE) - IXGBE_RXBUFFER_3K) +if (rx_buffer->page_offset >IXGBE_LAST_OFFSET )return false;#endif
@@ -1988,7 +1987,7 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,* the pagecnt_bias and page count so that we fully restock the * number of references the driver holds. */ -if (unlikely (pagecnt_bias ==1 )) { +if (unlikely (!pagecnt_bias)) {page_ref_add (page,USHRT_MAX ); rx_buffer->pagecnt_bias =USHRT_MAX ; }
@@ -2011,106 +2010,65 @@ static bool ixgbe_can_reuse_rx_page(struct ixgbe_ring *rx_ring,* The function will then update the page offset if necessary and return * true if the buffer can be reused by the adapter. **/ -static bool ixgbe_add_rx_frag (struct ixgbe_ring *rx_ring, +static void ixgbe_add_rx_frag (struct ixgbe_ring *rx_ring,struct ixgbe_rx_buffer *rx_buffer, -unsigned int size, -struct sk_buff *skb) +struct sk_buff *skb, +unsigned int size) { -struct page *page = rx_buffer->page; -unsigned char *va =page_address (page) + rx_buffer->page_offset;#if (PAGE_SIZE <8192 )unsigned int truesize =ixgbe_rx_pg_size (rx_ring) /2 ;#else unsigned int truesize =SKB_DATA_ALIGN (size);#endif - -if (unlikely (skb_is_nonlinear (skb))) - goto add_tail_frag; - -if (size <=IXGBE_RX_HDR_SIZE ) { -memcpy (__skb_put (skb, size), va,ALIGN (size,sizeof (long ))); - -/* page is not reserved, we can reuse buffer as-is */ -if (likely (!ixgbe_page_is_reserved (page))) -return true; - -/* this page cannot be reused so discard it */ -return false; - } - -add_tail_frag: -skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, page, +skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, rx_buffer->page, rx_buffer->page_offset, size, truesize); - -return ixgbe_can_reuse_rx_page (rx_ring, rx_buffer, page, truesize); +#if (PAGE_SIZE <8192 ) + rx_buffer->page_offset ^= truesize; +#else + rx_buffer->page_offset += truesize; +#endif } -static struct sk_buff *ixgbe_fetch_rx_buffer (struct ixgbe_ring *rx_ring, -union ixgbe_adv_rx_desc *rx_desc) +static struct ixgbe_rx_buffer *ixgbe_get_rx_buffer (struct ixgbe_ring *rx_ring, +union ixgbe_adv_rx_desc *rx_desc, +struct sk_buff **skb, +const unsigned int size) { -unsigned int size =le16_to_cpu (rx_desc->wb.upper.length);struct ixgbe_rx_buffer *rx_buffer; -struct sk_buff *skb; -struct page *page; rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean]; - page = rx_buffer->page; -prefetchw (page); - - skb = rx_buffer->skb; - -if (likely (!skb)) { -void *page_addr =page_address (page) + - rx_buffer->page_offset; +prefetchw (rx_buffer->page); + *skb = rx_buffer->skb; -/* prefetch first cache line of first page */ -prefetch (page_addr); -#if L1_CACHE_BYTES <128 -prefetch (page_addr +L1_CACHE_BYTES ); -#endif - -/* allocate a skb to store the frags */ - skb =napi_alloc_skb (&rx_ring->q_vector->napi, -IXGBE_RX_HDR_SIZE ); -if (unlikely (!skb)) { - rx_ring->rx_stats.alloc_rx_buff_failed++; -return NULL ; - } - -/* -* we will be copying header into skb->data in -* pskb_may_pull so it is in our interest to prefetch -* it now to avoid a possible cache miss -*/ -prefetchw (skb->data); - -/* -* Delay unmapping of the first packet. It carries the -* header information, HW may still access the header -* after the writeback. Only unmap it when EOP is -* reached -*/ -if (likely (ixgbe_test_staterr (rx_desc,IXGBE_RXD_STAT_EOP ))) - goto dma_sync; - -IXGBE_CB (skb)->dma = rx_buffer->dma; +/* Delay unmapping of the first packet. It carries the header +* information, HW may still access the header after the writeback. +* Only unmap it when EOP is reached +*/ +if (!ixgbe_test_staterr (rx_desc,IXGBE_RXD_STAT_EOP )) { +if (!*skb) + goto skip_sync; }else { -if (ixgbe_test_staterr (rx_desc,IXGBE_RXD_STAT_EOP )) -ixgbe_dma_sync_frag (rx_ring, skb); +if (*skb) +ixgbe_dma_sync_frag (rx_ring, *skb); + } -dma_sync: -/* we are reusing so sync this buffer for CPU use */ -dma_sync_single_range_for_cpu (rx_ring->dev, - rx_buffer->dma, - rx_buffer->page_offset, - size, -DMA_FROM_DEVICE ); +/* we are reusing so sync this buffer for CPU use */ +dma_sync_single_range_for_cpu (rx_ring->dev, + rx_buffer->dma, + rx_buffer->page_offset, + size, +DMA_FROM_DEVICE ); +skip_sync: + rx_buffer->pagecnt_bias--; - rx_buffer->skb =NULL ; - } +return rx_buffer; +} -/* pull page into skb */ -if (ixgbe_add_rx_frag (rx_ring, rx_buffer, size, skb)) { +static void ixgbe_put_rx_buffer (struct ixgbe_ring *rx_ring, +struct ixgbe_rx_buffer *rx_buffer, +struct sk_buff *skb) +{ +if (ixgbe_can_reuse_rx_page (rx_buffer)) {/* hand second half of page back to the ring */ ixgbe_reuse_rx_page (rx_ring, rx_buffer); }else {
@@ -2124,12 +2082,55 @@ static struct sk_buff *ixgbe_fetch_rx_buffer(struct ixgbe_ring *rx_ring,DMA_FROM_DEVICE , IXGBE_RX_DMA_ATTR); } -__page_frag_cache_drain (page, +__page_frag_cache_drain (rx_buffer->page, rx_buffer->pagecnt_bias); } -/* clear contents of buffer_info */ +/* clear contents of rx_buffer */ rx_buffer->page =NULL ; + rx_buffer->skb =NULL ; +} + +static struct sk_buff *ixgbe_construct_skb (struct ixgbe_ring *rx_ring, +struct ixgbe_rx_buffer *rx_buffer, +union ixgbe_adv_rx_desc *rx_desc, +unsigned int size) +{ +void *va =page_address (rx_buffer->page) + rx_buffer->page_offset; +#if (PAGE_SIZE <8192 ) +unsigned int truesize =ixgbe_rx_pg_size (rx_ring) /2 ; +#else +unsigned int truesize =ALIGN (size,L1_CACHE_BYTES ); +#endif +struct sk_buff *skb; + +/* prefetch first cache line of first page */ +prefetch (va); +#if L1_CACHE_BYTES <128 +prefetch (va +L1_CACHE_BYTES ); +#endif + +/* allocate a skb to store the frags */ + skb =napi_alloc_skb (&rx_ring->q_vector->napi,IXGBE_RX_HDR_SIZE ); +if (unlikely (!skb)) +return NULL ; + +if (size >IXGBE_RX_HDR_SIZE ) { +if (!ixgbe_test_staterr (rx_desc,IXGBE_RXD_STAT_EOP )) +IXGBE_CB (skb)->dma = rx_buffer->dma; + +skb_add_rx_frag (skb,0 , rx_buffer->page, + rx_buffer->page_offset, + size, truesize); +#if (PAGE_SIZE <8192 ) + rx_buffer->page_offset ^= truesize; +#else + rx_buffer->page_offset += truesize; +#endif + }else { +memcpy (__skb_put (skb, size), va,ALIGN (size,sizeof (long ))); + rx_buffer->pagecnt_bias++; + }return skb; }
@@ -2161,7 +2162,9 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,while (likely (total_rx_packets < budget)) {union ixgbe_adv_rx_desc *rx_desc; +struct ixgbe_rx_buffer *rx_buffer;struct sk_buff *skb; +unsigned int size;/* return some buffers to hardware, one@a time is too slow */ if (cleaned_count >=IXGBE_RX_BUFFER_WRITE ) {
@@ -2170,8 +2173,8 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector, } rx_desc =IXGBE_RX_DESC (rx_ring, rx_ring->next_to_clean); - -if (!rx_desc->wb.upper.length) + size =le16_to_cpu (rx_desc->wb.upper.length); +if (!size)break ;/* This memory barrier is needed to keep us from reading
@@ -2180,13 +2183,23 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,*/ dma_rmb (); + rx_buffer =ixgbe_get_rx_buffer (rx_ring, rx_desc, &skb, size); +/* retrieve a buffer from the ring */ - skb =ixgbe_fetch_rx_buffer (rx_ring, rx_desc); +if (skb) +ixgbe_add_rx_frag (rx_ring, rx_buffer, skb, size); +else + skb =ixgbe_construct_skb (rx_ring, rx_buffer, + rx_desc, size);/* exit if we failed to retrieve a buffer */ -if (!skb) +if (!skb) { + rx_ring->rx_stats.alloc_rx_buff_failed++; + rx_buffer->pagecnt_bias++;break ; + } +ixgbe_put_rx_buffer (rx_ring, rx_buffer, skb); cleaned_count++;/* place incomplete frames back on ring for completion */
Bowers
PATCH v2 7/11 ixgbe: Break out Rx buffer page management
Quoted 15 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:37 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 07/11] ixgbe: Break out Rx buffer page managementFrom: Alexander Duyck <[email protected]>
We are going to be expanding the number of Rx paths in the driver. Instead of duplicating all that code I am pulling it apart into separate functions so that we don't have so much code duplication.
Quoted 8 lines
Signed-off-by Alexander Duyck <[email protected]>---
v2: Update patch to incorporate addition of ring to
can_reuse_pagefunctiondrivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 231 +++++++++++++------ ------ 1 file changed, 122 insertions(+), 109 deletions(-)
PATCH v2 8/11 ixgbe: Add support for padding packet
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This patch adds support for providing a buffer with headroom and tailroom to allow for shared info, NET_SKB_PAD, and NET_IP_ALIGN. With this combined with the DMA changes we can start using build_skb to build frames around an incoming Rx buffer instead of having to memcpy the headers.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 17 ++++++++++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 43 +++++++++++++++++++++++-- 2 files changed, 56 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe.h +17 -0
@@ -94,6 +94,14 @@#define IXGBE_RXBUFFER_4K 4096#define IXGBE_MAX_RXBUFFER 16384/* largest size for a single descriptor */ +#define IXGBE_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN) +#if (PAGE_SIZE <8192 ) +#define IXGBE_MAX_FRAME_BUILD_SKB \ + (SKB_WITH_OVERHEAD(IXGBE_RXBUFFER_2K) - IXGBE_SKB_PAD) +#else +#define IGB_MAX_FRAME_BUILD_SKB IXGBE_RXBUFFER_2K +#endif +/* * NOTE: netdev_alloc_skb reserves up to 64 bytes, NET_IP_ALIGN means we * reserve 64 more, and skb_shared_info adds an additional 320 bytes more,
@@ -235,6 +243,7 @@ struct ixgbe_rx_queue_stats {enum ixgbe_ring_state_t { __IXGBE_RX_3K_BUFFER, + __IXGBE_RX_BUILD_SKB_ENABLED, __IXGBE_RX_RSC_ENABLED, __IXGBE_RX_CSUM_UDP_ZERO_ERR, __IXGBE_RX_FCOE,
@@ -244,6 +253,9 @@ enum ixgbe_ring_state_t { __IXGBE_HANG_CHECK_ARMED, }; +#define ring_uses_build_skb (ring) \ + test_bit(__IXGBE_RX_BUILD_SKB_ENABLED, &(ring)->state) +struct ixgbe_fwd_adapter {unsigned long active_vlans[BITS_TO_LONGS (VLAN_N_VID )];struct net_device *netdev;
@@ -355,6 +367,10 @@ static inline unsigned int ixgbe_rx_bufsz(struct ixgbe_ring *ring) {if (test_bit (__IXGBE_RX_3K_BUFFER, &ring->state))return IXGBE_RXBUFFER_3K ; +#if (PAGE_SIZE <8192 ) +if (ring_uses_build_skb (ring)) +return IXGBE_MAX_FRAME_BUILD_SKB ; +#endif return IXGBE_RXBUFFER_2K ; }
@@ -670,6 +686,7 @@ struct ixgbe_adapter {#define IXGBE_FLAG2_VLAN_PROMISC BIT(13)#define IXGBE_FLAG2_EEE_CAPABLE BIT(14)#define IXGBE_FLAG2_EEE_ENABLED BIT(15) +#define IXGBE_FLAG2_RX_LEGACY BIT(16)/* Tx fast path data */ int num_tx_queues;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +39 -4
91 lines not on this page · raw message
Bowers
PATCH v2 8/11 ixgbe: Add support for padding packet
Quoted 17 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:37 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 08/11] ixgbe: Add support for padding packetFrom: Alexander Duyck <[email protected]>
This patch adds support for providing a buffer with headroom and tailroom to allow for shared info,
NET_SKB_PAD, andNET_IP_ALIGN. With this combined with the DMA changes we can start usingbuild_skbto build frames around an incoming Rx buffer instead of having to memcpy the headers.
Quoted 6 lines
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 17 ++++++++++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 43 +++++++++++++++++++++++-- 2 files changed, 56 insertions(+), 4 deletions(-)
PATCH v2 9/11 ixgbe: Add private flag to control buffer mode
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
Since there are potential drawbacks to the new Rx allocation approach I thought it best to add a "chicken bit" so that we can turn the feature off if in the event that a problem is found.
It also provides a means of validating the legacy Rx path in the event that we are forced to fall back. At some point in the future when we are convinced we don't need it anymore we might be able to drop the legacy-rx flag.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 47 ++++++++++++++++++++++ 1 file changed, 47 insertions(+)
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +47 -0
@@ -151,6 +151,13 @@ struct ixgbe_stats { };#define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN +static const char ixgbe_priv_flags_strings[][ETH_GSTRING_LEN ] = { +#define IXGBE_PRIV_FLAGS_LEGACY_RX BIT(0) +"legacy-rx" , +}; + +#define IXGBE_PRIV_FLAGS_STR_LEN ARRAY_SIZE(ixgbe_priv_flags_strings) +/* currently supported speeds for 10G */ #define ADVRTSD_MSK_10G (SUPPORTED_10000baseT_Full | \ SUPPORTED_10000baseKX4_Full | \
@@ -1002,6 +1009,8 @@ static void ixgbe_get_drvinfo(struct net_device *netdev,strlcpy (drvinfo->bus_info,pci_name (adapter->pdev),sizeof (drvinfo->bus_info)); + + drvinfo->n_priv_flags =IXGBE_PRIV_FLAGS_STR_LEN ; } static void ixgbe_get_ringparam(struct net_device *netdev,
@@ -1141,6 +1150,8 @@ static int ixgbe_get_sset_count(struct net_device *netdev, int sset)return IXGBE_TEST_LEN ;case ETH_SS_STATS :return IXGBE_STATS_LEN ; +case ETH_SS_PRIV_FLAGS : +return IXGBE_PRIV_FLAGS_STR_LEN ;default :return -EOPNOTSUPP ; }
@@ -1305,6 +1316,9 @@ static void ixgbe_get_strings(struct net_device *netdev, u32 stringset, }/* BUG_ON(p - data != IXGBE_STATS_LEN * ETH_GSTRING_LEN); */ break ; +case ETH_SS_PRIV_FLAGS : +memcpy (data, ixgbe_priv_flags_strings, +IXGBE_PRIV_FLAGS_STR_LEN *ETH_GSTRING_LEN ); } }
@@ -3386,6 +3400,37 @@ static int ixgbe_set_eee(struct net_device *netdev, struct ethtool_eee *edata)return 0 ; } +static u32 ixgbe_get_priv_flags (struct net_device *netdev) +{ +struct ixgbe_adapter *adapter =netdev_priv (netdev); +u32 priv_flags =0 ; + +if (adapter->flags2 &IXGBE_FLAG2_RX_LEGACY ) + priv_flags |=IXGBE_PRIV_FLAGS_LEGACY_RX ; + +return priv_flags; +} + +static int ixgbe_set_priv_flags (struct net_device *netdev,u32 priv_flags) +{ +struct ixgbe_adapter *adapter =netdev_priv (netdev); +unsigned int flags2 = adapter->flags2; + + flags2 &= ~IXGBE_FLAG2_RX_LEGACY ; +if (priv_flags &IXGBE_PRIV_FLAGS_LEGACY_RX ) + flags2 |=IXGBE_FLAG2_RX_LEGACY ; + +if (flags2 != adapter->flags2) { + adapter->flags2 = flags2; + +/* reset interface to repopulate queues */ +if (netif_running (netdev)) +ixgbe_reinit_locked (adapter); + } + +return 0 ; +} +static const struct ethtool_ops ixgbe_ethtool_ops = { .get_settings = ixgbe_get_settings, .set_settings = ixgbe_set_settings,
@@ -3422,6 +3467,8 @@ static int ixgbe_set_eee(struct net_device *netdev, struct ethtool_eee *edata) .set_eee = ixgbe_set_eee, .get_channels = ixgbe_get_channels, .set_channels = ixgbe_set_channels, + .get_priv_flags = ixgbe_get_priv_flags, + .set_priv_flags = ixgbe_set_priv_flags, .get_ts_info = ixgbe_get_ts_info, .get_module_info = ixgbe_get_module_info, .get_module_eeprom = ixgbe_get_module_eeprom,
Bowers
PATCH v2 9/11 ixgbe: Add private flag to control buffer mode
Quoted 19 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:37 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 09/11] ixgbe: Add private flag to control buffer modeFrom: Alexander Duyck <[email protected]>
Since there are potential drawbacks to the new Rx allocation approach I thought it best to add a "chicken bit" so that we can turn the feature off if in the event that a problem is found.
It also provides a means of validating the legacy Rx path in the event that we are forced to fall back. At some point in the future when we are convinced we don't need it anymore we might be able to drop the legacy-rx flag.
Quoted 5 lines
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 47 ++++++++++++++++++++++ 1 file changed, 47 insertions(+)
PATCH v2 10/11 ixgbe: Add support for build_skb
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This patch adds build_skb support to the Rx path. There are several advantages to this change.
1. It avoids the memcpy and skb->head allocation for small packets which improves performance by about 5% in my tests.
2. It avoids the memcpy, skb->head allocation, and eth_get_headlen for larger packets improving performance by about 10% in my tests.
3. For VXLAN packets it allows the full header to be in skb->data which improves the performance by as much as 30% in some of my tests.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 49 ++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +48 -1
@@ -1913,7 +1913,7 @@ static bool ixgbe_cleanup_headers(struct ixgbe_ring *rx_ring, }/* place header in linear portion of buffer */ -if (skb_is_nonlinear (skb)) +if (!skb_headlen (skb))ixgbe_pull_tail (rx_ring, skb);#ifdef IXGBE_FCOE
@@ -2142,6 +2142,49 @@ static struct sk_buff *ixgbe_construct_skb(struct ixgbe_ring *rx_ring,return skb; } +static struct sk_buff *ixgbe_build_skb (struct ixgbe_ring *rx_ring, +struct ixgbe_rx_buffer *rx_buffer, +union ixgbe_adv_rx_desc *rx_desc, +unsigned int size) +{ +void *va =page_address (rx_buffer->page) + rx_buffer->page_offset; +#if (PAGE_SIZE <8192 ) +unsigned int truesize =ixgbe_rx_pg_size (rx_ring) /2 ; +#else +unsigned int truesize =SKB_DATA_ALIGN (sizeof (struct skb_shared_info )) + +SKB_DATA_ALIGN (IXGBE_SKB_PAD + size); +#endif +struct sk_buff *skb; + +/* prefetch first cache line of first page */ +prefetch (va); +#if L1_CACHE_BYTES <128 +prefetch (va +L1_CACHE_BYTES ); +#endif + +/* build an skb to around the page buffer */ + skb =build_skb (va -IXGBE_SKB_PAD , truesize); +if (unlikely (!skb)) +return NULL ; + +/* update pointers within the skb to store the data */ +skb_reserve (skb,IXGBE_SKB_PAD ); +__skb_put (skb, size); + +/* record DMA address if this is the start of a chain of buffers */ +if (!ixgbe_test_staterr (rx_desc,IXGBE_RXD_STAT_EOP )) +IXGBE_CB (skb)->dma = rx_buffer->dma; + +/* update buffer offset */ +#if (PAGE_SIZE <8192 ) + rx_buffer->page_offset ^= truesize; +#else + rx_buffer->page_offset += truesize; +#endif + +return skb; +} +/** * ixgbe_clean_rx_irq - Clean completed descriptors from Rx ring - bounce buf * @q_vector: structure containing interrupt and ring information
@@ -2195,6 +2238,9 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,/* retrieve a buffer from the ring */ if (skb)ixgbe_add_rx_frag (rx_ring, rx_buffer, skb, size); +else if (ring_uses_build_skb (rx_ring)) + skb =ixgbe_build_skb (rx_ring, rx_buffer, + rx_desc, size);else skb =ixgbe_construct_skb (rx_ring, rx_buffer, rx_desc, size);
@@ -3970,6 +4016,7 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter)if (test_bit (__IXGBE_RX_FCOE, &rx_ring->state))set_bit (__IXGBE_RX_3K_BUFFER, &rx_ring->state); +clear_bit (__IXGBE_RX_BUILD_SKB_ENABLED, &rx_ring->state);if (adapter->flags2 &IXGBE_FLAG2_RX_LEGACY )continue ;
Bowers
PATCH v2 10/11 ixgbe: Add support for build_skb
Quoted 21 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:37 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 10/11] ixgbe: Add support forbuild_skbFrom: Alexander Duyck <[email protected]>
This patch adds
build_skbsupport to the Rx path. There are several advantages to this change.1. It avoids the memcpy and
skb->headallocation for small packets which improves performance by about 5% in my tests.
2. It avoids the memcpy,skb->headallocation, andeth_get_headlenfor larger packets improving performance by about 10% in my tests.
3. For VXLAN packets it allows the full header to be inskb->datawhich improves the performance by as much as 30% in some of my tests.
Quoted 5 lines
Signed-off-by Alexander Duyck <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 49 ++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-)
PATCH v2 11/11 ixgbe: Don't bother clearing buffer memory for descriptor rings
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This patch makes it so that we don't need to bother with clearing the memory out for the descriptor rings. The general idea is to only free buffers associated with buffers in use which are located between the next_to_clean and next_to_use or next_to_alloc values. Everything outside of those regions can be safely ignored since they should have no buffers associated with them.
The advantage to doing things this way is that is should speed up bring-up and tear-down of the rings. Specifically we can avoid the 512 or more cycles required to memset the rings in tear-down. In the bring-up phase we then clear the memory as a part of initialization. The general idea is that the clearing in initialization can act as a prefetch of sorts for the buffer info structures so they are in the local CPU when we go to populate them. This should help to improve overall time needed to perform a suspend/resume.
---
v2: Fixed Tx mapping exception path to reset tx_buffer address
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 11 +- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 159 ++++++++++++---------- 2 files changed, 99 insertions(+), 71 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +10 -1
@@ -1945,7 +1945,16 @@ static u16 ixgbe_clean_test_rings(struct ixgbe_ring *rx_ring,/* unmap buffer on Tx side */ tx_buffer = &tx_ring->tx_buffer_info[tx_ntc]; -ixgbe_unmap_and_free_tx_resource (tx_ring, tx_buffer); + +/* Free all the Tx ring sk_buffs */ +dev_kfree_skb_any (tx_buffer->skb); + +/* unmap skb header data */ +dma_unmap_single (tx_ring->dev, +dma_unmap_addr (tx_buffer, dma), +dma_unmap_len (tx_buffer, len), +DMA_TO_DEVICE ); +dma_unmap_len_set (tx_buffer, len,0 );/* increment Rx/Tx next to clean counters */ rx_ntc++;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +89 -70
252 lines not on this page · raw message
Bowers
PATCH v2 11/11 ixgbe: Don't bother clearing buffer memory for descriptor rings
Quoted 27 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Alexander Duyck
Sent: Tuesday, January 17, 2017 8:37 AM
To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T <[email protected]>
Subject: [Intel-wired-lan] [next PATCH v2 11/11] ixgbe: Don't bother clearing buffer memory for descriptor ringsFrom: Alexander Duyck <[email protected]>
This patch makes it so that we don't need to bother with clearing the memory out for the descriptor rings. The general idea is to only free buffers associated with buffers in use which are located between the
next_to_cleanandnext_to_useornext_to_allocvalues. Everything outside of those regions can be safely ignored since they should have no buffers associated with them.The advantage to doing things this way is that is should speed up bring-up and tear-down of the rings. Specifically we can avoid the 512 or more cycles required to memset the rings in tear-down. In the bring-up phase we then clear the memory as a part of initialization. The general idea is that the clearing in initialization can act as a prefetch of sorts for the buffer info structures so they are in the local CPU when we go to populate them. This should help to improve overall time needed to perform a suspend/resume.
Signed-off-by Alexander Duyck <[email protected]>
Quoted 8 lines
---
v2: Fixed Tx mapping exception path to reset
tx_bufferaddressdrivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 11 +- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 159 ++++++++++++------ ---- 2 files changed, 99 insertions(+), 71 deletions(-)