igb: Add support for writable pages and build_skb
Subsystems (2)
v4 cover igb: Add support for writable pages and build_skb
This patch set enables support for using the recent changes that allow for unmapping pages without invalidating their contents via DMA_ATTR_SKIP_CPU_SYNC. With this change DMA pages should be writable and as a result we should be able to make use of build_skb which can be used to drop the skb->head memory allocation, header parsing, and memcpy from the receive path which can greatly help to improve performance.
My main concern at this point is that there might be an architecture where I didn't get DMA_ATTR_SKIP_CPU_SYNC implemented that might still need it. For that reason I have also added a ethtool private flag called out as "legacy-rx". If a platform encounters an issue where the Rx can possibly corrupt data it can be enbled by running:
ethtool --set-priv-flags DEVNAME legacy-rx on
The testing matrix for all of these patches is going to be pretty extensive. Basically we want to test these patches on as many platforms and architectures as possible with as many features being toggled as possible including SR-IOV and Jumbo Frames all while receiving traffic.
Within the patches there is also some intialization changes. Specifically I have updated the code paths to defer clearing the rings until we are about to initialize them and give them to hardware. By doing this we are able to avoid having to dirty memory we don't need to which should help to improve suspend/resume times for when we start looking at possibly using the suspend/resume approach for migration of interface in VMs.
v2: Minor fixes to address a few issues found in testing
v3: Fix minor build issues reported by kbuild test robot
v4: Split patch 6 up into 4 separate patches to allow for better bisection
---
Alexander Duyck (12):
igb: Add support for DMA_ATTR_WEAK_ORDERING
igb: Use length to determine if descriptor is done
igb: Clear Rx buffer_info in configure instead of clean
igb: Don't bother clearing Tx buffer_info in igb_clean_tx_ring
igb: Limit maximum frame Rx based on MTU
igb: Only sync size of expected frame in ethtool testing
igb: Use page_address offset from page instead of masking virtual address
igb: Add support for ethtool private flag to allow use of legacy Rx
igb: Add support for using order 1 pages to receive large frames
igb: Add support for padding packet
igb: Break out Rx buffer page management
igb: Re-add support for build_skb in igb
drivers/net/ethernet/intel/igb/igb.h | 58 +++ drivers/net/ethernet/intel/igb/igb_ethtool.c | 65 +++ drivers/net/ethernet/intel/igb/igb_main.c | 509 ++++++++++++++++---------- drivers/net/ethernet/intel/igb/igb_ptp.c | 3 4 files changed, 434 insertions(+), 201 deletions(-)
--
PATCH v4 1/12 igb: Add support for DMA_ATTR_WEAK_ORDERING
From: Alexander Duyck <[email protected]>
Since we are already using DMA attributes in igb for Rx there is no reason why we can't also apply DMA_ATTR_WEAK_ORDERING which is needed on some platforms to improve performance.
--- drivers/net/ethernet/intel/igb/igb.h | 3 +++ drivers/net/ethernet/intel/igb/igb_main.c | 6 +++--- 2 files changed, 6 insertions(+), 3 deletions(-)
drivers/net/ethernet/intel/igb/igb.h +3 -0
@@ -148,6 +148,9 @@ struct vf_data_storage {/* How many Rx Buffers do we bundle into one write to the hardware ? */ #define IGB_RX_BUFFER_WRITE 16/* Must be power of 2 */ +#define IGB_RX_DMA_ATTR \ + (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING) +#define AUTO_ALL_MODES 0#define IGB_EEPROM_APME 0x0400
drivers/net/ethernet/intel/igb/igb_main.c +3 -3
@@ -3963,7 +3963,7 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring) buffer_info->dma,PAGE_SIZE ,DMA_FROM_DEVICE , - DMA_ATTR_SKIP_CPU_SYNC); + IGB_RX_DMA_ATTR);__page_frag_cache_drain (buffer_info->page, buffer_info->pagecnt_bias);
@@ -6992,7 +6992,7 @@ static struct sk_buff *igb_fetch_rx_buffer(struct igb_ring *rx_ring,*/ dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma,PAGE_SIZE ,DMA_FROM_DEVICE , -DMA_ATTR_SKIP_CPU_SYNC ); +IGB_RX_DMA_ATTR );__page_frag_cache_drain (page, rx_buffer->pagecnt_bias); }
@@ -7252,7 +7252,7 @@ static bool igb_alloc_mapped_page(struct igb_ring *rx_ring,/* map page for use */ dma =dma_map_page_attrs (rx_ring->dev, page,0 ,PAGE_SIZE , -DMA_FROM_DEVICE ,DMA_ATTR_SKIP_CPU_SYNC ); +DMA_FROM_DEVICE ,IGB_RX_DMA_ATTR );/* if mapping failed free memory back to system since * there isn't much point in holding memory we can't use
PATCH v4 2/12 igb: Use length to determine if descriptor is done
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This change makes it so that we use the length of the packet instead of the DD status bit to determine if a new descriptor is ready to be processed. The obvious advantage is that it cuts down on reads as we don't really even need the DD bit if going from a 0 to a non-zero value on size is enough to inform us that the packet has been completed.
In addition I have updated the code so that we only reset the Rx descriptor length for descriptor zero when resetting a ring instead of having to do a memset with 0 over the entire ring. By doing this we can save some time on initialization.
---
v2: Update ethtool loopback test to use length check instead of DD check
drivers/net/ethernet/intel/igb/igb_ethtool.c | 2 +- drivers/net/ethernet/intel/igb/igb_main.c | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-)
drivers/net/ethernet/intel/igb/igb_ethtool.c +1 -1
@@ -1811,7 +1811,7 @@ static int igb_clean_test_rings(struct igb_ring *rx_ring, tx_ntc = tx_ring->next_to_clean; rx_desc =IGB_RX_DESC (rx_ring, rx_ntc); -while (igb_test_staterr (rx_desc,E1000_RXD_STAT_DD )) { +while (rx_desc->wb.upper.length) {/* check Rx buffer */ rx_buffer_info = &rx_ring->rx_buffer_info[rx_ntc];
drivers/net/ethernet/intel/igb/igb_main.c +8 -6
@@ -3720,6 +3720,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, struct igb_ring *ring) {struct e1000_hw *hw = &adapter->hw; +union e1000_adv_rx_desc *rx_desc;u64 rdba = ring->dma;int reg_idx = ring->reg_idx;u32 srrctl =0 , rxdctl =0 ;
@@ -3758,6 +3759,10 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, rxdctl |=IGB_RX_HTHRESH <<8 ; rxdctl |=IGB_RX_WTHRESH <<16 ; +/* initialize Rx descriptor 0 */ + rx_desc =IGB_RX_DESC (ring,0 ); + rx_desc->wb.upper.length =0 ; +/* enable receive descriptor fetching */ rxdctl |=E1000_RXDCTL_QUEUE_ENABLE ;wr32 (E1000_RXDCTL (reg_idx), rxdctl);
@@ -3973,9 +3978,6 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring) size =sizeof (struct igb_rx_buffer ) * rx_ring->count;memset (rx_ring->rx_buffer_info,0 , size); -/* Zero out the descriptor ring */ -memset (rx_ring->desc,0 , rx_ring->size); - rx_ring->next_to_alloc =0 ; rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ;
@@ -7174,7 +7176,7 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) rx_desc =IGB_RX_DESC (rx_ring, rx_ring->next_to_clean); -if (!rx_desc->wb.upper.status_error) +if (!rx_desc->wb.upper.length)break ;/* This memory barrier is needed to keep us from reading
@@ -7314,8 +7316,8 @@ void igb_alloc_rx_buffers(struct igb_ring *rx_ring, u16 cleaned_count) i -= rx_ring->count; } -/* clear the status bits for the next_to_use descriptor */ - rx_desc->wb.upper.status_error =0 ; +/* clear the length for the next_to_use descriptor */ + rx_desc->wb.upper.length =0 ; cleaned_count--; }while (cleaned_count);
PATCH v4 3/12 igb: Clear Rx buffer_info in configure instead of clean
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This change makes it so that instead of going through the entire ring on Rx cleanup we only go through the region that was designated to be cleaned up and stop when we reach the region where new allocations should start.
In addition we can avoid having to perform a memset on the Rx buffer_info structures until we are about to start using the ring again. By deferring this we can avoid dirtying the cache any more than we have to which can help to improve the time needed to bring the interface down and then back up again in a reset or suspend/resume cycle.
--- drivers/net/ethernet/intel/igb/igb_main.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +10 -14
@@ -3435,7 +3435,7 @@ int igb_setup_rx_resources(struct igb_ring *rx_ring) size =sizeof (struct igb_rx_buffer ) * rx_ring->count; - rx_ring->rx_buffer_info =vzalloc (size); + rx_ring->rx_buffer_info =vmalloc (size);if (!rx_ring->rx_buffer_info) goto err;
@@ -3759,6 +3759,10 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, rxdctl |=IGB_RX_HTHRESH <<8 ; rxdctl |=IGB_RX_WTHRESH <<16 ; +/* initialize rx_buffer_info */ +memset (ring->rx_buffer_info,0 , +sizeof (struct igb_rx_buffer ) * ring->count); +/* initialize Rx descriptor 0 */ rx_desc =IGB_RX_DESC (ring,0 ); rx_desc->wb.upper.length =0 ;
@@ -3937,23 +3941,16 @@ static void igb_free_all_rx_resources(struct igb_adapter *adapter)**/ static void igb_clean_rx_ring (struct igb_ring *rx_ring) { -unsigned long size; -u16 i; +u16 i = rx_ring->next_to_clean;if (rx_ring->skb)dev_kfree_skb (rx_ring->skb); rx_ring->skb =NULL ; -if (!rx_ring->rx_buffer_info) -return ; -/* Free all the Rx ring sk_buffs */ -for (i =0 ; i < rx_ring->count; i++) { +while (i != rx_ring->next_to_alloc) {struct igb_rx_buffer *buffer_info = &rx_ring->rx_buffer_info[i]; -if (!buffer_info->page) -continue ; -/* Invalidate cache lines that may have been written to by * device so that we avoid corrupting memory. */
@@ -3972,12 +3969,11 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring)__page_frag_cache_drain (buffer_info->page, buffer_info->pagecnt_bias); - buffer_info->page =NULL ; + i++; +if (i == rx_ring->count) + i =0 ; } - size =sizeof (struct igb_rx_buffer ) * rx_ring->count; -memset (rx_ring->rx_buffer_info,0 , size); - rx_ring->next_to_alloc =0 ; rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ;
PATCH v4 4/12 igb: Don't bother clearing Tx buffer_info in igb_clean_tx_ring
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
In the case of the Tx rings we need to only clear the Tx buffer_info when we are resetting the rings. Ideally we do this when we configure the ring to bring it back up instead of when we are taking it down in order to avoid dirtying pages we don't need to.
In addition we don't need to clear the Tx descriptor ring since we will fully repopulate it when we begin transmitting frames and next_to_watch can be cleared to prevent the ring from being cleaned beyond that point instead of needing to touch anything in the Tx descriptor ring.
Finally with these changes we can avoid having to reset the skb member of the Tx buffer_info structure in the cleanup path since the skb will always be associated with the first buffer which has next_to_watch set.
---
v2: Fixed Tx mapping exception path to reset tx_buffer address
drivers/net/ethernet/intel/igb/igb.h | 1 drivers/net/ethernet/intel/igb/igb_ethtool.c | 11 ++ drivers/net/ethernet/intel/igb/igb_main.c | 120 ++++++++++++++++---------- 3 files changed, 83 insertions(+), 49 deletions(-)
drivers/net/ethernet/intel/igb/igb.h +0 -1
@@ -594,7 +594,6 @@ enum igb_boards {void igb_setup_tctl (struct igb_adapter *);void igb_setup_rctl (struct igb_adapter *);netdev_tx_t igb_xmit_frame_ring (struct sk_buff *,struct igb_ring *); -void igb_unmap_and_free_tx_resource (struct igb_ring *,struct igb_tx_buffer *);void igb_alloc_rx_buffers (struct igb_ring *,u16 );void igb_update_stats (struct igb_adapter *,struct rtnl_link_stats64 *);bool igb_has_link (struct igb_adapter *adapter);
drivers/net/ethernet/intel/igb/igb_ethtool.c +10 -1
@@ -1833,7 +1833,16 @@ static int igb_clean_test_rings(struct igb_ring *rx_ring,/* unmap buffer on Tx side */ tx_buffer_info = &tx_ring->tx_buffer_info[tx_ntc]; -igb_unmap_and_free_tx_resource (tx_ring, tx_buffer_info); + +/* Free all the Tx ring sk_buffs */ +dev_kfree_skb_any (tx_buffer_info->skb); + +/* unmap skb header data */ +dma_unmap_single (tx_ring->dev, +dma_unmap_addr (tx_buffer_info, dma), +dma_unmap_len (tx_buffer_info, len), +DMA_TO_DEVICE ); +dma_unmap_len_set (tx_buffer_info, len,0 );/* increment Rx/Tx next to clean counters */ rx_ntc++;
drivers/net/ethernet/intel/igb/igb_main.c +73 -47
173 lines not on this page · raw message
PATCH v4 5/12 igb: Limit maximum frame Rx based on MTU
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
In order to support the use of build_skb going forward it will be necessary to place a maximum limit on the amount of data we can receive when jumbo frames is not enabled. In order to do this I am adding a new upper limit for receive based on the size of a 2K buffer minus padding.
--- drivers/net/ethernet/intel/igb/igb.h | 10 +++++++++- drivers/net/ethernet/intel/igb/igb_main.c | 16 ++++++++++++---- 2 files changed, 21 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/igb/igb.h +9 -1
@@ -143,8 +143,17 @@ struct vf_data_storage {#define IGB_RXBUFFER_256 256#define IGB_RXBUFFER_2048 2048#define IGB_RX_HDR_LEN IGB_RXBUFFER_256 +#define IGB_TS_HDR_LEN 16#define IGB_RX_BUFSZ IGB_RXBUFFER_2048 +#define IGB_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN) +#if (PAGE_SIZE <8192 ) +#define IGB_MAX_FRAME_BUILD_SKB \ + (SKB_WITH_OVERHEAD(IGB_RXBUFFER_2048) - IGB_SKB_PAD - IGB_TS_HDR_LEN) +#else +#define IGB_MAX_FRAME_BUILD_SKB (IGB_RXBUFFER_2048 - IGB_TS_HDR_LEN) +#endif +/* How many Rx Buffers do we bundle into one write to the hardware ? */ #define IGB_RX_BUFFER_WRITE 16/* Must be power of 2 */
@@ -561,7 +570,6 @@ struct igb_adapter {#define IGB_DMCTLX_DCFLUSH_DIS 0x80000000/* Disable DMA Coal Flush */ #define IGB_82576_TSYNC_SHIFT 19 -#define IGB_TS_HDR_LEN 16enum e1000_state_t { __IGB_TESTING, __IGB_RESETTING,
drivers/net/ethernet/intel/igb/igb_main.c +12 -4
@@ -4250,7 +4250,7 @@ static void igb_set_rx_mode(struct net_device *netdev)struct igb_adapter *adapter =netdev_priv (netdev);struct e1000_hw *hw = &adapter->hw;unsigned int vfn = adapter->vfs_allocated_count; -u32 rctl =0 , vmolr =0 ; +u32 rctl =0 , vmolr =0 , rlpml =MAX_JUMBO_FRAME_SIZE ;int count;/* Check for Promiscuous and All Multicast modes */
@@ -4322,12 +4322,20 @@ static void igb_set_rx_mode(struct net_device *netdev) vmolr |=rd32 (E1000_VMOLR (vfn)) & ~(E1000_VMOLR_ROPE |E1000_VMOLR_MPME |E1000_VMOLR_ROMPE ); -/* enable Rx jumbo frames, no need for restriction */ +/* enable Rx jumbo frames, restrict as needed to support build_skb */ vmolr &= ~E1000_VMOLR_RLPML_MASK ; - vmolr |=MAX_JUMBO_FRAME_SIZE |E1000_VMOLR_LPE ; +#if (PAGE_SIZE <8192 ) +if (adapter->max_frame_size <=IGB_MAX_FRAME_BUILD_SKB ) { +if (!adapter->vfs_allocated_count) + rlpml =IGB_MAX_FRAME_BUILD_SKB ; + vmolr |=IGB_MAX_FRAME_BUILD_SKB ; + } else +#endif + vmolr |=MAX_JUMBO_FRAME_SIZE ; + vmolr |=E1000_VMOLR_LPE ;wr32 (E1000_VMOLR (vfn), vmolr); -wr32 (E1000_RLPML ,MAX_JUMBO_FRAME_SIZE ); +wr32 (E1000_RLPML , rlpml);igb_restore_vf_multicasts (adapter); }
PATCH v4 6/12 igb: Only sync size of expected frame in ethtool testing
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
We only need to sync the size of the frame that is read to test. We don't need to sync the entire Rx buffer. This way the testing is more consistent with how we handle things in the receive path.
---
v4: Split this code out of patch 6 of the original series.
drivers/net/ethernet/intel/igb/igb_ethtool.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_ethtool.c +2 -2
@@ -1818,7 +1818,7 @@ static int igb_clean_test_rings(struct igb_ring *rx_ring,/* sync Rx buffer for CPU read */ dma_sync_single_for_cpu (rx_ring->dev, rx_buffer_info->dma, -IGB_RX_BUFSZ , + size,DMA_FROM_DEVICE );/* verify contents of skb */
@@ -1828,7 +1828,7 @@ static int igb_clean_test_rings(struct igb_ring *rx_ring,/* sync Rx buffer for device write */ dma_sync_single_for_device (rx_ring->dev, rx_buffer_info->dma, -IGB_RX_BUFSZ , + size,DMA_FROM_DEVICE );/* unmap buffer on Tx side */
PATCH v4 7/12 igb: Use page_address offset from page instead of masking virtual address
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
Update the handling of page addresses so that we always refer to them using a void pointer, and try to use the consistent name of va indicating we are working with a virtual address.
---
v4: Split this code out of patch 6 of the original series.
drivers/net/ethernet/intel/igb/igb.h | 2 +- drivers/net/ethernet/intel/igb/igb_main.c | 11 +++++------ drivers/net/ethernet/intel/igb/igb_ptp.c | 3 +-- 3 files changed, 7 insertions(+), 9 deletions(-)
drivers/net/ethernet/intel/igb/igb.h +1 -1
@@ -614,7 +614,7 @@ enum igb_boards {void igb_ptp_suspend (struct igb_adapter *adapter);void igb_ptp_rx_hang (struct igb_adapter *adapter);void igb_ptp_rx_rgtstamp (struct igb_q_vector *q_vector,struct sk_buff *skb); -void igb_ptp_rx_pktstamp (struct igb_q_vector *q_vector,unsigned char *va, +void igb_ptp_rx_pktstamp (struct igb_q_vector *q_vector,void *va,struct sk_buff *skb);int igb_ptp_set_ts_config (struct net_device *netdev,struct ifreq *ifr);int igb_ptp_get_ts_config (struct net_device *netdev,struct ifreq *ifr);
drivers/net/ethernet/intel/igb/igb_main.c +5 -6
@@ -6924,7 +6924,7 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring, struct sk_buff *skb) {struct page *page = rx_buffer->page; -unsigned char *va =page_address (page) + rx_buffer->page_offset; +void *va =page_address (page) + rx_buffer->page_offset;#if (PAGE_SIZE <8192 )unsigned int truesize =IGB_RX_BUFSZ ;#else
@@ -6966,7 +6966,7 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring, add_tail_frag:skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, page, - (unsigned long )va & ~PAGE_MASK , size, truesize); + va -page_address (page), size, truesize);return igb_can_reuse_rx_page (rx_buffer, page, truesize); }
@@ -6991,13 +6991,12 @@ static struct sk_buff *igb_fetch_rx_buffer(struct igb_ring *rx_ring, DMA_FROM_DEVICE);if (likely (!skb)) { -void *page_addr =page_address (page) + - rx_buffer->page_offset; +void *va =page_address (page) + rx_buffer->page_offset;/* prefetch first cache line of first page */ -prefetch (page_addr); +prefetch (va);#if L1_CACHE_BYTES <128 -prefetch (page_addr +L1_CACHE_BYTES ); +prefetch (va +L1_CACHE_BYTES );#endif /* allocate a skb to store the frags */
drivers/net/ethernet/intel/igb/igb_ptp.c +1 -2
@@ -764,8 +764,7 @@ static void igb_ptp_tx_hwtstamp(struct igb_adapter *adapter)* incoming frame. The value is stored in little endian format starting on * byte 8. **/ -void igb_ptp_rx_pktstamp (struct igb_q_vector *q_vector, -unsigned char *va, +void igb_ptp_rx_pktstamp (struct igb_q_vector *q_vector,void *va,struct sk_buff *skb) {__le64 *regval = (__le64 *)va;
PATCH v4 8/12 igb: Add support for ethtool private flag to allow use of legacy Rx
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
Since there are potential drawbacks to the new Rx allocation approach I thought it best to add a "chicken bit" so that we can turn the feature off if in the event that a problem is found.
It also provides a means of validating the legacy Rx path in the event that we are forced to fall back. At some point in the future when we are convinced we don't need it anymore we might be able to drop the legacy-rx flag.
---
v4: Move patch up so that we add support for RX_LEGACY flag before we add
support for the new approach.
Added actual creation of the RX_LEGACY flag to this patch.
drivers/net/ethernet/intel/igb/igb.h | 1 + drivers/net/ethernet/intel/igb/igb_ethtool.c | 48 ++++++++++++++++++++++++++ 2 files changed, 49 insertions(+)
drivers/net/ethernet/intel/igb/igb.h +1 -0
@@ -557,6 +557,7 @@ struct igb_adapter {#define IGB_FLAG_HAS_MSIX BIT(13)#define IGB_FLAG_EEE BIT(14)#define IGB_FLAG_VLAN_PROMISC BIT(15) +#define IGB_FLAG_RX_LEGACY BIT(16)/* Media Auto Sense */ #define IGB_MAS_ENABLE_0 0X0001
drivers/net/ethernet/intel/igb/igb_ethtool.c +48 -0
84 lines not on this page · raw message
PATCH v4 9/12 igb: Add support for using order 1 pages to receive large frames
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This patch adds support for using 3K buffers in order 1 pages the same way we were using 2K buffers in 4K pages. We are reserving 1K of room for now to have space available for future headroom and tailroom when we enable build_skb support.
One side effect of this patch is that we can end up using a larger buffer if jumbo frames is enabled. The impact shouldn't be too great, but it could hurt small packet performance for UDP workloads if jumbo frames is enabled as the truesize of frames will be larger.
---
v4: Split this code out of patch 6 of the original series.
drivers/net/ethernet/intel/igb/igb.h | 30 +++++++++++++- drivers/net/ethernet/intel/igb/igb_main.c | 64 +++++++++++++++++++++-------- 2 files changed, 76 insertions(+), 18 deletions(-)
drivers/net/ethernet/intel/igb/igb.h +29 -1
@@ -142,9 +142,9 @@ struct vf_data_storage {/* Supported Rx Buffer Sizes */ #define IGB_RXBUFFER_256 256#define IGB_RXBUFFER_2048 2048 +#define IGB_RXBUFFER_3072 3072#define IGB_RX_HDR_LEN IGB_RXBUFFER_256#define IGB_TS_HDR_LEN 16 -#define IGB_RX_BUFSZ IGB_RXBUFFER_2048#define IGB_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)#if (PAGE_SIZE <8192 )
@@ -313,12 +313,40 @@ struct igb_q_vector { };enum e1000_ring_flags_t { +IGB_RING_FLAG_RX_3K_BUFFER ,IGB_RING_FLAG_RX_SCTP_CSUM ,IGB_RING_FLAG_RX_LB_VLAN_BSWAP ,IGB_RING_FLAG_TX_CTX_IDX ,IGB_RING_FLAG_TX_DETECT_HANG }; +#define ring_uses_large_buffer (ring) \ + test_bit(IGB_RING_FLAG_RX_3K_BUFFER, &(ring)->flags) +#define set_ring_uses_large_buffer (ring) \ + set_bit(IGB_RING_FLAG_RX_3K_BUFFER, &(ring)->flags) +#define clear_ring_uses_large_buffer (ring) \ + clear_bit(IGB_RING_FLAG_RX_3K_BUFFER, &(ring)->flags) + +static inline unsigned int igb_rx_bufsz (struct igb_ring *ring) +{ +#if (PAGE_SIZE <8192 ) +if (ring_uses_large_buffer (ring)) +return IGB_RXBUFFER_3072 ; +#endif +return IGB_RXBUFFER_2048 ; +} + +static inline unsigned int igb_rx_pg_order (struct igb_ring *ring) +{ +#if (PAGE_SIZE <8192 ) +if (ring_uses_large_buffer (ring)) +return 1 ; +#endif +return 0 ; +} + +#define igb_rx_pg_size (_ring) (PAGE_SIZE << igb_rx_pg_order(_ring)) +#define IGB_TXD_DCMD (E1000_ADVTXD_DCMD_EOP | E1000_ADVTXD_DCMD_RS)#define IGB_RX_DESC (R , i) \
drivers/net/ethernet/intel/igb/igb_main.c +47 -17
162 lines not on this page · raw message
PATCH v4 10/12 igb: Add support for padding packet
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
With the size of the frame limited we can now write to an offset within the buffer instead of having to write at the very start of the buffer. The advantage to this is that it allows us to leave padding room for things like supporting XDP in the future.
---
v4: Split this code out of patch 6 of the original series.
drivers/net/ethernet/intel/igb/igb.h | 11 +++++++++++ drivers/net/ethernet/intel/igb/igb_main.c | 14 ++++++++++++-- 2 files changed, 23 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb.h +11 -0
@@ -314,6 +314,7 @@ struct igb_q_vector {enum e1000_ring_flags_t {IGB_RING_FLAG_RX_3K_BUFFER , +IGB_RING_FLAG_RX_BUILD_SKB_ENABLED ,IGB_RING_FLAG_RX_SCTP_CSUM ,IGB_RING_FLAG_RX_LB_VLAN_BSWAP ,IGB_RING_FLAG_TX_CTX_IDX ,
@@ -327,11 +328,21 @@ enum e1000_ring_flags_t {#define clear_ring_uses_large_buffer (ring) \ clear_bit(IGB_RING_FLAG_RX_3K_BUFFER, &(ring)->flags) +#define ring_uses_build_skb (ring) \ + test_bit(IGB_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags) +#define set_ring_build_skb_enabled (ring) \ + set_bit(IGB_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags) +#define clear_ring_build_skb_enabled (ring) \ + clear_bit(IGB_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags) +static inline unsigned int igb_rx_bufsz (struct igb_ring *ring) {#if (PAGE_SIZE <8192 )if (ring_uses_large_buffer (ring))return IGB_RXBUFFER_3072 ; + +if (ring_uses_build_skb (ring)) +return IGB_MAX_FRAME_BUILD_SKB +IGB_TS_HDR_LEN ;#endif return IGB_RXBUFFER_2048 ; }
drivers/net/ethernet/intel/igb/igb_main.c +12 -2
@@ -3783,11 +3783,14 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter, struct igb_ring *rx_ring) {/* set build_skb and buffer size flags */ +clear_ring_build_skb_enabled (rx_ring);clear_ring_uses_large_buffer (rx_ring);if (adapter->flags &IGB_FLAG_RX_LEGACY )return ; +set_ring_build_skb_enabled (rx_ring); +#if (PAGE_SIZE <8192 )if (adapter->max_frame_size <=IGB_MAX_FRAME_BUILD_SKB )return ;
@@ -6954,7 +6957,9 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring,#if (PAGE_SIZE <8192 )unsigned int truesize =igb_rx_pg_size (rx_ring) /2 ;#else -unsigned int truesize =SKB_DATA_ALIGN (size); +unsigned int truesize =ring_uses_build_skb (rx_ring) ? +SKB_DATA_ALIGN (IGB_SKB_PAD + size) : +SKB_DATA_ALIGN (size);#endif unsigned int pull_len;
@@ -7290,6 +7295,11 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)return total_packets; } +static inline unsigned int igb_rx_offset (struct igb_ring *rx_ring) +{ +return ring_uses_build_skb (rx_ring) ?IGB_SKB_PAD :0 ; +} +static bool igb_alloc_mapped_page (struct igb_ring *rx_ring,struct igb_rx_buffer *bi) {
@@ -7325,7 +7335,7 @@ static bool igb_alloc_mapped_page(struct igb_ring *rx_ring, bi->dma = dma; bi->page = page; - bi->page_offset =0 ; + bi->page_offset =igb_rx_offset (rx_ring); bi->pagecnt_bias =1 ;return true;
PATCH v4 11/12 igb: Break out Rx buffer page management
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
At this point we have 2 to 3 paths that can be taken depending on what Rx modes are enabled. In order to better support that and improve the maintainability I am breaking out the common bits from those paths and making them into their own functions.
---
v3: Rebase patch on v3 changes to fix pointer arithmetic mismatch
v4: Rebase on top of changes to split up buffer size and padding changes
drivers/net/ethernet/intel/igb/igb_main.c | 233 +++++++++++++++-------------- 1 file changed, 121 insertions(+), 112 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +121 -112
@@ -6883,8 +6883,14 @@ static void igb_reuse_rx_page(struct igb_ring *rx_ring, nta++; rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ; -/* transfer page from old buffer to new buffer */ - *new_buff = *old_buff; +/* Transfer page from old buffer to new buffer. +* Move each member individually to avoid possible store +* forwarding stalls. +*/ + new_buff->dma = old_buff->dma; + new_buff->page = old_buff->page; + new_buff->page_offset = old_buff->page_offset; + new_buff->pagecnt_bias = old_buff->pagecnt_bias; }static inline bool igb_page_is_reserved (struct page *page)
@@ -6892,11 +6898,10 @@ static inline bool igb_page_is_reserved(struct page *page)return (page_to_nid (page) !=numa_mem_id ()) ||page_is_pfmemalloc (page); } -static bool igb_can_reuse_rx_page (struct igb_rx_buffer *rx_buffer, -struct page *page, -const unsigned int truesize) +static bool igb_can_reuse_rx_page (struct igb_rx_buffer *rx_buffer) { -unsigned int pagecnt_bias = rx_buffer->pagecnt_bias--; +unsigned int pagecnt_bias = rx_buffer->pagecnt_bias; +struct page *page = rx_buffer->page;/* avoid re-using remote pages */ if (unlikely (igb_page_is_reserved (page)))
@@ -6904,14 +6909,9 @@ static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer,#if (PAGE_SIZE <8192 )/* if we are only owner of page we can reuse it */ -if (unlikely (page_ref_count (page) != pagecnt_bias)) +if (unlikely ((page_ref_count (page) - pagecnt_bias) >1 ))return false; - -/* flip page offset to other buffer */ - rx_buffer->page_offset ^= truesize;#else -/* move offset up to the next cache line */ - rx_buffer->page_offset += truesize;#define IGB_LAST_OFFSET \ (SKB_WITH_OVERHEAD(PAGE_SIZE) - IGB_RXBUFFER_2048)
@@ -6923,7 +6923,7 @@ static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer,* the pagecnt_bias and page count so that we fully restock the * number of references the driver holds. */ -if (unlikely (pagecnt_bias ==1 )) { +if (unlikely (!pagecnt_bias)) {page_ref_add (page,USHRT_MAX ); rx_buffer->pagecnt_bias =USHRT_MAX ; }
@@ -6935,22 +6935,15 @@ static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer,* igb_add_rx_frag - Add contents of Rx buffer to sk_buff * @rx_ring: rx descriptor ring to transact packets on * @rx_buffer: buffer containing page to add -* @rx_desc: descriptor containing length of buffer written by hardware * @skb: sk_buff to place the data into +* @size: size of buffer to be added * * This function will add the data contained in rx_buffer->page to the skb. -* This is done either through a direct copy if the data in the buffer is -* less than the skb header size, otherwise it will just attach the page as -* a frag to the skb. -* -* The function will then update the page offset if necessary and return -* true if the buffer can be reused by the adapter. **/ -static bool igb_add_rx_frag (struct igb_ring *rx_ring, +static void igb_add_rx_frag (struct igb_ring *rx_ring,struct igb_rx_buffer *rx_buffer, -unsigned int size, -union e1000_adv_rx_desc *rx_desc, -struct sk_buff *skb) +struct sk_buff *skb, +unsigned int size) {struct page *page = rx_buffer->page;void *va =page_address (page) + rx_buffer->page_offset;
@@ -6961,10 +6954,39 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring, SKB_DATA_ALIGN(IGB_SKB_PAD + size) :SKB_DATA_ALIGN (size);#endif -unsigned int pull_len; +skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, rx_buffer->page, + rx_buffer->page_offset, size, truesize); +#if (PAGE_SIZE <8192 ) + rx_buffer->page_offset ^= truesize; +#else + rx_buffer->page_offset += truesize; +#endif +} + +static struct sk_buff *igb_construct_skb (struct igb_ring *rx_ring, +struct igb_rx_buffer *rx_buffer, +union e1000_adv_rx_desc *rx_desc, +unsigned int size) +{ +void *va =page_address (rx_buffer->page) + rx_buffer->page_offset; +#if (PAGE_SIZE <8192 ) +unsigned int truesize =igb_rx_pg_size (rx_ring) /2 ; +#else +unsigned int truesize =SKB_DATA_ALIGN (size); +#endif +unsigned int headlen; +struct sk_buff *skb; -if (unlikely (skb_is_nonlinear (skb))) - goto add_tail_frag; +/* prefetch first cache line of first page */ +prefetch (va); +#if L1_CACHE_BYTES <128 +prefetch (va +L1_CACHE_BYTES ); +#endif + +/* allocate a skb to store the frags */ + skb =napi_alloc_skb (&rx_ring->q_vector->napi,IGB_RX_HDR_LEN ); +if (unlikely (!skb)) +return NULL ;if (unlikely (igb_test_staterr (rx_desc,E1000_RXDADV_STAT_TSIP ))) {igb_ptp_rx_pktstamp (rx_ring->q_vector, va, skb);
@@ -6972,95 +6994,29 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring, size -=IGB_TS_HDR_LEN ; } -if (likely (size <=IGB_RX_HDR_LEN )) { -memcpy (__skb_put (skb, size), va,ALIGN (size,sizeof (long ))); - -/* page is not reserved, we can reuse buffer as-is */ -if (likely (!igb_page_is_reserved (page))) -return true; - -/* this page cannot be reused so discard it */ -return false; - } - -/* we need the header to contain the greater of either ETH_HLEN or -* 60 bytes if the skb->len is less than 60 for skb_pad. -*/ - pull_len =eth_get_headlen (va,IGB_RX_HDR_LEN ); +/* Determine available headroom for copy */ + headlen = size; +if (headlen >IGB_RX_HDR_LEN ) + headlen =eth_get_headlen (va,IGB_RX_HDR_LEN );/* align pull length to size of long to optimize memcpy performance */ -memcpy (__skb_put (skb, pull_len), va,ALIGN (pull_len,sizeof (long ))); +memcpy (__skb_put (skb, headlen), va,ALIGN (headlen,sizeof (long )));/* update all of the pointers */ - va += pull_len; - size -= pull_len; - -add_tail_frag: -skb_add_rx_frag (skb,skb_shinfo (skb)->nr_frags, page, - va -page_address (page), size, truesize); - -return igb_can_reuse_rx_page (rx_buffer, page, truesize); -} - -static struct sk_buff *igb_fetch_rx_buffer (struct igb_ring *rx_ring, -union e1000_adv_rx_desc *rx_desc, -struct sk_buff *skb) -{ -unsigned int size =le16_to_cpu (rx_desc->wb.upper.length); -struct igb_rx_buffer *rx_buffer; -struct page *page; - - rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean]; - page = rx_buffer->page; -prefetchw (page); - -/* we are reusing so sync this buffer for CPU use */ -dma_sync_single_range_for_cpu (rx_ring->dev, - rx_buffer->dma, - rx_buffer->page_offset, - size, -DMA_FROM_DEVICE ); - -if (likely (!skb)) { -void *va =page_address (page) + rx_buffer->page_offset; - -/* prefetch first cache line of first page */ -prefetch (va); -#if L1_CACHE_BYTES <128 -prefetch (va +L1_CACHE_BYTES ); + size -= headlen; +if (size) { +skb_add_rx_frag (skb,0 , rx_buffer->page, + (va + headlen) -page_address (rx_buffer->page), + size, truesize); +#if (PAGE_SIZE <8192 ) + rx_buffer->page_offset ^= truesize; +#else + rx_buffer->page_offset += truesize;#endif - -/* allocate a skb to store the frags */ - skb =napi_alloc_skb (&rx_ring->q_vector->napi,IGB_RX_HDR_LEN ); -if (unlikely (!skb)) { - rx_ring->rx_stats.alloc_failed++; -return NULL ; - } - -/* we will be copying header into skb->data in -* pskb_may_pull so it is in our interest to prefetch -* it now to avoid a possible cache miss -*/ -prefetchw (skb->data); - } - -/* pull page into skb */ -if (igb_add_rx_frag (rx_ring, rx_buffer, size, rx_desc, skb)) { -/* hand second half of page back to the ring */ -igb_reuse_rx_page (rx_ring, rx_buffer); }else { -/* We are not reusing the buffer so unmap it and free -* any references we are holding to it -*/ -dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, -igb_rx_pg_size (rx_ring),DMA_FROM_DEVICE , -IGB_RX_DMA_ATTR ); -__page_frag_cache_drain (page, rx_buffer->pagecnt_bias); + rx_buffer->pagecnt_bias++; } -/* clear contents of rx_buffer */ - rx_buffer->page =NULL ; -return skb; }
@@ -7218,6 +7174,47 @@ static void igb_process_skb_fields(struct igb_ring *rx_ring, skb->protocol =eth_type_trans (skb, rx_ring->netdev); } +static struct igb_rx_buffer *igb_get_rx_buffer (struct igb_ring *rx_ring, +const unsigned int size) +{ +struct igb_rx_buffer *rx_buffer; + + rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean]; +prefetchw (rx_buffer->page); + +/* we are reusing so sync this buffer for CPU use */ +dma_sync_single_range_for_cpu (rx_ring->dev, + rx_buffer->dma, + rx_buffer->page_offset, + size, +DMA_FROM_DEVICE ); + + rx_buffer->pagecnt_bias--; + +return rx_buffer; +} + +static void igb_put_rx_buffer (struct igb_ring *rx_ring, +struct igb_rx_buffer *rx_buffer) +{ +if (igb_can_reuse_rx_page (rx_buffer)) { +/* hand second half of page back to the ring */ +igb_reuse_rx_page (rx_ring, rx_buffer); + }else { +/* We are not reusing the buffer so unmap it and free +* any references we are holding to it +*/ +dma_unmap_page_attrs (rx_ring->dev, rx_buffer->dma, +igb_rx_pg_size (rx_ring),DMA_FROM_DEVICE , +IGB_RX_DMA_ATTR ); +__page_frag_cache_drain (rx_buffer->page, + rx_buffer->pagecnt_bias); + } + +/* clear contents of rx_buffer */ + rx_buffer->page =NULL ; +} +static int igb_clean_rx_irq (struct igb_q_vector *q_vector,const int budget) {struct igb_ring *rx_ring = q_vector->rx.ring;
@@ -7227,6 +7224,8 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)while (likely (total_packets < budget)) {union e1000_adv_rx_desc *rx_desc; +struct igb_rx_buffer *rx_buffer; +unsigned int size;/* return some buffers to hardware, one@a time is too slow */ if (cleaned_count >=IGB_RX_BUFFER_WRITE ) {
@@ -7235,8 +7234,8 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) } rx_desc =IGB_RX_DESC (rx_ring, rx_ring->next_to_clean); - -if (!rx_desc->wb.upper.length) + size =le16_to_cpu (rx_desc->wb.upper.length); +if (!size)break ;/* This memory barrier is needed to keep us from reading
@@ -7245,13 +7244,23 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)*/ dma_rmb (); + rx_buffer =igb_get_rx_buffer (rx_ring, size); +/* retrieve a buffer from the ring */ - skb =igb_fetch_rx_buffer (rx_ring, rx_desc, skb); +if (skb) +igb_add_rx_frag (rx_ring, rx_buffer, skb, size); +else + skb =igb_construct_skb (rx_ring, rx_buffer, + rx_desc, size);/* exit if we failed to retrieve a buffer */ -if (!skb) +if (!skb) { + rx_ring->rx_stats.alloc_failed++; + rx_buffer->pagecnt_bias++;break ; + } +igb_put_rx_buffer (rx_ring, rx_buffer); cleaned_count++;/* fetch next buffer in frame if non-eop */
PATCH v4 12/12 igb: Re-add support for build_skb in igb
In reply to Alexander DuyckFrom: Alexander Duyck <[email protected]>
This reverts commit f9d40f6a9921 ("igb: Revert support for build_skb in igb") and adds a few changes to update it to work with the latest version of igb. We are now able to revert the removal of this due to the fact that with the recent changes to the page count and the use of DMA_ATTR_SKIP_CPU_SYNC we can make the pages writable so we should not be invalidating the additional data added when we call build_skb.
--- drivers/net/ethernet/intel/igb/igb_main.c | 47 +++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
drivers/net/ethernet/intel/igb/igb_main.c +47 -0
@@ -7020,6 +7020,51 @@ static struct sk_buff *igb_construct_skb(struct igb_ring *rx_ring,return skb; } +static struct sk_buff *igb_build_skb (struct igb_ring *rx_ring, +struct igb_rx_buffer *rx_buffer, +union e1000_adv_rx_desc *rx_desc, +unsigned int size) +{ +void *va =page_address (rx_buffer->page) + rx_buffer->page_offset; +#if (PAGE_SIZE <8192 ) +unsigned int truesize =igb_rx_pg_size (rx_ring) /2 ; +#else +unsigned int truesize =SKB_DATA_ALIGN (sizeof (struct skb_shared_info )) + +SKB_DATA_ALIGN (IGB_SKB_PAD + size); +#endif +struct sk_buff *skb; + +/* prefetch first cache line of first page */ +prefetch (va); +#if L1_CACHE_BYTES <128 +prefetch (va +L1_CACHE_BYTES ); +#endif + +/* build an skb to around the page buffer */ + skb =build_skb (va -IGB_SKB_PAD , truesize); +if (unlikely (!skb)) +return NULL ; + +/* update pointers within the skb to store the data */ +skb_reserve (skb,IGB_SKB_PAD ); +__skb_put (skb, size); + +/* pull timestamp out of packet data */ +if (igb_test_staterr (rx_desc,E1000_RXDADV_STAT_TSIP )) { +igb_ptp_rx_pktstamp (rx_ring->q_vector, skb->data, skb); +__skb_pull (skb,IGB_TS_HDR_LEN ); + } + +/* update buffer offset */ +#if (PAGE_SIZE <8192 ) + rx_buffer->page_offset ^= truesize; +#else + rx_buffer->page_offset += truesize; +#endif + +return skb; +} +static inline void igb_rx_checksum (struct igb_ring *ring,union e1000_adv_rx_desc *rx_desc,struct sk_buff *skb)
@@ -7249,6 +7294,8 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)/* retrieve a buffer from the ring */ if (skb)igb_add_rx_frag (rx_ring, rx_buffer, skb, size); +else if (ring_uses_build_skb (rx_ring)) + skb =igb_build_skb (rx_ring, rx_buffer, rx_desc, size);else skb =igb_construct_skb (rx_ring, rx_buffer, rx_desc, size);