gve: Introduce DQO descriptor format
Subsystems (4)
cover gve: Introduce DQO descriptor format
DQO is the descriptor format for our next generation virtual NIC. The existing descriptor format will be referred to as "GQI" in the patch set.
One major change with DQO is it uses dual descriptor rings for both TX and RX queues.
The TX path uses a TX queue to send descriptors to HW, and receives packet completion events on a TX completion queue.
The RX path posts buffers to HW using an RX buffer queue and receives incoming packets on an RX queue.
One important note is that DQO descriptors and doorbells are little endian. We continue to use the existing big endian control plane infrastructure.
The general format of the patch series is:
- Refactor existing code/data structures to be shared by DQO
- Expand admin queues to support DQO device setup
- Expand data structures and device setup to support DQO
- Add logic to setup DQO queues
- Implement datapath
Bailey Forrest (16):
gve: Update GVE documentation to describe DQO
gve: Move some static functions to a common file
gve: gve_rx_copy: Move padding to an argument
gve: Make gve_rx_slot_page_info.page_offset an absolute offset
gve: Introduce a new model for device options
gve: Introduce per netdev enum gve_queue_format
gve: adminq: DQO specific device descriptor logic
gve: Add support for DQO RX PTYPE map
gve: Add dqo descriptors
gve: Add DQO fields for core data structures
gve: Update adminq commands to support DQO queues
gve: DQO: Add core netdev features
gve: DQO: Add ring allocation and initialization
gve: DQO: Configure interrupts on device up
gve: DQO: Add TX path
gve: DQO: Add RX path
.../device_drivers/ethernet/google/gve.rst | 53 +- drivers/net/ethernet/google/Kconfig | 2 +- drivers/net/ethernet/google/gve/Makefile | 2 +- drivers/net/ethernet/google/gve/gve.h | 332 +++++- drivers/net/ethernet/google/gve/gve_adminq.c | 334 ++++-- drivers/net/ethernet/google/gve/gve_adminq.h | 112 +- .../net/ethernet/google/gve/gve_desc_dqo.h | 256 ++++ drivers/net/ethernet/google/gve/gve_dqo.h | 81 ++ drivers/net/ethernet/google/gve/gve_ethtool.c | 21 +- drivers/net/ethernet/google/gve/gve_main.c | 291 ++++- drivers/net/ethernet/google/gve/gve_rx.c | 54 +- drivers/net/ethernet/google/gve/gve_rx_dqo.c | 763 ++++++++++++ drivers/net/ethernet/google/gve/gve_tx.c | 25 +- drivers/net/ethernet/google/gve/gve_tx_dqo.c | 1031 +++++++++++++++++ drivers/net/ethernet/google/gve/gve_utils.c | 81 ++ drivers/net/ethernet/google/gve/gve_utils.h | 28 + 16 files changed, 3250 insertions(+), 216 deletions(-) create mode 100644 drivers/net/ethernet/google/gve/gve_desc_dqo.h create mode 100644 drivers/net/ethernet/google/gve/gve_dqo.h create mode 100644 drivers/net/ethernet/google/gve/gve_rx_dqo.c create mode 100644 drivers/net/ethernet/google/gve/gve_tx_dqo.c create mode 100644 drivers/net/ethernet/google/gve/gve_utils.c create mode 100644 drivers/net/ethernet/google/gve/gve_utils.h
base-commit: 35713d9b8f090d7a226e4aaeeb742265cde33c82
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 1/16 gve: Update GVE documentation to describe DQO
DQO is a new descriptor format for our next generation virtual NIC.
--- .../device_drivers/ethernet/google/gve.rst | 53 +++++++++++++++++-- 1 file changed, 48 insertions(+), 5 deletions(-)
Documentation/networking/device_drivers/ethernet/google/gve.rst +48 -5
@@ -47,13 +47,24 @@ The driver interacts with the device in the following ways: - Transmit and Receive Queues - See description below +Descriptor Formats +------------------ +GVE supports two descriptor formats: GQI and DQO. These two formats have +entirely different descriptors, which will be described below. + Registers --------- -All registers are MMIO and big endian. +All registers are MMIO. The registers are used for initializing and configuring the device as well as querying device status in response to management interrupts. +Endianness +---------- +- Admin Queue messages and registers are all Big Endian. +- GQI descriptors and datapath registers are Big Endian. +- DQO descriptors and datapath registers are Little Endian. + Admin Queue (AQ) ---------------- The Admin Queue is a PAGE_SIZE memory block, treated as an array of AQ
@@ -97,10 +108,10 @@ the queues associated with that interrupt. The handler for these irqs schedule the napi for that block to run and poll the queues. -Traffic Queues --------------- -gVNIC's queues are composed of a descriptor ring and a buffer and are -assigned to a notification block. +GQI Traffic Queues +------------------ +GQI queues are composed of a descriptor ring and a buffer and are assigned to a +notification block. The descriptor rings are power-of-two-sized ring buffers consisting of fixed-size descriptors. They advance their head pointer using a __be32
@@ -121,3 +132,35 @@ Receive The buffers for receive rings are put into a data ring that is the same length as the descriptor ring and the head and tail pointers advance over the rings together. + +DQO Traffic Queues +------------------ +- Every TX and RX queue is assigned a notification block. + +- TX and RX buffers queues, which send descriptors to the device, use MMIO + doorbells to notify the device of new descriptors. + +- RX and TX completion queues, which receive descriptors from the device, use a + "generation bit" to know when a descriptor was populated by the device. The + driver initializes all bits with the "current generation". The device will + populate received descriptors with the "next generation" which is inverted + from the current generation. When the ring wraps, the current/next generation + are swapped. + +- It's the driver's responsibility to ensure that the RX and TX completion + queues are not overrun. This can be accomplished by limiting the number of + descriptors posted to HW. + +- TX packets have a 16 bit completion_tag and RX buffers have a 16 bit + buffer_id. These will be returned on the TX completion and RX queues + respectively to let the driver know which packet/buffer was completed. + +Transmit +~~~~~~~~ +A packet's buffers are DMA mapped for the device to access before transmission. +After the packet was successfully transmitted, the buffers are unmapped. + +Receive +~~~~~~~ +The driver posts fixed sized buffers to HW on the RX buffer queue. The packet +received on the associated RX queue may span multiple descriptors.
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 2/16 gve: Move some static functions to a common file
In reply to Bailey ForrestThese functions will be shared by the GQI and DQO variants of the GVNIC driver as of follow-up patches in this series.
--- drivers/net/ethernet/google/gve/Makefile | 2 +- drivers/net/ethernet/google/gve/gve_rx.c | 42 +------------ drivers/net/ethernet/google/gve/gve_tx.c | 21 +------ drivers/net/ethernet/google/gve/gve_utils.c | 65 +++++++++++++++++++++ drivers/net/ethernet/google/gve/gve_utils.h | 24 ++++++++ 5 files changed, 94 insertions(+), 60 deletions(-) create mode 100644 drivers/net/ethernet/google/gve/gve_utils.c create mode 100644 drivers/net/ethernet/google/gve/gve_utils.h
drivers/net/ethernet/google/gve/Makefile +1 -1
@@ -1,4 +1,4 @@ # Makefile for the Google virtual Ethernet (gve) driver obj-$(CONFIG_GVE) += gve.o -gve-objs := gve_main.o gve_tx.o gve_rx.o gve_ethtool.o gve_adminq.o +gve-objs := gve_main.o gve_tx.o gve_rx.o gve_ethtool.o gve_adminq.o gve_utils.o
drivers/net/ethernet/google/gve/gve_rx.c +2 -40
66 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_tx.c +2 -19
@@ -1,11 +1,12 @@// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* Google virtual Ethernet (gve) driver * -* Copyright (C) 2015-2019 Google, Inc. +* Copyright (C) 2015-2021 Google, Inc. */ #include "gve.h" #include "gve_adminq.h" +#include "gve_utils.h" #include <linux/ip.h> #include <linux/tcp.h> #include <linux/vmalloc.h>
@@ -131,14 +132,6 @@ static void gve_tx_free_fifo(struct gve_tx_fifo *fifo, size_t bytes)atomic_add (bytes, &fifo->available); } -static void gve_tx_remove_from_block (struct gve_priv *priv,int queue_idx) -{ -struct gve_notify_block *block = - &priv->ntfy_blocks[gve_tx_idx_to_ntfy (priv, queue_idx)]; - - block->tx =NULL ; -} -static int gve_clean_tx_done (struct gve_priv *priv,struct gve_tx_ring *tx,u32 to_do,bool try_to_wake);
@@ -174,16 +167,6 @@ static void gve_tx_free_ring(struct gve_priv *priv, int idx)netif_dbg (priv, drv, priv->dev,"freed tx queue %d\n" , idx); } -static void gve_tx_add_to_block (struct gve_priv *priv,int queue_idx) -{ -int ntfy_idx =gve_tx_idx_to_ntfy (priv, queue_idx); -struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx]; -struct gve_tx_ring *tx = &priv->tx[queue_idx]; - - block->tx = tx; - tx->ntfy_id = ntfy_idx; -} -static int gve_tx_alloc_ring (struct gve_priv *priv,int idx) {struct gve_tx_ring *tx = &priv->tx[idx];
drivers/net/ethernet/google/gve/gve_utils.c new file +65 -0
65 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_utils.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR MIT) +* Google virtual Ethernet (gve) driver +* +* Copyright (C) 2015-2021 Google, Inc. +*/ + +#ifndef _GVE_UTILS_H +#define _GVE_UTILS_H + +#include <linux/etherdevice.h> + +#include "gve.h" + +void gve_tx_remove_from_block (struct gve_priv *priv,int queue_idx); +void gve_tx_add_to_block (struct gve_priv *priv,int queue_idx); + +void gve_rx_remove_from_block (struct gve_priv *priv,int queue_idx); +void gve_rx_add_to_block (struct gve_priv *priv,int queue_idx); + +struct sk_buff *gve_rx_copy (struct net_device *dev,struct napi_struct *napi, +struct gve_rx_slot_page_info *page_info,u16 len); + +#endif /* _GVE_UTILS_H */ +
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 3/16 gve: gve_rx_copy: Move padding to an argument
In reply to Bailey ForrestFuture use cases will have a different padding value.
--- drivers/net/ethernet/google/gve/gve_rx.c | 4 ++-- drivers/net/ethernet/google/gve/gve_utils.c | 5 +++-- drivers/net/ethernet/google/gve/gve_utils.h | 3 ++- 3 files changed, 7 insertions(+), 5 deletions(-)
drivers/net/ethernet/google/gve/gve_rx.c +2 -2
@@ -350,7 +350,7 @@ gve_rx_qpl(struct device *dev, struct net_device *netdev,gve_rx_flip_buff (page_info, &data_slot->qpl_offset); } }else { - skb =gve_rx_copy (netdev, napi, page_info, len); + skb =gve_rx_copy (netdev, napi, page_info, len,GVE_RX_PAD );if (skb) {u64_stats_update_begin (&rx->statss); rx->rx_copied_pkt++;
@@ -392,7 +392,7 @@ static bool gve_rx(struct gve_rx_ring *rx, struct gve_rx_desc *rx_desc,if (len <= priv->rx_copybreak) {/* Just copy small packets */ - skb =gve_rx_copy (dev, napi, page_info, len); + skb =gve_rx_copy (dev, napi, page_info, len,GVE_RX_PAD );u64_stats_update_begin (&rx->statss); rx->rx_copied_pkt++; rx->rx_copybreak_pkt++;
drivers/net/ethernet/google/gve/gve_utils.c +3 -2
@@ -45,10 +45,11 @@ void gve_rx_add_to_block(struct gve_priv *priv, int queue_idx) }struct sk_buff *gve_rx_copy (struct net_device *dev,struct napi_struct *napi, -struct gve_rx_slot_page_info *page_info,u16 len) +struct gve_rx_slot_page_info *page_info,u16 len, +u16 pad) {struct sk_buff *skb =napi_alloc_skb (napi, len); -void *va = page_info->page_address +GVE_RX_PAD + +void *va = page_info->page_address + pad + (page_info->page_offset ?PAGE_SIZE /2 :0 );if (unlikely (!skb))
drivers/net/ethernet/google/gve/gve_utils.h +2 -1
@@ -18,7 +18,8 @@ void gve_rx_remove_from_block(struct gve_priv *priv, int queue_idx);void gve_rx_add_to_block (struct gve_priv *priv,int queue_idx);struct sk_buff *gve_rx_copy (struct net_device *dev,struct napi_struct *napi, -struct gve_rx_slot_page_info *page_info,u16 len); +struct gve_rx_slot_page_info *page_info,u16 len, +u16 pad);#endif /* _GVE_UTILS_H */
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 4/16 gve: Make gve_rx_slot_page_info.page_offset an absolute offset
In reply to Bailey ForrestUsing page_offset like a boolean means a page may only be split into two sections. With page sizes larger than 4k, this can be very wasteful. Future commits in this patchset use struct gve_rx_slot_page_info in a way which supports a fixed buffer size and a variable page size.
--- drivers/net/ethernet/google/gve/gve.h | 4 ++-- drivers/net/ethernet/google/gve/gve_rx.c | 4 ++-- drivers/net/ethernet/google/gve/gve_utils.c | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)
drivers/net/ethernet/google/gve/gve.h +2 -2
@@ -1,7 +1,7 @@/* SPDX-License-Identifier: (GPL-2.0 OR MIT) * Google virtual Ethernet (gve) driver * -* Copyright (C) 2015-2019 Google, Inc. +* Copyright (C) 2015-2021 Google, Inc. */ #ifndef _GVE_H_
@@ -51,7 +51,7 @@ struct gve_rx_desc_queue {struct gve_rx_slot_page_info {struct page *page;void *page_address; -u8 page_offset;/* flipped to second half? */ +u32 page_offset;/* offset to write to in page */ u8 can_flip; };
drivers/net/ethernet/google/gve/gve_rx.c +2 -2
@@ -272,7 +272,7 @@ static struct sk_buff *gve_rx_add_frags(struct napi_struct *napi,return NULL ;skb_add_rx_frag (skb,0 , page_info->page, - (page_info->page_offset ?PAGE_SIZE /2 :0 ) + + page_info->page_offset +GVE_RX_PAD , len,PAGE_SIZE /2 );return skb;
@@ -283,7 +283,7 @@ static void gve_rx_flip_buff(struct gve_rx_slot_page_info *page_info, __be64 *slconst __be64 offset =cpu_to_be64 (PAGE_SIZE /2 );/* "flip" to other packet buffer on this page */ - page_info->page_offset ^=0x1 ; + page_info->page_offset ^=PAGE_SIZE /2 ; *(slot_addr) ^= offset; }
drivers/net/ethernet/google/gve/gve_utils.c +1 -1
@@ -50,7 +50,7 @@ struct sk_buff *gve_rx_copy(struct net_device *dev, struct napi_struct *napi, {struct sk_buff *skb =napi_alloc_skb (napi, len);void *va = page_info->page_address + pad + - (page_info->page_offset ?PAGE_SIZE /2 :0 ); + page_info->page_offset;if (unlikely (!skb))return NULL ;
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 5/16 gve: Introduce a new model for device options
In reply to Bailey ForrestThe current model uses an integer ID and a fixed size struct for the parameters of each device option.
The new model allows the device option structs to grow in size over time. A driver may assume that changes to device option structs will always be appended.
New device options will also generally have a supported_features_mask so that the driver knows which fields within a particular device option are enabled.
gve_device_option.feat_mask is changed to required_features_mask, and it is a bitmask which must match the value expected by the driver. This gives the device the ability to break backwards compatibility with old drivers for certain features by blocking the old drivers from trying to use the feature.
We maintain ABI compatibility with the old model for GVE_DEV_OPT_ID_RAW_ADDRESSING in case a driver is using a device which does not support the new model.
This patch introduces some new terminology:
RDA - Raw DMA Addressing - Buffers associated with SKBs are directly DMA
mapped and read/updated by the device.
QPL - Queue Page Lists - Driver uses bounce buffers which are DMA mapped
with the device for read/write and data is copied from/to SKBs.
--- drivers/net/ethernet/google/gve/gve_adminq.c | 172 +++++++++++++++---- drivers/net/ethernet/google/gve/gve_adminq.h | 50 +++++- 2 files changed, 179 insertions(+), 43 deletions(-)
drivers/net/ethernet/google/gve/gve_adminq.c +134 -38
@@ -1,7 +1,7 @@// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* Google virtual Ethernet (gve) driver * -* Copyright (C) 2015-2019 Google, Inc. +* Copyright (C) 2015-2021 Google, Inc. */ #include <linux/etherdevice.h>
@@ -18,6 +18,8 @@"Expected: length=%d, feature_mask=%x.\n" \ "Actual: length=%d, feature_mask=%x.\n" +#define GVE_DEVICE_OPTION_TOO_BIG_FMT "Length of %s option larger than expected. Possible older version of guest driver.\n" +static struct gve_device_option *gve_get_next_option (struct gve_device_descriptor *descriptor,struct gve_device_option *option)
@@ -33,28 +35,81 @@ struct gve_device_option *gve_get_next_option(struct gve_device_descriptor *descstatic void gve_parse_device_option (struct gve_priv *priv,struct gve_device_descriptor *device_descriptor, -struct gve_device_option *option) +struct gve_device_option *option, +struct gve_device_option_gqi_rda **dev_op_gqi_rda, +struct gve_device_option_gqi_qpl **dev_op_gqi_qpl, +struct gve_device_option_dqo_rda **dev_op_dqo_rda) { +u32 req_feat_mask =be32_to_cpu (option->required_features_mask);u16 option_length =be16_to_cpu (option->option_length);u16 option_id =be16_to_cpu (option->option_id); +/* If the length or feature mask doesn't match, continue without +* enabling the feature. +*/ switch (option_id) { -case GVE_DEV_OPT_ID_RAW_ADDRESSING : -/* If the length or feature mask doesn't match, -* continue without enabling the feature. -*/ -if (option_length !=GVE_DEV_OPT_LEN_RAW_ADDRESSING || - option->feat_mask !=cpu_to_be32 (GVE_DEV_OPT_FEAT_MASK_RAW_ADDRESSING )) { -dev_warn (&priv->pdev->dev,GVE_DEVICE_OPTION_ERROR_FMT ,"Raw Addressing" , -GVE_DEV_OPT_LEN_RAW_ADDRESSING , -cpu_to_be32 (GVE_DEV_OPT_FEAT_MASK_RAW_ADDRESSING ), - option_length, option->feat_mask); - priv->raw_addressing =0 ; - }else { -dev_info (&priv->pdev->dev, -"Raw addressing device option enabled.\n" ); - priv->raw_addressing =1 ; +case GVE_DEV_OPT_ID_GQI_RAW_ADDRESSING : +if (option_length !=GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING || + req_feat_mask !=GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING ) { +dev_warn (&priv->pdev->dev,GVE_DEVICE_OPTION_ERROR_FMT , +"Raw Addressing" , +GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING , +GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING , + option_length, req_feat_mask); +break ; + } + +dev_info (&priv->pdev->dev, +"Gqi raw addressing device option enabled.\n" ); + priv->raw_addressing =1 ; +break ; +case GVE_DEV_OPT_ID_GQI_RDA : +if (option_length <sizeof (**dev_op_gqi_rda) || + req_feat_mask !=GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA ) { +dev_warn (&priv->pdev->dev,GVE_DEVICE_OPTION_ERROR_FMT , +"GQI RDA" , (int )sizeof (**dev_op_gqi_rda), +GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA , + option_length, req_feat_mask); +break ; + } + +if (option_length >sizeof (**dev_op_gqi_rda)) { +dev_warn (&priv->pdev->dev, +GVE_DEVICE_OPTION_TOO_BIG_FMT ,"GQI RDA" ); + } + *dev_op_gqi_rda = (void *)(option +1 ); +break ; +case GVE_DEV_OPT_ID_GQI_QPL : +if (option_length <sizeof (**dev_op_gqi_qpl) || + req_feat_mask !=GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL ) { +dev_warn (&priv->pdev->dev,GVE_DEVICE_OPTION_ERROR_FMT , +"GQI QPL" , (int )sizeof (**dev_op_gqi_qpl), +GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL , + option_length, req_feat_mask); +break ; + } + +if (option_length >sizeof (**dev_op_gqi_qpl)) { +dev_warn (&priv->pdev->dev, +GVE_DEVICE_OPTION_TOO_BIG_FMT ,"GQI QPL" ); + } + *dev_op_gqi_qpl = (void *)(option +1 ); +break ; +case GVE_DEV_OPT_ID_DQO_RDA : +if (option_length <sizeof (**dev_op_dqo_rda) || + req_feat_mask !=GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA ) { +dev_warn (&priv->pdev->dev,GVE_DEVICE_OPTION_ERROR_FMT , +"DQO RDA" , (int )sizeof (**dev_op_dqo_rda), +GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA , + option_length, req_feat_mask); +break ; + } + +if (option_length >sizeof (**dev_op_dqo_rda)) { +dev_warn (&priv->pdev->dev, +GVE_DEVICE_OPTION_TOO_BIG_FMT ,"DQO RDA" ); } + *dev_op_dqo_rda = (void *)(option +1 );break ;default :/* If we don't recognize the option just continue
@@ -65,6 +120,39 @@ void gve_parse_device_option(struct gve_priv *priv, } } +/* Process all device options for a given describe device call. */ +static int +gve_process_device_options (struct gve_priv *priv, +struct gve_device_descriptor *descriptor, +struct gve_device_option_gqi_rda **dev_op_gqi_rda, +struct gve_device_option_gqi_qpl **dev_op_gqi_qpl, +struct gve_device_option_dqo_rda **dev_op_dqo_rda) +{ +const int num_options =be16_to_cpu (descriptor->num_device_options); +struct gve_device_option *dev_opt; +int i; + +/* The options struct directly follows the device descriptor. */ + dev_opt = (void *)(descriptor +1 ); +for (i =0 ; i < num_options; i++) { +struct gve_device_option *next_opt; + + next_opt =gve_get_next_option (descriptor, dev_opt); +if (!next_opt) { +dev_err (&priv->dev->dev, +"options exceed device_descriptor's total length.\n" ); +return -EINVAL ; + } + +gve_parse_device_option (priv, descriptor, dev_opt, + dev_op_gqi_rda, dev_op_gqi_qpl, + dev_op_dqo_rda); + dev_opt = next_opt; + } + +return 0 ; +} +int gve_adminq_alloc (struct device *dev,struct gve_priv *priv) { priv->adminq = dma_alloc_coherent(dev, PAGE_SIZE,
@@ -514,15 +602,15 @@ int gve_adminq_destroy_rx_queues(struct gve_priv *priv, u32 num_queues)int gve_adminq_describe_device (struct gve_priv *priv) { +struct gve_device_option_gqi_rda *dev_op_gqi_rda =NULL ; +struct gve_device_option_gqi_qpl *dev_op_gqi_qpl =NULL ; +struct gve_device_option_dqo_rda *dev_op_dqo_rda =NULL ;struct gve_device_descriptor *descriptor; -struct gve_device_option *dev_opt;union gve_adminq_command cmd;dma_addr_t descriptor_bus; -u16 num_options;int err =0 ;u8 *mac;u16 mtu; -int i;memset (&cmd,0 ,sizeof (cmd)); descriptor = dma_alloc_coherent(&priv->pdev->dev, PAGE_SIZE,
@@ -540,6 +628,31 @@ int gve_adminq_describe_device(struct gve_priv *priv)if (err) goto free_device_descriptor; + priv->raw_addressing =0 ; + err =gve_process_device_options (priv, descriptor, &dev_op_gqi_rda, + &dev_op_gqi_qpl, &dev_op_dqo_rda); +if (err) + goto free_device_descriptor; + +/* If the GQI_RAW_ADDRESSING option is not enabled and the queue format +* is not set to GqiRda, choose the queue format in a priority order: +* DqoRda, GqiRda, GqiQpl. Use GqiQpl as default. +*/ +if (priv->raw_addressing ==1 ) { +dev_info (&priv->pdev->dev, +"Driver is running with GQI RDA queue format.\n" ); + }else if (dev_op_dqo_rda) { +dev_info (&priv->pdev->dev, +"Driver is running with DQO RDA queue format.\n" ); + }else if (dev_op_gqi_rda) { +dev_info (&priv->pdev->dev, +"Driver is running with GQI RDA queue format.\n" ); + priv->raw_addressing =1 ; + }else { +dev_info (&priv->pdev->dev, +"Driver is running with GQI QPL queue format.\n" ); + } + priv->tx_desc_cnt =be16_to_cpu (descriptor->tx_queue_entries);if (priv->tx_desc_cnt *sizeof (priv->tx->desc[0 ]) <PAGE_SIZE ) {dev_err (&priv->pdev->dev,"Tx desc count %d too low\n" , priv->tx_desc_cnt);
@@ -576,26 +689,9 @@ int gve_adminq_describe_device(struct gve_priv *priv) priv->rx_desc_cnt = priv->rx_data_slot_cnt; } priv->default_num_queues =be16_to_cpu (descriptor->default_num_queues); - dev_opt = (void *)(descriptor +1 ); - - num_options =be16_to_cpu (descriptor->num_device_options); -for (i =0 ; i < num_options; i++) { -struct gve_device_option *next_opt; - - next_opt =gve_get_next_option (descriptor, dev_opt); -if (!next_opt) { -dev_err (&priv->dev->dev, -"options exceed device_descriptor's total length.\n" ); - err = -EINVAL ; - goto free_device_descriptor; - } - -gve_parse_device_option (priv, descriptor, dev_opt); - dev_opt = next_opt; - } free_device_descriptor: -dma_free_coherent (&priv->pdev->dev,sizeof (*descriptor), descriptor, +dma_free_coherent (&priv->pdev->dev,PAGE_SIZE , descriptor, descriptor_bus);return err; }
drivers/net/ethernet/google/gve/gve_adminq.h +45 -5
66 lines not on this page · raw message
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 6/16 gve: Introduce per netdev `enum gve_queue_format`
In reply to Bailey ForrestThe currently supported queue formats are:
- GQI_RDA - GQI with raw DMA addressing
- GQI_QPL - GQI with queue page list
- DQO_RDA - DQO with raw DMA addressing
The old gve_priv.raw_addressing value is only used for GQI_RDA, so we remove it in favor of just checking against GQI_RDA
--- drivers/net/ethernet/google/gve/gve.h | 24 +++++++++++++++++--- drivers/net/ethernet/google/gve/gve_adminq.c | 15 +++++++----- drivers/net/ethernet/google/gve/gve_main.c | 9 ++++---- drivers/net/ethernet/google/gve/gve_rx.c | 2 +- drivers/net/ethernet/google/gve/gve_tx.c | 2 +- 5 files changed, 37 insertions(+), 15 deletions(-)
drivers/net/ethernet/google/gve/gve.h +21 -3
@@ -194,6 +194,17 @@ struct gve_qpl_config {unsigned long *qpl_id_map;/* bitmap of used qpl ids */ }; +/* GVE_QUEUE_FORMAT_UNSPECIFIED must be zero since 0 is the default value +* when the entire configure_device_resources command is zeroed out and the +* queue_format is not specified. +*/ +enum gve_queue_format { +GVE_QUEUE_FORMAT_UNSPECIFIED =0x0 , +GVE_GQI_RDA_FORMAT =0x1 , +GVE_GQI_QPL_FORMAT =0x2 , +GVE_DQO_RDA_FORMAT =0x3 , +}; +struct gve_priv {struct net_device *dev;struct gve_tx_ring *tx;/* array of tx_cfg.num_queues */
@@ -216,7 +227,6 @@ struct gve_priv {u64 num_registered_pages;/* num pages registered with NIC */ u32 rx_copybreak;/* copy packets smaller than this */ u16 default_num_queues;/* default num queues to set up */ -u8 raw_addressing;/* 1 if this dev supports raw addressing, 0 otherwise */ struct gve_queue_config tx_cfg;struct gve_queue_config rx_cfg;
@@ -275,6 +285,8 @@ struct gve_priv {/* Gvnic device link speed from hypervisor. */ u64 link_speed; + +enum gve_queue_format queue_format; };enum gve_service_task_flags_bit {
@@ -454,14 +466,20 @@ static inline u32 gve_rx_idx_to_ntfy(struct gve_priv *priv, u32 queue_idx)*/ static inline u32 gve_num_tx_qpls (struct gve_priv *priv) { -return priv->raw_addressing ?0 : priv->tx_cfg.num_queues; +if (priv->queue_format !=GVE_GQI_QPL_FORMAT ) +return 0 ; + +return priv->tx_cfg.num_queues; }/* Returns the number of rx queue page lists */ static inline u32 gve_num_rx_qpls (struct gve_priv *priv) { -return priv->raw_addressing ?0 : priv->rx_cfg.num_queues; +if (priv->queue_format !=GVE_GQI_QPL_FORMAT ) +return 0 ; + +return priv->rx_cfg.num_queues; }/* Returns a pointer to the next available tx qpl in the list of qpls
drivers/net/ethernet/google/gve/gve_adminq.c +9 -6
@@ -61,7 +61,7 @@ void gve_parse_device_option(struct gve_priv *priv,dev_info (&priv->pdev->dev,"Gqi raw addressing device option enabled.\n" ); - priv->raw_addressing =1 ; + priv->queue_format =GVE_GQI_RDA_FORMAT ;break ;case GVE_DEV_OPT_ID_GQI_RDA : if (option_length < sizeof(**dev_op_gqi_rda) ||
@@ -460,7 +460,8 @@ static int gve_adminq_create_tx_queue(struct gve_priv *priv, u32 queue_index)u32 qpl_id;int err; - qpl_id = priv->raw_addressing ?GVE_RAW_ADDRESSING_QPL_ID : tx->tx_fifo.qpl->id; + qpl_id = priv->queue_format ==GVE_GQI_RDA_FORMAT ? +GVE_RAW_ADDRESSING_QPL_ID : tx->tx_fifo.qpl->id;memset (&cmd,0 ,sizeof (cmd)); cmd.opcode =cpu_to_be32 (GVE_ADMINQ_CREATE_TX_QUEUE ); cmd.create_tx_queue = (struct gve_adminq_create_tx_queue ) {
@@ -501,7 +502,8 @@ static int gve_adminq_create_rx_queue(struct gve_priv *priv, u32 queue_index)u32 qpl_id;int err; - qpl_id = priv->raw_addressing ?GVE_RAW_ADDRESSING_QPL_ID : rx->data.qpl->id; + qpl_id = priv->queue_format ==GVE_GQI_RDA_FORMAT ? +GVE_RAW_ADDRESSING_QPL_ID : rx->data.qpl->id;memset (&cmd,0 ,sizeof (cmd)); cmd.opcode =cpu_to_be32 (GVE_ADMINQ_CREATE_RX_QUEUE ); cmd.create_rx_queue = (struct gve_adminq_create_rx_queue ) {
@@ -628,7 +630,6 @@ int gve_adminq_describe_device(struct gve_priv *priv)if (err) goto free_device_descriptor; - priv->raw_addressing =0 ; err =gve_process_device_options (priv, descriptor, &dev_op_gqi_rda, &dev_op_gqi_qpl, &dev_op_dqo_rda);if (err)
@@ -638,17 +639,19 @@ int gve_adminq_describe_device(struct gve_priv *priv)* is not set to GqiRda, choose the queue format in a priority order: * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default. */ -if (priv->raw_addressing ==1 ) { +if (priv->queue_format ==GVE_GQI_RDA_FORMAT ) {dev_info (&priv->pdev->dev,"Driver is running with GQI RDA queue format.\n" ); }else if (dev_op_dqo_rda) { + priv->queue_format =GVE_DQO_RDA_FORMAT ;dev_info (&priv->pdev->dev,"Driver is running with DQO RDA queue format.\n" ); }else if (dev_op_gqi_rda) { + priv->queue_format =GVE_GQI_RDA_FORMAT ;dev_info (&priv->pdev->dev,"Driver is running with GQI RDA queue format.\n" ); - priv->raw_addressing =1 ; }else { + priv->queue_format =GVE_GQI_QPL_FORMAT ;dev_info (&priv->pdev->dev,"Driver is running with GQI QPL queue format.\n" ); }
drivers/net/ethernet/google/gve/gve_main.c +5 -4
@@ -1,7 +1,7 @@// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* Google virtual Ethernet (gve) driver * -* Copyright (C) 2015-2019 Google, Inc. +* Copyright (C) 2015-2021 Google, Inc. */ #include <linux/cpumask.h>
@@ -681,7 +681,7 @@ static int gve_alloc_qpls(struct gve_priv *priv)int err;/* Raw addressing means no QPLs */ -if (priv->raw_addressing) +if (priv->queue_format ==GVE_GQI_RDA_FORMAT )return 0 ; priv->qpls =kvzalloc (num_qpls *sizeof (*priv->qpls),GFP_KERNEL );
@@ -725,7 +725,7 @@ static void gve_free_qpls(struct gve_priv *priv)int i;/* Raw addressing means no QPLs */ -if (priv->raw_addressing) +if (priv->queue_format ==GVE_GQI_RDA_FORMAT )return ;kvfree (priv->qpl_cfg.qpl_id_map);
@@ -1088,7 +1088,7 @@ static int gve_init_priv(struct gve_priv *priv, bool skip_describe_device)if (skip_describe_device) goto setup_device; - priv->raw_addressing = false; + priv->queue_format =GVE_QUEUE_FORMAT_UNSPECIFIED ;/* Get the initial information we need from the device */ err =gve_adminq_describe_device (priv);if (err) {
@@ -1352,6 +1352,7 @@ static int gve_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto abort_with_wq;dev_info (&pdev->dev,"GVE version %s\n" , gve_version_str); +dev_info (&pdev->dev,"GVE queue format %d\n" , (int )priv->queue_format);gve_clear_probe_in_progress (priv);queue_work (priv->gve_wq, &priv->service_task);return 0 ;
drivers/net/ethernet/google/gve/gve_rx.c +1 -1
@@ -148,7 +148,7 @@ static int gve_rx_alloc_ring(struct gve_priv *priv, int idx) slots = priv->rx_data_slot_cnt; rx->mask = slots -1 ; - rx->data.raw_addressing = priv->raw_addressing; + rx->data.raw_addressing = priv->queue_format ==GVE_GQI_RDA_FORMAT ;/* alloc rx data ring */ bytes =sizeof (*rx->data.data_ring) * slots;
drivers/net/ethernet/google/gve/gve_tx.c +1 -1
@@ -191,7 +191,7 @@ static int gve_tx_alloc_ring(struct gve_priv *priv, int idx)if (!tx->desc) goto abort_with_info; - tx->raw_addressing = priv->raw_addressing; + tx->raw_addressing = priv->queue_format ==GVE_GQI_RDA_FORMAT ; tx->dev = &priv->pdev->dev;if (!tx->raw_addressing) { tx->tx_fifo.qpl =gve_assign_tx_qpl (priv);
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 7/16 gve: adminq: DQO specific device descriptor logic
In reply to Bailey Forrest- In addition to TX and RX queues, DQO has TX completion and RX buffer queues.
- TX completions are received when the device has completed sending a packet on the wire.
- RX buffers are posted on a separate queue form the RX completions.
- DQO descriptor rings are allowed to be smaller than PAGE_SIZE.
--- drivers/net/ethernet/google/gve/gve.h | 13 +++++ drivers/net/ethernet/google/gve/gve_adminq.c | 57 ++++++++++++++------ 2 files changed, 55 insertions(+), 15 deletions(-)
drivers/net/ethernet/google/gve/gve.h +13 -0
@@ -194,6 +194,11 @@ struct gve_qpl_config {unsigned long *qpl_id_map;/* bitmap of used qpl ids */ }; +struct gve_options_dqo_rda { +u16 tx_comp_ring_entries;/* number of tx_comp descriptors */ +u16 rx_buff_ring_entries;/* number of rx_buff descriptors */ +}; +/* GVE_QUEUE_FORMAT_UNSPECIFIED must be zero since 0 is the default value * when the entire configure_device_resources command is zeroed out and the * queue_format is not specified.
@@ -286,6 +291,8 @@ struct gve_priv {/* Gvnic device link speed from hypervisor. */ u64 link_speed; +struct gve_options_dqo_rda options_dqo_rda; +enum gve_queue_format queue_format; };
@@ -533,6 +540,12 @@ static inline enum dma_data_direction gve_qpl_dma_dir(struct gve_priv *priv,return DMA_FROM_DEVICE ; } +static inline bool gve_is_gqi (struct gve_priv *priv) +{ +return priv->queue_format ==GVE_GQI_RDA_FORMAT || + priv->queue_format ==GVE_GQI_QPL_FORMAT ; +} +/* buffers */ int gve_alloc_page(struct gve_priv *priv, struct device *dev, struct page **page, dma_addr_t *dma,
drivers/net/ethernet/google/gve/gve_adminq.c +42 -15
77 lines not on this page · raw message
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 8/16 gve: Add support for DQO RX PTYPE map
In reply to Bailey ForrestUnlike GQI, DQO RX descriptors do not contain the L3 and L4 type of the packet. L3 and L4 types are necessary in order to set the hash and csum on RX SKBs correctly.
DQO RX descriptors instead contain a 10 bit PTYPE index. The PTYPE map enables the device to tell the driver how to map from PTYPE index to L3/L4 type.
The device doesn't provide any guarantees about the range of possible PTYPEs, so we just use a 1024 entry array to implement a fast mapping structure.
--- drivers/net/ethernet/google/gve/gve.h | 15 +++++++ drivers/net/ethernet/google/gve/gve_adminq.c | 45 +++++++++++++++++++- drivers/net/ethernet/google/gve/gve_adminq.h | 44 ++++++++++++++++++- drivers/net/ethernet/google/gve/gve_main.c | 25 +++++++++++ 4 files changed, 127 insertions(+), 2 deletions(-)
drivers/net/ethernet/google/gve/gve.h +15 -0
@@ -11,6 +11,7 @@#include <linux/netdevice.h> #include <linux/pci.h> #include <linux/u64_stats_sync.h> +#include "gve_desc.h" #ifndef PCI_VENDOR_ID_GOOGLE
@@ -40,6 +41,9 @@#define GVE_DATA_SLOT_ADDR_PAGE_MASK (~(PAGE_SIZE - 1)) +/* PTYPEs are always 10 bits. */ +#define GVE_NUM_PTYPES 1024 +/* Each slot in the desc ring has a 1:1 mapping to a slot in the data ring */ struct gve_rx_desc_queue {struct gve_rx_desc *desc_ring;/* the descriptor ring */
@@ -199,6 +203,15 @@ struct gve_options_dqo_rda {u16 rx_buff_ring_entries;/* number of rx_buff descriptors */ }; +struct gve_ptype { +u8 l3_type;/* `gve_l3_type` in gve_adminq.h */ +u8 l4_type;/* `gve_l4_type` in gve_adminq.h */ +}; + +struct gve_ptype_lut { +struct gve_ptype ptypes[GVE_NUM_PTYPES ]; +}; +/* GVE_QUEUE_FORMAT_UNSPECIFIED must be zero since 0 is the default value * when the entire configure_device_resources command is zeroed out and the * queue_format is not specified.
@@ -266,6 +279,7 @@ struct gve_priv {u32 adminq_set_driver_parameter_cnt;u32 adminq_report_stats_cnt;u32 adminq_report_link_speed_cnt; +u32 adminq_get_ptype_map_cnt;/* Global stats */ u32 interface_up_cnt;/* count of times interface turned up since last reset */
@@ -292,6 +306,7 @@ struct gve_priv {u64 link_speed;struct gve_options_dqo_rda options_dqo_rda; +struct gve_ptype_lut *ptype_lut_dqo;enum gve_queue_format queue_format; };
drivers/net/ethernet/google/gve/gve_adminq.c +44 -1
66 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_adminq.h +43 -1
66 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_main.c +25 -0
@@ -346,6 +346,22 @@ static int gve_setup_device_resources(struct gve_priv *priv) err = -ENXIO ; goto abort_with_stats_report; } + +if (priv->queue_format ==GVE_DQO_RDA_FORMAT ) { + priv->ptype_lut_dqo =kvzalloc (sizeof (*priv->ptype_lut_dqo), +GFP_KERNEL ); +if (!priv->ptype_lut_dqo) { + err = -ENOMEM ; + goto abort_with_stats_report; + } + err =gve_adminq_get_ptype_map_dqo (priv, priv->ptype_lut_dqo); +if (err) { +dev_err (&priv->pdev->dev, +"Failed to get ptype map: err=%d\n" , err); + goto abort_with_ptype_lut; + } + } + err =gve_adminq_report_stats (priv, priv->stats_report_len, priv->stats_report_bus,GVE_STATS_REPORT_TIMER_PERIOD );
@@ -354,12 +370,17 @@ static int gve_setup_device_resources(struct gve_priv *priv) "Failed to report stats: err=%d\n", err);gve_set_device_resources_ok (priv);return 0 ; + +abort_with_ptype_lut: +kvfree (priv->ptype_lut_dqo); + priv->ptype_lut_dqo =NULL ; abort_with_stats_report:gve_free_stats_report (priv); abort_with_ntfy_blocks:gve_free_notify_blocks (priv); abort_with_counter:gve_free_counter_array (priv); +return err; }
@@ -386,6 +407,10 @@ static void gve_teardown_device_resources(struct gve_priv *priv)gve_trigger_reset (priv); } } + +kvfree (priv->ptype_lut_dqo); + priv->ptype_lut_dqo =NULL ; +gve_free_counter_array (priv);gve_free_notify_blocks (priv);gve_free_stats_report (priv);
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 9/16 gve: Add dqo descriptors
In reply to Bailey ForrestGeneral description of rings and descriptors:
TX ring is used for sending TX packet buffers to the NIC. It has the following descriptors:
- gve_tx_pkt_desc_dqo - Data buffer descriptor
- gve_tx_tso_context_desc_dqo - TSO context descriptor
- gve_tx_general_context_desc_dqo - Generic metadata descriptor
Metadata is a collection of 12 bytes. We define gve_tx_metadata_dqo which represents the logical interpetation of the metadata bytes. It's helpful to define this structure because the metadata bytes exist in multiple descriptor types (including gve_tx_tso_context_desc_dqo), and the device requires same field has the same value in all descriptors.
The TX completion ring is used to receive completions from the NIC. Having a separate ring allows for completions to be out of order. The completion descriptor gve_tx_compl_desc has several different types, most important are packet and descriptor completions. Descriptor completions are used to notify the driver when descriptors sent on the TX ring are done being consumed. The descriptor completion is only used to signal that space is cleared in the TX ring. A packet completion will be received when a packet transmitted on the TX queue is done being transmitted.
In addition there are "miss" and "reinjection" completions. The device implements a "flow-miss model". Most packets will simply receive a packet completion. The flow-miss system may choose to process a packet based on its contents. A TX packet which experiences a flow miss would receive a miss completion followed by a later reinjection completion. The miss-completion is received when the packet starts to be processed by the flow-miss system and the reinjection completion is received when the flow-miss system completes processing the packet and sends it on the wire.
The RX buffer ring is used to send buffers to HW via the gve_rx_desc_dqo descriptor.
Received packets are put into the RX queue by the device, which populates the gve_rx_compl_desc_dqo descriptor. The RX descriptors refer to buffers posted by the buffer queue. Received buffers may be returned out of order, such as when HW LRO is enabled.
Important concepts:
- "TX" and "RX buffer" queues, which send descriptors to the device, use MMIO doorbells to notify the device of new descriptors.
- "RX" and "TX completion" queues, which receive descriptors from the device, use a "generation bit" to know when a descriptor was populated by the device. The driver initializes all bits with the "current generation". The device will populate received descriptors with the "next generation" which is inverted from the current generation. When the ring wraps, the current/next generation are swapped.
- It's the driver's responsibility to ensure that the RX and TX completion queues are not overrun. This can be accomplished by limiting the number of descriptors posted to HW.
- TX packets have a 16 bit completion_tag and RX buffers have a 16 bit buffer_id. These will be returned on the TX completion and RX queues respectively to let the driver know which packet/buffer was completed.
Bitfields are used to describe descriptor fields. This notation is more concise and readable than shift-and-mask. It is possible because the driver is restricted to little endian platforms.
--- drivers/net/ethernet/google/Kconfig | 2 +- .../net/ethernet/google/gve/gve_desc_dqo.h | 256 ++++++++++++++++++ 2 files changed, 257 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/google/gve/gve_desc_dqo.h
drivers/net/ethernet/google/Kconfig +1 -1
@@ -17,7 +17,7 @@ if NET_VENDOR_GOOGLE config GVE tristate "Google Virtual NIC (gVNIC) support" - depends on PCI_MSI + depends on (PCI_MSI && (X86 || CPU_LITTLE_ENDIAN)) help This driver supports Google Virtual NIC (gVNIC)"
drivers/net/ethernet/google/gve/gve_desc_dqo.h new file +256 -0
256 lines not on this page · raw message
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 10/16 gve: Add DQO fields for core data structures
In reply to Bailey Forrest- Add new DQO datapath structures:
- gve_rx_buf_queue_dqo
- gve_rx_compl_queue_dqo
- gve_rx_buf_state_dqo
- gve_tx_desc_dqo
- gve_tx_pending_packet_dqo
- Incorporate these into the existing ring data structures:
- gve_rx_ring
- gve_tx_ring
Noteworthy mentions:
- gve_rx_buf_state represents an RX buffer which was posted to HW. Each RX queue has an array of these objects and the index into the array is used as the buffer_id when posted to HW.
- gve_tx_pending_packet_dqo is treated similarly for TX queues. The completion_tag is the index into the array.
- These two structures have links for linked lists which are represented by 16b indexes into a contiguous array of these structures. This reduces memory footprint compared to 64b pointers.
- We use unions for the writeable datapath structures to reduce cache footprint. GQI specific members will renamed like DQO members in a future patch.
--- drivers/net/ethernet/google/gve/gve.h | 262 ++++++++++++++++++++++++-- 1 file changed, 251 insertions(+), 11 deletions(-)
drivers/net/ethernet/google/gve/gve.h +251 -11
@@ -13,6 +13,7 @@#include <linux/u64_stats_sync.h> #include "gve_desc.h" +#include "gve_desc_dqo.h" #ifndef PCI_VENDOR_ID_GOOGLE #define PCI_VENDOR_ID_GOOGLE 0x1ae0
@@ -80,17 +81,117 @@ struct gve_rx_data_queue {struct gve_priv ; -/* An RX ring that contains a power-of-two sized desc and data ring. */ +/* RX buffer queue for posting buffers to HW. +* Each RX (completion) queue has a corresponding buffer queue. +*/ +struct gve_rx_buf_queue_dqo { +struct gve_rx_desc_dqo *desc_ring; +dma_addr_t bus; +u32 head;/* Pointer to start cleaning buffers at. */ +u32 tail;/* Last posted buffer index + 1 */ +u32 mask;/* Mask for indices to the size of the ring */ +}; + +/* RX completion queue to receive packets from HW. */ +struct gve_rx_compl_queue_dqo { +struct gve_rx_compl_desc_dqo *desc_ring; +dma_addr_t bus; + +/* Number of slots which did not have a buffer posted yet. We should not +* post more buffers than the queue size to avoid HW overrunning the +* queue. +*/ +int num_free_slots; + +/* HW uses a "generation bit" to notify SW of new descriptors. When a +* descriptor's generation bit is different from the current generation, +* that descriptor is ready to be consumed by SW. +*/ +u8 cur_gen_bit; + +/* Pointer into desc_ring where the next completion descriptor will be +* received. +*/ +u32 head; +u32 mask;/* Mask for indices to the size of the ring */ +}; + +/* Stores state for tracking buffers posted to HW */ +struct gve_rx_buf_state_dqo { +/* The page posted to HW. */ +struct gve_rx_slot_page_info page_info; + +/* The DMA address corresponding to `page_info`. */ +dma_addr_t addr; + +/* Last offset into the page when it only had a single reference, at +* which point every other offset is free to be reused. +*/ +u32 last_single_ref_offset; + +/* Linked list index to next element in the list, or -1 if none */ +s16 next; +}; + +/* `head` and `tail` are indices into an array, or -1 if empty. */ +struct gve_index_list { +s16 head; +s16 tail; +}; + +/* Contains datapath state used to represent an RX queue. */ struct gve_rx_ring {struct gve_priv *gve; -struct gve_rx_desc_queue desc; -struct gve_rx_data_queue data; +union { +/* GQI fields */ +struct { +struct gve_rx_desc_queue desc; +struct gve_rx_data_queue data; + +/* threshold for posting new buffs and descs */ +u32 db_threshold; + }; + +/* DQO fields. */ +struct { +struct gve_rx_buf_queue_dqo bufq; +struct gve_rx_compl_queue_dqo complq; + +struct gve_rx_buf_state_dqo *buf_states; +u16 num_buf_states; + +/* Linked list of gve_rx_buf_state_dqo. Index into +* buf_states, or -1 if empty. +*/ +s16 free_buf_states; + +/* Linked list of gve_rx_buf_state_dqo. Indexes into +* buf_states, or -1 if empty. +* +* This list contains buf_states which are pointing to +* valid buffers. +* +* We use a FIFO here in order to increase the +* probability that buffers can be reused by increasing +* the time between usages. +*/ +struct gve_index_list recycled_buf_states; + +/* Linked list of gve_rx_buf_state_dqo. Indexes into +* buf_states, or -1 if empty. +* +* This list contains buf_states which have buffers +* which cannot be reused yet. +*/ +struct gve_index_list used_buf_states; + } dqo; + }; +u64 rbytes;/* free-running bytes received */ u64 rpackets;/* free-running packets received */ u32 cnt;/* free-running total number of completed packets */ u32 fill_cnt;/* free-running total number of descs and buffs posted */ u32 mask;/* masks the cnt and fill_cnt to the size of the ring */ -u32 db_threshold;/* threshold for posting new buffs and descs */ u64 rx_copybreak_pkt;/* free-running count of copybreak packets */ u64 rx_copied_pkt;/* free-running total number of copied packets */ u64 rx_skb_alloc_fail;/* free-running count of skb alloc fails */
@@ -141,23 +242,161 @@ struct gve_tx_fifo {struct gve_queue_page_list *qpl;/* QPL mapped into this FIFO */ }; -/* A TX ring that contains a power-of-two sized desc ring and a FIFO buffer */ +/* TX descriptor for DQO format */ +union gve_tx_desc_dqo { +struct gve_tx_pkt_desc_dqo pkt; +struct gve_tx_tso_context_desc_dqo tso_ctx; +struct gve_tx_general_context_desc_dqo general_ctx; +}; + +enum gve_packet_state { +/* Packet is in free list, available to be allocated. +* This should always be zero since state is not explicitly initialized. +*/ +GVE_PACKET_STATE_UNALLOCATED , +/* Packet is expecting a regular data completion or miss completion */ +GVE_PACKET_STATE_PENDING_DATA_COMPL , +/* Packet has received a miss completion and is expecting a +* re-injection completion. +*/ +GVE_PACKET_STATE_PENDING_REINJECT_COMPL , +/* No valid completion received within the specified timeout. */ +GVE_PACKET_STATE_TIMED_OUT_COMPL , +}; + +struct gve_tx_pending_packet_dqo { +struct sk_buff *skb;/* skb for this packet */ + +/* 0th element corresponds to the linear portion of `skb`, should be +* unmapped with `dma_unmap_single`. +* +* All others correspond to `skb`'s frags and should be unmapped with +* `dma_unmap_page`. +*/ +struct gve_tx_dma_buf bufs[MAX_SKB_FRAGS +1 ]; +u16 num_bufs; + +/* Linked list index to next element in the list, or -1 if none */ +s16 next; + +/* Linked list index to prev element in the list, or -1 if none. +* Used for tracking either outstanding miss completions or prematurely +* freed packets. +*/ +s16 prev; + +/* Identifies the current state of the packet as defined in +* `enum gve_packet_state`. +*/ +u8 state; + +/* If packet is an outstanding miss completion, then the packet is +* freed if the corresponding re-injection completion is not received +* before kernel jiffies exceeds timeout_jiffies. +*/ +unsigned long timeout_jiffies; +}; + +/* Contains datapath state used to represent a TX queue. */ struct gve_tx_ring {/* Cacheline 0 -- Accessed & dirtied during transmit */ -struct gve_tx_fifo tx_fifo; -u32 req;/* driver tracked head pointer */ -u32 done;/* driver tracked tail pointer */ +union { +/* GQI fields */ +struct { +struct gve_tx_fifo tx_fifo; +u32 req;/* driver tracked head pointer */ +u32 done;/* driver tracked tail pointer */ + }; + +/* DQO fields. */ +struct { +/* Linked list of gve_tx_pending_packet_dqo. Index into +* pending_packets, or -1 if empty. +* +* This is a consumer list owned by the TX path. When it +* runs out, the producer list is stolen from the +* completion handling path +* (dqo_compl.free_pending_packets). +*/ +s16 free_pending_packets; + +/* Cached value of `dqo_compl.hw_tx_head` */ +u32 head; +u32 tail;/* Last posted buffer index + 1 */ + +/* Index of the last descriptor with "report event" bit +* set. +*/ +u32 last_re_idx; + } dqo_tx; + };/* Cacheline 1 -- Accessed & dirtied during gve_clean_tx_done */ -__be32 last_nic_done____cacheline_aligned ;/* NIC tail pointer */ +union { +/* GQI fields */ +struct { +/* NIC tail pointer */ +__be32 last_nic_done; + }; + +/* DQO fields. */ +struct { +u32 head;/* Last read on compl_desc */ + +/* Tracks the current gen bit of compl_q */ +u8 cur_gen_bit; + +/* Linked list of gve_tx_pending_packet_dqo. Index into +* pending_packets, or -1 if empty. +* +* This is the producer list, owned by the completion +* handling path. When the consumer list +* (dqo_tx.free_pending_packets) is runs out, this list +* will be stolen. +*/ +atomic_t free_pending_packets; + +/* Last TX ring index fetched by HW */ +atomic_t hw_tx_head; + +/* List to track pending packets which received a miss +* completion but not a corresponding reinjection. +*/ +struct gve_index_list miss_completions; + +/* List to track pending packets that were completed +* before receiving a valid completion because they +* reached a specified timeout. +*/ +struct gve_index_list timed_out_completions; + } dqo_compl; + }____cacheline_aligned ;u64 pkt_done;/* free-running - total packets completed */ u64 bytes_done;/* free-running - total bytes completed */ u64 dropped_pkt;/* free-running - total packets dropped */ u64 dma_mapping_error;/* count of dma mapping errors */ /* Cacheline 2 -- Read-mostly fields */ -union gve_tx_desc *desc____cacheline_aligned ; -struct gve_tx_buffer_state *info;/* Maps 1:1 to a desc */ +union { +/* GQI fields */ +struct { +union gve_tx_desc *desc; + +/* Maps 1:1 to a desc */ +struct gve_tx_buffer_state *info; + }; + +/* DQO fields. */ +struct { +union gve_tx_desc_dqo *tx_ring; +struct gve_tx_compl_desc *compl_ring; + +struct gve_tx_pending_packet_dqo *pending_packets; +s16 num_pending_packets; + +u32 complq_mask;/* complq size is complq_mask + 1 */ + } dqo; + }____cacheline_aligned ;struct netdev_queue *netdev_txq;struct gve_queue_resources *q_resources;/* head and tail pointer idx */ struct device *dev;
@@ -171,6 +410,7 @@ struct gve_tx_ring {u32 ntfy_id;/* notification block index */ dma_addr_t bus;/* dma address of the descr ring */ dma_addr_t q_resources_bus;/* dma address of the queue resources */ +dma_addr_t complq_bus_dqo;/* dma address of the dqo.compl_ring */ struct u64_stats_sync statss;/* sync stats for 32bit archs */ }____cacheline_aligned ;
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 11/16 gve: Update adminq commands to support DQO queues
In reply to Bailey ForrestDQO queue creation requires additional parameters:
- TX completion/RX buffer queue size
- TX completion/RX buffer queue address
- TX/RX queue size
- RX buffer size
--- drivers/net/ethernet/google/gve/gve.h | 3 + drivers/net/ethernet/google/gve/gve_adminq.c | 63 ++++++++++++------- drivers/net/ethernet/google/gve/gve_adminq.h | 18 ++++-- drivers/net/ethernet/google/gve/gve_ethtool.c | 9 ++- 4 files changed, 64 insertions(+), 29 deletions(-)
drivers/net/ethernet/google/gve/gve.h +3 -0
@@ -548,6 +548,9 @@ struct gve_priv {struct gve_options_dqo_rda options_dqo_rda;struct gve_ptype_lut *ptype_lut_dqo; +/* Must be a power of two. */ +int data_buffer_size_dqo; +enum gve_queue_format queue_format; };
drivers/net/ethernet/google/gve/gve_adminq.c +40 -23
103 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_adminq.h +14 -4
@@ -139,9 +139,11 @@ struct gve_adminq_configure_device_resources {__be32 num_irq_dbs;__be32 irq_db_stride;__be32 ntfy_blk_msix_base_idx; +u8 queue_format; +u8 padding[7 ]; }; -static_assert (sizeof (struct gve_adminq_configure_device_resources ) ==32 ); +static_assert (sizeof (struct gve_adminq_configure_device_resources ) ==40 );struct gve_adminq_register_page_list {__be32 page_list_id;
@@ -166,9 +168,13 @@ struct gve_adminq_create_tx_queue {__be64 tx_ring_addr;__be32 queue_page_list_id;__be32 ntfy_id; +__be64 tx_comp_ring_addr; +__be16 tx_ring_size; +__be16 tx_comp_ring_size; +u8 padding[4 ]; }; -static_assert (sizeof (struct gve_adminq_create_tx_queue ) ==32 ); +static_assert (sizeof (struct gve_adminq_create_tx_queue ) ==48 );struct gve_adminq_create_rx_queue {__be32 queue_id;
@@ -179,10 +185,14 @@ struct gve_adminq_create_rx_queue {__be64 rx_desc_ring_addr;__be64 rx_data_ring_addr;__be32 queue_page_list_id; -u8 padding[4 ]; +__be16 rx_ring_size; +__be16 packet_buffer_size; +__be16 rx_buff_ring_size; +u8 enable_rsc; +u8 padding[5 ]; }; -static_assert (sizeof (struct gve_adminq_create_rx_queue ) ==48 ); +static_assert (sizeof (struct gve_adminq_create_rx_queue ) ==56 );/* Queue resources that are shared with the device */ struct gve_queue_resources {
drivers/net/ethernet/google/gve/gve_ethtool.c +7 -2
@@ -1,7 +1,7 @@// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* Google virtual Ethernet (gve) driver * -* Copyright (C) 2015-2019 Google, Inc. +* Copyright (C) 2015-2021 Google, Inc. */ #include <linux/ethtool.h>
@@ -453,11 +453,16 @@ static int gve_set_tunable(struct net_device *netdev,switch (etuna->id) {case ETHTOOL_RX_COPYBREAK : + { +u32 max_copybreak =gve_is_gqi (priv) ? + (PAGE_SIZE /2 ) : priv->data_buffer_size_dqo; + len = *(u32 *)value; -if (len >PAGE_SIZE /2 ) +if (len > max_copybreak)return -EINVAL ; priv->rx_copybreak = len;return 0 ; + }default :return -EOPNOTSUPP ; }
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 12/16 gve: DQO: Add core netdev features
In reply to Bailey ForrestAdd napi netdev device registration, interrupt handling and initial tx and rx polling stubs. The stubs will be filled in follow-on patches.
Also:
- LRO feature advertisement and handling
- Also update ethtool logic
--- drivers/net/ethernet/google/gve/Makefile | 2 +- drivers/net/ethernet/google/gve/gve.h | 2 + drivers/net/ethernet/google/gve/gve_adminq.c | 2 + drivers/net/ethernet/google/gve/gve_dqo.h | 32 +++ drivers/net/ethernet/google/gve/gve_ethtool.c | 12 +- drivers/net/ethernet/google/gve/gve_main.c | 188 ++++++++++++++++-- drivers/net/ethernet/google/gve/gve_rx_dqo.c | 24 +++ drivers/net/ethernet/google/gve/gve_tx_dqo.c | 23 +++ 8 files changed, 260 insertions(+), 25 deletions(-) create mode 100644 drivers/net/ethernet/google/gve/gve_dqo.h create mode 100644 drivers/net/ethernet/google/gve/gve_rx_dqo.c create mode 100644 drivers/net/ethernet/google/gve/gve_tx_dqo.c
drivers/net/ethernet/google/gve/Makefile +1 -1
@@ -1,4 +1,4 @@ # Makefile for the Google virtual Ethernet (gve) driver obj-$(CONFIG_GVE) += gve.o -gve-objs := gve_main.o gve_tx.o gve_rx.o gve_ethtool.o gve_adminq.o gve_utils.o +gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.o
drivers/net/ethernet/google/gve/gve.h +2 -0
@@ -45,6 +45,8 @@/* PTYPEs are always 10 bits. */ #define GVE_NUM_PTYPES 1024 +#define GVE_RX_BUFFER_SIZE_DQO 2048 +/* Each slot in the desc ring has a 1:1 mapping to a slot in the data ring */ struct gve_rx_desc_queue {struct gve_rx_desc *desc_ring;/* the descriptor ring */
drivers/net/ethernet/google/gve/gve_adminq.c +2 -0
@@ -714,6 +714,8 @@ int gve_adminq_describe_device(struct gve_priv *priv)if (gve_is_gqi (priv)) { err =gve_set_desc_cnt (priv, descriptor); }else { +/* DQO supports LRO. */ + priv->dev->hw_features |=NETIF_F_LRO ; err =gve_set_desc_cnt_dqo (priv, descriptor, dev_op_dqo_rda); }if (err)
drivers/net/ethernet/google/gve/gve_dqo.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR MIT) +* Google virtual Ethernet (gve) driver +* +* Copyright (C) 2015-2021 Google, Inc. +*/ + +#ifndef _GVE_DQO_H_ +#define _GVE_DQO_H_ + +#include "gve_adminq.h" + +#define GVE_ITR_ENABLE_BIT_DQO BIT(0) +#define GVE_ITR_CLEAR_PBA_BIT_DQO BIT(1) +#define GVE_ITR_NO_UPDATE_DQO (3 << 3) + +#define GVE_TX_IRQ_RATELIMIT_US_DQO 50 +#define GVE_RX_IRQ_RATELIMIT_US_DQO 20 + +netdev_tx_t gve_tx_dqo (struct sk_buff *skb,struct net_device *dev); +bool gve_tx_poll_dqo (struct gve_notify_block *block,bool do_clean); +int gve_rx_poll_dqo (struct gve_notify_block *block,int budget); + +static inline void +gve_write_irq_doorbell_dqo (const struct gve_priv *priv, +const struct gve_notify_block *block,u32 val) +{ +u32 index =be32_to_cpu (block->irq_db_index); + +iowrite32 (val, &priv->db_bar2[index]); +} + +#endif /* _GVE_DQO_H_ */
drivers/net/ethernet/google/gve/gve_ethtool.c +10 -2
@@ -311,8 +311,16 @@ gve_get_ethtool_stats(struct net_device *netdev,for (ring =0 ; ring < priv->tx_cfg.num_queues; ring++) {struct gve_tx_ring *tx = &priv->tx[ring]; - data[i++] = tx->req; - data[i++] = tx->done; +if (gve_is_gqi (priv)) { + data[i++] = tx->req; + data[i++] = tx->done; + }else { +/* DQO doesn't currently support +* posted/completed descriptor counts; +*/ + data[i++] =0 ; + data[i++] =0 ; + } do { start = u64_stats_fetch_begin(&priv->tx[ring].statss);
drivers/net/ethernet/google/gve/gve_main.c +166 -22
292 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_rx_dqo.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) +/* Google virtual Ethernet (gve) driver +* +* Copyright (C) 2015-2021 Google, Inc. +*/ + +#include "gve.h" +#include "gve_dqo.h" +#include "gve_adminq.h" +#include "gve_utils.h" +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <linux/skbuff.h> +#include <linux/slab.h> +#include <net/ip6_checksum.h> +#include <net/ipv6.h> +#include <net/tcp.h> + +int gve_rx_poll_dqo (struct gve_notify_block *block,int budget) +{ +u32 work_done =0 ; + +return work_done; +}
drivers/net/ethernet/google/gve/gve_tx_dqo.c new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) +/* Google virtual Ethernet (gve) driver +* +* Copyright (C) 2015-2021 Google, Inc. +*/ + +#include "gve.h" +#include "gve_adminq.h" +#include "gve_utils.h" +#include "gve_dqo.h" +#include <linux/tcp.h> +#include <linux/slab.h> +#include <linux/skbuff.h> + +netdev_tx_t gve_tx_dqo (struct sk_buff *skb,struct net_device *dev) +{ +return NETDEV_TX_OK ; +} + +bool gve_tx_poll_dqo (struct gve_notify_block *block,bool do_clean) +{ +return false; +}
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
Re: [PATCH net-next 12/16] gve: DQO: Add core netdev features
Quoted 52 lines · Bailey Forrest
Add napi netdev device registration, interrupt handling and initial tx and rx polling stubs. The stubs will be filled in follow-on patches.
Also:
- LRO feature advertisement and handling
- Also update ethtool logicSigned-off-by Bailey Forrest <[email protected]>Reviewed-by Willem de Bruijn <[email protected]>Reviewed-by Catherine Sullivan <[email protected]>--- drivers/net/ethernet/google/gve/Makefile | 2 +- drivers/net/ethernet/google/gve/gve.h | 2 + drivers/net/ethernet/google/gve/gve_adminq.c | 2 + drivers/net/ethernet/google/gve/gve_dqo.h | 32 +++ drivers/net/ethernet/google/gve/gve_ethtool.c | 12 +- drivers/net/ethernet/google/gve/gve_main.c | 188 ++++++++++++++++-- drivers/net/ethernet/google/gve/gve_rx_dqo.c | 24 +++ drivers/net/ethernet/google/gve/gve_tx_dqo.c | 23 +++ 8 files changed, 260 insertions(+), 25 deletions(-)create mode 100644
drivers/net/ethernet/google/gve/gve_dqo.hcreate mode 100644drivers/net/ethernet/google/gve/gve_rx_dqo.ccreate mode 100644drivers/net/ethernet/google/gve/gve_tx_dqo.cdrivers/net/ethernet/google/gve/Makefile +1
-1@@ -1,4 +1,4 @@ # Makefile for the Google virtual Ethernet (gve) driver obj-$(CONFIG_GVE) += gve.o -gve-objs := gve_main.o gve_tx.o gve_rx.o gve_ethtool.o gve_adminq.o gve_utils.o +gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.odrivers/net/ethernet/google/gve/gve.h +2
-0@@ -45,6 +45,8 @@/* PTYPEs are always 10 bits. */ #define GVE_NUM_PTYPES 1024 +#define GVE_RX_BUFFER_SIZE_DQO 2048 +/* Each slot in the desc ring has a 1:1 mapping to a slot in the data ring */ struct gve_rx_desc_queue {struct gve_rx_desc *desc_ring;/* the descriptor ring */ drivers/net/ethernet/google/gve/gve_adminq.c +0
-0
drivers/net/ethernet/google/gve/gve_adminq.c int gve_adminq_describe_device(struct gve_priv *priv) · line 718
Quoted 5 lines
drivers/net/ethernet/google/gve/gve_adminq.c +2
-0@@ -714,6 +714,8 @@ int gve_adminq_describe_device(struct gve_priv *priv)if (gve_is_gqi (priv)) { err =gve_set_desc_cnt (priv, descriptor); }else { +/* DQO supports LRO. */ + priv->dev->hw_features |=NETIF_F_LRO ;
Shouldn't this be NETIF_F_HW_GRO?
Also, what does DQO stands for?
<snip>
Re: [PATCH net-next 12/16] gve: DQO: Add core netdev features
Quoted 54 lines · Sridhar
Quoted 52 lines · Bailey Forrest
Add napi netdev device registration, interrupt handling and initial tx and rx polling stubs. The stubs will be filled in follow-on patches.
Also:
- LRO feature advertisement and handling
- Also update ethtool logicSigned-off-by Bailey Forrest <[email protected]>Reviewed-by Willem de Bruijn <[email protected]>Reviewed-by Catherine Sullivan <[email protected]>--- drivers/net/ethernet/google/gve/Makefile | 2 +- drivers/net/ethernet/google/gve/gve.h | 2 + drivers/net/ethernet/google/gve/gve_adminq.c | 2 + drivers/net/ethernet/google/gve/gve_dqo.h | 32 +++ drivers/net/ethernet/google/gve/gve_ethtool.c | 12 +- drivers/net/ethernet/google/gve/gve_main.c | 188 ++++++++++++++++-- drivers/net/ethernet/google/gve/gve_rx_dqo.c | 24 +++ drivers/net/ethernet/google/gve/gve_tx_dqo.c | 23 +++ 8 files changed, 260 insertions(+), 25 deletions(-)create mode 100644
drivers/net/ethernet/google/gve/gve_dqo.hcreate mode 100644drivers/net/ethernet/google/gve/gve_rx_dqo.ccreate mode 100644drivers/net/ethernet/google/gve/gve_tx_dqo.cdrivers/net/ethernet/google/gve/Makefile +1
-1@@ -1,4 +1,4 @@ # Makefile for the Google virtual Ethernet (gve) driver obj-$(CONFIG_GVE) += gve.o -gve-objs := gve_main.o gve_tx.o gve_rx.o gve_ethtool.o gve_adminq.o gve_utils.o +gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.odrivers/net/ethernet/google/gve/gve.h +2
-0@@ -45,6 +45,8 @@/* PTYPEs are always 10 bits. */ #define GVE_NUM_PTYPES 1024 +#define GVE_RX_BUFFER_SIZE_DQO 2048 +/* Each slot in the desc ring has a 1:1 mapping to a slot in the data ring */ struct gve_rx_desc_queue {struct gve_rx_desc *desc_ring;/* the descriptor ring */ drivers/net/ethernet/google/gve/gve_adminq.c +0
-0@@ -714,6 +714,8 @@ int gve_adminq_describe_device(struct gve_priv *priv)
Quoted 8 lines
drivers/net/ethernet/google/gve/gve_adminq.c int gve_adminq_describe_device(struct gve_priv *priv) · line 718
Quoted 5 lines
drivers/net/ethernet/google/gve/gve_adminq.c +2
-0if (gve_is_gqi (priv)) { err =gve_set_desc_cnt (priv, descriptor); }else { +/* DQO supports LRO. */ + priv->dev->hw_features |=NETIF_F_LRO ;Shouldn't this be
NETIF_F_HW_GRO?
Also, what does DQO stands for?
DQO stands for "Dual Queue Out of order completions"
For now we only support LRO. HW GRO support may come in the future.
The reason for this is I was unable to get HW GRO to function in conjunction with napi_gro_frags(). On our system, we need to use napi_gro_frags() to achieve good performance. I did not see any other drivers which support NETIF_F_HW_GRO and also use napi_gro_frags().
Quoted 3 lines
<snip>
PATCH 13/16 gve: DQO: Add ring allocation and initialization
In reply to Bailey ForrestAllocate the buffer and completion ring structures. Do not populate the rings yet. That will happen in the respective rx and tx datapath follow-on patches
--- drivers/net/ethernet/google/gve/gve.h | 8 +- drivers/net/ethernet/google/gve/gve_dqo.h | 18 ++ drivers/net/ethernet/google/gve/gve_main.c | 53 ++++- drivers/net/ethernet/google/gve/gve_rx.c | 2 +- drivers/net/ethernet/google/gve/gve_rx_dqo.c | 157 +++++++++++++++ drivers/net/ethernet/google/gve/gve_tx.c | 2 +- drivers/net/ethernet/google/gve/gve_tx_dqo.c | 193 +++++++++++++++++++ 7 files changed, 420 insertions(+), 13 deletions(-)
drivers/net/ethernet/google/gve/gve.h +6 -2
@@ -204,6 +204,10 @@ struct gve_rx_ring {struct gve_queue_resources *q_resources;/* head and tail pointer idx */ dma_addr_t q_resources_bus;/* dma address for the queue resources */ struct u64_stats_sync statss;/* sync stats for 32bit archs */ + +/* head and tail of skb chain for the current packet or NULL if none */ +struct sk_buff *skb_head; +struct sk_buff *skb_tail; };/* A TX desc ring entry */
@@ -816,14 +820,14 @@ void gve_free_page(struct device *dev, struct page *page, dma_addr_t dma,netdev_tx_t gve_tx (struct sk_buff *skb,struct net_device *dev);bool gve_tx_poll (struct gve_notify_block *block,int budget);int gve_tx_alloc_rings (struct gve_priv *priv); -void gve_tx_free_rings (struct gve_priv *priv); +void gve_tx_free_rings_gqi (struct gve_priv *priv);__be32 gve_tx_load_event_counter (struct gve_priv *priv,struct gve_tx_ring *tx);/* rx handling */ void gve_rx_write_doorbell (struct gve_priv *priv,struct gve_rx_ring *rx);bool gve_rx_poll (struct gve_notify_block *block,int budget);int gve_rx_alloc_rings (struct gve_priv *priv); -void gve_rx_free_rings (struct gve_priv *priv); +void gve_rx_free_rings_gqi (struct gve_priv *priv);bool gve_clean_rx_done (struct gve_rx_ring *rx,int budget,netdev_features_t feat);/* Reset */
drivers/net/ethernet/google/gve/gve_dqo.h +18 -0
@@ -19,6 +19,24 @@netdev_tx_t gve_tx_dqo (struct sk_buff *skb,struct net_device *dev);bool gve_tx_poll_dqo (struct gve_notify_block *block,bool do_clean);int gve_rx_poll_dqo (struct gve_notify_block *block,int budget); +int gve_tx_alloc_rings_dqo (struct gve_priv *priv); +void gve_tx_free_rings_dqo (struct gve_priv *priv); +int gve_rx_alloc_rings_dqo (struct gve_priv *priv); +void gve_rx_free_rings_dqo (struct gve_priv *priv); +int gve_clean_tx_done_dqo (struct gve_priv *priv,struct gve_tx_ring *tx, +struct napi_struct *napi); +void gve_rx_post_buffers_dqo (struct gve_rx_ring *rx); +void gve_rx_write_doorbell_dqo (const struct gve_priv *priv,int queue_idx); + +static inline void +gve_tx_put_doorbell_dqo (const struct gve_priv *priv, +const struct gve_queue_resources *q_resources,u32 val) +{ +u64 index; + + index =be32_to_cpu (q_resources->db_index); +iowrite32 (val, &priv->db_bar2[index]); +} static inline void gve_write_irq_doorbell_dqo(const struct gve_priv *priv,
drivers/net/ethernet/google/gve/gve_main.c +44 -9
91 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_rx.c +1 -1
@@ -238,7 +238,7 @@ int gve_rx_alloc_rings(struct gve_priv *priv)return err; } -void gve_rx_free_rings (struct gve_priv *priv) +void gve_rx_free_rings_gqi (struct gve_priv *priv) {int i;
drivers/net/ethernet/google/gve/gve_rx_dqo.c +157 -0
163 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_tx.c +1 -1
@@ -256,7 +256,7 @@ int gve_tx_alloc_rings(struct gve_priv *priv)return err; } -void gve_tx_free_rings (struct gve_priv *priv) +void gve_tx_free_rings_gqi (struct gve_priv *priv) {int i;
drivers/net/ethernet/google/gve/gve_tx_dqo.c +193 -0
204 lines not on this page · raw message
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 14/16 gve: DQO: Configure interrupts on device up
In reply to Bailey ForrestWhen interrupts are first enabled, we also set the ratelimits, which will be static for the entire usage of the device.
--- drivers/net/ethernet/google/gve/gve_dqo.h | 19 +++++++++++++++++++ drivers/net/ethernet/google/gve/gve_main.c | 16 ++++++++++++++-- 2 files changed, 33 insertions(+), 2 deletions(-)
drivers/net/ethernet/google/gve/gve_dqo.h +19 -0
@@ -13,6 +13,9 @@#define GVE_ITR_CLEAR_PBA_BIT_DQO BIT(1)#define GVE_ITR_NO_UPDATE_DQO (3 << 3) +#define GVE_ITR_INTERVAL_DQO_SHIFT 5 +#define GVE_ITR_INTERVAL_DQO_MASK ((1 << 12) - 1) +#define GVE_TX_IRQ_RATELIMIT_US_DQO 50#define GVE_RX_IRQ_RATELIMIT_US_DQO 20
@@ -38,6 +41,22 @@ gve_tx_put_doorbell_dqo(const struct gve_priv *priv,iowrite32 (val, &priv->db_bar2[index]); } +/* Builds register value to write to DQO IRQ doorbell to enable with specified +* ratelimit. +*/ +static inline u32 gve_set_itr_ratelimit_dqo (u32 ratelimit_us) +{ +u32 result =GVE_ITR_ENABLE_BIT_DQO ; + +/* Interval has 2us granularity. */ + ratelimit_us >>=1 ; + + ratelimit_us &=GVE_ITR_INTERVAL_DQO_MASK ; + result |= (ratelimit_us <<GVE_ITR_INTERVAL_DQO_SHIFT ); + +return result; +} +static inline void gve_write_irq_doorbell_dqo (const struct gve_priv *priv,const struct gve_notify_block *block,u32 val)
drivers/net/ethernet/google/gve/gve_main.c +14 -2
@@ -1077,14 +1077,26 @@ static void gve_turnup(struct gve_priv *priv)struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];napi_enable (&block->napi); -iowrite32be (0 ,gve_irq_doorbell (priv, block)); +if (gve_is_gqi (priv)) { +iowrite32be (0 ,gve_irq_doorbell (priv, block)); + }else { +u32 val =gve_set_itr_ratelimit_dqo (GVE_TX_IRQ_RATELIMIT_US_DQO ); + +gve_write_irq_doorbell_dqo (priv, block, val); + } }for (idx =0 ; idx < priv->rx_cfg.num_queues; idx++) {int ntfy_idx =gve_rx_idx_to_ntfy (priv, idx);struct gve_notify_block *block = &priv->ntfy_blocks[ntfy_idx];napi_enable (&block->napi); -iowrite32be (0 ,gve_irq_doorbell (priv, block)); +if (gve_is_gqi (priv)) { +iowrite32be (0 ,gve_irq_doorbell (priv, block)); + }else { +u32 val =gve_set_itr_ratelimit_dqo (GVE_RX_IRQ_RATELIMIT_US_DQO ); + +gve_write_irq_doorbell_dqo (priv, block, val); + } }gve_set_napi_enabled (priv);
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
PATCH 15/16 gve: DQO: Add TX path
In reply to Bailey ForrestTX SKBs will have their buffers DMA mapped with the device. Each buffer will have at least one TX descriptor associated. Each SKB will also have a metadata descriptor.
Each TX queue maintains an array of gve_tx_pending_packet_dqo objects. Every TX SKB will have an associated pending_packet object. A TX SKB's descriptors will use its pending_packet's index as the completion tag, which will be returned on the TX completion queue.
The device implements a "flow-miss model". Most packets will simply receive a packet completion. The flow-miss system may choose to process a packet based on its contents. A TX packet which experiences a flow miss would receive a miss completion followed by a later reinjection completion. The miss-completion is received when the packet starts to be processed by the flow-miss system and the reinjection completion is received when the flow-miss system completes processing the packet and sends it on the wire.
Notable mentions:
- Buffers may be freed after receiving the miss-completion, but in order to avoid packet reordering, we do not complete the SKB until receiving the reinjection completion.
- The driver must robustly handle the unlikely scenario where a miss completion does not have an associated reinjection completion. This is accomplished by maintaining a list of packets which have a pending reinjection completion. After a short timeout (5 seconds), the SKB and buffers are released and the pending_packet is moved to a second list which has a longer timeout (60 seconds), where the pending_packet will not be reused. When the longer timeout elapses, the driver may assume the reinjection completion would never be received and the pending_packet may be reused.
- Completion handling is triggered by an interrupt and is done in the NAPI poll function. Because the TX path and completion exist in different threading contexts they maintain their own lists for free pending_packet objects. The TX path uses a lock-free approach to steal the list from the completion path.
- Both the TSO context and general context descriptors have metadata bytes. The device requires that if multiple descriptors contain the same field, each descriptor must have the same value set for that field.
--- drivers/net/ethernet/google/gve/gve_dqo.h | 12 + drivers/net/ethernet/google/gve/gve_tx_dqo.c | 819 ++++++++++++++++++- 2 files changed, 829 insertions(+), 2 deletions(-)
drivers/net/ethernet/google/gve/gve_dqo.h +12 -0
@@ -19,6 +19,18 @@#define GVE_TX_IRQ_RATELIMIT_US_DQO 50#define GVE_RX_IRQ_RATELIMIT_US_DQO 20 +/* Timeout in seconds to wait for a reinjection completion after receiving +* its corresponding miss completion. +*/ +#define GVE_REINJECT_COMPL_TIMEOUT 1 + +/* Timeout in seconds to deallocate the completion tag for a packet that was +* prematurely freed for not receiving a valid completion. This should be large +* enough to rule out the possibility of receiving the corresponding valid +* completion after this interval. +*/ +#define GVE_DEALLOCATE_COMPL_TIMEOUT 60 +netdev_tx_t gve_tx_dqo (struct sk_buff *skb,struct net_device *dev);bool gve_tx_poll_dqo (struct gve_notify_block *block,bool do_clean);int gve_rx_poll_dqo (struct gve_notify_block *block,int budget);
drivers/net/ethernet/google/gve/gve_tx_dqo.c +817 -2
841 lines not on this page · raw message
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
kernel test robot
build botRe: [PATCH net-next 15/16] gve: DQO: Add TX path
Thank you for the patch! Perhaps something to improve:
Hi Bailey,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on 35713d9b8f090d7a226e4aaeeb742265cde33c82]
url: https://github.com/0day-ci/linux/commits/Bailey-Forrest/gve-Introduce-DQO-descriptor-format/20210625-021110 base: 35713d9b8f090d7a226e4aaeeb742265cde33c82 config: i386-randconfig-a011-20210622 (attached as .config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/af0833aafca5d9abd931a16ee9e761e85f5ad965 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Bailey-Forrest/gve-Introduce-DQO-descriptor-format/20210625-021110 git checkout af0833aafca5d9abd931a16ee9e761e85f5ad965 # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
drivers/net/ethernet/google/gve/gve_tx_dqo.c: In function 'gve_tx_clean_pending_packets':
drivers/net/ethernet/google/gve/gve_tx_dqo.c:88:27: warning: unused variable 'buf' [-Wunused-variable]
88 | struct gve_tx_dma_buf *buf = &cur_state->bufs[j];
| ^~~
drivers/net/ethernet/google/gve/gve_tx_dqo.c: In function 'gve_tx_add_skb_no_copy_dqo':
drivers/net/ethernet/google/gve/gve_tx_dqo.c:496:26: warning: unused variable 'buf' [-Wunused-variable]
496 | struct gve_tx_dma_buf *buf =
| ^~~
drivers/net/ethernet/google/gve/gve_tx_dqo.c:515:26: warning: unused variable 'buf' [-Wunused-variable]
515 | struct gve_tx_dma_buf *buf =
| ^~~
drivers/net/ethernet/google/gve/gve_tx_dqo.c:556:26: warning: unused variable 'buf' [-Wunused-variable]
556 | struct gve_tx_dma_buf *buf = &pending_packet->bufs[i];
| ^~~
drivers/net/ethernet/google/gve/gve_tx_dqo.c: In function 'remove_from_list':
>> drivers/net/ethernet/google/gve/gve_tx_dqo.c:730:6: warning: variable 'index' set but not used [-Wunused-but-set-variable]
730 | s16 index, prev_index, next_index;
| ^~~~~
drivers/net/ethernet/google/gve/gve_tx_dqo.c: In function 'gve_unmap_packet':
>> drivers/net/ethernet/google/gve/gve_tx_dqo.c:753:25: warning: variable 'buf' set but not used [-Wunused-but-set-variable]
753 | struct gve_tx_dma_buf *buf;
| ^~~
In file included from include/linux/printk.h:7,
from include/linux/kernel.h:17,
from arch/x86/include/asm/percpu.h:27,
from arch/x86/include/asm/current.h:6,
from include/linux/sched.h:12,
from include/linux/ratelimit.h:6,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/linux/dma-mapping.h:7,
from drivers/net/ethernet/google/gve/gve.h:10,
from drivers/net/ethernet/google/gve/gve_tx_dqo.c:7:
drivers/net/ethernet/google/gve/gve_tx_dqo.c: In function 'remove_miss_completions':
>> include/linux/kern_levels.h:5:18: warning: format '%ld' expects argument of type 'long int', but argument 3 has type 'int' [-Wformat=]
5 | #define KERN_SOH "\001" /* ASCII Start Of Header */
| ^~~~~~
include/linux/kern_levels.h:11:18: note: in expansion of macro 'KERN_SOH'
11 | #define KERN_ERR KERN_SOH "3" /* error conditions */
| ^~~~~~~~
include/linux/printk.h:343:9: note: in expansion of macro 'KERN_ERR'
343 | printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__)
| ^~~~~~~~
include/linux/net.h:247:3: note: in expansion of macro 'pr_err'
247 | function(__VA_ARGS__); \
| ^~~~~~~~
include/linux/net.h:257:2: note: in expansion of macro 'net_ratelimited_function'
257 | net_ratelimited_function(pr_err, fmt, ##__VA_ARGS__)
| ^~~~~~~~~~~~~~~~~~~~~~~~
drivers/net/ethernet/google/gve/gve_tx_dqo.c:893:3: note: in expansion of macro 'net_err_ratelimited'
893 | net_err_ratelimited("%s: No reinjection completion was received for: %ld.\n",
| ^~~~~~~~~~~~~~~~~~~
drivers/net/ethernet/google/gve/gve_tx_dqo.c:893:74: note: format string is defined here
893 | net_err_ratelimited("%s: No reinjection completion was received for: %ld.\n",
| ~~^
| |
| long int
| %dvim +/index +730 drivers/net/ethernet/google/gve/gve_tx_dqo.c
447
448 /* Returns 0 on success, or < 0 on error.
449 *
450 * Before this function is called, the caller must ensure
451 * gve_has_pending_packet(tx) returns true.
452 */
453 static int gve_tx_add_skb_no_copy_dqo(struct gve_tx_ring *tx,
454 struct sk_buff *skb)
455 {
456 const struct skb_shared_info *shinfo = skb_shinfo(skb);
457 const bool is_gso = skb_is_gso(skb);
458 u32 desc_idx = tx->dqo_tx.tail;
459
460 struct gve_tx_pending_packet_dqo *pending_packet;
461 struct gve_tx_metadata_dqo metadata;
462 s16 completion_tag;
463 int i;
464
465 pending_packet = gve_alloc_pending_packet(tx);
466 pending_packet->skb = skb;
467 pending_packet->num_bufs = 0;
468 completion_tag = pending_packet - tx->dqo.pending_packets;
469
470 gve_extract_tx_metadata_dqo(skb, &metadata);
471 if (is_gso) {
472 int header_len = gve_prep_tso(skb);
473
474 if (unlikely(header_len < 0))
475 goto err;
476
477 gve_tx_fill_tso_ctx_desc(&tx->dqo.tx_ring[desc_idx].tso_ctx,
478 skb, &metadata, header_len);
479 desc_idx = (desc_idx + 1) & tx->mask;
480 }
481
482 gve_tx_fill_general_ctx_desc(&tx->dqo.tx_ring[desc_idx].general_ctx,
483 &metadata);
484 desc_idx = (desc_idx + 1) & tx->mask;
485
486 /* Note: HW requires that the size of a non-TSO packet be within the
487 * range of [17, 9728].
488 *
489 * We don't double check because
490 * - We limited netdev->min_mtu to ETH_MIN_MTU.
491 * - Hypervisor won't allow MTU larger than 9216.
492 */
493
494 /* Map the linear portion of skb */
495 {
496 struct gve_tx_dma_buf *buf =
497 &pending_packet->bufs[pending_packet->num_bufs];
498 u32 len = skb_headlen(skb);
499 dma_addr_t addr;
500
501 addr = dma_map_single(tx->dev, skb->data, len, DMA_TO_DEVICE);
502 if (unlikely(dma_mapping_error(tx->dev, addr)))
503 goto err;
504
505 dma_unmap_len_set(buf, len, len);
506 dma_unmap_addr_set(buf, dma, addr);
507 ++pending_packet->num_bufs;
508
509 gve_tx_fill_pkt_desc_dqo(tx, &desc_idx, skb, len, addr,
510 completion_tag,
511 /*eop=*/shinfo->nr_frags == 0, is_gso);
512 }
513
514 for (i = 0; i < shinfo->nr_frags; i++) {
515 struct gve_tx_dma_buf *buf =
516 &pending_packet->bufs[pending_packet->num_bufs];
517 const skb_frag_t *frag = &shinfo->frags[i];
518 bool is_eop = i == (shinfo->nr_frags - 1);
519 u32 len = skb_frag_size(frag);
520 dma_addr_t addr;
521
522 addr = skb_frag_dma_map(tx->dev, frag, 0, len, DMA_TO_DEVICE);
523 if (unlikely(dma_mapping_error(tx->dev, addr)))
524 goto err;
525
526 dma_unmap_len_set(buf, len, len);
527 dma_unmap_addr_set(buf, dma, addr);
528 ++pending_packet->num_bufs;
529
530 gve_tx_fill_pkt_desc_dqo(tx, &desc_idx, skb, len, addr,
531 completion_tag, is_eop, is_gso);
532 }
533
534 /* Commit the changes to our state */
535 tx->dqo_tx.tail = desc_idx;
536
537 /* Request a descriptor completion on the last descriptor of the
538 * packet if we are allowed to by the HW enforced interval.
539 */
540 {
541 u32 last_desc_idx = (desc_idx - 1) & tx->mask;
542 u32 last_report_event_interval =
543 (last_desc_idx - tx->dqo_tx.last_re_idx) & tx->mask;
544
545 if (unlikely(last_report_event_interval >=
546 GVE_TX_MIN_RE_INTERVAL)) {
547 tx->dqo.tx_ring[last_desc_idx].pkt.report_event = true;
548 tx->dqo_tx.last_re_idx = last_desc_idx;
549 }
550 }
551
552 return 0;
553
554 err:
555 for (i = 0; i < pending_packet->num_bufs; i++) {
> 556 struct gve_tx_dma_buf *buf = &pending_packet->bufs[i];
557
558 if (i == 0) {
559 dma_unmap_single(tx->dev, dma_unmap_addr(buf, dma),
560 dma_unmap_len(buf, len),
561 DMA_TO_DEVICE);
562 } else {
563 dma_unmap_page(tx->dev, dma_unmap_addr(buf, dma),
564 dma_unmap_len(buf, len), DMA_TO_DEVICE);
565 }
566 }
567
568 pending_packet->skb = NULL;
569 pending_packet->num_bufs = 0;
570 gve_free_pending_packet(tx, pending_packet);
571
572 return -1;
573 }
574
575 static int gve_num_descs_per_buf(size_t size)
576 {
577 return DIV_ROUND_UP(size, GVE_TX_MAX_BUF_SIZE_DQO);
578 }
579
580 static int gve_num_buffer_descs_needed(const struct sk_buff *skb)
581 {
582 const struct skb_shared_info *shinfo = skb_shinfo(skb);
583 int num_descs;
584 int i;
585
586 num_descs = gve_num_descs_per_buf(skb_headlen(skb));
587
588 for (i = 0; i < shinfo->nr_frags; i++) {
589 unsigned int frag_size = skb_frag_size(&shinfo->frags[i]);
590
591 num_descs += gve_num_descs_per_buf(frag_size);
592 }
593
594 return num_descs;
595 }
596
597 /* Returns true if HW is capable of sending TSO represented by `skb`.
598 *
599 * Each segment must not span more than GVE_TX_MAX_DATA_DESCS buffers.
600 * - The header is counted as one buffer for every single segment.
601 * - A buffer which is split between two segments is counted for both.
602 * - If a buffer contains both header and payload, it is counted as two buffers.
603 */
604 static bool gve_can_send_tso(const struct sk_buff *skb)
605 {
606 const int header_len = skb_checksum_start_offset(skb) + tcp_hdrlen(skb);
607 const int max_bufs_per_seg = GVE_TX_MAX_DATA_DESCS - 1;
608 const struct skb_shared_info *shinfo = skb_shinfo(skb);
609 const int gso_size = shinfo->gso_size;
610 int cur_seg_num_bufs;
611 int cur_seg_size;
612 int i;
613
614 cur_seg_size = skb_headlen(skb) - header_len;
615 cur_seg_num_bufs = cur_seg_size > 0;
616
617 for (i = 0; i < shinfo->nr_frags; i++) {
618 if (cur_seg_size >= gso_size) {
619 cur_seg_size %= gso_size;
620 cur_seg_num_bufs = cur_seg_size > 0;
621 }
622
623 if (unlikely(++cur_seg_num_bufs > max_bufs_per_seg))
624 return false;
625
626 cur_seg_size += skb_frag_size(&shinfo->frags[i]);
627 }
628
629 return true;
630 }
631
632 /* Attempt to transmit specified SKB.
633 *
634 * Returns 0 if the SKB was transmitted or dropped.
635 * Returns -1 if there is not currently enough space to transmit the SKB.
636 */
637 static int gve_try_tx_skb(struct gve_priv *priv, struct gve_tx_ring *tx,
638 struct sk_buff *skb)
639 {
640 int num_buffer_descs;
641 int total_num_descs;
642
643 if (skb_is_gso(skb)) {
644 /* If TSO doesn't meet HW requirements, attempt to linearize the
645 * packet.
646 */
647 if (unlikely(!gve_can_send_tso(skb) &&
648 skb_linearize(skb) < 0)) {
649 net_err_ratelimited("%s: Failed to transmit TSO packet\n",
650 priv->dev->name);
651 goto drop;
652 }
653
654 num_buffer_descs = gve_num_buffer_descs_needed(skb);
655 } else {
656 num_buffer_descs = gve_num_buffer_descs_needed(skb);
657
658 if (unlikely(num_buffer_descs > GVE_TX_MAX_DATA_DESCS)) {
659 if (unlikely(skb_linearize(skb) < 0))
660 goto drop;
661
662 num_buffer_descs = 1;
663 }
664 }
665
666 /* Metadata + (optional TSO) + data descriptors. */
667 total_num_descs = 1 + skb_is_gso(skb) + num_buffer_descs;
668 if (unlikely(gve_maybe_stop_tx_dqo(tx, total_num_descs +
669 GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP))) {
670 return -1;
671 }
672
673 if (unlikely(gve_tx_add_skb_no_copy_dqo(tx, skb) < 0))
674 goto drop;
675
676 netdev_tx_sent_queue(tx->netdev_txq, skb->len);
677 skb_tx_timestamp(skb);
678 return 0;
679
680 drop:
681 tx->dropped_pkt++;
682 dev_kfree_skb_any(skb);
683 return 0;
684 }
685
686 /* Transmit a given skb and ring the doorbell. */
687 netdev_tx_t gve_tx_dqo(struct sk_buff *skb, struct net_device *dev)
688 {
689 struct gve_priv *priv = netdev_priv(dev);
690 struct gve_tx_ring *tx;
691
692 tx = &priv->tx[skb_get_queue_mapping(skb)];
693 if (unlikely(gve_try_tx_skb(priv, tx, skb) < 0)) {
694 /* We need to ring the txq doorbell -- we have stopped the Tx
695 * queue for want of resources, but prior calls to gve_tx()
696 * may have added descriptors without ringing the doorbell.
697 */
698 gve_tx_put_doorbell_dqo(priv, tx->q_resources, tx->dqo_tx.tail);
699 return NETDEV_TX_BUSY;
700 }
701
702 if (!netif_xmit_stopped(tx->netdev_txq) && netdev_xmit_more())
703 return NETDEV_TX_OK;
704
705 gve_tx_put_doorbell_dqo(priv, tx->q_resources, tx->dqo_tx.tail);
706 return NETDEV_TX_OK;
707 }
708
709 static void add_to_list(struct gve_tx_ring *tx, struct gve_index_list *list,
710 struct gve_tx_pending_packet_dqo *pending_packet)
711 {
712 s16 old_tail, index;
713
714 index = pending_packet - tx->dqo.pending_packets;
715 old_tail = list->tail;
716 list->tail = index;
717 if (old_tail == -1)
718 list->head = index;
719 else
720 tx->dqo.pending_packets[old_tail].next = index;
721
722 pending_packet->next = -1;
723 pending_packet->prev = old_tail;
724 }
725
726 static void remove_from_list(struct gve_tx_ring *tx,
727 struct gve_index_list *list,
728 struct gve_tx_pending_packet_dqo *pending_packet)
729 {
> 730 s16 index, prev_index, next_index;
731
732 index = pending_packet - tx->dqo.pending_packets;
733 prev_index = pending_packet->prev;
734 next_index = pending_packet->next;
735
736 if (prev_index == -1) {
737 /* Node is head */
738 list->head = next_index;
739 } else {
740 tx->dqo.pending_packets[prev_index].next = next_index;
741 }
742 if (next_index == -1) {
743 /* Node is tail */
744 list->tail = prev_index;
745 } else {
746 tx->dqo.pending_packets[next_index].prev = prev_index;
747 }
748 }
749
750 static void gve_unmap_packet(struct device *dev,
751 struct gve_tx_pending_packet_dqo *pending_packet)
752 {
> 753 struct gve_tx_dma_buf *buf; 754 int i;
755
756 /* SKB linear portion is guaranteed to be mapped */
757 buf = &pending_packet->bufs[0];
758 dma_unmap_single(dev, dma_unmap_addr(buf, dma),
759 dma_unmap_len(buf, len), DMA_TO_DEVICE);
760 for (i = 1; i < pending_packet->num_bufs; i++) {
761 buf = &pending_packet->bufs[i];
762 dma_unmap_page(dev, dma_unmap_addr(buf, dma),
763 dma_unmap_len(buf, len), DMA_TO_DEVICE);
764 }
765 pending_packet->num_bufs = 0;
766 }
767
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
PATCH 16/16 gve: DQO: Add RX path
In reply to Bailey ForrestThe RX queue has an array of gve_rx_buf_state_dqo objects. All allocated pages have an associated buf_state object. When a buffer is posted on the RX buffer queue, the buffer ID will be the buf_state's index into the RX queue's array.
On packet reception, the RX queue will have one descriptor for each buffer associated with a received packet. Each RX descriptor will have a buffer_id that was posted on the buffer queue.
Notable mentions:
- We use a default buffer size of 2048 bytes. Based on page size, we may post separate sections of a single page as separate buffers.
- The driver holds an extra reference on pages passed up the receive path with an skb and keeps these pages on a list. When posting new buffers to the NIC, we check if any of these pages has only our reference, or another buffer sized segment of the page has no references. If so, it is free to reuse. This page recycling approach is a common netdev optimization that reduces page alloc/free calls.
- Pages in the free list have a page_count bias in order to avoid an atomic increment of pagecount every time we attempt to reuse a page.
# references = page_count() - bias
- In order to track when a page is safe to reuse, we keep track of the last offset which had a single SKB reference. When this occurs, it implies that every single other offset is reusable. Otherwise, we don't know if offsets can be safely reused.
- We maintain two free lists of pages. List #1 (recycled_buf_states) contains pages we know can be reused right away. List #2 (used_buf_states) contains pages which cannot be used right away. We only attempt to get pages from list #2 when list #1 is empty. We only attempt to use a small fixed number pages from list #2 before giving up and allocating a new page. Both lists are FIFOs in hope that by the time we attempt to reuse a page, the references were dropped.
--- drivers/net/ethernet/google/gve/gve.h | 1 + drivers/net/ethernet/google/gve/gve_rx_dqo.c | 582 +++++++++++++++++++ drivers/net/ethernet/google/gve/gve_utils.c | 15 + drivers/net/ethernet/google/gve/gve_utils.h | 3 + 4 files changed, 601 insertions(+)
drivers/net/ethernet/google/gve/gve.h +1 -0
@@ -59,6 +59,7 @@ struct gve_rx_slot_page_info {struct page *page;void *page_address;u32 page_offset;/* offset to write to in page */ +int pagecnt_bias;/* expected pagecnt if only the driver has a ref */ u8 can_flip; };
drivers/net/ethernet/google/gve/gve_rx_dqo.c +582 -0
608 lines not on this page · raw message
drivers/net/ethernet/google/gve/gve_utils.c +15 -0
@@ -64,3 +64,18 @@ struct sk_buff *gve_rx_copy(struct net_device *dev, struct napi_struct *napi,return skb; } +void gve_dec_pagecnt_bias (struct gve_rx_slot_page_info *page_info) +{ + page_info->pagecnt_bias--; +if (page_info->pagecnt_bias ==0 ) { +int pagecount =page_count (page_info->page); + +/* If we have run out of bias - set it back up to INT_MAX +* minus the existing refs. +*/ + page_info->pagecnt_bias =INT_MAX - pagecount; + +/* Set pagecount back up to max. */ +page_ref_add (page_info->page,INT_MAX - pagecount); + } +}
drivers/net/ethernet/google/gve/gve_utils.h +3 -0
@@ -21,5 +21,8 @@ struct sk_buff *gve_rx_copy(struct net_device *dev, struct napi_struct *napi, struct gve_rx_slot_page_info *page_info, u16 len, u16 pad); +/* Decrement pagecnt_bias. Set it back to INT_MAX if it reached zero. */ +void gve_dec_pagecnt_bias (struct gve_rx_slot_page_info *page_info); +#endif /* _GVE_UTILS_H */
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
This series was applied to netdev/net-next.git (refs/heads/master):
Hello:
This series was applied to netdev/net-next.git (refs/heads/master):
Quoted 10 lines · On Thu, 24 Jun 2021 11:06:16 -0700 you wrote:
DQO is the descriptor format for our next generation virtual NIC. The existing descriptor format will be referred to as "GQI" in the patch set.
One major change with DQO is it uses dual descriptor rings for both TX and RX queues.
The TX path uses a TX queue to send descriptors to HW, and receives packet completion events on a TX completion queue.
[...]
Here is the summary with links:
- [net-next,01/16] gve: Update GVE documentation to describe DQO
https://git.kernel.org/netdev/net-next/c/c6a7ed77ee63 - [net-next,02/16] gve: Move some static functions to a common file https://git.kernel.org/netdev/net-next/c/dbdaa6754051 - [net-next,03/16] gve: gve_rx_copy: Move padding to an argument https://git.kernel.org/netdev/net-next/c/35f9b2f43f8e - [net-next,04/16] gve: Make gve_rx_slot_page_info.page_offset an absolute offset https://git.kernel.org/netdev/net-next/c/920fb4519355 - [net-next,05/16] gve: Introduce a new model for device options https://git.kernel.org/netdev/net-next/c/8a39d3e0dadf - [net-next,06/16] gve: Introduce per netdev `enum gve_queue_format` https://git.kernel.org/netdev/net-next/c/a5886ef4f4bf - [net-next,07/16] gve: adminq: DQO specific device descriptor logic https://git.kernel.org/netdev/net-next/c/5ca2265eefc0 - [net-next,08/16] gve: Add support for DQO RX PTYPE map https://git.kernel.org/netdev/net-next/c/c4b87ac87635 - [net-next,09/16] gve: Add dqo descriptors https://git.kernel.org/netdev/net-next/c/223198183ff1 - [net-next,10/16] gve: Add DQO fields for core data structures https://git.kernel.org/netdev/net-next/c/a4aa1f1e69df - [net-next,11/16] gve: Update adminq commands to support DQO queues https://git.kernel.org/netdev/net-next/c/1f6228e459f8 - [net-next,12/16] gve: DQO: Add core netdev features https://git.kernel.org/netdev/net-next/c/5e8c5adf95f8 - [net-next,13/16] gve: DQO: Add ring allocation and initialization https://git.kernel.org/netdev/net-next/c/9c1a59a2f4bc - [net-next,14/16] gve: DQO: Configure interrupts on device up https://git.kernel.org/netdev/net-next/c/0dcc144a7994 - [net-next,15/16] gve: DQO: Add TX path https://git.kernel.org/netdev/net-next/c/a57e5de476be - [net-next,16/16] gve: DQO: Add RX path https://git.kernel.org/netdev/net-next/c/9b8dd5e5ea48
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html