introduce GVE PMD
Subsystems (2)
RE: [PATCH v7 8/8] net/gve: add support for Rx/Tx
In reply to Guo, JunfengQuoted 102 lines
-----Original Message-----
From: Guo, Junfeng
Sent: Monday, October 24, 2022 21:25
To: Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: RE: [PATCH v7 8/8] net/gve: add support for Rx/TxQuoted 13 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/TxQuoted 13 lines
Quoted 12 lines · Junfeng
Quoted 12 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:52
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, ChenboQuoted 58 lines
Quoted 57 lines
Quoted 52 lines
Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/TxQuoted 5 lines · Junfeng Guo
Add Rx/Tx of
GQI_QPLqueue format andGQI_RDAqueue format.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 26 lines
drivers/net/gve/gve_tx.c +26
-0+ +static inline void +gve_tx_clean_swr_qpl (struct gve_tx_queue *txq) +{ +uint32_t start = txq->sw_ntc; +uint32_t ntc, nb_clean; + + ntc = txq->sw_tail; + +if (ntc == start) +return ; + +/* if wrap around, free twice. */ +if (ntc < start) { + nb_clean = txq->nb_tx_desc - start; +if (nb_clean >GVE_TX_MAX_FREE_SZ ) + nb_clean =GVE_TX_MAX_FREE_SZ ; +gve_free_bulk_mbuf (&txq->sw_ring[start], nb_clean); + + txq->sw_nb_free += nb_clean; + start += nb_clean; +if (start == txq->nb_tx_desc) + start =0 ; + txq->sw_ntc = start; + } +drivers/net/gve/gve_tx.c line 138
Quoted 10 lines
drivers/net/gve/gve_tx.c +10
-0+if (ntc > start) { + nb_clean = ntc - start; +if (nb_clean >GVE_TX_MAX_FREE_SZ ) + nb_clean =GVE_TX_MAX_FREE_SZ ; +gve_free_bulk_mbuf (&txq->sw_ring[start], nb_clean); + txq->sw_nb_free += nb_clean; + start += nb_clean; + txq->sw_ntc = start; + } +}[copy/paste from previous version]
Quoted 2 lines
may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.
Yes, this 'if' block is dropped in v7 as suggested. Thanks!
Quoted 6 lines
Quoted 2 lines
This is v7 and please check above code which has the mentioned 'if' block exist, do you mean in coming v8?
Oh, sorry about this! There is another function with the same issue. I just updated the code there and forgot this. Will update this and check for the rest in the coming version. Thanks!
Sorry again about this part!!
After running the code and double checking the code, it seems that the 'if' blocks here cannot be removed directly. As you can see, in the above 'if' block, the parameter 'start' is not a fixed one and may be changed gradually. So these 'if' blocks are necessary for some certain cases.
I remembered that the queue page list is a tricky feature that can be wraparound to improve the utility or performance. Thus the queue page list should be freed twice and these two 'if' blocks here would run one by one.
I'll restore these modifications in the coming version. Thanks!
Quoted 36 lines
Quoted 36 lines
Quoted 20 lines
Quoted 20 lines
<...>
Review
Quoted 2 lines
(quoted diff) +2
-0+uint16_t +gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t
nb_pkts)drivers/net/gve/gve_tx.c line 461
Quoted 9 lines
drivers/net/gve/gve_tx.c +9
-0+{ +struct gve_tx_queue *txq = tx_queue; + +if (txq->is_gqi_qpl) +return gve_tx_burst_qpl (tx_queue, tx_pkts, nb_pkts); + +return gve_tx_burst_ra (tx_queue, tx_pkts, nb_pkts); +} +[copy/paste from previous version]
Quoted 3 lines
Quoted 3 lines
Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check ondatapath.
Quoted 12 lines
Quoted 3 lines
This is optimization and can be done later, it doesn't have to be in the set.
The exact queue type can be fetched from the backend via adminq in
priv->queue_format. So there won't be mix of the queue types. Currently, onlyGQI_QPLandGQI_RDAqueue format are supported in PMD. Also, onlyGQI_QPLqueue format is in use on GCP sinceGQI_RDAhasn't been released in production.
This part code will be optimized/refactored later when involving the queue typeDQO_RDA. Thanks!
Quoted 40 lines · Junfeng Guo
Introduce a new PMD for Google Virtual Ethernet (GVE).
gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.
Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.
Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic for the device description.
This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
-gve_adminq.c
-gve_adminq.h
-gve_desc.h
-gve_desc_dqo.h
-gve_register.hIt's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0v2:
fix some CI check error.v3:
refactor some code and fix some build error.v4:
move the Google base code files into DPDK base folder.v5:
reorder commit sequence and drop the stats feature.v6-v7:
improve the code.
Quoted 9 lines
Junfeng Guo (8):
net/gve/base: introduce base codenet/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx
Can you please check the build error reported by CI:
https://mails.dpdk.org/archives/test-report/2022-October/318054.html
Following link can be helpful:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225324
Quoted 51 lines · Ferruh Yigit
Quoted 40 lines · Junfeng Guo
Introduce a new PMD for Google Virtual Ethernet (GVE).
gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.
Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.
Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic for the device description.
This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
-gve_adminq.c
-gve_adminq.h
-gve_desc.h
-gve_desc_dqo.h
-gve_register.hIt's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0v2:
fix some CI check error.v3:
refactor some code and fix some build error.v4:
move the Google base code files into DPDK base folder.v5:
reorder commit sequence and drop the stats feature.v6-v7:
improve the code.Quoted 9 lines
Junfeng Guo (8):
net/gve/base: introduce base codenet/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx
Quoted 7 lines
Can you please check the build error reported by CI:
https://mails.dpdk.org/archives/test-report/2022-October/318054.htmlFollowing link can be helpful:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225324
Reminder if this build error, please send v8 with fix.
Quoted 78 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 0/8] introduce GVE PMDQuoted 4 lines · Ferruh Yigit
Quoted 3 lines · Junfeng Guo
Introduce a new PMD for Google Virtual Ethernet (GVE).
gve (or gVNIC) is the standard virtual ethernet interface on Google
Cloud
Quoted 5 lines
Quoted 5 lines
Platform (GCP), which is one of the multiple virtual interfaces from those
leading CSP customers in the world.Having a well maintained/optimized gve PMD on DPDK community can
help
Quoted 17 lines
Quoted 17 lines
those
cloud instance consumers with better experience of performance, maintenance
who wants to run their own VNFs on GCP.Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic for the device description.
This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
-gve_adminq.c
-gve_adminq.h
-gve_desc.h
-gve_desc_dqo.h
-gve_register.hQuoted 2 lines
Quoted 2 lines
It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0
Quoted 31 lines
Quoted 17 lines
v2:
fix some CI check error.v3:
refactor some code and fix some build error.v4:
move the Google base code files into DPDK base folder.v5:
reorder commit sequence and drop the stats feature.v6-v7:
improve the code.Quoted 9 lines
Junfeng Guo (8):
net/gve/base: introduce base codenet/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/TxCan you please check the build error reported by CI:
https://mails.dpdk.org/archives/test-report/2022-October/318054.html
Quoted 5 lines
Quoted 3 lines
Following link can be helpful:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225324Reminder if this build error, please send v8 with fix.
Sure, will try to fix this. Thanks for reminding this!
PATCH v5 2/8 net/gve/base: add OS specific implementation
In reply to Junfeng GuoAdd some MACRO definitions and memory operations which are specific for DPDK.
--- drivers/net/gve/base/gve_adminq.h | 2 + drivers/net/gve/base/gve_desc.h | 2 + drivers/net/gve/base/gve_desc_dqo.h | 2 + drivers/net/gve/base/gve_osdep.h | 159 ++++++++++++++++++++++++++++ drivers/net/gve/base/gve_register.h | 2 + 5 files changed, 167 insertions(+) create mode 100644 drivers/net/gve/base/gve_osdep.h
drivers/net/gve/base/gve_adminq.h +2 -0
@@ -8,6 +8,8 @@#ifndef _GVE_ADMINQ_H#define _GVE_ADMINQ_H +#include "gve_osdep.h" +/* Admin queue opcodes */ enum gve_adminq_opcodes { GVE_ADMINQ_DESCRIBE_DEVICE = 0x1,
drivers/net/gve/base/gve_desc.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_H_#define _GVE_DESC_H_ +#include "gve_osdep.h" +/* A note on seg_addrs * * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/base/gve_desc_dqo.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_DQO_H_#define _GVE_DESC_DQO_H_ +#include "gve_osdep.h" +#define GVE_TX_MAX_HDR_SIZE_DQO 255#define GVE_TX_MIN_TSO_MSS_DQO 88
drivers/net/gve/base/gve_osdep.h new file +159 -0
159 lines not on this page · raw message
drivers/net/gve/base/gve_register.h +2 -0
@@ -7,6 +7,8 @@ #ifndef _GVE_REGISTER_H_ #define _GVE_REGISTER_H_ +#include "gve_osdep.h" + /* Fixed Configuration Registers */ struct gve_registers { __be32 device_status;
Signature · 1 line
2.34.1
PATCH v5 3/8 net/gve: add support for device initialization
In reply to Junfeng GuoSupport device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close
Note that build system (including doc) is also added in this patch.
--- MAINTAINERS | 6 + doc/guides/nics/features/gve.ini | 10 + doc/guides/nics/gve.rst | 63 +++++ doc/guides/nics/index.rst | 1 + doc/guides/rel_notes/release_22_11.rst | 5 + drivers/net/gve/base/gve_adminq.c | 1 + drivers/net/gve/gve_ethdev.c | 371 +++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 225 +++++++++++++++ drivers/net/gve/gve_logs.h | 14 + drivers/net/gve/meson.build | 14 + drivers/net/gve/version.map | 3 + drivers/net/meson.build | 1 + 12 files changed, 714 insertions(+) create mode 100644 doc/guides/nics/features/gve.ini create mode 100644 doc/guides/nics/gve.rst create mode 100644 drivers/net/gve/gve_ethdev.c create mode 100644 drivers/net/gve/gve_ethdev.h create mode 100644 drivers/net/gve/gve_logs.h create mode 100644 drivers/net/gve/meson.build create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
@@ -700,6 +700,12 @@ F: drivers/net/enic/ F: doc/guides/nics/enic.rst F: doc/guides/nics/features/enic.ini +Google Virtual Ethernet +M: Junfeng Guo <[email protected]> +F: drivers/net/gve/ +F: doc/guides/nics/gve.rst +F: doc/guides/nics/features/gve.ini + Hisilicon hns3 M: Dongdong Liu <[email protected]> M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
@@ -0,0 +1,10 @@ +; +; Supported features of the Google Virtual Ethernet 'gve' poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/gve.rst new file +63 -0
63 lines not on this page · raw message
doc/guides/nics/index.rst +1 -0
@@ -29,6 +29,7 @@ Network Interface Controller Drivers enetfec enic fm10k + gve hinic hns3 i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -200,6 +200,11 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``. +* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernet devices. + * See the :doc:`../nics/gve` NIC guide for more details on this new driver. + Removed Items -------------
drivers/net/gve/base/gve_adminq.c +1 -0
@@ -5,6 +5,7 @@* Copyright(C) 2022 Intel Corporation */ +#include "../gve_ethdev.h" #include "gve_adminq.h" #include "gve_register.h"
drivers/net/gve/gve_ethdev.c new file +371 -0
371 lines not on this page · raw message
drivers/net/gve/gve_ethdev.h new file +225 -0
225 lines not on this page · raw message
drivers/net/gve/gve_logs.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2022 Intel Corporation +*/ + +#ifndef _GVE_LOGS_H_ +#define _GVE_LOGS_H_ + +extern int gve_logtype_driver; + +#define PMD_DRV_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \ + __func__, ## args) + +#endif
drivers/net/gve/meson.build new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2022 Intel Corporation + +if is_windows + build = false + reason = 'not supported on Windows' + subdir_done() +endif + +sources = files( + 'base/gve_adminq.c', + 'gve_ethdev.c', +) +includes += include_directories('base')
drivers/net/gve/version.map new file +3 -0
@@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +};
drivers/net/meson.build +1 -0
@@ -23,6 +23,7 @@ drivers = [ 'enic', 'failsafe', 'fm10k', + 'gve', 'hinic', 'hns3', 'i40e',
Signature · 1 line
2.34.1
Re: [PATCH v5 3/8] net/gve: add support for device initialization
Quoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +0
-0
doc/guides/rel_notes/release_22_11.rst New Features · line 208
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +5
-0@@ -200,6 +200,11 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``. +* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernet devices. + * See the :doc:`../nics/gve` NIC guide for more details on this new driver. +
Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?
<...>
Quoted 43 lines
doc/guides/rel_notes/release_22_11.rst +43
-0+static int +gve_dev_init(struct rte_eth_dev *eth_dev) +{ + struct gve_priv *priv = eth_dev->data->dev_private; + int max_tx_queues, max_rx_queues; + struct rte_pci_device *pci_dev; + struct gve_registers *reg_bar; + rte_be32_t *db_bar; + int err; + + eth_dev->dev_ops = &gve_eth_dev_ops; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + pci_dev = RTE_DEV_TO_PCI(eth_dev->device); + + reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr; + if (!reg_bar) { + PMD_DRV_LOG(ERR, "Failed to map pci bar!"); + return -ENOMEM; + } + + db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr; + if (!db_bar) { + PMD_DRV_LOG(ERR, "Failed to map doorbell bar!"); + return -ENOMEM; + } + + gve_write_version(®_bar->driver_version); + /* Get max queues to alloc etherdev */ + max_tx_queues = ioread32be(®_bar->max_tx_queues); + max_rx_queues = ioread32be(®_bar->max_rx_queues); + + priv->reg_bar0 = reg_bar; + priv->db_bar2 = db_bar; + priv->pci_dev = pci_dev; + priv->state_flags = 0x0; + + priv->max_nb_txq = max_tx_queues; + priv->max_nb_rxq = max_rx_queues; + + err = gve_init_priv(priv, false);
doc/guides/rel_notes/release_22_11.rst New Features
Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10
-0+ if (err) + return err; + + eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0); + if (!eth_dev->data->mac_addrs) { + PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac address"); + return -ENOMEM; + } + rte_ether_addr_copy(&priv->dev_addr, eth_dev->data->mac_addrs); +
Is anything assinged to 'priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly, instead of allocating memory for 'eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs = &priv->dev_addr"?
<...>
doc/guides/rel_notes/release_22_11.rst
Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12
-0+struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz; + + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */ + uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */
These fields are not used in this patch, I guess some will be used in datapath patch.
Can you please only add fields that is used in the patch? This way it will be clear in which functionality that field is used and enable to detect not used fields. We are accepting batch updates for base code, but this is dpdk related code, lets only add things that are used when they are used.
Same for all data structures.
<...>
drivers/net/gve/version.map line 1
Quoted 7 lines
drivers/net/gve/version.map new file +1
-0@@ -0,0 +1,3 @@ +DPDK_22 {
DPDK_23
RE: [PATCH v5 3/8] net/gve: add support for device initialization
Hi
Quoted 104 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +0
-0doc/guides/rel_notes/release_22_11.rst New Features · line 208
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +5
-0@@ -200,6 +200,11 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``. +* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernet devices. + * See the :doc:`../nics/gve` NIC guide for more details on this new driver. +Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?
<...>
Quoted 36 lines
doc/guides/rel_notes/release_22_11.rst +36
-0+static int +gve_dev_init(struct rte_eth_dev *eth_dev) { + struct gve_priv *priv = eth_dev->data->dev_private; + int max_tx_queues, max_rx_queues; + struct rte_pci_device *pci_dev; + struct gve_registers *reg_bar; + rte_be32_t *db_bar; + int err; + + eth_dev->dev_ops = &gve_eth_dev_ops; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + pci_dev = RTE_DEV_TO_PCI(eth_dev->device); + + reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr; + if (!reg_bar) { + PMD_DRV_LOG(ERR, "Failed to map pci bar!"); + return -ENOMEM; + } + + db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr; + if (!db_bar) { + PMD_DRV_LOG(ERR, "Failed to map doorbell bar!"); + return -ENOMEM; + } + + gve_write_version(®_bar->driver_version); + /* Get max queues to alloc etherdev */ + max_tx_queues = ioread32be(®_bar->max_tx_queues); + max_rx_queues = ioread32be(®_bar->max_rx_queues); + + priv->reg_bar0 = reg_bar; + priv->db_bar2 = db_bar; + priv->pci_dev = pci_dev;doc/guides/rel_notes/release_22_11.rst New Features
Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10
-0+ priv->state_flags = 0x0; + + priv->max_nb_txq = max_tx_queues; + priv->max_nb_rxq = max_rx_queues; + + err = gve_init_priv(priv, false); + if (err) + return err; + + eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct
rte_ether_addr), 0);doc/guides/rel_notes/release_22_11.rst
Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2
-0+ if (!eth_dev->data->mac_addrs) { + PMD_DRV_LOG(ERR, "Failed to allocate memory to store macaddress");
doc/guides/rel_notes/release_22_11.rst
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5
-0+ return -ENOMEM; + } + rte_ether_addr_copy(&priv->dev_addr, + eth_dev->data->mac_addrs); +
Quoted 4 lines
Is anything assinged to '
priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly, instead of allocating memory for 'eth_dev->data->mac_addrs'?
I mean why not "eth_dev->data->mac_addrs=&priv->dev_addr"?
Makes sense. There's no need to allocate a new memory. @Guo, Junfeng Can you update this?
Quoted 18 lines
<...>
doc/guides/rel_notes/release_22_11.rst
Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12
-0+struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz; + + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */ + uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */These fields are not used in this patch, I guess some will be used in datapath patch.
This is needed for base code gve_adminq.c not for datapath. Most of the stuff in gve_priv is for gve_adminq.c. The adminq will update this info which dpdk pmd will need later. Compiler will complain if these don't exsit.
Quoted 18 lines
Can you please only add fields that is used in the patch? This way it will be clear in which functionality that field is used and enable to detect not used fields.
We are accepting batch updates for base code, but this is dpdk related code, lets only add things that are used when they are used.
Same for all data structures.<...>
drivers/net/gve/version.map line 1
Quoted 6 lines
drivers/net/gve/version.map new file +1
-0@@ -0,0 +1,3 @@ +DPDK_22 {
DPDK_23
Re: [PATCH v5 3/8] net/gve: add support for device initialization
Quoted 132 lines · Xiaoyun
Hi
Quoted 104 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +0
-0doc/guides/rel_notes/release_22_11.rst New Features · line 208
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +5
-0@@ -200,6 +200,11 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``. +* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernet devices. + * See the :doc:`../nics/gve` NIC guide for more details on this new driver. +Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?
<...>
Quoted 36 lines
doc/guides/rel_notes/release_22_11.rst +36
-0+static int +gve_dev_init(struct rte_eth_dev *eth_dev) { + struct gve_priv *priv = eth_dev->data->dev_private; + int max_tx_queues, max_rx_queues; + struct rte_pci_device *pci_dev; + struct gve_registers *reg_bar; + rte_be32_t *db_bar; + int err; + + eth_dev->dev_ops = &gve_eth_dev_ops; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + pci_dev = RTE_DEV_TO_PCI(eth_dev->device); + + reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr; + if (!reg_bar) { + PMD_DRV_LOG(ERR, "Failed to map pci bar!"); + return -ENOMEM; + } + + db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr; + if (!db_bar) { + PMD_DRV_LOG(ERR, "Failed to map doorbell bar!"); + return -ENOMEM; + } + + gve_write_version(®_bar->driver_version); + /* Get max queues to alloc etherdev */ + max_tx_queues = ioread32be(®_bar->max_tx_queues); + max_rx_queues = ioread32be(®_bar->max_rx_queues); + + priv->reg_bar0 = reg_bar; + priv->db_bar2 = db_bar; + priv->pci_dev = pci_dev;doc/guides/rel_notes/release_22_11.rst New Features
Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10
-0+ priv->state_flags = 0x0; + + priv->max_nb_txq = max_tx_queues; + priv->max_nb_rxq = max_rx_queues; + + err = gve_init_priv(priv, false); + if (err) + return err; + + eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct
rte_ether_addr), 0);doc/guides/rel_notes/release_22_11.rst
Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2
-0+ if (!eth_dev->data->mac_addrs) { + PMD_DRV_LOG(ERR, "Failed to allocate memory to store macaddress");
doc/guides/rel_notes/release_22_11.rst
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5
-0+ return -ENOMEM; + } + rte_ether_addr_copy(&priv->dev_addr, + eth_dev->data->mac_addrs); +Quoted 4 lines
Is anything assinged to '
priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly, instead of allocating memory for 'eth_dev->data->mac_addrs'?
I mean why not "eth_dev->data->mac_addrs=&priv->dev_addr"?Makes sense. There's no need to allocate a new memory. @Guo, Junfeng Can you update this?
Quoted 18 lines
<...>
doc/guides/rel_notes/release_22_11.rst
Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12
-0+struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz; + + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */ + uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */These fields are not used in this patch, I guess some will be used in datapath patch.
Quoted 3 lines
This is needed for base code
gve_adminq.cnot for datapath. Most of the stuff ingve_privis forgve_adminq.c. The adminq will update this info which dpdk pmd will need later. Compiler will complain if these don't exsit.
You are right they are used by 'gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.
Quoted 18 lines
Quoted 18 lines
Can you please only add fields that is used in the patch? This way it will be clear in which functionality that field is used and enable to detect not used fields.
We are accepting batch updates for base code, but this is dpdk related code, lets only add things that are used when they are used.
Same for all data structures.<...>
drivers/net/gve/version.map line 1
Quoted 6 lines
drivers/net/gve/version.map new file +1
-0@@ -0,0 +1,3 @@ +DPDK_22 {
DPDK_23
RE: [PATCH v5 3/8] net/gve: add support for device initialization
Quoted 128 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 05:01
To: Li, Xiaoyun <[email protected]>; Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 11 lines · Xiaoyun
Hi
Quoted 8 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>;Wang,
Quoted 2 lines
Quoted 2 lines
Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for deviceinitialization
Quoted 26 lines
Quoted 26 lines · Junfeng Guo
Quoted 12 lines
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +0
-0@@ -200,6 +200,11 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.Quoted 3 lines
Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst New Features
Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst +3
-0+* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernetdevices.
Quoted 1 line
Quoted 1 line
doc/guides/rel_notes/release_22_11.rst New Features · line 206
Quoted 1 line
doc/guides/rel_notes/release_22_11.rst +1
-0+ * See the :doc:`../nics/gve` NIC guide for more details on this newdriver.
Quoted 46 lines
Quoted 46 lines
doc/guides/rel_notes/release_22_11.rst New Features · line 208
Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +1
-0+Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?
<...>
Quoted 38 lines
doc/guides/rel_notes/release_22_11.rst +38
-0+static int +gve_dev_init(struct rte_eth_dev *eth_dev) { + struct gve_priv *priv = eth_dev->data->dev_private; + int max_tx_queues, max_rx_queues; + struct rte_pci_device *pci_dev; + struct gve_registers *reg_bar; + rte_be32_t *db_bar; + int err; + + eth_dev->dev_ops = &gve_eth_dev_ops; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + pci_dev = RTE_DEV_TO_PCI(eth_dev->device); + + reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr; + if (!reg_bar) { + PMD_DRV_LOG(ERR, "Failed to map pci bar!"); + return -ENOMEM; + } + + db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr; + if (!db_bar) { + PMD_DRV_LOG(ERR, "Failed to map doorbell bar!"); + return -ENOMEM; + } + + gve_write_version(®_bar->driver_version); + /* Get max queues to alloc etherdev */ + max_tx_queues = ioread32be(®_bar->max_tx_queues); + max_rx_queues = ioread32be(®_bar->max_rx_queues); + + priv->reg_bar0 = reg_bar; + priv->db_bar2 = db_bar; + priv->pci_dev = pci_dev; + priv->state_flags = 0x0; +Quoted 8 lines
Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst
Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst +8
-0+ priv->max_nb_txq = max_tx_queues; + priv->max_nb_rxq = max_rx_queues; + + err = gve_init_priv(priv, false); + if (err) + return err; + + eth_dev->data->mac_addrs = rte_zmalloc("gve_mac",sizeof(struct
Quoted 3 lines
Quoted 3 lines
rte_ether_addr), 0);Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2
-0+ if (!eth_dev->data->mac_addrs) { + PMD_DRV_LOG(ERR, "Failed to allocate memory to storemac
Quoted 7 lines
Quoted 7 lines
address");
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5
-0+ return -ENOMEM; + } + rte_ether_addr_copy(&priv->dev_addr, + eth_dev->data->mac_addrs); +
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Is anything assinged to '
priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly,instead of
Quoted 4 lines
Quoted 2 lines
allocating memory for '
eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs=&priv->dev_addr"?Makes sense. There's no need to allocate a new memory. @Guo,
Junfeng Can you update this?
Thanks Xiaoyun and Ferruh for the comments!
I tried to update the code as suggested but may get "Invalid Memory" warning when quit the testpmd. I found it was caused at the function rte_eth_dev_release_port with " rte_free(eth_dev->data->mac_addrs); ". Seems that allocating memory for 'eth_dev->data->mac_addrs' is still needed. Please help correct me if I misunderstood this. Thanks! I'll keep this part unchanged for the coming patchset first.
Quoted 22 lines
Quoted 17 lines
Quoted 17 lines
<...>
doc/guides/rel_notes/release_22_11.rst
Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12
-0+struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz; + + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */ + uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */These fields are not used in this patch, I guess some will be used in
datapath
Quoted 3 lines
Quoted 1 line
patch.
This is needed for base code
gve_adminq.cnot for datapath. Most ofthe stuff in
gve_privis forgve_adminq.c.
Quoted 8 lines
Quoted 1 line
The adminq will update this info which dpdk pmd will need later.
Compiler will complain if these don't exsit.
Quoted 1 line
You are right they are used by '
gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.
Yes, we have already tried to move all the unused items to the corresponding stages patch by patch. Thanks for reminding this!
Quoted 23 lines
Quoted 3 lines
Quoted 3 lines
Can you please only add fields that is used in the patch? This way it will
be
Quoted 1 line
Quoted 1 line
clear in which functionality that field is used and enable to detect not
used
Quoted 2 lines
Quoted 2 lines
fields.
We are accepting batch updates for base code, but this is dpdk relatedcode,
Quoted 6 lines
Quoted 6 lines
lets only add things that are used when they are used. Same for all data structures.
<...>
Review
Quoted 1 line
(quoted diff) +0
-0b/drivers/net/gve/version.map
Quoted 7 lines
Quoted 7 lines
drivers/net/gve/version.map line 1
Quoted 5 lines
new file mode 100644 index 0000000000..c2e0723b4c
drivers/net/gve/version.map new file +1
-0@@ -0,0 +1,3 @@ +DPDK_22 {
DPDK_23
Re: [PATCH v5 3/8] net/gve: add support for device initialization
Quoted 138 lines · Junfeng
CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
Quoted 126 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 05:01
To: Li, Xiaoyun <[email protected]>; Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 11 lines · Xiaoyun
Hi
Quoted 8 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>;Wang,
Quoted 2 lines
Quoted 2 lines
Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for deviceinitialization
Quoted 26 lines
Quoted 26 lines · Junfeng Guo
Quoted 12 lines
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +0
-0@@ -200,6 +200,11 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.Quoted 3 lines
Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst New Features
Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst +3
-0+* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernetdevices.
Quoted 1 line
Quoted 1 line
doc/guides/rel_notes/release_22_11.rst New Features · line 206
Quoted 1 line
doc/guides/rel_notes/release_22_11.rst +1
-0+ * See the :doc:`../nics/gve` NIC guide for more details on this newdriver.
Quoted 46 lines
Quoted 46 lines
doc/guides/rel_notes/release_22_11.rst New Features · line 208
Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +1
-0+Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?
<...>
Quoted 38 lines
doc/guides/rel_notes/release_22_11.rst +38
-0+static int +gve_dev_init(struct rte_eth_dev *eth_dev) { + struct gve_priv *priv = eth_dev->data->dev_private; + int max_tx_queues, max_rx_queues; + struct rte_pci_device *pci_dev; + struct gve_registers *reg_bar; + rte_be32_t *db_bar; + int err; + + eth_dev->dev_ops = &gve_eth_dev_ops; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + pci_dev = RTE_DEV_TO_PCI(eth_dev->device); + + reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr; + if (!reg_bar) { + PMD_DRV_LOG(ERR, "Failed to map pci bar!"); + return -ENOMEM; + } + + db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr; + if (!db_bar) { + PMD_DRV_LOG(ERR, "Failed to map doorbell bar!"); + return -ENOMEM; + } + + gve_write_version(®_bar->driver_version); + /* Get max queues to alloc etherdev */ + max_tx_queues = ioread32be(®_bar->max_tx_queues); + max_rx_queues = ioread32be(®_bar->max_rx_queues); + + priv->reg_bar0 = reg_bar; + priv->db_bar2 = db_bar; + priv->pci_dev = pci_dev; + priv->state_flags = 0x0; +Quoted 8 lines
Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst
Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst +8
-0+ priv->max_nb_txq = max_tx_queues; + priv->max_nb_rxq = max_rx_queues; + + err = gve_init_priv(priv, false); + if (err) + return err; + + eth_dev->data->mac_addrs = rte_zmalloc("gve_mac",sizeof(struct
Quoted 3 lines
Quoted 3 lines
rte_ether_addr), 0);Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2
-0+ if (!eth_dev->data->mac_addrs) { + PMD_DRV_LOG(ERR, "Failed to allocate memory to storemac
Quoted 7 lines
Quoted 7 lines
address");
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5
-0+ return -ENOMEM; + } + rte_ether_addr_copy(&priv->dev_addr, + eth_dev->data->mac_addrs); +Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Is anything assinged to '
priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly,instead of
Quoted 4 lines
Quoted 2 lines
allocating memory for '
eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs=&priv->dev_addr"?Makes sense. There's no need to allocate a new memory. @Guo,
Junfeng Can you update this?
Quoted 8 lines
Thanks Xiaoyun and Ferruh for the comments!
I tried to update the code as suggested but may get "Invalid Memory" warning when quit the testpmd. I found it was caused at the functionrte_eth_dev_release_portwith "rte_free(eth_dev->data->mac_addrs); ". Seems that allocating memory for 'eth_dev->data->mac_addrs' is still needed. Please help correct me if I misunderstood this. Thanks! I'll keep this part unchanged for the coming patchset first.
No it is not needed, you need to set pointer to NULL on release path to prevent common code free it (the problem you are getting). There are samples in various PMDs, please check.
Quoted 26 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
<...>
doc/guides/rel_notes/release_22_11.rst
Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12
-0+struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz; + + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */ + uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */These fields are not used in this patch, I guess some will be used in
datapath
Quoted 3 lines
Quoted 1 line
patch.
This is needed for base code
gve_adminq.cnot for datapath. Most ofthe stuff in
gve_privis forgve_adminq.c.Quoted 1 line
The adminq will update this info which dpdk pmd will need later.
Compiler will complain if these don't exsit.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
You are right they are used by '
gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.Yes, we have already tried to move all the unused items to the corresponding stages patch by patch. Thanks for reminding this!
thanks.
Quoted 24 lines
Quoted 23 lines
Quoted 3 lines
Quoted 3 lines
Can you please only add fields that is used in the patch? This way it will
be
Quoted 1 line
Quoted 1 line
clear in which functionality that field is used and enable to detect not
used
Quoted 2 lines
Quoted 2 lines
fields.
We are accepting batch updates for base code, but this is dpdk relatedcode,
Quoted 6 lines
Quoted 6 lines
lets only add things that are used when they are used. Same for all data structures.
<...>
Review
Quoted 1 line
(quoted diff) +0
-0b/drivers/net/gve/version.map
Quoted 7 lines
Quoted 7 lines
drivers/net/gve/version.map line 1
Quoted 5 lines
new file mode 100644 index 0000000000..c2e0723b4c
drivers/net/gve/version.map new file +1
-0@@ -0,0 +1,3 @@ +DPDK_22 {
DPDK_23
RE: [PATCH v5 3/8] net/gve: add support for device initialization
Quoted 162 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 19:16
To: Guo, Junfeng <[email protected]>; Li, Xiaoyun <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 1 line · Junfeng
CAUTION: This message has originated from an External Source. Please
use proper judgment and caution when opening attachments, clicking links, or responding to this email.
Quoted 12 lines
Quoted 12 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 05:01
To: Li, Xiaoyun <[email protected]>; Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for deviceinitialization
Quoted 6 lines
Quoted 6 lines · Xiaoyun
Quoted 4 lines
Hi
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson,Bruce
Quoted 31 lines
Quoted 31 lines
Quoted 1 line
Quoted 1 line
<[email protected]>; Lin, Xueqin <[email protected]>;
Wang,
Quoted 2 lines
Quoted 2 lines
Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for deviceinitialization
Quoted 26 lines
Quoted 26 lines · Junfeng Guo
Quoted 12 lines
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +0
-0@@ -200,6 +200,11 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst New Features
Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst +3
-0+* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernetdevices.
Quoted 1 line
Quoted 1 line
Quoted 1 line
doc/guides/rel_notes/release_22_11.rst +1
-0+ * See the :doc:`../nics/gve` NIC guide for more details on thisnew
Quoted 73 lines
Quoted 73 lines
driver.
Quoted 46 lines
Quoted 46 lines
doc/guides/rel_notes/release_22_11.rst New Features · line 208
Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +1
-0+Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?
<...>
Quoted 38 lines
doc/guides/rel_notes/release_22_11.rst +38
-0+static int +gve_dev_init(struct rte_eth_dev *eth_dev) { + struct gve_priv *priv = eth_dev->data->dev_private; + int max_tx_queues, max_rx_queues; + struct rte_pci_device *pci_dev; + struct gve_registers *reg_bar; + rte_be32_t *db_bar; + int err; + + eth_dev->dev_ops = &gve_eth_dev_ops; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + pci_dev = RTE_DEV_TO_PCI(eth_dev->device); + + reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr; + if (!reg_bar) { + PMD_DRV_LOG(ERR, "Failed to map pci bar!"); + return -ENOMEM; + } + + db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr; + if (!db_bar) { + PMD_DRV_LOG(ERR, "Failed to map doorbell bar!"); + return -ENOMEM; + } + + gve_write_version(®_bar->driver_version); + /* Get max queues to alloc etherdev */ + max_tx_queues = ioread32be(®_bar->max_tx_queues); + max_rx_queues = ioread32be(®_bar->max_rx_queues); + + priv->reg_bar0 = reg_bar; + priv->db_bar2 = db_bar; + priv->pci_dev = pci_dev; + priv->state_flags = 0x0; +Quoted 8 lines
Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst
Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst +8
-0+ priv->max_nb_txq = max_tx_queues; + priv->max_nb_rxq = max_rx_queues; + + err = gve_init_priv(priv, false); + if (err) + return err; + + eth_dev->data->mac_addrs = rte_zmalloc("gve_mac",sizeof(struct
Quoted 3 lines
Quoted 3 lines
rte_ether_addr), 0);Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2
-0+ if (!eth_dev->data->mac_addrs) { + PMD_DRV_LOG(ERR, "Failed to allocate memory to storemac
Quoted 9 lines
Quoted 9 lines
address");
doc/guides/rel_notes/release_22_11.rst
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5
-0+ return -ENOMEM; + } + rte_ether_addr_copy(&priv->dev_addr, + eth_dev->data->mac_addrs); +Is anything assinged to '
priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly,instead of
Quoted 3 lines
Quoted 2 lines
allocating memory for '
eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs=&priv->dev_addr"?Quoted 7 lines
Quoted 2 lines
Quoted 1 line
Makes sense. There's no need to allocate a new memory. @Guo,
Junfeng Can you update this?
Thanks Xiaoyun and Ferruh for the comments!
I tried to update the code as suggested but may get "Invalid Memory" warning when quit the testpmd. I found it was caused at the functionrte_eth_dev_release_portwith "rte_free(eth_dev->data->mac_addrs);".
Quoted 8 lines
Quoted 4 lines
Seems that allocating memory for '
eth_dev->data->mac_addrs' is still needed. Please help correct me if I misunderstood this. Thanks! I'll keep this part unchanged for the coming patchset first.No it is not needed, you need to set pointer to NULL on release path to prevent common code free it (the problem you are getting). There are samples in various PMDs, please check.
Yes, make sense!
I'll double check for this and update in the coming version. Thanks!
Quoted 66 lines
Quoted 27 lines
Quoted 27 lines
Quoted 4 lines
Quoted 4 lines
<...>
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +6
-0+struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz;Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +6
-0+ + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */ + uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */These fields are not used in this patch, I guess some will be used in
datapath
Quoted 3 lines
Quoted 1 line
patch.
This is needed for base code
gve_adminq.cnot for datapath. Most ofthe stuff in
gve_privis forgve_adminq.c.Quoted 1 line
The adminq will update this info which dpdk pmd will need later.
Compiler will complain if these don't exsit.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
You are right they are used by '
gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.Yes, we have already tried to move all the unused items to the
corresponding
Quoted 2 lines
stages patch by patch. Thanks for reminding this!
thanks.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Can you please only add fields that is used in the patch? This way it
will
Quoted 2 lines
Quoted 2 lines
be
Quoted 1 line
Quoted 1 line
clear in which functionality that field is used and enable to detect
not
Quoted 3 lines
Quoted 3 lines
used
Quoted 2 lines
Quoted 2 lines
fields.
We are accepting batch updates for base code, but this is dpdkrelated
Quoted 16 lines
Quoted 15 lines
code,
Quoted 6 lines
Quoted 6 lines
lets only add things that are used when they are used. Same for all data structures.
<...>
Review
Quoted 1 line
(quoted diff) +0
-0b/drivers/net/gve/version.map
Quoted 7 lines
Quoted 7 lines
drivers/net/gve/version.map line 1
Quoted 5 lines
new file mode 100644 index 0000000000..c2e0723b4c
drivers/net/gve/version.map new file +1
-0@@ -0,0 +1,3 @@ +DPDK_22 {
DPDK_23
Re: [PATCH v5 3/8] net/gve: add support for device initialization
In reply to Junfeng GuoQuoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
doc/guides/nics/gve.rst line 9
Quoted 12 lines
doc/guides/nics/gve.rst new file +9
-0@@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2022 Intel Corporation. + +GVE poll mode driver +======================= + +The GVE PMD (**librte_net_gve**) provides poll mode driver support for +Google Virtual Ethernet device (also called as gVNIC). +
This is a virtual device, emulated in VM as PCI device, right? If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?
doc/guides/nics/gve.rst line 11
Quoted 2 lines
doc/guides/nics/gve.rst new file +2
-0+Current gVNIC is an alternative to the virtIO-based ethernet interface that can +support higher network bandwidths such as the 50-100 Gbps speeds.
This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.
Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report? Aren't there any other notable difference?
I think better to document as much as possible, cc'ed more virtio people.
RE: [PATCH v5 3/8] net/gve: add support for device initialization
Quoted 29 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 9:47 PM
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 14 lines
doc/guides/nics/gve.rst line 9
Quoted 12 lines
doc/guides/nics/gve.rst new file +9
-0@@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2022 Intel Corporation. + +GVE poll mode driver +======================= + +The GVE PMD (**librte_net_gve**) provides poll mode driver support for +Google Virtual Ethernet device (also called as gVNIC). +This is a virtual device, emulated in VM as PCI device, right?
Most likely yes. But I think this part is not public by Google. (Correct me if I am wrong)
Quoted 1 line
If so what emulates it, I mean can we use QEMU for it?
Yes if the user can have a emulated gVNIC device or real HW that supports gVNIC interface.
Quoted 1 line
And is there a kernel supported backend, as virtio has vhost?
I guess no if the back-end is not public
Quoted 7 lines
doc/guides/nics/gve.rst line 10
Quoted 2 lines
doc/guides/nics/gve.rst new file +1
-0+Current gVNIC is an alternative to the virtIO-based ethernet interfacethat can
doc/guides/nics/gve.rst line 11
Quoted 1 line
doc/guides/nics/gve.rst new file +1
-0+support higher network bandwidths such as the 50-100 Gbps speeds.This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.
I don't think it's good to compare these two. It's just two virtual interface, I don't see strong analysis to prove which one is better.
Like if you use Google cloud, you use gVNIC. Use other cloud that uses virtio, choose virtio.
Quoted 4 lines
Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?
Aren't there any other notable difference?
If we want to leave such description, I agree with Ferruh that we need to give some evidence.
But in my understanding, they are just two similar virtual interfaces. One has public standard, one is google-specific. Users need to use one of them when they choose to use one cloud service.
Thanks,
Chenbo
Quoted 3 lines
I think better to document as much as possible, cc'ed more virtio people.
RE: [PATCH v5 3/8] net/gve: add support for device initialization
In reply to Ferruh YigitQuoted 26 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 9:47 PM
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 16 lines
doc/guides/nics/gve.rst line 9
Quoted 12 lines
doc/guides/nics/gve.rst new file +9
-0@@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2022 Intel Corporation. + +GVE poll mode driver +======================= + +The GVE PMD (**librte_net_gve**) provides poll mode driver support +for Google Virtual Ethernet device (also called as gVNIC). +This is a virtual device, emulated in VM as PCI device, right? If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?
This is a virtual interface only provided on Google Cloud Platform (GCP), like ena on AWS, and virtio on Alibaba Cloud, etc.
The gve (gVNIC) is the standard virtual ethernet interface provided to users when anyone buys a cloud instance on GCP, which is ready there and everybody can access it. The backend details are not open to anyone outside of Google, I assume it is more like a business & technical secret of Google.
Quoted 3 lines
doc/guides/nics/gve.rst line 11
Quoted 3 lines
doc/guides/nics/gve.rst new file +2
-0+Current gVNIC is an alternative to the virtIO-based ethernet +interface that can support higher network bandwidths such as the 50-100 Gbps speeds.
I don't think it is an alternative to virtio. Gve is the driver name of gVNIC of GCP, which is the standard virtual ethernet interface on that cloud platform.
Quoted 4 lines
This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.
Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?
There is no hint that gve (or virtio) interface is faster than virtio (or gve). I think it heavily depends on the backend design, which can be SW or HW.
I would treat gve (actually the driver for gVNIC) as just one of the leading virtual ethernet interfaces on different cloud environments. Till now, gve is for GCP cloud environment only.
Hopefully my understanding is correct, as I am not the expert at Google cloud, and I got all the information from public.
Thanks for all the good questions! Hopefully my answers help!
Regards,
Helin
Quoted 3 lines
Aren't there any other notable difference?
I think better to document as much as possible, cc'ed more virtio people.
Re: [PATCH v5 3/8] net/gve: add support for device initialization
Quoted 42 lines · Helin
Quoted 26 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 9:47 PM
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 16 lines
doc/guides/nics/gve.rst line 9
Quoted 12 lines
doc/guides/nics/gve.rst new file +9
-0@@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2022 Intel Corporation. + +GVE poll mode driver +======================= + +The GVE PMD (**librte_net_gve**) provides poll mode driver support +for Google Virtual Ethernet device (also called as gVNIC). +This is a virtual device, emulated in VM as PCI device, right? If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?
Quoted 8 lines
This is a virtual interface only provided on Google Cloud Platform (GCP), like ena on AWS, and virtio on Alibaba Cloud, etc.
The gve (gVNIC) is the standard virtual ethernet interface provided to users when anyone buys a cloud instance on GCP, which is ready there and everybody can access it. The backend details are not open to anyone outside of Google, I assume it is more like a business & technical secret of Google.Quoted 3 lines
doc/guides/nics/gve.rst line 11
Quoted 3 lines
doc/guides/nics/gve.rst new file +2
-0+Current gVNIC is an alternative to the virtIO-based ethernet +interface that can support higher network bandwidths such as the 50-100 Gbps speeds.I don't think it is an alternative to virtio. Gve is the driver name of gVNIC of GCP, which is the standard virtual ethernet interface on that cloud platform.
Thanks Helin, Chenbo, Xiaoyun for clarification,
My understanding that gVNIC is another virtual interface spec like 'virtio' is not correct, so comparing this with virtio and to document it is not really necessary.
Perhaps "gVNIC is an alternative to the virtIO-based ethernet interface" part in the documentation can be updated slightly, because as far as I understand gVNIC is an alternative to virtio in the *Google Cloud platform*, not a general alternative.
Overall, agree to enable virtual network interface in Google cloud and as you all highlighted we have similar device support for other cloud platforms.
Thanks,
ferruh
Quoted 19 lines
Quoted 4 lines
This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.
Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?
There is no hint that gve (or virtio) interface is faster than virtio (or gve). I think it heavily depends on the backend design, which can be SW or HW.
I would treat gve (actually the driver for gVNIC) as just one of the leading virtual ethernet interfaces on different cloud environments. Till now, gve is for GCP cloud environment only.Hopefully my understanding is correct, as I am not the expert at Google cloud, and I got all the information from public.
Thanks for all the good questions! Hopefully my answers help!
Regards,
HelinQuoted 3 lines
Aren't there any other notable difference?
I think better to document as much as possible, cc'ed more virtio people.
RE: [PATCH v5 3/8] net/gve: add support for device initialization
In reply to Ferruh YigitHi
Quoted 31 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:47
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initializationQuoted 12 lines · Junfeng Guo
Support device init and add following devops skeleton:
-dev_configure
-dev_start
-dev_stop
-dev_closeNote that build system (including doc) is also added in this patch.
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 14 lines
doc/guides/nics/gve.rst line 9
Quoted 12 lines
doc/guides/nics/gve.rst new file +9
-0@@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2022 Intel Corporation. + +GVE poll mode driver +======================= + +The GVE PMD (**librte_net_gve**) provides poll mode driver support +for Google Virtual Ethernet device (also called as gVNIC). +This is a virtual device, emulated in VM as PCI device, right?
It's similar to ENA driver for AWS. This is Google's own device for GCP.
Quoted 2 lines
If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?
No. backend is not public.
Quoted 7 lines
doc/guides/nics/gve.rst line 11
Quoted 3 lines
doc/guides/nics/gve.rst new file +2
-0+Current gVNIC is an alternative to the virtIO-based ethernet +interface that can support higher network bandwidths such as the 50-100Gbps speeds.
This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.
In GCP, they provide two types of devices which users can choose for the interfaces of the instance, virtio or gvnic, if users choose gvnic, then gve driver is needed and if they choose virtio, vitio is used. I'm not sure we should suggest which one to choose.
I guess the best thing to do is just linking google's doc and let users to choose themselves.
https://cloud.google.com/compute/docs/networking/using-gvnic
Quoted 4 lines
Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?
Aren't there any other notable difference?
This is from https://cloud.google.com/compute/docs/networking/using-gvnic.
There is no official performance report. Google didn't provide any.
From our private testing, gvnic is faster than virtio using the same instance type but virtio is more stable. But our testing can be wrong. It all depends how you config and use cloud env.
Quoted 3 lines
I think better to document as much as possible, cc'ed more virtio people.
PATCH v5 4/8 net/gve: add support for link update
In reply to Junfeng GuoSupport dev_ops link_update.
--- doc/guides/nics/features/gve.ini | 1 + doc/guides/nics/gve.rst | 3 +++ drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 34 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
@@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y Linux = Y x86-32 = Y x86-64 = Y
doc/guides/nics/gve.rst +3 -0
@@ -55,6 +55,9 @@ Features and Limitations In this release, the GVE PMD provides the basic functionality of packet reception and transmission. +Supported features of the GVE PMD are: + +- Link state information Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD. Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +30 -0
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)return 0 ; } +static int +gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) +{ +struct gve_priv *priv = dev->data->dev_private; +struct rte_eth_link link; +int err; + +memset (&link,0 ,sizeof (link)); + link.link_duplex =RTE_ETH_LINK_FULL_DUPLEX ; + link.link_autoneg =RTE_ETH_LINK_AUTONEG ; + +if (!dev->data->dev_started) { + link.link_status =RTE_ETH_LINK_DOWN ; + link.link_speed =RTE_ETH_SPEED_NUM_NONE ; + }else { + link.link_status =RTE_ETH_LINK_UP ; +PMD_DRV_LOG (DEBUG ,"Get link status from hw" ); + err =gve_adminq_report_link_speed (priv); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to get link speed." ); + priv->link_speed =RTE_ETH_SPEED_NUM_UNKNOWN ; + } + link.link_speed = priv->link_speed; + } + +return rte_eth_linkstatus_set (dev, &link); +} +static int gve_dev_start (struct rte_eth_dev *dev) { dev->data->dev_started =1 ; +gve_link_update (dev,0 );return 0 ; }
@@ -70,6 +99,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = { .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, + .link_update = gve_link_update, }; static void
Signature · 1 line
2.34.1
PATCH v5 5/8 net/gve: add support for MTU setting
In reply to Junfeng GuoSupport dev_ops mtu_set.
--- doc/guides/nics/features/gve.ini | 1 + drivers/net/gve/gve_ethdev.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
@@ -5,6 +5,7 @@ ; [Features] Link status = Y +MTU update = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +29 -0
@@ -94,12 +94,41 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_mtu_set (struct rte_eth_dev *dev,uint16_t mtu) +{ +struct gve_priv *priv = dev->data->dev_private; +int err; + +if (mtu <RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) { +PMD_DRV_LOG (ERR ,"MIN MTU is %u MAX MTU is %u" ,RTE_ETHER_MIN_MTU , priv->max_mtu); +return -EINVAL ; + } + +/* mtu setting is forbidden if port is start */ +if (dev->data->dev_started) { +PMD_DRV_LOG (ERR ,"Port must be stopped before configuration" ); +return -EBUSY ; + } + + dev->data->dev_conf.rxmode.mtu = mtu +RTE_ETHER_HDR_LEN ; + + err =gve_adminq_set_mtu (priv, mtu); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to set mtu as %u err = %d" , mtu, err); +return err; + } + +return 0 ; +} +static const struct eth_dev_ops gve_eth_dev_ops = { .dev_configure = gve_dev_configure, .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, .link_update = gve_link_update, + .mtu_set = gve_dev_mtu_set, }; static void
Signature · 1 line
2.34.1
Re: [PATCH v5 5/8] net/gve: add support for MTU setting
Quoted 5 lines · Junfeng Guo
Support
dev_opsmtu_set.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
Quoted 9 lines
drivers/net/gve/gve_ethdev.c +8
-0+static int +gve_dev_mtu_set (struct rte_eth_dev *dev,uint16_t mtu) +{ +struct gve_priv *priv = dev->data->dev_private; +int err; + +if (mtu <RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) { +PMD_DRV_LOG (ERR ,"MIN MTU is %u MAX MTU is %u" ,RTE_ETHER_MIN_MTU , priv->max_mtu);
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 114
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+return -EINVAL ; + } + +/* mtu setting is forbidden if port is start */ +if (dev->data->dev_started) { +PMD_DRV_LOG (ERR ,"Port must be stopped before configuration" ); +return -EBUSY ; + } + + dev->data->dev_conf.rxmode.mtu = mtu +RTE_ETHER_HDR_LEN ;
it is 'dev->data->mtu' that holds latest MTU value.
'dev_conf.rxmode.mtu' is the config requested from user, no need to update that.
And since 'dev->data->mtu' already updated by 'rte_eth_dev_set_mtu()', can drop above line.
RE: [PATCH v5 5/8] net/gve: add support for MTU setting
Quoted 44 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:48
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 5/8] net/gve: add support for MTU settingQuoted 5 lines · Junfeng Guo
Support
dev_opsmtu_set.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)
Quoted 9 lines
drivers/net/gve/gve_ethdev.c +8
-0+static int +gve_dev_mtu_set (struct rte_eth_dev *dev,uint16_t mtu) +{ +struct gve_priv *priv = dev->data->dev_private; +int err; + +if (mtu <RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) { + PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u",
RTE_ETHER_MIN_MTU,priv->max_mtu);drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)
Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6
-0+return -EINVAL ; + } + +/* mtu setting is forbidden if port is start */ +if (dev->data->dev_started) { + PMD_DRV_LOG(ERR, "Port must be stopped beforeconfiguration");
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 114
Quoted 4 lines
drivers/net/gve/gve_ethdev.c +4
-0+return -EBUSY ; + } + + dev->data->dev_conf.rxmode.mtu = mtu +RTE_ETHER_HDR_LEN ;
Quoted 7 lines
it is '
dev->data->mtu' that holds latest MTU value.'dev_conf.rxmode.mtu' is the config requested from user, no need to update that.
And since '
dev->data->mtu' already updated by 'rte_eth_dev_set_mtu()', can drop above line.
Will fix this, thanks!
PATCH v5 6/8 net/gve: add support for dev info get and dev configure
In reply to Junfeng GuoAdd dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.
--- doc/guides/nics/features/gve.ini | 2 + doc/guides/nics/gve.rst | 1 + drivers/net/gve/gve_ethdev.c | 65 +++++++++++++++++++++++++++++++- 3 files changed, 67 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
@@ -4,8 +4,10 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Speed capabilities = Y Link status = Y MTU update = Y +RSS hash = Y Linux = Y x86-32 = Y x86-64 = Y
doc/guides/nics/gve.rst +1 -0
@@ -57,6 +57,7 @@ In this release, the GVE PMD provides the basic functionality of packet reception and transmission. Supported features of the GVE PMD are: +- Receiver Side Scaling (RSS) - Link state information Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
drivers/net/gve/gve_ethdev.c +64 -1
84 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure
Quoted 6 lines · Junfeng Guo
Add
dev_opsdev_infos_get.
Completedev_configurewith RX offloads configuration.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
Quoted 4 lines
drivers/net/gve/gve_ethdev.c +0
-0@@ -29,8 +29,16 @@ gve_write_version(uint8_t *driver_version_register) }
drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register) · line 38
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +6
-1static int -gve_dev_configure(__rte_unused struct rte_eth_dev *dev) +gve_dev_configure (struct rte_eth_dev *dev) { +struct gve_priv *priv = dev->data->dev_private; + +if (dev->data->dev_conf.rxmode.mq_mode &RTE_ETH_MQ_RX_RSS_FLAG ) + dev->data->dev_conf.rxmode.offloads |=RTE_ETH_RX_OFFLOAD_RSS_HASH ; +
This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.
Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?
Quoted 12 lines
drivers/net/gve/gve_ethdev.c +5
-0+if (dev->data->dev_conf.rxmode.offloads &RTE_ETH_RX_OFFLOAD_TCP_LRO ) + priv->enable_rsc =1 ; +return 0 ; }@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_info_get (struct rte_eth_dev *dev,struct rte_eth_dev_info *dev_info)
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+{ +struct gve_priv *priv = dev->data->dev_private; + + dev_info->device = dev->device; + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_queues = priv->max_nb_rxq; + dev_info->max_tx_queues = priv->max_nb_txq; + dev_info->min_rx_bufsize =GVE_MIN_BUF_SIZE ; + dev_info->max_rx_pktlen =GVE_MAX_RX_PKTLEN ; + dev_info->max_mtu =RTE_ETHER_MTU ;
Can you please confirm max MTU this PMD supports is 1500? Meaning it doesn't support jumbo frames etc...
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 126
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+ dev_info->min_mtu =RTE_ETHER_MIN_MTU ; + + dev_info->rx_offload_capa =0 ; + dev_info->tx_offload_capa = +RTE_ETH_TX_OFFLOAD_MULTI_SEGS | +RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | +RTE_ETH_TX_OFFLOAD_UDP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_CKSUM | +RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_TSO ;
Can you adverstise these capabilities in the patch that implements them?
RE: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure
Quoted 42 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configureQuoted 6 lines · Junfeng Guo
Add
dev_opsdev_infos_get.
Completedev_configurewith RX offloads configuration.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t
Quoted 3 lines
drivers/net/gve/gve_ethdev.c +0
-0@@ -29,8 +29,16 @@ gve_write_version(uint8_t
*driver_version_register)drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register)
Quoted 9 lines
drivers/net/gve/gve_ethdev.c +4
-1}static int -gve_dev_configure(__rte_unused struct rte_eth_dev *dev) +gve_dev_configure (struct rte_eth_dev *dev) { +struct gve_priv *priv = dev->data->dev_private; + + if (dev->data->dev_conf.rxmode.mq_mode &
RTE_ETH_MQ_RX_RSS_FLAG)drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 37
Quoted 1 line
drivers/net/gve/gve_ethdev.c +1
-0+ dev->data->dev_conf.rxmode.offloads |=
RTE_ETH_RX_OFFLOAD_RSS_HASH;drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 38
Quoted 1 line
drivers/net/gve/gve_ethdev.c +1
-0+
Quoted 7 lines
This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.
Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?
Yes, it seems force enabling this offloading is not quite reasonable here. This may just follow previous PMD convention, so we decided to remove this part in the coming version. Thanks!
Quoted 15 lines
drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 39
Quoted 2 lines
drivers/net/gve/gve_ethdev.c +1
-0+ if (dev->data->dev_conf.rxmode.offloads &
RTE_ETH_RX_OFFLOAD_TCP_LRO)drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 106
Quoted 11 lines
drivers/net/gve/gve_ethdev.c +4
-0+ priv->enable_rsc =1 ; +return 0 ; }@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info
*dev_info)
Quoted 13 lines
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+{ +struct gve_priv *priv = dev->data->dev_private; + + dev_info->device = dev->device; + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_queues = priv->max_nb_rxq; + dev_info->max_tx_queues = priv->max_nb_txq; + dev_info->min_rx_bufsize =GVE_MIN_BUF_SIZE ; + dev_info->max_rx_pktlen =GVE_MAX_RX_PKTLEN ; + dev_info->max_mtu =RTE_ETHER_MTU ;Can you please confirm max MTU this PMD supports is 1500? Meaning it doesn't support jumbo frames etc...
Actually here is just a workaround solution for the max_mtu info... We can only get the max_mtu value via adminq message from the backend. But the real one (i.e., priv->max_mtu) we get is 1460, which is less than 1500 Seems it is the GCP bug or something.
If we use "dev_info->max_mtu = priv->max_mtu", the testpmd cannot even be launched successfully...
I'll keep this part unchanged with some comments here if no other solutions. Please help correct me if you have any other idea. Thanks a lot!
Quoted 12 lines
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 126
Quoted 11 lines
drivers/net/gve/gve_ethdev.c +10
-0+ dev_info->min_mtu =RTE_ETHER_MIN_MTU ; + + dev_info->rx_offload_capa =0 ; + dev_info->tx_offload_capa = +RTE_ETH_TX_OFFLOAD_MULTI_SEGS | +RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | +RTE_ETH_TX_OFFLOAD_UDP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_CKSUM | +RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_TSO ;
Quoted 1 line
Can you adverstise these capabilities in the patch that implements them?
Will move this to the corresponding patch, thanks!
Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure
Quoted 83 lines · Junfeng
Quoted 42 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configureQuoted 6 lines · Junfeng Guo
Add
dev_opsdev_infos_get.
Completedev_configurewith RX offloads configuration.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t
Quoted 3 lines
drivers/net/gve/gve_ethdev.c +0
-0@@ -29,8 +29,16 @@ gve_write_version(uint8_t
*driver_version_register)drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register)
Quoted 9 lines
drivers/net/gve/gve_ethdev.c +4
-1}static int -gve_dev_configure(__rte_unused struct rte_eth_dev *dev) +gve_dev_configure (struct rte_eth_dev *dev) { +struct gve_priv *priv = dev->data->dev_private; + + if (dev->data->dev_conf.rxmode.mq_mode &
RTE_ETH_MQ_RX_RSS_FLAG)drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 37
Quoted 1 line
drivers/net/gve/gve_ethdev.c +1
-0+ dev->data->dev_conf.rxmode.offloads |=
RTE_ETH_RX_OFFLOAD_RSS_HASH;drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 38
Quoted 1 line
drivers/net/gve/gve_ethdev.c +1
-0+Quoted 7 lines
This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.
Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?
Yes, it seems force enabling this offloading is not quite reasonable here. This may just follow previous PMD convention, so we decided to remove this part in the coming version. Thanks!
Quoted 15 lines
drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 39
Quoted 2 lines
drivers/net/gve/gve_ethdev.c +1
-0+ if (dev->data->dev_conf.rxmode.offloads &
RTE_ETH_RX_OFFLOAD_TCP_LRO)drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 106
Quoted 11 lines
drivers/net/gve/gve_ethdev.c +4
-0+ priv->enable_rsc =1 ; +return 0 ; }@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info
*dev_info)Quoted 13 lines
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+{ +struct gve_priv *priv = dev->data->dev_private; + + dev_info->device = dev->device; + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_queues = priv->max_nb_rxq; + dev_info->max_tx_queues = priv->max_nb_txq; + dev_info->min_rx_bufsize =GVE_MIN_BUF_SIZE ; + dev_info->max_rx_pktlen =GVE_MAX_RX_PKTLEN ; + dev_info->max_mtu =RTE_ETHER_MTU ;Can you please confirm max MTU this PMD supports is 1500? Meaning it doesn't support jumbo frames etc...
Quoted 9 lines
Actually here is just a workaround solution for the
max_mtuinfo... We can only get themax_mtuvalue via adminq message from the backend. But the real one (i.e.,priv->max_mtu) we get is 1460, which is less than 1500 Seems it is the GCP bug or something.
If we use "dev_info->max_mtu=priv->max_mtu", the testpmd cannot even be launched successfully...
I'll keep this part unchanged with some comments here if no other solutions. Please help correct me if you have any other idea. Thanks a lot!
Getting actual value from device is correct thing to do, but it seems received value is not good, so OK to keep as it is. Can you please follow this with GVE?
Quoted 16 lines
Quoted 12 lines
drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 126
Quoted 11 lines
drivers/net/gve/gve_ethdev.c +10
-0+ dev_info->min_mtu =RTE_ETHER_MIN_MTU ; + + dev_info->rx_offload_capa =0 ; + dev_info->tx_offload_capa = +RTE_ETH_TX_OFFLOAD_MULTI_SEGS | +RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | +RTE_ETH_TX_OFFLOAD_UDP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_CKSUM | +RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_TSO ;Quoted 1 line
Can you adverstise these capabilities in the patch that implements them?
Will move this to the corresponding patch, thanks!
RE: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure
Quoted 110 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 19:20
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configureQuoted 11 lines · Junfeng
Quoted 11 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get anddev
Quoted 36 lines
Quoted 36 lines
configure
Quoted 6 lines · Junfeng Guo
Add
dev_opsdev_infos_get.
Completedev_configurewith RX offloads configuration.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t
Quoted 3 lines
drivers/net/gve/gve_ethdev.c +0
-0@@ -29,8 +29,16 @@ gve_write_version(uint8_t
*driver_version_register)drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register)
Quoted 9 lines
drivers/net/gve/gve_ethdev.c +4
-1}static int -gve_dev_configure(__rte_unused struct rte_eth_dev *dev) +gve_dev_configure (struct rte_eth_dev *dev) { +struct gve_priv *priv = dev->data->dev_private; + + if (dev->data->dev_conf.rxmode.mq_mode &
RTE_ETH_MQ_RX_RSS_FLAG)drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 37
Quoted 1 line
drivers/net/gve/gve_ethdev.c +1
-0+ dev->data->dev_conf.rxmode.offloads |=
RTE_ETH_RX_OFFLOAD_RSS_HASH;drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 38
Quoted 1 line
drivers/net/gve/gve_ethdev.c +1
-0+This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.
Quoted 5 lines
Quoted 2 lines
Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?
Yes, it seems force enabling this offloading is not quite reasonable here. This may just follow previous PMD convention, so we decided to
remove
Quoted 29 lines
this part in the coming version. Thanks!
Quoted 27 lines
drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 39
Quoted 2 lines
drivers/net/gve/gve_ethdev.c +1
-0+ if (dev->data->dev_conf.rxmode.offloads &
RTE_ETH_RX_OFFLOAD_TCP_LRO)drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 106
Quoted 11 lines
drivers/net/gve/gve_ethdev.c +4
-0+ priv->enable_rsc =1 ; +return 0 ; }@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info
*dev_info)drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+{ +struct gve_priv *priv = dev->data->dev_private; + + dev_info->device = dev->device; + dev_info->max_mac_addrs =1 ; + dev_info->max_rx_queues = priv->max_nb_rxq; + dev_info->max_tx_queues = priv->max_nb_txq; + dev_info->min_rx_bufsize =GVE_MIN_BUF_SIZE ; + dev_info->max_rx_pktlen =GVE_MAX_RX_PKTLEN ; + dev_info->max_mtu =RTE_ETHER_MTU ;Can you please confirm max MTU this PMD supports is 1500? Meaning
it
Quoted 4 lines
Quoted 1 line
doesn't support jumbo frames etc...
Actually here is just a workaround solution for the
max_mtuinfo... We can only get themax_mtuvalue via adminq message from thebackend.
Quoted 1 line
But the real one (i.e.,
priv->max_mtu) we get is 1460, which is less than1500
Quoted 2 lines
Seems it is the GCP bug or something.
If we use "dev_info->max_mtu=priv->max_mtu", the testpmd cannoteven
Quoted 2 lines
be launched successfully...
I'll keep this part unchanged with some comments here if no other
Quoted 7 lines
solutions.
Quoted 2 lines
Please help correct me if you have any other idea. Thanks a lot!
Getting actual value from device is correct thing to do, but it seems received value is not good, so OK to keep as it is.
Can you please follow this with GVE?
Sure, will update this in the coming version. Thanks!
Quoted 18 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
drivers/net/gve/gve_ethdev.c +10
-0+ dev_info->min_mtu =RTE_ETHER_MIN_MTU ; + + dev_info->rx_offload_capa =0 ; + dev_info->tx_offload_capa = +RTE_ETH_TX_OFFLOAD_MULTI_SEGS | +RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | +RTE_ETH_TX_OFFLOAD_UDP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_CKSUM | +RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | +RTE_ETH_TX_OFFLOAD_TCP_TSO ;Quoted 1 line
Quoted 1 line
Can you adverstise these capabilities in the patch that implements
them?
Quoted 3 lines
Will move this to the corresponding patch, thanks!
PATCH v5 7/8 net/gve: add support for queue operations
In reply to Junfeng GuoAdd support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues
--- drivers/net/gve/gve_ethdev.c | 204 +++++++++++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 52 +++++++++ drivers/net/gve/gve_rx.c | 212 ++++++++++++++++++++++++++++++++++ drivers/net/gve/gve_tx.c | 214 +++++++++++++++++++++++++++++++++++ drivers/net/gve/meson.build | 2 + 5 files changed, 684 insertions(+) create mode 100644 drivers/net/gve/gve_rx.c create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +204 -0
@@ -28,6 +28,68 @@ gve_write_version(uint8_t *driver_version_register)writeb ('\n' , driver_version_register); } +static int +gve_alloc_queue_page_list (struct gve_priv *priv,uint32_t id,uint32_t pages) +{ +char z_name[RTE_MEMZONE_NAMESIZE ]; +struct gve_queue_page_list *qpl; +const struct rte_memzone *mz; +dma_addr_t page_bus; +uint32_t i; + +if (priv->num_registered_pages + pages > + priv->max_registered_pages) { +PMD_DRV_LOG (ERR ,"Pages %" PRIu64" > max registered pages %" PRIu64, + priv->num_registered_pages + pages, + priv->max_registered_pages); +return -EINVAL ; + } + qpl = &priv->qpl[id]; +snprintf (z_name,sizeof (z_name),"gve_%s_qpl%d" , priv->pci_dev->device.name, id); + mz =rte_memzone_reserve_aligned (z_name, pages *PAGE_SIZE , +rte_socket_id (), +RTE_MEMZONE_IOVA_CONTIG ,PAGE_SIZE ); +if (mz ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc %s." , z_name); +return -ENOMEM ; + } + qpl->page_buses =rte_zmalloc ("qpl page buses" , pages *sizeof (dma_addr_t),0 ); +if (qpl->page_buses ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl %u page buses" , id); +return -ENOMEM ; + } + page_bus = mz->iova; +for (i =0 ; i < pages; i++) { + qpl->page_buses[i] = page_bus; + page_bus +=PAGE_SIZE ; + } + qpl->id = id; + qpl->mz = mz; + qpl->num_entries = pages; + + priv->num_registered_pages += pages; + +return 0 ; +} + +static void +gve_free_qpls (struct gve_priv *priv) +{ +uint16_t nb_txqs = priv->max_nb_txq; +uint16_t nb_rxqs = priv->max_nb_rxq; +uint32_t i; + +for (i =0 ; i < nb_txqs + nb_rxqs; i++) { +if (priv->qpl[i].mz !=NULL ) +rte_memzone_free (priv->qpl[i].mz); +if (priv->qpl[i].page_buses !=NULL ) +rte_free (priv->qpl[i].page_buses); + } + +if (priv->qpl !=NULL ) +rte_free (priv->qpl); +} +static int gve_dev_configure (struct rte_eth_dev *dev) {
@@ -42,6 +104,43 @@ gve_dev_configure(struct rte_eth_dev *dev)return 0 ; } +static int +gve_refill_pages (struct gve_rx_queue *rxq) +{ +struct rte_mbuf *nmb; +uint16_t i; +int diag; + + diag =rte_pktmbuf_alloc_bulk (rxq->mpool, &rxq->sw_ring[0 ], rxq->nb_rx_desc); +if (diag <0 ) { +for (i =0 ; i < rxq->nb_rx_desc -1 ; i++) { + nmb =rte_pktmbuf_alloc (rxq->mpool); +if (!nmb) +break ; + rxq->sw_ring[i] = nmb; + } +if (i < rxq->nb_rx_desc -1 ) +return -ENOMEM ; + } + rxq->nb_avail =0 ; + rxq->next_avail = rxq->nb_rx_desc -1 ; + +for (i =0 ; i < rxq->nb_rx_desc; i++) { +if (rxq->is_gqi_qpl) { + rxq->rx_data_ring[i].addr =rte_cpu_to_be_64 (i *PAGE_SIZE ); + }else { +if (i == rxq->nb_rx_desc -1 ) +break ; + nmb = rxq->sw_ring[i]; + rxq->rx_data_ring[i].addr =rte_cpu_to_be_64 (rte_mbuf_data_iova (nmb)); + } + } + +rte_write32 (rte_cpu_to_be_32 (rxq->next_avail), rxq->qrx_tail); + +return 0 ; +} +static int gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) {
@@ -73,16 +172,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)static int gve_dev_start (struct rte_eth_dev *dev) { +uint16_t num_queues = dev->data->nb_tx_queues; +struct gve_priv *priv = dev->data->dev_private; +struct gve_tx_queue *txq; +struct gve_rx_queue *rxq; +uint16_t i; +int err; + + priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues; + err =gve_adminq_create_tx_queues (priv, num_queues); +if (err) { +PMD_DRV_LOG (ERR ,"failed to create %u tx queues." , num_queues); +return err; + } +for (i =0 ; i < num_queues; i++) { + txq = priv->txqs[i]; + txq->qtx_tail = + &priv->db_bar2[rte_be_to_cpu_32 (txq->qres->db_index)]; + txq->qtx_head = + &priv->cnt_array[rte_be_to_cpu_32 (txq->qres->counter_index)]; + +rte_write32 (rte_cpu_to_be_32 (GVE_IRQ_MASK ), txq->ntfy_addr); + } + + num_queues = dev->data->nb_rx_queues; + priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues; + err =gve_adminq_create_rx_queues (priv, num_queues); +if (err) { +PMD_DRV_LOG (ERR ,"failed to create %u rx queues." , num_queues); + goto err_tx; + } +for (i =0 ; i < num_queues; i++) { + rxq = priv->rxqs[i]; + rxq->qrx_tail = + &priv->db_bar2[rte_be_to_cpu_32 (rxq->qres->db_index)]; + +rte_write32 (rte_cpu_to_be_32 (GVE_IRQ_MASK ), rxq->ntfy_addr); + + err =gve_refill_pages (rxq); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to refill for RX" ); + goto err_rx; + } + } + dev->data->dev_started =1 ;gve_link_update (dev,0 );return 0 ; + +err_rx: +gve_stop_rx_queues (dev); +err_tx: +gve_stop_tx_queues (dev); +return err; }static int gve_dev_stop (struct rte_eth_dev *dev) { dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; + +gve_stop_tx_queues (dev); +gve_stop_rx_queues (dev); + dev->data->dev_started =0 ;return 0 ;
@@ -91,7 +244,11 @@ gve_dev_stop(struct rte_eth_dev *dev)static int gve_dev_close (struct rte_eth_dev *dev) { +struct gve_priv *priv = dev->data->dev_private; +struct gve_tx_queue *txq; +struct gve_rx_queue *rxq;int err =0 ; +uint16_t i;if (dev->data->dev_started) { err =gve_dev_stop (dev);
@@ -99,6 +256,19 @@ gve_dev_close(struct rte_eth_dev *dev)PMD_DRV_LOG (ERR ,"Failed to stop dev." ); } +for (i =0 ; i < dev->data->nb_tx_queues; i++) { + txq = dev->data->tx_queues[i]; +gve_tx_queue_release (txq); + } + +for (i =0 ; i < dev->data->nb_rx_queues; i++) { + rxq = dev->data->rx_queues[i]; +gve_rx_queue_release (rxq); + } + +gve_free_qpls (priv); +rte_free (priv->adminq); +return err; }
@@ -190,6 +360,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = { .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, .dev_infos_get = gve_dev_info_get, + .rx_queue_setup = gve_rx_queue_setup, + .tx_queue_setup = gve_tx_queue_setup, .link_update = gve_link_update, .mtu_set = gve_dev_mtu_set, };
@@ -327,7 +499,9 @@ gve_setup_device_resources(struct gve_priv *priv)static int gve_init_priv (struct gve_priv *priv,bool skip_describe_device) { +uint16_t pages;int num_ntfy; +uint32_t i;int err;/* Set up the adminq */
@@ -378,10 +552,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)PMD_DRV_LOG (INFO ,"Max TX queues %d, Max RX queues %d" , priv->max_nb_txq, priv->max_nb_rxq); +/* In GQI_QPL queue format: +* Allocate queue page lists according to max queue number +* tx qpl id should start from 0 while rx qpl id should start +* from priv->max_nb_txq +*/ +if (priv->queue_format ==GVE_GQI_QPL_FORMAT ) { + priv->qpl =rte_zmalloc ("gve_qpl" , + (priv->max_nb_txq + priv->max_nb_rxq) * +sizeof (struct gve_queue_page_list ),0 ); +if (priv->qpl ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl." ); + err = -ENOMEM ; + goto free_adminq; + } + +for (i =0 ; i < priv->max_nb_txq + priv->max_nb_rxq; i++) { +if (i < priv->max_nb_txq) + pages = priv->tx_pages_per_qpl; +else + pages = priv->rx_data_slot_cnt; + err =gve_alloc_queue_page_list (priv, i, pages); +if (err !=0 ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl %u." , i); + goto err_qpl; + } + } + } + setup_device: err =gve_setup_device_resources (priv);if (!err)return 0 ; +err_qpl: +gve_free_qpls (priv); free_adminq:gve_adminq_free (priv);return err;
drivers/net/gve/gve_ethdev.h +52 -0
92 lines not on this page · raw message
drivers/net/gve/gve_rx.c new file +212 -0
212 lines not on this page · raw message
drivers/net/gve/gve_tx.c new file +214 -0
214 lines not on this page · raw message
drivers/net/gve/meson.build +2 -0
@@ -9,6 +9,8 @@ endif sources = files( 'base/gve_adminq.c', + 'gve_rx.c', + 'gve_tx.c', 'gve_ethdev.c', ) includes += include_directories('base')
Signature · 1 line
2.34.1
PATCH v5 8/8 net/gve: add support for Rx/Tx
In reply to Junfeng GuoAdd Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.
--- doc/guides/nics/features/gve.ini | 2 + doc/guides/nics/gve.rst | 4 + drivers/net/gve/gve_ethdev.c | 7 + drivers/net/gve/gve_ethdev.h | 18 ++ drivers/net/gve/gve_rx.c | 140 ++++++++++ drivers/net/gve/gve_tx.c | 455 +++++++++++++++++++++++++++++++ 6 files changed, 626 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
@@ -7,7 +7,9 @@ Speed capabilities = Y Link status = Y MTU update = Y +TSO = Y RSS hash = Y +L4 checksum offload = Y Linux = Y x86-32 = Y x86-64 = Y
doc/guides/nics/gve.rst +4 -0
@@ -57,8 +57,12 @@ In this release, the GVE PMD provides the basic functionality of packet reception and transmission. Supported features of the GVE PMD are: +- Multiple queues for TX and RX - Receiver Side Scaling (RSS) +- TSO offload - Link state information +- TX multi-segments (Scatter TX) +- Tx UDP/TCP/SCTP Checksum Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD. Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +7 -0
@@ -644,6 +644,13 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err; +if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + }else { +PMD_DRV_LOG (ERR ,"DQO_RDA is not implemented and will be added in the future" ); + } + eth_dev->data->mac_addrs =rte_zmalloc ("gve_mac" ,sizeof (struct rte_ether_addr ),0 );if (!eth_dev->data->mac_addrs) {PMD_DRV_LOG (ERR ,"Failed to allocate memory to store mac address" );
drivers/net/gve/gve_ethdev.h +18 -0
@@ -34,6 +34,18 @@ union gve_tx_desc {struct gve_tx_seg_desc seg;/* subsequent descs for a packet */ }; +/* Offload features */ +union gve_tx_offload { +uint64_t data; +struct { +uint64_t l2_len:7 ;/* L2 (MAC) Header Length. */ +uint64_t l3_len:9 ;/* L3 (IP) Header Length. */ +uint64_t l4_len:8 ;/* L4 Header Length. */ +uint64_t tso_segsz:16 ;/* TCP TSO segment size */ +/* uint64_t unused : 24; */ + }; +}; +struct gve_tx_iovec {uint32_t iov_base;/* offset in fifo */ uint32_t iov_len;
@@ -274,4 +286,10 @@ gve_stop_tx_queues(struct rte_eth_dev *dev);void gve_stop_rx_queues (struct rte_eth_dev *dev); +uint16_t +gve_rx_burst (void *rxq,struct rte_mbuf **rx_pkts,uint16_t nb_pkts); + +uint16_t +gve_tx_burst (void *txq,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +#endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +140 -0
146 lines not on this page · raw message
drivers/net/gve/gve_tx.c +455 -0
461 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v5 8/8] net/gve: add support for Rx/Tx
Quoted 5 lines · Junfeng Guo
Add Rx/Tx of
GQI_QPLqueue format andGQI_RDAqueue format.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
Quoted 10 lines
drivers/net/gve/gve_rx.c +10
-0+uint16_t +gve_rx_burst (void *rx_queue,struct rte_mbuf **rx_pkts,uint16_t nb_pkts) +{ +volatile struct gve_rx_desc *rxr, *rxd; +struct gve_rx_queue *rxq = rx_queue; +uint16_t rx_id = rxq->rx_tail; +struct rte_mbuf *rxe; +uint16_t nb_rx, len; +uint64_t addr; +
drivers/net/gve/gve_rx.c line 102
Quoted 10 lines
drivers/net/gve/gve_rx.c +10
-0+ rxr = rxq->rx_desc_ring; + +for (nb_rx =0 ; nb_rx < nb_pkts; nb_rx++) { + rxd = &rxr[rx_id]; +if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno) +break ; + +if (rxd->flags_seq &GVE_RXF_ERR ) +continue ; +
I think above code wrong.
Function returns 'nb_rx', if you continue on error 'nb_rx' kept increased, so application will receive wrong number of packets.
Also packets assigned as "rx_pkts[nb_rx] = rxe;", this will cause gaps in the arrays.
You can either break on the error, or keep another variable to store number of received packets.
drivers/net/gve/gve_rx.c line 105
Quoted 3 lines
drivers/net/gve/gve_rx.c +3
-0+ len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; + rxe = rxq->sw_ring[rx_id]; + rxe->data_off =RTE_PKTMBUF_HEADROOM ;
As far as I can see mbufs allocated using 'rte_pktmbuf_alloc()', if so no need to explicitly set the 'm->data_off'.
RE: [PATCH v5 8/8] net/gve: add support for Rx/Tx
Quoted 48 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:47
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 8/8] net/gve: add support for Rx/TxQuoted 5 lines · Junfeng Guo
Add Rx/Tx of
GQI_QPLqueue format andGQI_RDAqueue format.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Review
Quoted 2 lines
(quoted diff) +2
-0+uint16_t +gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t
nb_pkts)Quoted 8 lines
drivers/net/gve/gve_rx.c +8
-0+{ +volatile struct gve_rx_desc *rxr, *rxd; +struct gve_rx_queue *rxq = rx_queue; +uint16_t rx_id = rxq->rx_tail; +struct rte_mbuf *rxe; +uint16_t nb_rx, len; +uint64_t addr; +drivers/net/gve/gve_rx.c line 102
Quoted 10 lines
drivers/net/gve/gve_rx.c +10
-0+ rxr = rxq->rx_desc_ring; + +for (nb_rx =0 ; nb_rx < nb_pkts; nb_rx++) { + rxd = &rxr[rx_id]; +if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno) +break ; + +if (rxd->flags_seq &GVE_RXF_ERR ) +continue ; +I think above code wrong.
Function returns 'nb_rx', if you continue on error 'nb_rx' kept increased, so application will receive wrong number of packets.
Quoted 5 lines
Also packets assigned as "rx_pkts[nb_rx] = rxe;", this will cause gaps in the arrays.
You can either break on the error, or keep another variable to store number of received packets.
Thanks for pointing out this!
Yes, we will use another variable as index to go through all the nb_pkts and use nb_rx to store all valid pkts. In this way, there would be no gap in the rx_pkts array and the size can align with the final returned nb_rx. Thanks!
Quoted 7 lines
drivers/net/gve/gve_rx.c line 105
Quoted 4 lines
drivers/net/gve/gve_rx.c +3
-0+ len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; + rxe = rxq->sw_ring[rx_id]; + rxe->data_off =RTE_PKTMBUF_HEADROOM ;As far as I can see mbufs allocated using '
rte_pktmbuf_alloc()', if so no need to explicitly set the 'm->data_off'.
Will update this, thanks!
PATCH v4 2/9 net/gve/base: add logs and OS specific implementation
In reply to Junfeng GuoAdd GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.
--- drivers/net/gve/base/gve_adminq.h | 2 + drivers/net/gve/base/gve_desc.h | 2 + drivers/net/gve/base/gve_desc_dqo.h | 2 + drivers/net/gve/base/gve_osdep.h | 159 ++++++++++++++++++++++++++++ drivers/net/gve/base/gve_register.h | 2 + drivers/net/gve/gve_logs.h | 14 +++ 6 files changed, 181 insertions(+) create mode 100644 drivers/net/gve/base/gve_osdep.h create mode 100644 drivers/net/gve/gve_logs.h
drivers/net/gve/base/gve_adminq.h +2 -0
@@ -8,6 +8,8 @@#ifndef _GVE_ADMINQ_H#define _GVE_ADMINQ_H +#include "gve_osdep.h" +/* Admin queue opcodes */ enum gve_adminq_opcodes { GVE_ADMINQ_DESCRIBE_DEVICE = 0x1,
drivers/net/gve/base/gve_desc.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_H_#define _GVE_DESC_H_ +#include "gve_osdep.h" +/* A note on seg_addrs * * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/base/gve_desc_dqo.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_DQO_H_#define _GVE_DESC_DQO_H_ +#include "gve_osdep.h" +#define GVE_TX_MAX_HDR_SIZE_DQO 255#define GVE_TX_MIN_TSO_MSS_DQO 88
drivers/net/gve/base/gve_osdep.h new file +159 -0
159 lines not on this page · raw message
drivers/net/gve/base/gve_register.h +2 -0
@@ -7,6 +7,8 @@ #ifndef _GVE_REGISTER_H_ #define _GVE_REGISTER_H_ +#include "gve_osdep.h" + /* Fixed Configuration Registers */ struct gve_registers { __be32 device_status;
drivers/net/gve/gve_logs.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2022 Intel Corporation +*/ + +#ifndef _GVE_LOGS_H_ +#define _GVE_LOGS_H_ + +extern int gve_logtype_driver; + +#define PMD_DRV_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \ + __func__, ## args) + +#endif
Signature · 1 line
2.34.1
Re: [PATCH v4 2/9] net/gve/base: add logs and OS specific implementation
Quoted 8 lines · Junfeng Guo
Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
Quoted 6 lines
drivers/net/gve/gve_logs.h new file +3
-0@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2022 Intel Corporation +*/
drivers/net/gve/gve_logs.h line 13
Quoted 10 lines
drivers/net/gve/gve_logs.h new file +10
-0+ +#ifndef _GVE_LOGS_H_ +#define _GVE_LOGS_H_ + +extern int gve_logtype_driver; + +#define PMD_DRV_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \ + __func__, ## args) +
What do you think to move 'gve_logs.h' to next patch, since this is extern 'gve_logtype_driver' which is not added yet. Although files are not compiled yet, logically I think it suits better to next patch.
RE: [PATCH v4 2/9] net/gve/base: add logs and OS specific implementation
Quoted 40 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:20
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v4 2/9]net/gve/base: add logs and OS specific implementationQuoted 8 lines · Junfeng Guo
Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 13 lines
drivers/net/gve/gve_logs.h new file +10
-0@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2022 Intel Corporation +*/ + +#ifndef _GVE_LOGS_H_ +#define _GVE_LOGS_H_ + +extern int gve_logtype_driver; + +#define PMD_DRV_LOG(level, fmt, args...) \
Quoted 8 lines
drivers/net/gve/gve_logs.h line 13
Quoted 3 lines
drivers/net/gve/gve_logs.h new file +3
-0+ rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \ + __func__, ## args) +What do you think to move '
gve_logs.h' to next patch, since this is extern 'gve_logtype_driver' which is not added yet. Although files are not compiled yet, logically I think it suits better to next patch.
Sure, make sense. Thanks!
PATCH v4 3/9 net/gve: add support for device initialization
In reply to Junfeng GuoSupport device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close
--- MAINTAINERS | 6 + doc/guides/nics/features/gve.ini | 10 + doc/guides/nics/gve.rst | 69 +++++ doc/guides/nics/index.rst | 1 + doc/guides/rel_notes/release_22_11.rst | 5 + drivers/net/gve/base/gve_adminq.c | 1 + drivers/net/gve/gve_ethdev.c | 371 +++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 225 +++++++++++++++ drivers/net/gve/meson.build | 14 + drivers/net/gve/version.map | 3 + drivers/net/meson.build | 1 + 11 files changed, 706 insertions(+) create mode 100644 doc/guides/nics/features/gve.ini create mode 100644 doc/guides/nics/gve.rst create mode 100644 drivers/net/gve/gve_ethdev.c create mode 100644 drivers/net/gve/gve_ethdev.h create mode 100644 drivers/net/gve/meson.build create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
@@ -697,6 +697,12 @@ F: drivers/net/enic/ F: doc/guides/nics/enic.rst F: doc/guides/nics/features/enic.ini +Google Virtual Ethernet +M: Junfeng Guo <[email protected]> +F: drivers/net/gve/ +F: doc/guides/nics/gve.rst +F: doc/guides/nics/features/gve.ini + Hisilicon hns3 M: Dongdong Liu <[email protected]> M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
@@ -0,0 +1,10 @@ +; +; Supported features of the Google Virtual Ethernet 'gve' poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/gve.rst new file +69 -0
69 lines not on this page · raw message
doc/guides/nics/index.rst +1 -0
@@ -29,6 +29,7 @@ Network Interface Controller Drivers enetfec enic fm10k + gve hinic hns3 i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -59,6 +59,11 @@ New Features * Added flow subscription support. +* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernet devices. + * See the :doc:`../nics/gve` NIC guide for more details on this new driver. + Removed Items -------------
drivers/net/gve/base/gve_adminq.c +1 -0
@@ -5,6 +5,7 @@* Copyright(C) 2022 Intel Corporation */ +#include "../gve_ethdev.h" #include "gve_adminq.h" #include "gve_register.h"
drivers/net/gve/gve_ethdev.c new file +371 -0
371 lines not on this page · raw message
drivers/net/gve/gve_ethdev.h new file +225 -0
225 lines not on this page · raw message
drivers/net/gve/meson.build new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2022 Intel Corporation + +if is_windows + build = false + reason = 'not supported on Windows' + subdir_done() +endif + +sources = files( + 'base/gve_adminq.c', + 'gve_ethdev.c', +) +includes += include_directories('base')
drivers/net/gve/version.map new file +3 -0
@@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +};
drivers/net/meson.build +1 -0
@@ -23,6 +23,7 @@ drivers = [ 'enic', 'failsafe', 'fm10k', + 'gve', 'hinic', 'hns3', 'i40e',
Signature · 1 line
2.34.1
Re: [PATCH v4 3/9] net/gve: add support for device initialization
Quoted 2 lines · Junfeng Guo
Support device init and the fowllowing devops:
s/fowllowing/following/
Quoted 4 lines
-
dev_configure
-dev_start
-dev_stop
-dev_close
At this stage most of above are empty functions and not implemented yet, instead can you document in the commit log that build system and device initialization is added?
Quoted 4 lines
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
Quoted 5 lines
doc/guides/nics/gve.rst new file +2
-0@@ -0,0 +1,69 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2022 Intel Corporation.
doc/guides/nics/gve.rst line 12
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+ +GVE poll mode driver +======================= + +The GVE PMD (**librte_net_gve**) provides poll mode driver support for +Google Virtual Ethernet device. + +Please refer to https://cloud.google.com/compute/docs/networking/using-gvnic +for the device description. +
This seems another virtual interface, similar to iavf/virtio/idpf ...
Can you please briefly describe here the motivation to add yet another virtual interface, and again briefly describe cons/pros of the interface?
Quoted 42 lines
doc/guides/nics/gve.rst new file +42
-0+The base code is under MIT license and based on GVE kernel driver v1.3.0. +GVE base code files are: + +- gve_adminq.h +- gve_adminq.c +- gve_desc.h +- gve_desc_dqo.h +- gve_register.h +- gve.h + +Please refer to https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0/google/gve +to find the original base code. + +GVE has 3 queue formats: + +- GQI_QPL - GQI with queue page list +- GQI_RDA - GQI with raw DMA addressing +- DQO_RDA - DQO with raw DMA addressing + +GQI_QPL queue format is queue page list mode. Driver needs to allocate +memory and register this memory as a Queue Page List (QPL) in hardware +(Google Hypervisor/GVE Backend) first. Each queue has its own QPL. +Then Tx needs to copy packets to QPL memory and put this packet's offset +in the QPL memory into hardware descriptors so that hardware can get the +packets data. And Rx needs to read descriptors of offset in QPL to get +QPL address and copy packets from the address to get real packets data. + +GQI_RDA queue format works like usual NICs that driver can put packets' +physical address into hardware descriptors. + +DQO_RDA queue format has submission and completion queue pair for each +Tx/Rx queue. And similar as GQI_RDA, driver can put packets' physical +address into hardware descriptors. + +Please refer to https://www.kernel.org/doc/html/latest/networking/device_drivers/ethernet/google/gve.html +to get more information about GVE queue formats. + +Features and Limitations +------------------------ + +In this release, the GVE PMD provides the basic functionality of packet +reception and transmission.
doc/guides/nics/gve.rst line 64
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+Supported features of the GVE PMD are: + +- Multiple queues for TX and RX +- Receiver Side Scaling (RSS) +- TSO offload +- Port hardware statistics +- Link state information +- TX multi-segments (Scatter TX) +- Tx UDP/TCP/SCTP Checksum +
Can you build this list gradully, by adding relavent item in each patch that adds it? That way mapping with the code and documented feature becomes more obvious.
<...>
doc/guides/nics/gve.rst
Quoted 7 lines
doc/guides/nics/gve.rst new file +7
-0+static int +gve_dev_uninit(struct rte_eth_dev *eth_dev) +{ + struct gve_priv *priv = eth_dev->data->dev_private; + + eth_dev->data->mac_addrs = NULL; +
At this stage 'mac_addrs' is not freed, setting it to NULL prevents it to be freed.
<...>
doc/guides/nics/gve.rst
Quoted 4 lines
doc/guides/nics/gve.rst new file +4
-0+ +static struct rte_pci_driver rte_gve_pmd = { + .id_table = pci_id_gve_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
As far as I can see LSC interrupt is not supported, if that is correct should we drop the flag?
<...>
Quoted 58 lines
doc/guides/nics/gve.rst new file +58
-0+ +struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz; + + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */ + uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */ + + /* Only valid for DQO_RDA queue format */ + uint16_t tx_compq_size; /* tx completion queue size */ + uint16_t rx_bufq_size; /* rx buff queue size */ + + uint64_t max_registered_pages; + uint64_t num_registered_pages; /* num pages registered with NIC */ + uint16_t default_num_queues; /* default num queues to set up */ + enum gve_queue_format queue_format; /* see enum gve_queue_format */ + uint8_t enable_rsc; + + uint16_t max_nb_txq; + uint16_t max_nb_rxq; + uint32_t num_ntfy_blks; /* spilt between TX and RX so must be even */ + + struct gve_registers __iomem *reg_bar0; /* see gve_register.h */ + rte_be32_t __iomem *db_bar2; /* "array" of doorbells */ + struct rte_pci_device *pci_dev; + + /* Admin queue - see gve_adminq.h*/ + union gve_adminq_command *adminq; + struct gve_dma_mem adminq_dma_mem; + uint32_t adminq_mask; /* masks prod_cnt to adminq size */ + uint32_t adminq_prod_cnt; /* free-running count of AQ cmds executed */ + uint32_t adminq_cmd_fail; /* free-running count of AQ cmds failed */ + uint32_t adminq_timeouts; /* free-running count of AQ cmds timeouts */ + /* free-running count of per AQ cmd executed */ + uint32_t adminq_describe_device_cnt; + uint32_t adminq_cfg_device_resources_cnt; + uint32_t adminq_register_page_list_cnt; + uint32_t adminq_unregister_page_list_cnt; + uint32_t adminq_create_tx_queue_cnt; + uint32_t adminq_create_rx_queue_cnt; + uint32_t adminq_destroy_tx_queue_cnt; + uint32_t adminq_destroy_rx_queue_cnt; + uint32_t adminq_dcfg_device_resources_cnt; + uint32_t adminq_set_driver_parameter_cnt; + uint32_t adminq_report_stats_cnt; + uint32_t adminq_report_link_speed_cnt; + uint32_t adminq_get_ptype_map_cnt; + + volatile uint32_t state_flags; + + /* Gvnic device link speed from hypervisor. */ + uint64_t link_speed;
doc/guides/nics/gve.rst
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+ + uint16_t max_mtu; + struct rte_ether_addr dev_addr; /* mac address */ + + struct gve_queue_page_list *qpl; + + struct gve_tx_queue **txqs; + struct gve_rx_queue **rxqs; +}; +
Similar to previous comment, can you construct the headers by only adding used fields in that patch?
When batch copied an existing struct, it is very easy to add unused code and very hard to detect it. So if you only add what you need, that becomes easy to be sure all fields are used.
Also it makes more obvious which fields related to which feature.
<...>
drivers/net/gve/version.map line 3
Quoted 8 lines
new file mode 100644
index 0000000000..c2e0723b4cdrivers/net/gve/version.map new file +3
-0@@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +};
it is 'DPDK_23' now, hopefully we will have an update to get rid of empty map files, feel free to review:
https://patches.dpdk.org/project/dpdk/list/?series=25002
RE: [PATCH v4 3/9] net/gve: add support for device initialization
Quoted 20 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:23
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v4 3/9] net/gve: add support for device initializationQuoted 2 lines · Junfeng Guo
Support device init and the fowllowing devops:
s/fowllowing/following/
Quoted 8 lines
Quoted 4 lines
-
dev_configure
-dev_start
-dev_stop
-dev_closeAt this stage most of above are empty functions and not implemented yet, instead can you document in the commit log that build system and device initialization is added?
Agreed, will add this in the coming version. Thanks!
Quoted 23 lines
Quoted 5 lines
Signed-off-by Haiyue Wang <[email protected]>Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
doc/guides/nics/gve.rst line 7
Quoted 10 lines
doc/guides/nics/gve.rst new file +7
-0@@ -0,0 +1,69 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(C) 2022 Intel Corporation. + +GVE poll mode driver +======================= + +The GVE PMD (**librte_net_gve**) provides poll mode driver supportfor
doc/guides/nics/gve.rst
Quoted 3 lines
doc/guides/nics/gve.rst new file +3
-0+Google Virtual Ethernet device. + +Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic
Quoted 7 lines
doc/guides/nics/gve.rst line 12
Quoted 2 lines
doc/guides/nics/gve.rst new file +2
-0+for the device description. +This seems another virtual interface, similar to iavf/virtio/idpf ...
Can you please briefly describe here the motivation to add yet another virtual interface, and again briefly describe cons/pros of the interface?
Sure. According to the official gVNIC description, gVNIC is an alternative to the virtio driver that can provide higher network bandwidths. Will add the brief descriptions of the cons/pros in the coming version. Thanks!
Quoted 61 lines
doc/guides/nics/gve.rst line 13
Quoted 2 lines
doc/guides/nics/gve.rst new file +1
-0+The base code is under MIT license and based on GVE kernel driverv1.3.0.
doc/guides/nics/gve.rst
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+GVE base code files are: + +- gve_adminq.h +- gve_adminq.c +- gve_desc.h +- gve_desc_dqo.h +- gve_register.h +- gve.h + +Please refer to https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0/google/gve
doc/guides/nics/gve.rst
Quoted 9 lines
doc/guides/nics/gve.rst new file +9
-0+to find the original base code. + +GVE has 3 queue formats: + +- GQI_QPL - GQI with queue page list +- GQI_RDA - GQI with raw DMA addressing +- DQO_RDA - DQO with raw DMA addressing + +GQI_QPL queue format is queue page list mode. Driver needs toallocate
doc/guides/nics/gve.rst line 33
Quoted 1 line
doc/guides/nics/gve.rst new file +1
-0+memory and register this memory as a Queue Page List (QPL) inhardware
doc/guides/nics/gve.rst line 35
Quoted 2 lines
doc/guides/nics/gve.rst new file +2
-0+(Google Hypervisor/GVE Backend) first. Each queue has its own QPL. +Then Tx needs to copy packets to QPL memory and put this packet'soffset
doc/guides/nics/gve.rst line 36
Quoted 1 line
doc/guides/nics/gve.rst new file +1
-0+in the QPL memory into hardware descriptors so that hardware can getthe
doc/guides/nics/gve.rst line 38
Quoted 2 lines
doc/guides/nics/gve.rst new file +2
-0+packets data. And Rx needs to read descriptors of offset in QPL to get +QPL address and copy packets from the address to get real packetsdata.
doc/guides/nics/gve.rst line 40
Quoted 2 lines
doc/guides/nics/gve.rst new file +2
-0+ +GQI_RDA queue format works like usual NICs that driver can putpackets'
doc/guides/nics/gve.rst
Quoted 3 lines
doc/guides/nics/gve.rst new file +3
-0+physical address into hardware descriptors. + +DQO_RDA queue format has submission and completion queue pair foreach
doc/guides/nics/gve.rst
Quoted 4 lines
doc/guides/nics/gve.rst new file +4
-0+Tx/Rx queue. And similar as GQI_RDA, driver can put packets' physical +address into hardware descriptors. + +Please refer tohttps://www.kernel.org/doc/html/latest/networking/device_drivers/ethe
rnet/google/gve.htmlQuoted 14 lines
doc/guides/nics/gve.rst new file +14
-0+to get more information about GVE queue formats. + +Features and Limitations +------------------------ + +In this release, the GVE PMD provides the basic functionality of packet +reception and transmission. +Supported features of the GVE PMD are: + +- Multiple queues for TX and RX +- Receiver Side Scaling (RSS) +- TSO offload +- Port hardware statistics +- Link state information
Quoted 8 lines
doc/guides/nics/gve.rst line 64
Quoted 3 lines
doc/guides/nics/gve.rst new file +3
-0+- TX multi-segments (Scatter TX) +- Tx UDP/TCP/SCTP Checksum +Can you build this list gradully, by adding relavent item in each patch that adds it?
That way mapping with the code and documented feature becomes more obvious.
Sure... Will add the items of this list gradually in the coming version. Thanks!
Quoted 13 lines
<...>
doc/guides/nics/gve.rst
Quoted 7 lines
doc/guides/nics/gve.rst new file +7
-0+static int +gve_dev_uninit(struct rte_eth_dev *eth_dev) +{ + struct gve_priv *priv = eth_dev->data->dev_private; + + eth_dev->data->mac_addrs = NULL; +At this stage '
mac_addrs' is not freed, setting it to NULL prevents it to be freed.
Thanks for the catch! Will improve this. Thanks!
Quoted 11 lines
<...>
doc/guides/nics/gve.rst
Quoted 4 lines
doc/guides/nics/gve.rst new file +4
-0+ +static struct rte_pci_driver rte_gve_pmd = { + .id_table = pci_id_gve_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING |
RTE_PCI_DRV_INTR_LSC,As far as I can see LSC interrupt is not supported, if that is correct should we drop the flag?
Sure, seems this flag is not used in current GCP env. Will remove it in the coming version. Thanks!
Quoted 81 lines
<...>
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+ +struct gve_priv { + struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */ + const struct rte_memzone *irq_dbs_mz; + uint32_t mgmt_msix_idx; + rte_be32_t *cnt_array; /* array of num_event_counters */ + const struct rte_memzone *cnt_array_mz; + + uint16_t num_event_counters; + uint16_t tx_desc_cnt; /* txq size */doc/guides/nics/gve.rst
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+ uint16_t rx_desc_cnt; /* rxq size */ + uint16_t tx_pages_per_qpl; /* tx buffer length */ + uint16_t rx_data_slot_cnt; /* rx buffer length */ + + /* Only valid for DQO_RDA queue format */ + uint16_t tx_compq_size; /* tx completion queue size */ + uint16_t rx_bufq_size; /* rx buff queue size */ + + uint64_t max_registered_pages; + uint64_t num_registered_pages; /* num pages registered with NIC*/
doc/guides/nics/gve.rst
Quoted 2 lines
doc/guides/nics/gve.rst new file +2
-0+ uint16_t default_num_queues; /* default num queues to set up */ + enum gve_queue_format queue_format; /* see enum
gve_queue_format*/doc/guides/nics/gve.rst
Quoted 5 lines
doc/guides/nics/gve.rst new file +5
-0+ uint8_t enable_rsc; + + uint16_t max_nb_txq; + uint16_t max_nb_rxq; + uint32_t num_ntfy_blks; /* spilt between TX and RX so must beeven */
doc/guides/nics/gve.rst
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+ + struct gve_registers __iomem *reg_bar0; /* see gve_register.h */ + rte_be32_t __iomem *db_bar2; /* "array" of doorbells */ + struct rte_pci_device *pci_dev; + + /* Admin queue - see gve_adminq.h*/ + union gve_adminq_command *adminq; + struct gve_dma_mem adminq_dma_mem; + uint32_t adminq_mask; /* masks prod_cnt to adminq size */ + uint32_t adminq_prod_cnt; /* free-running count of AQ cmdsexecuted */
Quoted 1 line
+
uint32_tadminq_cmd_fail; /* free-running count of AQ cmds failed*/
Quoted 1 line
+
uint32_tadminq_timeouts; /* free-running count of AQ cmdstimeouts */
Quoted 19 lines
doc/guides/nics/gve.rst new file +19
-0+ /* free-running count of per AQ cmd executed */ + uint32_t adminq_describe_device_cnt; + uint32_t adminq_cfg_device_resources_cnt; + uint32_t adminq_register_page_list_cnt; + uint32_t adminq_unregister_page_list_cnt; + uint32_t adminq_create_tx_queue_cnt; + uint32_t adminq_create_rx_queue_cnt; + uint32_t adminq_destroy_tx_queue_cnt; + uint32_t adminq_destroy_rx_queue_cnt; + uint32_t adminq_dcfg_device_resources_cnt; + uint32_t adminq_set_driver_parameter_cnt; + uint32_t adminq_report_stats_cnt; + uint32_t adminq_report_link_speed_cnt; + uint32_t adminq_get_ptype_map_cnt; + + volatile uint32_t state_flags; + + /* Gvnic device link speed from hypervisor. */ + uint64_t link_speed;doc/guides/nics/gve.rst
Quoted 10 lines
doc/guides/nics/gve.rst new file +10
-0+ + uint16_t max_mtu; + struct rte_ether_addr dev_addr; /* mac address */ + + struct gve_queue_page_list *qpl; + + struct gve_tx_queue **txqs; + struct gve_rx_queue **rxqs; +}; +Similar to previous comment, can you construct the headers by only adding used fields in that patch?
Quoted 5 lines
When batch copied an existing struct, it is very easy to add unused code and very hard to detect it. So if you only add what you need, that becomes easy to be sure all fields are used.
Also it makes more obvious which fields related to which feature.
Sure... Will try best to construct the header structure items gradually. Thanks!
Quoted 15 lines
<...>
drivers/net/gve/version.map line 3
Quoted 8 lines
new file mode 100644
index 0000000000..c2e0723b4cdrivers/net/gve/version.map new file +3
-0@@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +};it is '
DPDK_23' now, hopefully we will have an update to get rid of empty map files, feel free to review:
https://patches.dpdk.org/project/dpdk/list/?series=25002
Sure, it looks much better. Thanks!
Quoted 1 line
PATCH v4 4/9 net/gve: add support for link update
In reply to Junfeng GuoSupport dev_ops link_update.
--- doc/guides/nics/features/gve.ini | 2 ++ drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
@@ -4,6 +4,8 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Speed capabilities = Y +Link status = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +30 -0
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)return 0 ; } +static int +gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) +{ +struct gve_priv *priv = dev->data->dev_private; +struct rte_eth_link link; +int err; + +memset (&link,0 ,sizeof (link)); + link.link_duplex =RTE_ETH_LINK_FULL_DUPLEX ; + link.link_autoneg =RTE_ETH_LINK_AUTONEG ; + +if (!dev->data->dev_started) { + link.link_status =RTE_ETH_LINK_DOWN ; + link.link_speed =RTE_ETH_SPEED_NUM_NONE ; + }else { + link.link_status =RTE_ETH_LINK_UP ; +PMD_DRV_LOG (DEBUG ,"Get link status from hw" ); + err =gve_adminq_report_link_speed (priv); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to get link speed." ); + priv->link_speed =RTE_ETH_SPEED_NUM_UNKNOWN ; + } + link.link_speed = priv->link_speed; + } + +return rte_eth_linkstatus_set (dev, &link); +} +static int gve_dev_start (struct rte_eth_dev *dev) { dev->data->dev_started =1 ; +gve_link_update (dev,0 );return 0 ; }
@@ -70,6 +99,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = { .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, + .link_update = gve_link_update, }; static void
Signature · 1 line
2.34.1
Re: [PATCH v4 4/9] net/gve: add support for link update
Quoted 15 lines · Junfeng Guo
Support
dev_opslink_update.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>--- doc/guides/nics/features/gve.ini | 2 ++ drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)doc/guides/nics/features/gve.ini +0
-0
doc/guides/nics/features/gve.ini line 7
Quoted 4 lines
doc/guides/nics/features/gve.ini +1
-0@@ -4,6 +4,8 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Speed capabilities = Y
'Speed capabilities' is when device reports supported speeds in 'rte_eth_dev_info_get()', so it shouldn't be in this patch.
Please check 'doc/guides/nics/features.rst' for more details.
RE: [PATCH v4 4/9] net/gve: add support for link update
Quoted 39 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:23
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 4/9] net/gve: add support for link updateQuoted 8 lines · Junfeng Guo
Support
dev_opslink_update.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>--- doc/guides/nics/features/gve.ini | 2 ++ drivers/net/gve/gve_ethdev.c | 30++++++++++++++++++++++++++++++
Review
Quoted 3 lines
2 files changed, 32 insertions(+)(quoted diff) +0
-0b/doc/guides/nics/features/gve.ini
doc/guides/nics/features/gve.ini line 7
Quoted 8 lines
index
44aec28009..d03e3ac89e100644doc/guides/nics/features/gve.ini +1
-0@@ -4,6 +4,8 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Speed capabilities = Y'Speed capabilities' is when device reports supported speeds in '
rte_eth_dev_info_get()', so it shouldn't be in this patch.
Quoted 1 line
Please check '
doc/guides/nics/features.rst' for more details.
Thanks for the comments!
Will check the features file and update accordingly.
PATCH v4 5/9 net/gve: add support for MTU setting
In reply to Junfeng GuoSupport dev_ops mtu_set.
--- doc/guides/nics/features/gve.ini | 1 + drivers/net/gve/gve_ethdev.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
@@ -6,6 +6,7 @@ [Features] Speed capabilities = Y Link status = Y +MTU update = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +29 -0
@@ -94,12 +94,41 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_mtu_set (struct rte_eth_dev *dev,uint16_t mtu) +{ +struct gve_priv *priv = dev->data->dev_private; +int err; + +if (mtu <RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) { +PMD_DRV_LOG (ERR ,"MIN MTU is %u MAX MTU is %u" ,RTE_ETHER_MIN_MTU , priv->max_mtu); +return -EINVAL ; + } + +/* mtu setting is forbidden if port is start */ +if (dev->data->dev_started) { +PMD_DRV_LOG (ERR ,"Port must be stopped before configuration" ); +return -EBUSY ; + } + + dev->data->dev_conf.rxmode.mtu = mtu +RTE_ETHER_HDR_LEN ; + + err =gve_adminq_set_mtu (priv, mtu); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to set mtu as %u err = %d" , mtu, err); +return err; + } + +return 0 ; +} +static const struct eth_dev_ops gve_eth_dev_ops = { .dev_configure = gve_dev_configure, .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, .link_update = gve_link_update, + .mtu_set = gve_dev_mtu_set, }; static void
Signature · 1 line
2.34.1
PATCH v4 6/9 net/gve: add support for queue operations
In reply to Junfeng GuoAdd support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues
--- drivers/net/gve/gve_ethdev.c | 206 +++++++++++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 48 ++++++++ drivers/net/gve/gve_rx.c | 212 ++++++++++++++++++++++++++++++++++ drivers/net/gve/gve_tx.c | 214 +++++++++++++++++++++++++++++++++++ drivers/net/gve/meson.build | 2 + 5 files changed, 682 insertions(+) create mode 100644 drivers/net/gve/gve_rx.c create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +206 -0
@@ -28,12 +28,111 @@ gve_write_version(uint8_t *driver_version_register)writeb ('\n' , driver_version_register); } +static int +gve_alloc_queue_page_list (struct gve_priv *priv,uint32_t id,uint32_t pages) +{ +char z_name[RTE_MEMZONE_NAMESIZE ]; +struct gve_queue_page_list *qpl; +const struct rte_memzone *mz; +dma_addr_t page_bus; +uint32_t i; + +if (priv->num_registered_pages + pages > + priv->max_registered_pages) { +PMD_DRV_LOG (ERR ,"Pages %" PRIu64" > max registered pages %" PRIu64, + priv->num_registered_pages + pages, + priv->max_registered_pages); +return -EINVAL ; + } + qpl = &priv->qpl[id]; +snprintf (z_name,sizeof (z_name),"gve_%s_qpl%d" , priv->pci_dev->device.name, id); + mz =rte_memzone_reserve_aligned (z_name, pages *PAGE_SIZE , +rte_socket_id (), +RTE_MEMZONE_IOVA_CONTIG ,PAGE_SIZE ); +if (mz ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc %s." , z_name); +return -ENOMEM ; + } + qpl->page_buses =rte_zmalloc ("qpl page buses" , pages *sizeof (dma_addr_t),0 ); +if (qpl->page_buses ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl %u page buses" , id); +return -ENOMEM ; + } + page_bus = mz->iova; +for (i =0 ; i < pages; i++) { + qpl->page_buses[i] = page_bus; + page_bus +=PAGE_SIZE ; + } + qpl->id = id; + qpl->mz = mz; + qpl->num_entries = pages; + + priv->num_registered_pages += pages; + +return 0 ; +} + +static void +gve_free_qpls (struct gve_priv *priv) +{ +uint16_t nb_txqs = priv->max_nb_txq; +uint16_t nb_rxqs = priv->max_nb_rxq; +uint32_t i; + +for (i =0 ; i < nb_txqs + nb_rxqs; i++) { +if (priv->qpl[i].mz !=NULL ) +rte_memzone_free (priv->qpl[i].mz); +if (priv->qpl[i].page_buses !=NULL ) +rte_free (priv->qpl[i].page_buses); + } + +if (priv->qpl !=NULL ) +rte_free (priv->qpl); +} +static int gve_dev_configure(__rte_unused struct rte_eth_dev *dev) {return 0 ; } +static int +gve_refill_pages (struct gve_rx_queue *rxq) +{ +struct rte_mbuf *nmb; +uint16_t i; +int diag; + + diag =rte_pktmbuf_alloc_bulk (rxq->mpool, &rxq->sw_ring[0 ], rxq->nb_rx_desc); +if (diag <0 ) { +for (i =0 ; i < rxq->nb_rx_desc -1 ; i++) { + nmb =rte_pktmbuf_alloc (rxq->mpool); +if (!nmb) +break ; + rxq->sw_ring[i] = nmb; + } +if (i < rxq->nb_rx_desc -1 ) +return -ENOMEM ; + } + rxq->nb_avail =0 ; + rxq->next_avail = rxq->nb_rx_desc -1 ; + +for (i =0 ; i < rxq->nb_rx_desc; i++) { +if (rxq->is_gqi_qpl) { + rxq->rx_data_ring[i].addr =rte_cpu_to_be_64 (i *PAGE_SIZE ); + }else { +if (i == rxq->nb_rx_desc -1 ) +break ; + nmb = rxq->sw_ring[i]; + rxq->rx_data_ring[i].addr =rte_cpu_to_be_64 (rte_mbuf_data_iova (nmb)); + } + } + +rte_write32 (rte_cpu_to_be_32 (rxq->next_avail), rxq->qrx_tail); + +return 0 ; +} +static int gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) {
@@ -65,16 +164,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)static int gve_dev_start (struct rte_eth_dev *dev) { +uint16_t num_queues = dev->data->nb_tx_queues; +struct gve_priv *priv = dev->data->dev_private; +struct gve_tx_queue *txq; +struct gve_rx_queue *rxq; +uint16_t i; +int err; + + priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues; + err =gve_adminq_create_tx_queues (priv, num_queues); +if (err) { +PMD_DRV_LOG (ERR ,"failed to create %u tx queues." , num_queues); +return err; + } +for (i =0 ; i < num_queues; i++) { + txq = priv->txqs[i]; + txq->qtx_tail = + &priv->db_bar2[rte_be_to_cpu_32 (txq->qres->db_index)]; + txq->qtx_head = + &priv->cnt_array[rte_be_to_cpu_32 (txq->qres->counter_index)]; + +rte_write32 (rte_cpu_to_be_32 (GVE_IRQ_MASK ), txq->ntfy_addr); + } + + num_queues = dev->data->nb_rx_queues; + priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues; + err =gve_adminq_create_rx_queues (priv, num_queues); +if (err) { +PMD_DRV_LOG (ERR ,"failed to create %u rx queues." , num_queues); + goto err_tx; + } +for (i =0 ; i < num_queues; i++) { + rxq = priv->rxqs[i]; + rxq->qrx_tail = + &priv->db_bar2[rte_be_to_cpu_32 (rxq->qres->db_index)]; + +rte_write32 (rte_cpu_to_be_32 (GVE_IRQ_MASK ), rxq->ntfy_addr); + + err =gve_refill_pages (rxq); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to refill for RX" ); + goto err_rx; + } + } + dev->data->dev_started =1 ;gve_link_update (dev,0 );return 0 ; + +err_rx: +gve_stop_rx_queues (dev); +err_tx: +gve_stop_tx_queues (dev); +return err; }static int gve_dev_stop (struct rte_eth_dev *dev) { dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; + +gve_stop_tx_queues (dev); +gve_stop_rx_queues (dev); + dev->data->dev_started =0 ;return 0 ;
@@ -83,7 +236,11 @@ gve_dev_stop(struct rte_eth_dev *dev)static int gve_dev_close (struct rte_eth_dev *dev) { +struct gve_priv *priv = dev->data->dev_private; +struct gve_tx_queue *txq; +struct gve_rx_queue *rxq;int err =0 ; +uint16_t i;if (dev->data->dev_started) { err =gve_dev_stop (dev);
@@ -91,6 +248,21 @@ gve_dev_close(struct rte_eth_dev *dev)PMD_DRV_LOG (ERR ,"Failed to stop dev." ); } +for (i =0 ; i < dev->data->nb_tx_queues; i++) { + txq = dev->data->tx_queues[i]; +gve_tx_queue_release (txq); + } + +for (i =0 ; i < dev->data->nb_rx_queues; i++) { + rxq = dev->data->rx_queues[i]; +gve_rx_queue_release (rxq); + } + +gve_free_qpls (priv); +rte_free (priv->adminq); +rte_free (priv->qpl); +rte_free (priv); +return err; }
@@ -127,6 +299,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = { .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, + .rx_queue_setup = gve_rx_queue_setup, + .tx_queue_setup = gve_tx_queue_setup, .link_update = gve_link_update, .mtu_set = gve_dev_mtu_set, };
@@ -264,7 +438,9 @@ gve_setup_device_resources(struct gve_priv *priv)static int gve_init_priv (struct gve_priv *priv,bool skip_describe_device) { +uint16_t pages;int num_ntfy; +uint32_t i;int err;/* Set up the adminq */
@@ -315,10 +491,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)PMD_DRV_LOG (INFO ,"Max TX queues %d, Max RX queues %d" , priv->max_nb_txq, priv->max_nb_rxq); +/* In GQI_QPL queue format: +* Allocate queue page lists according to max queue number +* tx qpl id should start from 0 while rx qpl id should start +* from priv->max_nb_txq +*/ +if (priv->queue_format ==GVE_GQI_QPL_FORMAT ) { + priv->qpl =rte_zmalloc ("gve_qpl" , + (priv->max_nb_txq + priv->max_nb_rxq) * +sizeof (struct gve_queue_page_list ),0 ); +if (priv->qpl ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl." ); + err = -ENOMEM ; + goto free_adminq; + } + +for (i =0 ; i < priv->max_nb_txq + priv->max_nb_rxq; i++) { +if (i < priv->max_nb_txq) + pages = priv->tx_pages_per_qpl; +else + pages = priv->rx_data_slot_cnt; + err =gve_alloc_queue_page_list (priv, i, pages); +if (err !=0 ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl %u." , i); + goto err_qpl; + } + } + } + setup_device: err =gve_setup_device_resources (priv);if (!err)return 0 ; +err_qpl: +gve_free_qpls (priv); free_adminq:gve_adminq_free (priv);return err;
drivers/net/gve/gve_ethdev.h +48 -0
88 lines not on this page · raw message
drivers/net/gve/gve_rx.c new file +212 -0
212 lines not on this page · raw message
drivers/net/gve/gve_tx.c new file +214 -0
214 lines not on this page · raw message
drivers/net/gve/meson.build +2 -0
@@ -9,6 +9,8 @@ endif sources = files( 'base/gve_adminq.c', + 'gve_rx.c', + 'gve_tx.c', 'gve_ethdev.c', ) includes += include_directories('base')
Signature · 1 line
2.34.1
PATCH v4 7/9 net/gve: add support for Rx/Tx
In reply to Junfeng GuoAdd Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.
--- doc/guides/nics/features/gve.ini | 2 + drivers/net/gve/gve_ethdev.c | 5 + drivers/net/gve/gve_ethdev.h | 16 ++ drivers/net/gve/gve_rx.c | 143 ++++++++++ drivers/net/gve/gve_tx.c | 455 +++++++++++++++++++++++++++++++ 5 files changed, 621 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
@@ -7,6 +7,8 @@ Speed capabilities = Y Link status = Y MTU update = Y +TSO = Y +L4 checksum offload = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +5 -0
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err; +if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + } + eth_dev->data->mac_addrs =rte_zmalloc ("gve_mac" ,sizeof (struct rte_ether_addr ),0 );if (!eth_dev->data->mac_addrs) {PMD_DRV_LOG (ERR ,"Failed to allocate memory to store mac address" );
drivers/net/gve/gve_ethdev.h +16 -0
@@ -34,6 +34,18 @@ union gve_tx_desc {struct gve_tx_seg_desc seg;/* subsequent descs for a packet */ }; +/* Offload features */ +union gve_tx_offload { +uint64_t data; +struct { +uint64_t l2_len:7 ;/* L2 (MAC) Header Length. */ +uint64_t l3_len:9 ;/* L3 (IP) Header Length. */ +uint64_t l4_len:8 ;/* L4 Header Length. */ +uint64_t tso_segsz:16 ;/* TCP TSO segment size */ +/* uint64_t unused : 24; */ + }; +}; +struct gve_tx_iovec {uint32_t iov_base;/* offset in fifo */ uint32_t iov_len;
@@ -270,4 +282,8 @@ void gve_stop_tx_queues(struct rte_eth_dev *dev);void gve_stop_rx_queues (struct rte_eth_dev *dev); +uint16_t gve_rx_burst (void *rxq,struct rte_mbuf **rx_pkts,uint16_t nb_pkts); + +uint16_t gve_tx_burst (void *txq,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +#endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +143 -0
149 lines not on this page · raw message
drivers/net/gve/gve_tx.c +455 -0
461 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v4 7/9] net/gve: add support for Rx/Tx
Quoted 5 lines · Junfeng Guo
Add Rx/Tx of
GQI_QPLqueue format andGQI_RDAqueue format.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>
<...>
drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590
Quoted 11 lines
drivers/net/gve/gve_ethdev.c +5
-0@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err; +if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + } +
What do you think to add a log here for 'else' case, to inform user why datapath is not working?
<...>
Quoted 17 lines
drivers/net/gve/gve_ethdev.c +17
-0+uint16_t +gve_rx_burst (void *rx_queue,struct rte_mbuf **rx_pkts,uint16_t nb_pkts) +{ +volatile struct gve_rx_desc *rxr, *rxd; +struct gve_rx_queue *rxq = rx_queue; +uint16_t rx_id = rxq->rx_tail; +struct rte_mbuf *rxe; +uint16_t nb_rx, len; +uint64_t addr; + + rxr = rxq->rx_desc_ring; + +for (nb_rx =0 ; nb_rx < nb_pkts; nb_rx++) { + rxd = &rxr[rx_id]; +if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno) +break ; +
drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev)
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+if (rxd->flags_seq &GVE_RXF_ERR ) +continue ; + + len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; + rxe = rxq->sw_ring[rx_id]; + rxe->data_off =RTE_PKTMBUF_HEADROOM ; +if (rxq->is_gqi_qpl) { + addr = (uint64_t )(rxq->qpl->mz->addr) + rx_id *PAGE_SIZE +GVE_RX_PAD ; +rte_memcpy ((void *)((size_t )rxe->buf_addr + rxe->data_off), + (void *)(size_t )addr, len);
Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?
drivers/net/gve/gve_ethdev.c
Quoted 9 lines
drivers/net/gve/gve_ethdev.c +9
-0+ } + rxe->nb_segs =1 ; + rxe->next =NULL ; + rxe->pkt_len = len; + rxe->data_len = len; + rxe->port = rxq->port_id; + rxe->packet_type =0 ; + rxe->ol_flags =0 ; +
As far as I can see 'sw_ring[]' filled using 'rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.
drivers/net/gve/gve_ethdev.c
Quoted 9 lines
drivers/net/gve/gve_ethdev.c +9
-0+if (rxd->flags_seq &GVE_RXF_TCP ) + rxe->packet_type |=RTE_PTYPE_L4_TCP ; +if (rxd->flags_seq &GVE_RXF_UDP ) + rxe->packet_type |=RTE_PTYPE_L4_UDP ; +if (rxd->flags_seq &GVE_RXF_IPV4 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV4 ; +if (rxd->flags_seq &GVE_RXF_IPV6 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV6 ; +
If you are setting packet_type, it is better to implement 'dev_supported_ptypes_get()' dev_ops too, to announce host which packet type parsin supporting. (+ dev_ptypes_set() dev_ops) And later driver can announce "Packet type parsing" feature in .ini file.
RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx
Quoted 28 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 7/9] net/gve: add support for Rx/TxQuoted 5 lines · Junfeng Guo
Add Rx/Tx of
GQI_QPLqueue format andGQI_RDAqueue format.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 6 lines
drivers/net/gve/gve_ethdev.c +0
-0@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err;
Quoted 8 lines
drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590
Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5
-0+if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + } +What do you think to add a log here for 'else' case, to inform user why datapath is not working?
Agreed, make sense!
Currently only one queue mode (i.e., qpl mode) is supported on the GCP env. Will add a log to inform this in the else case. Thanks!
Quoted 30 lines
<...>
drivers/net/gve/gve_ethdev.c
Quoted 2 lines
drivers/net/gve/gve_ethdev.c +2
-0+uint16_t +gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t
nb_pkts)Quoted 13 lines
drivers/net/gve/gve_ethdev.c +13
-0+{ +volatile struct gve_rx_desc *rxr, *rxd; +struct gve_rx_queue *rxq = rx_queue; +uint16_t rx_id = rxq->rx_tail; +struct rte_mbuf *rxe; +uint16_t nb_rx, len; +uint64_t addr; + + rxr = rxq->rx_desc_ring; + +for (nb_rx =0 ; nb_rx < nb_pkts; nb_rx++) { + rxd = &rxr[rx_id]; +if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno)drivers/net/gve/gve_ethdev.c
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+break ; + +if (rxd->flags_seq &GVE_RXF_ERR ) +continue ; + + len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; + rxe = rxq->sw_ring[rx_id]; + rxe->data_off =RTE_PKTMBUF_HEADROOM ; +if (rxq->is_gqi_qpl) { + addr = (uint64_t )(rxq->qpl->mz->addr) + rx_id *PAGE_SIZE +
GVE_RX_PAD;
Quoted 5 lines
Quoted 3 lines
+ rte_memcpy((void *)((size_t)rxe->buf_addr + rxe-
data_off),
+ (void *)(size_t)addr, len);Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?
Well, only qpl (queue page list) mode supported on the GCP env now. So the DMA may not be used in current case.
Quoted 11 lines
drivers/net/gve/gve_ethdev.c
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9
-0+ } + rxe->nb_segs =1 ; + rxe->next =NULL ; + rxe->pkt_len = len; + rxe->data_len = len; + rxe->port = rxq->port_id; + rxe->packet_type =0 ; + rxe->ol_flags =0 ; +
Quoted 3 lines
As far as I can see 'sw_ring[]' filled using '
rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.
Yes, some fields are already assigned at 'rte_pktmbuf_reset()'. Will remove the redundant ones in the coming version. Thanks!
Quoted 11 lines
drivers/net/gve/gve_ethdev.c
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9
-0+if (rxd->flags_seq &GVE_RXF_TCP ) + rxe->packet_type |=RTE_PTYPE_L4_TCP ; +if (rxd->flags_seq &GVE_RXF_UDP ) + rxe->packet_type |=RTE_PTYPE_L4_UDP ; +if (rxd->flags_seq &GVE_RXF_IPV4 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV4 ; +if (rxd->flags_seq &GVE_RXF_IPV6 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV6 ; +
Quoted 5 lines
If you are setting
packet_type, it is better to implement 'dev_supported_ptypes_get()'dev_opstoo, to announce host which packet
type parsin supporting. (+dev_ptypes_set()dev_ops)
And later driver can announce "Packet type parsing" feature in .ini file.
Well, on current GCP env, the APIs for supported ptypes get/set have not been exposed even in the base code. The only one in the base code is for the dqo mode (gve_adminq_get_ptype_map_dqo). But this also cannot be used on current GCP env. We can only implement this once they are supported and exposed at GCP. Thanks!
RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx
Hi
Quoted 47 lines
-----Original Message-----
From: Guo, Junfeng <[email protected]>
Sent: Sunday, October 9, 2022 10:15
To: Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: RE: [PATCH v4 7/9] net/gve: add support for Rx/TxQuoted 26 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 7/9] net/gve: add support for Rx/TxQuoted 5 lines · Junfeng Guo
Add Rx/Tx of
GQI_QPLqueue format andGQI_RDAqueue format.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 6 lines
drivers/net/gve/gve_ethdev.c +0
-0@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err;Quoted 8 lines
drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590
Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5
-0+if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + } +What do you think to add a log here for 'else' case, to inform user why datapath is not working?
Quoted 3 lines
Agreed, make sense!
Currently only one queue mode (i.e., qpl mode) is supported on the GCP env. Will add a log to inform this in the else case. Thanks!
This explanation is not correct. Only QPL mode is supported in GCP now. This is env limitation but not related to the else code here. gve_is_gqi() includes two modes GQI_QPL and GQI_RDA. And both of these datapath is supported in rxtx.
GQI means its queue model is single queue model (txq for tx and rxq for rx). And there're 2 ways for this queue model QPL and RDA. QPL needs to copy packets from/to several reserved pages negotiated with backend. RDA is just like normal device and uses PA in descs.
The datapath not supported is DQO_RDA which uses different hardware so different queue model (split/double queue model). Tx will use txq and tx_completion_q and Rx will use rxq and rx_completion_q. This is not implemented in the datapath for now and will be implemented in the future.
So if you want to add comment here. Please say "DQO_RDA is not implemented and will be added in the future". Don't say it's not available in GCP env which is not the reason.
Quoted 32 lines
Quoted 32 lines
<...>
drivers/net/gve/gve_ethdev.c
Quoted 2 lines
drivers/net/gve/gve_ethdev.c +2
-0+uint16_t +gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t
nb_pkts)Quoted 14 lines
drivers/net/gve/gve_ethdev.c +14
-0+{ +volatile struct gve_rx_desc *rxr, *rxd; +struct gve_rx_queue *rxq = rx_queue; +uint16_t rx_id = rxq->rx_tail; +struct rte_mbuf *rxe; +uint16_t nb_rx, len; +uint64_t addr; + + rxr = rxq->rx_desc_ring; + +for (nb_rx =0 ; nb_rx < nb_pkts; nb_rx++) { + rxd = &rxr[rx_id]; +if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno) +break ;drivers/net/gve/gve_ethdev.c
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+ +if (rxd->flags_seq &GVE_RXF_ERR ) +continue ; + + len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; + rxe = rxq->sw_ring[rx_id]; + rxe->data_off =RTE_PKTMBUF_HEADROOM ; +if (rxq->is_gqi_qpl) { + addr = (uint64_t )(rxq->qpl->mz->addr) + + rx_id *PAGE_SIZE +
GVE_RX_PAD;
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
+ rte_memcpy((void *)((size_t)rxe->buf_addr +
+ rxe-data_off),
+ (void *)(size_t)addr, len);Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?
When queue model is gpi_qpl (this is negotiated and gotten using adminq with backend), the device needs to register a block of memory (called page list). And tx needs to copy the packets to this memory and rx will get packets from this area. Backend will be responsible for getting(tx)/giving(rx) packets from this memory to the device/line (We don't really know how backend does this).
Please refer to https://www.kernel.org/doc/html/v5.4/networking/device_drivers/google/gve.html. There's a bit more explanation about this queue format.
Quoted 3 lines
Well, only qpl (queue page list) mode supported on the GCP env now. So the DMA may not be used in current case.
And yes, it's because GCP doesn't support GQI_RDA for now so GQI_QPL has to be implemented. But even if GCP env supports RDA in the future, unless they completely remove QPL support, QPL is still needed. Because queue format/model is getting from backend through gve_adminq_describe_device(). You may just get the QPI version. The device can't really control which queue format to get.
Quoted 35 lines
Quoted 12 lines
drivers/net/gve/gve_ethdev.c
Quoted 11 lines
drivers/net/gve/gve_ethdev.c +9
-0+ } + rxe->nb_segs =1 ; + rxe->next =NULL ; + rxe->pkt_len = len; + rxe->data_len = len; + rxe->port = rxq->port_id; + rxe->packet_type =0 ; + rxe->ol_flags =0 ; +Quoted 3 lines
As far as I can see 'sw_ring[]' filled using '
rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.Yes, some fields are already assigned at '
rte_pktmbuf_reset()'. Will remove the redundant ones in the coming version. Thanks!Quoted 11 lines
drivers/net/gve/gve_ethdev.c
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9
-0+if (rxd->flags_seq &GVE_RXF_TCP ) + rxe->packet_type |=RTE_PTYPE_L4_TCP ; +if (rxd->flags_seq &GVE_RXF_UDP ) + rxe->packet_type |=RTE_PTYPE_L4_UDP ; +if (rxd->flags_seq &GVE_RXF_IPV4 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV4 ; +if (rxd->flags_seq &GVE_RXF_IPV6 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV6 ; +Quoted 4 lines
If you are setting
packet_type, it is better to implement 'dev_supported_ptypes_get()'dev_opstoo, to announce host which packet type parsin supporting. (+dev_ptypes_set()dev_ops) And later driver can announce "Packet type parsing" feature in .ini file.
Quoted 5 lines
Well, on current GCP env, the APIs for supported ptypes get/set have not been exposed even in the base code. The only one in the base code is for the dqo mode (
gve_adminq_get_ptype_map_dqo). But this also cannot be used on current GCP env. We can only implement this once they are supported and exposed at GCP. Thanks!
You're mixing the concept again. GCP env only supports QPL is not an excuse.
The packet type is supported even in QPL. It's just very limited to L4_TCP/UDP and L3_IPV4/6. Ptypes_get is possible and it'll be RTE_PTYPE_L3_IPV4/6 and RTE_PTYPE_L4_UDP/TCP.
For DQO mode you mentioned, it'll be more flexible and have more support. I'm not sure what's your plan but it can be implemented whenever based on the plan not GCP env availability. The base code is there. It's just you may not be able to timely verify and debug it.
Ptype_set is not supported since the hardware doesn't support it (There's no such adminq).
RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx
Thanks Xiaoyun for helping explain, it helps a lot!
Quoted 81 lines
-----Original Message-----
From: Li, Xiaoyun <[email protected]>
Sent: Monday, October 10, 2022 17:40
To: Guo, Junfeng <[email protected]>; Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: RE: [PATCH v4 7/9] net/gve: add support for Rx/TxHi
Quoted 19 lines
-----Original Message-----
From: Guo, Junfeng <[email protected]>
Sent: Sunday, October 9, 2022 10:15
To: Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: RE: [PATCH v4 7/9] net/gve: add support for Rx/TxQuoted 7 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson,Bruce
Quoted 28 lines
Quoted 19 lines
<[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 7/9] net/gve: add support for Rx/TxQuoted 5 lines · Junfeng Guo
Add Rx/Tx of
GQI_QPLqueue format andGQI_RDAqueue format.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]><...>
Quoted 6 lines
drivers/net/gve/gve_ethdev.c +0
-0@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err;Quoted 8 lines
drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590
Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5
-0+if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + } +What do you think to add a log here for 'else' case, to inform user why datapath is not working?
Quoted 2 lines
Agreed, make sense!
Currently only one queue mode (i.e., qpl mode) is supported on the GCPenv.
Quoted 1 line
Will add a log to inform this in the else case. Thanks!
This explanation is not correct. Only QPL mode is supported in GCP now. This is env limitation but not related to the else code here.
gve_is_gqi()includes two modesGQI_QPLandGQI_RDA. And both of these datapath is supported in rxtx.
GQI means its queue model is single queue model (txq for tx and rxq for rx). And there're 2 ways for this queue model QPL and RDA. QPL needs to copy packets from/to several reserved pages negotiated with backend. RDA is just like normal device and uses PA in descs.The datapath not supported is
DQO_RDAwhich uses different hardware so different queue model (split/double queue model). Tx will use txq andtx_completion_qand Rx will use rxq andrx_completion_q. This is not implemented in the datapath for now and will be implemented in the future.
Quoted 3 lines
So if you want to add comment here. Please say "
DQO_RDAis not implemented and will be added in the future". Don't say it's not available in GCP env which is not the reason.
Okay, will add this in the coming version. Thanks!
Quoted 54 lines
Quoted 33 lines
Quoted 33 lines
<...>
drivers/net/gve/gve_ethdev.c
Quoted 2 lines
drivers/net/gve/gve_ethdev.c +2
-0+uint16_t +gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t
nb_pkts)Quoted 14 lines
drivers/net/gve/gve_ethdev.c +14
-0+{ +volatile struct gve_rx_desc *rxr, *rxd; +struct gve_rx_queue *rxq = rx_queue; +uint16_t rx_id = rxq->rx_tail; +struct rte_mbuf *rxe; +uint16_t nb_rx, len; +uint64_t addr; + + rxr = rxq->rx_desc_ring; + +for (nb_rx =0 ; nb_rx < nb_pkts; nb_rx++) { + rxd = &rxr[rx_id]; +if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno) +break ;drivers/net/gve/gve_ethdev.c
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+ +if (rxd->flags_seq &GVE_RXF_ERR ) +continue ; + + len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; + rxe = rxq->sw_ring[rx_id]; + rxe->data_off =RTE_PKTMBUF_HEADROOM ; +if (rxq->is_gqi_qpl) { + addr = (uint64_t )(rxq->qpl->mz->addr) + + rx_id *PAGE_SIZE +
GVE_RX_PAD;Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
+ rte_memcpy((void *)((size_t)rxe->buf_addr +
+ rxe-data_off),
+ (void *)(size_t)addr, len);Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?
When queue model is
gpi_qpl(this is negotiated and gotten using adminq with backend), the device needs to register a block of memory (called page list). And tx needs to copy the packets to this memory and rx will get packets from this area.
Backend will be responsible for getting(tx)/giving(rx) packets from this memory to the device/line (We don't really know how backend does this). Please refer to
https://www.kernel.org/doc/html/v5.4/networking/device_drivers/googl
e/gve.html. There's a bit more explanation about this queue format.Quoted 3 lines
Well, only qpl (queue page list) mode supported on the GCP env now. So the DMA may not be used in current case.
Quoted 6 lines
And yes, it's because GCP doesn't support
GQI_RDAfor now soGQI_QPLhas to be implemented. But even if GCP env supports RDA in the future, unless they completely remove QPL support, QPL is still needed. Because queue format/model is getting from backend throughgve_adminq_describe_device(). You may just get the QPI version. The device can't really control which queue format to get.
Thanks for the explanation!
Quoted 55 lines
Quoted 31 lines
Quoted 13 lines
drivers/net/gve/gve_ethdev.c
Quoted 12 lines
drivers/net/gve/gve_ethdev.c +9
-0+ } + rxe->nb_segs =1 ; + rxe->next =NULL ; + rxe->pkt_len = len; + rxe->data_len = len; + rxe->port = rxq->port_id; + rxe->packet_type =0 ; + rxe->ol_flags =0 ; +Quoted 3 lines
As far as I can see 'sw_ring[]' filled using '
rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.Yes, some fields are already assigned at '
rte_pktmbuf_reset()'. Will remove the redundant ones in the coming version. Thanks!Quoted 11 lines
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9
-0+if (rxd->flags_seq &GVE_RXF_TCP ) + rxe->packet_type |=RTE_PTYPE_L4_TCP ; +if (rxd->flags_seq &GVE_RXF_UDP ) + rxe->packet_type |=RTE_PTYPE_L4_UDP ; +if (rxd->flags_seq &GVE_RXF_IPV4 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV4 ; +if (rxd->flags_seq &GVE_RXF_IPV6 ) + rxe->packet_type |=RTE_PTYPE_L3_IPV6 ; +Quoted 6 lines
Quoted 4 lines
If you are setting
packet_type, it is better to implement 'dev_supported_ptypes_get()'dev_opstoo, to announce host which packet type parsin supporting. (+dev_ptypes_set()dev_ops) And later driver can announce "Packet type parsing" feature in .ini file.Well, on current GCP env, the APIs for supported ptypes get/set have
not
Quoted 1 line
been exposed even in the base code. The only one in the base code is
for
Quoted 1 line
the dqo mode (
gve_adminq_get_ptype_map_dqo). But this also cannotbe
Quoted 2 lines
used on current GCP env. We can only implement this once they are supported and exposed at GCP. Thanks!
You're mixing the concept again. GCP env only supports QPL is not an excuse.
The packet type is supported even in QPL. It's just very limited to L4_TCP/UDP and L3_IPV4/6.Ptypes_getis possible and it'll be RTE_PTYPE_L3_IPV4/6 and RTE_PTYPE_L4_UDP/TCP.
For DQO mode you mentioned, it'll be more flexible and have more support. I'm not sure what's your plan but it can be implemented whenever based on the plan not GCP env availability. The base code is there. It's just you may not be able to timely verify and debug it.
Quoted 2 lines
Ptype_setis not supported since the hardware doesn't support it (There's no such adminq).
Okay... no much bandwidth to implement at this point. Maybe next release, thanks!
PATCH v4 8/9 net/gve: add support for dev info get and dev configure
In reply to Junfeng GuoAdd dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.
--- doc/guides/nics/features/gve.ini | 1 + drivers/net/gve/gve_ethdev.c | 65 +++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +1 -0
@@ -8,6 +8,7 @@ Speed capabilities = Y Link status = Y MTU update = Y TSO = Y +RSS hash = Y L4 checksum offload = Y Linux = Y x86-32 = Y
drivers/net/gve/gve_ethdev.c +64 -1
84 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v4 8/9] net/gve: add support for dev info get and dev configure
Quoted 4 lines · Junfeng Guo
Add
dev_opsdev_infos_get.
Completedev_configurewith RX offloads configuration.
I think better to have this before datapath patches (6/9 & 7/9), because this is more fundamental step.
RE: [PATCH v4 8/9] net/gve: add support for dev info get and dev configure
Quoted 16 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 8/9] net/gve: add support for dev info get and dev configureQuoted 1 line · Junfeng Guo
Quoted 6 lines
Quoted 3 lines
Add
dev_opsdev_infos_get.
Completedev_configurewith RX offloads configuration.I think better to have this before datapath patches (6/9 & 7/9), because this is more fundamental step.
Sure, make sense! Will reorder the patches in the coming version patchset. Thanks!
PATCH v4 9/9 net/gve: add support for stats
In reply to Junfeng GuoUpdate stats add support of dev_ops stats_get/reset.
--- doc/guides/nics/features/gve.ini | 2 + drivers/net/gve/gve_ethdev.c | 71 ++++++++++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 11 +++++ drivers/net/gve/gve_rx.c | 15 ++++++- drivers/net/gve/gve_tx.c | 13 ++++++ 5 files changed, 110 insertions(+), 2 deletions(-)
doc/guides/nics/features/gve.ini +2 -0
@@ -10,6 +10,8 @@ MTU update = Y TSO = Y RSS hash = Y L4 checksum offload = Y +Basic stats = Y +Stats per queue = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +71 -0
83 lines not on this page · raw message
drivers/net/gve/gve_ethdev.h +11 -0
@@ -76,6 +76,11 @@ struct gve_tx_queue {struct gve_queue_page_list *qpl;struct gve_tx_iovec *iov_ring; +/* Stats */ +uint64_t errors; +uint64_t packets; +uint64_t bytes; +uint16_t port_id;uint16_t queue_id;
@@ -114,6 +119,12 @@ struct gve_rx_queue {/* only valid for GQI_QPL queue format */ struct gve_queue_page_list *qpl; +/* stats */ +uint64_t no_mbufs; +uint64_t errors; +uint64_t packets; +uint64_t bytes; +struct gve_priv *hw;const struct rte_memzone *qres_mz;struct gve_queue_resources *qres;
drivers/net/gve/gve_rx.c +13 -2
@@ -26,8 +26,10 @@ gve_rx_refill(struct gve_rx_queue *rxq)break ; rxq->sw_ring[idx + i] = nmb; } -if (i != nb_alloc) +if (i != nb_alloc) { + rxq->no_mbufs += nb_alloc - i; nb_alloc = i; + } } rxq->nb_avail -= nb_alloc; next_avail += nb_alloc;
@@ -88,6 +90,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)uint16_t rx_id = rxq->rx_tail;struct rte_mbuf *rxe;uint16_t nb_rx, len; +uint64_t bytes =0 ;uint64_t addr; rxr = rxq->rx_desc_ring;
@@ -97,8 +100,10 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno)break ; -if (rxd->flags_seq &GVE_RXF_ERR ) +if (rxd->flags_seq &GVE_RXF_ERR ) { + rxq->errors++;continue ; + } len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; rxe = rxq->sw_ring[rx_id];
@@ -137,6 +142,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rx_id =0 ; rx_pkts[nb_rx] = rxe; + bytes += len; } rxq->nb_avail += nb_rx;
@@ -145,6 +151,11 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)if (rxq->nb_avail > rxq->free_thresh)gve_rx_refill (rxq); +if (nb_rx) { + rxq->packets += nb_rx; + rxq->bytes += bytes; + } +return nb_rx; }
drivers/net/gve/gve_tx.c +13 -0
@@ -260,6 +260,7 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)struct rte_mbuf *tx_pkt, *first;uint16_t sw_id = txq->sw_tail;uint16_t nb_used, i; +uint64_t bytes =0 ;uint16_t nb_tx =0 ;uint32_t hlen;
@@ -355,6 +356,8 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) txq->nb_free -= nb_used; txq->sw_nb_free -= first->nb_segs; tx_tail += nb_used; + + bytes += first->pkt_len; } end_of_tx:
@@ -362,6 +365,10 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)rte_write32 (rte_cpu_to_be_32 (tx_tail), txq->qtx_tail); txq->tx_tail = tx_tail; txq->sw_tail = sw_id; + + txq->errors += nb_pkts - nb_tx; + txq->packets += nb_tx; + txq->bytes += bytes; }return nb_tx;
@@ -380,6 +387,7 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)struct rte_mbuf *tx_pkt, *first;uint16_t nb_used, hlen, i;uint64_t ol_flags, addr; +uint64_t bytes =0 ;uint16_t nb_tx =0 ; txr = txq->tx_desc_ring;
@@ -438,12 +446,17 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) txq->nb_free -= nb_used; tx_tail += nb_used; + + bytes += first->pkt_len; } end_of_tx:if (nb_tx) {rte_write32 (rte_cpu_to_be_32 (tx_tail), txq->qtx_tail); txq->tx_tail = tx_tail; + + txq->packets += nb_tx; + txq->bytes += bytes; }return nb_tx;
Signature · 1 line
2.34.1
Re: [PATCH v4 9/9] net/gve: add support for stats
Quoted 18 lines · Junfeng Guo
Update stats add support of
dev_opsstats_get/reset.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>--- doc/guides/nics/features/gve.ini | 2 + drivers/net/gve/gve_ethdev.c | 71 ++++++++++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 11 +++++ drivers/net/gve/gve_rx.c | 15 ++++++- drivers/net/gve/gve_tx.c | 13 ++++++ 5 files changed, 110 insertions(+), 2 deletions(-)doc/guides/nics/features/gve.ini +0
-0
doc/guides/nics/features/gve.ini MTU update = Y · line 14
Quoted 5 lines
doc/guides/nics/features/gve.ini +2
-0@@ -10,6 +10,8 @@ MTU update = Y TSO = Y RSS hash = Y L4 checksum offload = Y +Basic stats = Y +Stats per queue = Y
"stats per queue" is something else, agree that it is bad naming, please check features.rst file.
Quoted 18 lines
doc/guides/nics/features/gve.ini +0
-0Linux = Y x86-32 = Y x86-64 = Ydrivers/net/gve/gve_ethdev.c +7
-0@@ -328,6 +328,75 @@ gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)return 0 ; } +static int +gve_dev_stats_get (struct rte_eth_dev *dev,struct rte_eth_stats *stats) +{ +uint16_t i; + +for (i =0 ; i < dev->data->nb_tx_queues; i++) { +struct gve_tx_queue *txq = dev->data->tx_queues[i];
drivers/net/gve/gve_ethdev.c gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) · line 347
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+if (txq ==NULL ) +continue ; + + stats->opackets += txq->packets; + stats->obytes += txq->bytes; + stats->oerrors += txq->errors; + +if (i <RTE_ETHDEV_QUEUE_STAT_CNTRS ) { + stats->q_opackets[i] = txq->packets; + stats->q_obytes[i] = txq->bytes;
Queue stats update is moved to xstat [1], we are waiting existing PMDs to adopt it but for new drivers it is better to implement new method.
Can you please either drop queue stats completely, or implement it via xstats?
[1]
https://elixir.bootlin.com/dpdk/v22.07/source/doc/guides/rel_notes/deprecation.rst#L118
RE: [PATCH v4 9/9] net/gve: add support for stats
Quoted 30 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:26
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 9/9] net/gve: add support for statsQuoted 8 lines · Junfeng Guo
Update stats add support of
dev_opsstats_get/reset.Signed-off-by Xiaoyun Li <[email protected]>Signed-off-by Junfeng Guo <[email protected]>--- doc/guides/nics/features/gve.ini | 2 + drivers/net/gve/gve_ethdev.c | 71++++++++++++++++++++++++++++++++
Review
Quoted 6 lines
drivers/net/gve/gve_ethdev.h | 11 +++++ drivers/net/gve/gve_rx.c | 15 ++++++- drivers/net/gve/gve_tx.c | 13 ++++++ 5 files changed, 110 insertions(+), 2 deletions(-)(quoted diff) +0
-0b/doc/guides/nics/features/gve.ini
Quoted 12 lines
doc/guides/nics/features/gve.ini MTU update = Y · line 14
Quoted 9 lines
index
cdc46b08a3..180408aa80100644doc/guides/nics/features/gve.ini +2
-0@@ -10,6 +10,8 @@ MTU update = Y TSO = Y RSS hash = Y L4 checksum offload = Y +Basic stats = Y +Stats per queue = Y"stats per queue" is something else, agree that it is bad naming, please check
features.rstfile.
Sure, will check the features file and update accordingly. Thanks!
Quoted 35 lines
Review
Quoted 5 lines
doc/guides/nics/features/gve.ini +0
-0Linux = Y x86-32 = Y x86-64 = Y(quoted diff) +0
-0
b/drivers/net/gve/gve_ethdev.cdrivers/net/gve/gve_ethdev.c gve_dev_info_get(struct rte_eth_dev *dev,
Quoted 4 lines
index
b9b8e51b02..cd474b8128100644drivers/net/gve/gve_ethdev.c +0
-0@@ -328,6 +328,75 @@ gve_dev_info_get(struct rte_eth_dev *dev,struct
rte_eth_dev_info*dev_info)Quoted 10 lines
drivers/net/gve/gve_ethdev.c +7
-0return 0 ; } +static int +gve_dev_stats_get (struct rte_eth_dev *dev,struct rte_eth_stats *stats) +{ +uint16_t i; + +for (i =0 ; i < dev->data->nb_tx_queues; i++) { +struct gve_tx_queue *txq = dev->data->tx_queues[i];drivers/net/gve/gve_ethdev.c gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) · line 347
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10
-0+if (txq ==NULL ) +continue ; + + stats->opackets += txq->packets; + stats->obytes += txq->bytes; + stats->oerrors += txq->errors; + +if (i <RTE_ETHDEV_QUEUE_STAT_CNTRS ) { + stats->q_opackets[i] = txq->packets; + stats->q_obytes[i] = txq->bytes;Queue stats update is moved to xstat [1], we are waiting existing PMDs to adopt it but for new drivers it is better to implement new method.
Quoted 6 lines
Can you please either drop queue stats completely, or implement it via xstats?
[1]
https://elixir.bootlin.com/dpdk/v22.07/source/doc/guides/rel_notes/dep
recation.rst#L118
Sure, thanks for reminding this!
Seems that it would be better to drop the stats feature at this point since lack of time and bandwidth... But we can still measure the performance via other tools like TRex. And we can plan the implementation of the new method in the coming release. Thanks!
PATCH v3 2/9 net/gve: add logs and OS specific implementation
In reply to Junfeng GuoAdd GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.
--- drivers/net/gve/gve_adminq.h | 2 + drivers/net/gve/gve_desc.h | 2 + drivers/net/gve/gve_desc_dqo.h | 2 + drivers/net/gve/gve_logs.h | 14 +++ drivers/net/gve/gve_osdep.h | 159 +++++++++++++++++++++++++++++++++ drivers/net/gve/gve_register.h | 2 + 6 files changed, 181 insertions(+) create mode 100644 drivers/net/gve/gve_logs.h create mode 100644 drivers/net/gve/gve_osdep.h
drivers/net/gve/gve_adminq.h +2 -0
@@ -8,6 +8,8 @@#ifndef _GVE_ADMINQ_H#define _GVE_ADMINQ_H +#include "gve_osdep.h" +/* Admin queue opcodes */ enum gve_adminq_opcodes { GVE_ADMINQ_DESCRIBE_DEVICE = 0x1,
drivers/net/gve/gve_desc.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_H_#define _GVE_DESC_H_ +#include "gve_osdep.h" +/* A note on seg_addrs * * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/gve_desc_dqo.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_DQO_H_#define _GVE_DESC_DQO_H_ +#include "gve_osdep.h" +#define GVE_TX_MAX_HDR_SIZE_DQO 255#define GVE_TX_MIN_TSO_MSS_DQO 88
drivers/net/gve/gve_logs.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2022 Intel Corporation +*/ + +#ifndef _GVE_LOGS_H_ +#define _GVE_LOGS_H_ + +extern int gve_logtype_driver; + +#define PMD_DRV_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \ + __func__, ## args) + +#endif
drivers/net/gve/gve_osdep.h new file +159 -0
159 lines not on this page · raw message
drivers/net/gve/gve_register.h +2 -0
@@ -7,6 +7,8 @@ #ifndef _GVE_REGISTER_H_ #define _GVE_REGISTER_H_ +#include "gve_osdep.h" + /* Fixed Configuration Registers */ struct gve_registers { __be32 device_status;
Signature · 1 line
2.34.1
Re: [PATCH v3 2/9] net/gve: add logs and OS specific implementation
drivers/net/gve/gve_logs.h line 12
Quoted 4 lines · On Fri, 23 Sep 2022 17:38:22 +0800 Junfeng Guo <[email protected]> wrote:
drivers/net/gve/gve_logs.h new file +4
-0+ +#define PMD_DRV_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \ + __func__, ## args)
Many of your existing log messages already have newline, so using this common definition will create double spaced log messages.
Please audit all usages and print one newline.
RE: [PATCH v3 2/9] net/gve: add logs and OS specific implementation
Quoted 21 lines
-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Saturday, September 24, 2022 03:01
To: Guo, Junfeng <[email protected]>
Cc: Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v3 2/9] net/gve: add logs and OS specific implementationdrivers/net/gve/gve_logs.h
Quoted 3 lines · On Fri, 23 Sep 2022 17:38:22 +0800 Junfeng Guo <[email protected]> wrote:
drivers/net/gve/gve_logs.h new file +3
-0+ +#define PMD_DRV_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n",\
Quoted 6 lines
drivers/net/gve/gve_logs.h line 12
Quoted 1 line
drivers/net/gve/gve_logs.h new file +1
-0+ __func__, ## args)Many of your existing log messages already have newline, so using this common definition will create double spaced log messages.
Please audit all usages and print one newline.
Sure, will double check all the usages and update in the coming version. Thanks!
PATCH v3 3/9 net/gve: support device initialization
In reply to Junfeng GuoSupport device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close
--- MAINTAINERS | 6 + doc/guides/nics/features/gve.ini | 10 + doc/guides/nics/gve.rst | 69 +++++ doc/guides/nics/index.rst | 1 + doc/guides/rel_notes/release_22_11.rst | 5 + drivers/net/gve/gve_adminq.c | 1 + drivers/net/gve/gve_ethdev.c | 371 +++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 225 +++++++++++++++ drivers/net/gve/meson.build | 13 + drivers/net/gve/version.map | 3 + drivers/net/meson.build | 1 + 11 files changed, 705 insertions(+) create mode 100644 doc/guides/nics/features/gve.ini create mode 100644 doc/guides/nics/gve.rst create mode 100644 drivers/net/gve/gve_ethdev.c create mode 100644 drivers/net/gve/gve_ethdev.h create mode 100644 drivers/net/gve/meson.build create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
@@ -697,6 +697,12 @@ F: drivers/net/enic/ F: doc/guides/nics/enic.rst F: doc/guides/nics/features/enic.ini +Google Virtual Ethernet +M: Junfeng Guo <[email protected]> +F: drivers/net/gve/ +F: doc/guides/nics/gve.rst +F: doc/guides/nics/features/gve.ini + Hisilicon hns3 M: Dongdong Liu <[email protected]> M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
@@ -0,0 +1,10 @@ +; +; Supported features of the Google Virtual Ethernet 'gve' poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Linux = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/gve.rst new file +69 -0
69 lines not on this page · raw message
doc/guides/nics/index.rst +1 -0
@@ -29,6 +29,7 @@ Network Interface Controller Drivers enetfec enic fm10k + gve hinic hns3 i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -59,6 +59,11 @@ New Features * Added flow subscription support. +* **Added GVE net PMD** + + * Added the new ``gve`` net driver for Google Virtual Ethernet devices. + * See the :doc:`../nics/gve` NIC guide for more details on this new driver. + Removed Items -------------
drivers/net/gve/gve_adminq.c +1 -0
@@ -5,6 +5,7 @@* Copyright(C) 2022 Intel Corporation */ +#include "gve_ethdev.h" #include "gve_adminq.h" #include "gve_register.h"
drivers/net/gve/gve_ethdev.c new file +371 -0
371 lines not on this page · raw message
drivers/net/gve/gve_ethdev.h new file +225 -0
225 lines not on this page · raw message
drivers/net/gve/meson.build new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2022 Intel Corporation + +if is_windows + build = false + reason = 'not supported on Windows' + subdir_done() +endif + +sources = files( + 'gve_adminq.c', + 'gve_ethdev.c', +)
drivers/net/gve/version.map new file +3 -0
@@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +};
drivers/net/meson.build +1 -0
@@ -23,6 +23,7 @@ drivers = [ 'enic', 'failsafe', 'fm10k', + 'gve', 'hinic', 'hns3', 'i40e',
Signature · 1 line
2.34.1
PATCH v3 4/9 net/gve: add link update support
In reply to Junfeng GuoSupport dev_ops link_update.
--- doc/guides/nics/features/gve.ini | 2 ++ drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
@@ -4,6 +4,8 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Speed capabilities = Y +Link status = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +30 -0
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)return 0 ; } +static int +gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) +{ +struct gve_priv *priv = dev->data->dev_private; +struct rte_eth_link link; +int err; + +memset (&link,0 ,sizeof (link)); + link.link_duplex =RTE_ETH_LINK_FULL_DUPLEX ; + link.link_autoneg =RTE_ETH_LINK_AUTONEG ; + +if (!dev->data->dev_started) { + link.link_status =RTE_ETH_LINK_DOWN ; + link.link_speed =RTE_ETH_SPEED_NUM_NONE ; + }else { + link.link_status =RTE_ETH_LINK_UP ; +PMD_DRV_LOG (DEBUG ,"Get link status from hw" ); + err =gve_adminq_report_link_speed (priv); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to get link speed." ); + priv->link_speed =RTE_ETH_SPEED_NUM_UNKNOWN ; + } + link.link_speed = priv->link_speed; + } + +return rte_eth_linkstatus_set (dev, &link); +} +static int gve_dev_start (struct rte_eth_dev *dev) { dev->data->dev_started =1 ; +gve_link_update (dev,0 );return 0 ; }
@@ -70,6 +99,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = { .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, + .link_update = gve_link_update, }; static void
Signature · 1 line
2.34.1
PATCH v3 5/9 net/gve: add MTU set support
In reply to Junfeng GuoSupport dev_ops mtu_set.
--- doc/guides/nics/features/gve.ini | 1 + drivers/net/gve/gve_ethdev.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
@@ -6,6 +6,7 @@ [Features] Speed capabilities = Y Link status = Y +MTU update = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +29 -0
@@ -94,12 +94,41 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_mtu_set (struct rte_eth_dev *dev,uint16_t mtu) +{ +struct gve_priv *priv = dev->data->dev_private; +int err; + +if (mtu <RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) { +PMD_DRV_LOG (ERR ,"MIN MTU is %u MAX MTU is %u" ,RTE_ETHER_MIN_MTU , priv->max_mtu); +return -EINVAL ; + } + +/* mtu setting is forbidden if port is start */ +if (dev->data->dev_started) { +PMD_DRV_LOG (ERR ,"Port must be stopped before configuration" ); +return -EBUSY ; + } + + dev->data->dev_conf.rxmode.mtu = mtu +RTE_ETHER_HDR_LEN ; + + err =gve_adminq_set_mtu (priv, mtu); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to set mtu as %u err = %d" , mtu, err); +return err; + } + +return 0 ; +} +static const struct eth_dev_ops gve_eth_dev_ops = { .dev_configure = gve_dev_configure, .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, .link_update = gve_link_update, + .mtu_set = gve_dev_mtu_set, }; static void
Signature · 1 line
2.34.1
PATCH v3 6/9 net/gve: add queue operations
In reply to Junfeng GuoAdd support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues
--- drivers/net/gve/gve_ethdev.c | 206 +++++++++++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 47 ++++++++ drivers/net/gve/gve_rx.c | 212 ++++++++++++++++++++++++++++++++++ drivers/net/gve/gve_tx.c | 214 +++++++++++++++++++++++++++++++++++ drivers/net/gve/meson.build | 2 + 5 files changed, 681 insertions(+) create mode 100644 drivers/net/gve/gve_rx.c create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +206 -0
@@ -28,12 +28,111 @@ gve_write_version(uint8_t *driver_version_register)writeb ('\n' , driver_version_register); } +static int +gve_alloc_queue_page_list (struct gve_priv *priv,uint32_t id,uint32_t pages) +{ +char z_name[RTE_MEMZONE_NAMESIZE ]; +struct gve_queue_page_list *qpl; +const struct rte_memzone *mz; +dma_addr_t page_bus; +uint32_t i; + +if (priv->num_registered_pages + pages > + priv->max_registered_pages) { +PMD_DRV_LOG (ERR ,"Pages %" PRIu64" > max registered pages %" PRIu64, + priv->num_registered_pages + pages, + priv->max_registered_pages); +return -EINVAL ; + } + qpl = &priv->qpl[id]; +snprintf (z_name,sizeof (z_name),"gve_%s_qpl%d" , priv->pci_dev->device.name, id); + mz =rte_memzone_reserve_aligned (z_name, pages *PAGE_SIZE , +rte_socket_id (), +RTE_MEMZONE_IOVA_CONTIG ,PAGE_SIZE ); +if (mz ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc %s." , z_name); +return -ENOMEM ; + } + qpl->page_buses =rte_zmalloc ("qpl page buses" , pages *sizeof (dma_addr_t),0 ); +if (qpl->page_buses ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl %u page buses" , id); +return -ENOMEM ; + } + page_bus = mz->iova; +for (i =0 ; i < pages; i++) { + qpl->page_buses[i] = page_bus; + page_bus +=PAGE_SIZE ; + } + qpl->id = id; + qpl->mz = mz; + qpl->num_entries = pages; + + priv->num_registered_pages += pages; + +return 0 ; +} + +static void +gve_free_qpls (struct gve_priv *priv) +{ +uint16_t nb_txqs = priv->max_nb_txq; +uint16_t nb_rxqs = priv->max_nb_rxq; +uint32_t i; + +for (i =0 ; i < nb_txqs + nb_rxqs; i++) { +if (priv->qpl[i].mz !=NULL ) +rte_memzone_free (priv->qpl[i].mz); +if (priv->qpl[i].page_buses !=NULL ) +rte_free (priv->qpl[i].page_buses); + } + +if (priv->qpl !=NULL ) +rte_free (priv->qpl); +} +static int gve_dev_configure(__rte_unused struct rte_eth_dev *dev) {return 0 ; } +static int +gve_refill_pages (struct gve_rx_queue *rxq) +{ +struct rte_mbuf *nmb; +uint16_t i; +int diag; + + diag =rte_pktmbuf_alloc_bulk (rxq->mpool, &rxq->sw_ring[0 ], rxq->nb_rx_desc); +if (diag <0 ) { +for (i =0 ; i < rxq->nb_rx_desc -1 ; i++) { + nmb =rte_pktmbuf_alloc (rxq->mpool); +if (!nmb) +break ; + rxq->sw_ring[i] = nmb; + } +if (i < rxq->nb_rx_desc -1 ) +return -ENOMEM ; + } + rxq->nb_avail =0 ; + rxq->next_avail = rxq->nb_rx_desc -1 ; + +for (i =0 ; i < rxq->nb_rx_desc; i++) { +if (rxq->is_gqi_qpl) { + rxq->rx_data_ring[i].addr =rte_cpu_to_be_64 (i *PAGE_SIZE ); + }else { +if (i == rxq->nb_rx_desc -1 ) +break ; + nmb = rxq->sw_ring[i]; + rxq->rx_data_ring[i].addr =rte_cpu_to_be_64 (rte_mbuf_data_iova (nmb)); + } + } + +rte_write32 (rte_cpu_to_be_32 (rxq->next_avail), rxq->qrx_tail); + +return 0 ; +} +static int gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) {
@@ -65,16 +164,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)static int gve_dev_start (struct rte_eth_dev *dev) { +uint16_t num_queues = dev->data->nb_tx_queues; +struct gve_priv *priv = dev->data->dev_private; +struct gve_tx_queue *txq; +struct gve_rx_queue *rxq; +uint16_t i; +int err; + + priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues; + err =gve_adminq_create_tx_queues (priv, num_queues); +if (err) { +PMD_DRV_LOG (ERR ,"failed to create %u tx queues." , num_queues); +return err; + } +for (i =0 ; i < num_queues; i++) { + txq = priv->txqs[i]; + txq->qtx_tail = + &priv->db_bar2[rte_be_to_cpu_32 (txq->qres->db_index)]; + txq->qtx_head = + &priv->cnt_array[rte_be_to_cpu_32 (txq->qres->counter_index)]; + +rte_write32 (rte_cpu_to_be_32 (GVE_IRQ_MASK ), txq->ntfy_addr); + } + + num_queues = dev->data->nb_rx_queues; + priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues; + err =gve_adminq_create_rx_queues (priv, num_queues); +if (err) { +PMD_DRV_LOG (ERR ,"failed to create %u rx queues." , num_queues); + goto err_tx; + } +for (i =0 ; i < num_queues; i++) { + rxq = priv->rxqs[i]; + rxq->qrx_tail = + &priv->db_bar2[rte_be_to_cpu_32 (rxq->qres->db_index)]; + +rte_write32 (rte_cpu_to_be_32 (GVE_IRQ_MASK ), rxq->ntfy_addr); + + err =gve_refill_pages (rxq); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to refill for RX" ); + goto err_rx; + } + } + dev->data->dev_started =1 ;gve_link_update (dev,0 );return 0 ; + +err_rx: +gve_stop_rx_queues (dev); +err_tx: +gve_stop_tx_queues (dev); +return err; }static int gve_dev_stop (struct rte_eth_dev *dev) { dev->data->dev_link.link_status =RTE_ETH_LINK_DOWN ; + +gve_stop_tx_queues (dev); +gve_stop_rx_queues (dev); + dev->data->dev_started =0 ;return 0 ;
@@ -83,7 +236,11 @@ gve_dev_stop(struct rte_eth_dev *dev)static int gve_dev_close (struct rte_eth_dev *dev) { +struct gve_priv *priv = dev->data->dev_private; +struct gve_tx_queue *txq; +struct gve_rx_queue *rxq;int err =0 ; +uint16_t i;if (dev->data->dev_started) { err =gve_dev_stop (dev);
@@ -91,6 +248,21 @@ gve_dev_close(struct rte_eth_dev *dev)PMD_DRV_LOG (ERR ,"Failed to stop dev." ); } +for (i =0 ; i < dev->data->nb_tx_queues; i++) { + txq = dev->data->tx_queues[i]; +gve_tx_queue_release (txq); + } + +for (i =0 ; i < dev->data->nb_rx_queues; i++) { + rxq = dev->data->rx_queues[i]; +gve_rx_queue_release (rxq); + } + +gve_free_qpls (priv); +rte_free (priv->adminq); +rte_free (priv->qpl); +rte_free (priv); +return err; }
@@ -127,6 +299,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = { .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, + .rx_queue_setup = gve_rx_queue_setup, + .tx_queue_setup = gve_tx_queue_setup, .link_update = gve_link_update, .mtu_set = gve_dev_mtu_set, };
@@ -264,7 +438,9 @@ gve_setup_device_resources(struct gve_priv *priv)static int gve_init_priv (struct gve_priv *priv,bool skip_describe_device) { +uint16_t pages;int num_ntfy; +uint32_t i;int err;/* Set up the adminq */
@@ -315,10 +491,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)PMD_DRV_LOG (INFO ,"Max TX queues %d, Max RX queues %d" , priv->max_nb_txq, priv->max_nb_rxq); +/* In GQI_QPL queue format: +* Allocate queue page lists according to max queue number +* tx qpl id should start from 0 while rx qpl id should start +* from priv->max_nb_txq +*/ +if (priv->queue_format ==GVE_GQI_QPL_FORMAT ) { + priv->qpl =rte_zmalloc ("gve_qpl" , + (priv->max_nb_txq + priv->max_nb_rxq) * +sizeof (struct gve_queue_page_list ),0 ); +if (priv->qpl ==NULL ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl." ); + err = -ENOMEM ; + goto free_adminq; + } + +for (i =0 ; i < priv->max_nb_txq + priv->max_nb_rxq; i++) { +if (i < priv->max_nb_txq) + pages = priv->tx_pages_per_qpl; +else + pages = priv->rx_data_slot_cnt; + err =gve_alloc_queue_page_list (priv, i, pages); +if (err !=0 ) { +PMD_DRV_LOG (ERR ,"Failed to alloc qpl %u." , i); + goto err_qpl; + } + } + } + setup_device: err =gve_setup_device_resources (priv);if (!err)return 0 ; +err_qpl: +gve_free_qpls (priv); free_adminq:gve_adminq_free (priv);return err;
drivers/net/gve/gve_ethdev.h +47 -0
87 lines not on this page · raw message
drivers/net/gve/gve_rx.c new file +212 -0
212 lines not on this page · raw message
drivers/net/gve/gve_tx.c new file +214 -0
214 lines not on this page · raw message
drivers/net/gve/meson.build +2 -0
@@ -9,5 +9,7 @@ endif sources = files( 'gve_adminq.c', + 'gve_rx.c', + 'gve_tx.c', 'gve_ethdev.c', )
Signature · 1 line
2.34.1
PATCH v3 7/9 net/gve: add Rx/Tx support
In reply to Junfeng GuoAdd Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.
--- doc/guides/nics/features/gve.ini | 2 + drivers/net/gve/gve_ethdev.c | 5 + drivers/net/gve/gve_ethdev.h | 16 ++ drivers/net/gve/gve_rx.c | 143 ++++++++++ drivers/net/gve/gve_tx.c | 455 +++++++++++++++++++++++++++++++ 5 files changed, 621 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
@@ -7,6 +7,8 @@ Speed capabilities = Y Link status = Y MTU update = Y +TSO = Y +L4 checksum offload = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +5 -0
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err; +if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + } + eth_dev->data->mac_addrs =rte_zmalloc ("gve_mac" ,sizeof (struct rte_ether_addr ),0 );if (!eth_dev->data->mac_addrs) {PMD_DRV_LOG (ERR ,"Failed to allocate memory to store mac address" );
drivers/net/gve/gve_ethdev.h +16 -0
@@ -34,6 +34,18 @@ union gve_tx_desc {struct gve_tx_seg_desc seg;/* subsequent descs for a packet */ }; +/* Offload features */ +union gve_tx_offload { +uint64_t data; +struct { +uint64_t l2_len:7 ;/* L2 (MAC) Header Length. */ +uint64_t l3_len:9 ;/* L3 (IP) Header Length. */ +uint64_t l4_len:8 ;/* L4 Header Length. */ +uint64_t tso_segsz:16 ;/* TCP TSO segment size */ +/* uint64_t unused : 24; */ + }; +}; +struct gve_tx_iovec {uint32_t iov_base;/* offset in fifo */ uint32_t iov_len;
@@ -269,4 +281,8 @@ void gve_stop_tx_queues(struct rte_eth_dev *dev);void gve_stop_rx_queues (struct rte_eth_dev *dev); +uint16_t gve_rx_burst (void *rxq,struct rte_mbuf **rx_pkts,uint16_t nb_pkts); + +uint16_t gve_tx_burst (void *txq,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +#endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +143 -0
149 lines not on this page · raw message
drivers/net/gve/gve_tx.c +455 -0
461 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 8/9 net/gve: add support to get dev info and configure dev
In reply to Junfeng GuoAdd dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.
--- doc/guides/nics/features/gve.ini | 1 + drivers/net/gve/gve_ethdev.c | 63 ++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
@@ -8,6 +8,7 @@ Speed capabilities = Y Link status = Y MTU update = Y TSO = Y +RSS hash = Y L4 checksum offload = Y Linux = Y x86-32 = Y
drivers/net/gve/gve_ethdev.c +63 -0
81 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 9/9 net/gve: add stats support
In reply to Junfeng GuoUpdate stats add support of dev_ops stats_get/reset.
--- doc/guides/nics/features/gve.ini | 2 + drivers/net/gve/gve_ethdev.c | 71 ++++++++++++++++++++++++++++++++ drivers/net/gve/gve_ethdev.h | 12 ++++++ drivers/net/gve/gve_rx.c | 15 ++++++- drivers/net/gve/gve_tx.c | 13 ++++++ 5 files changed, 111 insertions(+), 2 deletions(-)
doc/guides/nics/features/gve.ini +2 -0
@@ -10,6 +10,8 @@ MTU update = Y TSO = Y RSS hash = Y L4 checksum offload = Y +Basic stats = Y +Stats per queue = Y Linux = Y x86-32 = Y x86-64 = Y
drivers/net/gve/gve_ethdev.c +71 -0
83 lines not on this page · raw message
drivers/net/gve/gve_ethdev.h +12 -0
@@ -76,6 +76,11 @@ struct gve_tx_queue {struct gve_queue_page_list *qpl;struct gve_tx_iovec *iov_ring; +/* Stats */ +uint64_t errors; +uint64_t packets; +uint64_t bytes; +uint16_t port_id;uint16_t queue_id;
@@ -114,6 +119,12 @@ struct gve_rx_queue {/* only valid for GQI_QPL queue format */ struct gve_queue_page_list *qpl; +/* stats */ +uint64_t no_mbufs; +uint64_t errors; +uint64_t packets; +uint64_t bytes; +struct gve_priv *hw;const struct rte_memzone *qres_mz;struct gve_queue_resources *qres;
@@ -125,6 +136,7 @@ struct gve_rx_queue {/* Only valid for DQO_RDA queue format */ struct gve_rx_queue *bufq; +uint8_t is_gqi_qpl; };
drivers/net/gve/gve_rx.c +13 -2
@@ -26,8 +26,10 @@ gve_rx_refill(struct gve_rx_queue *rxq)break ; rxq->sw_ring[idx + i] = nmb; } -if (i != nb_alloc) +if (i != nb_alloc) { + rxq->no_mbufs += nb_alloc - i; nb_alloc = i; + } } rxq->nb_avail -= nb_alloc; next_avail += nb_alloc;
@@ -88,6 +90,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)uint16_t rx_id = rxq->rx_tail;struct rte_mbuf *rxe;uint16_t nb_rx, len; +uint64_t bytes =0 ;uint64_t addr; rxr = rxq->rx_desc_ring;
@@ -97,8 +100,10 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno)break ; -if (rxd->flags_seq &GVE_RXF_ERR ) +if (rxd->flags_seq &GVE_RXF_ERR ) { + rxq->errors++;continue ; + } len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; rxe = rxq->sw_ring[rx_id];
@@ -137,6 +142,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rx_id =0 ; rx_pkts[nb_rx] = rxe; + bytes += len; } rxq->nb_avail += nb_rx;
@@ -145,6 +151,11 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)if (rxq->nb_avail > rxq->free_thresh)gve_rx_refill (rxq); +if (nb_rx) { + rxq->packets += nb_rx; + rxq->bytes += bytes; + } +return nb_rx; }
drivers/net/gve/gve_tx.c +13 -0
@@ -260,6 +260,7 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)struct rte_mbuf *tx_pkt, *first;uint16_t sw_id = txq->sw_tail;uint16_t nb_used, i; +uint64_t bytes =0 ;uint16_t nb_tx =0 ;uint32_t hlen;
@@ -355,6 +356,8 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) txq->nb_free -= nb_used; txq->sw_nb_free -= first->nb_segs; tx_tail += nb_used; + + bytes += first->pkt_len; } end_of_tx:
@@ -362,6 +365,10 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)rte_write32 (rte_cpu_to_be_32 (tx_tail), txq->qtx_tail); txq->tx_tail = tx_tail; txq->sw_tail = sw_id; + + txq->errors += nb_pkts - nb_tx; + txq->packets += nb_tx; + txq->bytes += bytes; }return nb_tx;
@@ -380,6 +387,7 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)struct rte_mbuf *tx_pkt, *first;uint16_t nb_used, hlen, i;uint64_t ol_flags, addr; +uint64_t bytes =0 ;uint16_t nb_tx =0 ; txr = txq->tx_desc_ring;
@@ -438,12 +446,17 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) txq->nb_free -= nb_used; tx_tail += nb_used; + + bytes += first->pkt_len; } end_of_tx:if (nb_tx) {rte_write32 (rte_cpu_to_be_32 (tx_tail), txq->qtx_tail); txq->tx_tail = tx_tail; + + txq->packets += nb_tx; + txq->bytes += bytes; }return nb_tx;
Signature · 1 line
2.34.1
Quoted 17 lines · Junfeng Guo
Introduce a new PMD for Google Virtual Ethernet (GVE).
This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
-gve_adminq.c
-gve_adminq.h
-gve_desc.h
-gve_desc_dqo.h
-gve_register.hIt's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0v2:
fix some CI check error.
Quoted 12 lines
Junfeng Guo (10):
net/gve: introduce GVE PMD base code
net/gve: add logs and OS specific implementation net/gve: support device initialization net/gve: add link update support
net/gve: add MTU set support
net/gve: add queue operations
net/gve: add Rx/Tx support
net/gve: add support to get dev info and configure dev net/gve: add stats support
doc: update documentation
Please check build error reported by CI:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/
I am also getting various build errors, even not able to reach patch by patch build stage where I expect some issues, can you please verify patch by patch build in next version?
Quoted 45 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:19
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH v2 00/10] introduce GVE PMDQuoted 11 lines · Junfeng Guo
Introduce a new PMD for Google Virtual Ethernet (GVE).
This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
-gve_adminq.c
-gve_adminq.h
-gve_desc.h
-gve_desc_dqo.h
-gve_register.hQuoted 2 lines
It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0
Quoted 4 lines
v2:
fix some CI check error.Quoted 12 lines
Junfeng Guo (10):
net/gve: introduce GVE PMD base code
net/gve: add logs and OS specific implementation net/gve: support device initialization net/gve: add link update support
net/gve: add MTU set support
net/gve: add queue operations
net/gve: add Rx/Tx support
net/gve: add support to get dev info and configure dev net/gve: add stats support
doc: update documentation
Quoted 7 lines
Please check build error reported by CI:
https://patches.dpdk.org/project/dpdk/patch/20220829084127.934183-
[email protected]/I am also getting various build errors, even not able to reach patch by patch build stage where I expect some issues, can you please verify patch by patch build in next version?
Sure, thanks for reminding!
The compile/build issues are being handled in process now. Thanks!
PATCH 2/10 net/gve: add logs and OS specific implementation
In reply to Xiaoyun LiAdd GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.
--- drivers/net/gve/gve_adminq.h | 2 + drivers/net/gve/gve_desc.h | 2 + drivers/net/gve/gve_desc_dqo.h | 2 + drivers/net/gve/gve_logs.h | 22 +++++ drivers/net/gve/gve_osdep.h | 149 +++++++++++++++++++++++++++++++++ drivers/net/gve/gve_register.h | 2 + 6 files changed, 179 insertions(+) create mode 100644 drivers/net/gve/gve_logs.h create mode 100644 drivers/net/gve/gve_osdep.h
drivers/net/gve/gve_adminq.h +2 -0
@@ -8,6 +8,8 @@#ifndef _GVE_ADMINQ_H#define _GVE_ADMINQ_H +#include "gve_osdep.h" +/* Admin queue opcodes */ enum gve_adminq_opcodes { GVE_ADMINQ_DESCRIBE_DEVICE = 0x1,
drivers/net/gve/gve_desc.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_H_#define _GVE_DESC_H_ +#include "gve_osdep.h" +/* A note on seg_addrs * * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/gve_desc_dqo.h +2 -0
@@ -9,6 +9,8 @@#ifndef _GVE_DESC_DQO_H_#define _GVE_DESC_DQO_H_ +#include "gve_osdep.h" +#define GVE_TX_MAX_HDR_SIZE_DQO 255#define GVE_TX_MIN_TSO_MSS_DQO 88
drivers/net/gve/gve_logs.h new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2022 Intel Corporation +*/ + +#ifndef _GVE_LOGS_H_ +#define _GVE_LOGS_H_ + +extern int gve_logtype_init; +extern int gve_logtype_driver; + +#define PMD_INIT_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_init, "%s(): " fmt "\n", \ + __func__, ##args) + +#define PMD_DRV_LOG_RAW(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt, \ + __func__, ## args) + +#define PMD_DRV_LOG(level, fmt, args...) \ + PMD_DRV_LOG_RAW(level, fmt "\n", ## args) + +#endif
drivers/net/gve/gve_osdep.h new file +149 -0
149 lines not on this page · raw message
drivers/net/gve/gve_register.h +2 -0
@@ -7,6 +7,8 @@ #ifndef _GVE_REGISTER_H_ #define _GVE_REGISTER_H_ +#include "gve_osdep.h" + /* Fixed Configuration Registers */ struct gve_registers { __be32 device_status;
Signature · 1 line
2.25.1
PATCH 3/10 net/gve: support device initialization
In reply to Xiaoyun LiSupport device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close
--- drivers/net/gve/gve.h | 249 +++++++++++++++++++++++ drivers/net/gve/gve_adminq.c | 1 + drivers/net/gve/gve_ethdev.c | 375 +++++++++++++++++++++++++++++++++++ drivers/net/gve/meson.build | 13 ++ drivers/net/gve/version.map | 3 + drivers/net/meson.build | 1 + 6 files changed, 642 insertions(+) create mode 100644 drivers/net/gve/gve.h create mode 100644 drivers/net/gve/gve_ethdev.c create mode 100644 drivers/net/gve/meson.build create mode 100644 drivers/net/gve/version.map
drivers/net/gve/gve.h new file +249 -0
@@ -0,0 +1,249 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(C) 2022 Intel Corporation +*/ + +#ifndef _GVE_H_ +#define _GVE_H_ + +#include <ethdev_driver.h> +#include <ethdev_pci.h> +#include <rte_ether.h> + +#include "gve_desc.h" + +#ifndef GOOGLE_VENDOR_ID +#define GOOGLE_VENDOR_ID 0x1ae0 +#endif + +#define GVE_DEV_ID 0x0042 + +#define GVE_REG_BAR 0 +#define GVE_DB_BAR 2 + +/* 1 for management, 1 for rx, 1 for tx */ +#define GVE_MIN_MSIX 3 + +/* PTYPEs are always 10 bits. */ +#define GVE_NUM_PTYPES 1024 + +/* A list of pages registered with the device during setup and used by a queue +* as buffers +*/ +struct gve_queue_page_list { +uint32_t id;/* unique id */ +uint32_t num_entries; +dma_addr_t *page_buses;/* the dma addrs of the pages */ +const struct rte_memzone *mz; +}; + +/* A TX desc ring entry */ +union gve_tx_desc { +struct gve_tx_pkt_desc pkt;/* first desc for a packet */ +struct gve_tx_seg_desc seg;/* subsequent descs for a packet */ +}; + +struct gve_tx_queue { +volatile union gve_tx_desc *tx_desc_ring; +const struct rte_memzone *mz; +uint64_t tx_ring_phys_addr; + +uint16_t nb_tx_desc; + +/* Only valid for DQO_QPL queue format */ +struct gve_queue_page_list *qpl; + +uint16_t port_id; +uint16_t queue_id; + +uint16_t ntfy_id; +volatile rte_be32_t *ntfy_addr; + +struct gve_priv *hw; +const struct rte_memzone *qres_mz; +struct gve_queue_resources *qres; + +/* Only valid for DQO_RDA queue format */ +struct gve_tx_queue *complq; +}; + +struct gve_rx_queue { +volatile struct gve_rx_desc *rx_desc_ring; +volatile union gve_rx_data_slot *rx_data_ring; +const struct rte_memzone *mz; +const struct rte_memzone *data_mz; +uint64_t rx_ring_phys_addr; + +uint16_t nb_rx_desc; + +volatile rte_be32_t *ntfy_addr; + +/* only valid for GQI_QPL queue format */ +struct gve_queue_page_list *qpl; + +struct gve_priv *hw; +const struct rte_memzone *qres_mz; +struct gve_queue_resources *qres; + +uint16_t port_id; +uint16_t queue_id; +uint16_t ntfy_id; +uint16_t rx_buf_len; + +/* Only valid for DQO_RDA queue format */ +struct gve_rx_queue *bufq; +}; + +struct gve_irq_db { +rte_be32_t id; +}____cacheline_aligned ; + +struct gve_ptype { +uint8_t l3_type;/* `gve_l3_type` in gve_adminq.h */ +uint8_t l4_type;/* `gve_l4_type` in gve_adminq.h */ +}; + +struct gve_ptype_lut { +struct gve_ptype ptypes[GVE_NUM_PTYPES ]; +}; + +enum gve_queue_format { +GVE_QUEUE_FORMAT_UNSPECIFIED =0x0 ,/* default unspecified */ +GVE_GQI_RDA_FORMAT =0x1 ,/* GQI Raw Addressing */ +GVE_GQI_QPL_FORMAT =0x2 ,/* GQI Queue Page List */ +GVE_DQO_RDA_FORMAT =0x3 ,/* DQO Raw Addressing */ +}; + +struct gve_priv { +struct gve_irq_db *irq_dbs;/* array of num_ntfy_blks */ +const struct rte_memzone *irq_dbs_mz; +uint32_t mgmt_msix_idx; +rte_be32_t *cnt_array;/* array of num_event_counters */ +const struct rte_memzone *cnt_array_mz; + +uint16_t num_event_counters; +uint16_t tx_desc_cnt;/* txq size */ +uint16_t rx_desc_cnt;/* rxq size */ +uint16_t tx_pages_per_qpl;/* tx buffer length */ +uint16_t rx_data_slot_cnt;/* rx buffer length */ + +/* Only valid for DQO_RDA queue format */ +uint16_t tx_compq_size;/* tx completion queue size */ +uint16_t rx_bufq_size;/* rx buff queue size */ + +uint64_t max_registered_pages; +uint64_t num_registered_pages;/* num pages registered with NIC */ +uint16_t default_num_queues;/* default num queues to set up */ +enum gve_queue_format queue_format;/* see enum gve_queue_format */ +uint8_t enable_lsc; + +uint16_t max_nb_txq; +uint16_t max_nb_rxq; +uint32_t num_ntfy_blks;/* spilt between TX and RX so must be even */ + +struct gve_registers __iomem *reg_bar0;/* see gve_register.h */ +rte_be32_t __iomem *db_bar2;/* "array" of doorbells */ +struct rte_pci_device *pci_dev; + +/* Admin queue - see gve_adminq.h*/ +union gve_adminq_command *adminq; +struct gve_dma_mem adminq_dma_mem; +uint32_t adminq_mask;/* masks prod_cnt to adminq size */ +uint32_t adminq_prod_cnt;/* free-running count of AQ cmds executed */ +uint32_t adminq_cmd_fail;/* free-running count of AQ cmds failed */ +uint32_t adminq_timeouts;/* free-running count of AQ cmds timeouts */ +/* free-running count of per AQ cmd executed */ +uint32_t adminq_describe_device_cnt; +uint32_t adminq_cfg_device_resources_cnt; +uint32_t adminq_register_page_list_cnt; +uint32_t adminq_unregister_page_list_cnt; +uint32_t adminq_create_tx_queue_cnt; +uint32_t adminq_create_rx_queue_cnt; +uint32_t adminq_destroy_tx_queue_cnt; +uint32_t adminq_destroy_rx_queue_cnt; +uint32_t adminq_dcfg_device_resources_cnt; +uint32_t adminq_set_driver_parameter_cnt; +uint32_t adminq_report_stats_cnt; +uint32_t adminq_report_link_speed_cnt; +uint32_t adminq_get_ptype_map_cnt; + +volatile uint32_t state_flags; + +/* Gvnic device link speed from hypervisor. */ +uint64_t link_speed; + +uint16_t max_mtu; +struct rte_ether_addr dev_addr;/* mac address */ + +struct gve_queue_page_list *qpl; + +struct gve_tx_queue **txqs; +struct gve_rx_queue **rxqs; +}; + +enum gve_state_flags_bit { +GVE_PRIV_FLAGS_ADMIN_QUEUE_OK =1 , +GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK =2 , +GVE_PRIV_FLAGS_DEVICE_RINGS_OK =3 , +GVE_PRIV_FLAGS_NAPI_ENABLED =4 , +}; + +static inline bool gve_is_gqi (struct gve_priv *priv) +{ +return priv->queue_format ==GVE_GQI_RDA_FORMAT || + priv->queue_format ==GVE_GQI_QPL_FORMAT ; +} + +static inline bool gve_get_admin_queue_ok (struct gve_priv *priv) +{ +return !!rte_bit_relaxed_get32 (GVE_PRIV_FLAGS_ADMIN_QUEUE_OK , + &priv->state_flags); +} + +static inline void gve_set_admin_queue_ok (struct gve_priv *priv) +{ +rte_bit_relaxed_set32 (GVE_PRIV_FLAGS_ADMIN_QUEUE_OK , + &priv->state_flags); +} + +static inline void gve_clear_admin_queue_ok (struct gve_priv *priv) +{ +rte_bit_relaxed_clear32 (GVE_PRIV_FLAGS_ADMIN_QUEUE_OK , + &priv->state_flags); +} + +static inline bool gve_get_device_resources_ok (struct gve_priv *priv) +{ +return !!rte_bit_relaxed_get32 (GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK , + &priv->state_flags); +} + +static inline void gve_set_device_resources_ok (struct gve_priv *priv) +{ +rte_bit_relaxed_set32 (GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK , + &priv->state_flags); +} + +static inline void gve_clear_device_resources_ok (struct gve_priv *priv) +{ +rte_bit_relaxed_clear32 (GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK , + &priv->state_flags); +} + +static inline bool gve_get_device_rings_ok (struct gve_priv *priv) +{ +return !!rte_bit_relaxed_get32 (GVE_PRIV_FLAGS_DEVICE_RINGS_OK , + &priv->state_flags); +} + +static inline void gve_set_device_rings_ok (struct gve_priv *priv) +{ +rte_bit_relaxed_set32 (GVE_PRIV_FLAGS_DEVICE_RINGS_OK , + &priv->state_flags); +} + +static inline void gve_clear_device_rings_ok (struct gve_priv *priv) +{ +rte_bit_relaxed_clear32 (GVE_PRIV_FLAGS_DEVICE_RINGS_OK , + &priv->state_flags); +} +#endif /* _GVE_H_ */
drivers/net/gve/gve_adminq.c +1 -0
@@ -5,6 +5,7 @@* Copyright(C) 2022 Intel Corporation */ +#include "gve.h" #include "gve_adminq.h" #include "gve_register.h"
drivers/net/gve/gve_ethdev.c new file +375 -0
375 lines not on this page · raw message
drivers/net/gve/meson.build new file +13 -0
@@ -0,0 +1,13 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(C) 2022 Intel Corporation + +if is_windows + build = false + reason = 'not supported on Windows' + subdir_done() +endif + +sources = files( + 'gve_adminq.c', + 'gve_ethdev.c', +)
drivers/net/gve/version.map new file +3 -0
@@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +};
drivers/net/meson.build +1 -0
@@ -23,6 +23,7 @@ drivers = [ 'enic', 'failsafe', 'fm10k', + 'gve', 'hinic', 'hns3', 'i40e',
Signature · 1 line
2.25.1
PATCH 4/10 net/gve: add link update support
In reply to Xiaoyun LiSupport dev_ops link_update.
--- drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
drivers/net/gve/gve_ethdev.c +30 -0
@@ -37,10 +37,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)return 0 ; } +static int +gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) +{ +struct gve_priv *priv = dev->data->dev_private; +struct rte_eth_link link; +int err; + +memset (&link,0 ,sizeof (link)); + link.link_duplex =RTE_ETH_LINK_FULL_DUPLEX ; + link.link_autoneg =RTE_ETH_LINK_AUTONEG ; + +if (!dev->data->dev_started) { + link.link_status =RTE_ETH_LINK_DOWN ; + link.link_speed =RTE_ETH_SPEED_NUM_NONE ; + }else { + link.link_status =RTE_ETH_LINK_UP ; +PMD_INIT_LOG (DEBUG ,"Get link status from hw" ); + err =gve_adminq_report_link_speed (priv); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to get link speed." ); + priv->link_speed =RTE_ETH_SPEED_NUM_UNKNOWN ; + } + link.link_speed = priv->link_speed; + } + +return rte_eth_linkstatus_set (dev, &link); +} +static int gve_dev_start (struct rte_eth_dev *dev) { dev->data->dev_started =1 ; +gve_link_update (dev,0 );return 0 ; }
@@ -73,6 +102,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = { .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, + .link_update = gve_link_update, }; static void
Signature · 1 line
2.25.1
PATCH 5/10 net/gve: add MTU set support
In reply to Xiaoyun LiSupport dev_ops mtu_set.
--- drivers/net/gve/gve_ethdev.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
drivers/net/gve/gve_ethdev.c +29 -0
@@ -97,12 +97,41 @@ gve_dev_close(struct rte_eth_dev *dev)return err; } +static int +gve_dev_mtu_set (struct rte_eth_dev *dev,uint16_t mtu) +{ +struct gve_priv *priv = dev->data->dev_private; +int err; + +if (mtu <RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) { +PMD_DRV_LOG (ERR ,"MIN MTU is %u MAX MTU is %u" ,RTE_ETHER_MIN_MTU , priv->max_mtu); +return -EINVAL ; + } + +/* mtu setting is forbidden if port is start */ +if (dev->data->dev_started) { +PMD_DRV_LOG (ERR ,"Port must be stopped before configuration" ); +return -EBUSY ; + } + + dev->data->dev_conf.rxmode.mtu = mtu +RTE_ETHER_HDR_LEN ; + + err =gve_adminq_set_mtu (priv, mtu); +if (err) { +PMD_DRV_LOG (ERR ,"Failed to set mtu as %u err = %d" , mtu, err); +return err; + } + +return 0 ; +} +static const struct eth_dev_ops gve_eth_dev_ops = { .dev_configure = gve_dev_configure, .dev_start = gve_dev_start, .dev_stop = gve_dev_stop, .dev_close = gve_dev_close, .link_update = gve_link_update, + .mtu_set = gve_dev_mtu_set, }; static void
Signature · 1 line
2.25.1
PATCH 6/10 net/gve: add queue operations
In reply to Xiaoyun LiAdd support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues
--- drivers/net/gve/gve.h | 52 +++++++++ drivers/net/gve/gve_ethdev.c | 203 +++++++++++++++++++++++++++++++++ drivers/net/gve/gve_rx.c | 212 ++++++++++++++++++++++++++++++++++ drivers/net/gve/gve_tx.c | 214 +++++++++++++++++++++++++++++++++++ drivers/net/gve/meson.build | 2 + 5 files changed, 683 insertions(+) create mode 100644 drivers/net/gve/gve_rx.c create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve.h +52 -0
@@ -23,6 +23,9 @@/* 1 for management, 1 for rx, 1 for tx */ #define GVE_MIN_MSIX 3 +#define GVE_DEFAULT_RX_FREE_THRESH 512 +#define GVE_DEFAULT_TX_FREE_THRESH 256 +/* PTYPEs are always 10 bits. */ #define GVE_NUM_PTYPES 1024
@@ -42,15 +45,35 @@ union gve_tx_desc {struct gve_tx_seg_desc seg;/* subsequent descs for a packet */ }; +struct gve_tx_iovec { +uint32_t iov_base;/* offset in fifo */ +uint32_t iov_len; +}; +struct gve_tx_queue {volatile union gve_tx_desc *tx_desc_ring;const struct rte_memzone *mz;uint64_t tx_ring_phys_addr; +struct rte_mbuf **sw_ring; +volatile rte_be32_t *qtx_tail; +volatile rte_be32_t *qtx_head; +uint32_t tx_tail;uint16_t nb_tx_desc; +uint16_t nb_free; +uint32_t next_to_clean; +uint16_t free_thresh;/* Only valid for DQO_QPL queue format */ +uint16_t sw_tail; +uint16_t sw_ntc; +uint16_t sw_nb_free; +uint32_t fifo_size; +uint32_t fifo_head; +uint32_t fifo_avail; +uint64_t fifo_base;struct gve_queue_page_list *qpl; +struct gve_tx_iovec *iov_ring;uint16_t port_id;uint16_t queue_id;
@@ -64,6 +87,8 @@ struct gve_tx_queue {/* Only valid for DQO_RDA queue format */ struct gve_tx_queue *complq; + +uint8_t is_gqi_qpl; };struct gve_rx_queue {
@@ -72,9 +97,17 @@ struct gve_rx_queue {const struct rte_memzone *mz;const struct rte_memzone *data_mz;uint64_t rx_ring_phys_addr; +struct rte_mbuf **sw_ring; +struct rte_mempool *mpool; +uint16_t rx_tail;uint16_t nb_rx_desc; +uint16_t expected_seqno;/* the next expected seqno */ +uint16_t free_thresh; +uint32_t next_avail; +uint32_t nb_avail; +volatile rte_be32_t *qrx_tail;volatile rte_be32_t *ntfy_addr;/* only valid for GQI_QPL queue format */
@@ -91,6 +124,8 @@ struct gve_rx_queue {/* Only valid for DQO_RDA queue format */ struct gve_rx_queue *bufq; + +uint8_t is_gqi_qpl; };struct gve_irq_db {
@@ -246,4 +281,21 @@ static inline void gve_clear_device_rings_ok(struct gve_priv *priv)rte_bit_relaxed_clear32 (GVE_PRIV_FLAGS_DEVICE_RINGS_OK , &priv->state_flags); } + +int +gve_rx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_desc, +unsigned int socket_id,const struct rte_eth_rxconf *conf, +struct rte_mempool *pool); +int +gve_tx_queue_setup (struct rte_eth_dev *dev,uint16_t queue_id,uint16_t nb_desc, +unsigned int socket_id,const struct rte_eth_txconf *conf); + +void gve_tx_queue_release (void *txq); + +void gve_rx_queue_release (void *rxq); + +void gve_stop_tx_queues (struct rte_eth_dev *dev); + +void gve_stop_rx_queues (struct rte_eth_dev *dev); +#endif /* _GVE_H_ */
drivers/net/gve/gve_ethdev.c +203 -0
267 lines not on this page · raw message
drivers/net/gve/gve_rx.c new file +212 -0
212 lines not on this page · raw message
drivers/net/gve/gve_tx.c new file +214 -0
214 lines not on this page · raw message
drivers/net/gve/meson.build +2 -0
@@ -9,5 +9,7 @@ endif sources = files( 'gve_adminq.c', + 'gve_rx.c', + 'gve_tx.c', 'gve_ethdev.c', )
Signature · 1 line
2.25.1
PATCH 7/10 net/gve: add Rx/Tx support
In reply to Xiaoyun LiAdd Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.
--- drivers/net/gve/gve.h | 17 ++ drivers/net/gve/gve_ethdev.c | 5 + drivers/net/gve/gve_rx.c | 143 +++++++++++ drivers/net/gve/gve_tx.c | 452 +++++++++++++++++++++++++++++++++++ 4 files changed, 617 insertions(+)
drivers/net/gve/gve.h +17 -0
@@ -25,6 +25,7 @@#define GVE_DEFAULT_RX_FREE_THRESH 512#define GVE_DEFAULT_TX_FREE_THRESH 256 +#define GVE_TX_MAX_FREE_SZ 512/* PTYPEs are always 10 bits. */ #define GVE_NUM_PTYPES 1024
@@ -45,6 +46,18 @@ union gve_tx_desc {struct gve_tx_seg_desc seg;/* subsequent descs for a packet */ }; +/* Offload features */ +union gve_tx_offload { +uint64_t data; +struct { +uint64_t l2_len:7 ;/* L2 (MAC) Header Length. */ +uint64_t l3_len:9 ;/* L3 (IP) Header Length. */ +uint64_t l4_len:8 ;/* L4 Header Length. */ +uint64_t tso_segsz:16 ;/* TCP TSO segment size */ +/* uint64_t unused : 24; */ + }; +}; +struct gve_tx_iovec {uint32_t iov_base;/* offset in fifo */ uint32_t iov_len;
@@ -298,4 +311,8 @@ void gve_stop_tx_queues(struct rte_eth_dev *dev);void gve_stop_rx_queues (struct rte_eth_dev *dev); +uint16_t gve_rx_burst (void *rxq,struct rte_mbuf **rx_pkts,uint16_t nb_pkts); + +uint16_t gve_tx_burst (void *txq,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +#endif /* _GVE_H_ */
drivers/net/gve/gve_ethdev.c +5 -0
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)if (err)return err; +if (gve_is_gqi (priv)) { + eth_dev->rx_pkt_burst = gve_rx_burst; + eth_dev->tx_pkt_burst = gve_tx_burst; + } + eth_dev->data->mac_addrs =rte_zmalloc ("gve_mac" ,sizeof (struct rte_ether_addr ),0 );if (!eth_dev->data->mac_addrs) {PMD_INIT_LOG (ERR ,"Failed to allocate memory to store mac address" );
drivers/net/gve/gve_rx.c +143 -0
149 lines not on this page · raw message
drivers/net/gve/gve_tx.c +452 -0
458 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 8/10 net/gve: add support to get dev info and configure dev
In reply to Xiaoyun LiAdd dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.
--- drivers/net/gve/gve.h | 3 ++ drivers/net/gve/gve_ethdev.c | 61 ++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+)
drivers/net/gve/gve.h +3 -0
@@ -27,6 +27,9 @@#define GVE_DEFAULT_TX_FREE_THRESH 256#define GVE_TX_MAX_FREE_SZ 512 +#define GVE_MIN_BUF_SIZE 1024 +#define GVE_MAX_RX_PKTLEN 65535 +/* PTYPEs are always 10 bits. */ #define GVE_NUM_PTYPES 1024
drivers/net/gve/gve_ethdev.c +61 -0
79 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 9/10 net/gve: add stats support
In reply to Xiaoyun LiUpdate stats add support of dev_ops stats_get/reset.
--- drivers/net/gve/gve.h | 10 ++++++ drivers/net/gve/gve_ethdev.c | 69 ++++++++++++++++++++++++++++++++++++ drivers/net/gve/gve_rx.c | 15 ++++++-- drivers/net/gve/gve_tx.c | 12 +++++++ 4 files changed, 104 insertions(+), 2 deletions(-)
drivers/net/gve/gve.h +10 -0
@@ -91,6 +91,10 @@ struct gve_tx_queue {struct gve_queue_page_list *qpl;struct gve_tx_iovec *iov_ring; +/* Stats */ +uint64_t packets; +uint64_t bytes; +uint16_t port_id;uint16_t queue_id;
@@ -129,6 +133,12 @@ struct gve_rx_queue {/* only valid for GQI_QPL queue format */ struct gve_queue_page_list *qpl; +/* stats */ +uint64_t no_mbufs; +uint64_t errors; +uint64_t packets; +uint64_t bytes; +struct gve_priv *hw;const struct rte_memzone *qres_mz;struct gve_queue_resources *qres;
drivers/net/gve/gve_ethdev.c +69 -0
81 lines not on this page · raw message
drivers/net/gve/gve_rx.c +13 -2
@@ -26,8 +26,10 @@ gve_rx_refill(struct gve_rx_queue *rxq)break ; rxq->sw_ring[idx + i] = nmb; } -if (i != nb_alloc) +if (i != nb_alloc) { + rxq->no_mbufs += nb_alloc - i; nb_alloc = i; + } } rxq->nb_avail -= nb_alloc; next_avail += nb_alloc;
@@ -88,6 +90,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)uint16_t rx_id = rxq->rx_tail;struct rte_mbuf *rxe;uint16_t nb_rx, len; +uint64_t bytes =0 ;uint64_t addr; rxr = rxq->rx_desc_ring;
@@ -97,8 +100,10 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)if (GVE_SEQNO (rxd->flags_seq) != rxq->expected_seqno)break ; -if (rxd->flags_seq &GVE_RXF_ERR ) +if (rxd->flags_seq &GVE_RXF_ERR ) { + rxq->errors++;continue ; + } len =rte_be_to_cpu_16 (rxd->len) -GVE_RX_PAD ; rxe = rxq->sw_ring[rx_id];
@@ -137,6 +142,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rx_id =0 ; rx_pkts[nb_rx] = rxe; + bytes += len; } rxq->nb_avail += nb_rx;
@@ -145,6 +151,11 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)if (rxq->nb_avail > rxq->free_thresh)gve_rx_refill (rxq); +if (nb_rx) { + rxq->packets += nb_rx; + rxq->bytes += bytes; + } +return nb_rx; }
drivers/net/gve/gve_tx.c +12 -0
@@ -260,6 +260,7 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)struct rte_mbuf *tx_pkt, *first;uint16_t sw_id = txq->sw_tail;uint16_t nb_used, i; +uint64_t bytes =0 ;uint16_t nb_tx =0 ;uint32_t hlen;
@@ -352,6 +353,8 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) txq->nb_free -= nb_used; txq->sw_nb_free -= first->nb_segs; tx_tail += nb_used; + + bytes += first->pkt_len; } end_of_tx:
@@ -359,6 +362,9 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)rte_write32 (rte_cpu_to_be_32 (tx_tail), txq->qtx_tail); txq->tx_tail = tx_tail; txq->sw_tail = sw_id; + + txq->packets += nb_tx; + txq->bytes += bytes; }return nb_tx;
@@ -377,6 +383,7 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)struct rte_mbuf *tx_pkt, *first;uint16_t nb_used, hlen, i;uint64_t ol_flags, addr; +uint64_t bytes =0 ;uint16_t nb_tx =0 ; txr = txq->tx_desc_ring;
@@ -435,12 +442,17 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) txq->nb_free -= nb_used; tx_tail += nb_used; + + bytes += first->pkt_len; } end_of_tx:if (nb_tx) {rte_write32 (rte_cpu_to_be_32 (tx_tail), txq->qtx_tail); txq->tx_tail = tx_tail; + + txq->packets += nb_tx; + txq->bytes += bytes; }return nb_tx;
Signature · 1 line
2.25.1
PATCH 10/10 doc: update documentation
In reply to Xiaoyun LiUpdate documentation of GVE PMD and release note.
Add Junfeng Guo as GVE PMD maintainer since he'll work on GVE PMD in the future and maintain it and I won't be available for maintaining.
--- MAINTAINERS | 6 +++ doc/guides/nics/features/gve.ini | 18 +++++++ doc/guides/nics/gve.rst | 65 ++++++++++++++++++++++++++ doc/guides/rel_notes/release_22_11.rst | 4 ++ 4 files changed, 93 insertions(+) create mode 100644 doc/guides/nics/features/gve.ini create mode 100644 doc/guides/nics/gve.rst
MAINTAINERS +6 -0
@@ -697,6 +697,12 @@ F: drivers/net/enic/ F: doc/guides/nics/enic.rst F: doc/guides/nics/features/enic.ini +Google Virtual Ethernet +M: Junfeng Guo <[email protected]> +F: drivers/net/gve/ +F: doc/guides/nics/gve.rst +F: doc/guides/nics/features/gve.ini + Hisilicon hns3 M: Dongdong Liu <[email protected]> M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +18 -0
@@ -0,0 +1,18 @@ +; +; Supported features of the Google Virtual Ethernet 'gve' poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Speed capabilities = Y +Link status = Y +MTU update = Y +TSO = Y +RSS hash = Y +L4 checksum offload = Y +Basic stats = Y +Stats per queue = Y +Linux = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y
doc/guides/nics/gve.rst new file +65 -0
65 lines not on this page · raw message
doc/guides/rel_notes/release_22_11.rst +4 -0
@@ -55,6 +55,10 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added GVE net PMD** + + Added the new ``gve`` net driver for Google Virtual Ethernet devices. + See the :doc:`../nics/gve` NIC guide for more details on this new driver. Removed Items -------------
Signature · 1 line
2.25.1