Skip to contentSkip to conversation

introduce GVE PMD

PATCH In review Xiaoyun Li 9 participants 192 messages Jul 29, 2022 – Oct 26, 2022 Page 2 of 2

Subsystems (2)
lore
Share & tools
lore
#119 lore raw

RE: [PATCH v7 8/8] net/gve: add support for Rx/Tx

In reply to Guo, Junfeng
Quoted 102 lines

-----Original Message-----
From: Guo, Junfeng
Sent: Monday, October 24, 2022 21:25
To: Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: RE: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 13 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 13 lines
Quoted 12 lines · Junfeng
Quoted 12 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:52
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo

<[email protected]>;

Quoted 58 lines
Quoted 57 lines
Quoted 52 lines

Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 26 lines
drivers/net/gve/gve_tx.c +26 -0
+
+static inline void
+gve_tx_clean_swr_qpl(struct gve_tx_queue *txq)
+{
+       uint32_t start = txq->sw_ntc;
+       uint32_t ntc, nb_clean;
+
+       ntc = txq->sw_tail;
+
+       if (ntc == start)
+               return;
+
+       /* if wrap around, free twice. */
+       if (ntc < start) {
+               nb_clean = txq->nb_tx_desc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               if (start == txq->nb_tx_desc)
+                       start = 0;
+               txq->sw_ntc = start;
+       }
+

drivers/net/gve/gve_tx.c line 138

Quoted 10 lines
drivers/net/gve/gve_tx.c +10 -0
+       if (ntc > start) {
+               nb_clean = ntc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               txq->sw_ntc = start;
+       }
+}

[copy/paste from previous version]

Quoted 2 lines

may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.

Yes, this 'if' block is dropped in v7 as suggested. Thanks!

Quoted 6 lines
Quoted 2 lines

This is v7 and please check above code which has the mentioned 'if' block exist, do you mean in coming v8?

Oh, sorry about this! There is another function with the same issue. I just updated the code there and forgot this. Will update this and check for the rest in the coming version. Thanks!

Sorry again about this part!!

After running the code and double checking the code, it seems that the 'if' blocks here cannot be removed directly. As you can see, in the above 'if' block, the parameter 'start' is not a fixed one and may be changed gradually. So these 'if' blocks are necessary for some certain cases.

I remembered that the queue page list is a tricky feature that can be wraparound to improve the utility or performance. Thus the queue page list should be freed twice and these two 'if' blocks here would run one by one.

I'll restore these modifications in the coming version. Thanks!

Quoted 36 lines
Quoted 36 lines
Quoted 20 lines
Quoted 20 lines

<...>

Review

Quoted 2 lines
(quoted diff) +2 -0
+uint16_t
+gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t

nb_pkts)

drivers/net/gve/gve_tx.c line 461

Quoted 9 lines
drivers/net/gve/gve_tx.c +9 -0
+{
+       struct gve_tx_queue *txq = tx_queue;
+
+       if (txq->is_gqi_qpl)
+               return gve_tx_burst_qpl(tx_queue, tx_pkts, nb_pkts);
+
+       return gve_tx_burst_ra(tx_queue, tx_pkts, nb_pkts);
+}
+

[copy/paste from previous version]

Quoted 3 lines
Quoted 3 lines

Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check on

datapath.

Quoted 12 lines
Quoted 3 lines

This is optimization and can be done later, it doesn't have to be in the set.

The exact queue type can be fetched from the backend via adminq in priv->queue_format. So there won't be mix of the queue types. Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD. Also, only GQI_QPL queue format is in use on GCP since GQI_RDA hasn't been released in production.
This part code will be optimized/refactored later when involving the queue type DQO_RDA. Thanks!

#120 lore raw
In reply to Junfeng Guo
Quoted 40 lines · Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.

Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6-v7:
improve the code.

Quoted 9 lines

Junfeng Guo (8):
net/gve/base: introduce base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

Can you please check the build error reported by CI:
https://mails.dpdk.org/archives/test-report/2022-October/318054.html

Following link can be helpful:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225324

#121 lore raw
Quoted 51 lines · Ferruh Yigit
Quoted 40 lines · Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.

Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
  - gve_adminq.c
  - gve_adminq.h
  - gve_desc.h
  - gve_desc_dqo.h
  - gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6-v7:
improve the code.

Quoted 9 lines

Junfeng Guo (8):
   net/gve/base: introduce base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
   net/gve: add support for MTU setting
   net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

Quoted 7 lines

Can you please check the build error reported by CI:
https://mails.dpdk.org/archives/test-report/2022-October/318054.html

Following link can be helpful:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225324

Reminder if this build error, please send v8 with fix.

#122 lore raw
Quoted 78 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 0/8] introduce GVE PMD

Quoted 4 lines · Ferruh Yigit
Quoted 3 lines · Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google

Cloud

Quoted 5 lines
Quoted 5 lines

Platform (GCP), which is one of the multiple virtual interfaces from those
leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can

help

Quoted 17 lines
Quoted 17 lines

those
cloud instance consumers with better experience of performance, maintenance
who wants to run their own VNFs on GCP.

Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
  - gve_adminq.c
  - gve_adminq.h
  - gve_desc.h
  - gve_desc_dqo.h
  - gve_register.h

Quoted 2 lines
Quoted 2 lines

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/tree/v1.3.0

Quoted 31 lines
Quoted 17 lines

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6-v7:
improve the code.

Quoted 9 lines

Junfeng Guo (8):
   net/gve/base: introduce base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
   net/gve: add support for MTU setting
   net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

Can you please check the build error reported by CI:
https://mails.dpdk.org/archives/test-report/2022-October/318054.html

Quoted 5 lines
Quoted 3 lines

Following link can be helpful:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225324

Reminder if this build error, please send v8 with fix.

Sure, will try to fix this. Thanks for reminding this!

#123 lore raw

PATCH v5 2/8 net/gve/base: add OS specific implementation

In reply to Junfeng Guo

Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/base/gve_adminq.h   |   2 +
 drivers/net/gve/base/gve_desc.h     |   2 +
 drivers/net/gve/base/gve_desc_dqo.h |   2 +
 drivers/net/gve/base/gve_osdep.h    | 159 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_register.h |   2 +
 5 files changed, 167 insertions(+)
 create mode 100644 drivers/net/gve/base/gve_osdep.h
drivers/net/gve/base/gve_adminq.h +2 -0
index c7114cc883..cd496760ae 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/base/gve_desc.h +2 -0
index 358755b7e0..627b9120dc 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/base/gve_desc_dqo.h +2 -0
index 0d533abcd1..5031752b43 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/base/gve_osdep.h new file +159 -0

159 lines not on this page · raw message

drivers/net/gve/base/gve_register.h +2 -0
index b65f336be2..a599c1a08e 100644
@@ -7,6 +7,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
Signature · 1 line
2.34.1
#124 lore raw

PATCH v5 3/8 net/gve: add support for device initialization

In reply to Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

---
 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  10 +
 doc/guides/nics/gve.rst                |  63 +++++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve_adminq.c      |   1 +
 drivers/net/gve/gve_ethdev.c           | 371 +++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 225 +++++++++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/meson.build            |  14 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 12 files changed, 714 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
index 3757ccc3b3..a959f710a8 100644
@@ -700,6 +700,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
new file mode 100644
index 0000000000..44aec28009
@@ -0,0 +1,10 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +63 -0

63 lines not on this page · raw message

doc/guides/nics/index.rst +1 -0
index 32c7544968..4d40ea29a3 100644
@@ -29,6 +29,7 @@ Network Interface Controller Drivers
     enetfec
     enic
     fm10k
+    gve
     hinic
     hns3
     i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
index fbb575255f..c1162ea1a4 100644
@@ -200,6 +200,11 @@ New Features
   into single event containing ``rte_event_vector``
   whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 
 Removed Items
 -------------
drivers/net/gve/base/gve_adminq.c +1 -0
index 2344100f1a..ceafa05286 100644
@@ -5,6 +5,7 @@
  * Copyright(C) 2022 Intel Corporation
  */
 
+#include "../gve_ethdev.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +371 -0

371 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h new file +225 -0

225 lines not on this page · raw message

drivers/net/gve/gve_logs.h new file +14 -0
new file mode 100644
index 0000000000..0d02da46e1
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+		__func__, ## args)
+
+#endif
drivers/net/gve/meson.build new file +14 -0
new file mode 100644
index 0000000000..d8ec64b3a3
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'base/gve_adminq.c',
+        'gve_ethdev.c',
+)
+includes += include_directories('base')
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index 35bfa78dee..355dbd07e9 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.34.1
#125 lore raw

Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
index fbb575255f..c1162ea1a4 100644

doc/guides/rel_notes/release_22_11.rst New Features · line 208

Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -200,6 +200,11 @@ New Features
    into single event containing ``rte_event_vector``
    whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 

Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?

<...>

Quoted 43 lines
doc/guides/rel_notes/release_22_11.rst +43 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);

doc/guides/rel_notes/release_22_11.rst New Features

Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10 -0
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac address");
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr, eth_dev->data->mac_addrs);
+

Is anything assinged to 'priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly, instead of allocating memory for 'eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs = &priv->dev_addr"?

<...>

doc/guides/rel_notes/release_22_11.rst

Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */

These fields are not used in this patch, I guess some will be used in datapath patch.

Can you please only add fields that is used in the patch? This way it will be clear in which functionality that field is used and enable to detect not used fields. We are accepting batch updates for base code, but this is dpdk related code, lets only add things that are used when they are used.
Same for all data structures.

<...>

drivers/net/gve/version.map line 1

Quoted 7 lines
drivers/net/gve/version.map new file +1 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

#126 lore raw

RE: [PATCH v5 3/8] net/gve: add support for device initialization

Hi

Quoted 104 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
index fbb575255f..c1162ea1a4 100644

doc/guides/rel_notes/release_22_11.rst New Features · line 208

Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -200,6 +200,11 @@ New Features
    into single event containing ``rte_event_vector``
    whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 

Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?

<...>

Quoted 36 lines
doc/guides/rel_notes/release_22_11.rst +36 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev) {
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;

doc/guides/rel_notes/release_22_11.rst New Features

Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10 -0
+       priv->state_flags = 0x0;
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct

rte_ether_addr), 0);

doc/guides/rel_notes/release_22_11.rst

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac

address");

doc/guides/rel_notes/release_22_11.rst

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr,
+ eth_dev->data->mac_addrs);
+
Quoted 4 lines

Is anything assinged to 'priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly, instead of allocating memory for 'eth_dev->data->mac_addrs'?
I mean why not "eth_dev->data->mac_addrs = &priv->dev_addr"?

Makes sense. There's no need to allocate a new memory. @Guo, Junfeng Can you update this?

Quoted 18 lines

<...>

doc/guides/rel_notes/release_22_11.rst

Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */

These fields are not used in this patch, I guess some will be used in datapath patch.

This is needed for base code gve_adminq.c not for datapath. Most of the stuff in gve_priv is for gve_adminq.c. The adminq will update this info which dpdk pmd will need later. Compiler will complain if these don't exsit.

Quoted 18 lines

Can you please only add fields that is used in the patch? This way it will be clear in which functionality that field is used and enable to detect not used fields.
We are accepting batch updates for base code, but this is dpdk related code, lets only add things that are used when they are used.
Same for all data structures.

<...>

drivers/net/gve/version.map line 1

Quoted 6 lines
drivers/net/gve/version.map new file +1 -0
new file mode 100644 index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

#127 lore raw

Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 132 lines · Xiaoyun

Hi

Quoted 104 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
index fbb575255f..c1162ea1a4 100644

doc/guides/rel_notes/release_22_11.rst New Features · line 208

Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -200,6 +200,11 @@ New Features
     into single event containing ``rte_event_vector``
     whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 

Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?

<...>

Quoted 36 lines
doc/guides/rel_notes/release_22_11.rst +36 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev) {
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;

doc/guides/rel_notes/release_22_11.rst New Features

Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10 -0
+       priv->state_flags = 0x0;
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct

rte_ether_addr), 0);

doc/guides/rel_notes/release_22_11.rst

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac

address");

doc/guides/rel_notes/release_22_11.rst

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr,
+ eth_dev->data->mac_addrs);
+
Quoted 4 lines

Is anything assinged to 'priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly, instead of allocating memory for 'eth_dev->data->mac_addrs'?
I mean why not "eth_dev->data->mac_addrs = &priv->dev_addr"?

Makes sense. There's no need to allocate a new memory. @Guo, Junfeng Can you update this?

Quoted 18 lines

<...>

doc/guides/rel_notes/release_22_11.rst

Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */

These fields are not used in this patch, I guess some will be used in datapath patch.

Quoted 3 lines

This is needed for base code gve_adminq.c not for datapath. Most of the stuff in gve_priv is for gve_adminq.c. The adminq will update this info which dpdk pmd will need later. Compiler will complain if these don't exsit.

You are right they are used by 'gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.

Quoted 18 lines
Quoted 18 lines

Can you please only add fields that is used in the patch? This way it will be clear in which functionality that field is used and enable to detect not used fields.
We are accepting batch updates for base code, but this is dpdk related code, lets only add things that are used when they are used.
Same for all data structures.

<...>

drivers/net/gve/version.map line 1

Quoted 6 lines
drivers/net/gve/version.map new file +1 -0
new file mode 100644 index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

#128 lore raw

RE: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 128 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 05:01
To: Li, Xiaoyun <[email protected]>; Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 11 lines · Xiaoyun

Hi

Quoted 8 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>;

Wang,

Quoted 2 lines
Quoted 2 lines

Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device

initialization

Quoted 26 lines
Quoted 26 lines · Junfeng Guo
Quoted 12 lines

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
index fbb575255f..c1162ea1a4 100644
@@ -200,6 +200,11 @@ New Features
     into single event containing ``rte_event_vector``
     whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
Quoted 3 lines
Quoted 3 lines

doc/guides/rel_notes/release_22_11.rst New Features

Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst +3 -0
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet

devices.

Quoted 1 line
Quoted 1 line

doc/guides/rel_notes/release_22_11.rst New Features · line 206

Quoted 1 line
doc/guides/rel_notes/release_22_11.rst +1 -0
+  * See the :doc:`../nics/gve` NIC guide for more details on this new

driver.

Quoted 46 lines
Quoted 46 lines

doc/guides/rel_notes/release_22_11.rst New Features · line 208

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +1 -0
+
 

Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?

<...>

Quoted 38 lines
doc/guides/rel_notes/release_22_11.rst +38 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev) {
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;
+
Quoted 8 lines
Quoted 8 lines

doc/guides/rel_notes/release_22_11.rst

Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst +8 -0
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac",

sizeof(struct

Quoted 3 lines
Quoted 3 lines

rte_ether_addr), 0);

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store

mac

Quoted 7 lines
Quoted 7 lines

address");

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr,
+ eth_dev->data->mac_addrs);
+
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Is anything assinged to 'priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly,

instead of

Quoted 4 lines
Quoted 2 lines

allocating memory for 'eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs = &priv->dev_addr"?

Makes sense. There's no need to allocate a new memory. @Guo,

Junfeng Can you update this?

Thanks Xiaoyun and Ferruh for the comments!
I tried to update the code as suggested but may get "Invalid Memory" warning when quit the testpmd. I found it was caused at the function rte_eth_dev_release_port with " rte_free(eth_dev->data->mac_addrs); ". Seems that allocating memory for 'eth_dev->data->mac_addrs' is still needed. Please help correct me if I misunderstood this. Thanks! I'll keep this part unchanged for the coming patchset first.

Quoted 22 lines
Quoted 17 lines
Quoted 17 lines

<...>

doc/guides/rel_notes/release_22_11.rst

Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */

These fields are not used in this patch, I guess some will be used in

datapath

Quoted 3 lines
Quoted 1 line

patch.

This is needed for base code gve_adminq.c not for datapath. Most of

the stuff in gve_priv is for gve_adminq.c.

Quoted 8 lines
Quoted 1 line

The adminq will update this info which dpdk pmd will need later.

Compiler will complain if these don't exsit.

Quoted 1 line

You are right they are used by 'gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.

Yes, we have already tried to move all the unused items to the corresponding stages patch by patch. Thanks for reminding this!

Quoted 23 lines
Quoted 3 lines
Quoted 3 lines

Can you please only add fields that is used in the patch? This way it will

be

Quoted 1 line
Quoted 1 line

clear in which functionality that field is used and enable to detect not

used

Quoted 2 lines
Quoted 2 lines

fields.
We are accepting batch updates for base code, but this is dpdk related

code,

Quoted 6 lines
Quoted 6 lines

lets only add things that are used when they are used. Same for all data structures.

<...>

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/net/gve/version.map

Quoted 7 lines
Quoted 7 lines

drivers/net/gve/version.map line 1

Quoted 5 lines

new file mode 100644 index 0000000000..c2e0723b4c

drivers/net/gve/version.map new file +1 -0
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

#129 lore raw

Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 138 lines · Junfeng

CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

Quoted 126 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 05:01
To: Li, Xiaoyun <[email protected]>; Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 11 lines · Xiaoyun

Hi

Quoted 8 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>;

Wang,

Quoted 2 lines
Quoted 2 lines

Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device

initialization

Quoted 26 lines
Quoted 26 lines · Junfeng Guo
Quoted 12 lines

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
index fbb575255f..c1162ea1a4 100644
@@ -200,6 +200,11 @@ New Features
      into single event containing ``rte_event_vector``
      whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
Quoted 3 lines
Quoted 3 lines

doc/guides/rel_notes/release_22_11.rst New Features

Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst +3 -0
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet

devices.

Quoted 1 line
Quoted 1 line

doc/guides/rel_notes/release_22_11.rst New Features · line 206

Quoted 1 line
doc/guides/rel_notes/release_22_11.rst +1 -0
+  * See the :doc:`../nics/gve` NIC guide for more details on this new

driver.

Quoted 46 lines
Quoted 46 lines

doc/guides/rel_notes/release_22_11.rst New Features · line 208

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +1 -0
+
 

Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?

<...>

Quoted 38 lines
doc/guides/rel_notes/release_22_11.rst +38 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev) {
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;
+
Quoted 8 lines
Quoted 8 lines

doc/guides/rel_notes/release_22_11.rst

Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst +8 -0
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac",

sizeof(struct

Quoted 3 lines
Quoted 3 lines

rte_ether_addr), 0);

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store

mac

Quoted 7 lines
Quoted 7 lines

address");

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr,
+ eth_dev->data->mac_addrs);
+
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Is anything assinged to 'priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly,

instead of

Quoted 4 lines
Quoted 2 lines

allocating memory for 'eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs = &priv->dev_addr"?

Makes sense. There's no need to allocate a new memory. @Guo,

Junfeng Can you update this?

Quoted 8 lines

Thanks Xiaoyun and Ferruh for the comments!
I tried to update the code as suggested but may get "Invalid Memory" warning when quit the testpmd. I found it was caused at the function rte_eth_dev_release_port with " rte_free(eth_dev->data->mac_addrs); ". Seems that allocating memory for 'eth_dev->data->mac_addrs' is still needed. Please help correct me if I misunderstood this. Thanks! I'll keep this part unchanged for the coming patchset first.

No it is not needed, you need to set pointer to NULL on release path to prevent common code free it (the problem you are getting). There are samples in various PMDs, please check.

Quoted 26 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines

<...>

doc/guides/rel_notes/release_22_11.rst

Quoted 12 lines
doc/guides/rel_notes/release_22_11.rst +12 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */

These fields are not used in this patch, I guess some will be used in

datapath

Quoted 3 lines
Quoted 1 line

patch.

This is needed for base code gve_adminq.c not for datapath. Most of

the stuff in gve_priv is for gve_adminq.c.

Quoted 1 line

The adminq will update this info which dpdk pmd will need later.

Compiler will complain if these don't exsit.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

You are right they are used by 'gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.

Yes, we have already tried to move all the unused items to the corresponding stages patch by patch. Thanks for reminding this!

thanks.

Quoted 24 lines
Quoted 23 lines
Quoted 3 lines
Quoted 3 lines

Can you please only add fields that is used in the patch? This way it will

be

Quoted 1 line
Quoted 1 line

clear in which functionality that field is used and enable to detect not

used

Quoted 2 lines
Quoted 2 lines

fields.
We are accepting batch updates for base code, but this is dpdk related

code,

Quoted 6 lines
Quoted 6 lines

lets only add things that are used when they are used. Same for all data structures.

<...>

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/net/gve/version.map

Quoted 7 lines
Quoted 7 lines

drivers/net/gve/version.map line 1

Quoted 5 lines

new file mode 100644 index 0000000000..c2e0723b4c

drivers/net/gve/version.map new file +1 -0
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

#130 lore raw

RE: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 162 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 19:16
To: Guo, Junfeng <[email protected]>; Li, Xiaoyun <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 1 line · Junfeng

CAUTION: This message has originated from an External Source. Please

use proper judgment and caution when opening attachments, clicking links, or responding to this email.

Quoted 12 lines
Quoted 12 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 05:01
To: Li, Xiaoyun <[email protected]>; Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device

initialization

Quoted 6 lines
Quoted 6 lines · Xiaoyun
Quoted 4 lines

Hi

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson,

Bruce

Quoted 31 lines
Quoted 31 lines
Quoted 1 line
Quoted 1 line

<[email protected]>; Lin, Xueqin <[email protected]>;

Wang,

Quoted 2 lines
Quoted 2 lines

Haiyue <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device

initialization

Quoted 26 lines
Quoted 26 lines · Junfeng Guo
Quoted 12 lines

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +0 -0
index fbb575255f..c1162ea1a4 100644
@@ -200,6 +200,11 @@ New Features
      into single event containing ``rte_event_vector``
      whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

doc/guides/rel_notes/release_22_11.rst New Features

Quoted 3 lines
doc/guides/rel_notes/release_22_11.rst +3 -0
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet

devices.

Quoted 1 line
Quoted 1 line
Quoted 1 line
doc/guides/rel_notes/release_22_11.rst +1 -0
+  * See the :doc:`../nics/gve` NIC guide for more details on this

new

Quoted 73 lines
Quoted 73 lines

driver.

Quoted 46 lines
Quoted 46 lines

doc/guides/rel_notes/release_22_11.rst New Features · line 208

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +1 -0
+
 

Can you please move the block amaong the other ethdev drivers, as alphabetically sorted?

<...>

Quoted 38 lines
doc/guides/rel_notes/release_22_11.rst +38 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev) {
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;
+
Quoted 8 lines
Quoted 8 lines

doc/guides/rel_notes/release_22_11.rst

Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst +8 -0
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac",

sizeof(struct

Quoted 3 lines
Quoted 3 lines

rte_ether_addr), 0);

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store

mac

Quoted 9 lines
Quoted 9 lines

address");

doc/guides/rel_notes/release_22_11.rst

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +5 -0
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr,
+ eth_dev->data->mac_addrs);
+

Is anything assinged to 'priv->dev_addr' to copy?
Also since there is a 'priv->dev_addr' field, why not use it directly,

instead of

Quoted 3 lines
Quoted 2 lines

allocating memory for 'eth_dev->data->mac_addrs'? I mean why not "eth_dev->data->mac_addrs = &priv->dev_addr"?

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

Makes sense. There's no need to allocate a new memory. @Guo,

Junfeng Can you update this?

Thanks Xiaoyun and Ferruh for the comments!
I tried to update the code as suggested but may get "Invalid Memory" warning when quit the testpmd. I found it was caused at the function rte_eth_dev_release_port with " rte_free(eth_dev->data->mac_addrs);

".

Quoted 8 lines
Quoted 4 lines

Seems that allocating memory for 'eth_dev->data->mac_addrs' is still needed. Please help correct me if I misunderstood this. Thanks! I'll keep this part unchanged for the coming patchset first.

No it is not needed, you need to set pointer to NULL on release path to prevent common code free it (the problem you are getting). There are samples in various PMDs, please check.

Yes, make sense!
I'll double check for this and update in the coming version. Thanks!

Quoted 66 lines
Quoted 27 lines
Quoted 27 lines
Quoted 4 lines
Quoted 4 lines

<...>

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +6 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
Quoted 8 lines

doc/guides/rel_notes/release_22_11.rst

Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +6 -0
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */

These fields are not used in this patch, I guess some will be used in

datapath

Quoted 3 lines
Quoted 1 line

patch.

This is needed for base code gve_adminq.c not for datapath. Most of

the stuff in gve_priv is for gve_adminq.c.

Quoted 1 line

The adminq will update this info which dpdk pmd will need later.

Compiler will complain if these don't exsit.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

You are right they are used by 'gve_adminq.c', so OK to keep them, if there are ones not used at this stage, can you add them whenever they are used, or remove them if not used at all. If all used/required, no change required.

Yes, we have already tried to move all the unused items to the

corresponding

Quoted 2 lines

stages patch by patch. Thanks for reminding this!

thanks.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Can you please only add fields that is used in the patch? This way it

will

Quoted 2 lines
Quoted 2 lines

be

Quoted 1 line
Quoted 1 line

clear in which functionality that field is used and enable to detect

not

Quoted 3 lines
Quoted 3 lines

used

Quoted 2 lines
Quoted 2 lines

fields.
We are accepting batch updates for base code, but this is dpdk

related

Quoted 16 lines
Quoted 15 lines

code,

Quoted 6 lines
Quoted 6 lines

lets only add things that are used when they are used. Same for all data structures.

<...>

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/net/gve/version.map

Quoted 7 lines
Quoted 7 lines

drivers/net/gve/version.map line 1

Quoted 5 lines

new file mode 100644 index 0000000000..c2e0723b4c

drivers/net/gve/version.map new file +1 -0
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

#131 lore raw

Re: [PATCH v5 3/8] net/gve: add support for device initialization

In reply to Junfeng Guo
Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

doc/guides/nics/gve.rst line 9

Quoted 12 lines
doc/guides/nics/gve.rst new file +9 -0
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support for
+Google Virtual Ethernet device (also called as gVNIC).
+

This is a virtual device, emulated in VM as PCI device, right? If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?

doc/guides/nics/gve.rst line 11

Quoted 2 lines
doc/guides/nics/gve.rst new file +2 -0
+Current gVNIC is an alternative to the virtIO-based ethernet interface that can
+support higher network bandwidths such as the 50-100 Gbps speeds.

This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.

Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report? Aren't there any other notable difference?

I think better to document as much as possible, cc'ed more virtio people.

#132 lore raw

RE: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 29 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 9:47 PM
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 14 lines

doc/guides/nics/gve.rst line 9

Quoted 12 lines
doc/guides/nics/gve.rst new file +9 -0
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support for
+Google Virtual Ethernet device (also called as gVNIC).
+

This is a virtual device, emulated in VM as PCI device, right?

Most likely yes. But I think this part is not public by Google. (Correct me if I am wrong)

Quoted 1 line

If so what emulates it, I mean can we use QEMU for it?

Yes if the user can have a emulated gVNIC device or real HW that supports gVNIC interface.

Quoted 1 line

And is there a kernel supported backend, as virtio has vhost?

I guess no if the back-end is not public

Quoted 7 lines

doc/guides/nics/gve.rst line 10

Quoted 2 lines
doc/guides/nics/gve.rst new file +1 -0
+Current gVNIC is an alternative to the virtIO-based ethernet interface

that can

doc/guides/nics/gve.rst line 11

Quoted 1 line
doc/guides/nics/gve.rst new file +1 -0
+support higher network bandwidths such as the 50-100 Gbps speeds.

This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.

I don't think it's good to compare these two. It's just two virtual interface, I don't see strong analysis to prove which one is better.

Like if you use Google cloud, you use gVNIC. Use other cloud that uses virtio, choose virtio.

Quoted 4 lines

Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?
Aren't there any other notable difference?

If we want to leave such description, I agree with Ferruh that we need to give some evidence.

But in my understanding, they are just two similar virtual interfaces. One has public standard, one is google-specific. Users need to use one of them when they choose to use one cloud service.

Thanks,
Chenbo

Quoted 3 lines

I think better to document as much as possible, cc'ed more virtio people.

#133 lore raw

RE: [PATCH v5 3/8] net/gve: add support for device initialization

In reply to Ferruh Yigit
Quoted 26 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 9:47 PM
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 16 lines

doc/guides/nics/gve.rst line 9

Quoted 12 lines
doc/guides/nics/gve.rst new file +9 -0
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support 
+for Google Virtual Ethernet device (also called as gVNIC).
+

This is a virtual device, emulated in VM as PCI device, right? If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?

This is a virtual interface only provided on Google Cloud Platform (GCP), like ena on AWS, and virtio on Alibaba Cloud, etc.
The gve (gVNIC) is the standard virtual ethernet interface provided to users when anyone buys a cloud instance on GCP, which is ready there and everybody can access it. The backend details are not open to anyone outside of Google, I assume it is more like a business & technical secret of Google.

Quoted 3 lines

doc/guides/nics/gve.rst line 11

Quoted 3 lines
doc/guides/nics/gve.rst new file +2 -0
+Current gVNIC is an alternative to the virtIO-based ethernet 
+interface that can support higher network bandwidths such as the 50-100 Gbps speeds.

I don't think it is an alternative to virtio. Gve is the driver name of gVNIC of GCP, which is the standard virtual ethernet interface on that cloud platform.

Quoted 4 lines

This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.

Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?

There is no hint that gve (or virtio) interface is faster than virtio (or gve). I think it heavily depends on the backend design, which can be SW or HW.
I would treat gve (actually the driver for gVNIC) as just one of the leading virtual ethernet interfaces on different cloud environments. Till now, gve is for GCP cloud environment only.

Hopefully my understanding is correct, as I am not the expert at Google cloud, and I got all the information from public.

Thanks for all the good questions! Hopefully my answers help!

Regards,
Helin

Quoted 3 lines

Aren't there any other notable difference?

I think better to document as much as possible, cc'ed more virtio people.

#134 lore raw

Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 42 lines · Helin
Quoted 26 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 9:47 PM
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 16 lines

doc/guides/nics/gve.rst line 9

Quoted 12 lines
doc/guides/nics/gve.rst new file +9 -0
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support
+for Google Virtual Ethernet device (also called as gVNIC).
+

This is a virtual device, emulated in VM as PCI device, right? If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?

Quoted 8 lines

This is a virtual interface only provided on Google Cloud Platform (GCP), like ena on AWS, and virtio on Alibaba Cloud, etc.
The gve (gVNIC) is the standard virtual ethernet interface provided to users when anyone buys a cloud instance on GCP, which is ready there and everybody can access it. The backend details are not open to anyone outside of Google, I assume it is more like a business & technical secret of Google.

Quoted 3 lines

doc/guides/nics/gve.rst line 11

Quoted 3 lines
doc/guides/nics/gve.rst new file +2 -0
+Current gVNIC is an alternative to the virtIO-based ethernet
+interface that can support higher network bandwidths such as the 50-100 Gbps speeds.

I don't think it is an alternative to virtio. Gve is the driver name of gVNIC of GCP, which is the standard virtual ethernet interface on that cloud platform.

Thanks Helin, Chenbo, Xiaoyun for clarification,

My understanding that gVNIC is another virtual interface spec like 'virtio' is not correct, so comparing this with virtio and to document it is not really necessary.

Perhaps "gVNIC is an alternative to the virtIO-based ethernet interface" part in the documentation can be updated slightly, because as far as I understand gVNIC is an alternative to virtio in the *Google Cloud platform*, not a general alternative.

Overall, agree to enable virtual network interface in Google cloud and as you all highlighted we have similar device support for other cloud platforms.

Thanks,
ferruh

Quoted 19 lines
Quoted 4 lines

This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.

Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?

There is no hint that gve (or virtio) interface is faster than virtio (or gve). I think it heavily depends on the backend design, which can be SW or HW.
I would treat gve (actually the driver for gVNIC) as just one of the leading virtual ethernet interfaces on different cloud environments. Till now, gve is for GCP cloud environment only.

Hopefully my understanding is correct, as I am not the expert at Google cloud, and I got all the information from public.

Thanks for all the good questions! Hopefully my answers help!

Regards,
Helin

Quoted 3 lines

Aren't there any other notable difference?

I think better to document as much as possible, cc'ed more virtio people.

#135 lore raw

RE: [PATCH v5 3/8] net/gve: add support for device initialization

In reply to Ferruh Yigit

Hi

Quoted 31 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:47
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>; Zhang, Helin <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH v5 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 14 lines

doc/guides/nics/gve.rst line 9

Quoted 12 lines
doc/guides/nics/gve.rst new file +9 -0
@@ -0,0 +1,63 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support
+for Google Virtual Ethernet device (also called as gVNIC).
+

This is a virtual device, emulated in VM as PCI device, right?

It's similar to ENA driver for AWS. This is Google's own device for GCP.

Quoted 2 lines

If so what emulates it, I mean can we use QEMU for it? And is there a kernel supported backend, as virtio has vhost?

No. backend is not public.

Quoted 7 lines

doc/guides/nics/gve.rst line 11

Quoted 3 lines
doc/guides/nics/gve.rst new file +2 -0
+Current gVNIC is an alternative to the virtIO-based ethernet
+interface that can support higher network bandwidths such as the 50-100

Gbps speeds.

This is an alternative to virtio, and it would be good to document pros/cons of this device/approach, to help users to chose one or other.

In GCP, they provide two types of devices which users can choose for the interfaces of the instance, virtio or gvnic, if users choose gvnic, then gve driver is needed and if they choose virtio, vitio is used. I'm not sure we should suggest which one to choose.

I guess the best thing to do is just linking google's doc and let users to choose themselves.
https://cloud.google.com/compute/docs/networking/using-gvnic

Quoted 4 lines

Is "support higher network bandwidths" means this device is faster than virtio? Is there any performance report?
Aren't there any other notable difference?

This is from https://cloud.google.com/compute/docs/networking/using-gvnic.
There is no official performance report. Google didn't provide any.

From our private testing, gvnic is faster than virtio using the same instance type but virtio is more stable. But our testing can be wrong. It all depends how you config and use cloud env.

Quoted 3 lines

I think better to document as much as possible, cc'ed more virtio people.

#136 lore raw

PATCH v5 4/8 net/gve: add support for link update

In reply to Junfeng Guo

Support dev_ops link_update.

---
 doc/guides/nics/features/gve.ini |  1 +
 doc/guides/nics/gve.rst          |  3 +++
 drivers/net/gve/gve_ethdev.c     | 30 ++++++++++++++++++++++++++++++
 3 files changed, 34 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index 44aec28009..ae466ad677 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +3 -0
index 4fbef8f9dc..6e95528f55 100644
@@ -55,6 +55,9 @@ Features and Limitations
 
 In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
+Supported features of the GVE PMD are:
+
+- Link state information
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +30 -0
index f8ff3b1923..ca4a467140 100644
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_DRV_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -70,6 +99,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.34.1
#137 lore raw

PATCH v5 5/8 net/gve: add support for MTU setting

In reply to Junfeng Guo

Support dev_ops mtu_set.

---
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index ae466ad677..d1703d8dab 100644
@@ -5,6 +5,7 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +29 -0
index ca4a467140..e9c68964ac 100644
@@ -94,12 +94,41 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	dev->data->dev_conf.rxmode.mtu = mtu + RTE_ETHER_HDR_LEN;
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.34.1
#138 lore raw

Re: [PATCH v5 5/8] net/gve: add support for MTU setting

Quoted 5 lines · Junfeng Guo

Support dev_ops mtu_set.

<...>

Quoted 9 lines
drivers/net/gve/gve_ethdev.c +8 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 114

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+               return -EBUSY;
+       }
+
+       dev->data->dev_conf.rxmode.mtu = mtu + RTE_ETHER_HDR_LEN;

it is 'dev->data->mtu' that holds latest MTU value.

'dev_conf.rxmode.mtu' is the config requested from user, no need to update that.

And since 'dev->data->mtu' already updated by 'rte_eth_dev_set_mtu()', can drop above line.

#139 lore raw

RE: [PATCH v5 5/8] net/gve: add support for MTU setting

Quoted 44 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:48
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 5/8] net/gve: add support for MTU setting

Quoted 5 lines · Junfeng Guo

Support dev_ops mtu_set.

<...>

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 9 lines
drivers/net/gve/gve_ethdev.c +8 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u",

RTE_ETHER_MIN_MTU, priv->max_mtu);

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before

configuration");

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 114

Quoted 4 lines
drivers/net/gve/gve_ethdev.c +4 -0
+               return -EBUSY;
+       }
+
+       dev->data->dev_conf.rxmode.mtu = mtu + RTE_ETHER_HDR_LEN;
Quoted 7 lines

it is 'dev->data->mtu' that holds latest MTU value.

'dev_conf.rxmode.mtu' is the config requested from user, no need to update that.

And since 'dev->data->mtu' already updated by 'rte_eth_dev_set_mtu()', can drop above line.

Will fix this, thanks!

#140 lore raw

PATCH v5 6/8 net/gve: add support for dev info get and dev configure

In reply to Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
 doc/guides/nics/features/gve.ini |  2 +
 doc/guides/nics/gve.rst          |  1 +
 drivers/net/gve/gve_ethdev.c     | 65 +++++++++++++++++++++++++++++++-
 3 files changed, 67 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
index d1703d8dab..986df7f94a 100644
@@ -4,8 +4,10 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+RSS hash             = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +1 -0
index 6e95528f55..d76f9bd5b9 100644
@@ -57,6 +57,7 @@ In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
 Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)
 - Link state information
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
drivers/net/gve/gve_ethdev.c +64 -1

84 lines not on this page · raw message

Signature · 1 line
2.34.1
#141 lore raw

Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

<...>

Quoted 4 lines
drivers/net/gve/gve_ethdev.c +0 -0
@@ -29,8 +29,16 @@ gve_write_version(uint8_t *driver_version_register)
  }

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register) · line 38

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +6 -1
 
  static int
-gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
+gve_dev_configure(struct rte_eth_dev *dev)
  {
+       struct gve_priv *priv = dev->data->dev_private;
+
+       if (dev->data->dev_conf.rxmode.mq_mode & RTE_ETH_MQ_RX_RSS_FLAG)
+               dev->data->dev_conf.rxmode.offloads |= RTE_ETH_RX_OFFLOAD_RSS_HASH;
+

This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.

Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?

Quoted 12 lines
drivers/net/gve/gve_ethdev.c +5 -0
+       if (dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_TCP_LRO)
+               priv->enable_rsc = 1;
+
         return 0;
  }
 
@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)
         return err;
  }
 
+static int
+gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+{
+       struct gve_priv *priv = dev->data->dev_private;
+
+       dev_info->device = dev->device;
+       dev_info->max_mac_addrs = 1;
+       dev_info->max_rx_queues = priv->max_nb_rxq;
+       dev_info->max_tx_queues = priv->max_nb_txq;
+       dev_info->min_rx_bufsize = GVE_MIN_BUF_SIZE;
+       dev_info->max_rx_pktlen = GVE_MAX_RX_PKTLEN;
+       dev_info->max_mtu = RTE_ETHER_MTU;

Can you please confirm max MTU this PMD supports is 1500? Meaning it doesn't support jumbo frames etc...

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 126

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+       dev_info->min_mtu = RTE_ETHER_MIN_MTU;
+
+       dev_info->rx_offload_capa = 0;
+       dev_info->tx_offload_capa =
+               RTE_ETH_TX_OFFLOAD_MULTI_SEGS   |
+               RTE_ETH_TX_OFFLOAD_IPV4_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_UDP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_TCP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_SCTP_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_TCP_TSO;

Can you adverstise these capabilities in the patch that implements them?

#142 lore raw

RE: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure

Quoted 42 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

<...>

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t

Quoted 3 lines
drivers/net/gve/gve_ethdev.c +0 -0
@@ -29,8 +29,16 @@ gve_write_version(uint8_t

*driver_version_register)

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register)

Quoted 9 lines
drivers/net/gve/gve_ethdev.c +4 -1
  }
 
  static int
-gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
+gve_dev_configure(struct rte_eth_dev *dev)
  {
+       struct gve_priv *priv = dev->data->dev_private;
+
+       if (dev->data->dev_conf.rxmode.mq_mode &

RTE_ETH_MQ_RX_RSS_FLAG)

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 37

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+               dev->data->dev_conf.rxmode.offloads |=

RTE_ETH_RX_OFFLOAD_RSS_HASH;

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 38

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+
Quoted 7 lines

This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.

Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?

Yes, it seems force enabling this offloading is not quite reasonable here. This may just follow previous PMD convention, so we decided to remove this part in the coming version. Thanks!

Quoted 15 lines

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 39

Quoted 2 lines
drivers/net/gve/gve_ethdev.c +1 -0
+       if (dev->data->dev_conf.rxmode.offloads &

RTE_ETH_RX_OFFLOAD_TCP_LRO)

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 106

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +4 -0
+               priv->enable_rsc = 1;
+
         return 0;
  }
 
@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)
         return err;
  }
 
+static int
+gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info

*dev_info)

Quoted 13 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+{
+       struct gve_priv *priv = dev->data->dev_private;
+
+       dev_info->device = dev->device;
+       dev_info->max_mac_addrs = 1;
+       dev_info->max_rx_queues = priv->max_nb_rxq;
+       dev_info->max_tx_queues = priv->max_nb_txq;
+       dev_info->min_rx_bufsize = GVE_MIN_BUF_SIZE;
+       dev_info->max_rx_pktlen = GVE_MAX_RX_PKTLEN;
+       dev_info->max_mtu = RTE_ETHER_MTU;

Can you please confirm max MTU this PMD supports is 1500? Meaning it doesn't support jumbo frames etc...

Actually here is just a workaround solution for the max_mtu info... We can only get the max_mtu value via adminq message from the backend. But the real one (i.e., priv->max_mtu) we get is 1460, which is less than 1500 Seems it is the GCP bug or something.
If we use "dev_info->max_mtu = priv->max_mtu", the testpmd cannot even be launched successfully...
I'll keep this part unchanged with some comments here if no other solutions. Please help correct me if you have any other idea. Thanks a lot!

Quoted 12 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 126

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +10 -0
+       dev_info->min_mtu = RTE_ETHER_MIN_MTU;
+
+       dev_info->rx_offload_capa = 0;
+       dev_info->tx_offload_capa =
+               RTE_ETH_TX_OFFLOAD_MULTI_SEGS   |
+               RTE_ETH_TX_OFFLOAD_IPV4_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_UDP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_TCP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_SCTP_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_TCP_TSO;
Quoted 1 line

Can you adverstise these capabilities in the patch that implements them?

Will move this to the corresponding patch, thanks!

#143 lore raw

Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure

Quoted 83 lines · Junfeng
Quoted 42 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

<...>

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t

Quoted 3 lines
drivers/net/gve/gve_ethdev.c +0 -0
@@ -29,8 +29,16 @@ gve_write_version(uint8_t

*driver_version_register)

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register)

Quoted 9 lines
drivers/net/gve/gve_ethdev.c +4 -1
   }
 
   static int
-gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
+gve_dev_configure(struct rte_eth_dev *dev)
   {
+       struct gve_priv *priv = dev->data->dev_private;
+
+       if (dev->data->dev_conf.rxmode.mq_mode &

RTE_ETH_MQ_RX_RSS_FLAG)

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 37

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+               dev->data->dev_conf.rxmode.offloads |=

RTE_ETH_RX_OFFLOAD_RSS_HASH;

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 38

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+
Quoted 7 lines

This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.

Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?

Yes, it seems force enabling this offloading is not quite reasonable here. This may just follow previous PMD convention, so we decided to remove this part in the coming version. Thanks!

Quoted 15 lines

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 39

Quoted 2 lines
drivers/net/gve/gve_ethdev.c +1 -0
+       if (dev->data->dev_conf.rxmode.offloads &

RTE_ETH_RX_OFFLOAD_TCP_LRO)

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 106

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +4 -0
+               priv->enable_rsc = 1;
+
          return 0;
   }
 
@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)
          return err;
   }
 
+static int
+gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info

*dev_info)

Quoted 13 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+{
+       struct gve_priv *priv = dev->data->dev_private;
+
+       dev_info->device = dev->device;
+       dev_info->max_mac_addrs = 1;
+       dev_info->max_rx_queues = priv->max_nb_rxq;
+       dev_info->max_tx_queues = priv->max_nb_txq;
+       dev_info->min_rx_bufsize = GVE_MIN_BUF_SIZE;
+       dev_info->max_rx_pktlen = GVE_MAX_RX_PKTLEN;
+       dev_info->max_mtu = RTE_ETHER_MTU;

Can you please confirm max MTU this PMD supports is 1500? Meaning it doesn't support jumbo frames etc...

Quoted 9 lines

Actually here is just a workaround solution for the max_mtu info... We can only get the max_mtu value via adminq message from the backend. But the real one (i.e., priv->max_mtu) we get is 1460, which is less than 1500 Seems it is the GCP bug or something.
If we use "dev_info->max_mtu = priv->max_mtu", the testpmd cannot even be launched successfully...
I'll keep this part unchanged with some comments here if no other solutions. Please help correct me if you have any other idea. Thanks a lot!

Getting actual value from device is correct thing to do, but it seems received value is not good, so OK to keep as it is. Can you please follow this with GVE?

Quoted 16 lines
Quoted 12 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 126

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +10 -0
+       dev_info->min_mtu = RTE_ETHER_MIN_MTU;
+
+       dev_info->rx_offload_capa = 0;
+       dev_info->tx_offload_capa =
+               RTE_ETH_TX_OFFLOAD_MULTI_SEGS   |
+               RTE_ETH_TX_OFFLOAD_IPV4_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_UDP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_TCP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_SCTP_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_TCP_TSO;
Quoted 1 line

Can you adverstise these capabilities in the patch that implements them?

Will move this to the corresponding patch, thanks!

#144 lore raw

RE: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure

Quoted 110 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 19:20
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get and dev configure

Quoted 11 lines · Junfeng
Quoted 11 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 6/8] net/gve: add support for dev info get and

dev

Quoted 36 lines
Quoted 36 lines

configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

<...>

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t

Quoted 3 lines
drivers/net/gve/gve_ethdev.c +0 -0
@@ -29,8 +29,16 @@ gve_write_version(uint8_t

*driver_version_register)

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t *driver_version_register)

Quoted 9 lines
drivers/net/gve/gve_ethdev.c +4 -1
   }
 
   static int
-gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
+gve_dev_configure(struct rte_eth_dev *dev)
   {
+       struct gve_priv *priv = dev->data->dev_private;
+
+       if (dev->data->dev_conf.rxmode.mq_mode &

RTE_ETH_MQ_RX_RSS_FLAG)

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 37

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+               dev->data->dev_conf.rxmode.offloads |=

RTE_ETH_RX_OFFLOAD_RSS_HASH;

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 38

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+

This is force enabling the feature, we are doing this for PMDs that has the hash value anyway and no additional work or performance loss observed to enable this offload. Otherwise drivers shouldn't update 'dev_conf.rxmode'.

Quoted 5 lines
Quoted 2 lines

Can you please confirm this PMD fits above description? And can you please add a coment that says force enabling the feature?

Yes, it seems force enabling this offloading is not quite reasonable here. This may just follow previous PMD convention, so we decided to

remove

Quoted 29 lines

this part in the coming version. Thanks!

Quoted 27 lines

drivers/net/gve/gve_ethdev.c gve_write_version(uint8_t · line 39

Quoted 2 lines
drivers/net/gve/gve_ethdev.c +1 -0
+       if (dev->data->dev_conf.rxmode.offloads &

RTE_ETH_RX_OFFLOAD_TCP_LRO)

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 106

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +4 -0
+               priv->enable_rsc = 1;
+
          return 0;
   }
 
@@ -94,6 +102,60 @@ gve_dev_close(struct rte_eth_dev *dev)
          return err;
   }
 
+static int
+gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info

*dev_info)

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 116

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+{
+       struct gve_priv *priv = dev->data->dev_private;
+
+       dev_info->device = dev->device;
+       dev_info->max_mac_addrs = 1;
+       dev_info->max_rx_queues = priv->max_nb_rxq;
+       dev_info->max_tx_queues = priv->max_nb_txq;
+       dev_info->min_rx_bufsize = GVE_MIN_BUF_SIZE;
+       dev_info->max_rx_pktlen = GVE_MAX_RX_PKTLEN;
+       dev_info->max_mtu = RTE_ETHER_MTU;

Can you please confirm max MTU this PMD supports is 1500? Meaning

it

Quoted 4 lines
Quoted 1 line

doesn't support jumbo frames etc...

Actually here is just a workaround solution for the max_mtu info... We can only get the max_mtu value via adminq message from the

backend.

Quoted 1 line

But the real one (i.e., priv->max_mtu) we get is 1460, which is less than

1500

Quoted 2 lines

Seems it is the GCP bug or something.
If we use "dev_info->max_mtu = priv->max_mtu", the testpmd cannot

even

Quoted 2 lines

be launched successfully...
I'll keep this part unchanged with some comments here if no other

Quoted 7 lines

solutions.

Quoted 2 lines

Please help correct me if you have any other idea. Thanks a lot!

Getting actual value from device is correct thing to do, but it seems received value is not good, so OK to keep as it is.
Can you please follow this with GVE?

Sure, will update this in the coming version. Thanks!

Quoted 18 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
drivers/net/gve/gve_ethdev.c +10 -0
+       dev_info->min_mtu = RTE_ETHER_MIN_MTU;
+
+       dev_info->rx_offload_capa = 0;
+       dev_info->tx_offload_capa =
+               RTE_ETH_TX_OFFLOAD_MULTI_SEGS   |
+               RTE_ETH_TX_OFFLOAD_IPV4_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_UDP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_TCP_CKSUM    |
+               RTE_ETH_TX_OFFLOAD_SCTP_CKSUM   |
+               RTE_ETH_TX_OFFLOAD_TCP_TSO;
Quoted 1 line
Quoted 1 line

Can you adverstise these capabilities in the patch that implements

them?

Quoted 3 lines

Will move this to the corresponding patch, thanks!

#145 lore raw

PATCH v5 7/8 net/gve: add support for queue operations

In reply to Junfeng Guo

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve_ethdev.c | 204 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h |  52 +++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 684 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +204 -0
index 2f914b55ed..5c568268fa 100644
@@ -28,6 +28,68 @@ gve_write_version(uint8_t *driver_version_register)
 	writeb('\n', driver_version_register);
 }
 
+static int
+gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pages)
+{
+	char z_name[RTE_MEMZONE_NAMESIZE];
+	struct gve_queue_page_list *qpl;
+	const struct rte_memzone *mz;
+	dma_addr_t page_bus;
+	uint32_t i;
+
+	if (priv->num_registered_pages + pages >
+	    priv->max_registered_pages) {
+		PMD_DRV_LOG(ERR, "Pages %" PRIu64 " > max registered pages %" PRIu64,
+			    priv->num_registered_pages + pages,
+			    priv->max_registered_pages);
+		return -EINVAL;
+	}
+	qpl = &priv->qpl[id];
+	snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_dev->device.name, id);
+	mz = rte_memzone_reserve_aligned(z_name, pages * PAGE_SIZE,
+					 rte_socket_id(),
+					 RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
+	if (mz == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name);
+		return -ENOMEM;
+	}
+	qpl->page_buses = rte_zmalloc("qpl page buses", pages * sizeof(dma_addr_t), 0);
+	if (qpl->page_buses == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page buses", id);
+		return -ENOMEM;
+	}
+	page_bus = mz->iova;
+	for (i = 0; i < pages; i++) {
+		qpl->page_buses[i] = page_bus;
+		page_bus += PAGE_SIZE;
+	}
+	qpl->id = id;
+	qpl->mz = mz;
+	qpl->num_entries = pages;
+
+	priv->num_registered_pages += pages;
+
+	return 0;
+}
+
+static void
+gve_free_qpls(struct gve_priv *priv)
+{
+	uint16_t nb_txqs = priv->max_nb_txq;
+	uint16_t nb_rxqs = priv->max_nb_rxq;
+	uint32_t i;
+
+	for (i = 0; i < nb_txqs + nb_rxqs; i++) {
+		if (priv->qpl[i].mz != NULL)
+			rte_memzone_free(priv->qpl[i].mz);
+		if (priv->qpl[i].page_buses != NULL)
+			rte_free(priv->qpl[i].page_buses);
+	}
+
+	if (priv->qpl != NULL)
+		rte_free(priv->qpl);
+}
+
 static int
 gve_dev_configure(struct rte_eth_dev *dev)
 {
@@ -42,6 +104,43 @@ gve_dev_configure(struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_refill_pages(struct gve_rx_queue *rxq)
+{
+	struct rte_mbuf *nmb;
+	uint16_t i;
+	int diag;
+
+	diag = rte_pktmbuf_alloc_bulk(rxq->mpool, &rxq->sw_ring[0], rxq->nb_rx_desc);
+	if (diag < 0) {
+		for (i = 0; i < rxq->nb_rx_desc - 1; i++) {
+			nmb = rte_pktmbuf_alloc(rxq->mpool);
+			if (!nmb)
+				break;
+			rxq->sw_ring[i] = nmb;
+		}
+		if (i < rxq->nb_rx_desc - 1)
+			return -ENOMEM;
+	}
+	rxq->nb_avail = 0;
+	rxq->next_avail = rxq->nb_rx_desc - 1;
+
+	for (i = 0; i < rxq->nb_rx_desc; i++) {
+		if (rxq->is_gqi_qpl) {
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(i * PAGE_SIZE);
+		} else {
+			if (i == rxq->nb_rx_desc - 1)
+				break;
+			nmb = rxq->sw_ring[i];
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(rte_mbuf_data_iova(nmb));
+		}
+	}
+
+	rte_write32(rte_cpu_to_be_32(rxq->next_avail), rxq->qrx_tail);
+
+	return 0;
+}
+
 static int
 gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 {
@@ -73,16 +172,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
+	uint16_t num_queues = dev->data->nb_tx_queues;
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
+	uint16_t i;
+	int err;
+
+	priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues;
+	err = gve_adminq_create_tx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u tx queues.", num_queues);
+		return err;
+	}
+	for (i = 0; i < num_queues; i++) {
+		txq = priv->txqs[i];
+		txq->qtx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(txq->qres->db_index)];
+		txq->qtx_head =
+		&priv->cnt_array[rte_be_to_cpu_32(txq->qres->counter_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), txq->ntfy_addr);
+	}
+
+	num_queues = dev->data->nb_rx_queues;
+	priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues;
+	err = gve_adminq_create_rx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u rx queues.", num_queues);
+		goto err_tx;
+	}
+	for (i = 0; i < num_queues; i++) {
+		rxq = priv->rxqs[i];
+		rxq->qrx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(rxq->qres->db_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), rxq->ntfy_addr);
+
+		err = gve_refill_pages(rxq);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to refill for RX");
+			goto err_rx;
+		}
+	}
+
 	dev->data->dev_started = 1;
 	gve_link_update(dev, 0);
 
 	return 0;
+
+err_rx:
+	gve_stop_rx_queues(dev);
+err_tx:
+	gve_stop_tx_queues(dev);
+	return err;
 }
 
 static int
 gve_dev_stop(struct rte_eth_dev *dev)
 {
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+
+	gve_stop_tx_queues(dev);
+	gve_stop_rx_queues(dev);
+
 	dev->data->dev_started = 0;
 
 	return 0;
@@ -91,7 +244,11 @@ gve_dev_stop(struct rte_eth_dev *dev)
 static int
 gve_dev_close(struct rte_eth_dev *dev)
 {
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
 	int err = 0;
+	uint16_t i;
 
 	if (dev->data->dev_started) {
 		err = gve_dev_stop(dev);
@@ -99,6 +256,19 @@ gve_dev_close(struct rte_eth_dev *dev)
 			PMD_DRV_LOG(ERR, "Failed to stop dev.");
 	}
 
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		txq = dev->data->tx_queues[i];
+		gve_tx_queue_release(txq);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		rxq = dev->data->rx_queues[i];
+		gve_rx_queue_release(rxq);
+	}
+
+	gve_free_qpls(priv);
+	rte_free(priv->adminq);
+
 	return err;
 }
 
@@ -190,6 +360,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.dev_infos_get        = gve_dev_info_get,
+	.rx_queue_setup       = gve_rx_queue_setup,
+	.tx_queue_setup       = gve_tx_queue_setup,
 	.link_update          = gve_link_update,
 	.mtu_set              = gve_dev_mtu_set,
 };
@@ -327,7 +499,9 @@ gve_setup_device_resources(struct gve_priv *priv)
 static int
 gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 {
+	uint16_t pages;
 	int num_ntfy;
+	uint32_t i;
 	int err;
 
 	/* Set up the adminq */
@@ -378,10 +552,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 	PMD_DRV_LOG(INFO, "Max TX queues %d, Max RX queues %d",
 		    priv->max_nb_txq, priv->max_nb_rxq);
 
+	/* In GQI_QPL queue format:
+	 * Allocate queue page lists according to max queue number
+	 * tx qpl id should start from 0 while rx qpl id should start
+	 * from priv->max_nb_txq
+	 */
+	if (priv->queue_format == GVE_GQI_QPL_FORMAT) {
+		priv->qpl = rte_zmalloc("gve_qpl",
+					(priv->max_nb_txq + priv->max_nb_rxq) *
+					sizeof(struct gve_queue_page_list), 0);
+		if (priv->qpl == NULL) {
+			PMD_DRV_LOG(ERR, "Failed to alloc qpl.");
+			err = -ENOMEM;
+			goto free_adminq;
+		}
+
+		for (i = 0; i < priv->max_nb_txq + priv->max_nb_rxq; i++) {
+			if (i < priv->max_nb_txq)
+				pages = priv->tx_pages_per_qpl;
+			else
+				pages = priv->rx_data_slot_cnt;
+			err = gve_alloc_queue_page_list(priv, i, pages);
+			if (err != 0) {
+				PMD_DRV_LOG(ERR, "Failed to alloc qpl %u.", i);
+				goto err_qpl;
+			}
+		}
+	}
+
 setup_device:
 	err = gve_setup_device_resources(priv);
 	if (!err)
 		return 0;
+err_qpl:
+	gve_free_qpls(priv);
 free_adminq:
 	gve_adminq_free(priv);
 	return err;
drivers/net/gve/gve_ethdev.h +52 -0

92 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index d8ec64b3a3..af0010c01c 100644
@@ -9,6 +9,8 @@ endif
 
 sources = files(
         'base/gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
 includes += include_directories('base')
Signature · 1 line
2.34.1
#146 lore raw

PATCH v5 8/8 net/gve: add support for Rx/Tx

In reply to Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 doc/guides/nics/features/gve.ini |   2 +
 doc/guides/nics/gve.rst          |   4 +
 drivers/net/gve/gve_ethdev.c     |   7 +
 drivers/net/gve/gve_ethdev.h     |  18 ++
 drivers/net/gve/gve_rx.c         | 140 ++++++++++
 drivers/net/gve/gve_tx.c         | 455 +++++++++++++++++++++++++++++++
 6 files changed, 626 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
index 986df7f94a..cdc46b08a3 100644
@@ -7,7 +7,9 @@
 Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+TSO                  = Y
 RSS hash             = Y
+L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +4 -0
index d76f9bd5b9..f675679625 100644
@@ -57,8 +57,12 @@ In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
 Supported features of the GVE PMD are:
 
+- Multiple queues for TX and RX
 - Receiver Side Scaling (RSS)
+- TSO offload
 - Link state information
+- TX multi-segments (Scatter TX)
+- Tx UDP/TCP/SCTP Checksum
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +7 -0
index 5c568268fa..47ffb3afff 100644
@@ -644,6 +644,13 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	} else {
+		PMD_DRV_LOG(ERR, "DQO_RDA is not implemented and will be added in the future");
+	}
+
 	eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
 	if (!eth_dev->data->mac_addrs) {
 		PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac address");
drivers/net/gve/gve_ethdev.h +18 -0
index 20fe57781e..266b831a01 100644
@@ -34,6 +34,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -274,4 +286,10 @@ gve_stop_tx_queues(struct rte_eth_dev *dev);
 void
 gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t
+gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t
+gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +140 -0

146 lines not on this page · raw message

drivers/net/gve/gve_tx.c +455 -0

461 lines not on this page · raw message

Signature · 1 line
2.34.1
#147 lore raw

Re: [PATCH v5 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 10 lines
drivers/net/gve/gve_rx.c +10 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+

drivers/net/gve/gve_rx.c line 102

Quoted 10 lines
drivers/net/gve/gve_rx.c +10 -0
+       rxr = rxq->rx_desc_ring;
+
+       for (nb_rx = 0; nb_rx < nb_pkts; nb_rx++) {
+               rxd = &rxr[rx_id];
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
+                       break;
+
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+

I think above code wrong.
Function returns 'nb_rx', if you continue on error 'nb_rx' kept increased, so application will receive wrong number of packets.

Also packets assigned as "rx_pkts[nb_rx] = rxe;", this will cause gaps in the arrays.

You can either break on the error, or keep another variable to store number of received packets.

drivers/net/gve/gve_rx.c line 105

Quoted 3 lines
drivers/net/gve/gve_rx.c +3 -0
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               rxe->data_off = RTE_PKTMBUF_HEADROOM;

As far as I can see mbufs allocated using 'rte_pktmbuf_alloc()', if so no need to explicitly set the 'm->data_off'.

#148 lore raw

RE: [PATCH v5 8/8] net/gve: add support for Rx/Tx

Quoted 48 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:47
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v5 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Review

Quoted 2 lines
(quoted diff) +2 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t

nb_pkts)

Quoted 8 lines
drivers/net/gve/gve_rx.c +8 -0
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+

drivers/net/gve/gve_rx.c line 102

Quoted 10 lines
drivers/net/gve/gve_rx.c +10 -0
+       rxr = rxq->rx_desc_ring;
+
+       for (nb_rx = 0; nb_rx < nb_pkts; nb_rx++) {
+               rxd = &rxr[rx_id];
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
+                       break;
+
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+

I think above code wrong.
Function returns 'nb_rx', if you continue on error 'nb_rx' kept increased, so application will receive wrong number of packets.

Quoted 5 lines

Also packets assigned as "rx_pkts[nb_rx] = rxe;", this will cause gaps in the arrays.

You can either break on the error, or keep another variable to store number of received packets.

Thanks for pointing out this!
Yes, we will use another variable as index to go through all the nb_pkts and use nb_rx to store all valid pkts. In this way, there would be no gap in the rx_pkts array and the size can align with the final returned nb_rx. Thanks!

Quoted 7 lines

drivers/net/gve/gve_rx.c line 105

Quoted 4 lines
drivers/net/gve/gve_rx.c +3 -0
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               rxe->data_off = RTE_PKTMBUF_HEADROOM;

As far as I can see mbufs allocated using 'rte_pktmbuf_alloc()', if so no need to explicitly set the 'm->data_off'.

Will update this, thanks!

#149 lore raw

PATCH v4 2/9 net/gve/base: add logs and OS specific implementation

In reply to Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/base/gve_adminq.h   |   2 +
 drivers/net/gve/base/gve_desc.h     |   2 +
 drivers/net/gve/base/gve_desc_dqo.h |   2 +
 drivers/net/gve/base/gve_osdep.h    | 159 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_register.h |   2 +
 drivers/net/gve/gve_logs.h          |  14 +++
 6 files changed, 181 insertions(+)
 create mode 100644 drivers/net/gve/base/gve_osdep.h
 create mode 100644 drivers/net/gve/gve_logs.h
drivers/net/gve/base/gve_adminq.h +2 -0
index c7114cc883..cd496760ae 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/base/gve_desc.h +2 -0
index 358755b7e0..627b9120dc 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/base/gve_desc_dqo.h +2 -0
index 0d533abcd1..5031752b43 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/base/gve_osdep.h new file +159 -0

159 lines not on this page · raw message

drivers/net/gve/base/gve_register.h +2 -0
index b65f336be2..a599c1a08e 100644
@@ -7,6 +7,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
drivers/net/gve/gve_logs.h new file +14 -0
new file mode 100644
index 0000000000..0d02da46e1
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+		__func__, ## args)
+
+#endif
Signature · 1 line
2.34.1
#150 lore raw

Re: [PATCH v4 2/9] net/gve/base: add logs and OS specific implementation

Quoted 8 lines · Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

<...>

Quoted 6 lines
drivers/net/gve/gve_logs.h new file +3 -0
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */

drivers/net/gve/gve_logs.h line 13

Quoted 10 lines
drivers/net/gve/gve_logs.h new file +10 -0
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+       rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+               __func__, ## args)
+

What do you think to move 'gve_logs.h' to next patch, since this is extern 'gve_logtype_driver' which is not added yet. Although files are not compiled yet, logically I think it suits better to next patch.

#151 lore raw

RE: [PATCH v4 2/9] net/gve/base: add logs and OS specific implementation

Quoted 40 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:20
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v4 2/9] net/gve/base: add logs and OS specific implementation

Quoted 8 lines · Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

<...>

Quoted 13 lines
drivers/net/gve/gve_logs.h new file +10 -0
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
Quoted 8 lines

drivers/net/gve/gve_logs.h line 13

Quoted 3 lines
drivers/net/gve/gve_logs.h new file +3 -0
+       rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+               __func__, ## args)
+

What do you think to move 'gve_logs.h' to next patch, since this is extern 'gve_logtype_driver' which is not added yet. Although files are not compiled yet, logically I think it suits better to next patch.

Sure, make sense. Thanks!

#152 lore raw

PATCH v4 3/9 net/gve: add support for device initialization

In reply to Junfeng Guo

Support device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close

---
 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  10 +
 doc/guides/nics/gve.rst                |  69 +++++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve_adminq.c      |   1 +
 drivers/net/gve/gve_ethdev.c           | 371 +++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 225 +++++++++++++++
 drivers/net/gve/meson.build            |  14 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 11 files changed, 706 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
index 32ffdd1a61..474f41f0de 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
new file mode 100644
index 0000000000..44aec28009
@@ -0,0 +1,10 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +69 -0

69 lines not on this page · raw message

doc/guides/nics/index.rst +1 -0
index f48e9f815c..64388adad0 100644
@@ -29,6 +29,7 @@ Network Interface Controller Drivers
     enetfec
     enic
     fm10k
+    gve
     hinic
     hns3
     i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
index bb77a03e24..20d9dcaafd 100644
@@ -59,6 +59,11 @@ New Features
 
   * Added flow subscription support.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 
 Removed Items
 -------------
drivers/net/gve/base/gve_adminq.c +1 -0
index 95ec6b015c..072fbee539 100644
@@ -5,6 +5,7 @@
  * Copyright(C) 2022 Intel Corporation
  */
 
+#include "../gve_ethdev.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +371 -0

371 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h new file +225 -0

225 lines not on this page · raw message

drivers/net/gve/meson.build new file +14 -0
new file mode 100644
index 0000000000..d8ec64b3a3
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'base/gve_adminq.c',
+        'gve_ethdev.c',
+)
+includes += include_directories('base')
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index e35652fe63..f1a0ee2cef 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.34.1
#153 lore raw

Re: [PATCH v4 3/9] net/gve: add support for device initialization

Quoted 2 lines · Junfeng Guo

Support device init and the fowllowing devops:

s/fowllowing/following/

Quoted 4 lines

- dev_configure
- dev_start
- dev_stop
- dev_close

At this stage most of above are empty functions and not implemented yet, instead can you document in the commit log that build system and device initialization is added?

Quoted 4 lines

<...>

Quoted 5 lines
doc/guides/nics/gve.rst new file +2 -0
@@ -0,0 +1,69 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.

doc/guides/nics/gve.rst line 12

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support for
+Google Virtual Ethernet device.
+
+Please refer to https://cloud.google.com/compute/docs/networking/using-gvnic
+for the device description.
+

This seems another virtual interface, similar to iavf/virtio/idpf ...

Can you please briefly describe here the motivation to add yet another virtual interface, and again briefly describe cons/pros of the interface?

Quoted 42 lines
doc/guides/nics/gve.rst new file +42 -0
+The base code is under MIT license and based on GVE kernel driver v1.3.0.
+GVE base code files are:
+
+- gve_adminq.h
+- gve_adminq.c
+- gve_desc.h
+- gve_desc_dqo.h
+- gve_register.h
+- gve.h
+
+Please refer to https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0/google/gve
+to find the original base code.
+
+GVE has 3 queue formats:
+
+- GQI_QPL - GQI with queue page list
+- GQI_RDA - GQI with raw DMA addressing
+- DQO_RDA - DQO with raw DMA addressing
+
+GQI_QPL queue format is queue page list mode. Driver needs to allocate
+memory and register this memory as a Queue Page List (QPL) in hardware
+(Google Hypervisor/GVE Backend) first. Each queue has its own QPL.
+Then Tx needs to copy packets to QPL memory and put this packet's offset
+in the QPL memory into hardware descriptors so that hardware can get the
+packets data. And Rx needs to read descriptors of offset in QPL to get
+QPL address and copy packets from the address to get real packets data.
+
+GQI_RDA queue format works like usual NICs that driver can put packets'
+physical address into hardware descriptors.
+
+DQO_RDA queue format has submission and completion queue pair for each
+Tx/Rx queue. And similar as GQI_RDA, driver can put packets' physical
+address into hardware descriptors.
+
+Please refer to https://www.kernel.org/doc/html/latest/networking/device_drivers/ethernet/google/gve.html
+to get more information about GVE queue formats.
+
+Features and Limitations
+------------------------
+
+In this release, the GVE PMD provides the basic functionality of packet
+reception and transmission.

doc/guides/nics/gve.rst line 64

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+Supported features of the GVE PMD are:
+
+- Multiple queues for TX and RX
+- Receiver Side Scaling (RSS)
+- TSO offload
+- Port hardware statistics
+- Link state information
+- TX multi-segments (Scatter TX)
+- Tx UDP/TCP/SCTP Checksum
+

Can you build this list gradully, by adding relavent item in each patch that adds it? That way mapping with the code and documented feature becomes more obvious.

<...>

doc/guides/nics/gve.rst

Quoted 7 lines
doc/guides/nics/gve.rst new file +7 -0
+static int
+gve_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+
+       eth_dev->data->mac_addrs = NULL;
+

At this stage 'mac_addrs' is not freed, setting it to NULL prevents it to be freed.

<...>

doc/guides/nics/gve.rst

Quoted 4 lines
doc/guides/nics/gve.rst new file +4 -0
+
+static struct rte_pci_driver rte_gve_pmd = {
+       .id_table = pci_id_gve_map,
+       .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,

As far as I can see LSC interrupt is not supported, if that is correct should we drop the flag?

<...>

Quoted 58 lines
doc/guides/nics/gve.rst new file +58 -0
+
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */
+
+       /* Only valid for DQO_RDA queue format */
+       uint16_t tx_compq_size; /* tx completion queue size */
+       uint16_t rx_bufq_size; /* rx buff queue size */
+
+       uint64_t max_registered_pages;
+       uint64_t num_registered_pages; /* num pages registered with NIC */
+       uint16_t default_num_queues; /* default num queues to set up */
+       enum gve_queue_format queue_format; /* see enum gve_queue_format */
+       uint8_t enable_rsc;
+
+       uint16_t max_nb_txq;
+       uint16_t max_nb_rxq;
+       uint32_t num_ntfy_blks; /* spilt between TX and RX so must be even */
+
+       struct gve_registers __iomem *reg_bar0; /* see gve_register.h */
+       rte_be32_t __iomem *db_bar2; /* "array" of doorbells */
+       struct rte_pci_device *pci_dev;
+
+       /* Admin queue - see gve_adminq.h*/
+       union gve_adminq_command *adminq;
+       struct gve_dma_mem adminq_dma_mem;
+       uint32_t adminq_mask; /* masks prod_cnt to adminq size */
+       uint32_t adminq_prod_cnt; /* free-running count of AQ cmds executed */
+       uint32_t adminq_cmd_fail; /* free-running count of AQ cmds failed */
+       uint32_t adminq_timeouts; /* free-running count of AQ cmds timeouts */
+       /* free-running count of per AQ cmd executed */
+       uint32_t adminq_describe_device_cnt;
+       uint32_t adminq_cfg_device_resources_cnt;
+       uint32_t adminq_register_page_list_cnt;
+       uint32_t adminq_unregister_page_list_cnt;
+       uint32_t adminq_create_tx_queue_cnt;
+       uint32_t adminq_create_rx_queue_cnt;
+       uint32_t adminq_destroy_tx_queue_cnt;
+       uint32_t adminq_destroy_rx_queue_cnt;
+       uint32_t adminq_dcfg_device_resources_cnt;
+       uint32_t adminq_set_driver_parameter_cnt;
+       uint32_t adminq_report_stats_cnt;
+       uint32_t adminq_report_link_speed_cnt;
+       uint32_t adminq_get_ptype_map_cnt;
+
+       volatile uint32_t state_flags;
+
+       /* Gvnic device link speed from hypervisor. */
+       uint64_t link_speed;

doc/guides/nics/gve.rst

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+
+       uint16_t max_mtu;
+       struct rte_ether_addr dev_addr; /* mac address */
+
+       struct gve_queue_page_list *qpl;
+
+       struct gve_tx_queue **txqs;
+       struct gve_rx_queue **rxqs;
+};
+

Similar to previous comment, can you construct the headers by only adding used fields in that patch?

When batch copied an existing struct, it is very easy to add unused code and very hard to detect it. So if you only add what you need, that becomes easy to be sure all fields are used.

Also it makes more obvious which fields related to which feature.

<...>

drivers/net/gve/version.map line 3

Quoted 8 lines

new file mode 100644
index 0000000000..c2e0723b4c

drivers/net/gve/version.map new file +3 -0
@@ -0,0 +1,3 @@
+DPDK_22 {
+       local: *;
+};

it is 'DPDK_23' now, hopefully we will have an update to get rid of empty map files, feel free to review:
https://patches.dpdk.org/project/dpdk/list/?series=25002

#154 lore raw

RE: [PATCH v4 3/9] net/gve: add support for device initialization

Quoted 20 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:23
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v4 3/9] net/gve: add support for device initialization

Quoted 2 lines · Junfeng Guo

Support device init and the fowllowing devops:

s/fowllowing/following/

Quoted 8 lines
Quoted 4 lines

- dev_configure
- dev_start
- dev_stop
- dev_close

At this stage most of above are empty functions and not implemented yet, instead can you document in the commit log that build system and device initialization is added?

Agreed, will add this in the coming version. Thanks!

Quoted 23 lines
Quoted 5 lines

<...>

doc/guides/nics/gve.rst line 7

Quoted 10 lines
doc/guides/nics/gve.rst new file +7 -0
@@ -0,0 +1,69 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support

for

doc/guides/nics/gve.rst

Quoted 3 lines
doc/guides/nics/gve.rst new file +3 -0
+Google Virtual Ethernet device.
+
+Please refer to

https://cloud.google.com/compute/docs/networking/using-gvnic

Quoted 7 lines

doc/guides/nics/gve.rst line 12

Quoted 2 lines
doc/guides/nics/gve.rst new file +2 -0
+for the device description.
+

This seems another virtual interface, similar to iavf/virtio/idpf ...

Can you please briefly describe here the motivation to add yet another virtual interface, and again briefly describe cons/pros of the interface?

Sure. According to the official gVNIC description, gVNIC is an alternative to the virtio driver that can provide higher network bandwidths. Will add the brief descriptions of the cons/pros in the coming version. Thanks!

Quoted 61 lines

doc/guides/nics/gve.rst line 13

Quoted 2 lines
doc/guides/nics/gve.rst new file +1 -0
+The base code is under MIT license and based on GVE kernel driver

v1.3.0.

doc/guides/nics/gve.rst

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+GVE base code files are:
+
+- gve_adminq.h
+- gve_adminq.c
+- gve_desc.h
+- gve_desc_dqo.h
+- gve_register.h
+- gve.h
+
+Please refer to https://github.com/GoogleCloudPlatform/compute-

virtual-ethernet-linux/tree/v1.3.0/google/gve

doc/guides/nics/gve.rst

Quoted 9 lines
doc/guides/nics/gve.rst new file +9 -0
+to find the original base code.
+
+GVE has 3 queue formats:
+
+- GQI_QPL - GQI with queue page list
+- GQI_RDA - GQI with raw DMA addressing
+- DQO_RDA - DQO with raw DMA addressing
+
+GQI_QPL queue format is queue page list mode. Driver needs to

allocate

doc/guides/nics/gve.rst line 33

Quoted 1 line
doc/guides/nics/gve.rst new file +1 -0
+memory and register this memory as a Queue Page List (QPL) in

hardware

doc/guides/nics/gve.rst line 35

Quoted 2 lines
doc/guides/nics/gve.rst new file +2 -0
+(Google Hypervisor/GVE Backend) first. Each queue has its own QPL.
+Then Tx needs to copy packets to QPL memory and put this packet's

offset

doc/guides/nics/gve.rst line 36

Quoted 1 line
doc/guides/nics/gve.rst new file +1 -0
+in the QPL memory into hardware descriptors so that hardware can get

the

doc/guides/nics/gve.rst line 38

Quoted 2 lines
doc/guides/nics/gve.rst new file +2 -0
+packets data. And Rx needs to read descriptors of offset in QPL to get
+QPL address and copy packets from the address to get real packets

data.

doc/guides/nics/gve.rst line 40

Quoted 2 lines
doc/guides/nics/gve.rst new file +2 -0
+
+GQI_RDA queue format works like usual NICs that driver can put

packets'

doc/guides/nics/gve.rst

Quoted 3 lines
doc/guides/nics/gve.rst new file +3 -0
+physical address into hardware descriptors.
+
+DQO_RDA queue format has submission and completion queue pair for

each

doc/guides/nics/gve.rst

Quoted 4 lines
doc/guides/nics/gve.rst new file +4 -0
+Tx/Rx queue. And similar as GQI_RDA, driver can put packets' physical
+address into hardware descriptors.
+
+Please refer to

https://www.kernel.org/doc/html/latest/networking/device_drivers/ethe
rnet/google/gve.html

Quoted 14 lines
doc/guides/nics/gve.rst new file +14 -0
+to get more information about GVE queue formats.
+
+Features and Limitations
+------------------------
+
+In this release, the GVE PMD provides the basic functionality of packet
+reception and transmission.
+Supported features of the GVE PMD are:
+
+- Multiple queues for TX and RX
+- Receiver Side Scaling (RSS)
+- TSO offload
+- Port hardware statistics
+- Link state information
Quoted 8 lines

doc/guides/nics/gve.rst line 64

Quoted 3 lines
doc/guides/nics/gve.rst new file +3 -0
+- TX multi-segments (Scatter TX)
+- Tx UDP/TCP/SCTP Checksum
+

Can you build this list gradully, by adding relavent item in each patch that adds it?
That way mapping with the code and documented feature becomes more obvious.

Sure... Will add the items of this list gradually in the coming version. Thanks!

Quoted 13 lines

<...>

doc/guides/nics/gve.rst

Quoted 7 lines
doc/guides/nics/gve.rst new file +7 -0
+static int
+gve_dev_uninit(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+
+       eth_dev->data->mac_addrs = NULL;
+

At this stage 'mac_addrs' is not freed, setting it to NULL prevents it to be freed.

Thanks for the catch! Will improve this. Thanks!

Quoted 11 lines

<...>

doc/guides/nics/gve.rst

Quoted 4 lines
doc/guides/nics/gve.rst new file +4 -0
+
+static struct rte_pci_driver rte_gve_pmd = {
+       .id_table = pci_id_gve_map,
+       .drv_flags = RTE_PCI_DRV_NEED_MAPPING |

RTE_PCI_DRV_INTR_LSC,

As far as I can see LSC interrupt is not supported, if that is correct should we drop the flag?

Sure, seems this flag is not used in current GCP env. Will remove it in the coming version. Thanks!

Quoted 81 lines

<...>

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */

doc/guides/nics/gve.rst

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */
+
+       /* Only valid for DQO_RDA queue format */
+       uint16_t tx_compq_size; /* tx completion queue size */
+       uint16_t rx_bufq_size; /* rx buff queue size */
+
+       uint64_t max_registered_pages;
+       uint64_t num_registered_pages; /* num pages registered with NIC

*/

doc/guides/nics/gve.rst

Quoted 2 lines
doc/guides/nics/gve.rst new file +2 -0
+       uint16_t default_num_queues; /* default num queues to set up */
+       enum gve_queue_format queue_format; /* see enum

gve_queue_format */

doc/guides/nics/gve.rst

Quoted 5 lines
doc/guides/nics/gve.rst new file +5 -0
+       uint8_t enable_rsc;
+
+       uint16_t max_nb_txq;
+       uint16_t max_nb_rxq;
+       uint32_t num_ntfy_blks; /* spilt between TX and RX so must be

even */

doc/guides/nics/gve.rst

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+
+       struct gve_registers __iomem *reg_bar0; /* see gve_register.h */
+       rte_be32_t __iomem *db_bar2; /* "array" of doorbells */
+       struct rte_pci_device *pci_dev;
+
+       /* Admin queue - see gve_adminq.h*/
+       union gve_adminq_command *adminq;
+       struct gve_dma_mem adminq_dma_mem;
+       uint32_t adminq_mask; /* masks prod_cnt to adminq size */
+       uint32_t adminq_prod_cnt; /* free-running count of AQ cmds

executed */

Quoted 1 line

+ uint32_t adminq_cmd_fail; /* free-running count of AQ cmds failed

*/

Quoted 1 line

+ uint32_t adminq_timeouts; /* free-running count of AQ cmds

timeouts */

Quoted 19 lines
doc/guides/nics/gve.rst new file +19 -0
+       /* free-running count of per AQ cmd executed */
+       uint32_t adminq_describe_device_cnt;
+       uint32_t adminq_cfg_device_resources_cnt;
+       uint32_t adminq_register_page_list_cnt;
+       uint32_t adminq_unregister_page_list_cnt;
+       uint32_t adminq_create_tx_queue_cnt;
+       uint32_t adminq_create_rx_queue_cnt;
+       uint32_t adminq_destroy_tx_queue_cnt;
+       uint32_t adminq_destroy_rx_queue_cnt;
+       uint32_t adminq_dcfg_device_resources_cnt;
+       uint32_t adminq_set_driver_parameter_cnt;
+       uint32_t adminq_report_stats_cnt;
+       uint32_t adminq_report_link_speed_cnt;
+       uint32_t adminq_get_ptype_map_cnt;
+
+       volatile uint32_t state_flags;
+
+       /* Gvnic device link speed from hypervisor. */
+       uint64_t link_speed;

doc/guides/nics/gve.rst

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+
+       uint16_t max_mtu;
+       struct rte_ether_addr dev_addr; /* mac address */
+
+       struct gve_queue_page_list *qpl;
+
+       struct gve_tx_queue **txqs;
+       struct gve_rx_queue **rxqs;
+};
+

Similar to previous comment, can you construct the headers by only adding used fields in that patch?

Quoted 5 lines

When batch copied an existing struct, it is very easy to add unused code and very hard to detect it. So if you only add what you need, that becomes easy to be sure all fields are used.

Also it makes more obvious which fields related to which feature.

Sure... Will try best to construct the header structure items gradually. Thanks!

Quoted 15 lines

<...>

drivers/net/gve/version.map line 3

Quoted 8 lines

new file mode 100644
index 0000000000..c2e0723b4c

drivers/net/gve/version.map new file +3 -0
@@ -0,0 +1,3 @@
+DPDK_22 {
+       local: *;
+};

it is 'DPDK_23' now, hopefully we will have an update to get rid of empty map files, feel free to review:
https://patches.dpdk.org/project/dpdk/list/?series=25002

Sure, it looks much better. Thanks!

Quoted 1 line
#155 lore raw

PATCH v4 4/9 net/gve: add support for link update

In reply to Junfeng Guo

Support dev_ops link_update.

---
 doc/guides/nics/features/gve.ini |  2 ++
 drivers/net/gve/gve_ethdev.c     | 30 ++++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
index 44aec28009..d03e3ac89e 100644
@@ -4,6 +4,8 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Speed capabilities   = Y
+Link status          = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +30 -0
index df698c1b02..43112d901a 100644
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_DRV_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -70,6 +99,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.34.1
#156 lore raw

Re: [PATCH v4 4/9] net/gve: add support for link update

Quoted 15 lines · Junfeng Guo

Support dev_ops link_update.

---
  doc/guides/nics/features/gve.ini |  2 ++
  drivers/net/gve/gve_ethdev.c     | 30 ++++++++++++++++++++++++++++++
  2 files changed, 32 insertions(+)
doc/guides/nics/features/gve.ini +0 -0
index 44aec28009..d03e3ac89e 100644

doc/guides/nics/features/gve.ini line 7

Quoted 4 lines
doc/guides/nics/features/gve.ini +1 -0
@@ -4,6 +4,8 @@
  ; Refer to default.ini for the full list of available PMD features.
  ;
  [Features]
+Speed capabilities   = Y

'Speed capabilities' is when device reports supported speeds in 'rte_eth_dev_info_get()', so it shouldn't be in this patch.

Please check 'doc/guides/nics/features.rst' for more details.

#157 lore raw

RE: [PATCH v4 4/9] net/gve: add support for link update

Quoted 39 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:23
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 4/9] net/gve: add support for link update

Quoted 8 lines · Junfeng Guo

Support dev_ops link_update.

---
  doc/guides/nics/features/gve.ini |  2 ++
  drivers/net/gve/gve_ethdev.c     | 30

++++++++++++++++++++++++++++++

Review

Quoted 3 lines
  2 files changed, 32 insertions(+)
(quoted diff) +0 -0

b/doc/guides/nics/features/gve.ini

doc/guides/nics/features/gve.ini line 7

Quoted 8 lines

index 44aec28009..d03e3ac89e 100644

doc/guides/nics/features/gve.ini +1 -0
@@ -4,6 +4,8 @@
  ; Refer to default.ini for the full list of available PMD features.
  ;
  [Features]
+Speed capabilities   = Y

'Speed capabilities' is when device reports supported speeds in 'rte_eth_dev_info_get()', so it shouldn't be in this patch.

Quoted 1 line

Please check 'doc/guides/nics/features.rst' for more details.

Thanks for the comments!
Will check the features file and update accordingly.

#158 lore raw

PATCH v4 5/9 net/gve: add support for MTU setting

In reply to Junfeng Guo

Support dev_ops mtu_set.

---
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index d03e3ac89e..fbff0a5462 100644
@@ -6,6 +6,7 @@
 [Features]
 Speed capabilities   = Y
 Link status          = Y
+MTU update           = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +29 -0
index 43112d901a..5bcc9ab3a0 100644
@@ -94,12 +94,41 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	dev->data->dev_conf.rxmode.mtu = mtu + RTE_ETHER_HDR_LEN;
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.34.1
#159 lore raw

PATCH v4 6/9 net/gve: add support for queue operations

In reply to Junfeng Guo

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve_ethdev.c | 206 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h |  48 ++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 682 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +206 -0
index 5bcc9ab3a0..7a3695aec1 100644
@@ -28,12 +28,111 @@ gve_write_version(uint8_t *driver_version_register)
 	writeb('\n', driver_version_register);
 }
 
+static int
+gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pages)
+{
+	char z_name[RTE_MEMZONE_NAMESIZE];
+	struct gve_queue_page_list *qpl;
+	const struct rte_memzone *mz;
+	dma_addr_t page_bus;
+	uint32_t i;
+
+	if (priv->num_registered_pages + pages >
+	    priv->max_registered_pages) {
+		PMD_DRV_LOG(ERR, "Pages %" PRIu64 " > max registered pages %" PRIu64,
+			    priv->num_registered_pages + pages,
+			    priv->max_registered_pages);
+		return -EINVAL;
+	}
+	qpl = &priv->qpl[id];
+	snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_dev->device.name, id);
+	mz = rte_memzone_reserve_aligned(z_name, pages * PAGE_SIZE,
+					 rte_socket_id(),
+					 RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
+	if (mz == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name);
+		return -ENOMEM;
+	}
+	qpl->page_buses = rte_zmalloc("qpl page buses", pages * sizeof(dma_addr_t), 0);
+	if (qpl->page_buses == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page buses", id);
+		return -ENOMEM;
+	}
+	page_bus = mz->iova;
+	for (i = 0; i < pages; i++) {
+		qpl->page_buses[i] = page_bus;
+		page_bus += PAGE_SIZE;
+	}
+	qpl->id = id;
+	qpl->mz = mz;
+	qpl->num_entries = pages;
+
+	priv->num_registered_pages += pages;
+
+	return 0;
+}
+
+static void
+gve_free_qpls(struct gve_priv *priv)
+{
+	uint16_t nb_txqs = priv->max_nb_txq;
+	uint16_t nb_rxqs = priv->max_nb_rxq;
+	uint32_t i;
+
+	for (i = 0; i < nb_txqs + nb_rxqs; i++) {
+		if (priv->qpl[i].mz != NULL)
+			rte_memzone_free(priv->qpl[i].mz);
+		if (priv->qpl[i].page_buses != NULL)
+			rte_free(priv->qpl[i].page_buses);
+	}
+
+	if (priv->qpl != NULL)
+		rte_free(priv->qpl);
+}
+
 static int
 gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 {
 	return 0;
 }
 
+static int
+gve_refill_pages(struct gve_rx_queue *rxq)
+{
+	struct rte_mbuf *nmb;
+	uint16_t i;
+	int diag;
+
+	diag = rte_pktmbuf_alloc_bulk(rxq->mpool, &rxq->sw_ring[0], rxq->nb_rx_desc);
+	if (diag < 0) {
+		for (i = 0; i < rxq->nb_rx_desc - 1; i++) {
+			nmb = rte_pktmbuf_alloc(rxq->mpool);
+			if (!nmb)
+				break;
+			rxq->sw_ring[i] = nmb;
+		}
+		if (i < rxq->nb_rx_desc - 1)
+			return -ENOMEM;
+	}
+	rxq->nb_avail = 0;
+	rxq->next_avail = rxq->nb_rx_desc - 1;
+
+	for (i = 0; i < rxq->nb_rx_desc; i++) {
+		if (rxq->is_gqi_qpl) {
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(i * PAGE_SIZE);
+		} else {
+			if (i == rxq->nb_rx_desc - 1)
+				break;
+			nmb = rxq->sw_ring[i];
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(rte_mbuf_data_iova(nmb));
+		}
+	}
+
+	rte_write32(rte_cpu_to_be_32(rxq->next_avail), rxq->qrx_tail);
+
+	return 0;
+}
+
 static int
 gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 {
@@ -65,16 +164,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
+	uint16_t num_queues = dev->data->nb_tx_queues;
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
+	uint16_t i;
+	int err;
+
+	priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues;
+	err = gve_adminq_create_tx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u tx queues.", num_queues);
+		return err;
+	}
+	for (i = 0; i < num_queues; i++) {
+		txq = priv->txqs[i];
+		txq->qtx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(txq->qres->db_index)];
+		txq->qtx_head =
+		&priv->cnt_array[rte_be_to_cpu_32(txq->qres->counter_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), txq->ntfy_addr);
+	}
+
+	num_queues = dev->data->nb_rx_queues;
+	priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues;
+	err = gve_adminq_create_rx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u rx queues.", num_queues);
+		goto err_tx;
+	}
+	for (i = 0; i < num_queues; i++) {
+		rxq = priv->rxqs[i];
+		rxq->qrx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(rxq->qres->db_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), rxq->ntfy_addr);
+
+		err = gve_refill_pages(rxq);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to refill for RX");
+			goto err_rx;
+		}
+	}
+
 	dev->data->dev_started = 1;
 	gve_link_update(dev, 0);
 
 	return 0;
+
+err_rx:
+	gve_stop_rx_queues(dev);
+err_tx:
+	gve_stop_tx_queues(dev);
+	return err;
 }
 
 static int
 gve_dev_stop(struct rte_eth_dev *dev)
 {
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+
+	gve_stop_tx_queues(dev);
+	gve_stop_rx_queues(dev);
+
 	dev->data->dev_started = 0;
 
 	return 0;
@@ -83,7 +236,11 @@ gve_dev_stop(struct rte_eth_dev *dev)
 static int
 gve_dev_close(struct rte_eth_dev *dev)
 {
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
 	int err = 0;
+	uint16_t i;
 
 	if (dev->data->dev_started) {
 		err = gve_dev_stop(dev);
@@ -91,6 +248,21 @@ gve_dev_close(struct rte_eth_dev *dev)
 			PMD_DRV_LOG(ERR, "Failed to stop dev.");
 	}
 
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		txq = dev->data->tx_queues[i];
+		gve_tx_queue_release(txq);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		rxq = dev->data->rx_queues[i];
+		gve_rx_queue_release(rxq);
+	}
+
+	gve_free_qpls(priv);
+	rte_free(priv->adminq);
+	rte_free(priv->qpl);
+	rte_free(priv);
+
 	return err;
 }
 
@@ -127,6 +299,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.rx_queue_setup       = gve_rx_queue_setup,
+	.tx_queue_setup       = gve_tx_queue_setup,
 	.link_update          = gve_link_update,
 	.mtu_set              = gve_dev_mtu_set,
 };
@@ -264,7 +438,9 @@ gve_setup_device_resources(struct gve_priv *priv)
 static int
 gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 {
+	uint16_t pages;
 	int num_ntfy;
+	uint32_t i;
 	int err;
 
 	/* Set up the adminq */
@@ -315,10 +491,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 	PMD_DRV_LOG(INFO, "Max TX queues %d, Max RX queues %d",
 		    priv->max_nb_txq, priv->max_nb_rxq);
 
+	/* In GQI_QPL queue format:
+	 * Allocate queue page lists according to max queue number
+	 * tx qpl id should start from 0 while rx qpl id should start
+	 * from priv->max_nb_txq
+	 */
+	if (priv->queue_format == GVE_GQI_QPL_FORMAT) {
+		priv->qpl = rte_zmalloc("gve_qpl",
+					(priv->max_nb_txq + priv->max_nb_rxq) *
+					sizeof(struct gve_queue_page_list), 0);
+		if (priv->qpl == NULL) {
+			PMD_DRV_LOG(ERR, "Failed to alloc qpl.");
+			err = -ENOMEM;
+			goto free_adminq;
+		}
+
+		for (i = 0; i < priv->max_nb_txq + priv->max_nb_rxq; i++) {
+			if (i < priv->max_nb_txq)
+				pages = priv->tx_pages_per_qpl;
+			else
+				pages = priv->rx_data_slot_cnt;
+			err = gve_alloc_queue_page_list(priv, i, pages);
+			if (err != 0) {
+				PMD_DRV_LOG(ERR, "Failed to alloc qpl %u.", i);
+				goto err_qpl;
+			}
+		}
+	}
+
 setup_device:
 	err = gve_setup_device_resources(priv);
 	if (!err)
 		return 0;
+err_qpl:
+	gve_free_qpls(priv);
 free_adminq:
 	gve_adminq_free(priv);
 	return err;
drivers/net/gve/gve_ethdev.h +48 -0

88 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index d8ec64b3a3..af0010c01c 100644
@@ -9,6 +9,8 @@ endif
 
 sources = files(
         'base/gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
 includes += include_directories('base')
Signature · 1 line
2.34.1
#160 lore raw

PATCH v4 7/9 net/gve: add support for Rx/Tx

In reply to Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 doc/guides/nics/features/gve.ini |   2 +
 drivers/net/gve/gve_ethdev.c     |   5 +
 drivers/net/gve/gve_ethdev.h     |  16 ++
 drivers/net/gve/gve_rx.c         | 143 ++++++++++
 drivers/net/gve/gve_tx.c         | 455 +++++++++++++++++++++++++++++++
 5 files changed, 621 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
index fbff0a5462..38dc7024d6 100644
@@ -7,6 +7,8 @@
 Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+TSO                  = Y
+L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +5 -0
index 7a3695aec1..0aae447b9b 100644
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	}
+
 	eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
 	if (!eth_dev->data->mac_addrs) {
 		PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac address");
drivers/net/gve/gve_ethdev.h +16 -0
index b0391f7df5..502ba88dc3 100644
@@ -34,6 +34,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -270,4 +282,8 @@ void gve_stop_tx_queues(struct rte_eth_dev *dev);
 
 void gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +143 -0

149 lines not on this page · raw message

drivers/net/gve/gve_tx.c +455 -0

461 lines not on this page · raw message

Signature · 1 line
2.34.1
#161 lore raw

Re: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +5 -0
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
         if (err)
                 return err;
 
+       if (gve_is_gqi(priv)) {
+               eth_dev->rx_pkt_burst = gve_rx_burst;
+               eth_dev->tx_pkt_burst = gve_tx_burst;
+       }
+

What do you think to add a log here for 'else' case, to inform user why datapath is not working?

<...>

Quoted 17 lines
drivers/net/gve/gve_ethdev.c +17 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+
+       rxr = rxq->rx_desc_ring;
+
+       for (nb_rx = 0; nb_rx < nb_pkts; nb_rx++) {
+               rxd = &rxr[rx_id];
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
+                       break;
+

drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev)

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               rxe->data_off = RTE_PKTMBUF_HEADROOM;
+               if (rxq->is_gqi_qpl) {
+                       addr = (uint64_t)(rxq->qpl->mz->addr) + rx_id * PAGE_SIZE + GVE_RX_PAD;
+                       rte_memcpy((void *)((size_t)rxe->buf_addr + rxe->data_off),
+                                  (void *)(size_t)addr, len);

Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?

drivers/net/gve/gve_ethdev.c

Quoted 9 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               }
+               rxe->nb_segs = 1;
+               rxe->next = NULL;
+               rxe->pkt_len = len;
+               rxe->data_len = len;
+               rxe->port = rxq->port_id;
+               rxe->packet_type = 0;
+               rxe->ol_flags = 0;
+

As far as I can see 'sw_ring[]' filled using 'rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.

drivers/net/gve/gve_ethdev.c

Quoted 9 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               if (rxd->flags_seq & GVE_RXF_TCP)
+                       rxe->packet_type |= RTE_PTYPE_L4_TCP;
+               if (rxd->flags_seq & GVE_RXF_UDP)
+                       rxe->packet_type |= RTE_PTYPE_L4_UDP;
+               if (rxd->flags_seq & GVE_RXF_IPV4)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV4;
+               if (rxd->flags_seq & GVE_RXF_IPV6)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV6;
+

If you are setting packet_type, it is better to implement 'dev_supported_ptypes_get()' dev_ops too, to announce host which packet type parsin supporting. (+ dev_ptypes_set() dev_ops) And later driver can announce "Packet type parsing" feature in .ini file.

#162 lore raw

RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Quoted 28 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +0 -0
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
         if (err)
                 return err;
 
Quoted 8 lines

drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5 -0
+       if (gve_is_gqi(priv)) {
+               eth_dev->rx_pkt_burst = gve_rx_burst;
+               eth_dev->tx_pkt_burst = gve_tx_burst;
+       }
+

What do you think to add a log here for 'else' case, to inform user why datapath is not working?

Agreed, make sense!
Currently only one queue mode (i.e., qpl mode) is supported on the GCP env. Will add a log to inform this in the else case. Thanks!

Quoted 30 lines

<...>

drivers/net/gve/gve_ethdev.c

Quoted 2 lines
drivers/net/gve/gve_ethdev.c +2 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t

nb_pkts)

Quoted 13 lines
drivers/net/gve/gve_ethdev.c +13 -0
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+
+       rxr = rxq->rx_desc_ring;
+
+       for (nb_rx = 0; nb_rx < nb_pkts; nb_rx++) {
+               rxd = &rxr[rx_id];
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)

drivers/net/gve/gve_ethdev.c

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+                       break;
+
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               rxe->data_off = RTE_PKTMBUF_HEADROOM;
+               if (rxq->is_gqi_qpl) {
+                       addr = (uint64_t)(rxq->qpl->mz->addr) + rx_id * PAGE_SIZE

+ GVE_RX_PAD;

Quoted 5 lines
Quoted 3 lines

+ rte_memcpy((void *)((size_t)rxe->buf_addr + rxe-
data_off),
+ (void *)(size_t)addr, len);

Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?

Well, only qpl (queue page list) mode supported on the GCP env now. So the DMA may not be used in current case.

Quoted 11 lines

drivers/net/gve/gve_ethdev.c

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               }
+               rxe->nb_segs = 1;
+               rxe->next = NULL;
+               rxe->pkt_len = len;
+               rxe->data_len = len;
+               rxe->port = rxq->port_id;
+               rxe->packet_type = 0;
+               rxe->ol_flags = 0;
+
Quoted 3 lines

As far as I can see 'sw_ring[]' filled using 'rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.

Yes, some fields are already assigned at 'rte_pktmbuf_reset()'. Will remove the redundant ones in the coming version. Thanks!

Quoted 11 lines

drivers/net/gve/gve_ethdev.c

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               if (rxd->flags_seq & GVE_RXF_TCP)
+                       rxe->packet_type |= RTE_PTYPE_L4_TCP;
+               if (rxd->flags_seq & GVE_RXF_UDP)
+                       rxe->packet_type |= RTE_PTYPE_L4_UDP;
+               if (rxd->flags_seq & GVE_RXF_IPV4)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV4;
+               if (rxd->flags_seq & GVE_RXF_IPV6)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV6;
+
Quoted 5 lines

If you are setting packet_type, it is better to implement 'dev_supported_ptypes_get()' dev_ops too, to announce host which packet
type parsin supporting. (+ dev_ptypes_set() dev_ops)
And later driver can announce "Packet type parsing" feature in .ini file.

Well, on current GCP env, the APIs for supported ptypes get/set have not been exposed even in the base code. The only one in the base code is for the dqo mode (gve_adminq_get_ptype_map_dqo). But this also cannot be used on current GCP env. We can only implement this once they are supported and exposed at GCP. Thanks!

#163 lore raw

RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Hi

Quoted 47 lines

-----Original Message-----
From: Guo, Junfeng <[email protected]>
Sent: Sunday, October 9, 2022 10:15
To: Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Quoted 26 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +0 -0
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
         if (err)
                 return err;
 
Quoted 8 lines

drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5 -0
+       if (gve_is_gqi(priv)) {
+               eth_dev->rx_pkt_burst = gve_rx_burst;
+               eth_dev->tx_pkt_burst = gve_tx_burst;
+       }
+

What do you think to add a log here for 'else' case, to inform user why datapath is not working?

Quoted 3 lines

Agreed, make sense!
Currently only one queue mode (i.e., qpl mode) is supported on the GCP env. Will add a log to inform this in the else case. Thanks!

This explanation is not correct. Only QPL mode is supported in GCP now. This is env limitation but not related to the else code here. gve_is_gqi() includes two modes GQI_QPL and GQI_RDA. And both of these datapath is supported in rxtx.
GQI means its queue model is single queue model (txq for tx and rxq for rx). And there're 2 ways for this queue model QPL and RDA. QPL needs to copy packets from/to several reserved pages negotiated with backend. RDA is just like normal device and uses PA in descs.

The datapath not supported is DQO_RDA which uses different hardware so different queue model (split/double queue model). Tx will use txq and tx_completion_q and Rx will use rxq and rx_completion_q. This is not implemented in the datapath for now and will be implemented in the future.

So if you want to add comment here. Please say "DQO_RDA is not implemented and will be added in the future". Don't say it's not available in GCP env which is not the reason.

Quoted 32 lines
Quoted 32 lines

<...>

drivers/net/gve/gve_ethdev.c

Quoted 2 lines
drivers/net/gve/gve_ethdev.c +2 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t

nb_pkts)

Quoted 14 lines
drivers/net/gve/gve_ethdev.c +14 -0
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+
+       rxr = rxq->rx_desc_ring;
+
+       for (nb_rx = 0; nb_rx < nb_pkts; nb_rx++) {
+               rxd = &rxr[rx_id];
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
+                       break;

drivers/net/gve/gve_ethdev.c

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               rxe->data_off = RTE_PKTMBUF_HEADROOM;
+               if (rxq->is_gqi_qpl) {
+                       addr = (uint64_t)(rxq->qpl->mz->addr) +
+ rx_id * PAGE_SIZE

+ GVE_RX_PAD;

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

+ rte_memcpy((void *)((size_t)rxe->buf_addr +
+ rxe-
data_off),
+ (void *)(size_t)addr, len);

Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?

When queue model is gpi_qpl (this is negotiated and gotten using adminq with backend), the device needs to register a block of memory (called page list). And tx needs to copy the packets to this memory and rx will get packets from this area. Backend will be responsible for getting(tx)/giving(rx) packets from this memory to the device/line (We don't really know how backend does this).
Please refer to https://www.kernel.org/doc/html/v5.4/networking/device_drivers/google/gve.html. There's a bit more explanation about this queue format.

Quoted 3 lines

Well, only qpl (queue page list) mode supported on the GCP env now. So the DMA may not be used in current case.

And yes, it's because GCP doesn't support GQI_RDA for now so GQI_QPL has to be implemented. But even if GCP env supports RDA in the future, unless they completely remove QPL support, QPL is still needed. Because queue format/model is getting from backend through gve_adminq_describe_device(). You may just get the QPI version. The device can't really control which queue format to get.

Quoted 35 lines
Quoted 12 lines

drivers/net/gve/gve_ethdev.c

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               }
+               rxe->nb_segs = 1;
+               rxe->next = NULL;
+               rxe->pkt_len = len;
+               rxe->data_len = len;
+               rxe->port = rxq->port_id;
+               rxe->packet_type = 0;
+               rxe->ol_flags = 0;
+
Quoted 3 lines

As far as I can see 'sw_ring[]' filled using 'rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.

Yes, some fields are already assigned at 'rte_pktmbuf_reset()'. Will remove the redundant ones in the coming version. Thanks!

Quoted 11 lines

drivers/net/gve/gve_ethdev.c

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               if (rxd->flags_seq & GVE_RXF_TCP)
+                       rxe->packet_type |= RTE_PTYPE_L4_TCP;
+               if (rxd->flags_seq & GVE_RXF_UDP)
+                       rxe->packet_type |= RTE_PTYPE_L4_UDP;
+               if (rxd->flags_seq & GVE_RXF_IPV4)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV4;
+               if (rxd->flags_seq & GVE_RXF_IPV6)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV6;
+
Quoted 4 lines

If you are setting packet_type, it is better to implement 'dev_supported_ptypes_get()' dev_ops too, to announce host which packet type parsin supporting. (+ dev_ptypes_set() dev_ops) And later driver can announce "Packet type parsing" feature in .ini file.

Quoted 5 lines

Well, on current GCP env, the APIs for supported ptypes get/set have not been exposed even in the base code. The only one in the base code is for the dqo mode (gve_adminq_get_ptype_map_dqo). But this also cannot be used on current GCP env. We can only implement this once they are supported and exposed at GCP. Thanks!

You're mixing the concept again. GCP env only supports QPL is not an excuse.
The packet type is supported even in QPL. It's just very limited to L4_TCP/UDP and L3_IPV4/6. Ptypes_get is possible and it'll be RTE_PTYPE_L3_IPV4/6 and RTE_PTYPE_L4_UDP/TCP.
For DQO mode you mentioned, it'll be more flexible and have more support. I'm not sure what's your plan but it can be implemented whenever based on the plan not GCP env availability. The base code is there. It's just you may not be able to timely verify and debug it.

Ptype_set is not supported since the hardware doesn't support it (There's no such adminq).

#164 lore raw

RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Thanks Xiaoyun for helping explain, it helps a lot!

Quoted 81 lines

-----Original Message-----
From: Li, Xiaoyun <[email protected]>
Sent: Monday, October 10, 2022 17:40
To: Guo, Junfeng <[email protected]>; Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Hi

Quoted 19 lines

-----Original Message-----
From: Guo, Junfeng <[email protected]>
Sent: Sunday, October 9, 2022 10:15
To: Ferruh Yigit <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: RE: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Quoted 7 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson,

Bruce

Quoted 28 lines
Quoted 19 lines

<[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 7/9] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +0 -0
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
         if (err)
                 return err;
 
Quoted 8 lines

drivers/net/gve/gve_ethdev.c gve_dev_init(struct rte_eth_dev *eth_dev) · line 590

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5 -0
+       if (gve_is_gqi(priv)) {
+               eth_dev->rx_pkt_burst = gve_rx_burst;
+               eth_dev->tx_pkt_burst = gve_tx_burst;
+       }
+

What do you think to add a log here for 'else' case, to inform user why datapath is not working?

Quoted 2 lines

Agreed, make sense!
Currently only one queue mode (i.e., qpl mode) is supported on the GCP

env.

Quoted 1 line

Will add a log to inform this in the else case. Thanks!

This explanation is not correct. Only QPL mode is supported in GCP now. This is env limitation but not related to the else code here. gve_is_gqi() includes two modes GQI_QPL and GQI_RDA. And both of these datapath is supported in rxtx.
GQI means its queue model is single queue model (txq for tx and rxq for rx). And there're 2 ways for this queue model QPL and RDA. QPL needs to copy packets from/to several reserved pages negotiated with backend. RDA is just like normal device and uses PA in descs.

The datapath not supported is DQO_RDA which uses different hardware so different queue model (split/double queue model). Tx will use txq and tx_completion_q and Rx will use rxq and rx_completion_q. This is not implemented in the datapath for now and will be implemented in the future.

Quoted 3 lines

So if you want to add comment here. Please say "DQO_RDA is not implemented and will be added in the future". Don't say it's not available in GCP env which is not the reason.

Okay, will add this in the coming version. Thanks!

Quoted 54 lines
Quoted 33 lines
Quoted 33 lines

<...>

drivers/net/gve/gve_ethdev.c

Quoted 2 lines
drivers/net/gve/gve_ethdev.c +2 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t

nb_pkts)

Quoted 14 lines
drivers/net/gve/gve_ethdev.c +14 -0
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+
+       rxr = rxq->rx_desc_ring;
+
+       for (nb_rx = 0; nb_rx < nb_pkts; nb_rx++) {
+               rxd = &rxr[rx_id];
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
+                       break;

drivers/net/gve/gve_ethdev.c

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               rxe->data_off = RTE_PKTMBUF_HEADROOM;
+               if (rxq->is_gqi_qpl) {
+                       addr = (uint64_t)(rxq->qpl->mz->addr) +
+ rx_id * PAGE_SIZE

+ GVE_RX_PAD;

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

+ rte_memcpy((void *)((size_t)rxe->buf_addr +
+ rxe-
data_off),
+ (void *)(size_t)addr, len);

Why a 'memcpy' is needed? Can't it DMA to mbuf data buffer?

When queue model is gpi_qpl (this is negotiated and gotten using adminq with backend), the device needs to register a block of memory (called page list). And tx needs to copy the packets to this memory and rx will get packets from this area.
Backend will be responsible for getting(tx)/giving(rx) packets from this memory to the device/line (We don't really know how backend does this). Please refer to
https://www.kernel.org/doc/html/v5.4/networking/device_drivers/googl
e/gve.html. There's a bit more explanation about this queue format.

Quoted 3 lines

Well, only qpl (queue page list) mode supported on the GCP env now. So the DMA may not be used in current case.

Quoted 6 lines

And yes, it's because GCP doesn't support GQI_RDA for now so GQI_QPL has to be implemented. But even if GCP env supports RDA in the future, unless they completely remove QPL support, QPL is still needed. Because queue format/model is getting from backend through gve_adminq_describe_device(). You may just get the QPI version. The device can't really control which queue format to get.

Thanks for the explanation!

Quoted 55 lines
Quoted 31 lines
Quoted 13 lines

drivers/net/gve/gve_ethdev.c

Quoted 12 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               }
+               rxe->nb_segs = 1;
+               rxe->next = NULL;
+               rxe->pkt_len = len;
+               rxe->data_len = len;
+               rxe->port = rxq->port_id;
+               rxe->packet_type = 0;
+               rxe->ol_flags = 0;
+
Quoted 3 lines

As far as I can see 'sw_ring[]' filled using 'rte_pktmbuf_alloc_bulk()' API, which should reset mbuf fields to default values, so some of the assignment above can be redundant.

Yes, some fields are already assigned at 'rte_pktmbuf_reset()'. Will remove the redundant ones in the coming version. Thanks!

Quoted 11 lines
Quoted 10 lines
drivers/net/gve/gve_ethdev.c +9 -0
+               if (rxd->flags_seq & GVE_RXF_TCP)
+                       rxe->packet_type |= RTE_PTYPE_L4_TCP;
+               if (rxd->flags_seq & GVE_RXF_UDP)
+                       rxe->packet_type |= RTE_PTYPE_L4_UDP;
+               if (rxd->flags_seq & GVE_RXF_IPV4)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV4;
+               if (rxd->flags_seq & GVE_RXF_IPV6)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV6;
+
Quoted 6 lines
Quoted 4 lines

If you are setting packet_type, it is better to implement 'dev_supported_ptypes_get()' dev_ops too, to announce host which packet type parsin supporting. (+ dev_ptypes_set() dev_ops) And later driver can announce "Packet type parsing" feature in .ini file.

Well, on current GCP env, the APIs for supported ptypes get/set have

not

Quoted 1 line

been exposed even in the base code. The only one in the base code is

for

Quoted 1 line

the dqo mode (gve_adminq_get_ptype_map_dqo). But this also cannot

be

Quoted 2 lines

used on current GCP env. We can only implement this once they are supported and exposed at GCP. Thanks!

You're mixing the concept again. GCP env only supports QPL is not an excuse.
The packet type is supported even in QPL. It's just very limited to L4_TCP/UDP and L3_IPV4/6. Ptypes_get is possible and it'll be RTE_PTYPE_L3_IPV4/6 and RTE_PTYPE_L4_UDP/TCP.
For DQO mode you mentioned, it'll be more flexible and have more support. I'm not sure what's your plan but it can be implemented whenever based on the plan not GCP env availability. The base code is there. It's just you may not be able to timely verify and debug it.

Quoted 2 lines

Ptype_set is not supported since the hardware doesn't support it (There's no such adminq).

Okay... no much bandwidth to implement at this point. Maybe next release, thanks!

#165 lore raw

PATCH v4 8/9 net/gve: add support for dev info get and dev configure

In reply to Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 65 +++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +1 -0
index 38dc7024d6..cdc46b08a3 100644
@@ -8,6 +8,7 @@ Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
 TSO                  = Y
+RSS hash             = Y
 L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
drivers/net/gve/gve_ethdev.c +64 -1

84 lines not on this page · raw message

Signature · 1 line
2.34.1
#166 lore raw

Re: [PATCH v4 8/9] net/gve: add support for dev info get and dev configure

Quoted 4 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

I think better to have this before datapath patches (6/9 & 7/9), because this is more fundamental step.

#167 lore raw

RE: [PATCH v4 8/9] net/gve: add support for dev info get and dev configure

Quoted 16 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:25
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 8/9] net/gve: add support for dev info get and dev configure

Quoted 1 line · Junfeng Guo
Quoted 6 lines
Quoted 3 lines

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

I think better to have this before datapath patches (6/9 & 7/9), because this is more fundamental step.

Sure, make sense! Will reorder the patches in the coming version patchset. Thanks!

#168 lore raw

PATCH v4 9/9 net/gve: add support for stats

In reply to Junfeng Guo

Update stats add support of dev_ops stats_get/reset.

---
 doc/guides/nics/features/gve.ini |  2 +
 drivers/net/gve/gve_ethdev.c     | 71 ++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h     | 11 +++++
 drivers/net/gve/gve_rx.c         | 15 ++++++-
 drivers/net/gve/gve_tx.c         | 13 ++++++
 5 files changed, 110 insertions(+), 2 deletions(-)
doc/guides/nics/features/gve.ini +2 -0
index cdc46b08a3..180408aa80 100644
@@ -10,6 +10,8 @@ MTU update           = Y
 TSO                  = Y
 RSS hash             = Y
 L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +71 -0

83 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h +11 -0
index 502ba88dc3..7d9283f8fa 100644
@@ -76,6 +76,11 @@ struct gve_tx_queue {
 	struct gve_queue_page_list *qpl;
 	struct gve_tx_iovec *iov_ring;
 
+	/* Stats */
+	uint64_t errors;
+	uint64_t packets;
+	uint64_t bytes;
+
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -114,6 +119,12 @@ struct gve_rx_queue {
 	/* only valid for GQI_QPL queue format */
 	struct gve_queue_page_list *qpl;
 
+	/* stats */
+	uint64_t no_mbufs;
+	uint64_t errors;
+	uint64_t packets;
+	uint64_t bytes;
+
 	struct gve_priv *hw;
 	const struct rte_memzone *qres_mz;
 	struct gve_queue_resources *qres;
drivers/net/gve/gve_rx.c +13 -2
index 3634a2762f..6928afe96e 100644
@@ -26,8 +26,10 @@ gve_rx_refill(struct gve_rx_queue *rxq)
 					break;
 				rxq->sw_ring[idx + i] = nmb;
 			}
-			if (i != nb_alloc)
+			if (i != nb_alloc) {
+				rxq->no_mbufs += nb_alloc - i;
 				nb_alloc = i;
+			}
 		}
 		rxq->nb_avail -= nb_alloc;
 		next_avail += nb_alloc;
@@ -88,6 +90,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	uint16_t rx_id = rxq->rx_tail;
 	struct rte_mbuf *rxe;
 	uint16_t nb_rx, len;
+	uint64_t bytes = 0;
 	uint64_t addr;
 
 	rxr = rxq->rx_desc_ring;
@@ -97,8 +100,10 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
 			break;
 
-		if (rxd->flags_seq & GVE_RXF_ERR)
+		if (rxd->flags_seq & GVE_RXF_ERR) {
+			rxq->errors++;
 			continue;
+		}
 
 		len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
 		rxe = rxq->sw_ring[rx_id];
@@ -137,6 +142,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 			rx_id = 0;
 
 		rx_pkts[nb_rx] = rxe;
+		bytes += len;
 	}
 
 	rxq->nb_avail += nb_rx;
@@ -145,6 +151,11 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	if (rxq->nb_avail > rxq->free_thresh)
 		gve_rx_refill(rxq);
 
+	if (nb_rx) {
+		rxq->packets += nb_rx;
+		rxq->bytes += bytes;
+	}
+
 	return nb_rx;
 }
 
drivers/net/gve/gve_tx.c +13 -0
index d94b1186a4..0f2c3f8288 100644
@@ -260,6 +260,7 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t sw_id = txq->sw_tail;
 	uint16_t nb_used, i;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 	uint32_t hlen;
 
@@ -355,6 +356,8 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		txq->nb_free -= nb_used;
 		txq->sw_nb_free -= first->nb_segs;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
@@ -362,6 +365,10 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
 		txq->sw_tail = sw_id;
+
+		txq->errors += nb_pkts - nb_tx;
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
@@ -380,6 +387,7 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t nb_used, hlen, i;
 	uint64_t ol_flags, addr;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 
 	txr = txq->tx_desc_ring;
@@ -438,12 +446,17 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		txq->nb_free -= nb_used;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
 	if (nb_tx) {
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
+
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
Signature · 1 line
2.34.1
#169 lore raw

Re: [PATCH v4 9/9] net/gve: add support for stats

Quoted 18 lines · Junfeng Guo

Update stats add support of dev_ops stats_get/reset.

---
  doc/guides/nics/features/gve.ini |  2 +
  drivers/net/gve/gve_ethdev.c     | 71 ++++++++++++++++++++++++++++++++
  drivers/net/gve/gve_ethdev.h     | 11 +++++
  drivers/net/gve/gve_rx.c         | 15 ++++++-
  drivers/net/gve/gve_tx.c         | 13 ++++++
  5 files changed, 110 insertions(+), 2 deletions(-)
doc/guides/nics/features/gve.ini +0 -0
index cdc46b08a3..180408aa80 100644

doc/guides/nics/features/gve.ini MTU update = Y · line 14

Quoted 5 lines
doc/guides/nics/features/gve.ini +2 -0
@@ -10,6 +10,8 @@ MTU update           = Y
  TSO                  = Y
  RSS hash             = Y
  L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y

"stats per queue" is something else, agree that it is bad naming, please check features.rst file.

Quoted 18 lines
doc/guides/nics/features/gve.ini +0 -0
  Linux                = Y
  x86-32               = Y
  x86-64               = Y
drivers/net/gve/gve_ethdev.c +7 -0
index b9b8e51b02..cd474b8128 100644
@@ -328,6 +328,75 @@ gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
         return 0;
  }
 
+static int
+gve_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
+{
+       uint16_t i;
+
+       for (i = 0; i < dev->data->nb_tx_queues; i++) {
+               struct gve_tx_queue *txq = dev->data->tx_queues[i];

drivers/net/gve/gve_ethdev.c gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) · line 347

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+               if (txq == NULL)
+                       continue;
+
+               stats->opackets += txq->packets;
+               stats->obytes += txq->bytes;
+               stats->oerrors += txq->errors;
+
+               if (i < RTE_ETHDEV_QUEUE_STAT_CNTRS) {
+                       stats->q_opackets[i] = txq->packets;
+                       stats->q_obytes[i] = txq->bytes;

Queue stats update is moved to xstat [1], we are waiting existing PMDs to adopt it but for new drivers it is better to implement new method.

Can you please either drop queue stats completely, or implement it via xstats?

[1]
https://elixir.bootlin.com/dpdk/v22.07/source/doc/guides/rel_notes/deprecation.rst#L118

#170 lore raw

RE: [PATCH v4 9/9] net/gve: add support for stats

Quoted 30 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:26
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>
Subject: Re: [PATCH v4 9/9] net/gve: add support for stats

Quoted 8 lines · Junfeng Guo

Update stats add support of dev_ops stats_get/reset.

---
  doc/guides/nics/features/gve.ini |  2 +
  drivers/net/gve/gve_ethdev.c     | 71

++++++++++++++++++++++++++++++++

Review

Quoted 6 lines
  drivers/net/gve/gve_ethdev.h     | 11 +++++
  drivers/net/gve/gve_rx.c         | 15 ++++++-
  drivers/net/gve/gve_tx.c         | 13 ++++++
  5 files changed, 110 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/doc/guides/nics/features/gve.ini

Quoted 12 lines

doc/guides/nics/features/gve.ini MTU update = Y · line 14

Quoted 9 lines

index cdc46b08a3..180408aa80 100644

doc/guides/nics/features/gve.ini +2 -0
@@ -10,6 +10,8 @@ MTU update           = Y
  TSO                  = Y
  RSS hash             = Y
  L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y

"stats per queue" is something else, agree that it is bad naming, please check features.rst file.

Sure, will check the features file and update accordingly. Thanks!

Quoted 35 lines

Review

Quoted 5 lines
doc/guides/nics/features/gve.ini +0 -0
  Linux                = Y
  x86-32               = Y
  x86-64               = Y
(quoted diff) +0 -0

b/drivers/net/gve/gve_ethdev.c

drivers/net/gve/gve_ethdev.c gve_dev_info_get(struct rte_eth_dev *dev,

Quoted 4 lines

index b9b8e51b02..cd474b8128 100644

drivers/net/gve/gve_ethdev.c +0 -0
@@ -328,6 +328,75 @@ gve_dev_info_get(struct rte_eth_dev *dev,

struct rte_eth_dev_info *dev_info)

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +7 -0
         return 0;
  }
 
+static int
+gve_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
+{
+       uint16_t i;
+
+       for (i = 0; i < dev->data->nb_tx_queues; i++) {
+               struct gve_tx_queue *txq = dev->data->tx_queues[i];

drivers/net/gve/gve_ethdev.c gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) · line 347

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+               if (txq == NULL)
+                       continue;
+
+               stats->opackets += txq->packets;
+               stats->obytes += txq->bytes;
+               stats->oerrors += txq->errors;
+
+               if (i < RTE_ETHDEV_QUEUE_STAT_CNTRS) {
+                       stats->q_opackets[i] = txq->packets;
+                       stats->q_obytes[i] = txq->bytes;

Queue stats update is moved to xstat [1], we are waiting existing PMDs to adopt it but for new drivers it is better to implement new method.

Quoted 6 lines

Can you please either drop queue stats completely, or implement it via xstats?

[1]
https://elixir.bootlin.com/dpdk/v22.07/source/doc/guides/rel_notes/dep
recation.rst#L118

Sure, thanks for reminding this!
Seems that it would be better to drop the stats feature at this point since lack of time and bandwidth... But we can still measure the performance via other tools like TRex. And we can plan the implementation of the new method in the coming release. Thanks!

#171 lore raw

PATCH v3 2/9 net/gve: add logs and OS specific implementation

In reply to Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/gve_adminq.h   |   2 +
 drivers/net/gve/gve_desc.h     |   2 +
 drivers/net/gve/gve_desc_dqo.h |   2 +
 drivers/net/gve/gve_logs.h     |  14 +++
 drivers/net/gve/gve_osdep.h    | 159 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_register.h |   2 +
 6 files changed, 181 insertions(+)
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_osdep.h
drivers/net/gve/gve_adminq.h +2 -0
index c7114cc883..cd496760ae 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/gve_desc.h +2 -0
index 358755b7e0..627b9120dc 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/gve_desc_dqo.h +2 -0
index 0d533abcd1..5031752b43 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/gve_logs.h new file +14 -0
new file mode 100644
index 0000000000..0d02da46e1
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+		__func__, ## args)
+
+#endif
drivers/net/gve/gve_osdep.h new file +159 -0

159 lines not on this page · raw message

drivers/net/gve/gve_register.h +2 -0
index b65f336be2..a599c1a08e 100644
@@ -7,6 +7,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
Signature · 1 line
2.34.1
#172 lore raw

Re: [PATCH v3 2/9] net/gve: add logs and OS specific implementation

drivers/net/gve/gve_logs.h line 12

Quoted 4 lines · On Fri, 23 Sep 2022 17:38:22 +0800 Junfeng Guo <[email protected]> wrote:
drivers/net/gve/gve_logs.h new file +4 -0
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+		__func__, ## args)

Many of your existing log messages already have newline, so using this common definition will create double spaced log messages.

Please audit all usages and print one newline.

#173 lore raw

RE: [PATCH v3 2/9] net/gve: add logs and OS specific implementation

Quoted 21 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Saturday, September 24, 2022 03:01
To: Guo, Junfeng <[email protected]>
Cc: Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v3 2/9] net/gve: add logs and OS specific implementation

drivers/net/gve/gve_logs.h

Quoted 3 lines · On Fri, 23 Sep 2022 17:38:22 +0800 Junfeng Guo <[email protected]> wrote:
drivers/net/gve/gve_logs.h new file +3 -0
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n",

\

Quoted 6 lines

drivers/net/gve/gve_logs.h line 12

Quoted 1 line
drivers/net/gve/gve_logs.h new file +1 -0
+		__func__, ## args)

Many of your existing log messages already have newline, so using this common definition will create double spaced log messages.

Please audit all usages and print one newline.

Sure, will double check all the usages and update in the coming version. Thanks!

#174 lore raw

PATCH v3 3/9 net/gve: support device initialization

In reply to Junfeng Guo

Support device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close

---
 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  10 +
 doc/guides/nics/gve.rst                |  69 +++++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/gve_adminq.c           |   1 +
 drivers/net/gve/gve_ethdev.c           | 371 +++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 225 +++++++++++++++
 drivers/net/gve/meson.build            |  13 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 11 files changed, 705 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
index 32ffdd1a61..474f41f0de 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
new file mode 100644
index 0000000000..44aec28009
@@ -0,0 +1,10 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +69 -0

69 lines not on this page · raw message

doc/guides/nics/index.rst +1 -0
index f48e9f815c..64388adad0 100644
@@ -29,6 +29,7 @@ Network Interface Controller Drivers
     enetfec
     enic
     fm10k
+    gve
     hinic
     hns3
     i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
index bb77a03e24..20d9dcaafd 100644
@@ -59,6 +59,11 @@ New Features
 
   * Added flow subscription support.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 
 Removed Items
 -------------
drivers/net/gve/gve_adminq.c +1 -0
index 06f2ac2315..dcc573c077 100644
@@ -5,6 +5,7 @@
  * Copyright(C) 2022 Intel Corporation
  */
 
+#include "gve_ethdev.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +371 -0

371 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h new file +225 -0

225 lines not on this page · raw message

drivers/net/gve/meson.build new file +13 -0
new file mode 100644
index 0000000000..9a22cc9abe
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'gve_adminq.c',
+        'gve_ethdev.c',
+)
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index e35652fe63..f1a0ee2cef 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.34.1
#175 lore raw

PATCH v3 4/9 net/gve: add link update support

In reply to Junfeng Guo

Support dev_ops link_update.

---
 doc/guides/nics/features/gve.ini |  2 ++
 drivers/net/gve/gve_ethdev.c     | 30 ++++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
index 44aec28009..d03e3ac89e 100644
@@ -4,6 +4,8 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Speed capabilities   = Y
+Link status          = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +30 -0
index 4bb73b188d..7eb93d7366 100644
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_DRV_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -70,6 +99,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.34.1
#176 lore raw

PATCH v3 5/9 net/gve: add MTU set support

In reply to Junfeng Guo

Support dev_ops mtu_set.

---
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index d03e3ac89e..fbff0a5462 100644
@@ -6,6 +6,7 @@
 [Features]
 Speed capabilities   = Y
 Link status          = Y
+MTU update           = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +29 -0
index 7eb93d7366..c510938832 100644
@@ -94,12 +94,41 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	dev->data->dev_conf.rxmode.mtu = mtu + RTE_ETHER_HDR_LEN;
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.34.1
#177 lore raw

PATCH v3 6/9 net/gve: add queue operations

In reply to Junfeng Guo

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve_ethdev.c | 206 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h |  47 ++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 681 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +206 -0
index c510938832..72e7a78ace 100644
@@ -28,12 +28,111 @@ gve_write_version(uint8_t *driver_version_register)
 	writeb('\n', driver_version_register);
 }
 
+static int
+gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pages)
+{
+	char z_name[RTE_MEMZONE_NAMESIZE];
+	struct gve_queue_page_list *qpl;
+	const struct rte_memzone *mz;
+	dma_addr_t page_bus;
+	uint32_t i;
+
+	if (priv->num_registered_pages + pages >
+	    priv->max_registered_pages) {
+		PMD_DRV_LOG(ERR, "Pages %" PRIu64 " > max registered pages %" PRIu64,
+			    priv->num_registered_pages + pages,
+			    priv->max_registered_pages);
+		return -EINVAL;
+	}
+	qpl = &priv->qpl[id];
+	snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_dev->device.name, id);
+	mz = rte_memzone_reserve_aligned(z_name, pages * PAGE_SIZE,
+					 rte_socket_id(),
+					 RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
+	if (mz == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name);
+		return -ENOMEM;
+	}
+	qpl->page_buses = rte_zmalloc("qpl page buses", pages * sizeof(dma_addr_t), 0);
+	if (qpl->page_buses == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page buses", id);
+		return -ENOMEM;
+	}
+	page_bus = mz->iova;
+	for (i = 0; i < pages; i++) {
+		qpl->page_buses[i] = page_bus;
+		page_bus += PAGE_SIZE;
+	}
+	qpl->id = id;
+	qpl->mz = mz;
+	qpl->num_entries = pages;
+
+	priv->num_registered_pages += pages;
+
+	return 0;
+}
+
+static void
+gve_free_qpls(struct gve_priv *priv)
+{
+	uint16_t nb_txqs = priv->max_nb_txq;
+	uint16_t nb_rxqs = priv->max_nb_rxq;
+	uint32_t i;
+
+	for (i = 0; i < nb_txqs + nb_rxqs; i++) {
+		if (priv->qpl[i].mz != NULL)
+			rte_memzone_free(priv->qpl[i].mz);
+		if (priv->qpl[i].page_buses != NULL)
+			rte_free(priv->qpl[i].page_buses);
+	}
+
+	if (priv->qpl != NULL)
+		rte_free(priv->qpl);
+}
+
 static int
 gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 {
 	return 0;
 }
 
+static int
+gve_refill_pages(struct gve_rx_queue *rxq)
+{
+	struct rte_mbuf *nmb;
+	uint16_t i;
+	int diag;
+
+	diag = rte_pktmbuf_alloc_bulk(rxq->mpool, &rxq->sw_ring[0], rxq->nb_rx_desc);
+	if (diag < 0) {
+		for (i = 0; i < rxq->nb_rx_desc - 1; i++) {
+			nmb = rte_pktmbuf_alloc(rxq->mpool);
+			if (!nmb)
+				break;
+			rxq->sw_ring[i] = nmb;
+		}
+		if (i < rxq->nb_rx_desc - 1)
+			return -ENOMEM;
+	}
+	rxq->nb_avail = 0;
+	rxq->next_avail = rxq->nb_rx_desc - 1;
+
+	for (i = 0; i < rxq->nb_rx_desc; i++) {
+		if (rxq->is_gqi_qpl) {
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(i * PAGE_SIZE);
+		} else {
+			if (i == rxq->nb_rx_desc - 1)
+				break;
+			nmb = rxq->sw_ring[i];
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(rte_mbuf_data_iova(nmb));
+		}
+	}
+
+	rte_write32(rte_cpu_to_be_32(rxq->next_avail), rxq->qrx_tail);
+
+	return 0;
+}
+
 static int
 gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 {
@@ -65,16 +164,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
+	uint16_t num_queues = dev->data->nb_tx_queues;
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
+	uint16_t i;
+	int err;
+
+	priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues;
+	err = gve_adminq_create_tx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u tx queues.", num_queues);
+		return err;
+	}
+	for (i = 0; i < num_queues; i++) {
+		txq = priv->txqs[i];
+		txq->qtx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(txq->qres->db_index)];
+		txq->qtx_head =
+		&priv->cnt_array[rte_be_to_cpu_32(txq->qres->counter_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), txq->ntfy_addr);
+	}
+
+	num_queues = dev->data->nb_rx_queues;
+	priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues;
+	err = gve_adminq_create_rx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u rx queues.", num_queues);
+		goto err_tx;
+	}
+	for (i = 0; i < num_queues; i++) {
+		rxq = priv->rxqs[i];
+		rxq->qrx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(rxq->qres->db_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), rxq->ntfy_addr);
+
+		err = gve_refill_pages(rxq);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to refill for RX");
+			goto err_rx;
+		}
+	}
+
 	dev->data->dev_started = 1;
 	gve_link_update(dev, 0);
 
 	return 0;
+
+err_rx:
+	gve_stop_rx_queues(dev);
+err_tx:
+	gve_stop_tx_queues(dev);
+	return err;
 }
 
 static int
 gve_dev_stop(struct rte_eth_dev *dev)
 {
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+
+	gve_stop_tx_queues(dev);
+	gve_stop_rx_queues(dev);
+
 	dev->data->dev_started = 0;
 
 	return 0;
@@ -83,7 +236,11 @@ gve_dev_stop(struct rte_eth_dev *dev)
 static int
 gve_dev_close(struct rte_eth_dev *dev)
 {
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
 	int err = 0;
+	uint16_t i;
 
 	if (dev->data->dev_started) {
 		err = gve_dev_stop(dev);
@@ -91,6 +248,21 @@ gve_dev_close(struct rte_eth_dev *dev)
 			PMD_DRV_LOG(ERR, "Failed to stop dev.");
 	}
 
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		txq = dev->data->tx_queues[i];
+		gve_tx_queue_release(txq);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		rxq = dev->data->rx_queues[i];
+		gve_rx_queue_release(rxq);
+	}
+
+	gve_free_qpls(priv);
+	rte_free(priv->adminq);
+	rte_free(priv->qpl);
+	rte_free(priv);
+
 	return err;
 }
 
@@ -127,6 +299,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.rx_queue_setup       = gve_rx_queue_setup,
+	.tx_queue_setup       = gve_tx_queue_setup,
 	.link_update          = gve_link_update,
 	.mtu_set              = gve_dev_mtu_set,
 };
@@ -264,7 +438,9 @@ gve_setup_device_resources(struct gve_priv *priv)
 static int
 gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 {
+	uint16_t pages;
 	int num_ntfy;
+	uint32_t i;
 	int err;
 
 	/* Set up the adminq */
@@ -315,10 +491,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 	PMD_DRV_LOG(INFO, "Max TX queues %d, Max RX queues %d",
 		    priv->max_nb_txq, priv->max_nb_rxq);
 
+	/* In GQI_QPL queue format:
+	 * Allocate queue page lists according to max queue number
+	 * tx qpl id should start from 0 while rx qpl id should start
+	 * from priv->max_nb_txq
+	 */
+	if (priv->queue_format == GVE_GQI_QPL_FORMAT) {
+		priv->qpl = rte_zmalloc("gve_qpl",
+					(priv->max_nb_txq + priv->max_nb_rxq) *
+					sizeof(struct gve_queue_page_list), 0);
+		if (priv->qpl == NULL) {
+			PMD_DRV_LOG(ERR, "Failed to alloc qpl.");
+			err = -ENOMEM;
+			goto free_adminq;
+		}
+
+		for (i = 0; i < priv->max_nb_txq + priv->max_nb_rxq; i++) {
+			if (i < priv->max_nb_txq)
+				pages = priv->tx_pages_per_qpl;
+			else
+				pages = priv->rx_data_slot_cnt;
+			err = gve_alloc_queue_page_list(priv, i, pages);
+			if (err != 0) {
+				PMD_DRV_LOG(ERR, "Failed to alloc qpl %u.", i);
+				goto err_qpl;
+			}
+		}
+	}
+
 setup_device:
 	err = gve_setup_device_resources(priv);
 	if (!err)
 		return 0;
+err_qpl:
+	gve_free_qpls(priv);
 free_adminq:
 	gve_adminq_free(priv);
 	return err;
drivers/net/gve/gve_ethdev.h +47 -0

87 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index 9a22cc9abe..c4fd013ef2 100644
@@ -9,5 +9,7 @@ endif
 
 sources = files(
         'gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
Signature · 1 line
2.34.1
#178 lore raw

PATCH v3 7/9 net/gve: add Rx/Tx support

In reply to Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 doc/guides/nics/features/gve.ini |   2 +
 drivers/net/gve/gve_ethdev.c     |   5 +
 drivers/net/gve/gve_ethdev.h     |  16 ++
 drivers/net/gve/gve_rx.c         | 143 ++++++++++
 drivers/net/gve/gve_tx.c         | 455 +++++++++++++++++++++++++++++++
 5 files changed, 621 insertions(+)
doc/guides/nics/features/gve.ini +2 -0
index fbff0a5462..38dc7024d6 100644
@@ -7,6 +7,8 @@
 Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+TSO                  = Y
+L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +5 -0
index 72e7a78ace..dcf79ddb23 100644
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	}
+
 	eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
 	if (!eth_dev->data->mac_addrs) {
 		PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac address");
drivers/net/gve/gve_ethdev.h +16 -0
index 44e075e166..0624085517 100644
@@ -34,6 +34,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -269,4 +281,8 @@ void gve_stop_tx_queues(struct rte_eth_dev *dev);
 
 void gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +143 -0

149 lines not on this page · raw message

drivers/net/gve/gve_tx.c +455 -0

461 lines not on this page · raw message

Signature · 1 line
2.34.1
#179 lore raw

PATCH v3 8/9 net/gve: add support to get dev info and configure dev

In reply to Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 63 ++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index 38dc7024d6..cdc46b08a3 100644
@@ -8,6 +8,7 @@ Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
 TSO                  = Y
+RSS hash             = Y
 L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
drivers/net/gve/gve_ethdev.c +63 -0

81 lines not on this page · raw message

Signature · 1 line
2.34.1
#180 lore raw

PATCH v3 9/9 net/gve: add stats support

In reply to Junfeng Guo

Update stats add support of dev_ops stats_get/reset.

---
 doc/guides/nics/features/gve.ini |  2 +
 drivers/net/gve/gve_ethdev.c     | 71 ++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h     | 12 ++++++
 drivers/net/gve/gve_rx.c         | 15 ++++++-
 drivers/net/gve/gve_tx.c         | 13 ++++++
 5 files changed, 111 insertions(+), 2 deletions(-)
doc/guides/nics/features/gve.ini +2 -0
index cdc46b08a3..180408aa80 100644
@@ -10,6 +10,8 @@ MTU update           = Y
 TSO                  = Y
 RSS hash             = Y
 L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +71 -0

83 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h +12 -0
index 0624085517..a07c438b5d 100644
@@ -76,6 +76,11 @@ struct gve_tx_queue {
 	struct gve_queue_page_list *qpl;
 	struct gve_tx_iovec *iov_ring;
 
+	/* Stats */
+	uint64_t errors;
+	uint64_t packets;
+	uint64_t bytes;
+
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -114,6 +119,12 @@ struct gve_rx_queue {
 	/* only valid for GQI_QPL queue format */
 	struct gve_queue_page_list *qpl;
 
+	/* stats */
+	uint64_t no_mbufs;
+	uint64_t errors;
+	uint64_t packets;
+	uint64_t bytes;
+
 	struct gve_priv *hw;
 	const struct rte_memzone *qres_mz;
 	struct gve_queue_resources *qres;
@@ -125,6 +136,7 @@ struct gve_rx_queue {
 
 	/* Only valid for DQO_RDA queue format */
 	struct gve_rx_queue *bufq;
+
 	uint8_t is_gqi_qpl;
 };
 
drivers/net/gve/gve_rx.c +13 -2
index e29f979a4e..8d3ee35472 100644
@@ -26,8 +26,10 @@ gve_rx_refill(struct gve_rx_queue *rxq)
 					break;
 				rxq->sw_ring[idx + i] = nmb;
 			}
-			if (i != nb_alloc)
+			if (i != nb_alloc) {
+				rxq->no_mbufs += nb_alloc - i;
 				nb_alloc = i;
+			}
 		}
 		rxq->nb_avail -= nb_alloc;
 		next_avail += nb_alloc;
@@ -88,6 +90,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	uint16_t rx_id = rxq->rx_tail;
 	struct rte_mbuf *rxe;
 	uint16_t nb_rx, len;
+	uint64_t bytes = 0;
 	uint64_t addr;
 
 	rxr = rxq->rx_desc_ring;
@@ -97,8 +100,10 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
 			break;
 
-		if (rxd->flags_seq & GVE_RXF_ERR)
+		if (rxd->flags_seq & GVE_RXF_ERR) {
+			rxq->errors++;
 			continue;
+		}
 
 		len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
 		rxe = rxq->sw_ring[rx_id];
@@ -137,6 +142,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 			rx_id = 0;
 
 		rx_pkts[nb_rx] = rxe;
+		bytes += len;
 	}
 
 	rxq->nb_avail += nb_rx;
@@ -145,6 +151,11 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	if (rxq->nb_avail > rxq->free_thresh)
 		gve_rx_refill(rxq);
 
+	if (nb_rx) {
+		rxq->packets += nb_rx;
+		rxq->bytes += bytes;
+	}
+
 	return nb_rx;
 }
 
drivers/net/gve/gve_tx.c +13 -0
index 6196c29e24..81778840cf 100644
@@ -260,6 +260,7 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t sw_id = txq->sw_tail;
 	uint16_t nb_used, i;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 	uint32_t hlen;
 
@@ -355,6 +356,8 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		txq->nb_free -= nb_used;
 		txq->sw_nb_free -= first->nb_segs;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
@@ -362,6 +365,10 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
 		txq->sw_tail = sw_id;
+
+		txq->errors += nb_pkts - nb_tx;
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
@@ -380,6 +387,7 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t nb_used, hlen, i;
 	uint64_t ol_flags, addr;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 
 	txr = txq->tx_desc_ring;
@@ -438,12 +446,17 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		txq->nb_free -= nb_used;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
 	if (nb_tx) {
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
+
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
Signature · 1 line
2.34.1
#181 lore raw
In reply to Junfeng Guo
Quoted 17 lines · Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

Quoted 12 lines

Junfeng Guo (10):
net/gve: introduce GVE PMD base code
net/gve: add logs and OS specific implementation net/gve: support device initialization net/gve: add link update support
net/gve: add MTU set support
net/gve: add queue operations
net/gve: add Rx/Tx support
net/gve: add support to get dev info and configure dev net/gve: add stats support
doc: update documentation

Please check build error reported by CI:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/

I am also getting various build errors, even not able to reach patch by patch build stage where I expect some issues, can you please verify patch by patch build in next version?

#182 lore raw
Quoted 45 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:19
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH v2 00/10] introduce GVE PMD

Quoted 11 lines · Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

Quoted 2 lines

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/tree/v1.3.0

Quoted 4 lines

v2:
fix some CI check error.

Quoted 12 lines

Junfeng Guo (10):
net/gve: introduce GVE PMD base code
net/gve: add logs and OS specific implementation net/gve: support device initialization net/gve: add link update support
net/gve: add MTU set support
net/gve: add queue operations
net/gve: add Rx/Tx support
net/gve: add support to get dev info and configure dev net/gve: add stats support
doc: update documentation

Quoted 7 lines

Please check build error reported by CI:
https://patches.dpdk.org/project/dpdk/patch/20220829084127.934183-
[email protected]/

I am also getting various build errors, even not able to reach patch by patch build stage where I expect some issues, can you please verify patch by patch build in next version?

Sure, thanks for reminding!
The compile/build issues are being handled in process now. Thanks!

author
#183 lore raw

PATCH 2/10 net/gve: add logs and OS specific implementation

In reply to Xiaoyun Li

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/gve_adminq.h   |   2 +
 drivers/net/gve/gve_desc.h     |   2 +
 drivers/net/gve/gve_desc_dqo.h |   2 +
 drivers/net/gve/gve_logs.h     |  22 +++++
 drivers/net/gve/gve_osdep.h    | 149 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_register.h |   2 +
 6 files changed, 179 insertions(+)
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_osdep.h
drivers/net/gve/gve_adminq.h +2 -0
index c7114cc883..cd496760ae 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/gve_desc.h +2 -0
index b531669bc0..049792b43e 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/gve_desc_dqo.h +2 -0
index 0d533abcd1..5031752b43 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/gve_logs.h new file +22 -0
new file mode 100644
index 0000000000..a050253f59
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_init;
+extern int gve_logtype_driver;
+
+#define PMD_INIT_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_init, "%s(): " fmt "\n", \
+		__func__, ##args)
+
+#define PMD_DRV_LOG_RAW(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt, \
+		__func__, ## args)
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	PMD_DRV_LOG_RAW(level, fmt "\n", ## args)
+
+#endif
drivers/net/gve/gve_osdep.h new file +149 -0

149 lines not on this page · raw message

drivers/net/gve/gve_register.h +2 -0
index b65f336be2..a599c1a08e 100644
@@ -7,6 +7,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
Signature · 1 line
2.25.1
author
#184 lore raw

PATCH 3/10 net/gve: support device initialization

In reply to Xiaoyun Li

Support device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close

---
 drivers/net/gve/gve.h        | 249 +++++++++++++++++++++++
 drivers/net/gve/gve_adminq.c |   1 +
 drivers/net/gve/gve_ethdev.c | 375 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |  13 ++
 drivers/net/gve/version.map  |   3 +
 drivers/net/meson.build      |   1 +
 6 files changed, 642 insertions(+)
 create mode 100644 drivers/net/gve/gve.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
drivers/net/gve/gve.h new file +249 -0
new file mode 100644
index 0000000000..704c88983c
@@ -0,0 +1,249 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include <ethdev_driver.h>
+#include <ethdev_pci.h>
+#include <rte_ether.h>
+
+#include "gve_desc.h"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR	0
+#define GVE_DB_BAR	2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES	1024
+
+/* A list of pages registered with the device during setup and used by a queue
+ * as buffers
+ */
+struct gve_queue_page_list {
+	uint32_t id; /* unique id */
+	uint32_t num_entries;
+	dma_addr_t *page_buses; /* the dma addrs of the pages */
+	const struct rte_memzone *mz;
+};
+
+/* A TX desc ring entry */
+union gve_tx_desc {
+	struct gve_tx_pkt_desc pkt; /* first desc for a packet */
+	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
+};
+
+struct gve_tx_queue {
+	volatile union gve_tx_desc *tx_desc_ring;
+	const struct rte_memzone *mz;
+	uint64_t tx_ring_phys_addr;
+
+	uint16_t nb_tx_desc;
+
+	/* Only valid for DQO_QPL queue format */
+	struct gve_queue_page_list *qpl;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+
+	uint16_t ntfy_id;
+	volatile rte_be32_t *ntfy_addr;
+
+	struct gve_priv *hw;
+	const struct rte_memzone *qres_mz;
+	struct gve_queue_resources *qres;
+
+	/* Only valid for DQO_RDA queue format */
+	struct gve_tx_queue *complq;
+};
+
+struct gve_rx_queue {
+	volatile struct gve_rx_desc *rx_desc_ring;
+	volatile union gve_rx_data_slot *rx_data_ring;
+	const struct rte_memzone *mz;
+	const struct rte_memzone *data_mz;
+	uint64_t rx_ring_phys_addr;
+
+	uint16_t nb_rx_desc;
+
+	volatile rte_be32_t *ntfy_addr;
+
+	/* only valid for GQI_QPL queue format */
+	struct gve_queue_page_list *qpl;
+
+	struct gve_priv *hw;
+	const struct rte_memzone *qres_mz;
+	struct gve_queue_resources *qres;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+	uint16_t ntfy_id;
+	uint16_t rx_buf_len;
+
+	/* Only valid for DQO_RDA queue format */
+	struct gve_rx_queue *bufq;
+};
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+struct gve_priv {
+	struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+	const struct rte_memzone *irq_dbs_mz;
+	uint32_t mgmt_msix_idx;
+	rte_be32_t *cnt_array; /* array of num_event_counters */
+	const struct rte_memzone *cnt_array_mz;
+
+	uint16_t num_event_counters;
+	uint16_t tx_desc_cnt; /* txq size */
+	uint16_t rx_desc_cnt; /* rxq size */
+	uint16_t tx_pages_per_qpl; /* tx buffer length */
+	uint16_t rx_data_slot_cnt; /* rx buffer length */
+
+	/* Only valid for DQO_RDA queue format */
+	uint16_t tx_compq_size; /* tx completion queue size */
+	uint16_t rx_bufq_size; /* rx buff queue size */
+
+	uint64_t max_registered_pages;
+	uint64_t num_registered_pages; /* num pages registered with NIC */
+	uint16_t default_num_queues; /* default num queues to set up */
+	enum gve_queue_format queue_format; /* see enum gve_queue_format */
+	uint8_t enable_lsc;
+
+	uint16_t max_nb_txq;
+	uint16_t max_nb_rxq;
+	uint32_t num_ntfy_blks; /* spilt between TX and RX so must be even */
+
+	struct gve_registers __iomem *reg_bar0; /* see gve_register.h */
+	rte_be32_t __iomem *db_bar2; /* "array" of doorbells */
+	struct rte_pci_device *pci_dev;
+
+	/* Admin queue - see gve_adminq.h*/
+	union gve_adminq_command *adminq;
+	struct gve_dma_mem adminq_dma_mem;
+	uint32_t adminq_mask; /* masks prod_cnt to adminq size */
+	uint32_t adminq_prod_cnt; /* free-running count of AQ cmds executed */
+	uint32_t adminq_cmd_fail; /* free-running count of AQ cmds failed */
+	uint32_t adminq_timeouts; /* free-running count of AQ cmds timeouts */
+	/* free-running count of per AQ cmd executed */
+	uint32_t adminq_describe_device_cnt;
+	uint32_t adminq_cfg_device_resources_cnt;
+	uint32_t adminq_register_page_list_cnt;
+	uint32_t adminq_unregister_page_list_cnt;
+	uint32_t adminq_create_tx_queue_cnt;
+	uint32_t adminq_create_rx_queue_cnt;
+	uint32_t adminq_destroy_tx_queue_cnt;
+	uint32_t adminq_destroy_rx_queue_cnt;
+	uint32_t adminq_dcfg_device_resources_cnt;
+	uint32_t adminq_set_driver_parameter_cnt;
+	uint32_t adminq_report_stats_cnt;
+	uint32_t adminq_report_link_speed_cnt;
+	uint32_t adminq_get_ptype_map_cnt;
+
+	volatile uint32_t state_flags;
+
+	/* Gvnic device link speed from hypervisor. */
+	uint64_t link_speed;
+
+	uint16_t max_mtu;
+	struct rte_ether_addr dev_addr; /* mac address */
+
+	struct gve_queue_page_list *qpl;
+
+	struct gve_tx_queue **txqs;
+	struct gve_rx_queue **rxqs;
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+static inline bool gve_is_gqi(struct gve_priv *priv)
+{
+	return priv->queue_format == GVE_GQI_RDA_FORMAT ||
+		priv->queue_format == GVE_GQI_QPL_FORMAT;
+}
+
+static inline bool gve_get_admin_queue_ok(struct gve_priv *priv)
+{
+	return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+				       &priv->state_flags);
+}
+
+static inline void gve_set_admin_queue_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_set32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+			      &priv->state_flags);
+}
+
+static inline void gve_clear_admin_queue_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+				&priv->state_flags);
+}
+
+static inline bool gve_get_device_resources_ok(struct gve_priv *priv)
+{
+	return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+				       &priv->state_flags);
+}
+
+static inline void gve_set_device_resources_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+			      &priv->state_flags);
+}
+
+static inline void gve_clear_device_resources_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+				&priv->state_flags);
+}
+
+static inline bool gve_get_device_rings_ok(struct gve_priv *priv)
+{
+	return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+				       &priv->state_flags);
+}
+
+static inline void gve_set_device_rings_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+			      &priv->state_flags);
+}
+
+static inline void gve_clear_device_rings_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+				&priv->state_flags);
+}
+#endif /* _GVE_H_ */
drivers/net/gve/gve_adminq.c +1 -0
index 8a724f12c6..438ca2070e 100644
@@ -5,6 +5,7 @@
  * Copyright(C) 2022 Intel Corporation
  */
 
+#include "gve.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +375 -0

375 lines not on this page · raw message

drivers/net/gve/meson.build new file +13 -0
new file mode 100644
index 0000000000..9a22cc9abe
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'gve_adminq.c',
+        'gve_ethdev.c',
+)
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index e35652fe63..f1a0ee2cef 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.25.1
author
#185 lore raw

PATCH 4/10 net/gve: add link update support

In reply to Xiaoyun Li

Support dev_ops link_update.

---
 drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
drivers/net/gve/gve_ethdev.c +30 -0
index f10f273f7d..435115c047 100644
@@ -37,10 +37,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_INIT_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -73,6 +102,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.25.1
author
#186 lore raw

PATCH 5/10 net/gve: add MTU set support

In reply to Xiaoyun Li

Support dev_ops mtu_set.

---
 drivers/net/gve/gve_ethdev.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
drivers/net/gve/gve_ethdev.c +29 -0
index 435115c047..26b45fde6f 100644
@@ -97,12 +97,41 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	dev->data->dev_conf.rxmode.mtu = mtu + RTE_ETHER_HDR_LEN;
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.25.1
author
#187 lore raw

PATCH 6/10 net/gve: add queue operations

In reply to Xiaoyun Li

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve.h        |  52 +++++++++
 drivers/net/gve/gve_ethdev.c | 203 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 683 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve.h +52 -0
index 704c88983c..a53a852a5f 100644
@@ -23,6 +23,9 @@
 /* 1 for management, 1 for rx, 1 for tx */
 #define GVE_MIN_MSIX		3
 
+#define GVE_DEFAULT_RX_FREE_THRESH  512
+#define GVE_DEFAULT_TX_FREE_THRESH  256
+
 /* PTYPEs are always 10 bits. */
 #define GVE_NUM_PTYPES	1024
 
@@ -42,15 +45,35 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+struct gve_tx_iovec {
+	uint32_t iov_base; /* offset in fifo */
+	uint32_t iov_len;
+};
+
 struct gve_tx_queue {
 	volatile union gve_tx_desc *tx_desc_ring;
 	const struct rte_memzone *mz;
 	uint64_t tx_ring_phys_addr;
+	struct rte_mbuf **sw_ring;
+	volatile rte_be32_t *qtx_tail;
+	volatile rte_be32_t *qtx_head;
 
+	uint32_t tx_tail;
 	uint16_t nb_tx_desc;
+	uint16_t nb_free;
+	uint32_t next_to_clean;
+	uint16_t free_thresh;
 
 	/* Only valid for DQO_QPL queue format */
+	uint16_t sw_tail;
+	uint16_t sw_ntc;
+	uint16_t sw_nb_free;
+	uint32_t fifo_size;
+	uint32_t fifo_head;
+	uint32_t fifo_avail;
+	uint64_t fifo_base;
 	struct gve_queue_page_list *qpl;
+	struct gve_tx_iovec *iov_ring;
 
 	uint16_t port_id;
 	uint16_t queue_id;
@@ -64,6 +87,8 @@ struct gve_tx_queue {
 
 	/* Only valid for DQO_RDA queue format */
 	struct gve_tx_queue *complq;
+
+	uint8_t is_gqi_qpl;
 };
 
 struct gve_rx_queue {
@@ -72,9 +97,17 @@ struct gve_rx_queue {
 	const struct rte_memzone *mz;
 	const struct rte_memzone *data_mz;
 	uint64_t rx_ring_phys_addr;
+	struct rte_mbuf **sw_ring;
+	struct rte_mempool *mpool;
 
+	uint16_t rx_tail;
 	uint16_t nb_rx_desc;
+	uint16_t expected_seqno; /* the next expected seqno */
+	uint16_t free_thresh;
+	uint32_t next_avail;
+	uint32_t nb_avail;
 
+	volatile rte_be32_t *qrx_tail;
 	volatile rte_be32_t *ntfy_addr;
 
 	/* only valid for GQI_QPL queue format */
@@ -91,6 +124,8 @@ struct gve_rx_queue {
 
 	/* Only valid for DQO_RDA queue format */
 	struct gve_rx_queue *bufq;
+
+	uint8_t is_gqi_qpl;
 };
 
 struct gve_irq_db {
@@ -246,4 +281,21 @@ static inline void gve_clear_device_rings_ok(struct gve_priv *priv)
 	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
 				&priv->state_flags);
 }
+
+int
+gve_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_desc,
+		   unsigned int socket_id, const struct rte_eth_rxconf *conf,
+		   struct rte_mempool *pool);
+int
+gve_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_desc,
+		   unsigned int socket_id, const struct rte_eth_txconf *conf);
+
+void gve_tx_queue_release(void *txq);
+
+void gve_rx_queue_release(void *rxq);
+
+void gve_stop_tx_queues(struct rte_eth_dev *dev);
+
+void gve_stop_rx_queues(struct rte_eth_dev *dev);
+
 #endif /* _GVE_H_ */
drivers/net/gve/gve_ethdev.c +203 -0

267 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index 9a22cc9abe..c4fd013ef2 100644
@@ -9,5 +9,7 @@ endif
 
 sources = files(
         'gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
Signature · 1 line
2.25.1
author
#188 lore raw

PATCH 7/10 net/gve: add Rx/Tx support

In reply to Xiaoyun Li

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 drivers/net/gve/gve.h        |  17 ++
 drivers/net/gve/gve_ethdev.c |   5 +
 drivers/net/gve/gve_rx.c     | 143 +++++++++++
 drivers/net/gve/gve_tx.c     | 452 +++++++++++++++++++++++++++++++++++
 4 files changed, 617 insertions(+)
drivers/net/gve/gve.h +17 -0
index a53a852a5f..7f4d0e37f3 100644
@@ -25,6 +25,7 @@
 
 #define GVE_DEFAULT_RX_FREE_THRESH  512
 #define GVE_DEFAULT_TX_FREE_THRESH  256
+#define GVE_TX_MAX_FREE_SZ          512
 
 /* PTYPEs are always 10 bits. */
 #define GVE_NUM_PTYPES	1024
@@ -45,6 +46,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -298,4 +311,8 @@ void gve_stop_tx_queues(struct rte_eth_dev *dev);
 
 void gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_H_ */
drivers/net/gve/gve_ethdev.c +5 -0
index 5201398664..5ebe2c30ea 100644
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	}
+
 	eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
 	if (!eth_dev->data->mac_addrs) {
 		PMD_INIT_LOG(ERR, "Failed to allocate memory to store mac address");
drivers/net/gve/gve_rx.c +143 -0

149 lines not on this page · raw message

drivers/net/gve/gve_tx.c +452 -0

458 lines not on this page · raw message

Signature · 1 line
2.25.1
author
#189 lore raw

PATCH 8/10 net/gve: add support to get dev info and configure dev

In reply to Xiaoyun Li

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
 drivers/net/gve/gve.h        |  3 ++
 drivers/net/gve/gve_ethdev.c | 61 ++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
drivers/net/gve/gve.h +3 -0
index 7f4d0e37f3..004e0a75ca 100644
@@ -27,6 +27,9 @@
 #define GVE_DEFAULT_TX_FREE_THRESH  256
 #define GVE_TX_MAX_FREE_SZ          512
 
+#define GVE_MIN_BUF_SIZE	    1024
+#define GVE_MAX_RX_PKTLEN	    65535
+
 /* PTYPEs are always 10 bits. */
 #define GVE_NUM_PTYPES	1024
 
drivers/net/gve/gve_ethdev.c +61 -0

79 lines not on this page · raw message

Signature · 1 line
2.25.1
author
#190 lore raw

PATCH 9/10 net/gve: add stats support

In reply to Xiaoyun Li

Update stats add support of dev_ops stats_get/reset.

---
 drivers/net/gve/gve.h        | 10 ++++++
 drivers/net/gve/gve_ethdev.c | 69 ++++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_rx.c     | 15 ++++++--
 drivers/net/gve/gve_tx.c     | 12 +++++++
 4 files changed, 104 insertions(+), 2 deletions(-)
drivers/net/gve/gve.h +10 -0
index 004e0a75ca..e256a2bec2 100644
@@ -91,6 +91,10 @@ struct gve_tx_queue {
 	struct gve_queue_page_list *qpl;
 	struct gve_tx_iovec *iov_ring;
 
+	/* Stats */
+	uint64_t packets;
+	uint64_t bytes;
+
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -129,6 +133,12 @@ struct gve_rx_queue {
 	/* only valid for GQI_QPL queue format */
 	struct gve_queue_page_list *qpl;
 
+	/* stats */
+	uint64_t no_mbufs;
+	uint64_t errors;
+	uint64_t packets;
+	uint64_t bytes;
+
 	struct gve_priv *hw;
 	const struct rte_memzone *qres_mz;
 	struct gve_queue_resources *qres;
drivers/net/gve/gve_ethdev.c +69 -0

81 lines not on this page · raw message

drivers/net/gve/gve_rx.c +13 -2
index 8f560ae592..3a8a869980 100644
@@ -26,8 +26,10 @@ gve_rx_refill(struct gve_rx_queue *rxq)
 					break;
 				rxq->sw_ring[idx + i] = nmb;
 			}
-			if (i != nb_alloc)
+			if (i != nb_alloc) {
+				rxq->no_mbufs += nb_alloc - i;
 				nb_alloc = i;
+			}
 		}
 		rxq->nb_avail -= nb_alloc;
 		next_avail += nb_alloc;
@@ -88,6 +90,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	uint16_t rx_id = rxq->rx_tail;
 	struct rte_mbuf *rxe;
 	uint16_t nb_rx, len;
+	uint64_t bytes = 0;
 	uint64_t addr;
 
 	rxr = rxq->rx_desc_ring;
@@ -97,8 +100,10 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
 			break;
 
-		if (rxd->flags_seq & GVE_RXF_ERR)
+		if (rxd->flags_seq & GVE_RXF_ERR) {
+			rxq->errors++;
 			continue;
+		}
 
 		len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
 		rxe = rxq->sw_ring[rx_id];
@@ -137,6 +142,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 			rx_id = 0;
 
 		rx_pkts[nb_rx] = rxe;
+		bytes += len;
 	}
 
 	rxq->nb_avail += nb_rx;
@@ -145,6 +151,11 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	if (rxq->nb_avail > rxq->free_thresh)
 		gve_rx_refill(rxq);
 
+	if (nb_rx) {
+		rxq->packets += nb_rx;
+		rxq->bytes += bytes;
+	}
+
 	return nb_rx;
 }
 
drivers/net/gve/gve_tx.c +12 -0
index 2dc3411672..d99e6eb009 100644
@@ -260,6 +260,7 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t sw_id = txq->sw_tail;
 	uint16_t nb_used, i;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 	uint32_t hlen;
 
@@ -352,6 +353,8 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		txq->nb_free -= nb_used;
 		txq->sw_nb_free -= first->nb_segs;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
@@ -359,6 +362,9 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
 		txq->sw_tail = sw_id;
+
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
@@ -377,6 +383,7 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t nb_used, hlen, i;
 	uint64_t ol_flags, addr;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 
 	txr = txq->tx_desc_ring;
@@ -435,12 +442,17 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		txq->nb_free -= nb_used;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
 	if (nb_tx) {
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
+
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
Signature · 1 line
2.25.1
author
#191 lore raw

PATCH 10/10 doc: update documentation

In reply to Xiaoyun Li

Update documentation of GVE PMD and release note.
Add Junfeng Guo as GVE PMD maintainer since he'll work on GVE PMD in the future and maintain it and I won't be available for maintaining.

---
 MAINTAINERS                            |  6 +++
 doc/guides/nics/features/gve.ini       | 18 +++++++
 doc/guides/nics/gve.rst                | 65 ++++++++++++++++++++++++++
 doc/guides/rel_notes/release_22_11.rst |  4 ++
 4 files changed, 93 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
MAINTAINERS +6 -0
index 32ffdd1a61..474f41f0de 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +18 -0
new file mode 100644
index 0000000000..180408aa80
@@ -0,0 +1,18 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Speed capabilities   = Y
+Link status          = Y
+MTU update           = Y
+TSO                  = Y
+RSS hash             = Y
+L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +65 -0

65 lines not on this page · raw message

doc/guides/rel_notes/release_22_11.rst +4 -0
index 8c021cf050..6674f4cf6f 100644
@@ -55,6 +55,10 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added GVE net PMD**
+
+  Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  See the :doc:`../nics/gve` NIC guide for more details on this new driver.
 
 Removed Items
 -------------
Signature · 1 line
2.25.1