Skip to contentSkip to conversation

introduce GVE PMD

PATCH In review Xiaoyun Li 9 participants 192 messages Jul 29, 2022 – Oct 26, 2022 Page 1 of 2

Subsystems (2)
lore
Share & tools
lore

Xiaoyun Li

author
#0 lore raw

cover introduce GVE PMD

Introduce a new PMD for Google Virtual Ethernet (GVE).

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

Xiaoyun Li (10):
net/gve: introduce GVE PMD base code
net/gve: add logs and OS specific implementation net/gve: support device initialization net/gve: add link update support
net/gve: add MTU set support
net/gve: add queue operations
net/gve: add Rx/Tx support
net/gve: add support to get dev info and configure dev net/gve: add stats support
doc: update documentation

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  18 +
 doc/guides/nics/gve.rst                |  65 ++
 doc/guides/rel_notes/release_22_11.rst |   4 +
 drivers/net/gve/gve.h                  | 331 +++++++++
 drivers/net/gve/gve_adminq.c           | 926 +++++++++++++++++++++++++
 drivers/net/gve/gve_adminq.h           | 383 ++++++++++
 drivers/net/gve/gve_desc.h             | 139 ++++
 drivers/net/gve/gve_desc_dqo.h         | 256 +++++++
 drivers/net/gve/gve_ethdev.c           | 772 +++++++++++++++++++++
 drivers/net/gve/gve_logs.h             |  22 +
 drivers/net/gve/gve_osdep.h            | 149 ++++
 drivers/net/gve/gve_register.h         |  30 +
 drivers/net/gve/gve_rx.c               | 366 ++++++++++
 drivers/net/gve/gve_tx.c               | 678 ++++++++++++++++++
 drivers/net/gve/meson.build            |  15 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 18 files changed, 4164 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve.h
 create mode 100644 drivers/net/gve/gve_adminq.c
 create mode 100644 drivers/net/gve/gve_adminq.h
 create mode 100644 drivers/net/gve/gve_desc.h
 create mode 100644 drivers/net/gve/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_osdep.h
 create mode 100644 drivers/net/gve/gve_register.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/10 net/gve: introduce GVE PMD base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

---
 drivers/net/gve/gve_adminq.c   | 925 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_adminq.h   | 381 ++++++++++++++
 drivers/net/gve/gve_desc.h     | 137 +++++
 drivers/net/gve/gve_desc_dqo.h | 254 +++++++++
 drivers/net/gve/gve_register.h |  28 +
 5 files changed, 1725 insertions(+)
 create mode 100644 drivers/net/gve/gve_adminq.c
 create mode 100644 drivers/net/gve/gve_adminq.h
 create mode 100644 drivers/net/gve/gve_desc.h
 create mode 100644 drivers/net/gve/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/gve_register.h
drivers/net/gve/gve_adminq.c new file +925 -0
new file mode 100644
index 0000000000..8a724f12c6
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#include "gve_adminq.h"
+#include "gve_register.h"
+
+#define GVE_MAX_ADMINQ_RELEASE_CHECK	500
+#define GVE_ADMINQ_SLEEP_LEN		20
+#define GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK	100
+
+#define GVE_DEVICE_OPTION_ERROR_FMT "%s option error:\n" \
+"Expected: length=%d, feature_mask=%x.\n" \
+"Actual: length=%d, feature_mask=%x."
+
+#define GVE_DEVICE_OPTION_TOO_BIG_FMT "Length of %s option larger than expected. Possible older version of guest driver."
+
+static
+struct gve_device_option *gve_get_next_option(struct gve_device_descriptor *descriptor,
+					      struct gve_device_option *option)
+{
+	uintptr_t option_end, descriptor_end;
+
+	option_end = (uintptr_t)option + sizeof(*option) + be16_to_cpu(option->option_length);
+	descriptor_end = (uintptr_t)descriptor + be16_to_cpu(descriptor->total_length);
+
+	return option_end > descriptor_end ? NULL : (struct gve_device_option *)option_end;
+}
+
+static
+void gve_parse_device_option(struct gve_priv *priv,
+			     struct gve_device_option *option,
+			     struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+			     struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+			     struct gve_device_option_dqo_rda **dev_op_dqo_rda,
+			     struct gve_device_option_jumbo_frames **dev_op_jumbo_frames)
+{
+	u32 req_feat_mask = be32_to_cpu(option->required_features_mask);
+	u16 option_length = be16_to_cpu(option->option_length);
+	u16 option_id = be16_to_cpu(option->option_id);
+
+	/* If the length or feature mask doesn't match, continue without
+	 * enabling the feature.
+	 */
+	switch (option_id) {
+	case GVE_DEV_OPT_ID_GQI_RAW_ADDRESSING:
+		if (option_length != GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "Raw Addressing",
+				    GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING,
+				    GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		PMD_DRV_LOG(INFO, "Gqi raw addressing device option enabled.");
+		priv->queue_format = GVE_GQI_RDA_FORMAT;
+		break;
+	case GVE_DEV_OPT_ID_GQI_RDA:
+		if (option_length < sizeof(**dev_op_gqi_rda) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "GQI RDA", (int)sizeof(**dev_op_gqi_rda),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_gqi_rda)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT, "GQI RDA");
+		}
+		*dev_op_gqi_rda = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	case GVE_DEV_OPT_ID_GQI_QPL:
+		if (option_length < sizeof(**dev_op_gqi_qpl) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "GQI QPL", (int)sizeof(**dev_op_gqi_qpl),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_gqi_qpl)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT, "GQI QPL");
+		}
+		*dev_op_gqi_qpl = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	case GVE_DEV_OPT_ID_DQO_RDA:
+		if (option_length < sizeof(**dev_op_dqo_rda) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "DQO RDA", (int)sizeof(**dev_op_dqo_rda),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_dqo_rda)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT, "DQO RDA");
+		}
+		*dev_op_dqo_rda = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	case GVE_DEV_OPT_ID_JUMBO_FRAMES:
+		if (option_length < sizeof(**dev_op_jumbo_frames) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_JUMBO_FRAMES) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "Jumbo Frames",
+				    (int)sizeof(**dev_op_jumbo_frames),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_JUMBO_FRAMES,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_jumbo_frames)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT,
+				    "Jumbo Frames");
+		}
+		*dev_op_jumbo_frames = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	default:
+		/* If we don't recognize the option just continue
+		 * without doing anything.
+		 */
+		PMD_DRV_LOG(DEBUG, "Unrecognized device option 0x%hx not enabled.\n",
+			    option_id);
+	}
+}
+
+/* Process all device options for a given describe device call. */
+static int
+gve_process_device_options(struct gve_priv *priv,
+			   struct gve_device_descriptor *descriptor,
+			   struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+			   struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+			   struct gve_device_option_dqo_rda **dev_op_dqo_rda,
+			   struct gve_device_option_jumbo_frames **dev_op_jumbo_frames)
+{
+	const int num_options = be16_to_cpu(descriptor->num_device_options);
+	struct gve_device_option *dev_opt;
+	int i;
+
+	/* The options struct directly follows the device descriptor. */
+	dev_opt = RTE_PTR_ADD(descriptor, sizeof(*descriptor));
+	for (i = 0; i < num_options; i++) {
+		struct gve_device_option *next_opt;
+
+		next_opt = gve_get_next_option(descriptor, dev_opt);
+		if (!next_opt) {
+			PMD_DRV_LOG(ERR,
+				    "options exceed device_descriptor's total length.\n");
+			return -EINVAL;
+		}
+
+		gve_parse_device_option(priv, dev_opt,
+					dev_op_gqi_rda, dev_op_gqi_qpl,
+					dev_op_dqo_rda, dev_op_jumbo_frames);
+		dev_opt = next_opt;
+	}
+
+	return 0;
+}
+
+int gve_adminq_alloc(struct gve_priv *priv)
+{
+	priv->adminq = gve_alloc_dma_mem(&priv->adminq_dma_mem, PAGE_SIZE);
+	if (unlikely(!priv->adminq))
+		return -ENOMEM;
+
+	priv->adminq_mask = (PAGE_SIZE / sizeof(union gve_adminq_command)) - 1;
+	priv->adminq_prod_cnt = 0;
+	priv->adminq_cmd_fail = 0;
+	priv->adminq_timeouts = 0;
+	priv->adminq_describe_device_cnt = 0;
+	priv->adminq_cfg_device_resources_cnt = 0;
+	priv->adminq_register_page_list_cnt = 0;
+	priv->adminq_unregister_page_list_cnt = 0;
+	priv->adminq_create_tx_queue_cnt = 0;
+	priv->adminq_create_rx_queue_cnt = 0;
+	priv->adminq_destroy_tx_queue_cnt = 0;
+	priv->adminq_destroy_rx_queue_cnt = 0;
+	priv->adminq_dcfg_device_resources_cnt = 0;
+	priv->adminq_set_driver_parameter_cnt = 0;
+	priv->adminq_report_stats_cnt = 0;
+	priv->adminq_report_link_speed_cnt = 0;
+	priv->adminq_get_ptype_map_cnt = 0;
+
+	/* Setup Admin queue with the device */
+	iowrite32be(priv->adminq_dma_mem.pa / PAGE_SIZE,
+		    &priv->reg_bar0->adminq_pfn);
+
+	gve_set_admin_queue_ok(priv);
+	return 0;
+}
+
+void gve_adminq_release(struct gve_priv *priv)
+{
+	int i = 0;
+
+	/* Tell the device the adminq is leaving */
+	iowrite32be(0x0, &priv->reg_bar0->adminq_pfn);
+	while (ioread32be(&priv->reg_bar0->adminq_pfn)) {
+		/* If this is reached the device is unrecoverable and still
+		 * holding memory. Continue looping to avoid memory corruption,
+		 * but WARN so it is visible what is going on.
+		 */
+		if (i == GVE_MAX_ADMINQ_RELEASE_CHECK)
+			PMD_DRV_LOG(WARNING, "Unrecoverable platform error!");
+		i++;
+		msleep(GVE_ADMINQ_SLEEP_LEN);
+	}
+	gve_clear_device_rings_ok(priv);
+	gve_clear_device_resources_ok(priv);
+	gve_clear_admin_queue_ok(priv);
+}
+
+void gve_adminq_free(struct gve_priv *priv)
+{
+	if (!gve_get_admin_queue_ok(priv))
+		return;
+	gve_adminq_release(priv);
+	gve_free_dma_mem(&priv->adminq_dma_mem);
+	gve_clear_admin_queue_ok(priv);
+}
+
+static void gve_adminq_kick_cmd(struct gve_priv *priv, u32 prod_cnt)
+{
+	iowrite32be(prod_cnt, &priv->reg_bar0->adminq_doorbell);
+}
+
+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32 prod_cnt)
+{
+	int i;
+
+	for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++) {
+		if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+		    == prod_cnt)
+			return true;
+		msleep(GVE_ADMINQ_SLEEP_LEN);
+	}
+
+	return false;
+}
+
+static int gve_adminq_parse_err(struct gve_priv *priv, u32 status)
+{
+	if (status != GVE_ADMINQ_COMMAND_PASSED &&
+	    status != GVE_ADMINQ_COMMAND_UNSET) {
+		PMD_DRV_LOG(ERR, "AQ command failed with status %d", status);
+		priv->adminq_cmd_fail++;
+	}
+	switch (status) {
+	case GVE_ADMINQ_COMMAND_PASSED:
+		return 0;
+	case GVE_ADMINQ_COMMAND_UNSET:
+		PMD_DRV_LOG(ERR, "parse_aq_err: err and status both unset, this should not be possible.");
+		return -EINVAL;
+	case GVE_ADMINQ_COMMAND_ERROR_ABORTED:
+	case GVE_ADMINQ_COMMAND_ERROR_CANCELLED:
+	case GVE_ADMINQ_COMMAND_ERROR_DATALOSS:
+	case GVE_ADMINQ_COMMAND_ERROR_FAILED_PRECONDITION:
+	case GVE_ADMINQ_COMMAND_ERROR_UNAVAILABLE:
+		return -EAGAIN;
+	case GVE_ADMINQ_COMMAND_ERROR_ALREADY_EXISTS:
+	case GVE_ADMINQ_COMMAND_ERROR_INTERNAL_ERROR:
+	case GVE_ADMINQ_COMMAND_ERROR_INVALID_ARGUMENT:
+	case GVE_ADMINQ_COMMAND_ERROR_NOT_FOUND:
+	case GVE_ADMINQ_COMMAND_ERROR_OUT_OF_RANGE:
+	case GVE_ADMINQ_COMMAND_ERROR_UNKNOWN_ERROR:
+		return -EINVAL;
+	case GVE_ADMINQ_COMMAND_ERROR_DEADLINE_EXCEEDED:
+		return -ETIME;
+	case GVE_ADMINQ_COMMAND_ERROR_PERMISSION_DENIED:
+	case GVE_ADMINQ_COMMAND_ERROR_UNAUTHENTICATED:
+		return -EACCES;
+	case GVE_ADMINQ_COMMAND_ERROR_RESOURCE_EXHAUSTED:
+		return -ENOMEM;
+	case GVE_ADMINQ_COMMAND_ERROR_UNIMPLEMENTED:
+		return -ENOTSUP;
+	default:
+		PMD_DRV_LOG(ERR, "parse_aq_err: unknown status code %d",
+			    status);
+		return -EINVAL;
+	}
+}
+
+/* Flushes all AQ commands currently queued and waits for them to complete.
+ * If there are failures, it will return the first error.
+ */
+static int gve_adminq_kick_and_wait(struct gve_priv *priv)
+{
+	u32 tail, head;
+	u32 i;
+
+	tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+	head = priv->adminq_prod_cnt;
+
+	gve_adminq_kick_cmd(priv, head);
+	if (!gve_adminq_wait_for_cmd(priv, head)) {
+		PMD_DRV_LOG(ERR, "AQ commands timed out, need to reset AQ");
+		priv->adminq_timeouts++;
+		return -ENOTRECOVERABLE;
+	}
+
+	for (i = tail; i < head; i++) {
+		union gve_adminq_command *cmd;
+		u32 status, err;
+
+		cmd = &priv->adminq[i & priv->adminq_mask];
+		status = be32_to_cpu(READ_ONCE32(cmd->status));
+		err = gve_adminq_parse_err(priv, status);
+		if (err)
+			/* Return the first error if we failed. */
+			return err;
+	}
+
+	return 0;
+}
+
+/* This function is not threadsafe - the caller is responsible for any
+ * necessary locks.
+ */
+static int gve_adminq_issue_cmd(struct gve_priv *priv,
+				union gve_adminq_command *cmd_orig)
+{
+	union gve_adminq_command *cmd;
+	u32 opcode;
+	u32 tail;
+
+	tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+
+	/* Check if next command will overflow the buffer. */
+	if (((priv->adminq_prod_cnt + 1) & priv->adminq_mask) ==
+	    (tail & priv->adminq_mask)) {
+		int err;
+
+		/* Flush existing commands to make room. */
+		err = gve_adminq_kick_and_wait(priv);
+		if (err)
+			return err;
+
+		/* Retry. */
+		tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+		if (((priv->adminq_prod_cnt + 1) & priv->adminq_mask) ==
+		    (tail & priv->adminq_mask)) {
+			/* This should never happen. We just flushed the
+			 * command queue so there should be enough space.
+			 */
+			return -ENOMEM;
+		}
+	}
+
+	cmd = &priv->adminq[priv->adminq_prod_cnt & priv->adminq_mask];
+	priv->adminq_prod_cnt++;
+
+	memcpy(cmd, cmd_orig, sizeof(*cmd_orig));
+	opcode = be32_to_cpu(READ_ONCE32(cmd->opcode));
+
+	switch (opcode) {
+	case GVE_ADMINQ_DESCRIBE_DEVICE:
+		priv->adminq_describe_device_cnt++;
+		break;
+	case GVE_ADMINQ_CONFIGURE_DEVICE_RESOURCES:
+		priv->adminq_cfg_device_resources_cnt++;
+		break;
+	case GVE_ADMINQ_REGISTER_PAGE_LIST:
+		priv->adminq_register_page_list_cnt++;
+		break;
+	case GVE_ADMINQ_UNREGISTER_PAGE_LIST:
+		priv->adminq_unregister_page_list_cnt++;
+		break;
+	case GVE_ADMINQ_CREATE_TX_QUEUE:
+		priv->adminq_create_tx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_CREATE_RX_QUEUE:
+		priv->adminq_create_rx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_DESTROY_TX_QUEUE:
+		priv->adminq_destroy_tx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_DESTROY_RX_QUEUE:
+		priv->adminq_destroy_rx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_DECONFIGURE_DEVICE_RESOURCES:
+		priv->adminq_dcfg_device_resources_cnt++;
+		break;
+	case GVE_ADMINQ_SET_DRIVER_PARAMETER:
+		priv->adminq_set_driver_parameter_cnt++;
+		break;
+	case GVE_ADMINQ_REPORT_STATS:
+		priv->adminq_report_stats_cnt++;
+		break;
+	case GVE_ADMINQ_REPORT_LINK_SPEED:
+		priv->adminq_report_link_speed_cnt++;
+		break;
+	case GVE_ADMINQ_GET_PTYPE_MAP:
+		priv->adminq_get_ptype_map_cnt++;
+		break;
+	default:
+		PMD_DRV_LOG(ERR, "unknown AQ command opcode %d", opcode);
+	}
+
+	return 0;
+}
+
+/* This function is not threadsafe - the caller is responsible for any
+ * necessary locks.
+ * The caller is also responsible for making sure there are no commands
+ * waiting to be executed.
+ */
+static int gve_adminq_execute_cmd(struct gve_priv *priv,
+				  union gve_adminq_command *cmd_orig)
+{
+	u32 tail, head;
+	int err;
+
+	tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+	head = priv->adminq_prod_cnt;
+	if (tail != head)
+		/* This is not a valid path */
+		return -EINVAL;
+
+	err = gve_adminq_issue_cmd(priv, cmd_orig);
+	if (err)
+		return err;
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+/* The device specifies that the management vector can either be the first irq
+ * or the last irq. ntfy_blk_msix_base_idx indicates the first irq assigned to
+ * the ntfy blks. It if is 0 then the management vector is last, if it is 1 then
+ * the management vector is first.
+ *
+ * gve arranges the msix vectors so that the management vector is last.
+ */
+#define GVE_NTFY_BLK_BASE_MSIX_IDX	0
+int gve_adminq_configure_device_resources(struct gve_priv *priv,
+					  dma_addr_t counter_array_bus_addr,
+					  u32 num_counters,
+					  dma_addr_t db_array_bus_addr,
+					  u32 num_ntfy_blks)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_CONFIGURE_DEVICE_RESOURCES);
+	cmd.configure_device_resources =
+		(struct gve_adminq_configure_device_resources) {
+		.counter_array = cpu_to_be64(counter_array_bus_addr),
+		.num_counters = cpu_to_be32(num_counters),
+		.irq_db_addr = cpu_to_be64(db_array_bus_addr),
+		.num_irq_dbs = cpu_to_be32(num_ntfy_blks),
+		.irq_db_stride = cpu_to_be32(sizeof(*priv->irq_dbs)),
+		.ntfy_blk_msix_base_idx =
+					cpu_to_be32(GVE_NTFY_BLK_BASE_MSIX_IDX),
+		.queue_format = priv->queue_format,
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_deconfigure_device_resources(struct gve_priv *priv)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DECONFIGURE_DEVICE_RESOURCES);
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+static int gve_adminq_create_tx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	struct gve_tx_queue *txq = priv->txqs[queue_index];
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_CREATE_TX_QUEUE);
+	cmd.create_tx_queue = (struct gve_adminq_create_tx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+		.queue_resources_addr =
+			cpu_to_be64(txq->qres_mz->iova),
+		.tx_ring_addr = cpu_to_be64(txq->tx_ring_phys_addr),
+		.ntfy_id = cpu_to_be32(txq->ntfy_id),
+	};
+
+	if (gve_is_gqi(priv)) {
+		u32 qpl_id = priv->queue_format == GVE_GQI_RDA_FORMAT ?
+			GVE_RAW_ADDRESSING_QPL_ID : txq->qpl->id;
+
+		cmd.create_tx_queue.queue_page_list_id = cpu_to_be32(qpl_id);
+	} else {
+		cmd.create_tx_queue.tx_ring_size =
+			cpu_to_be16(txq->nb_tx_desc);
+		cmd.create_tx_queue.tx_comp_ring_addr =
+			cpu_to_be64(txq->complq->tx_ring_phys_addr);
+		cmd.create_tx_queue.tx_comp_ring_size =
+			cpu_to_be16(priv->tx_compq_size);
+	}
+
+	return gve_adminq_issue_cmd(priv, &cmd);
+}
+
+int gve_adminq_create_tx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_create_tx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_adminq_create_rx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	struct gve_rx_queue *rxq = priv->rxqs[queue_index];
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_CREATE_RX_QUEUE);
+	cmd.create_rx_queue = (struct gve_adminq_create_rx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+		.ntfy_id = cpu_to_be32(rxq->ntfy_id),
+		.queue_resources_addr = cpu_to_be64(rxq->qres_mz->iova),
+	};
+
+	if (gve_is_gqi(priv)) {
+		u32 qpl_id = priv->queue_format == GVE_GQI_RDA_FORMAT ?
+			GVE_RAW_ADDRESSING_QPL_ID : rxq->qpl->id;
+
+		cmd.create_rx_queue.rx_desc_ring_addr =
+			cpu_to_be64(rxq->mz->iova),
+		cmd.create_rx_queue.rx_data_ring_addr =
+			cpu_to_be64(rxq->data_mz->iova),
+		cmd.create_rx_queue.index = cpu_to_be32(queue_index);
+		cmd.create_rx_queue.queue_page_list_id = cpu_to_be32(qpl_id);
+		cmd.create_rx_queue.packet_buffer_size = cpu_to_be16(rxq->rx_buf_len);
+	} else {
+		cmd.create_rx_queue.rx_ring_size =
+			cpu_to_be16(priv->rx_desc_cnt);
+		cmd.create_rx_queue.rx_desc_ring_addr =
+			cpu_to_be64(rxq->rx_ring_phys_addr);
+		cmd.create_rx_queue.rx_data_ring_addr =
+			cpu_to_be64(rxq->bufq->rx_ring_phys_addr);
+		cmd.create_rx_queue.packet_buffer_size =
+			cpu_to_be16(rxq->rx_buf_len);
+		cmd.create_rx_queue.rx_buff_ring_size =
+			cpu_to_be16(priv->rx_bufq_size);
+		cmd.create_rx_queue.enable_rsc = !!(priv->enable_lsc);
+	}
+
+	return gve_adminq_issue_cmd(priv, &cmd);
+}
+
+int gve_adminq_create_rx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_create_rx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_adminq_destroy_tx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	union gve_adminq_command cmd;
+	int err;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESTROY_TX_QUEUE);
+	cmd.destroy_tx_queue = (struct gve_adminq_destroy_tx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+	};
+
+	err = gve_adminq_issue_cmd(priv, &cmd);
+	if (err)
+		return err;
+
+	return 0;
+}
+
+int gve_adminq_destroy_tx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_destroy_tx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_adminq_destroy_rx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	union gve_adminq_command cmd;
+	int err;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESTROY_RX_QUEUE);
+	cmd.destroy_rx_queue = (struct gve_adminq_destroy_rx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+	};
+
+	err = gve_adminq_issue_cmd(priv, &cmd);
+	if (err)
+		return err;
+
+	return 0;
+}
+
+int gve_adminq_destroy_rx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_destroy_rx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_set_desc_cnt(struct gve_priv *priv,
+			    struct gve_device_descriptor *descriptor)
+{
+	priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries);
+	if (priv->tx_desc_cnt * sizeof(priv->txqs[0]->tx_desc_ring[0])
+	    < PAGE_SIZE) {
+		PMD_DRV_LOG(ERR, "Tx desc count %d too low", priv->tx_desc_cnt);
+		return -EINVAL;
+	}
+	priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries);
+	if (priv->rx_desc_cnt * sizeof(priv->rxqs[0]->rx_desc_ring[0])
+	    < PAGE_SIZE) {
+		PMD_DRV_LOG(ERR, "Rx desc count %d too low", priv->rx_desc_cnt);
+		return -EINVAL;
+	}
+	return 0;
+}
+
+static int
+gve_set_desc_cnt_dqo(struct gve_priv *priv,
+		     const struct gve_device_descriptor *descriptor,
+		     const struct gve_device_option_dqo_rda *dev_op_dqo_rda)
+{
+	priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries);
+	priv->tx_compq_size = be16_to_cpu(dev_op_dqo_rda->tx_comp_ring_entries);
+	priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries);
+	priv->rx_bufq_size = be16_to_cpu(dev_op_dqo_rda->rx_buff_ring_entries);
+
+	return 0;
+}
+
+static void gve_enable_supported_features(struct gve_priv *priv,
+					  u32 supported_features_mask,
+					  const struct gve_device_option_jumbo_frames
+						  *dev_op_jumbo_frames)
+{
+	/* Before control reaches this point, the page-size-capped max MTU from
+	 * the gve_device_descriptor field has already been stored in
+	 * priv->dev->max_mtu. We overwrite it with the true max MTU below.
+	 */
+	if (dev_op_jumbo_frames &&
+	    (supported_features_mask & GVE_SUP_JUMBO_FRAMES_MASK)) {
+		PMD_DRV_LOG(INFO, "JUMBO FRAMES device option enabled.");
+		priv->max_mtu = be16_to_cpu(dev_op_jumbo_frames->max_mtu);
+	}
+}
+
+int gve_adminq_describe_device(struct gve_priv *priv)
+{
+	struct gve_device_option_jumbo_frames *dev_op_jumbo_frames = NULL;
+	struct gve_device_option_gqi_rda *dev_op_gqi_rda = NULL;
+	struct gve_device_option_gqi_qpl *dev_op_gqi_qpl = NULL;
+	struct gve_device_option_dqo_rda *dev_op_dqo_rda = NULL;
+	struct gve_device_descriptor *descriptor;
+	struct gve_dma_mem descriptor_dma_mem;
+	u32 supported_features_mask = 0;
+	union gve_adminq_command cmd;
+	int err = 0;
+	u8 *mac;
+	u16 mtu;
+
+	memset(&cmd, 0, sizeof(cmd));
+	descriptor = gve_alloc_dma_mem(&descriptor_dma_mem, PAGE_SIZE);
+	if (!descriptor)
+		return -ENOMEM;
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESCRIBE_DEVICE);
+	cmd.describe_device.device_descriptor_addr =
+					cpu_to_be64(descriptor_dma_mem.pa);
+	cmd.describe_device.device_descriptor_version =
+			cpu_to_be32(GVE_ADMINQ_DEVICE_DESCRIPTOR_VERSION);
+	cmd.describe_device.available_length = cpu_to_be32(PAGE_SIZE);
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	if (err)
+		goto free_device_descriptor;
+
+	err = gve_process_device_options(priv, descriptor, &dev_op_gqi_rda,
+					 &dev_op_gqi_qpl, &dev_op_dqo_rda,
+					 &dev_op_jumbo_frames);
+	if (err)
+		goto free_device_descriptor;
+
+	/* If the GQI_RAW_ADDRESSING option is not enabled and the queue format
+	 * is not set to GqiRda, choose the queue format in a priority order:
+	 * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default.
+	 */
+	if (dev_op_dqo_rda) {
+		priv->queue_format = GVE_DQO_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with DQO RDA queue format.");
+		supported_features_mask =
+			be32_to_cpu(dev_op_dqo_rda->supported_features_mask);
+	} else if (dev_op_gqi_rda) {
+		priv->queue_format = GVE_GQI_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+		supported_features_mask =
+			be32_to_cpu(dev_op_gqi_rda->supported_features_mask);
+	} else if (priv->queue_format == GVE_GQI_RDA_FORMAT) {
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+	} else {
+		priv->queue_format = GVE_GQI_QPL_FORMAT;
+		if (dev_op_gqi_qpl)
+			supported_features_mask =
+				be32_to_cpu(dev_op_gqi_qpl->supported_features_mask);
+		PMD_DRV_LOG(INFO, "Driver is running with GQI QPL queue format.");
+	}
+	if (gve_is_gqi(priv)) {
+		err = gve_set_desc_cnt(priv, descriptor);
+	} else {
+		/* DQO supports LRO. */
+		err = gve_set_desc_cnt_dqo(priv, descriptor, dev_op_dqo_rda);
+	}
+	if (err)
+		goto free_device_descriptor;
+
+	priv->max_registered_pages =
+				be64_to_cpu(descriptor->max_registered_pages);
+	mtu = be16_to_cpu(descriptor->mtu);
+	if (mtu < ETH_MIN_MTU) {
+		PMD_DRV_LOG(ERR, "MTU %d below minimum MTU", mtu);
+		err = -EINVAL;
+		goto free_device_descriptor;
+	}
+	priv->max_mtu = mtu;
+	priv->num_event_counters = be16_to_cpu(descriptor->counters);
+	rte_memcpy(priv->dev_addr.addr_bytes, descriptor->mac, ETH_ALEN);
+	mac = descriptor->mac;
+	PMD_DRV_LOG(INFO, "MAC addr: %02x:%02x:%02x:%02x:%02x:%02x",
+		    mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+	priv->tx_pages_per_qpl = be16_to_cpu(descriptor->tx_pages_per_qpl);
+	priv->rx_data_slot_cnt = be16_to_cpu(descriptor->rx_pages_per_qpl);
+
+	if (gve_is_gqi(priv) && priv->rx_data_slot_cnt < priv->rx_desc_cnt) {
+		PMD_DRV_LOG(ERR, "rx_data_slot_cnt cannot be smaller than rx_desc_cnt, setting rx_desc_cnt down to %d",
+			    priv->rx_data_slot_cnt);
+		priv->rx_desc_cnt = priv->rx_data_slot_cnt;
+	}
+	priv->default_num_queues = be16_to_cpu(descriptor->default_num_queues);
+
+	gve_enable_supported_features(priv, supported_features_mask,
+				      dev_op_jumbo_frames);
+
+free_device_descriptor:
+	gve_free_dma_mem(&descriptor_dma_mem);
+	return err;
+}
+
+int gve_adminq_register_page_list(struct gve_priv *priv,
+				  struct gve_queue_page_list *qpl)
+{
+	struct gve_dma_mem page_list_dma_mem;
+	u32 num_entries = qpl->num_entries;
+	u32 size = num_entries * sizeof(qpl->page_buses[0]);
+	union gve_adminq_command cmd;
+	__be64 *page_list;
+	int err;
+	u32 i;
+
+	memset(&cmd, 0, sizeof(cmd));
+	page_list = gve_alloc_dma_mem(&page_list_dma_mem, size);
+	if (!page_list)
+		return -ENOMEM;
+
+	for (i = 0; i < num_entries; i++)
+		page_list[i] = cpu_to_be64(qpl->page_buses[i]);
+
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_REGISTER_PAGE_LIST);
+	cmd.reg_page_list = (struct gve_adminq_register_page_list) {
+		.page_list_id = cpu_to_be32(qpl->id),
+		.num_pages = cpu_to_be32(num_entries),
+		.page_address_list_addr = cpu_to_be64(page_list_dma_mem.pa),
+	};
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	gve_free_dma_mem(&page_list_dma_mem);
+	return err;
+}
+
+int gve_adminq_unregister_page_list(struct gve_priv *priv, u32 page_list_id)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_UNREGISTER_PAGE_LIST);
+	cmd.unreg_page_list = (struct gve_adminq_unregister_page_list) {
+		.page_list_id = cpu_to_be32(page_list_id),
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_set_mtu(struct gve_priv *priv, u64 mtu)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_SET_DRIVER_PARAMETER);
+	cmd.set_driver_param = (struct gve_adminq_set_driver_parameter) {
+		.parameter_type = cpu_to_be32(GVE_SET_PARAM_MTU),
+		.parameter_value = cpu_to_be64(mtu),
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_report_stats(struct gve_priv *priv, u64 stats_report_len,
+			    dma_addr_t stats_report_addr, u64 interval)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_REPORT_STATS);
+	cmd.report_stats = (struct gve_adminq_report_stats) {
+		.stats_report_len = cpu_to_be64(stats_report_len),
+		.stats_report_addr = cpu_to_be64(stats_report_addr),
+		.interval = cpu_to_be64(interval),
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_report_link_speed(struct gve_priv *priv)
+{
+	struct gve_dma_mem link_speed_region_dma_mem;
+	union gve_adminq_command gvnic_cmd;
+	u64 *link_speed_region;
+	int err;
+
+	link_speed_region = gve_alloc_dma_mem(&link_speed_region_dma_mem,
+					      sizeof(*link_speed_region));
+
+	if (!link_speed_region)
+		return -ENOMEM;
+
+	memset(&gvnic_cmd, 0, sizeof(gvnic_cmd));
+	gvnic_cmd.opcode = cpu_to_be32(GVE_ADMINQ_REPORT_LINK_SPEED);
+	gvnic_cmd.report_link_speed.link_speed_address =
+		cpu_to_be64(link_speed_region_dma_mem.pa);
+
+	err = gve_adminq_execute_cmd(priv, &gvnic_cmd);
+
+	priv->link_speed = be64_to_cpu(*link_speed_region);
+	gve_free_dma_mem(&link_speed_region_dma_mem);
+	return err;
+}
+
+int gve_adminq_get_ptype_map_dqo(struct gve_priv *priv,
+				 struct gve_ptype_lut *ptype_lut)
+{
+	struct gve_dma_mem ptype_map_dma_mem;
+	struct gve_ptype_map *ptype_map;
+	union gve_adminq_command cmd;
+	int err = 0;
+	int i;
+
+	memset(&cmd, 0, sizeof(cmd));
+	ptype_map = gve_alloc_dma_mem(&ptype_map_dma_mem, sizeof(*ptype_map));
+	if (!ptype_map)
+		return -ENOMEM;
+
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_GET_PTYPE_MAP);
+	cmd.get_ptype_map = (struct gve_adminq_get_ptype_map) {
+		.ptype_map_len = cpu_to_be64(sizeof(*ptype_map)),
+		.ptype_map_addr = cpu_to_be64(ptype_map_dma_mem.pa),
+	};
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	if (err)
+		goto err;
+
+	/* Populate ptype_lut. */
+	for (i = 0; i < GVE_NUM_PTYPES; i++) {
+		ptype_lut->ptypes[i].l3_type =
+			ptype_map->ptypes[i].l3_type;
+		ptype_lut->ptypes[i].l4_type =
+			ptype_map->ptypes[i].l4_type;
+	}
+err:
+	gve_free_dma_mem(&ptype_map_dma_mem);
+	return err;
+}
drivers/net/gve/gve_adminq.h new file +381 -0

381 lines not on this page · raw message

drivers/net/gve/gve_desc.h new file +137 -0

137 lines not on this page · raw message

drivers/net/gve/gve_desc_dqo.h new file +254 -0

254 lines not on this page · raw message

drivers/net/gve/gve_register.h new file +28 -0
new file mode 100644
index 0000000000..b65f336be2
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH 01/10] net/gve: introduce GVE PMD base code

drivers/net/gve/gve_adminq.c line 7

Quoted 13 lines · On Fri, 29 Jul 2022 19:30:33 +0000 Xiaoyun Li <[email protected]> wrote:
drivers/net/gve/gve_adminq.c new file +7 -0
new file mode 100644
index 0000000000..8a724f12c6
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+

This would require special license exception approval by the tech board. Can you make it GPL or dual licensed instead please?

#3 lore raw

Re: [PATCH 01/10] net/gve: introduce GVE PMD base code

Quoted 19 lines · On Fri, 29 Jul 2022 15:42:48 -0700 Stephen Hemminger…

drivers/net/gve/gve_adminq.c line 7

Quoted 13 lines · On Fri, 29 Jul 2022 19:30:33 +0000 Xiaoyun Li <[email protected]> wrote:
drivers/net/gve/gve_adminq.c new file +7 -0
new file mode 100644
index 0000000000..8a724f12c6
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+  

This would require special license exception approval by the tech board. Can you make it GPL or dual licensed instead please?

I meant BSD

#4 lore raw

RE: [PATCH 01/10] net/gve: introduce GVE PMD base code

Hi Bruce,

Could you help give some comments about the License? Thanks!

Regards,
Junfeng Guo

Quoted 35 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Saturday, July 30, 2022 06:45
To: Li, Xiaoyun <[email protected]>
Cc: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH 01/10] net/gve: introduce GVE PMD base code

Quoted 4 lines · On Fri, 29 Jul 2022 15:42:48 -0700 Stephen Hemminger…

Review

Quoted 1 line · On Fri, 29 Jul 2022 19:30:33 +0000 Xiaoyun Li <[email protected]> wrote:
(quoted diff) +0 -0

b/drivers/net/gve/gve_adminq.c

Quoted 7 lines
Quoted 7 lines

new file mode 100644
index 0000000000..8a724f12c6

drivers/net/gve/gve_adminq.c new file +2 -0
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
Quoted 8 lines

drivers/net/gve/gve_adminq.c line 7

Quoted 5 lines
drivers/net/gve/gve_adminq.c new file +5 -0
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+

This would require special license exception approval by the tech board. Can you make it GPL or dual licensed instead please?

I meant BSD

#5 lore raw

v2 cover introduce GVE PMD

In reply to Xiaoyun Li

Introduce a new PMD for Google Virtual Ethernet (GVE).

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

Junfeng Guo (10):
net/gve: introduce GVE PMD base code
net/gve: add logs and OS specific implementation net/gve: support device initialization net/gve: add link update support
net/gve: add MTU set support
net/gve: add queue operations
net/gve: add Rx/Tx support
net/gve: add support to get dev info and configure dev net/gve: add stats support
doc: update documentation

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  18 +
 doc/guides/nics/gve.rst                |  65 ++
 doc/guides/rel_notes/release_22_11.rst |   4 +
 drivers/net/gve/gve.h                  | 331 +++++++++
 drivers/net/gve/gve_adminq.c           | 926 +++++++++++++++++++++++++
 drivers/net/gve/gve_adminq.h           | 383 ++++++++++
 drivers/net/gve/gve_desc.h             | 139 ++++
 drivers/net/gve/gve_desc_dqo.h         | 256 +++++++
 drivers/net/gve/gve_ethdev.c           | 772 +++++++++++++++++++++
 drivers/net/gve/gve_logs.h             |  22 +
 drivers/net/gve/gve_osdep.h            | 149 ++++
 drivers/net/gve/gve_register.h         |  30 +
 drivers/net/gve/gve_rx.c               | 366 ++++++++++
 drivers/net/gve/gve_tx.c               | 678 ++++++++++++++++++
 drivers/net/gve/meson.build            |  15 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 18 files changed, 4164 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve.h
 create mode 100644 drivers/net/gve/gve_adminq.c
 create mode 100644 drivers/net/gve/gve_adminq.h
 create mode 100644 drivers/net/gve/gve_desc.h
 create mode 100644 drivers/net/gve/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_osdep.h
 create mode 100644 drivers/net/gve/gve_register.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.34.1
#6 lore raw

PATCH v2 1/10 net/gve: introduce GVE PMD base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

---
 drivers/net/gve/gve_adminq.c   | 925 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_adminq.h   | 381 ++++++++++++++
 drivers/net/gve/gve_desc.h     | 137 +++++
 drivers/net/gve/gve_desc_dqo.h | 254 +++++++++
 drivers/net/gve/gve_register.h |  28 +
 5 files changed, 1725 insertions(+)
 create mode 100644 drivers/net/gve/gve_adminq.c
 create mode 100644 drivers/net/gve/gve_adminq.h
 create mode 100644 drivers/net/gve/gve_desc.h
 create mode 100644 drivers/net/gve/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/gve_register.h
drivers/net/gve/gve_adminq.c new file +925 -0
new file mode 100644
index 0000000000..8a724f12c6
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#include "gve_adminq.h"
+#include "gve_register.h"
+
+#define GVE_MAX_ADMINQ_RELEASE_CHECK	500
+#define GVE_ADMINQ_SLEEP_LEN		20
+#define GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK	100
+
+#define GVE_DEVICE_OPTION_ERROR_FMT "%s option error:\n" \
+"Expected: length=%d, feature_mask=%x.\n" \
+"Actual: length=%d, feature_mask=%x."
+
+#define GVE_DEVICE_OPTION_TOO_BIG_FMT "Length of %s option larger than expected. Possible older version of guest driver."
+
+static
+struct gve_device_option *gve_get_next_option(struct gve_device_descriptor *descriptor,
+					      struct gve_device_option *option)
+{
+	uintptr_t option_end, descriptor_end;
+
+	option_end = (uintptr_t)option + sizeof(*option) + be16_to_cpu(option->option_length);
+	descriptor_end = (uintptr_t)descriptor + be16_to_cpu(descriptor->total_length);
+
+	return option_end > descriptor_end ? NULL : (struct gve_device_option *)option_end;
+}
+
+static
+void gve_parse_device_option(struct gve_priv *priv,
+			     struct gve_device_option *option,
+			     struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+			     struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+			     struct gve_device_option_dqo_rda **dev_op_dqo_rda,
+			     struct gve_device_option_jumbo_frames **dev_op_jumbo_frames)
+{
+	u32 req_feat_mask = be32_to_cpu(option->required_features_mask);
+	u16 option_length = be16_to_cpu(option->option_length);
+	u16 option_id = be16_to_cpu(option->option_id);
+
+	/* If the length or feature mask doesn't match, continue without
+	 * enabling the feature.
+	 */
+	switch (option_id) {
+	case GVE_DEV_OPT_ID_GQI_RAW_ADDRESSING:
+		if (option_length != GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "Raw Addressing",
+				    GVE_DEV_OPT_LEN_GQI_RAW_ADDRESSING,
+				    GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RAW_ADDRESSING,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		PMD_DRV_LOG(INFO, "Gqi raw addressing device option enabled.");
+		priv->queue_format = GVE_GQI_RDA_FORMAT;
+		break;
+	case GVE_DEV_OPT_ID_GQI_RDA:
+		if (option_length < sizeof(**dev_op_gqi_rda) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "GQI RDA", (int)sizeof(**dev_op_gqi_rda),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_GQI_RDA,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_gqi_rda)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT, "GQI RDA");
+		}
+		*dev_op_gqi_rda = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	case GVE_DEV_OPT_ID_GQI_QPL:
+		if (option_length < sizeof(**dev_op_gqi_qpl) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "GQI QPL", (int)sizeof(**dev_op_gqi_qpl),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_GQI_QPL,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_gqi_qpl)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT, "GQI QPL");
+		}
+		*dev_op_gqi_qpl = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	case GVE_DEV_OPT_ID_DQO_RDA:
+		if (option_length < sizeof(**dev_op_dqo_rda) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "DQO RDA", (int)sizeof(**dev_op_dqo_rda),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_DQO_RDA,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_dqo_rda)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT, "DQO RDA");
+		}
+		*dev_op_dqo_rda = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	case GVE_DEV_OPT_ID_JUMBO_FRAMES:
+		if (option_length < sizeof(**dev_op_jumbo_frames) ||
+		    req_feat_mask != GVE_DEV_OPT_REQ_FEAT_MASK_JUMBO_FRAMES) {
+			PMD_DRV_LOG(WARNING, GVE_DEVICE_OPTION_ERROR_FMT,
+				    "Jumbo Frames",
+				    (int)sizeof(**dev_op_jumbo_frames),
+				    GVE_DEV_OPT_REQ_FEAT_MASK_JUMBO_FRAMES,
+				    option_length, req_feat_mask);
+			break;
+		}
+
+		if (option_length > sizeof(**dev_op_jumbo_frames)) {
+			PMD_DRV_LOG(WARNING,
+				    GVE_DEVICE_OPTION_TOO_BIG_FMT,
+				    "Jumbo Frames");
+		}
+		*dev_op_jumbo_frames = RTE_PTR_ADD(option, sizeof(*option));
+		break;
+	default:
+		/* If we don't recognize the option just continue
+		 * without doing anything.
+		 */
+		PMD_DRV_LOG(DEBUG, "Unrecognized device option 0x%hx not enabled.\n",
+			    option_id);
+	}
+}
+
+/* Process all device options for a given describe device call. */
+static int
+gve_process_device_options(struct gve_priv *priv,
+			   struct gve_device_descriptor *descriptor,
+			   struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+			   struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+			   struct gve_device_option_dqo_rda **dev_op_dqo_rda,
+			   struct gve_device_option_jumbo_frames **dev_op_jumbo_frames)
+{
+	const int num_options = be16_to_cpu(descriptor->num_device_options);
+	struct gve_device_option *dev_opt;
+	int i;
+
+	/* The options struct directly follows the device descriptor. */
+	dev_opt = RTE_PTR_ADD(descriptor, sizeof(*descriptor));
+	for (i = 0; i < num_options; i++) {
+		struct gve_device_option *next_opt;
+
+		next_opt = gve_get_next_option(descriptor, dev_opt);
+		if (!next_opt) {
+			PMD_DRV_LOG(ERR,
+				    "options exceed device_descriptor's total length.\n");
+			return -EINVAL;
+		}
+
+		gve_parse_device_option(priv, dev_opt,
+					dev_op_gqi_rda, dev_op_gqi_qpl,
+					dev_op_dqo_rda, dev_op_jumbo_frames);
+		dev_opt = next_opt;
+	}
+
+	return 0;
+}
+
+int gve_adminq_alloc(struct gve_priv *priv)
+{
+	priv->adminq = gve_alloc_dma_mem(&priv->adminq_dma_mem, PAGE_SIZE);
+	if (unlikely(!priv->adminq))
+		return -ENOMEM;
+
+	priv->adminq_mask = (PAGE_SIZE / sizeof(union gve_adminq_command)) - 1;
+	priv->adminq_prod_cnt = 0;
+	priv->adminq_cmd_fail = 0;
+	priv->adminq_timeouts = 0;
+	priv->adminq_describe_device_cnt = 0;
+	priv->adminq_cfg_device_resources_cnt = 0;
+	priv->adminq_register_page_list_cnt = 0;
+	priv->adminq_unregister_page_list_cnt = 0;
+	priv->adminq_create_tx_queue_cnt = 0;
+	priv->adminq_create_rx_queue_cnt = 0;
+	priv->adminq_destroy_tx_queue_cnt = 0;
+	priv->adminq_destroy_rx_queue_cnt = 0;
+	priv->adminq_dcfg_device_resources_cnt = 0;
+	priv->adminq_set_driver_parameter_cnt = 0;
+	priv->adminq_report_stats_cnt = 0;
+	priv->adminq_report_link_speed_cnt = 0;
+	priv->adminq_get_ptype_map_cnt = 0;
+
+	/* Setup Admin queue with the device */
+	iowrite32be(priv->adminq_dma_mem.pa / PAGE_SIZE,
+		    &priv->reg_bar0->adminq_pfn);
+
+	gve_set_admin_queue_ok(priv);
+	return 0;
+}
+
+void gve_adminq_release(struct gve_priv *priv)
+{
+	int i = 0;
+
+	/* Tell the device the adminq is leaving */
+	iowrite32be(0x0, &priv->reg_bar0->adminq_pfn);
+	while (ioread32be(&priv->reg_bar0->adminq_pfn)) {
+		/* If this is reached the device is unrecoverable and still
+		 * holding memory. Continue looping to avoid memory corruption,
+		 * but WARN so it is visible what is going on.
+		 */
+		if (i == GVE_MAX_ADMINQ_RELEASE_CHECK)
+			PMD_DRV_LOG(WARNING, "Unrecoverable platform error!");
+		i++;
+		msleep(GVE_ADMINQ_SLEEP_LEN);
+	}
+	gve_clear_device_rings_ok(priv);
+	gve_clear_device_resources_ok(priv);
+	gve_clear_admin_queue_ok(priv);
+}
+
+void gve_adminq_free(struct gve_priv *priv)
+{
+	if (!gve_get_admin_queue_ok(priv))
+		return;
+	gve_adminq_release(priv);
+	gve_free_dma_mem(&priv->adminq_dma_mem);
+	gve_clear_admin_queue_ok(priv);
+}
+
+static void gve_adminq_kick_cmd(struct gve_priv *priv, u32 prod_cnt)
+{
+	iowrite32be(prod_cnt, &priv->reg_bar0->adminq_doorbell);
+}
+
+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32 prod_cnt)
+{
+	int i;
+
+	for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++) {
+		if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+		    == prod_cnt)
+			return true;
+		msleep(GVE_ADMINQ_SLEEP_LEN);
+	}
+
+	return false;
+}
+
+static int gve_adminq_parse_err(struct gve_priv *priv, u32 status)
+{
+	if (status != GVE_ADMINQ_COMMAND_PASSED &&
+	    status != GVE_ADMINQ_COMMAND_UNSET) {
+		PMD_DRV_LOG(ERR, "AQ command failed with status %d", status);
+		priv->adminq_cmd_fail++;
+	}
+	switch (status) {
+	case GVE_ADMINQ_COMMAND_PASSED:
+		return 0;
+	case GVE_ADMINQ_COMMAND_UNSET:
+		PMD_DRV_LOG(ERR, "parse_aq_err: err and status both unset, this should not be possible.");
+		return -EINVAL;
+	case GVE_ADMINQ_COMMAND_ERROR_ABORTED:
+	case GVE_ADMINQ_COMMAND_ERROR_CANCELLED:
+	case GVE_ADMINQ_COMMAND_ERROR_DATALOSS:
+	case GVE_ADMINQ_COMMAND_ERROR_FAILED_PRECONDITION:
+	case GVE_ADMINQ_COMMAND_ERROR_UNAVAILABLE:
+		return -EAGAIN;
+	case GVE_ADMINQ_COMMAND_ERROR_ALREADY_EXISTS:
+	case GVE_ADMINQ_COMMAND_ERROR_INTERNAL_ERROR:
+	case GVE_ADMINQ_COMMAND_ERROR_INVALID_ARGUMENT:
+	case GVE_ADMINQ_COMMAND_ERROR_NOT_FOUND:
+	case GVE_ADMINQ_COMMAND_ERROR_OUT_OF_RANGE:
+	case GVE_ADMINQ_COMMAND_ERROR_UNKNOWN_ERROR:
+		return -EINVAL;
+	case GVE_ADMINQ_COMMAND_ERROR_DEADLINE_EXCEEDED:
+		return -ETIME;
+	case GVE_ADMINQ_COMMAND_ERROR_PERMISSION_DENIED:
+	case GVE_ADMINQ_COMMAND_ERROR_UNAUTHENTICATED:
+		return -EACCES;
+	case GVE_ADMINQ_COMMAND_ERROR_RESOURCE_EXHAUSTED:
+		return -ENOMEM;
+	case GVE_ADMINQ_COMMAND_ERROR_UNIMPLEMENTED:
+		return -ENOTSUP;
+	default:
+		PMD_DRV_LOG(ERR, "parse_aq_err: unknown status code %d",
+			    status);
+		return -EINVAL;
+	}
+}
+
+/* Flushes all AQ commands currently queued and waits for them to complete.
+ * If there are failures, it will return the first error.
+ */
+static int gve_adminq_kick_and_wait(struct gve_priv *priv)
+{
+	u32 tail, head;
+	u32 i;
+
+	tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+	head = priv->adminq_prod_cnt;
+
+	gve_adminq_kick_cmd(priv, head);
+	if (!gve_adminq_wait_for_cmd(priv, head)) {
+		PMD_DRV_LOG(ERR, "AQ commands timed out, need to reset AQ");
+		priv->adminq_timeouts++;
+		return -ENOTRECOVERABLE;
+	}
+
+	for (i = tail; i < head; i++) {
+		union gve_adminq_command *cmd;
+		u32 status, err;
+
+		cmd = &priv->adminq[i & priv->adminq_mask];
+		status = be32_to_cpu(READ_ONCE32(cmd->status));
+		err = gve_adminq_parse_err(priv, status);
+		if (err)
+			/* Return the first error if we failed. */
+			return err;
+	}
+
+	return 0;
+}
+
+/* This function is not threadsafe - the caller is responsible for any
+ * necessary locks.
+ */
+static int gve_adminq_issue_cmd(struct gve_priv *priv,
+				union gve_adminq_command *cmd_orig)
+{
+	union gve_adminq_command *cmd;
+	u32 opcode;
+	u32 tail;
+
+	tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+
+	/* Check if next command will overflow the buffer. */
+	if (((priv->adminq_prod_cnt + 1) & priv->adminq_mask) ==
+	    (tail & priv->adminq_mask)) {
+		int err;
+
+		/* Flush existing commands to make room. */
+		err = gve_adminq_kick_and_wait(priv);
+		if (err)
+			return err;
+
+		/* Retry. */
+		tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+		if (((priv->adminq_prod_cnt + 1) & priv->adminq_mask) ==
+		    (tail & priv->adminq_mask)) {
+			/* This should never happen. We just flushed the
+			 * command queue so there should be enough space.
+			 */
+			return -ENOMEM;
+		}
+	}
+
+	cmd = &priv->adminq[priv->adminq_prod_cnt & priv->adminq_mask];
+	priv->adminq_prod_cnt++;
+
+	memcpy(cmd, cmd_orig, sizeof(*cmd_orig));
+	opcode = be32_to_cpu(READ_ONCE32(cmd->opcode));
+
+	switch (opcode) {
+	case GVE_ADMINQ_DESCRIBE_DEVICE:
+		priv->adminq_describe_device_cnt++;
+		break;
+	case GVE_ADMINQ_CONFIGURE_DEVICE_RESOURCES:
+		priv->adminq_cfg_device_resources_cnt++;
+		break;
+	case GVE_ADMINQ_REGISTER_PAGE_LIST:
+		priv->adminq_register_page_list_cnt++;
+		break;
+	case GVE_ADMINQ_UNREGISTER_PAGE_LIST:
+		priv->adminq_unregister_page_list_cnt++;
+		break;
+	case GVE_ADMINQ_CREATE_TX_QUEUE:
+		priv->adminq_create_tx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_CREATE_RX_QUEUE:
+		priv->adminq_create_rx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_DESTROY_TX_QUEUE:
+		priv->adminq_destroy_tx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_DESTROY_RX_QUEUE:
+		priv->adminq_destroy_rx_queue_cnt++;
+		break;
+	case GVE_ADMINQ_DECONFIGURE_DEVICE_RESOURCES:
+		priv->adminq_dcfg_device_resources_cnt++;
+		break;
+	case GVE_ADMINQ_SET_DRIVER_PARAMETER:
+		priv->adminq_set_driver_parameter_cnt++;
+		break;
+	case GVE_ADMINQ_REPORT_STATS:
+		priv->adminq_report_stats_cnt++;
+		break;
+	case GVE_ADMINQ_REPORT_LINK_SPEED:
+		priv->adminq_report_link_speed_cnt++;
+		break;
+	case GVE_ADMINQ_GET_PTYPE_MAP:
+		priv->adminq_get_ptype_map_cnt++;
+		break;
+	default:
+		PMD_DRV_LOG(ERR, "unknown AQ command opcode %d", opcode);
+	}
+
+	return 0;
+}
+
+/* This function is not threadsafe - the caller is responsible for any
+ * necessary locks.
+ * The caller is also responsible for making sure there are no commands
+ * waiting to be executed.
+ */
+static int gve_adminq_execute_cmd(struct gve_priv *priv,
+				  union gve_adminq_command *cmd_orig)
+{
+	u32 tail, head;
+	int err;
+
+	tail = ioread32be(&priv->reg_bar0->adminq_event_counter);
+	head = priv->adminq_prod_cnt;
+	if (tail != head)
+		/* This is not a valid path */
+		return -EINVAL;
+
+	err = gve_adminq_issue_cmd(priv, cmd_orig);
+	if (err)
+		return err;
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+/* The device specifies that the management vector can either be the first irq
+ * or the last irq. ntfy_blk_msix_base_idx indicates the first irq assigned to
+ * the ntfy blks. It if is 0 then the management vector is last, if it is 1 then
+ * the management vector is first.
+ *
+ * gve arranges the msix vectors so that the management vector is last.
+ */
+#define GVE_NTFY_BLK_BASE_MSIX_IDX	0
+int gve_adminq_configure_device_resources(struct gve_priv *priv,
+					  dma_addr_t counter_array_bus_addr,
+					  u32 num_counters,
+					  dma_addr_t db_array_bus_addr,
+					  u32 num_ntfy_blks)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_CONFIGURE_DEVICE_RESOURCES);
+	cmd.configure_device_resources =
+		(struct gve_adminq_configure_device_resources) {
+		.counter_array = cpu_to_be64(counter_array_bus_addr),
+		.num_counters = cpu_to_be32(num_counters),
+		.irq_db_addr = cpu_to_be64(db_array_bus_addr),
+		.num_irq_dbs = cpu_to_be32(num_ntfy_blks),
+		.irq_db_stride = cpu_to_be32(sizeof(*priv->irq_dbs)),
+		.ntfy_blk_msix_base_idx =
+					cpu_to_be32(GVE_NTFY_BLK_BASE_MSIX_IDX),
+		.queue_format = priv->queue_format,
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_deconfigure_device_resources(struct gve_priv *priv)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DECONFIGURE_DEVICE_RESOURCES);
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+static int gve_adminq_create_tx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	struct gve_tx_queue *txq = priv->txqs[queue_index];
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_CREATE_TX_QUEUE);
+	cmd.create_tx_queue = (struct gve_adminq_create_tx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+		.queue_resources_addr =
+			cpu_to_be64(txq->qres_mz->iova),
+		.tx_ring_addr = cpu_to_be64(txq->tx_ring_phys_addr),
+		.ntfy_id = cpu_to_be32(txq->ntfy_id),
+	};
+
+	if (gve_is_gqi(priv)) {
+		u32 qpl_id = priv->queue_format == GVE_GQI_RDA_FORMAT ?
+			GVE_RAW_ADDRESSING_QPL_ID : txq->qpl->id;
+
+		cmd.create_tx_queue.queue_page_list_id = cpu_to_be32(qpl_id);
+	} else {
+		cmd.create_tx_queue.tx_ring_size =
+			cpu_to_be16(txq->nb_tx_desc);
+		cmd.create_tx_queue.tx_comp_ring_addr =
+			cpu_to_be64(txq->complq->tx_ring_phys_addr);
+		cmd.create_tx_queue.tx_comp_ring_size =
+			cpu_to_be16(priv->tx_compq_size);
+	}
+
+	return gve_adminq_issue_cmd(priv, &cmd);
+}
+
+int gve_adminq_create_tx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_create_tx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_adminq_create_rx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	struct gve_rx_queue *rxq = priv->rxqs[queue_index];
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_CREATE_RX_QUEUE);
+	cmd.create_rx_queue = (struct gve_adminq_create_rx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+		.ntfy_id = cpu_to_be32(rxq->ntfy_id),
+		.queue_resources_addr = cpu_to_be64(rxq->qres_mz->iova),
+	};
+
+	if (gve_is_gqi(priv)) {
+		u32 qpl_id = priv->queue_format == GVE_GQI_RDA_FORMAT ?
+			GVE_RAW_ADDRESSING_QPL_ID : rxq->qpl->id;
+
+		cmd.create_rx_queue.rx_desc_ring_addr =
+			cpu_to_be64(rxq->mz->iova),
+		cmd.create_rx_queue.rx_data_ring_addr =
+			cpu_to_be64(rxq->data_mz->iova),
+		cmd.create_rx_queue.index = cpu_to_be32(queue_index);
+		cmd.create_rx_queue.queue_page_list_id = cpu_to_be32(qpl_id);
+		cmd.create_rx_queue.packet_buffer_size = cpu_to_be16(rxq->rx_buf_len);
+	} else {
+		cmd.create_rx_queue.rx_ring_size =
+			cpu_to_be16(priv->rx_desc_cnt);
+		cmd.create_rx_queue.rx_desc_ring_addr =
+			cpu_to_be64(rxq->rx_ring_phys_addr);
+		cmd.create_rx_queue.rx_data_ring_addr =
+			cpu_to_be64(rxq->bufq->rx_ring_phys_addr);
+		cmd.create_rx_queue.packet_buffer_size =
+			cpu_to_be16(rxq->rx_buf_len);
+		cmd.create_rx_queue.rx_buff_ring_size =
+			cpu_to_be16(priv->rx_bufq_size);
+		cmd.create_rx_queue.enable_rsc = !!(priv->enable_lsc);
+	}
+
+	return gve_adminq_issue_cmd(priv, &cmd);
+}
+
+int gve_adminq_create_rx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_create_rx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_adminq_destroy_tx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	union gve_adminq_command cmd;
+	int err;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESTROY_TX_QUEUE);
+	cmd.destroy_tx_queue = (struct gve_adminq_destroy_tx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+	};
+
+	err = gve_adminq_issue_cmd(priv, &cmd);
+	if (err)
+		return err;
+
+	return 0;
+}
+
+int gve_adminq_destroy_tx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_destroy_tx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_adminq_destroy_rx_queue(struct gve_priv *priv, u32 queue_index)
+{
+	union gve_adminq_command cmd;
+	int err;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESTROY_RX_QUEUE);
+	cmd.destroy_rx_queue = (struct gve_adminq_destroy_rx_queue) {
+		.queue_id = cpu_to_be32(queue_index),
+	};
+
+	err = gve_adminq_issue_cmd(priv, &cmd);
+	if (err)
+		return err;
+
+	return 0;
+}
+
+int gve_adminq_destroy_rx_queues(struct gve_priv *priv, u32 num_queues)
+{
+	int err;
+	u32 i;
+
+	for (i = 0; i < num_queues; i++) {
+		err = gve_adminq_destroy_rx_queue(priv, i);
+		if (err)
+			return err;
+	}
+
+	return gve_adminq_kick_and_wait(priv);
+}
+
+static int gve_set_desc_cnt(struct gve_priv *priv,
+			    struct gve_device_descriptor *descriptor)
+{
+	priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries);
+	if (priv->tx_desc_cnt * sizeof(priv->txqs[0]->tx_desc_ring[0])
+	    < PAGE_SIZE) {
+		PMD_DRV_LOG(ERR, "Tx desc count %d too low", priv->tx_desc_cnt);
+		return -EINVAL;
+	}
+	priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries);
+	if (priv->rx_desc_cnt * sizeof(priv->rxqs[0]->rx_desc_ring[0])
+	    < PAGE_SIZE) {
+		PMD_DRV_LOG(ERR, "Rx desc count %d too low", priv->rx_desc_cnt);
+		return -EINVAL;
+	}
+	return 0;
+}
+
+static int
+gve_set_desc_cnt_dqo(struct gve_priv *priv,
+		     const struct gve_device_descriptor *descriptor,
+		     const struct gve_device_option_dqo_rda *dev_op_dqo_rda)
+{
+	priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries);
+	priv->tx_compq_size = be16_to_cpu(dev_op_dqo_rda->tx_comp_ring_entries);
+	priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries);
+	priv->rx_bufq_size = be16_to_cpu(dev_op_dqo_rda->rx_buff_ring_entries);
+
+	return 0;
+}
+
+static void gve_enable_supported_features(struct gve_priv *priv,
+					  u32 supported_features_mask,
+					  const struct gve_device_option_jumbo_frames
+						  *dev_op_jumbo_frames)
+{
+	/* Before control reaches this point, the page-size-capped max MTU from
+	 * the gve_device_descriptor field has already been stored in
+	 * priv->dev->max_mtu. We overwrite it with the true max MTU below.
+	 */
+	if (dev_op_jumbo_frames &&
+	    (supported_features_mask & GVE_SUP_JUMBO_FRAMES_MASK)) {
+		PMD_DRV_LOG(INFO, "JUMBO FRAMES device option enabled.");
+		priv->max_mtu = be16_to_cpu(dev_op_jumbo_frames->max_mtu);
+	}
+}
+
+int gve_adminq_describe_device(struct gve_priv *priv)
+{
+	struct gve_device_option_jumbo_frames *dev_op_jumbo_frames = NULL;
+	struct gve_device_option_gqi_rda *dev_op_gqi_rda = NULL;
+	struct gve_device_option_gqi_qpl *dev_op_gqi_qpl = NULL;
+	struct gve_device_option_dqo_rda *dev_op_dqo_rda = NULL;
+	struct gve_device_descriptor *descriptor;
+	struct gve_dma_mem descriptor_dma_mem;
+	u32 supported_features_mask = 0;
+	union gve_adminq_command cmd;
+	int err = 0;
+	u8 *mac;
+	u16 mtu;
+
+	memset(&cmd, 0, sizeof(cmd));
+	descriptor = gve_alloc_dma_mem(&descriptor_dma_mem, PAGE_SIZE);
+	if (!descriptor)
+		return -ENOMEM;
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESCRIBE_DEVICE);
+	cmd.describe_device.device_descriptor_addr =
+					cpu_to_be64(descriptor_dma_mem.pa);
+	cmd.describe_device.device_descriptor_version =
+			cpu_to_be32(GVE_ADMINQ_DEVICE_DESCRIPTOR_VERSION);
+	cmd.describe_device.available_length = cpu_to_be32(PAGE_SIZE);
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	if (err)
+		goto free_device_descriptor;
+
+	err = gve_process_device_options(priv, descriptor, &dev_op_gqi_rda,
+					 &dev_op_gqi_qpl, &dev_op_dqo_rda,
+					 &dev_op_jumbo_frames);
+	if (err)
+		goto free_device_descriptor;
+
+	/* If the GQI_RAW_ADDRESSING option is not enabled and the queue format
+	 * is not set to GqiRda, choose the queue format in a priority order:
+	 * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default.
+	 */
+	if (dev_op_dqo_rda) {
+		priv->queue_format = GVE_DQO_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with DQO RDA queue format.");
+		supported_features_mask =
+			be32_to_cpu(dev_op_dqo_rda->supported_features_mask);
+	} else if (dev_op_gqi_rda) {
+		priv->queue_format = GVE_GQI_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+		supported_features_mask =
+			be32_to_cpu(dev_op_gqi_rda->supported_features_mask);
+	} else if (priv->queue_format == GVE_GQI_RDA_FORMAT) {
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+	} else {
+		priv->queue_format = GVE_GQI_QPL_FORMAT;
+		if (dev_op_gqi_qpl)
+			supported_features_mask =
+				be32_to_cpu(dev_op_gqi_qpl->supported_features_mask);
+		PMD_DRV_LOG(INFO, "Driver is running with GQI QPL queue format.");
+	}
+	if (gve_is_gqi(priv)) {
+		err = gve_set_desc_cnt(priv, descriptor);
+	} else {
+		/* DQO supports LRO. */
+		err = gve_set_desc_cnt_dqo(priv, descriptor, dev_op_dqo_rda);
+	}
+	if (err)
+		goto free_device_descriptor;
+
+	priv->max_registered_pages =
+				be64_to_cpu(descriptor->max_registered_pages);
+	mtu = be16_to_cpu(descriptor->mtu);
+	if (mtu < ETH_MIN_MTU) {
+		PMD_DRV_LOG(ERR, "MTU %d below minimum MTU", mtu);
+		err = -EINVAL;
+		goto free_device_descriptor;
+	}
+	priv->max_mtu = mtu;
+	priv->num_event_counters = be16_to_cpu(descriptor->counters);
+	rte_memcpy(priv->dev_addr.addr_bytes, descriptor->mac, ETH_ALEN);
+	mac = descriptor->mac;
+	PMD_DRV_LOG(INFO, "MAC addr: %02x:%02x:%02x:%02x:%02x:%02x",
+		    mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+	priv->tx_pages_per_qpl = be16_to_cpu(descriptor->tx_pages_per_qpl);
+	priv->rx_data_slot_cnt = be16_to_cpu(descriptor->rx_pages_per_qpl);
+
+	if (gve_is_gqi(priv) && priv->rx_data_slot_cnt < priv->rx_desc_cnt) {
+		PMD_DRV_LOG(ERR, "rx_data_slot_cnt cannot be smaller than rx_desc_cnt, setting rx_desc_cnt down to %d",
+			    priv->rx_data_slot_cnt);
+		priv->rx_desc_cnt = priv->rx_data_slot_cnt;
+	}
+	priv->default_num_queues = be16_to_cpu(descriptor->default_num_queues);
+
+	gve_enable_supported_features(priv, supported_features_mask,
+				      dev_op_jumbo_frames);
+
+free_device_descriptor:
+	gve_free_dma_mem(&descriptor_dma_mem);
+	return err;
+}
+
+int gve_adminq_register_page_list(struct gve_priv *priv,
+				  struct gve_queue_page_list *qpl)
+{
+	struct gve_dma_mem page_list_dma_mem;
+	u32 num_entries = qpl->num_entries;
+	u32 size = num_entries * sizeof(qpl->page_buses[0]);
+	union gve_adminq_command cmd;
+	__be64 *page_list;
+	int err;
+	u32 i;
+
+	memset(&cmd, 0, sizeof(cmd));
+	page_list = gve_alloc_dma_mem(&page_list_dma_mem, size);
+	if (!page_list)
+		return -ENOMEM;
+
+	for (i = 0; i < num_entries; i++)
+		page_list[i] = cpu_to_be64(qpl->page_buses[i]);
+
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_REGISTER_PAGE_LIST);
+	cmd.reg_page_list = (struct gve_adminq_register_page_list) {
+		.page_list_id = cpu_to_be32(qpl->id),
+		.num_pages = cpu_to_be32(num_entries),
+		.page_address_list_addr = cpu_to_be64(page_list_dma_mem.pa),
+	};
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	gve_free_dma_mem(&page_list_dma_mem);
+	return err;
+}
+
+int gve_adminq_unregister_page_list(struct gve_priv *priv, u32 page_list_id)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_UNREGISTER_PAGE_LIST);
+	cmd.unreg_page_list = (struct gve_adminq_unregister_page_list) {
+		.page_list_id = cpu_to_be32(page_list_id),
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_set_mtu(struct gve_priv *priv, u64 mtu)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_SET_DRIVER_PARAMETER);
+	cmd.set_driver_param = (struct gve_adminq_set_driver_parameter) {
+		.parameter_type = cpu_to_be32(GVE_SET_PARAM_MTU),
+		.parameter_value = cpu_to_be64(mtu),
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_report_stats(struct gve_priv *priv, u64 stats_report_len,
+			    dma_addr_t stats_report_addr, u64 interval)
+{
+	union gve_adminq_command cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_REPORT_STATS);
+	cmd.report_stats = (struct gve_adminq_report_stats) {
+		.stats_report_len = cpu_to_be64(stats_report_len),
+		.stats_report_addr = cpu_to_be64(stats_report_addr),
+		.interval = cpu_to_be64(interval),
+	};
+
+	return gve_adminq_execute_cmd(priv, &cmd);
+}
+
+int gve_adminq_report_link_speed(struct gve_priv *priv)
+{
+	struct gve_dma_mem link_speed_region_dma_mem;
+	union gve_adminq_command gvnic_cmd;
+	u64 *link_speed_region;
+	int err;
+
+	link_speed_region = gve_alloc_dma_mem(&link_speed_region_dma_mem,
+					      sizeof(*link_speed_region));
+
+	if (!link_speed_region)
+		return -ENOMEM;
+
+	memset(&gvnic_cmd, 0, sizeof(gvnic_cmd));
+	gvnic_cmd.opcode = cpu_to_be32(GVE_ADMINQ_REPORT_LINK_SPEED);
+	gvnic_cmd.report_link_speed.link_speed_address =
+		cpu_to_be64(link_speed_region_dma_mem.pa);
+
+	err = gve_adminq_execute_cmd(priv, &gvnic_cmd);
+
+	priv->link_speed = be64_to_cpu(*link_speed_region);
+	gve_free_dma_mem(&link_speed_region_dma_mem);
+	return err;
+}
+
+int gve_adminq_get_ptype_map_dqo(struct gve_priv *priv,
+				 struct gve_ptype_lut *ptype_lut)
+{
+	struct gve_dma_mem ptype_map_dma_mem;
+	struct gve_ptype_map *ptype_map;
+	union gve_adminq_command cmd;
+	int err = 0;
+	int i;
+
+	memset(&cmd, 0, sizeof(cmd));
+	ptype_map = gve_alloc_dma_mem(&ptype_map_dma_mem, sizeof(*ptype_map));
+	if (!ptype_map)
+		return -ENOMEM;
+
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_GET_PTYPE_MAP);
+	cmd.get_ptype_map = (struct gve_adminq_get_ptype_map) {
+		.ptype_map_len = cpu_to_be64(sizeof(*ptype_map)),
+		.ptype_map_addr = cpu_to_be64(ptype_map_dma_mem.pa),
+	};
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	if (err)
+		goto err;
+
+	/* Populate ptype_lut. */
+	for (i = 0; i < GVE_NUM_PTYPES; i++) {
+		ptype_lut->ptypes[i].l3_type =
+			ptype_map->ptypes[i].l3_type;
+		ptype_lut->ptypes[i].l4_type =
+			ptype_map->ptypes[i].l4_type;
+	}
+err:
+	gve_free_dma_mem(&ptype_map_dma_mem);
+	return err;
+}
drivers/net/gve/gve_adminq.h new file +381 -0

381 lines not on this page · raw message

drivers/net/gve/gve_desc.h new file +137 -0

137 lines not on this page · raw message

drivers/net/gve/gve_desc_dqo.h new file +254 -0

254 lines not on this page · raw message

drivers/net/gve/gve_register.h new file +28 -0
new file mode 100644
index 0000000000..b65f336be2
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH v2 01/10] net/gve: introduce GVE PMD base code

Quoted 34 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

---
  drivers/net/gve/gve_adminq.c   | 925 +++++++++++++++++++++++++++++++++
  drivers/net/gve/gve_adminq.h   | 381 ++++++++++++++
  drivers/net/gve/gve_desc.h     | 137 +++++
  drivers/net/gve/gve_desc_dqo.h | 254 +++++++++
  drivers/net/gve/gve_register.h |  28 +
  5 files changed, 1725 insertions(+)

create mode 100644 drivers/net/gve/gve_adminq.c create mode 100644 drivers/net/gve/gve_adminq.h create mode 100644 drivers/net/gve/gve_desc.h create mode 100644 drivers/net/gve/gve_desc_dqo.h create mode 100644 drivers/net/gve/gve_register.h

drivers/net/gve/gve_adminq.c new file +0 -0
new file mode 100644
index 0000000000..8a724f12c6

drivers/net/gve/gve_adminq.c line 7

Quoted 7 lines
drivers/net/gve/gve_adminq.c new file +7 -0
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+

Can you please get approval for the MIT license from techboard, as Stephen highlighted in previous version?

#8 lore raw

Re: [PATCH v2 01/10] net/gve: introduce GVE PMD base code

Quoted 17 lines · On Thu, 1 Sep 2022 18:19:22 +0100 Ferruh Yigit <[email protected]> wrote:
Quoted 7 lines
drivers/net/gve/gve_adminq.c new file +0 -0
new file mode 100644
index 0000000000..8a724f12c6

drivers/net/gve/gve_adminq.c line 7

Quoted 7 lines
drivers/net/gve/gve_adminq.c new file +7 -0
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+  

Can you please get approval for the MIT license from techboard, as Stephen highlighted in previous version?

I would prefer that it be BSD or dual licensed.
Although MIT and BSD-3 licenses are compatible, this is not something techboard can decide it requires a statement from a knowledgeable open source lawyer (Intel or LF).

Please fix the license to BSD and save lots of trouble.

#9 lore raw

Re: [PATCH v2 01/10] net/gve: introduce GVE PMD base code

01/09/2022 20:23, Stephen Hemminger:

Quoted 22 lines
Quoted 17 lines · On Thu, 1 Sep 2022 18:19:22 +0100 Ferruh Yigit <[email protected]> wrote:
Quoted 7 lines
drivers/net/gve/gve_adminq.c new file +0 -0
new file mode 100644
index 0000000000..8a724f12c6

drivers/net/gve/gve_adminq.c line 7

Quoted 7 lines
drivers/net/gve/gve_adminq.c new file +7 -0
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+  

Can you please get approval for the MIT license from techboard, as Stephen highlighted in previous version?

Quoted 5 lines

I would prefer that it be BSD or dual licensed.
Although MIT and BSD-3 licenses are compatible, this is not something techboard can decide it requires a statement from a knowledgeable open source lawyer (Intel or LF).

Please fix the license to BSD and save lots of trouble.

+1 to change to BSD to avoid trouble.

#10 lore raw

RE: [PATCH v2 01/10] net/gve: introduce GVE PMD base code

Quoted 41 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Friday, September 2, 2022 04:50
To: Ferruh Yigit <[email protected]>; [email protected]
Cc: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Hemant Agrawal <[email protected]>; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Wang, Haiyue <[email protected]>; [email protected]; Stephen Hemminger <[email protected]>
Subject: Re: [PATCH v2 01/10] net/gve: introduce GVE PMD base code

01/09/2022 20:23, Stephen Hemminger:

Quoted 5 lines
Quoted 2 lines · On Thu, 1 Sep 2022 18:19:22 +0100 Ferruh Yigit <[email protected]> wrote:

Review

Quoted 2 lines
(quoted diff) +0 -0

b/drivers/net/gve/gve_adminq.c

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines

new file mode 100644
index 0000000000..8a724f12c6

drivers/net/gve/gve_adminq.c new file +2 -0
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
Quoted 8 lines

drivers/net/gve/gve_adminq.c line 7

Quoted 5 lines
drivers/net/gve/gve_adminq.c new file +5 -0
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+

Can you please get approval for the MIT license from techboard, as Stephen highlighted in previous version?

Quoted 2 lines

I would prefer that it be BSD or dual licensed.
Although MIT and BSD-3 licenses are compatible, this is not something

Quoted 7 lines

techboard can decide

Quoted 1 line

it requires a statement from a knowledgeable open source lawyer (Intel

or LF).

Quoted 2 lines

Please fix the license to BSD and save lots of trouble.

+1 to change to BSD to avoid trouble.

Thanks for your concern and comments!
Yes, we are also willing to have these base code under BSD license.

Note that these code are not Intel files and they come from the kernel community. Everyone can reach the code at:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0.
The base code here has the statement of SPDX-License-Identifier: (GPL-2.0 OR MIT).

Thus, we may not be in the good position to re-license these code, and we didn't find the BSD-licensed version at any open community, so we just follow the required MIT license as an exception to DPDK.

Regards,
Junfeng

Quoted 2 lines
#11 lore raw

Re: [PATCH v2 01/10] net/gve: introduce GVE PMD base code

06/09/2022 11:31, Guo, Junfeng:

Quoted 38 lines

From: Thomas Monjalon <[email protected]>

Quoted 26 lines

01/09/2022 20:23, Stephen Hemminger:

Quoted 5 lines
Quoted 2 lines · On Thu, 1 Sep 2022 18:19:22 +0100 Ferruh Yigit <[email protected]> wrote:

Review

Quoted 2 lines
(quoted diff) +0 -0

b/drivers/net/gve/gve_adminq.c

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines

new file mode 100644
index 0000000000..8a724f12c6

drivers/net/gve/gve_adminq.c new file +2 -0
@@ -0,0 +1,925 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
Quoted 8 lines

drivers/net/gve/gve_adminq.c line 7

Quoted 5 lines
drivers/net/gve/gve_adminq.c new file +5 -0
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+

Can you please get approval for the MIT license from techboard, as Stephen highlighted in previous version?

Quoted 2 lines

I would prefer that it be BSD or dual licensed.
Although MIT and BSD-3 licenses are compatible, this is not something

Quoted 7 lines

techboard can decide

Quoted 1 line

it requires a statement from a knowledgeable open source lawyer (Intel

or LF).

Quoted 2 lines

Please fix the license to BSD and save lots of trouble.

+1 to change to BSD to avoid trouble.

Thanks for your concern and comments!
Yes, we are also willing to have these base code under BSD license.

Quoted 8 lines

Note that these code are not Intel files and they come from the kernel community. Everyone can reach the code at:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0.
The base code here has the statement of SPDX-License-Identifier: (GPL-2.0 OR MIT).

Thus, we may not be in the good position to re-license these code, and we didn't find the BSD-licensed version at any open community, so we just follow the required MIT license as an exception to DPDK.

I understand we are in trouble here.
We need the techboard to decide what to do.
If we want to go in the MIT direction, we need to ask the govboard.

#12 lore raw

PATCH v2 2/10 net/gve: add logs and OS specific implementation

In reply to Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/gve_adminq.h   |   2 +
 drivers/net/gve/gve_desc.h     |   2 +
 drivers/net/gve/gve_desc_dqo.h |   2 +
 drivers/net/gve/gve_logs.h     |  22 +++++
 drivers/net/gve/gve_osdep.h    | 149 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_register.h |   2 +
 6 files changed, 179 insertions(+)
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_osdep.h
drivers/net/gve/gve_adminq.h +2 -0
index c7114cc883..cd496760ae 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/gve_desc.h +2 -0
index 358755b7e0..627b9120dc 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/gve_desc_dqo.h +2 -0
index 0d533abcd1..5031752b43 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/gve_logs.h new file +22 -0
new file mode 100644
index 0000000000..a050253f59
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_init;
+extern int gve_logtype_driver;
+
+#define PMD_INIT_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_init, "%s(): " fmt "\n", \
+		__func__, ##args)
+
+#define PMD_DRV_LOG_RAW(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt, \
+		__func__, ## args)
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	PMD_DRV_LOG_RAW(level, fmt "\n", ## args)
+
+#endif
drivers/net/gve/gve_osdep.h new file +149 -0

149 lines not on this page · raw message

drivers/net/gve/gve_register.h +2 -0
index b65f336be2..a599c1a08e 100644
@@ -7,6 +7,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
Signature · 1 line
2.34.1
#13 lore raw

Re: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 8 lines · Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

<...>

Quoted 16 lines
drivers/net/gve/gve_logs.h new file +10 -0
new file mode 100644
index 0000000000..a050253f59
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_init;
+extern int gve_logtype_driver;
+

drivers/net/gve/gve_logs.h line 20

Quoted 10 lines
drivers/net/gve/gve_logs.h new file +10 -0
+#define PMD_INIT_LOG(level, fmt, args...) \
+       rte_log(RTE_LOG_ ## level, gve_logtype_init, "%s(): " fmt "\n", \
+               __func__, ##args)
+
+#define PMD_DRV_LOG_RAW(level, fmt, args...) \
+       rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt, \
+               __func__, ## args)
+ > +#define PMD_DRV_LOG(level, fmt, args...) \
+       PMD_DRV_LOG_RAW(level, fmt "\n", ## args)
+

Why 'PMD_DRV_LOG_RAW' is needed, why not directly use 'PMD_DRV_LOG'?

Do you really need two different log types? How do you differentiate 'init' & 'driver' types? As far as I can see there is mixed usage of them.

#14 lore raw

RE: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 47 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:21
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 8 lines · Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

<...>

Quoted 21 lines
drivers/net/gve/gve_logs.h new file +15 -0
new file mode 100644
index 0000000000..a050253f59
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_init;
+extern int gve_logtype_driver;
+
+#define PMD_INIT_LOG(level, fmt, args...) \
+       rte_log(RTE_LOG_ ## level, gve_logtype_init, "%s(): " fmt "\n", \
+               __func__, ##args)
+
+#define PMD_DRV_LOG_RAW(level, fmt, args...) \
Quoted 8 lines

drivers/net/gve/gve_logs.h line 21

Quoted 5 lines
drivers/net/gve/gve_logs.h new file +5 -0
+       rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt, \
+               __func__, ## args)
+ > +#define PMD_DRV_LOG(level, fmt, args...) \
+       PMD_DRV_LOG_RAW(level, fmt "\n", ## args)
+

Why 'PMD_DRV_LOG_RAW' is needed, why not directly use 'PMD_DRV_LOG'?

It seems that the _RAW macro was first introduced at i40e driver logs file. Since sometimes the trailing '\n' is added at the end of the log message in the base code, the PMD_DRV_LOG_RAW macro that will not add one is used to keep consistent of the new line character.

Well, looks that the macro PMD_DRV_LOG_RAW is somewhat redundant. I think it's ok to remove PMD_DRV_LOG_RAW and keep all the log messages end without the trailing '\n'. Thanks!

Quoted 4 lines

Do you really need two different log types? How do you differentiate 'init' & 'driver' types? As far as I can see there is mixed usage of them.

The PMD_INIT_LOG is used at the init stage, while the PMD_DRV_LOG is used at the driver normal running stage. I agree that there might be mixed usage of these two macros. I'll try to check all these usages and update them at correct conditions in the coming versions.
If you insist that only one log type is needed to keep the code clean, then I could update them as you expected. Thanks!

Regards,
Junfeng

#15 lore raw

Re: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 57 lines · Junfeng

CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

Quoted 45 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:21
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 8 lines · Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are specific for DPDK.

<...>

Quoted 21 lines
drivers/net/gve/gve_logs.h new file +15 -0
new file mode 100644
index 0000000000..a050253f59
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_init;
+extern int gve_logtype_driver;
+
+#define PMD_INIT_LOG(level, fmt, args...) \
+       rte_log(RTE_LOG_ ## level, gve_logtype_init, "%s(): " fmt "\n", \
+               __func__, ##args)
+
+#define PMD_DRV_LOG_RAW(level, fmt, args...) \
Quoted 8 lines

drivers/net/gve/gve_logs.h line 21

Quoted 5 lines
drivers/net/gve/gve_logs.h new file +5 -0
+       rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt, \
+               __func__, ## args)
+ > +#define PMD_DRV_LOG(level, fmt, args...) \
+       PMD_DRV_LOG_RAW(level, fmt "\n", ## args)
+

Why 'PMD_DRV_LOG_RAW' is needed, why not directly use 'PMD_DRV_LOG'?

Quoted 9 lines

It seems that the _RAW macro was first introduced at i40e driver logs file. Since sometimes the trailing '\n' is added at the end of the log message in the base code, the PMD_DRV_LOG_RAW macro that will not add one is used to keep consistent of the new line character.

Well, looks that the macro PMD_DRV_LOG_RAW is somewhat redundant. I think it's ok to remove PMD_DRV_LOG_RAW and keep all the log messages end without the trailing '\n'. Thanks!

Or you can add '\n' to 'PMD_DRV_LOG', to not change all logs. Only having two macro seems unnecessary.

Quoted 5 lines
Quoted 4 lines

Do you really need two different log types? How do you differentiate 'init' & 'driver' types? As far as I can see there is mixed usage of them.

Quoted 7 lines

The PMD_INIT_LOG is used at the init stage, while the PMD_DRV_LOG is used at the driver normal running stage. I agree that there might be mixed usage of these two macros. I'll try to check all these usages and update them at correct conditions in the coming versions.
If you insist that only one log type is needed to keep the code clean, then I could update them as you expected. Thanks!

I do not insist, but it looks like you are complicating things, is there really a benefit to have two different log types?

#16 lore raw

RE: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 83 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, September 7, 2022 19:17
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 1 line · Junfeng

CAUTION: This message has originated from an External Source. Please

use proper judgment and caution when opening attachments, clicking links, or responding to this email.

Quoted 10 lines
Quoted 10 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:21
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Wang, Haiyue

<[email protected]>

Quoted 8 lines
Quoted 8 lines

Subject: Re: [PATCH v2 02/10] net/gve: add logs and OS specific implementation

Quoted 3 lines · Junfeng Guo

Add GVE PMD logs.
Add some MACRO definitions and memory operations which are

specific

Quoted 38 lines
Quoted 29 lines
Quoted 5 lines

for DPDK.

<...>

Quoted 21 lines
drivers/net/gve/gve_logs.h new file +15 -0
new file mode 100644
index 0000000000..a050253f59
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_init;
+extern int gve_logtype_driver;
+
+#define PMD_INIT_LOG(level, fmt, args...) \
+       rte_log(RTE_LOG_ ## level, gve_logtype_init, "%s(): " fmt "\n", \
+               __func__, ##args)
+
+#define PMD_DRV_LOG_RAW(level, fmt, args...) \
Quoted 8 lines

drivers/net/gve/gve_logs.h line 21

Quoted 5 lines
drivers/net/gve/gve_logs.h new file +5 -0
+       rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt, \
+               __func__, ## args)
+ > +#define PMD_DRV_LOG(level, fmt, args...) \
+       PMD_DRV_LOG_RAW(level, fmt "\n", ## args)
+

Why 'PMD_DRV_LOG_RAW' is needed, why not directly use 'PMD_DRV_LOG'?

Quoted 1 line

It seems that the _RAW macro was first introduced at i40e driver logs

file.

Quoted 1 line

Since sometimes the trailing '\n' is added at the end of the log message

in

Quoted 4 lines

the base code, the PMD_DRV_LOG_RAW macro that will not add one is used to keep consistent of the new line character.

Well, looks that the macro PMD_DRV_LOG_RAW is somewhat

Quoted 8 lines

redundant.

Quoted 1 line

I think it's ok to remove PMD_DRV_LOG_RAW and keep all the log

messages

Quoted 2 lines

end without the trailing '\n'. Thanks!

Or you can add '\n' to 'PMD_DRV_LOG', to not change all logs. Only having two macro seems unnecessary.

Yes, already did as this form in the coming version gve pmd code. Thanks!

Quoted 14 lines
Quoted 6 lines
Quoted 5 lines

Do you really need two different log types? How do you differentiate 'init' & 'driver' types? As far as I can see there is mixed usage of them.

Quoted 7 lines

The PMD_INIT_LOG is used at the init stage, while the PMD_DRV_LOG is used at the driver normal running stage. I agree that there might be mixed usage of these two macros. I'll try to check all these usages and update them at correct conditions in the coming versions.
If you insist that only one log type is needed to keep the code clean, then I could update them as you expected. Thanks!

Quoted 2 lines

I do not insist, but it looks like you are complicating things, is there really a benefit to have two different log types?

Well, these two types may be used to show init/driver logs, respectively. But It seems that there is no such specific need to use two log types in the GVE PMD. Anyway, I think it is good time to keep the code clean and not just inherit from previous drivers. We can add new log type in the future if it's required. Thanks!

#17 lore raw

PATCH v2 3/10 net/gve: support device initialization

In reply to Junfeng Guo

Support device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close

---
 drivers/net/gve/gve.h        | 249 +++++++++++++++++++++++
 drivers/net/gve/gve_adminq.c |   1 +
 drivers/net/gve/gve_ethdev.c | 375 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |  13 ++
 drivers/net/gve/version.map  |   3 +
 drivers/net/meson.build      |   1 +
 6 files changed, 642 insertions(+)
 create mode 100644 drivers/net/gve/gve.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
drivers/net/gve/gve.h new file +249 -0
new file mode 100644
index 0000000000..704c88983c
@@ -0,0 +1,249 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include <ethdev_driver.h>
+#include <ethdev_pci.h>
+#include <rte_ether.h>
+
+#include "gve_desc.h"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR	0
+#define GVE_DB_BAR	2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES	1024
+
+/* A list of pages registered with the device during setup and used by a queue
+ * as buffers
+ */
+struct gve_queue_page_list {
+	uint32_t id; /* unique id */
+	uint32_t num_entries;
+	dma_addr_t *page_buses; /* the dma addrs of the pages */
+	const struct rte_memzone *mz;
+};
+
+/* A TX desc ring entry */
+union gve_tx_desc {
+	struct gve_tx_pkt_desc pkt; /* first desc for a packet */
+	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
+};
+
+struct gve_tx_queue {
+	volatile union gve_tx_desc *tx_desc_ring;
+	const struct rte_memzone *mz;
+	uint64_t tx_ring_phys_addr;
+
+	uint16_t nb_tx_desc;
+
+	/* Only valid for DQO_QPL queue format */
+	struct gve_queue_page_list *qpl;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+
+	uint16_t ntfy_id;
+	volatile rte_be32_t *ntfy_addr;
+
+	struct gve_priv *hw;
+	const struct rte_memzone *qres_mz;
+	struct gve_queue_resources *qres;
+
+	/* Only valid for DQO_RDA queue format */
+	struct gve_tx_queue *complq;
+};
+
+struct gve_rx_queue {
+	volatile struct gve_rx_desc *rx_desc_ring;
+	volatile union gve_rx_data_slot *rx_data_ring;
+	const struct rte_memzone *mz;
+	const struct rte_memzone *data_mz;
+	uint64_t rx_ring_phys_addr;
+
+	uint16_t nb_rx_desc;
+
+	volatile rte_be32_t *ntfy_addr;
+
+	/* only valid for GQI_QPL queue format */
+	struct gve_queue_page_list *qpl;
+
+	struct gve_priv *hw;
+	const struct rte_memzone *qres_mz;
+	struct gve_queue_resources *qres;
+
+	uint16_t port_id;
+	uint16_t queue_id;
+	uint16_t ntfy_id;
+	uint16_t rx_buf_len;
+
+	/* Only valid for DQO_RDA queue format */
+	struct gve_rx_queue *bufq;
+};
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+struct gve_priv {
+	struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+	const struct rte_memzone *irq_dbs_mz;
+	uint32_t mgmt_msix_idx;
+	rte_be32_t *cnt_array; /* array of num_event_counters */
+	const struct rte_memzone *cnt_array_mz;
+
+	uint16_t num_event_counters;
+	uint16_t tx_desc_cnt; /* txq size */
+	uint16_t rx_desc_cnt; /* rxq size */
+	uint16_t tx_pages_per_qpl; /* tx buffer length */
+	uint16_t rx_data_slot_cnt; /* rx buffer length */
+
+	/* Only valid for DQO_RDA queue format */
+	uint16_t tx_compq_size; /* tx completion queue size */
+	uint16_t rx_bufq_size; /* rx buff queue size */
+
+	uint64_t max_registered_pages;
+	uint64_t num_registered_pages; /* num pages registered with NIC */
+	uint16_t default_num_queues; /* default num queues to set up */
+	enum gve_queue_format queue_format; /* see enum gve_queue_format */
+	uint8_t enable_lsc;
+
+	uint16_t max_nb_txq;
+	uint16_t max_nb_rxq;
+	uint32_t num_ntfy_blks; /* spilt between TX and RX so must be even */
+
+	struct gve_registers __iomem *reg_bar0; /* see gve_register.h */
+	rte_be32_t __iomem *db_bar2; /* "array" of doorbells */
+	struct rte_pci_device *pci_dev;
+
+	/* Admin queue - see gve_adminq.h*/
+	union gve_adminq_command *adminq;
+	struct gve_dma_mem adminq_dma_mem;
+	uint32_t adminq_mask; /* masks prod_cnt to adminq size */
+	uint32_t adminq_prod_cnt; /* free-running count of AQ cmds executed */
+	uint32_t adminq_cmd_fail; /* free-running count of AQ cmds failed */
+	uint32_t adminq_timeouts; /* free-running count of AQ cmds timeouts */
+	/* free-running count of per AQ cmd executed */
+	uint32_t adminq_describe_device_cnt;
+	uint32_t adminq_cfg_device_resources_cnt;
+	uint32_t adminq_register_page_list_cnt;
+	uint32_t adminq_unregister_page_list_cnt;
+	uint32_t adminq_create_tx_queue_cnt;
+	uint32_t adminq_create_rx_queue_cnt;
+	uint32_t adminq_destroy_tx_queue_cnt;
+	uint32_t adminq_destroy_rx_queue_cnt;
+	uint32_t adminq_dcfg_device_resources_cnt;
+	uint32_t adminq_set_driver_parameter_cnt;
+	uint32_t adminq_report_stats_cnt;
+	uint32_t adminq_report_link_speed_cnt;
+	uint32_t adminq_get_ptype_map_cnt;
+
+	volatile uint32_t state_flags;
+
+	/* Gvnic device link speed from hypervisor. */
+	uint64_t link_speed;
+
+	uint16_t max_mtu;
+	struct rte_ether_addr dev_addr; /* mac address */
+
+	struct gve_queue_page_list *qpl;
+
+	struct gve_tx_queue **txqs;
+	struct gve_rx_queue **rxqs;
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+static inline bool gve_is_gqi(struct gve_priv *priv)
+{
+	return priv->queue_format == GVE_GQI_RDA_FORMAT ||
+		priv->queue_format == GVE_GQI_QPL_FORMAT;
+}
+
+static inline bool gve_get_admin_queue_ok(struct gve_priv *priv)
+{
+	return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+				       &priv->state_flags);
+}
+
+static inline void gve_set_admin_queue_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_set32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+			      &priv->state_flags);
+}
+
+static inline void gve_clear_admin_queue_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+				&priv->state_flags);
+}
+
+static inline bool gve_get_device_resources_ok(struct gve_priv *priv)
+{
+	return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+				       &priv->state_flags);
+}
+
+static inline void gve_set_device_resources_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+			      &priv->state_flags);
+}
+
+static inline void gve_clear_device_resources_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+				&priv->state_flags);
+}
+
+static inline bool gve_get_device_rings_ok(struct gve_priv *priv)
+{
+	return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+				       &priv->state_flags);
+}
+
+static inline void gve_set_device_rings_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+			      &priv->state_flags);
+}
+
+static inline void gve_clear_device_rings_ok(struct gve_priv *priv)
+{
+	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+				&priv->state_flags);
+}
+#endif /* _GVE_H_ */
drivers/net/gve/gve_adminq.c +1 -0
index 8a724f12c6..438ca2070e 100644
@@ -5,6 +5,7 @@
  * Copyright(C) 2022 Intel Corporation
  */
 
+#include "gve.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +375 -0

375 lines not on this page · raw message

drivers/net/gve/meson.build new file +13 -0
new file mode 100644
index 0000000000..9a22cc9abe
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'gve_adminq.c',
+        'gve_ethdev.c',
+)
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index e35652fe63..f1a0ee2cef 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.34.1
#18 lore raw

Re: [PATCH v2 03/10] net/gve: support device initialization

Quoted 134 lines · Junfeng Guo

Support device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close

---
  drivers/net/gve/gve.h        | 249 +++++++++++++++++++++++
  drivers/net/gve/gve_adminq.c |   1 +
  drivers/net/gve/gve_ethdev.c | 375 +++++++++++++++++++++++++++++++++++
  drivers/net/gve/meson.build  |  13 ++
  drivers/net/gve/version.map  |   3 +
  drivers/net/meson.build      |   1 +
  6 files changed, 642 insertions(+)

create mode 100644 drivers/net/gve/gve.h
create mode 100644 drivers/net/gve/gve_ethdev.c
create mode 100644 drivers/net/gve/meson.build
create mode 100644 drivers/net/gve/version.map

drivers/net/gve/gve.h new file +105 -0
new file mode 100644
index 0000000000..704c88983c
@@ -0,0 +1,249 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include <ethdev_driver.h>
+#include <ethdev_pci.h>
+#include <rte_ether.h>
+
+#include "gve_desc.h"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID       0x1ae0
+#endif
+
+#define GVE_DEV_ID             0x0042
+
+#define GVE_REG_BAR    0
+#define GVE_DB_BAR     2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX           3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES 1024
+
+/* A list of pages registered with the device during setup and used by a queue
+ * as buffers
+ */
+struct gve_queue_page_list {
+       uint32_t id; /* unique id */
+       uint32_t num_entries;
+       dma_addr_t *page_buses; /* the dma addrs of the pages */
+       const struct rte_memzone *mz;
+};
+
+/* A TX desc ring entry */
+union gve_tx_desc {
+       struct gve_tx_pkt_desc pkt; /* first desc for a packet */
+       struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
+};
+
+struct gve_tx_queue {
+       volatile union gve_tx_desc *tx_desc_ring;
+       const struct rte_memzone *mz;
+       uint64_t tx_ring_phys_addr;
+
+       uint16_t nb_tx_desc;
+
+       /* Only valid for DQO_QPL queue format */
+       struct gve_queue_page_list *qpl;
+
+       uint16_t port_id;
+       uint16_t queue_id;
+
+       uint16_t ntfy_id;
+       volatile rte_be32_t *ntfy_addr;
+
+       struct gve_priv *hw;
+       const struct rte_memzone *qres_mz;
+       struct gve_queue_resources *qres;
+
+       /* Only valid for DQO_RDA queue format */
+       struct gve_tx_queue *complq;
+};
+
+struct gve_rx_queue {
+       volatile struct gve_rx_desc *rx_desc_ring;
+       volatile union gve_rx_data_slot *rx_data_ring;
+       const struct rte_memzone *mz;
+       const struct rte_memzone *data_mz;
+       uint64_t rx_ring_phys_addr;
+
+       uint16_t nb_rx_desc;
+
+       volatile rte_be32_t *ntfy_addr;
+
+       /* only valid for GQI_QPL queue format */
+       struct gve_queue_page_list *qpl;
+
+       struct gve_priv *hw;
+       const struct rte_memzone *qres_mz;
+       struct gve_queue_resources *qres;
+
+       uint16_t port_id;
+       uint16_t queue_id;
+       uint16_t ntfy_id;
+       uint16_t rx_buf_len;
+
+       /* Only valid for DQO_RDA queue format */
+       struct gve_rx_queue *bufq;
+};
+
+struct gve_irq_db {
+       rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+       uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+       uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {

drivers/net/gve/gve.h line 115

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+       struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+       GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+       GVE_GQI_RDA_FORMAT           = 0x1, /* GQI Raw Addressing */
+       GVE_GQI_QPL_FORMAT           = 0x2, /* GQI Queue Page List */
+       GVE_DQO_RDA_FORMAT           = 0x3, /* DQO Raw Addressing */
+};
+

Shouldn't these queue format information be part of 'base' file? Both for licensing issues also to cover the case it is updated in the google repo. But if some dpdk related information is required, what do you think to split this file as one for base folder, other for dpdk?

Quoted 57 lines
drivers/net/gve/gve.h new file +57 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */
+
+       /* Only valid for DQO_RDA queue format */
+       uint16_t tx_compq_size; /* tx completion queue size */
+       uint16_t rx_bufq_size; /* rx buff queue size */
+
+       uint64_t max_registered_pages;
+       uint64_t num_registered_pages; /* num pages registered with NIC */
+       uint16_t default_num_queues; /* default num queues to set up */
+       enum gve_queue_format queue_format; /* see enum gve_queue_format */
+       uint8_t enable_lsc;
+
+       uint16_t max_nb_txq;
+       uint16_t max_nb_rxq;
+       uint32_t num_ntfy_blks; /* spilt between TX and RX so must be even */
+
+       struct gve_registers __iomem *reg_bar0; /* see gve_register.h */
+       rte_be32_t __iomem *db_bar2; /* "array" of doorbells */
+       struct rte_pci_device *pci_dev;
+
+       /* Admin queue - see gve_adminq.h*/
+       union gve_adminq_command *adminq;
+       struct gve_dma_mem adminq_dma_mem;
+       uint32_t adminq_mask; /* masks prod_cnt to adminq size */
+       uint32_t adminq_prod_cnt; /* free-running count of AQ cmds executed */
+       uint32_t adminq_cmd_fail; /* free-running count of AQ cmds failed */
+       uint32_t adminq_timeouts; /* free-running count of AQ cmds timeouts */
+       /* free-running count of per AQ cmd executed */
+       uint32_t adminq_describe_device_cnt;
+       uint32_t adminq_cfg_device_resources_cnt;
+       uint32_t adminq_register_page_list_cnt;
+       uint32_t adminq_unregister_page_list_cnt;
+       uint32_t adminq_create_tx_queue_cnt;
+       uint32_t adminq_create_rx_queue_cnt;
+       uint32_t adminq_destroy_tx_queue_cnt;
+       uint32_t adminq_destroy_rx_queue_cnt;
+       uint32_t adminq_dcfg_device_resources_cnt;
+       uint32_t adminq_set_driver_parameter_cnt;
+       uint32_t adminq_report_stats_cnt;
+       uint32_t adminq_report_link_speed_cnt;
+       uint32_t adminq_get_ptype_map_cnt;
+
+       volatile uint32_t state_flags;
+
+       /* Gvnic device link speed from hypervisor. */
+       uint64_t link_speed;

drivers/net/gve/gve.h line 182

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+
+       uint16_t max_mtu;
+       struct rte_ether_addr dev_addr; /* mac address */
+
+       struct gve_queue_page_list *qpl;
+
+       struct gve_tx_queue **txqs;
+       struct gve_rx_queue **rxqs;
+};
+

The device private data is provided fully here, as well as 'gve_rx_queue' and 'gve_tx_queue' structs etc, although most of the files not used at this stage and not clear why needed.

Instead of adding full structs, can you please add here bare minimum structs, whatever used at this point, and as they are used in .c file, keep adding them in structs too? This clarifies what is used/added for specific features, also lets us easily figure out unused fields / clutter in the header files.

Quoted 56 lines
drivers/net/gve/gve.h new file +56 -0
+enum gve_state_flags_bit {
+       GVE_PRIV_FLAGS_ADMIN_QUEUE_OK           = 1,
+       GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK      = 2,
+       GVE_PRIV_FLAGS_DEVICE_RINGS_OK          = 3,
+       GVE_PRIV_FLAGS_NAPI_ENABLED             = 4,
+};
+
+static inline bool gve_is_gqi(struct gve_priv *priv)
+{
+       return priv->queue_format == GVE_GQI_RDA_FORMAT ||
+               priv->queue_format == GVE_GQI_QPL_FORMAT;
+}
+
+static inline bool gve_get_admin_queue_ok(struct gve_priv *priv)
+{
+       return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+                                      &priv->state_flags);
+}
+
+static inline void gve_set_admin_queue_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_set32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+                             &priv->state_flags);
+}
+
+static inline void gve_clear_admin_queue_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+                               &priv->state_flags);
+}
+
+static inline bool gve_get_device_resources_ok(struct gve_priv *priv)
+{
+       return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+                                      &priv->state_flags);
+}
+
+static inline void gve_set_device_resources_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+                             &priv->state_flags);
+}
+
+static inline void gve_clear_device_resources_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+                               &priv->state_flags);
+}
+
+static inline bool gve_get_device_rings_ok(struct gve_priv *priv)
+{
+       return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+                                      &priv->state_flags);
+}
+
+static inline void gve_set_device_rings_ok(struct gve_priv *priv)

drivers/net/gve/gve.h line 248

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+{
+       rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+                             &priv->state_flags);
+}
+
+static inline void gve_clear_device_rings_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+                               &priv->state_flags);
+}

If this is dpdk file, not base file, please follow dpdk coding convention, like return type should be in separate line.

Quoted 21 lines
drivers/net/gve/gve.h new file +1 -0
+#endif /* _GVE_H_ */
drivers/net/gve/gve_adminq.c +1 -0
index 8a724f12c6..438ca2070e 100644
@@ -5,6 +5,7 @@
   * Copyright(C) 2022 Intel Corporation
   */
 
+#include "gve.h"
  #include "gve_adminq.h"
  #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +2 -0
new file mode 100644
index 0000000000..f10f273f7d
@@ -0,0 +1,375 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation

drivers/net/gve/gve_ethdev.c line 12

Quoted 10 lines
drivers/net/gve/gve_ethdev.c new file +10 -0
+ */
+#include <linux/pci_regs.h>
+
+#include "gve.h"
+#include "gve_adminq.h"
+#include "gve_register.h"
+
+#define GVE_VERSION            "1.3.0"
+#define GVE_VERSION_PREFIX     "GVE-"
+

Again, shouldn't these come from base file (google repo)?

Perhaps can be good to discuss what is the base file update strategy/plan for future?
It can be easier if you can drop some external files with minimum change in dpdk files, that can be done by grouping external content in spefic files. Qi has lots of experince on this and I believe he can privide useful insight.

Quoted 45 lines
drivers/net/gve/gve_ethdev.c new file +45 -0
+const char gve_version_str[] = GVE_VERSION;
+static const char gve_version_prefix[] = GVE_VERSION_PREFIX;
+
+static void
+gve_write_version(uint8_t *driver_version_register)
+{
+       const char *c = gve_version_prefix;
+
+       while (*c) {
+               writeb(*c, driver_version_register);
+               c++;
+       }
+
+       c = gve_version_str;
+       while (*c) {
+               writeb(*c, driver_version_register);
+               c++;
+       }
+       writeb('\n', driver_version_register);
+}
+
+static int
+gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+       return 0;
+}
+
+static int
+gve_dev_start(struct rte_eth_dev *dev)
+{
+       dev->data->dev_started = 1;
+
+       return 0;
+}
+
+static int
+gve_dev_stop(struct rte_eth_dev *dev)
+{
+       dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+       dev->data->dev_started = 0;
+
+       return 0;
+}
+
+static int

drivers/net/gve/gve_ethdev.c line 67

Quoted 10 lines
drivers/net/gve/gve_ethdev.c new file +10 -0
+gve_dev_close(struct rte_eth_dev *dev)
+{
+       int err = 0;
+
+       if (dev->data->dev_started) {
+               err = gve_dev_stop(dev);
+               if (err != 0)
+                       PMD_DRV_LOG(ERR, "Failed to stop dev.");
+       }
+

Just a reminder that in 'close' driver should free all the resources, if there is previously allocated memomory, it should be freed now.

#19 lore raw

RE: [PATCH v2 03/10] net/gve: support device initialization

Quoted 156 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:22
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v2 03/10] net/gve: support device initialization

Quoted 7 lines · Junfeng Guo

Support device init and the fowllowing devops:
- dev_configure
- dev_start
- dev_stop
- dev_close

Quoted 7 lines
---
  drivers/net/gve/gve.h        | 249 +++++++++++++++++++++++
  drivers/net/gve/gve_adminq.c |   1 +
  drivers/net/gve/gve_ethdev.c | 375

+++++++++++++++++++++++++++++++++++

Quoted 34 lines
  drivers/net/gve/meson.build  |  13 ++
  drivers/net/gve/version.map  |   3 +
  drivers/net/meson.build      |   1 +
  6 files changed, 642 insertions(+)

create mode 100644 drivers/net/gve/gve.h
create mode 100644 drivers/net/gve/gve_ethdev.c create mode 100644 drivers/net/gve/meson.build create mode 100644 drivers/net/gve/version.map

drivers/net/gve/gve.h new file +19 -0
new file mode 100644
index 0000000000..704c88983c
@@ -0,0 +1,249 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include <ethdev_driver.h>
+#include <ethdev_pci.h>
+#include <rte_ether.h>
+
+#include "gve_desc.h"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID       0x1ae0
+#endif
+
+#define GVE_DEV_ID             0x0042
+

drivers/net/gve/gve.h line 29

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+#define GVE_REG_BAR    0
+#define GVE_DB_BAR     2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX           3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES 1024
+
+/* A list of pages registered with the device during setup and used by a

queue

Quoted 71 lines
drivers/net/gve/gve.h new file +71 -0
+ * as buffers
+ */
+struct gve_queue_page_list {
+       uint32_t id; /* unique id */
+       uint32_t num_entries;
+       dma_addr_t *page_buses; /* the dma addrs of the pages */
+       const struct rte_memzone *mz;
+};
+
+/* A TX desc ring entry */
+union gve_tx_desc {
+       struct gve_tx_pkt_desc pkt; /* first desc for a packet */
+       struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
+};
+
+struct gve_tx_queue {
+       volatile union gve_tx_desc *tx_desc_ring;
+       const struct rte_memzone *mz;
+       uint64_t tx_ring_phys_addr;
+
+       uint16_t nb_tx_desc;
+
+       /* Only valid for DQO_QPL queue format */
+       struct gve_queue_page_list *qpl;
+
+       uint16_t port_id;
+       uint16_t queue_id;
+
+       uint16_t ntfy_id;
+       volatile rte_be32_t *ntfy_addr;
+
+       struct gve_priv *hw;
+       const struct rte_memzone *qres_mz;
+       struct gve_queue_resources *qres;
+
+       /* Only valid for DQO_RDA queue format */
+       struct gve_tx_queue *complq;
+};
+
+struct gve_rx_queue {
+       volatile struct gve_rx_desc *rx_desc_ring;
+       volatile union gve_rx_data_slot *rx_data_ring;
+       const struct rte_memzone *mz;
+       const struct rte_memzone *data_mz;
+       uint64_t rx_ring_phys_addr;
+
+       uint16_t nb_rx_desc;
+
+       volatile rte_be32_t *ntfy_addr;
+
+       /* only valid for GQI_QPL queue format */
+       struct gve_queue_page_list *qpl;
+
+       struct gve_priv *hw;
+       const struct rte_memzone *qres_mz;
+       struct gve_queue_resources *qres;
+
+       uint16_t port_id;
+       uint16_t queue_id;
+       uint16_t ntfy_id;
+       uint16_t rx_buf_len;
+
+       /* Only valid for DQO_RDA queue format */
+       struct gve_rx_queue *bufq;
+};
+
+struct gve_irq_db {
+       rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {

drivers/net/gve/gve.h

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+       uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+       uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+       struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+       GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified

*/

Quoted 11 lines

drivers/net/gve/gve.h line 115

Quoted 5 lines
drivers/net/gve/gve.h new file +5 -0
+       GVE_GQI_RDA_FORMAT           = 0x1, /* GQI Raw Addressing */
+       GVE_GQI_QPL_FORMAT           = 0x2, /* GQI Queue Page List */
+       GVE_DQO_RDA_FORMAT           = 0x3, /* DQO Raw Addressing */
+};
+

Shouldn't these queue format information be part of 'base' file? Both for licensing issues also to cover the case it is updated in the google repo.
But if some dpdk related information is required, what do you think to split this file as one for base folder, other for dpdk?

Yes, current solution is to move these base-related code into another gve.h file and rename this file as gve_ethdev.h to match the dpdk style. Will update the solution in the coming version.
Thanks!

Quoted 79 lines
Quoted 10 lines
drivers/net/gve/gve.h new file +9 -0
+struct gve_priv {
+       struct gve_irq_db *irq_dbs; /* array of num_ntfy_blks */
+       const struct rte_memzone *irq_dbs_mz;
+       uint32_t mgmt_msix_idx;
+       rte_be32_t *cnt_array; /* array of num_event_counters */
+       const struct rte_memzone *cnt_array_mz;
+
+       uint16_t num_event_counters;
+       uint16_t tx_desc_cnt; /* txq size */

drivers/net/gve/gve.h

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+       uint16_t rx_desc_cnt; /* rxq size */
+       uint16_t tx_pages_per_qpl; /* tx buffer length */
+       uint16_t rx_data_slot_cnt; /* rx buffer length */
+
+       /* Only valid for DQO_RDA queue format */
+       uint16_t tx_compq_size; /* tx completion queue size */
+       uint16_t rx_bufq_size; /* rx buff queue size */
+
+       uint64_t max_registered_pages;
+       uint64_t num_registered_pages; /* num pages registered with NIC

*/

drivers/net/gve/gve.h line 136

Quoted 2 lines
drivers/net/gve/gve.h new file +2 -0
+       uint16_t default_num_queues; /* default num queues to set up */
+       enum gve_queue_format queue_format; /* see enum

gve_queue_format */

drivers/net/gve/gve.h

Quoted 5 lines
drivers/net/gve/gve.h new file +5 -0
+       uint8_t enable_lsc;
+
+       uint16_t max_nb_txq;
+       uint16_t max_nb_rxq;
+       uint32_t num_ntfy_blks; /* spilt between TX and RX so must be

even */

drivers/net/gve/gve.h

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+
+       struct gve_registers __iomem *reg_bar0; /* see gve_register.h */
+       rte_be32_t __iomem *db_bar2; /* "array" of doorbells */
+       struct rte_pci_device *pci_dev;
+
+       /* Admin queue - see gve_adminq.h*/
+       union gve_adminq_command *adminq;
+       struct gve_dma_mem adminq_dma_mem;
+       uint32_t adminq_mask; /* masks prod_cnt to adminq size */
+       uint32_t adminq_prod_cnt; /* free-running count of AQ cmds

executed */

drivers/net/gve/gve.h line 152

Quoted 1 line
drivers/net/gve/gve.h new file +1 -0
+       uint32_t adminq_cmd_fail; /* free-running count of AQ cmds failed

*/

drivers/net/gve/gve.h line 153

Quoted 1 line
drivers/net/gve/gve.h new file +1 -0
+       uint32_t adminq_timeouts; /* free-running count of AQ cmds

timeouts */

Quoted 19 lines
drivers/net/gve/gve.h new file +19 -0
+       /* free-running count of per AQ cmd executed */
+       uint32_t adminq_describe_device_cnt;
+       uint32_t adminq_cfg_device_resources_cnt;
+       uint32_t adminq_register_page_list_cnt;
+       uint32_t adminq_unregister_page_list_cnt;
+       uint32_t adminq_create_tx_queue_cnt;
+       uint32_t adminq_create_rx_queue_cnt;
+       uint32_t adminq_destroy_tx_queue_cnt;
+       uint32_t adminq_destroy_rx_queue_cnt;
+       uint32_t adminq_dcfg_device_resources_cnt;
+       uint32_t adminq_set_driver_parameter_cnt;
+       uint32_t adminq_report_stats_cnt;
+       uint32_t adminq_report_link_speed_cnt;
+       uint32_t adminq_get_ptype_map_cnt;
+
+       volatile uint32_t state_flags;
+
+       /* Gvnic device link speed from hypervisor. */
+       uint64_t link_speed;

drivers/net/gve/gve.h line 182

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+
+       uint16_t max_mtu;
+       struct rte_ether_addr dev_addr; /* mac address */
+
+       struct gve_queue_page_list *qpl;
+
+       struct gve_tx_queue **txqs;
+       struct gve_rx_queue **rxqs;
+};
+

The device private data is provided fully here, as well as 'gve_rx_queue' and 'gve_tx_queue' structs etc, although most of the files not used at this stage and not clear why needed.

Quoted 5 lines

Instead of adding full structs, can you please add here bare minimum structs, whatever used at this point, and as they are used in .c file, keep adding them in structs too?
This clarifies what is used/added for specific features, also lets us easily figure out unused fields / clutter in the header files.

Sure, will update in the coming version. Thanks!

Quoted 67 lines
Quoted 7 lines
drivers/net/gve/gve.h new file +6 -0
+enum gve_state_flags_bit {
+       GVE_PRIV_FLAGS_ADMIN_QUEUE_OK           = 1,
+       GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK      = 2,
+       GVE_PRIV_FLAGS_DEVICE_RINGS_OK          = 3,
+       GVE_PRIV_FLAGS_NAPI_ENABLED             = 4,
+};

drivers/net/gve/gve.h line 201

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+
+static inline bool gve_is_gqi(struct gve_priv *priv)
+{
+       return priv->queue_format == GVE_GQI_RDA_FORMAT ||
+               priv->queue_format == GVE_GQI_QPL_FORMAT;
+}
+
+static inline bool gve_get_admin_queue_ok(struct gve_priv *priv)
+{
+

return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,

Quoted 8 lines
drivers/net/gve/gve.h new file +8 -0
+                                      &priv->state_flags);
+}
+
+static inline void gve_set_admin_queue_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_set32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+                             &priv->state_flags);
+}

drivers/net/gve/gve.h line 219

Quoted 10 lines
drivers/net/gve/gve.h new file +10 -0
+
+static inline void gve_clear_admin_queue_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_ADMIN_QUEUE_OK,
+                               &priv->state_flags);
+}
+
+static inline bool gve_get_device_resources_ok(struct gve_priv *priv)
+{
+

return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_O K,

drivers/net/gve/gve.h line 231

Quoted 12 lines
drivers/net/gve/gve.h new file +12 -0
+                                      &priv->state_flags);
+}
+
+static inline void gve_set_device_resources_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,
+                             &priv->state_flags);
+}
+
+static inline void gve_clear_device_resources_ok(struct gve_priv *priv)
+{
+

rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK,

drivers/net/gve/gve.h line 237

Quoted 6 lines
drivers/net/gve/gve.h new file +6 -0
+                               &priv->state_flags);
+}
+
+static inline bool gve_get_device_rings_ok(struct gve_priv *priv)
+{
+

return !!rte_bit_relaxed_get32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,

Quoted 9 lines
drivers/net/gve/gve.h new file +9 -0
+                                      &priv->state_flags);
+}
+
+static inline void gve_set_device_rings_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_set32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+                             &priv->state_flags);
+}
+
Quoted 8 lines

drivers/net/gve/gve.h line 248

Quoted 5 lines
drivers/net/gve/gve.h new file +5 -0
+static inline void gve_clear_device_rings_ok(struct gve_priv *priv)
+{
+       rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
+                               &priv->state_flags);
+}

If this is dpdk file, not base file, please follow dpdk coding convention, like return type should be in separate line.

Sure, will update in the coming version patchset. Thanks!

Quoted 37 lines

Review

Quoted 3 lines
drivers/net/gve/gve.h new file +1 -0
+#endif /* _GVE_H_ */
(quoted diff) +0 -0

b/drivers/net/gve/gve_adminq.c

Review

Quoted 12 lines

index 8a724f12c6..438ca2070e 100644

drivers/net/gve/gve_adminq.c +1 -0
@@ -5,6 +5,7 @@
   * Copyright(C) 2022 Intel Corporation
   */
 
+#include "gve.h"
  #include "gve_adminq.h"
  #include "gve_register.h"
 
(quoted diff) +0 -0

b/drivers/net/gve/gve_ethdev.c

Quoted 7 lines

new file mode 100644
index 0000000000..f10f273f7d

drivers/net/gve/gve_ethdev.c new file +2 -0
@@ -0,0 +1,375 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation

drivers/net/gve/gve_ethdev.c line 12

Quoted 10 lines
drivers/net/gve/gve_ethdev.c new file +10 -0
+ */
+#include <linux/pci_regs.h>
+
+#include "gve.h"
+#include "gve_adminq.h"
+#include "gve_register.h"
+
+#define GVE_VERSION            "1.3.0"
+#define GVE_VERSION_PREFIX     "GVE-"
+

Again, shouldn't these come from base file (google repo)?

Quoted 7 lines

Perhaps can be good to discuss what is the base file update strategy/plan for future?
It can be easier if you can drop some external files with minimum change in dpdk files, that can be done by grouping external content in spefic files.
Qi has lots of experince on this and I believe he can privide useful insight.

Yes, need to keep the base file at the base folder in the future. Thanks!

Quoted 51 lines
Quoted 51 lines
drivers/net/gve/gve_ethdev.c new file +50 -0
+const char gve_version_str[] = GVE_VERSION;
+static const char gve_version_prefix[] = GVE_VERSION_PREFIX;
+
+static void
+gve_write_version(uint8_t *driver_version_register)
+{
+       const char *c = gve_version_prefix;
+
+       while (*c) {
+               writeb(*c, driver_version_register);
+               c++;
+       }
+
+       c = gve_version_str;
+       while (*c) {
+               writeb(*c, driver_version_register);
+               c++;
+       }
+       writeb('\n', driver_version_register);
+}
+
+static int
+gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
+{
+       return 0;
+}
+
+static int
+gve_dev_start(struct rte_eth_dev *dev)
+{
+       dev->data->dev_started = 1;
+
+       return 0;
+}
+
+static int
+gve_dev_stop(struct rte_eth_dev *dev)
+{
+       dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+       dev->data->dev_started = 0;
+
+       return 0;
+}
+
+static int
+gve_dev_close(struct rte_eth_dev *dev)
+{
+       int err = 0;
+
+       if (dev->data->dev_started) {
Quoted 8 lines

drivers/net/gve/gve_ethdev.c line 67

Quoted 5 lines
drivers/net/gve/gve_ethdev.c new file +5 -0
+               err = gve_dev_stop(dev);
+               if (err != 0)
+                       PMD_DRV_LOG(ERR, "Failed to stop dev.");
+       }
+

Just a reminder that in 'close' driver should free all the resources, if there is previously allocated memomory, it should be freed now.

Thanks! Will update in the coming version.

#20 lore raw

Re: [PATCH v2 03/10] net/gve: support device initialization

In reply to Junfeng Guo

drivers/net/gve/version.map line 3

Quoted 9 lines · Junfeng Guo
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {
+       local: *;
+};

It should be 'DPDK_23' now.

@Bruce, @David, @Thomas, @Andrew,

This is one of the common errors as far as I can see, and many times the .map is just empty and required because of the build process.

Can we add a build system magic to let libraries/drivers use a common template for the .map file, unless library/driver needs something special? This prevents this trivial error also reduces overhead for the
'yy.mm-rc0' commit.

#21 lore raw

PATCH v2 4/10 net/gve: add link update support

In reply to Junfeng Guo

Support dev_ops link_update.

---
 drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
drivers/net/gve/gve_ethdev.c +30 -0
index f10f273f7d..435115c047 100644
@@ -37,10 +37,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_INIT_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -73,6 +102,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.34.1
#22 lore raw

Re: [PATCH v2 04/10] net/gve: add link update support

Quoted 25 lines · Junfeng Guo

Support dev_ops link_update.

---
  drivers/net/gve/gve_ethdev.c | 30 ++++++++++++++++++++++++++++++
  1 file changed, 30 insertions(+)
drivers/net/gve/gve_ethdev.c +8 -0
index f10f273f7d..435115c047 100644
@@ -37,10 +37,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
         return 0;
  }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       struct rte_eth_link link;
+       int err;
+
+       memset(&link, 0, sizeof(link));

drivers/net/gve/gve_ethdev.c gve_dev_configure(__rte_unused struct rte_eth_dev *dev) · line 57

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+       link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+       link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+       if (!dev->data->dev_started) {
+               link.link_status = RTE_ETH_LINK_DOWN;
+               link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+       } else {
+               link.link_status = RTE_ETH_LINK_UP;
+               PMD_INIT_LOG(DEBUG, "Get link status from hw");
+               err = gve_adminq_report_link_speed(priv);

As far as I can see the API is calling an adminq command, is this command blocking until link is up? If so is there a non blocking version to utilize 'wait_to_complete', instead of ignoring it?

Also what will happen if 'start()' dev_ops called but cable is not plugged in at all, won't this set link status still to "UP"?

#23 lore raw

RE: [PATCH v2 04/10] net/gve: add link update support

Quoted 49 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:23
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH v2 04/10] net/gve: add link update support

Quoted 7 lines · Junfeng Guo

Support dev_ops link_update.

---
  drivers/net/gve/gve_ethdev.c | 30

++++++++++++++++++++++++++++++

Review

Quoted 3 lines
  1 file changed, 30 insertions(+)
(quoted diff) +0 -0

b/drivers/net/gve/gve_ethdev.c

drivers/net/gve/gve_ethdev.c gve_dev_configure(__rte_unused struct

Quoted 4 lines

index f10f273f7d..435115c047 100644

drivers/net/gve/gve_ethdev.c +0 -0
@@ -37,10 +37,39 @@ gve_dev_configure(__rte_unused struct

rte_eth_dev *dev)

drivers/net/gve/gve_ethdev.c gve_dev_configure(__rte_unused struct rte_eth_dev *dev)

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +2 -0
         return 0;
  }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int

wait_to_complete)

Quoted 12 lines
drivers/net/gve/gve_ethdev.c +12 -0
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       struct rte_eth_link link;
+       int err;
+
+       memset(&link, 0, sizeof(link));
+       link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+       link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+       if (!dev->data->dev_started) {
+               link.link_status = RTE_ETH_LINK_DOWN;
+               link.link_speed = RTE_ETH_SPEED_NUM_NONE;
Quoted 8 lines

drivers/net/gve/gve_ethdev.c gve_dev_configure(__rte_unused struct rte_eth_dev *dev) · line 57

Quoted 4 lines
drivers/net/gve/gve_ethdev.c +4 -0
+       } else {
+               link.link_status = RTE_ETH_LINK_UP;
+               PMD_INIT_LOG(DEBUG, "Get link status from hw");
+               err = gve_adminq_report_link_speed(priv);

As far as I can see the API is calling an adminq command, is this command blocking until link is up? If so is there a non blocking version to utilize 'wait_to_complete', instead of ignoring it?

Yes, getting the link speed via an adminq command here is an blocking until, and that's the only method we can utilize. As for the non-blocking version, it depends on the real behavior of the HW (or backend), which is like a black-box to us. There is no HW register to store the link status info. Seems the 'wait_to_complete' cannot be used here. Thanks!

Quoted 3 lines

Also what will happen if 'start()' dev_ops called but cable is not plugged in at all, won't this set link status still to "UP"?

Yes, that could be a terrible situation. Since this driver is running on the cloud environment, we can only assume that the HW & backend part work well when we run our driver. Thanks!

Quoted 1 line
#24 lore raw

PATCH v2 5/10 net/gve: add MTU set support

In reply to Junfeng Guo

Support dev_ops mtu_set.

---
 drivers/net/gve/gve_ethdev.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
drivers/net/gve/gve_ethdev.c +29 -0
index 435115c047..26b45fde6f 100644
@@ -97,12 +97,41 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	dev->data->dev_conf.rxmode.mtu = mtu + RTE_ETHER_HDR_LEN;
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.34.1
#25 lore raw

PATCH v2 6/10 net/gve: add queue operations

In reply to Junfeng Guo

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve.h        |  52 +++++++++
 drivers/net/gve/gve_ethdev.c | 203 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 683 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve.h +52 -0
index 704c88983c..a53a852a5f 100644
@@ -23,6 +23,9 @@
 /* 1 for management, 1 for rx, 1 for tx */
 #define GVE_MIN_MSIX		3
 
+#define GVE_DEFAULT_RX_FREE_THRESH  512
+#define GVE_DEFAULT_TX_FREE_THRESH  256
+
 /* PTYPEs are always 10 bits. */
 #define GVE_NUM_PTYPES	1024
 
@@ -42,15 +45,35 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+struct gve_tx_iovec {
+	uint32_t iov_base; /* offset in fifo */
+	uint32_t iov_len;
+};
+
 struct gve_tx_queue {
 	volatile union gve_tx_desc *tx_desc_ring;
 	const struct rte_memzone *mz;
 	uint64_t tx_ring_phys_addr;
+	struct rte_mbuf **sw_ring;
+	volatile rte_be32_t *qtx_tail;
+	volatile rte_be32_t *qtx_head;
 
+	uint32_t tx_tail;
 	uint16_t nb_tx_desc;
+	uint16_t nb_free;
+	uint32_t next_to_clean;
+	uint16_t free_thresh;
 
 	/* Only valid for DQO_QPL queue format */
+	uint16_t sw_tail;
+	uint16_t sw_ntc;
+	uint16_t sw_nb_free;
+	uint32_t fifo_size;
+	uint32_t fifo_head;
+	uint32_t fifo_avail;
+	uint64_t fifo_base;
 	struct gve_queue_page_list *qpl;
+	struct gve_tx_iovec *iov_ring;
 
 	uint16_t port_id;
 	uint16_t queue_id;
@@ -64,6 +87,8 @@ struct gve_tx_queue {
 
 	/* Only valid for DQO_RDA queue format */
 	struct gve_tx_queue *complq;
+
+	uint8_t is_gqi_qpl;
 };
 
 struct gve_rx_queue {
@@ -72,9 +97,17 @@ struct gve_rx_queue {
 	const struct rte_memzone *mz;
 	const struct rte_memzone *data_mz;
 	uint64_t rx_ring_phys_addr;
+	struct rte_mbuf **sw_ring;
+	struct rte_mempool *mpool;
 
+	uint16_t rx_tail;
 	uint16_t nb_rx_desc;
+	uint16_t expected_seqno; /* the next expected seqno */
+	uint16_t free_thresh;
+	uint32_t next_avail;
+	uint32_t nb_avail;
 
+	volatile rte_be32_t *qrx_tail;
 	volatile rte_be32_t *ntfy_addr;
 
 	/* only valid for GQI_QPL queue format */
@@ -91,6 +124,8 @@ struct gve_rx_queue {
 
 	/* Only valid for DQO_RDA queue format */
 	struct gve_rx_queue *bufq;
+
+	uint8_t is_gqi_qpl;
 };
 
 struct gve_irq_db {
@@ -246,4 +281,21 @@ static inline void gve_clear_device_rings_ok(struct gve_priv *priv)
 	rte_bit_relaxed_clear32(GVE_PRIV_FLAGS_DEVICE_RINGS_OK,
 				&priv->state_flags);
 }
+
+int
+gve_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_desc,
+		   unsigned int socket_id, const struct rte_eth_rxconf *conf,
+		   struct rte_mempool *pool);
+int
+gve_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id, uint16_t nb_desc,
+		   unsigned int socket_id, const struct rte_eth_txconf *conf);
+
+void gve_tx_queue_release(void *txq);
+
+void gve_rx_queue_release(void *rxq);
+
+void gve_stop_tx_queues(struct rte_eth_dev *dev);
+
+void gve_stop_rx_queues(struct rte_eth_dev *dev);
+
 #endif /* _GVE_H_ */
drivers/net/gve/gve_ethdev.c +203 -0

267 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index 9a22cc9abe..c4fd013ef2 100644
@@ -9,5 +9,7 @@ endif
 
 sources = files(
         'gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
Signature · 1 line
2.34.1
#26 lore raw

PATCH v2 7/10 net/gve: add Rx/Tx support

In reply to Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 drivers/net/gve/gve.h        |  17 ++
 drivers/net/gve/gve_ethdev.c |   5 +
 drivers/net/gve/gve_rx.c     | 143 +++++++++++
 drivers/net/gve/gve_tx.c     | 452 +++++++++++++++++++++++++++++++++++
 4 files changed, 617 insertions(+)
drivers/net/gve/gve.h +17 -0
index a53a852a5f..7f4d0e37f3 100644
@@ -25,6 +25,7 @@
 
 #define GVE_DEFAULT_RX_FREE_THRESH  512
 #define GVE_DEFAULT_TX_FREE_THRESH  256
+#define GVE_TX_MAX_FREE_SZ          512
 
 /* PTYPEs are always 10 bits. */
 #define GVE_NUM_PTYPES	1024
@@ -45,6 +46,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -298,4 +311,8 @@ void gve_stop_tx_queues(struct rte_eth_dev *dev);
 
 void gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_H_ */
drivers/net/gve/gve_ethdev.c +5 -0
index 5201398664..5ebe2c30ea 100644
@@ -583,6 +583,11 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	}
+
 	eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
 	if (!eth_dev->data->mac_addrs) {
 		PMD_INIT_LOG(ERR, "Failed to allocate memory to store mac address");
drivers/net/gve/gve_rx.c +143 -0

149 lines not on this page · raw message

drivers/net/gve/gve_tx.c +452 -0

458 lines not on this page · raw message

Signature · 1 line
2.34.1
#27 lore raw

PATCH v2 8/10 net/gve: add support to get dev info and configure dev

In reply to Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
 drivers/net/gve/gve.h        |  3 ++
 drivers/net/gve/gve_ethdev.c | 61 ++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
drivers/net/gve/gve.h +3 -0
index 7f4d0e37f3..004e0a75ca 100644
@@ -27,6 +27,9 @@
 #define GVE_DEFAULT_TX_FREE_THRESH  256
 #define GVE_TX_MAX_FREE_SZ          512
 
+#define GVE_MIN_BUF_SIZE	    1024
+#define GVE_MAX_RX_PKTLEN	    65535
+
 /* PTYPEs are always 10 bits. */
 #define GVE_NUM_PTYPES	1024
 
drivers/net/gve/gve_ethdev.c +61 -0

79 lines not on this page · raw message

Signature · 1 line
2.34.1
#28 lore raw

Re: [PATCH v2 08/10] net/gve: add support to get dev info and configure dev

Quoted 30 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
  drivers/net/gve/gve.h        |  3 ++
  drivers/net/gve/gve_ethdev.c | 61 ++++++++++++++++++++++++++++++++++++
  2 files changed, 64 insertions(+)
drivers/net/gve/gve.h +3 -0
index 7f4d0e37f3..004e0a75ca 100644
@@ -27,6 +27,9 @@
  #define GVE_DEFAULT_TX_FREE_THRESH  256
  #define GVE_TX_MAX_FREE_SZ          512
 
+#define GVE_MIN_BUF_SIZE           1024
+#define GVE_MAX_RX_PKTLEN          65535
+
  /* PTYPEs are always 10 bits. */
  #define GVE_NUM_PTYPES 1024
 
drivers/net/gve/gve_ethdev.c +0 -0
index 5ebe2c30ea..6bc7bf4519 100644

drivers/net/gve/gve_ethdev.c gve_free_qpls(struct gve_priv *priv) · line 105

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +7 -0
@@ -96,6 +96,14 @@ gve_free_qpls(struct gve_priv *priv)
  static int
  gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
  {
+       struct gve_priv *priv = dev->data->dev_private;
+
+       if (dev->data->dev_conf.rxmode.mq_mode & RTE_ETH_MQ_RX_RSS_FLAG)
+               dev->data->dev_conf.rxmode.offloads |= RTE_ETH_RX_OFFLOAD_RSS_HASH;
+
+       if (dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_TCP_LRO)
+               priv->enable_lsc = 1;

What is the relation between LRO and LSC? Is it a typo?

And does driver support LSC at all? Or any interrupt?

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +3 -0
+
         return 0;
  }
 
@@ -266,6 +274,58 @@ gve_dev_close(struct rte_eth_dev *dev)
         return err;
  }
 
+static int
+gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 288

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+{
+       struct gve_priv *priv = dev->data->dev_private;
+
+       dev_info->device = dev->device;
+       dev_info->max_mac_addrs = 1;
+       dev_info->max_rx_queues = priv->max_nb_rxq;
+       dev_info->max_tx_queues = priv->max_nb_txq;
+       dev_info->min_rx_bufsize = GVE_MIN_BUF_SIZE;
+       dev_info->max_rx_pktlen = GVE_MAX_RX_PKTLEN;
+

Can you please provide 'max_mtu' & 'min_mtu' values too?

#29 lore raw

RE: [PATCH v2 08/10] net/gve: add support to get dev info and configure dev

Quoted 54 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:24
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH v2 08/10] net/gve: add support to get dev info and configure dev

Quoted 9 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
  drivers/net/gve/gve.h        |  3 ++
  drivers/net/gve/gve_ethdev.c | 61

++++++++++++++++++++++++++++++++++++

Review

Quoted 17 lines
  2 files changed, 64 insertions(+)
drivers/net/gve/gve.h +3 -0
index 7f4d0e37f3..004e0a75ca 100644
@@ -27,6 +27,9 @@
  #define GVE_DEFAULT_TX_FREE_THRESH  256
  #define GVE_TX_MAX_FREE_SZ          512
 
+#define GVE_MIN_BUF_SIZE           1024
+#define GVE_MAX_RX_PKTLEN          65535
+
  /* PTYPEs are always 10 bits. */
  #define GVE_NUM_PTYPES 1024
 
(quoted diff) +0 -0

b/drivers/net/gve/gve_ethdev.c

drivers/net/gve/gve_ethdev.c gve_free_qpls(struct gve_priv *priv) · line 101

Quoted 10 lines

index 5ebe2c30ea..6bc7bf4519 100644

drivers/net/gve/gve_ethdev.c +3 -0
@@ -96,6 +96,14 @@ gve_free_qpls(struct gve_priv *priv)
  static int
  gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
  {
+       struct gve_priv *priv = dev->data->dev_private;
+
+       if (dev->data->dev_conf.rxmode.mq_mode &

RTE_ETH_MQ_RX_RSS_FLAG)

Quoted 8 lines

drivers/net/gve/gve_ethdev.c gve_free_qpls(struct gve_priv *priv) · line 102

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+               dev->data->dev_conf.rxmode.offloads |=

RTE_ETH_RX_OFFLOAD_RSS_HASH;

drivers/net/gve/gve_ethdev.c gve_free_qpls(struct gve_priv *priv) · line 104

Quoted 2 lines
drivers/net/gve/gve_ethdev.c +2 -0
+
+       if (dev->data->dev_conf.rxmode.offloads &

RTE_ETH_RX_OFFLOAD_TCP_LRO)

drivers/net/gve/gve_ethdev.c gve_free_qpls(struct gve_priv *priv) · line 105

Quoted 1 line
drivers/net/gve/gve_ethdev.c +1 -0
+               priv->enable_lsc = 1;

What is the relation between LRO and LSC? Is it a typo?

Yes, this is just a typo and it should be 'enable_rsc' to indicate Receive Segment Coalescing for TCP Large Receive Offload. Thanks for reminding!

Quoted 2 lines

And does driver support LSC at all? Or any interrupt?

Looks that current base code has not provided these functions.

Quoted 12 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 278

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +3 -0
+
         return 0;
  }
 
@@ -266,6 +274,58 @@ gve_dev_close(struct rte_eth_dev *dev)
         return err;
  }
 
+static int
+gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info

*dev_info)

Quoted 12 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 288

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+{
+       struct gve_priv *priv = dev->data->dev_private;
+
+       dev_info->device = dev->device;
+       dev_info->max_mac_addrs = 1;
+       dev_info->max_rx_queues = priv->max_nb_rxq;
+       dev_info->max_tx_queues = priv->max_nb_txq;
+       dev_info->min_rx_bufsize = GVE_MIN_BUF_SIZE;
+       dev_info->max_rx_pktlen = GVE_MAX_RX_PKTLEN;
+

Can you please provide 'max_mtu' & 'min_mtu' values too?

Yes, will add these in the coming version patchset. Thanks!

#30 lore raw

PATCH v2 9/10 net/gve: add stats support

In reply to Junfeng Guo

Update stats add support of dev_ops stats_get/reset.

---
 drivers/net/gve/gve.h        | 10 ++++++
 drivers/net/gve/gve_ethdev.c | 69 ++++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_rx.c     | 15 ++++++--
 drivers/net/gve/gve_tx.c     | 12 +++++++
 4 files changed, 104 insertions(+), 2 deletions(-)
drivers/net/gve/gve.h +10 -0
index 004e0a75ca..e256a2bec2 100644
@@ -91,6 +91,10 @@ struct gve_tx_queue {
 	struct gve_queue_page_list *qpl;
 	struct gve_tx_iovec *iov_ring;
 
+	/* Stats */
+	uint64_t packets;
+	uint64_t bytes;
+
 	uint16_t port_id;
 	uint16_t queue_id;
 
@@ -129,6 +133,12 @@ struct gve_rx_queue {
 	/* only valid for GQI_QPL queue format */
 	struct gve_queue_page_list *qpl;
 
+	/* stats */
+	uint64_t no_mbufs;
+	uint64_t errors;
+	uint64_t packets;
+	uint64_t bytes;
+
 	struct gve_priv *hw;
 	const struct rte_memzone *qres_mz;
 	struct gve_queue_resources *qres;
drivers/net/gve/gve_ethdev.c +69 -0

81 lines not on this page · raw message

drivers/net/gve/gve_rx.c +13 -2
index 8f560ae592..3a8a869980 100644
@@ -26,8 +26,10 @@ gve_rx_refill(struct gve_rx_queue *rxq)
 					break;
 				rxq->sw_ring[idx + i] = nmb;
 			}
-			if (i != nb_alloc)
+			if (i != nb_alloc) {
+				rxq->no_mbufs += nb_alloc - i;
 				nb_alloc = i;
+			}
 		}
 		rxq->nb_avail -= nb_alloc;
 		next_avail += nb_alloc;
@@ -88,6 +90,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	uint16_t rx_id = rxq->rx_tail;
 	struct rte_mbuf *rxe;
 	uint16_t nb_rx, len;
+	uint64_t bytes = 0;
 	uint64_t addr;
 
 	rxr = rxq->rx_desc_ring;
@@ -97,8 +100,10 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
 			break;
 
-		if (rxd->flags_seq & GVE_RXF_ERR)
+		if (rxd->flags_seq & GVE_RXF_ERR) {
+			rxq->errors++;
 			continue;
+		}
 
 		len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
 		rxe = rxq->sw_ring[rx_id];
@@ -137,6 +142,7 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 			rx_id = 0;
 
 		rx_pkts[nb_rx] = rxe;
+		bytes += len;
 	}
 
 	rxq->nb_avail += nb_rx;
@@ -145,6 +151,11 @@ gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	if (rxq->nb_avail > rxq->free_thresh)
 		gve_rx_refill(rxq);
 
+	if (nb_rx) {
+		rxq->packets += nb_rx;
+		rxq->bytes += bytes;
+	}
+
 	return nb_rx;
 }
 
drivers/net/gve/gve_tx.c +12 -0
index 2dc3411672..d99e6eb009 100644
@@ -260,6 +260,7 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t sw_id = txq->sw_tail;
 	uint16_t nb_used, i;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 	uint32_t hlen;
 
@@ -352,6 +353,8 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		txq->nb_free -= nb_used;
 		txq->sw_nb_free -= first->nb_segs;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
@@ -359,6 +362,9 @@ gve_tx_burst_qpl(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
 		txq->sw_tail = sw_id;
+
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
@@ -377,6 +383,7 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt, *first;
 	uint16_t nb_used, hlen, i;
 	uint64_t ol_flags, addr;
+	uint64_t bytes = 0;
 	uint16_t nb_tx = 0;
 
 	txr = txq->tx_desc_ring;
@@ -435,12 +442,17 @@ gve_tx_burst_ra(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		txq->nb_free -= nb_used;
 		tx_tail += nb_used;
+
+		bytes += first->pkt_len;
 	}
 
 end_of_tx:
 	if (nb_tx) {
 		rte_write32(rte_cpu_to_be_32(tx_tail), txq->qtx_tail);
 		txq->tx_tail = tx_tail;
+
+		txq->packets += nb_tx;
+		txq->bytes += bytes;
 	}
 
 	return nb_tx;
Signature · 1 line
2.34.1
#31 lore raw

Re: [PATCH v2 09/10] net/gve: add stats support

Quoted 17 lines · Junfeng Guo

Update stats add support of dev_ops stats_get/reset.

---
  drivers/net/gve/gve.h        | 10 ++++++
  drivers/net/gve/gve_ethdev.c | 69 ++++++++++++++++++++++++++++++++++++
  drivers/net/gve/gve_rx.c     | 15 ++++++--
  drivers/net/gve/gve_tx.c     | 12 +++++++
  4 files changed, 104 insertions(+), 2 deletions(-)
drivers/net/gve/gve.h +0 -0
index 004e0a75ca..e256a2bec2 100644

drivers/net/gve/gve.h struct gve_tx_queue { · line 97

Quoted 7 lines
drivers/net/gve/gve.h +4 -0
@@ -91,6 +91,10 @@ struct gve_tx_queue {
         struct gve_queue_page_list *qpl;
         struct gve_tx_iovec *iov_ring;
 
+       /* Stats */
+       uint64_t packets;
+       uint64_t bytes;
+

Can't you get stats for 'errors' in Tx path?

#32 lore raw

RE: [PATCH v2 09/10] net/gve: add stats support

Quoted 35 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, September 2, 2022 01:24
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH v2 09/10] net/gve: add stats support

Quoted 8 lines · Junfeng Guo

Update stats add support of dev_ops stats_get/reset.

---
  drivers/net/gve/gve.h        | 10 ++++++
  drivers/net/gve/gve_ethdev.c | 69

++++++++++++++++++++++++++++++++++++

Quoted 12 lines
  drivers/net/gve/gve_rx.c     | 15 ++++++--
  drivers/net/gve/gve_tx.c     | 12 +++++++
  4 files changed, 104 insertions(+), 2 deletions(-)
drivers/net/gve/gve.h +0 -0
index 004e0a75ca..e256a2bec2 100644
@@ -91,6 +91,10 @@ struct gve_tx_queue {
         struct gve_queue_page_list *qpl;
         struct gve_tx_iovec *iov_ring;
 
Quoted 6 lines

drivers/net/gve/gve.h struct gve_tx_queue { · line 97

Quoted 4 lines
drivers/net/gve/gve.h +4 -0
+       /* Stats */
+       uint64_t packets;
+       uint64_t bytes;
+

Can't you get stats for 'errors' in Tx path?

Yes, will add in the coming version. Thanks!

Quoted 1 line
#33 lore raw

PATCH v2 10/10 doc: update documentation

In reply to Junfeng Guo

Update documentation of GVE PMD and release note.

---
 MAINTAINERS                            |  6 +++
 doc/guides/nics/features/gve.ini       | 18 +++++++
 doc/guides/nics/gve.rst                | 65 ++++++++++++++++++++++++++
 doc/guides/rel_notes/release_22_11.rst |  4 ++
 4 files changed, 93 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
MAINTAINERS +6 -0
index 32ffdd1a61..474f41f0de 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +18 -0
new file mode 100644
index 0000000000..180408aa80
@@ -0,0 +1,18 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Speed capabilities   = Y
+Link status          = Y
+MTU update           = Y
+TSO                  = Y
+RSS hash             = Y
+L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +65 -0

65 lines not on this page · raw message

doc/guides/rel_notes/release_22_11.rst +4 -0
index 8c021cf050..6674f4cf6f 100644
@@ -55,6 +55,10 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added GVE net PMD**
+
+  Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  See the :doc:`../nics/gve` NIC guide for more details on this new driver.
 
 Removed Items
 -------------
Signature · 1 line
2.34.1
#34 lore raw

Re: [PATCH v2 10/10] doc: update documentation

Quoted 7 lines · Junfeng Guo

CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

Update documentation of GVE PMD and release note.

Quoted 9 lines
---
  MAINTAINERS                            |  6 +++
  doc/guides/nics/features/gve.ini       | 18 +++++++
  doc/guides/nics/gve.rst                | 65 ++++++++++++++++++++++++++
  doc/guides/rel_notes/release_22_11.rst |  4 ++
  4 files changed, 93 insertions(+)

create mode 100644 doc/guides/nics/features/gve.ini create mode 100644 doc/guides/nics/gve.rst

Need to update index file (doc/guides/nics/index.rst) to make new gve.rst file visible. Did you test build and viewing the documentation?

Quoted 31 lines
MAINTAINERS +6 -0
index 32ffdd1a61..474f41f0de 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
  F: doc/guides/nics/enic.rst
  F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
  Hisilicon hns3
  M: Dongdong Liu <[email protected]>
  M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +8 -0
new file mode 100644
index 0000000000..180408aa80
@@ -0,0 +1,18 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Speed capabilities   = Y
+Link status          = Y

doc/guides/nics/features/gve.ini line 18

Quoted 10 lines
doc/guides/nics/features/gve.ini new file +10 -0
+MTU update           = Y
+TSO                  = Y
+RSS hash             = Y
+L4 checksum offload  = Y
+Basic stats          = Y
+Stats per queue      = Y
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y

Can you please add this patch as first patch with infrastructure files (like meson files, .map file etc..) and later update the .ini with each patch, as a patch introduces a feature update .ini to add that feature.

Quoted 6 lines
doc/guides/nics/gve.rst new file +0 -0
new file mode 100644
index 0000000000..20cda5031b

doc/guides/nics/gve.rst line 9

Quoted 9 lines
doc/guides/nics/gve.rst new file +9 -0
@@ -0,0 +1,65 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(C) 2022 Intel Corporation.
+
+GVE poll mode driver
+=======================
+
+The GVE PMD (**librte_net_gve**) provides poll mode driver support for
+Google Virtual Ethernet device.
+

Can you please provide some references for the device? Official product link etc...

doc/guides/nics/gve.rst line 18

Quoted 9 lines
doc/guides/nics/gve.rst new file +9 -0
+The base code is under MIT license and based on GVE kernel driver v1.3.0.
+GVE base code files are:
+
+- gve_adminq.h
+- gve_adminq.c
+- gve_desc.h
+- gve_desc_dqo.h
+- gve_register.h
+

Instead of listing these files in the documentation, what do you think to place under a 'base' folder (drivers/net/gve/base/*) which will clarify that these are base code files.

Quoted 32 lines
doc/guides/nics/gve.rst new file +32 -0
+Please refer to https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0/google/gve
+to find the original base code.
+
+GVE has 3 queue formats:
+
+- GQI_QPL - GQI with queue page list
+- GQI_RDA - GQI with raw DMA addressing
+- DQO_RDA - DQO with raw DMA addressing
+
+GQI_QPL queue format is queue page list mode. Driver needs to allocate
+memory and register this memory as a Queue Page List (QPL) in hardware
+(Google Hypervisor/GVE Backend) first. Each queue has its own QPL.
+Then Tx needs to copy packets to QPL memory and put this packet's offset
+in the QPL memory into hardware descriptors so that hardware can get the
+packets data. And Rx needs to read descriptors of offset in QPL to get
+QPL address and copy packets from the address to get real packets data.
+
+GQI_RDA queue format works like usual NICs that driver can put packets'
+physical address into hardware descriptors.
+
+DQO_RDA queue format has submission and completion queue pair for each
+Tx/Rx queue. And similar as GQI_RDA, driver can put packets' physical
+address into hardware descriptors.
+
+Please refer to https://www.kernel.org/doc/html/latest/networking/device_drivers/ethernet/google/gve.html
+to get more information about GVE queue formats.
+
+Features and Limitations
+------------------------
+
+In this release, the GVE PMD provides the basic functionality of packet
+reception and transmission.

doc/guides/nics/gve.rst line 60

Quoted 10 lines
doc/guides/nics/gve.rst new file +10 -0
+Supported features of the GVE PMD are:
+
+- Multiple queues for TX and RX
+- Receiver Side Scaling (RSS)
+- TSO offload
+- Port hardware statistics
+- Link state information
+- TX multi-segments (Scatter TX)
+- Tx UDP/TCP/SCTP Checksum
+

Same comment as .ini file above. Let's build the features list gradually as code adds them, instead of adding it as batch at the end.

#35 lore raw

v3 cover introduce GVE PMD

In reply to Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

Junfeng Guo (9):
net/gve: introduce GVE PMD base code
net/gve: add logs and OS specific implementation net/gve: support device initialization net/gve: add link update support
net/gve: add MTU set support
net/gve: add queue operations
net/gve: add Rx/Tx support
net/gve: add support to get dev info and configure dev net/gve: add stats support

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  18 +
 doc/guides/nics/gve.rst                |  69 ++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/gve.h                  |  58 ++
 drivers/net/gve/gve_adminq.c           | 926 +++++++++++++++++++++++++
 drivers/net/gve/gve_adminq.h           | 383 ++++++++++
 drivers/net/gve/gve_desc.h             | 139 ++++
 drivers/net/gve/gve_desc_dqo.h         | 256 +++++++
 drivers/net/gve/gve_ethdev.c           | 775 +++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 300 ++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/gve_osdep.h            | 159 +++++
 drivers/net/gve/gve_register.h         |  30 +
 drivers/net/gve/gve_rx.c               | 366 ++++++++++
 drivers/net/gve/gve_tx.c               | 682 ++++++++++++++++++
 drivers/net/gve/meson.build            |  15 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 20 files changed, 4206 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve.h
 create mode 100644 drivers/net/gve/gve_adminq.c
 create mode 100644 drivers/net/gve/gve_adminq.h
 create mode 100644 drivers/net/gve/gve_desc.h
 create mode 100644 drivers/net/gve/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_osdep.h
 create mode 100644 drivers/net/gve/gve_register.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.34.1
#36 lore raw

PATCH v3 1/9 net/gve: introduce GVE PMD base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

---
 drivers/net/gve/gve.h          |  58 +++
 drivers/net/gve/gve_adminq.c   | 925 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_adminq.h   | 381 ++++++++++++++
 drivers/net/gve/gve_desc.h     | 137 +++++
 drivers/net/gve/gve_desc_dqo.h | 254 +++++++++
 drivers/net/gve/gve_register.h |  28 +
 6 files changed, 1783 insertions(+)
 create mode 100644 drivers/net/gve/gve.h
 create mode 100644 drivers/net/gve/gve_adminq.c
 create mode 100644 drivers/net/gve/gve_adminq.h
 create mode 100644 drivers/net/gve/gve_desc.h
 create mode 100644 drivers/net/gve/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/gve_register.h
drivers/net/gve/gve.h new file +58 -0
new file mode 100644
index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION		"1.3.0"
+#define GVE_VERSION_PREFIX	"GVE-"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR		0
+#define GVE_DB_BAR		2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES		1024
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+#endif /* _GVE_H_ */
drivers/net/gve/gve_adminq.c new file +925 -0

925 lines not on this page · raw message

drivers/net/gve/gve_adminq.h new file +381 -0

381 lines not on this page · raw message

drivers/net/gve/gve_desc.h new file +137 -0

137 lines not on this page · raw message

drivers/net/gve/gve_desc_dqo.h new file +254 -0

254 lines not on this page · raw message

drivers/net/gve/gve_register.h new file +28 -0
new file mode 100644
index 0000000000..b65f336be2
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.34.1
#37 lore raw

Re: [PATCH v3 1/9] net/gve: introduce GVE PMD base code

Quoted 1 line · On Fri, 23 Sep 2022 17:38:21 +0800 Junfeng Guo <[email protected]> wrote:

+#define GVE_DEVICE_OPTION_TOO_BIG_FMT "Length of %s option larger than expected. Possible older version of guest driver."

Why do you need #define for this?

+#define GVE_DEVICE_OPTION_ERROR_FMT "%s option error:\n" \
+"Expected: length=%d, feature_mask=%x.\n" \
+"Actual: length=%d, feature_mask=%x."
+

Why such a wordy multi-line message, please use single line

#38 lore raw

RE: [PATCH v3 1/9] net/gve: introduce GVE PMD base code

Quoted 13 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Saturday, September 24, 2022 02:58
To: Guo, Junfeng <[email protected]>
Cc: Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v3 1/9] net/gve: introduce GVE PMD base code

Quoted 7 lines
Quoted 1 line · On Fri, 23 Sep 2022 17:38:21 +0800 Junfeng Guo <[email protected]> wrote:

+#define GVE_DEVICE_OPTION_TOO_BIG_FMT "Length of %s option

larger than expected. Possible older version of guest driver."

Why do you need #define for this?

This macro is used by gve kernel driver and we just keep it here. Thanks!

Quoted 7 lines
drivers/net/gve/gve_adminq.c new file +4 -0
+#define GVE_DEVICE_OPTION_ERROR_FMT "%s option error:\n" \
+"Expected: length=%d, feature_mask=%x.\n" \
+"Actual: length=%d, feature_mask=%x."
+

Why such a wordy multi-line message, please use single line

This one is also from gve kernel driver.
Will update with single line message in the coming version. Thanks!

#39 lore raw

Re: [PATCH v3 1/9] net/gve: introduce GVE PMD base code

In reply to Junfeng Guo
Quoted 4 lines · On Fri, 23 Sep 2022 17:38:21 +0800 Junfeng Guo <[email protected]> wrote:

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

Using MIT license in DPDK will require approval from TAB and from the DPDK governing board. So it probably won't make this release.

#40 lore raw

RE: [PATCH v3 1/9] net/gve: introduce GVE PMD base code

Quoted 16 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Saturday, September 24, 2022 02:59
To: Guo, Junfeng <[email protected]>
Cc: Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v3 1/9] net/gve: introduce GVE PMD base code

On Fri, 23 Sep 2022 17:38:21 +0800
Junfeng Guo <[email protected]> wrote:

Quoted 7 lines
Quoted 4 lines

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

Using MIT license in DPDK will require approval from TAB and from the DPDK governing board. So it probably won't make this release.

Yes, the approval is under process now. Thanks!

#41 lore raw

v4 cover introduce GVE PMD

In reply to Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

Junfeng Guo (9):
net/gve/base: introduce GVE PMD base code net/gve/base: add logs and OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for queue operations net/gve: add support for Rx/Tx
net/gve: add support for dev info get and dev configure net/gve: add support for stats

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  18 +
 doc/guides/nics/gve.rst                |  69 ++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve.h             |  58 ++
 drivers/net/gve/base/gve_adminq.c      | 924 +++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h      | 383 ++++++++++
 drivers/net/gve/base/gve_desc.h        | 139 ++++
 drivers/net/gve/base/gve_desc_dqo.h    | 256 +++++++
 drivers/net/gve/base/gve_osdep.h       | 159 +++++
 drivers/net/gve/base/gve_register.h    |  30 +
 drivers/net/gve/gve_ethdev.c           | 775 +++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 300 ++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/gve_rx.c               | 366 ++++++++++
 drivers/net/gve/gve_tx.c               | 682 ++++++++++++++++++
 drivers/net/gve/meson.build            |  16 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 20 files changed, 4205 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_osdep.h
 create mode 100644 drivers/net/gve/base/gve_register.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.34.1
#42 lore raw

PATCH v4 1/9 net/gve/base: introduce GVE PMD base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

---
 drivers/net/gve/base/gve.h          |  58 ++
 drivers/net/gve/base/gve_adminq.c   | 923 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h   | 381 ++++++++++++
 drivers/net/gve/base/gve_desc.h     | 137 +++++
 drivers/net/gve/base/gve_desc_dqo.h | 254 ++++++++
 drivers/net/gve/base/gve_register.h |  28 +
 6 files changed, 1781 insertions(+)
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_register.h
drivers/net/gve/base/gve.h new file +58 -0
new file mode 100644
index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION		"1.3.0"
+#define GVE_VERSION_PREFIX	"GVE-"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR		0
+#define GVE_DB_BAR		2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES		1024
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+#endif /* _GVE_H_ */
drivers/net/gve/base/gve_adminq.c new file +923 -0

923 lines not on this page · raw message

drivers/net/gve/base/gve_adminq.h new file +381 -0

381 lines not on this page · raw message

drivers/net/gve/base/gve_desc.h new file +137 -0

137 lines not on this page · raw message

drivers/net/gve/base/gve_desc_dqo.h new file +254 -0

254 lines not on this page · raw message

drivers/net/gve/base/gve_register.h new file +28 -0
new file mode 100644
index 0000000000..b65f336be2
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.34.1
#43 lore raw

Re: [PATCH v4 1/9] net/gve/base: introduce GVE PMD base code

Quoted 22 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

Quoted 25 lines
drivers/net/gve/base/gve_adminq.c new file +25 -0
+/* Process all device options for a given describe device call. */
+static int
+gve_process_device_options(struct gve_priv *priv,
+                          struct gve_device_descriptor *descriptor,
+                          struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+                          struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+                          struct gve_device_option_dqo_rda **dev_op_dqo_rda,
+                          struct gve_device_option_jumbo_frames **dev_op_jumbo_frames)
+{
+       const int num_options = be16_to_cpu(descriptor->num_device_options);
+       struct gve_device_option *dev_opt;
+       int i;
+
+       /* The options struct directly follows the device descriptor. */
+       dev_opt = RTE_PTR_ADD(descriptor, sizeof(*descriptor));
+       for (i = 0; i < num_options; i++) {
+               struct gve_device_option *next_opt;
+
+               next_opt = gve_get_next_option(descriptor, dev_opt);
+               if (!next_opt) {
+                       PMD_DRV_LOG(ERR,
+                                   "options exceed device_descriptor's total length.");
+                       return -EINVAL;
+               }
+

drivers/net/gve/base/gve_adminq.c line 170

Quoted 10 lines
drivers/net/gve/base/gve_adminq.c new file +10 -0
+               gve_parse_device_option(priv, dev_opt,
+                                       dev_op_gqi_rda, dev_op_gqi_qpl,
+                                       dev_op_dqo_rda, dev_op_jumbo_frames);
+               dev_opt = next_opt;
+       }
+
+       return 0;
+}
+
+int gve_adminq_alloc(struct gve_priv *priv)

Can you please be consistent in the syntax, at least within same file, if this file has slightly different syntax because it is base file, keep the file syntax instead of mixing with DPDK syntax, like return type of function should be on separate line.

A generic comment for all base files.

<...>

Quoted 76 lines
drivers/net/gve/base/gve_adminq.c new file +76 -0
+int gve_adminq_describe_device(struct gve_priv *priv)
+{
+       struct gve_device_option_jumbo_frames *dev_op_jumbo_frames = NULL;
+       struct gve_device_option_gqi_rda *dev_op_gqi_rda = NULL;
+       struct gve_device_option_gqi_qpl *dev_op_gqi_qpl = NULL;
+       struct gve_device_option_dqo_rda *dev_op_dqo_rda = NULL;
+       struct gve_device_descriptor *descriptor;
+       struct gve_dma_mem descriptor_dma_mem;
+       u32 supported_features_mask = 0;
+       union gve_adminq_command cmd;
+       int err = 0;
+       u8 *mac;
+       u16 mtu;
+
+       memset(&cmd, 0, sizeof(cmd));
+       descriptor = gve_alloc_dma_mem(&descriptor_dma_mem, PAGE_SIZE);
+       if (!descriptor)
+               return -ENOMEM;
+       cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESCRIBE_DEVICE);
+       cmd.describe_device.device_descriptor_addr =
+                                       cpu_to_be64(descriptor_dma_mem.pa);
+       cmd.describe_device.device_descriptor_version =
+                       cpu_to_be32(GVE_ADMINQ_DEVICE_DESCRIPTOR_VERSION);
+       cmd.describe_device.available_length = cpu_to_be32(PAGE_SIZE);
+
+       err = gve_adminq_execute_cmd(priv, &cmd);
+       if (err)
+               goto free_device_descriptor;
+
+       err = gve_process_device_options(priv, descriptor, &dev_op_gqi_rda,
+                                        &dev_op_gqi_qpl, &dev_op_dqo_rda,
+                                        &dev_op_jumbo_frames);
+       if (err)
+               goto free_device_descriptor;
+
+       /* If the GQI_RAW_ADDRESSING option is not enabled and the queue format
+        * is not set to GqiRda, choose the queue format in a priority order:
+        * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default.
+        */
+       if (dev_op_dqo_rda) {
+               priv->queue_format = GVE_DQO_RDA_FORMAT;
+               PMD_DRV_LOG(INFO, "Driver is running with DQO RDA queue format.");
+               supported_features_mask =
+                       be32_to_cpu(dev_op_dqo_rda->supported_features_mask);
+       } else if (dev_op_gqi_rda) {
+               priv->queue_format = GVE_GQI_RDA_FORMAT;
+               PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+               supported_features_mask =
+                       be32_to_cpu(dev_op_gqi_rda->supported_features_mask);
+       } else if (priv->queue_format == GVE_GQI_RDA_FORMAT) {
+               PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+       } else {
+               priv->queue_format = GVE_GQI_QPL_FORMAT;
+               if (dev_op_gqi_qpl)
+                       supported_features_mask =
+                               be32_to_cpu(dev_op_gqi_qpl->supported_features_mask);
+               PMD_DRV_LOG(INFO, "Driver is running with GQI QPL queue format.");
+       }
+       if (gve_is_gqi(priv)) {
+               err = gve_set_desc_cnt(priv, descriptor);
+       } else {
+               /* DQO supports LRO. */
+               err = gve_set_desc_cnt_dqo(priv, descriptor, dev_op_dqo_rda);
+       }
+       if (err)
+               goto free_device_descriptor;
+
+       priv->max_registered_pages =
+                               be64_to_cpu(descriptor->max_registered_pages);
+       mtu = be16_to_cpu(descriptor->mtu);
+       if (mtu < ETH_MIN_MTU) {
+               PMD_DRV_LOG(ERR, "MTU %d below minimum MTU", mtu);
+               err = -EINVAL;
+               goto free_device_descriptor;
+       }
+       priv->max_mtu = mtu;

drivers/net/gve/base/gve_adminq.c line 776

Quoted 10 lines
drivers/net/gve/base/gve_adminq.c new file +10 -0
+       priv->num_event_counters = be16_to_cpu(descriptor->counters);
+       rte_memcpy(priv->dev_addr.addr_bytes, descriptor->mac, ETH_ALEN);
+       mac = descriptor->mac;
+       PMD_DRV_LOG(INFO, "MAC addr: %02x:%02x:%02x:%02x:%02x:%02x",
+                   mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+       priv->tx_pages_per_qpl = be16_to_cpu(descriptor->tx_pages_per_qpl);
+       priv->rx_data_slot_cnt = be16_to_cpu(descriptor->rx_pages_per_qpl);
+
+       if (gve_is_gqi(priv) && priv->rx_data_slot_cnt < priv->rx_desc_cnt) {
+               PMD_DRV_LOG(ERR, "rx_data_slot_cnt cannot be smaller than rx_desc_cnt, setting rx_desc_cnt down to %d",

Can you try to reduce the line length as;
PMD_DRV_LOG(ERR,
"rx_data_slot_cnt cannot be smaller than rx_desc_cnt, setting
rx_desc_cnt down to %d",

#44 lore raw

RE: [PATCH v4 1/9] net/gve/base: introduce GVE PMD base code

Quoted 84 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 6, 2022 22:20
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v4 1/9] net/gve/base: introduce GVE PMD base code

Quoted 10 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

Quoted 2 lines

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/\

Quoted 10 lines

tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

drivers/net/gve/base/gve_adminq.c

Quoted 8 lines
drivers/net/gve/base/gve_adminq.c new file +8 -0
+/* Process all device options for a given describe device call. */
+static int
+gve_process_device_options(struct gve_priv *priv,
+                          struct gve_device_descriptor *descriptor,
+                          struct gve_device_option_gqi_rda **dev_op_gqi_rda,
+                          struct gve_device_option_gqi_qpl **dev_op_gqi_qpl,
+                          struct gve_device_option_dqo_rda **dev_op_dqo_rda,
+                          struct gve_device_option_jumbo_frames

**dev_op_jumbo_frames)

Quoted 18 lines
(quoted diff) +2 -0
+{
+       const int num_options = be16_to_cpu(descriptor-

num_device_options);

drivers/net/gve/base/gve_adminq.c new file +15 -0
+       struct gve_device_option *dev_opt;
+       int i;
+
+       /* The options struct directly follows the device descriptor. */
+       dev_opt = RTE_PTR_ADD(descriptor, sizeof(*descriptor));
+       for (i = 0; i < num_options; i++) {
+               struct gve_device_option *next_opt;
+
+               next_opt = gve_get_next_option(descriptor, dev_opt);
+               if (!next_opt) {
+                       PMD_DRV_LOG(ERR,
+                                   "options exceed device_descriptor's total length.");
+                       return -EINVAL;
+               }
+

drivers/net/gve/base/gve_adminq.c line 170

Quoted 10 lines
drivers/net/gve/base/gve_adminq.c new file +10 -0
+               gve_parse_device_option(priv, dev_opt,
+                                       dev_op_gqi_rda, dev_op_gqi_qpl,
+                                       dev_op_dqo_rda, dev_op_jumbo_frames);
+               dev_opt = next_opt;
+       }
+
+       return 0;
+}
+
+int gve_adminq_alloc(struct gve_priv *priv)

Can you please be consistent in the syntax, at least within same file, if this file has slightly different syntax because it is base file, keep the file syntax instead of mixing with DPDK syntax,
like return type of function should be on separate line.

Quoted 1 line

A generic comment for all base files.

Thanks for reminding!
Will keep the same syntax with kernel code for the files within the base folder. And use DPDK syntax for the files in the upper folder of the base. For the base files, it would be better to keep most of the code with the origins.

Quoted 105 lines

<...>

drivers/net/gve/base/gve_adminq.c

Quoted 3 lines
drivers/net/gve/base/gve_adminq.c new file +3 -0
+int gve_adminq_describe_device(struct gve_priv *priv)
+{
+       struct gve_device_option_jumbo_frames *dev_op_jumbo_frames

= NULL;

drivers/net/gve/base/gve_adminq.c

Quoted 13 lines
drivers/net/gve/base/gve_adminq.c new file +13 -0
+       struct gve_device_option_gqi_rda *dev_op_gqi_rda = NULL;
+       struct gve_device_option_gqi_qpl *dev_op_gqi_qpl = NULL;
+       struct gve_device_option_dqo_rda *dev_op_dqo_rda = NULL;
+       struct gve_device_descriptor *descriptor;
+       struct gve_dma_mem descriptor_dma_mem;
+       u32 supported_features_mask = 0;
+       union gve_adminq_command cmd;
+       int err = 0;
+       u8 *mac;
+       u16 mtu;
+
+       memset(&cmd, 0, sizeof(cmd));
+       descriptor = gve_alloc_dma_mem(&descriptor_dma_mem,

PAGE_SIZE);

drivers/net/gve/base/gve_adminq.c line 715

Quoted 7 lines
drivers/net/gve/base/gve_adminq.c new file +7 -0
+       if (!descriptor)
+               return -ENOMEM;
+       cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESCRIBE_DEVICE);
+       cmd.describe_device.device_descriptor_addr =
+                                       cpu_to_be64(descriptor_dma_mem.pa);
+       cmd.describe_device.device_descriptor_version =
+

cpu_to_be32(GVE_ADMINQ_DEVICE_DESCRIPTOR_VERSION);

drivers/net/gve/base/gve_adminq.c

Quoted 7 lines
drivers/net/gve/base/gve_adminq.c new file +7 -0
+       cmd.describe_device.available_length = cpu_to_be32(PAGE_SIZE);
+
+       err = gve_adminq_execute_cmd(priv, &cmd);
+       if (err)
+               goto free_device_descriptor;
+
+       err = gve_process_device_options(priv, descriptor,

&dev_op_gqi_rda,

drivers/net/gve/base/gve_adminq.c

Quoted 6 lines
drivers/net/gve/base/gve_adminq.c new file +6 -0
+                                        &dev_op_gqi_qpl, &dev_op_dqo_rda,
+                                        &dev_op_jumbo_frames);
+       if (err)
+               goto free_device_descriptor;
+
+       /* If the GQI_RAW_ADDRESSING option is not enabled and the

queue format

drivers/net/gve/base/gve_adminq.c

Quoted 6 lines
drivers/net/gve/base/gve_adminq.c new file +6 -0
+        * is not set to GqiRda, choose the queue format in a priority order:
+        * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default.
+        */
+       if (dev_op_dqo_rda) {
+               priv->queue_format = GVE_DQO_RDA_FORMAT;
+               PMD_DRV_LOG(INFO, "Driver is running with DQO RDA queue

format.");

drivers/net/gve/base/gve_adminq.c

Quoted 6 lines
(quoted diff) +2 -0
+               supported_features_mask =
+                       be32_to_cpu(dev_op_dqo_rda-

supported_features_mask);

drivers/net/gve/base/gve_adminq.c new file +3 -0
+       } else if (dev_op_gqi_rda) {
+               priv->queue_format = GVE_GQI_RDA_FORMAT;
+               PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue

format.");

Review

Quoted 5 lines
(quoted diff) +2 -0
+               supported_features_mask =
+                       be32_to_cpu(dev_op_gqi_rda-

supported_features_mask);

(quoted diff) +2 -0
+       } else if (priv->queue_format == GVE_GQI_RDA_FORMAT) {
+               PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue

format.");

Quoted 7 lines
drivers/net/gve/base/gve_adminq.c new file +5 -0
+       } else {
+               priv->queue_format = GVE_GQI_QPL_FORMAT;
+               if (dev_op_gqi_qpl)
+                       supported_features_mask =
+                               be32_to_cpu(dev_op_gqi_qpl-

supported_features_mask);
+ PMD_DRV_LOG(INFO, "Driver is running with GQI QPL queue

format.");

drivers/net/gve/base/gve_adminq.c

Quoted 6 lines
drivers/net/gve/base/gve_adminq.c new file +6 -0
+       }
+       if (gve_is_gqi(priv)) {
+               err = gve_set_desc_cnt(priv, descriptor);
+       } else {
+               /* DQO supports LRO. */
+               err = gve_set_desc_cnt_dqo(priv, descriptor,

dev_op_dqo_rda);

Quoted 5 lines
drivers/net/gve/base/gve_adminq.c new file +5 -0
+       }
+       if (err)
+               goto free_device_descriptor;
+
+       priv->max_registered_pages =

drivers/net/gve/base/gve_adminq.c

Quoted 10 lines
drivers/net/gve/base/gve_adminq.c new file +10 -0
+                               be64_to_cpu(descriptor->max_registered_pages);
+       mtu = be16_to_cpu(descriptor->mtu);
+       if (mtu < ETH_MIN_MTU) {
+               PMD_DRV_LOG(ERR, "MTU %d below minimum MTU", mtu);
+               err = -EINVAL;
+               goto free_device_descriptor;
+       }
+       priv->max_mtu = mtu;
+       priv->num_event_counters = be16_to_cpu(descriptor->counters);
+       rte_memcpy(priv->dev_addr.addr_bytes, descriptor->mac,

ETH_ALEN);

Review

Quoted 2 lines
(quoted diff) +2 -0
+       mac = descriptor->mac;
+       PMD_DRV_LOG(INFO, "MAC

addr: %02x:%02x:%02x:%02x:%02x:%02x",

Review

Quoted 7 lines
(quoted diff) +2 -0
+                   mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+       priv->tx_pages_per_qpl = be16_to_cpu(descriptor-

tx_pages_per_qpl);
+ priv->rx_data_slot_cnt = be16_to_cpu(descriptor-
rx_pages_per_qpl);

(quoted diff) +2 -0
+
+       if (gve_is_gqi(priv) && priv->rx_data_slot_cnt < priv->rx_desc_cnt)
Quoted 8 lines

{

Quoted 1 line

+ PMD_DRV_LOG(ERR, "rx_data_slot_cnt cannot be smaller

than rx_desc_cnt, setting rx_desc_cnt down to %d",

Can you try to reduce the line length as;
PMD_DRV_LOG(ERR,
"rx_data_slot_cnt cannot be smaller than rx_desc_cnt, setting
rx_desc_cnt down to %d",

Sure, it could be. But it would still reach the 100 characters limit even started with a new line... Good news is that it is only log that won't trigger warnings while running the build.

#45 lore raw

v5 cover introduce GVE PMD

In reply to Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

Junfeng Guo (8):
net/gve/base: introduce GVE PMD base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  16 +
 doc/guides/nics/gve.rst                |  71 ++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve.h             |  58 ++
 drivers/net/gve/base/gve_adminq.c      | 925 +++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h      | 383 ++++++++++
 drivers/net/gve/base/gve_desc.h        | 139 ++++
 drivers/net/gve/base/gve_desc_dqo.h    | 256 +++++++
 drivers/net/gve/base/gve_osdep.h       | 159 +++++
 drivers/net/gve/base/gve_register.h    |  30 +
 drivers/net/gve/gve_ethdev.c           | 704 +++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 295 ++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/gve_rx.c               | 352 ++++++++++
 drivers/net/gve/gve_tx.c               | 669 ++++++++++++++++++
 drivers/net/gve/meson.build            |  16 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 20 files changed, 4103 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_osdep.h
 create mode 100644 drivers/net/gve/base/gve_register.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.34.1
#46 lore raw

PATCH v5 1/8 net/gve/base: introduce GVE PMD base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

---
 drivers/net/gve/base/gve.h          |  58 ++
 drivers/net/gve/base/gve_adminq.c   | 924 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h   | 381 ++++++++++++
 drivers/net/gve/base/gve_desc.h     | 137 +++++
 drivers/net/gve/base/gve_desc_dqo.h | 254 ++++++++
 drivers/net/gve/base/gve_register.h |  28 +
 6 files changed, 1782 insertions(+)
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_register.h
drivers/net/gve/base/gve.h new file +58 -0
new file mode 100644
index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION		"1.3.0"
+#define GVE_VERSION_PREFIX	"GVE-"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR		0
+#define GVE_DB_BAR		2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES		1024
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+#endif /* _GVE_H_ */
drivers/net/gve/base/gve_adminq.c new file +924 -0

924 lines not on this page · raw message

drivers/net/gve/base/gve_adminq.h new file +381 -0

381 lines not on this page · raw message

drivers/net/gve/base/gve_desc.h new file +137 -0

137 lines not on this page · raw message

drivers/net/gve/base/gve_desc_dqo.h new file +254 -0

254 lines not on this page · raw message

drivers/net/gve/base/gve_register.h new file +28 -0
new file mode 100644
index 0000000000..b65f336be2
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.34.1
#47 lore raw

Re: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 21 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

drivers/net/gve/base/gve.h line 3

Quoted 9 lines
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644
index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

There is a version macro in the code, is version information required in the file comment?

drivers/net/gve/base/gve.h line 5

Quoted 2 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation

I don't know if it is OK to add Intel copyright, as far as I know this requires big enough contribution to the code, if this is copy of existing code, may be only original copyright should exist.

cc'ed @Hemant and @Stephen for more comment.

#48 lore raw

RE: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 51 lines · Junfeng Guo
Quoted 9 lines

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

Quoted 3 lines

The original code is in:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
ub.com%2FGoogleCloudPlatform%2Fcompute-virtual-ethernet-

linux%2F%2F&am

Quoted 1 line

p;data=05%7C01%7Chemant.agrawal%40nxp.com%7C45cbc9718dcc40d04e4 508dab1

Quoted 1 line

d82440%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380178391 21579415

Quoted 1 line

%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi

LCJBTiI6I

Quoted 1 line

k1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=LHunq53xMl8i W6%2B3scjZ

Quoted 11 lines

q0Bx7oF08yLWk424aw5lnwA%3D&amp;reserved=0
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

drivers/net/gve/base/gve.h line 3

Quoted 8 lines
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644 index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

There is a version macro in the code, is version information required in the file comment?

Quoted 7 lines

drivers/net/gve/base/gve.h line 5

Quoted 2 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation

I don't know if it is OK to add Intel copyright, as far as I know this requires big enough contribution to the code, if this is copy of existing code, may be only original copyright should exist.

[Hemant] Yes, the general guideline is that one should add their copyright if they have big enough contribution. But at the end it is a guideline - not the rule. It is up-to the original copyright holder to object.

Quoted 1 line

cc'ed @Hemant and @Stephen for more comment.

#49 lore raw

Re: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 54 lines · Hemant Agrawal
Quoted 54 lines · Junfeng Guo
Quoted 9 lines

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

Quoted 3 lines

The original code is in:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
ub.com%2FGoogleCloudPlatform%2Fcompute-virtual-ethernet-

linux%2F%2F&am

Quoted 1 line

p;data=05%7C01%7Chemant.agrawal%40nxp.com%7C45cbc9718dcc40d04e4 508dab1

Quoted 1 line

d82440%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380178391 21579415

Quoted 1 line

%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi

LCJBTiI6I

Quoted 1 line

k1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=LHunq53xMl8i W6%2B3scjZ

Quoted 11 lines

q0Bx7oF08yLWk424aw5lnwA%3D&amp;reserved=0
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

drivers/net/gve/base/gve.h line 3

Quoted 8 lines
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644 index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

There is a version macro in the code, is version information required in the file comment?

Quoted 2 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
Quoted 6 lines
Quoted 4 lines

I don't know if it is OK to add Intel copyright, as far as I know this requires big enough contribution to the code, if this is copy of existing code, may be only original copyright should exist.

[Hemant] Yes, the general guideline is that one should add their copyright if they have big enough contribution. But at the end it is a guideline - not the rule. It is up-to the original copyright holder to object.

Does this mean as long as original copyright holder did not object, it is OK to add more copyright?
I don't think they are represented or aware of it this change at all, I believe we (as community) also have responsibility to make these things correct, in our capacity.

Quoted 2 lines
Quoted 2 lines

cc'ed @Hemant and @Stephen for more comment.

#50 lore raw

RE: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 67 lines · Hemant Agrawal
Quoted 10 lines
Quoted 10 lines · Junfeng Guo
Quoted 9 lines

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

The original code is in:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
th ub.com%2FGoogleCloudPlatform%2Fcompute-virtual-ethernet-

linux%2F%2F&am

Quoted 1 line

p;data=05%7C01%7Chemant.agrawal%40nxp.com%7C45cbc9718dcc40d04e4

Quoted 3 lines
Quoted 3 lines

508dab1

Quoted 1 line

d82440%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380178391

Quoted 2 lines
Quoted 2 lines

21579415

Quoted 1 line

%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu

MzIi

Quoted 3 lines
Quoted 3 lines

LCJBTiI6I

Quoted 1 line

k1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=LHunq53xMl8i

Quoted 27 lines
Quoted 27 lines

W6%2B3scjZ

Quoted 11 lines

q0Bx7oF08yLWk424aw5lnwA%3D&amp;reserved=0
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

drivers/net/gve/base/gve.h line 3

Quoted 8 lines
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644 index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

There is a version macro in the code, is version information required in the file comment?

Quoted 8 lines
Quoted 7 lines

drivers/net/gve/base/gve.h line 5

Quoted 2 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation

I don't know if it is OK to add Intel copyright, as far as I know this requires big enough contribution to the code, if this is copy of existing code, may be only original copyright should exist.

[Hemant] Yes, the general guideline is that one should add their copyright if

they have big enough contribution. But at the end it is a guideline - not the rule.

Quoted 7 lines
Quoted 1 line

It is up-to the original copyright holder to object.

Does this mean as long as original copyright holder did not object, it is OK to add more copyright?
I don't think they are represented or aware of it this change at all, I believe we (as community) also have responsibility to make these things correct, in our capacity.

[Hemant] I tried to convey the same in decent words.
Yes, it is incorrect to add copyright without major contribution change.
Intel team shall provide details about what is their contribution over the original code. Or they should remove their copyright

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cc'ed @Hemant and @Stephen for more comment.

#51 lore raw

RE: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

In reply to Ferruh Yigit

Hi

Quoted 38 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:45
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Hemant Agrawal <[email protected]>; Stephen Hemminger <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 22 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/
\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

Quoted 11 lines

drivers/net/gve/base/gve.h line 3

Quoted 8 lines
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644 index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

There is a version macro in the code, is version information required in the file comment?

Different versions of gve kernel driver change a lot. So for reference, I think adding the version info for the base code makes sense. Just tell the following update people which version is used.

Quoted 7 lines

drivers/net/gve/base/gve.h line 5

Quoted 3 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation

I don't know if it is OK to add Intel copyright, as far as I know this requires big enough contribution to the code, if this is copy of existing code, may be only original copyright should exist.

It's not just directly copy. Directly copy like gve_desc.h doesn't have Intel copyright.
But gve.h, in gve kernel driver, it has a lot of info dpdk doesn't need or dpdk has its own version like txq/rxq info.
I'm not sure the contribution is a lot or not. But I suppose this patchset is following the principle that if the code is changed, intel copy right is added, otherwise, only google's copyright.

Quoted 2 lines

cc'ed @Hemant and @Stephen for more comment.

#52 lore raw

Re: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 52 lines · Xiaoyun

Hi

Quoted 38 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:45
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Hemant Agrawal <[email protected]>; Stephen Hemminger <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 22 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/
\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

Quoted 11 lines

drivers/net/gve/base/gve.h line 3

Quoted 8 lines
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644 index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

There is a version macro in the code, is version information required in the file comment?

Quoted 2 lines

Different versions of gve kernel driver change a lot. So for reference, I think adding the version info for the base code makes sense. Just tell the following update people which version is used.

No problem to add version to base code, it is already in the code as macro:

#define GVE_VERSION "1.3.0"
#define GVE_VERSION_PREFIX "GVE-"

My question is if to have is in the *file comment* or not, for
it is duplicate and another thing to maintain (I won't get surprised in the future someone update the macro but not file comment).

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
Quoted 8 lines
Quoted 3 lines

I don't know if it is OK to add Intel copyright, as far as I know this requires big enough contribution to the code, if this is copy of existing code, may be only original copyright should exist.

It's not just directly copy. Directly copy like gve_desc.h doesn't have Intel copyright.
But gve.h, in gve kernel driver, it has a lot of info dpdk doesn't need or dpdk has its own version like txq/rxq info.
I'm not sure the contribution is a lot or not. But I suppose this patchset is following the principle that if the code is changed, intel copy right is added, otherwise, only google's copyright.

Thanks Xiaoyun confirming it is not direct copy, can someone in your end check Intel's addition to the file, and if it justifies Copyright or not?

Quoted 2 lines
Quoted 2 lines

cc'ed @Hemant and @Stephen for more comment.

#53 lore raw

RE: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 73 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 21:53
To: Li, Xiaoyun <[email protected]>; Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Hemant Agrawal <[email protected]>; Stephen Hemminger <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 27 lines · Xiaoyun

Hi

Quoted 25 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Wednesday, October 19, 2022 14:45
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Hemant Agrawal <[email protected]>; Stephen Hemminger <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Lin, Xueqin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v5 1/8] net/gve/base: introduce GVE PMD base code

Quoted 11 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linu

Quoted 27 lines
Quoted 15 lines
Quoted 12 lines

x/
\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

Quoted 11 lines

drivers/net/gve/base/gve.h line 3

Quoted 8 lines
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644 index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

There is a version macro in the code, is version information required in the file comment?

Quoted 1 line

Different versions of gve kernel driver change a lot. So for reference, I

think adding the version info for the base code makes sense. Just tell the following update people which version is used.

Quoted 1 line
Quoted 8 lines

No problem to add version to base code, it is already in the code as macro:

  #define GVE_VERSION		"1.3.0"
  #define GVE_VERSION_PREFIX	"GVE-"

My question is if to have is in the *file comment* or not, for it is duplicate and another thing to maintain (I won't get surprised in the future someone update the macro but not file comment).

Forgot about this one. Then it's better to remove it from copyright place then. @Guo, Junfeng

Quoted 12 lines
Quoted 10 lines
Quoted 8 lines

drivers/net/gve/base/gve.h line 5

Quoted 4 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation

I don't know if it is OK to add Intel copyright, as far as I know this requires big enough contribution to the code, if this is copy of existing code, may be only original copyright should exist.

It's not just directly copy. Directly copy like gve_desc.h doesn't have Intel

copyright.

Quoted 1 line

But gve.h, in gve kernel driver, it has a lot of info dpdk doesn't need or

Quoted 8 lines

dpdk has its own version like txq/rxq info.

Quoted 1 line

I'm not sure the contribution is a lot or not. But I suppose this patchset is

following the principle that if the code is changed, intel copy right is added, otherwise, only google's copyright.

Quoted 1 line

Thanks Xiaoyun confirming it is not direct copy, can someone in your end check Intel's addition to the file, and if it justifies Copyright or not?

@Guo, Junfeng Can you update this? I think you can ask Qi's opinion.
If people want to keep the copyright, add one short sentence under copyright to explain what's changed.

In my opinion, there're many changes but those changes are only for adapting kernel code to DPDK. I'm not sure that counts a lot or not and I think Intel's copyright can be removed for the base code. Afterall, Intel's main contribution is ethdev part and dpdk datapath.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

cc'ed @Hemant and @Stephen for more comment.

#54 lore raw

v6 cover introduce GVE PMD

In reply to Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.

Please refer to https://cloud.google.com/compute/docs/networking/using-gvnic
for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

Please info us if there is any object for the copyright concern. Thanks!

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6:
improve the code.

Junfeng Guo (8):
net/gve/base: introduce GVE PMD base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  16 +
 doc/guides/nics/gve.rst                |  76 ++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve.h             |  58 ++
 drivers/net/gve/base/gve_adminq.c      | 925 +++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h      | 383 ++++++++++
 drivers/net/gve/base/gve_desc.h        | 139 ++++
 drivers/net/gve/base/gve_desc_dqo.h    | 256 +++++++
 drivers/net/gve/base/gve_osdep.h       | 159 +++++
 drivers/net/gve/base/gve_register.h    |  30 +
 drivers/net/gve/gve_ethdev.c           | 699 +++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 295 ++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/gve_rx.c               | 354 ++++++++++
 drivers/net/gve/gve_tx.c               | 669 ++++++++++++++++++
 drivers/net/gve/meson.build            |  16 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 20 files changed, 4105 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_osdep.h
 create mode 100644 drivers/net/gve/base/gve_register.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.34.1
#55 lore raw

PATCH v6 1/8 net/gve/base: introduce GVE PMD base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

---
 drivers/net/gve/base/gve.h          |  58 ++
 drivers/net/gve/base/gve_adminq.c   | 924 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h   | 381 ++++++++++++
 drivers/net/gve/base/gve_desc.h     | 137 +++++
 drivers/net/gve/base/gve_desc_dqo.h | 254 ++++++++
 drivers/net/gve/base/gve_register.h |  28 +
 6 files changed, 1782 insertions(+)
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_register.h
drivers/net/gve/base/gve.h new file +58 -0
new file mode 100644
index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION		"1.3.0"
+#define GVE_VERSION_PREFIX	"GVE-"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR		0
+#define GVE_DB_BAR		2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES		1024
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+#endif /* _GVE_H_ */
drivers/net/gve/base/gve_adminq.c new file +924 -0

924 lines not on this page · raw message

drivers/net/gve/base/gve_adminq.h new file +381 -0

381 lines not on this page · raw message

drivers/net/gve/base/gve_desc.h new file +137 -0

137 lines not on this page · raw message

drivers/net/gve/base/gve_desc_dqo.h new file +254 -0

254 lines not on this page · raw message

drivers/net/gve/base/gve_register.h new file +28 -0
new file mode 100644
index 0000000000..b65f336be2
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.34.1
#56 lore raw

Re: [PATCH v6 1/8] net/gve/base: introduce GVE PMD base code

drivers/net/gve/base/gve.h line 3

Quoted 9 lines · Junfeng Guo
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644
index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

[1]

drivers/net/gve/base/gve.h line 5

Quoted 2 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation

[2]

drivers/net/gve/base/gve.h line 15

Quoted 10 lines
drivers/net/gve/base/gve.h new file +10 -0
+ */
+
+#ifndef_GVE_H_
+#define_GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION            "1.3.0"
+#define GVE_VERSION_PREFIX     "GVE-"
+

Is it clarified/decided to keep version in the file comment [1] and keep Intel copyright [2], or is this just not addressed yet?

#57 lore raw

RE: [PATCH v6 1/8] net/gve/base: introduce GVE PMD base code

Quoted 34 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 22:39
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>; Li, Xiaoyun <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v6 1/8] net/gve/base: introduce GVE PMD base code

drivers/net/gve/base/gve.h line 3

Quoted 9 lines · Junfeng Guo
drivers/net/gve/base/gve.h new file +3 -0
new file mode 100644
index 0000000000..1b0d59b639
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Version: 1.3.0

[1]

drivers/net/gve/base/gve.h line 5

Quoted 2 lines
drivers/net/gve/base/gve.h new file +2 -0
+ * Copyright (C) 2015-2022 Google, Inc.
+ * Copyright(C) 2022 Intel Corporation

[2]

Quoted 13 lines

drivers/net/gve/base/gve.h line 15

Quoted 10 lines
drivers/net/gve/base/gve.h new file +10 -0
+ */
+
+#ifndef_GVE_H_
+#define_GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION            "1.3.0"
+#define GVE_VERSION_PREFIX     "GVE-"
+

Is it clarified/decided to keep version in the file comment [1] and keep Intel copyright [2], or is this just not addressed yet?

Yes, we will remove these in the coming version. Thanks!

#58 lore raw

Re: [PATCH v6 1/8] net/gve/base: introduce GVE PMD base code

In reply to Junfeng Guo
Quoted 10 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

Quoted 8 lines

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

Can drop "GVE PMD" from patch title, since 'net/gve/base:' already implies it, like:
net/gve/base: introduce base code

Quoted 4 lines

<...>

drivers/net/gve/base/gve_adminq.c line 243

Quoted 7 lines
drivers/net/gve/base/gve_adminq.c new file +7 -0
+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32 prod_cnt)
+{
+       int i;
+
+       for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++) {
+               if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+                   == prod_cnt)

Syntax, why not move second half of the equation in above line? Unless this is coming from google code and updating it brings maintanance cost.

#59 lore raw

RE: [PATCH v6 1/8] net/gve/base: introduce GVE PMD base code

Quoted 33 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 22:41
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v6 1/8] net/gve/base: introduce GVE PMD base code

Quoted 10 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

Quoted 2 lines

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/\

Quoted 2 lines

tree/v1.3.0/google/gve

Quoted 8 lines
Quoted 4 lines

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

Can drop "GVE PMD" from patch title, since 'net/gve/base:' already implies it, like:
net/gve/base: introduce base code

Sure, make sense!
Will update this in the coming version, thanks!

Quoted 10 lines
Quoted 5 lines

<...>

Quoted 1 line

+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32

prod_cnt)

Quoted 10 lines

drivers/net/gve/base/gve_adminq.c line 243

Quoted 6 lines
drivers/net/gve/base/gve_adminq.c new file +6 -0
+{
+       int i;
+
+       for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++) {
+               if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+                   == prod_cnt)

Syntax, why not move second half of the equation in above line? Unless this is coming from google code and updating it brings maintanance cost.

Yes, this is basic adminq processing function and is coming from google code without any change. Better to keep this unchanged with the origin. Thanks!

#60 lore raw

PATCH v6 2/8 net/gve/base: add OS specific implementation

In reply to Junfeng Guo

Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/base/gve_adminq.h   |   2 +
 drivers/net/gve/base/gve_desc.h     |   2 +
 drivers/net/gve/base/gve_desc_dqo.h |   2 +
 drivers/net/gve/base/gve_osdep.h    | 159 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_register.h |   2 +
 5 files changed, 167 insertions(+)
 create mode 100644 drivers/net/gve/base/gve_osdep.h
drivers/net/gve/base/gve_adminq.h +2 -0
index c7114cc883..cd496760ae 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/base/gve_desc.h +2 -0
index 358755b7e0..627b9120dc 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/base/gve_desc_dqo.h +2 -0
index 0d533abcd1..5031752b43 100644
@@ -9,6 +9,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/base/gve_osdep.h new file +159 -0

159 lines not on this page · raw message

drivers/net/gve/base/gve_register.h +2 -0
index b65f336be2..a599c1a08e 100644
@@ -7,6 +7,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
Signature · 1 line
2.34.1
#61 lore raw

PATCH v6 3/8 net/gve: add support for device initialization

In reply to Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

---
 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  10 +
 doc/guides/nics/gve.rst                |  68 +++++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve_adminq.c      |   1 +
 drivers/net/gve/gve_ethdev.c           | 371 +++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 225 +++++++++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/meson.build            |  14 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 12 files changed, 719 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
index 92b381bc30..2d06a76efe 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
new file mode 100644
index 0000000000..44aec28009
@@ -0,0 +1,10 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +68 -0

68 lines not on this page · raw message

doc/guides/nics/index.rst +1 -0
index 32c7544968..4d40ea29a3 100644
@@ -29,6 +29,7 @@ Network Interface Controller Drivers
     enetfec
     enic
     fm10k
+    gve
     hinic
     hns3
     i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
index 1c3daf141d..715013fa35 100644
@@ -140,6 +140,11 @@ New Features
 
   * Made compatible with libbpf v0.8.0 (when used with libxdp).
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 * **Updated AMD Pensando ionic driver.**
 
   Updated the ionic PMD with new features and improvements, including:
drivers/net/gve/base/gve_adminq.c +1 -0
index 2344100f1a..ceafa05286 100644
@@ -5,6 +5,7 @@
  * Copyright(C) 2022 Intel Corporation
  */
 
+#include "../gve_ethdev.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +371 -0

371 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h new file +225 -0

225 lines not on this page · raw message

drivers/net/gve/gve_logs.h new file +14 -0
new file mode 100644
index 0000000000..0d02da46e1
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+		__func__, ## args)
+
+#endif
drivers/net/gve/meson.build new file +14 -0
new file mode 100644
index 0000000000..d8ec64b3a3
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'base/gve_adminq.c',
+        'gve_ethdev.c',
+)
+includes += include_directories('base')
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index 35bfa78dee..355dbd07e9 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.34.1
#62 lore raw

Re: [PATCH v6 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

doc/guides/rel_notes/release_22_11.rst New Features · line 147

Quoted 12 lines

index 1c3daf141d..715013fa35 100644

doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -140,6 +140,11 @@ New Features
 
    * Made compatible with libbpf v0.8.0 (when used with libxdp).
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+

Can you please move it one more down, just above 'Intel', to sort it alphabetically based on Vendor name, in this case 'G' I guess. We are almost there :)

<...>

Quoted 43 lines
doc/guides/rel_notes/release_22_11.rst +43 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);

doc/guides/rel_notes/release_22_11.rst New Features

Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10 -0
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac address");
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr, eth_dev->data->mac_addrs);
+

What is the value in 'priv->dev_addr'?
Even allocating memory for 'eth_dev->data->mac_addrs' removed or not, as we discussed, independent from it, need to set a valid value to 'priv->dev_addr'.

<...>

drivers/net/gve/version.map line 1

Quoted 7 lines
drivers/net/gve/version.map new file +1 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

In case it is not clear, since this comment skipped in previous a few versions, the ABI version should be 'DPDK_23', so the content of this file should be;

DPDK_23 {
        local: *;
};
#63 lore raw

RE: [PATCH v6 3/8] net/gve: add support for device initialization

Quoted 45 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 22:42
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v6 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

doc/guides/rel_notes/release_22_11.rst New Features · line 146

Quoted 11 lines

index 1c3daf141d..715013fa35 100644

doc/guides/rel_notes/release_22_11.rst +4 -0
@@ -140,6 +140,11 @@ New Features
 
    * Made compatible with libbpf v0.8.0 (when used with libxdp).
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new

driver.

Quoted 5 lines

doc/guides/rel_notes/release_22_11.rst New Features · line 147

Quoted 1 line
doc/guides/rel_notes/release_22_11.rst +1 -0
+

Can you please move it one more down, just above 'Intel', to sort it alphabetically based on Vendor name, in this case 'G' I guess. We are almost there :)

Sure, thanks for reminding this!

Quoted 60 lines

<...>

Quoted 37 lines
doc/guides/rel_notes/release_22_11.rst +37 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;

doc/guides/rel_notes/release_22_11.rst

Quoted 10 lines
doc/guides/rel_notes/release_22_11.rst +10 -0
+       priv->state_flags = 0x0;
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct

rte_ether_addr), 0);

doc/guides/rel_notes/release_22_11.rst

Quoted 2 lines
doc/guides/rel_notes/release_22_11.rst +2 -0
+       if (!eth_dev->data->mac_addrs) {
+               PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac

address");

Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +3 -0
+               return -ENOMEM;
+       }
+       rte_ether_addr_copy(&priv->dev_addr, eth_dev->data-

mac_addrs);
+

Quoted 5 lines

What is the value in 'priv->dev_addr'?
Even allocating memory for 'eth_dev->data->mac_addrs' removed or not, as
we discussed, independent from it, need to set a valid value to 'priv->dev_addr'.

Again, thanks for the explanation!
Will update this and re-validate for this. The addr value could be like " 42:01:0A:00:08:03". Thanks!

Quoted 13 lines

<...>

drivers/net/gve/version.map line 1

Quoted 7 lines
drivers/net/gve/version.map new file +1 -0
new file mode 100644
index 0000000000..c2e0723b4c
@@ -0,0 +1,3 @@
+DPDK_22 {

DPDK_23

Quoted 7 lines

In case it is not clear, since this comment skipped in previous a few versions, the ABI version should be 'DPDK_23', so the content of this file should be;

DPDK_23 {
         local: *;
};

Sure, will update this in the coming version. Thanks a lot!

#64 lore raw

PATCH v6 4/8 net/gve: add support for link update

In reply to Junfeng Guo

Support dev_ops link_update.

---
 doc/guides/nics/features/gve.ini |  1 +
 doc/guides/nics/gve.rst          |  3 +++
 drivers/net/gve/gve_ethdev.c     | 30 ++++++++++++++++++++++++++++++
 3 files changed, 34 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index 44aec28009..ae466ad677 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +3 -0
index 703fbcc5de..c42ff23841 100644
@@ -60,6 +60,9 @@ Features and Limitations
 
 In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
+Supported features of the GVE PMD are:
+
+- Link state information
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +30 -0
index f8ff3b1923..ca4a467140 100644
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_DRV_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -70,6 +99,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.34.1
#65 lore raw

PATCH v6 5/8 net/gve: add support for MTU setting

In reply to Junfeng Guo

Support dev_ops mtu_set.

---
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index ae466ad677..d1703d8dab 100644
@@ -5,6 +5,7 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +27 -0
index ca4a467140..1968f38eb6 100644
@@ -94,12 +94,39 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.34.1
#66 lore raw

Re: [PATCH v6 5/8] net/gve: add support for MTU setting

Quoted 28 lines · Junfeng Guo

Support dev_ops mtu_set.

---
  doc/guides/nics/features/gve.ini |  1 +
  drivers/net/gve/gve_ethdev.c     | 27 +++++++++++++++++++++++++++
  2 files changed, 28 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index ae466ad677..d1703d8dab 100644
@@ -5,6 +5,7 @@
  ;
  [Features]
  Link status          = Y
+MTU update           = Y
  Linux                = Y
  x86-32               = Y
  x86-64               = Y
drivers/net/gve/gve_ethdev.c +0 -0
index ca4a467140..1968f38eb6 100644
@@ -94,12 +94,39 @@ gve_dev_close(struct rte_eth_dev *dev)
         return err;

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 104

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +8 -0
  }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u", RTE_ETHER_MIN_MTU, priv->max_mtu);

Although this is within new 100 column limit, it is easy to break it without sacrificing the readability, can you break it as something like:

PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u",

	RTE_ETHER_MIN_MTU, priv->max_mtu);
Quoted 7 lines
drivers/net/gve/gve_ethdev.c +7 -0
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+               return -EBUSY;

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 121

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+       }
+
+       err = gve_adminq_set_mtu(priv, mtu);
+       if (err) {
+               PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+               return err;
+       }
+
+       return 0;
+}

configure() (gve_dev_configure()) also get 'mtu' as user config ('eth_conf->rxmode.mtu') which is ignored right now,

since there is 'gve_adminq_set_mtu()' command already what do you think to use it within 'gve_dev_configure()'?

Quoted 14 lines
drivers/net/gve/gve_ethdev.c +2 -1
+
  static const struct eth_dev_ops gve_eth_dev_ops = {
         .dev_configure        = gve_dev_configure,
         .dev_start            = gve_dev_start,
         .dev_stop             = gve_dev_stop,
         .dev_close            = gve_dev_close,
         .link_update          = gve_link_update,
+       .mtu_set              = gve_dev_mtu_set,
  };
 
  static void
--

2.34.1

#67 lore raw

RE: [PATCH v6 5/8] net/gve: add support for MTU setting

Quoted 59 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 22:45
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v6 5/8] net/gve: add support for MTU setting

Review

Quoted 11 lines · Junfeng Guo

Support dev_ops mtu_set.

---
  doc/guides/nics/features/gve.ini |  1 +
  drivers/net/gve/gve_ethdev.c     | 27 +++++++++++++++++++++++++++
  2 files changed, 28 insertions(+)
(quoted diff) +0 -0

b/doc/guides/nics/features/gve.ini

Review

Quoted 12 lines

index ae466ad677..d1703d8dab 100644

doc/guides/nics/features/gve.ini +1 -0
@@ -5,6 +5,7 @@
  ;
  [Features]
  Link status          = Y
+MTU update           = Y
  Linux                = Y
  x86-32               = Y
  x86-64               = Y
(quoted diff) +0 -0

b/drivers/net/gve/gve_ethdev.c

Quoted 5 lines

index ca4a467140..1968f38eb6 100644

drivers/net/gve/gve_ethdev.c +0 -0
@@ -94,12 +94,39 @@ gve_dev_close(struct rte_eth_dev *dev)
         return err;

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 104

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +8 -0
  }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u",

RTE_ETHER_MIN_MTU, priv->max_mtu);

Quoted 5 lines

Although this is within new 100 column limit, it is easy to break it without sacrificing the readability, can you break it as something like:

PMD_DRV_LOG(ERR, "MIN MTU is %u MAX MTU is %u",

	RTE_ETHER_MIN_MTU, priv->max_mtu);

Sure, will improve this. Thanks!

Quoted 25 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 7 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before

configuration");

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return -EBUSY;
+       }
+
+       err = gve_adminq_set_mtu(priv, mtu);
+       if (err) {
+               PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu,

err);

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 121

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5 -0
+               return err;
+       }
+
+       return 0;
+}

configure() (gve_dev_configure()) also get 'mtu' as user config ('eth_conf->rxmode.mtu') which is ignored right now,

Quoted 3 lines

since there is 'gve_adminq_set_mtu()' command already what do you think
to use it within 'gve_dev_configure()'?

Do you mean to set the mtu with the user config value like:
'gve_dev_mtu_set(dev, dev->data->dev_conf.rxmode.mtu)' within 'gve_dev_configure()'?

The ' dev->data->dev_conf.rxmode.mtu' I get at dev configure stage is also 1500, which is lager than priv->max_mtu (1460). And this may still cause the testpmd launch failed...

So I'll keep this part unchanged and do more investigations to figure out the mtu issues we met. Thanks!

Quoted 15 lines
Quoted 15 lines
drivers/net/gve/gve_ethdev.c +2 -1
+
  static const struct eth_dev_ops gve_eth_dev_ops = {
         .dev_configure        = gve_dev_configure,
         .dev_start            = gve_dev_start,
         .dev_stop             = gve_dev_stop,
         .dev_close            = gve_dev_close,
         .link_update          = gve_link_update,
+       .mtu_set              = gve_dev_mtu_set,
  };
 
  static void
--

2.34.1

#68 lore raw

PATCH v6 6/8 net/gve: add support for dev info get and dev configure

In reply to Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
 doc/guides/nics/features/gve.ini |  2 ++
 doc/guides/nics/gve.rst          |  1 +
 drivers/net/gve/gve_ethdev.c     | 56 +++++++++++++++++++++++++++++++-
 3 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
index d1703d8dab..986df7f94a 100644
@@ -4,8 +4,10 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+RSS hash             = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +1 -0
index c42ff23841..8c09a5a7fa 100644
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
 Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)
 - Link state information
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
drivers/net/gve/gve_ethdev.c +55 -1

75 lines not on this page · raw message

Signature · 1 line
2.34.1
#69 lore raw

Re: [PATCH v6 6/8] net/gve: add support for dev info get and dev configure

Quoted 17 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
  doc/guides/nics/features/gve.ini |  2 ++
  doc/guides/nics/gve.rst          |  1 +
  drivers/net/gve/gve_ethdev.c     | 56 +++++++++++++++++++++++++++++++-
  3 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +0 -0
index d1703d8dab..986df7f94a 100644

doc/guides/nics/features/gve.ini line 10

Quoted 7 lines
doc/guides/nics/features/gve.ini +2 -0
@@ -4,8 +4,10 @@
  ; Refer to default.ini for the full list of available PMD features.
  ;
  [Features]
+Speed capabilities   = Y
  Link status          = Y
  MTU update           = Y
+RSS hash             = Y

I think this was added because of 'RTE_ETH_RX_OFFLOAD_RSS_HASH', it is OK to keep this feature if you add force enabling above offload, otherwise please remove the feature.

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic functionality of packet · line 65

Quoted 12 lines
doc/guides/nics/features/gve.ini +0 -0
  Linux                = Y
  x86-32               = Y
  x86-64               = Y
doc/guides/nics/gve.rst +1 -0
index c42ff23841..8c09a5a7fa 100644
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic functionality of packet
  reception and transmission.
  Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)

I am not sure if driver can claim this, I can see a RSS hash is provided but is it possible to update which hash function to use or update key or RETA table to configure which queue packets goes?

Right now what is RSS calculated on?

Perpaps RSS support can be documented as limited?

And not sure if this update belongs this patch, it should be to the one that has the datapath.

#70 lore raw

RE: [PATCH v6 6/8] net/gve: add support for dev info get and dev configure

Quoted 33 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 22:46
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v6 6/8] net/gve: add support for dev info get and dev configure

Quoted 10 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads configuration.

---
  doc/guides/nics/features/gve.ini |  2 ++
  doc/guides/nics/gve.rst          |  1 +
  drivers/net/gve/gve_ethdev.c     | 56

+++++++++++++++++++++++++++++++-

Review

Quoted 3 lines
  3 files changed, 58 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/doc/guides/nics/features/gve.ini

Quoted 16 lines

doc/guides/nics/features/gve.ini line 10

Quoted 11 lines

index d1703d8dab..986df7f94a 100644

doc/guides/nics/features/gve.ini +2 -0
@@ -4,8 +4,10 @@
  ; Refer to default.ini for the full list of available PMD features.
  ;
  [Features]
+Speed capabilities   = Y
  Link status          = Y
  MTU update           = Y
+RSS hash             = Y

I think this was added because of 'RTE_ETH_RX_OFFLOAD_RSS_HASH', it is
OK to keep this feature if you add force enabling above offload, otherwise please remove the feature.

Sure, will keep this with the force enabling code at dev config stage. Thanks!

Quoted 19 lines

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic

Quoted 9 lines
doc/guides/nics/features/gve.ini +0 -0
  Linux                = Y
  x86-32               = Y
  x86-64               = Y
doc/guides/nics/gve.rst +0 -0
index c42ff23841..8c09a5a7fa 100644
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic

functionality of packet

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic functionality of packet · line 65

Quoted 4 lines
doc/guides/nics/gve.rst +1 -0
  reception and transmission.
  Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)

I am not sure if driver can claim this, I can see a RSS hash is provided but is it possible to update which hash function to use or update key or RETA table to configure which queue packets goes?

Quoted 6 lines

Right now what is RSS calculated on?

Perpaps RSS support can be documented as limited?

And not sure if this update belongs this patch, it should be to the one that has the datapath.

Same for this, thanks!

Quoted 1 line
#71 lore raw

PATCH v6 7/8 net/gve: add support for queue operations

In reply to Junfeng Guo

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve_ethdev.c | 204 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h |  52 +++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 684 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +204 -0
index 5be8d664f3..52c335bc16 100644
@@ -28,6 +28,68 @@ gve_write_version(uint8_t *driver_version_register)
 	writeb('\n', driver_version_register);
 }
 
+static int
+gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pages)
+{
+	char z_name[RTE_MEMZONE_NAMESIZE];
+	struct gve_queue_page_list *qpl;
+	const struct rte_memzone *mz;
+	dma_addr_t page_bus;
+	uint32_t i;
+
+	if (priv->num_registered_pages + pages >
+	    priv->max_registered_pages) {
+		PMD_DRV_LOG(ERR, "Pages %" PRIu64 " > max registered pages %" PRIu64,
+			    priv->num_registered_pages + pages,
+			    priv->max_registered_pages);
+		return -EINVAL;
+	}
+	qpl = &priv->qpl[id];
+	snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_dev->device.name, id);
+	mz = rte_memzone_reserve_aligned(z_name, pages * PAGE_SIZE,
+					 rte_socket_id(),
+					 RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
+	if (mz == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name);
+		return -ENOMEM;
+	}
+	qpl->page_buses = rte_zmalloc("qpl page buses", pages * sizeof(dma_addr_t), 0);
+	if (qpl->page_buses == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page buses", id);
+		return -ENOMEM;
+	}
+	page_bus = mz->iova;
+	for (i = 0; i < pages; i++) {
+		qpl->page_buses[i] = page_bus;
+		page_bus += PAGE_SIZE;
+	}
+	qpl->id = id;
+	qpl->mz = mz;
+	qpl->num_entries = pages;
+
+	priv->num_registered_pages += pages;
+
+	return 0;
+}
+
+static void
+gve_free_qpls(struct gve_priv *priv)
+{
+	uint16_t nb_txqs = priv->max_nb_txq;
+	uint16_t nb_rxqs = priv->max_nb_rxq;
+	uint32_t i;
+
+	for (i = 0; i < nb_txqs + nb_rxqs; i++) {
+		if (priv->qpl[i].mz != NULL)
+			rte_memzone_free(priv->qpl[i].mz);
+		if (priv->qpl[i].page_buses != NULL)
+			rte_free(priv->qpl[i].page_buses);
+	}
+
+	if (priv->qpl != NULL)
+		rte_free(priv->qpl);
+}
+
 static int
 gve_dev_configure(struct rte_eth_dev *dev)
 {
@@ -39,6 +101,43 @@ gve_dev_configure(struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_refill_pages(struct gve_rx_queue *rxq)
+{
+	struct rte_mbuf *nmb;
+	uint16_t i;
+	int diag;
+
+	diag = rte_pktmbuf_alloc_bulk(rxq->mpool, &rxq->sw_ring[0], rxq->nb_rx_desc);
+	if (diag < 0) {
+		for (i = 0; i < rxq->nb_rx_desc - 1; i++) {
+			nmb = rte_pktmbuf_alloc(rxq->mpool);
+			if (!nmb)
+				break;
+			rxq->sw_ring[i] = nmb;
+		}
+		if (i < rxq->nb_rx_desc - 1)
+			return -ENOMEM;
+	}
+	rxq->nb_avail = 0;
+	rxq->next_avail = rxq->nb_rx_desc - 1;
+
+	for (i = 0; i < rxq->nb_rx_desc; i++) {
+		if (rxq->is_gqi_qpl) {
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(i * PAGE_SIZE);
+		} else {
+			if (i == rxq->nb_rx_desc - 1)
+				break;
+			nmb = rxq->sw_ring[i];
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(rte_mbuf_data_iova(nmb));
+		}
+	}
+
+	rte_write32(rte_cpu_to_be_32(rxq->next_avail), rxq->qrx_tail);
+
+	return 0;
+}
+
 static int
 gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 {
@@ -70,16 +169,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
+	uint16_t num_queues = dev->data->nb_tx_queues;
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
+	uint16_t i;
+	int err;
+
+	priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues;
+	err = gve_adminq_create_tx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u tx queues.", num_queues);
+		return err;
+	}
+	for (i = 0; i < num_queues; i++) {
+		txq = priv->txqs[i];
+		txq->qtx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(txq->qres->db_index)];
+		txq->qtx_head =
+		&priv->cnt_array[rte_be_to_cpu_32(txq->qres->counter_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), txq->ntfy_addr);
+	}
+
+	num_queues = dev->data->nb_rx_queues;
+	priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues;
+	err = gve_adminq_create_rx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u rx queues.", num_queues);
+		goto err_tx;
+	}
+	for (i = 0; i < num_queues; i++) {
+		rxq = priv->rxqs[i];
+		rxq->qrx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(rxq->qres->db_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), rxq->ntfy_addr);
+
+		err = gve_refill_pages(rxq);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to refill for RX");
+			goto err_rx;
+		}
+	}
+
 	dev->data->dev_started = 1;
 	gve_link_update(dev, 0);
 
 	return 0;
+
+err_rx:
+	gve_stop_rx_queues(dev);
+err_tx:
+	gve_stop_tx_queues(dev);
+	return err;
 }
 
 static int
 gve_dev_stop(struct rte_eth_dev *dev)
 {
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+
+	gve_stop_tx_queues(dev);
+	gve_stop_rx_queues(dev);
+
 	dev->data->dev_started = 0;
 
 	return 0;
@@ -88,7 +241,11 @@ gve_dev_stop(struct rte_eth_dev *dev)
 static int
 gve_dev_close(struct rte_eth_dev *dev)
 {
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
 	int err = 0;
+	uint16_t i;
 
 	if (dev->data->dev_started) {
 		err = gve_dev_stop(dev);
@@ -96,6 +253,19 @@ gve_dev_close(struct rte_eth_dev *dev)
 			PMD_DRV_LOG(ERR, "Failed to stop dev.");
 	}
 
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		txq = dev->data->tx_queues[i];
+		gve_tx_queue_release(txq);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		rxq = dev->data->rx_queues[i];
+		gve_rx_queue_release(rxq);
+	}
+
+	gve_free_qpls(priv);
+	rte_free(priv->adminq);
+
 	return err;
 }
 
@@ -179,6 +349,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.dev_infos_get        = gve_dev_info_get,
+	.rx_queue_setup       = gve_rx_queue_setup,
+	.tx_queue_setup       = gve_tx_queue_setup,
 	.link_update          = gve_link_update,
 	.mtu_set              = gve_dev_mtu_set,
 };
@@ -316,7 +488,9 @@ gve_setup_device_resources(struct gve_priv *priv)
 static int
 gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 {
+	uint16_t pages;
 	int num_ntfy;
+	uint32_t i;
 	int err;
 
 	/* Set up the adminq */
@@ -367,10 +541,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 	PMD_DRV_LOG(INFO, "Max TX queues %d, Max RX queues %d",
 		    priv->max_nb_txq, priv->max_nb_rxq);
 
+	/* In GQI_QPL queue format:
+	 * Allocate queue page lists according to max queue number
+	 * tx qpl id should start from 0 while rx qpl id should start
+	 * from priv->max_nb_txq
+	 */
+	if (priv->queue_format == GVE_GQI_QPL_FORMAT) {
+		priv->qpl = rte_zmalloc("gve_qpl",
+					(priv->max_nb_txq + priv->max_nb_rxq) *
+					sizeof(struct gve_queue_page_list), 0);
+		if (priv->qpl == NULL) {
+			PMD_DRV_LOG(ERR, "Failed to alloc qpl.");
+			err = -ENOMEM;
+			goto free_adminq;
+		}
+
+		for (i = 0; i < priv->max_nb_txq + priv->max_nb_rxq; i++) {
+			if (i < priv->max_nb_txq)
+				pages = priv->tx_pages_per_qpl;
+			else
+				pages = priv->rx_data_slot_cnt;
+			err = gve_alloc_queue_page_list(priv, i, pages);
+			if (err != 0) {
+				PMD_DRV_LOG(ERR, "Failed to alloc qpl %u.", i);
+				goto err_qpl;
+			}
+		}
+	}
+
 setup_device:
 	err = gve_setup_device_resources(priv);
 	if (!err)
 		return 0;
+err_qpl:
+	gve_free_qpls(priv);
 free_adminq:
 	gve_adminq_free(priv);
 	return err;
drivers/net/gve/gve_ethdev.h +52 -0

92 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index d8ec64b3a3..af0010c01c 100644
@@ -9,6 +9,8 @@ endif
 
 sources = files(
         'base/gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
 includes += include_directories('base')
Signature · 1 line
2.34.1
#72 lore raw

PATCH v6 8/8 net/gve: add support for Rx/Tx

In reply to Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 doc/guides/nics/features/gve.ini |   2 +
 doc/guides/nics/gve.rst          |   4 +
 drivers/net/gve/gve_ethdev.c     |  15 +-
 drivers/net/gve/gve_ethdev.h     |  18 ++
 drivers/net/gve/gve_rx.c         | 142 ++++++++++
 drivers/net/gve/gve_tx.c         | 455 +++++++++++++++++++++++++++++++
 6 files changed, 635 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
index 986df7f94a..cdc46b08a3 100644
@@ -7,7 +7,9 @@
 Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+TSO                  = Y
 RSS hash             = Y
+L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +4 -0
index 8c09a5a7fa..1042852fd6 100644
@@ -62,8 +62,12 @@ In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
 Supported features of the GVE PMD are:
 
+- Multiple queues for TX and RX
 - Receiver Side Scaling (RSS)
+- TSO offload
 - Link state information
+- TX multi-segments (Scatter TX)
+- Tx UDP/TCP/SCTP Checksum
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +14 -1
index 52c335bc16..91f930957d 100644
@@ -284,7 +284,13 @@ gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->min_mtu = RTE_ETHER_MIN_MTU;
 
 	dev_info->rx_offload_capa = 0;
-	dev_info->tx_offload_capa = 0;
+	dev_info->tx_offload_capa =
+		RTE_ETH_TX_OFFLOAD_MULTI_SEGS	|
+		RTE_ETH_TX_OFFLOAD_IPV4_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_UDP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_TCP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_SCTP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_TCP_TSO;
 
 	if (priv->queue_format == GVE_DQO_RDA_FORMAT)
 		dev_info->rx_offload_capa |= RTE_ETH_RX_OFFLOAD_TCP_LRO;
@@ -633,6 +639,13 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	} else {
+		PMD_DRV_LOG(ERR, "DQO_RDA is not implemented and will be added in the future");
+	}
+
 	eth_dev->data->mac_addrs = rte_zmalloc("gve_mac", sizeof(struct rte_ether_addr), 0);
 	if (!eth_dev->data->mac_addrs) {
 		PMD_DRV_LOG(ERR, "Failed to allocate memory to store mac address");
drivers/net/gve/gve_ethdev.h +18 -0
index 20fe57781e..266b831a01 100644
@@ -34,6 +34,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -274,4 +286,10 @@ gve_stop_tx_queues(struct rte_eth_dev *dev);
 void
 gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t
+gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t
+gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +142 -0

148 lines not on this page · raw message

drivers/net/gve/gve_tx.c +455 -0

461 lines not on this page · raw message

Signature · 1 line
2.34.1
#73 lore raw

Re: [PATCH v6 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 36 lines
drivers/net/gve/gve_rx.c +36 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+       uint16_t i;
+
+       rxr = rxq->rx_desc_ring;
+       nb_rx = 0;
+
+       for (i = 0; i < nb_pkts; i++) {
+               rxd = &rxr[rx_id];
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
+                       break;
+
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               if (rxq->is_gqi_qpl) {
+                       addr = (uint64_t)(rxq->qpl->mz->addr) + rx_id * PAGE_SIZE + GVE_RX_PAD;
+                       rte_memcpy((void *)((size_t)rxe->buf_addr + rxe->data_off),
+                                  (void *)(size_t)addr, len);
+               }
+               rxe->pkt_len = len;
+               rxe->data_len = len;
+               rxe->port = rxq->port_id;
+               rxe->ol_flags = 0;
+
+               if (rxd->flags_seq & GVE_RXF_TCP)
+                       rxe->packet_type |= RTE_PTYPE_L4_TCP;

drivers/net/gve/gve_rx.c line 128

Quoted 10 lines
drivers/net/gve/gve_rx.c +10 -0
+               if (rxd->flags_seq & GVE_RXF_UDP)
+                       rxe->packet_type |= RTE_PTYPE_L4_UDP;
+               if (rxd->flags_seq & GVE_RXF_IPV4)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV4;
+               if (rxd->flags_seq & GVE_RXF_IPV6)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV6;
+
+               if (gve_needs_rss(rxd->flags_seq)) {
+                       rxe->ol_flags |= RTE_MBUF_F_RX_RSS_HASH;
+                       rxe->hash.rss = rte_be_to_cpu_32(rxd->rss_hash);

You are updating "m->hash.rss" anyway, and if this is without and cost you can force enable as done in previous version:
'dev->data->dev_conf.rxmode.offloads |= RTE_ETH_RX_OFFLOAD_RSS_HASH;'

<...>

Quoted 7 lines
drivers/net/gve/gve_tx.c +7 -0
+static inline void
+gve_free_bulk_mbuf(struct rte_mbuf **txep, int num)
+{
+       struct rte_mbuf *m, *free[GVE_TX_MAX_FREE_SZ];
+       int nb_free = 0;
+       int i, s;
+

drivers/net/gve/gve_tx.c line 24

Quoted 10 lines
drivers/net/gve/gve_tx.c +10 -0
+       if (unlikely(num == 0))
+               return;
+
+       /* Find the 1st mbuf which needs to be free */
+       for (s = 0; s < num; s++) {
+               if (txep[s] != NULL) {
+                       m = rte_pktmbuf_prefree_seg(txep[s]);
+                       if (m != NULL)
+                               break;
+                       }

'}' indentation is wrong.

<...>

Quoted 16 lines
drivers/net/gve/gve_tx.c +16 -0
+static inline void
+gve_tx_clean_swr_qpl(struct gve_tx_queue *txq)
+{
+       uint32_t start = txq->sw_ntc;
+       uint32_t ntc, nb_clean;
+
+       ntc = txq->sw_tail;
+
+       if (ntc == start)
+               return;
+
+       /* if wrap around, free twice. */
+       if (ntc < start) {
+               nb_clean = txq->nb_tx_desc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;

drivers/net/gve/gve_tx.c line 130

Quoted 10 lines
drivers/net/gve/gve_tx.c +10 -0
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               if (start == txq->nb_tx_desc)
+                       start = 0;
+               txq->sw_ntc = start;
+       }
+
+       if (ntc > start) {

may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.

<...>

drivers/net/gve/gve_tx.c line 462

Quoted 11 lines
drivers/net/gve/gve_tx.c +11 -0
+uint16_t
+gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+       struct gve_tx_queue *txq = tx_queue;
+
+       if (txq->is_gqi_qpl)
+               return gve_tx_burst_qpl(tx_queue, tx_pkts, nb_pkts);
+
+       return gve_tx_burst_ra(tx_queue, tx_pkts, nb_pkts);
+}
+

Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check on datapath.

This is optimization and can be done later, it doesn't have to be in the set.

#74 lore raw

RE: [PATCH v6 8/8] net/gve: add support for Rx/Tx

Quoted 71 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, October 20, 2022 22:47
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v6 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Review

Quoted 2 lines
(quoted diff) +2 -0
+uint16_t
+gve_rx_burst(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t

nb_pkts)

Quoted 14 lines
drivers/net/gve/gve_rx.c +14 -0
+{
+       volatile struct gve_rx_desc *rxr, *rxd;
+       struct gve_rx_queue *rxq = rx_queue;
+       uint16_t rx_id = rxq->rx_tail;
+       struct rte_mbuf *rxe;
+       uint16_t nb_rx, len;
+       uint64_t addr;
+       uint16_t i;
+
+       rxr = rxq->rx_desc_ring;
+       nb_rx = 0;
+
+       for (i = 0; i < nb_pkts; i++) {
+               rxd = &rxr[rx_id];

drivers/net/gve/gve_rx.c

Quoted 10 lines
drivers/net/gve/gve_rx.c +10 -0
+               if (GVE_SEQNO(rxd->flags_seq) != rxq->expected_seqno)
+                       break;
+
+               if (rxd->flags_seq & GVE_RXF_ERR)
+                       continue;
+
+               len = rte_be_to_cpu_16(rxd->len) - GVE_RX_PAD;
+               rxe = rxq->sw_ring[rx_id];
+               if (rxq->is_gqi_qpl) {
+                       addr = (uint64_t)(rxq->qpl->mz->addr) + rx_id * PAGE_SIZE

+ GVE_RX_PAD;

Quoted 18 lines

+ rte_memcpy((void *)((size_t)rxe->buf_addr + rxe-
data_off),

drivers/net/gve/gve_rx.c +16 -0
+                                  (void *)(size_t)addr, len);
+               }
+               rxe->pkt_len = len;
+               rxe->data_len = len;
+               rxe->port = rxq->port_id;
+               rxe->ol_flags = 0;
+
+               if (rxd->flags_seq & GVE_RXF_TCP)
+                       rxe->packet_type |= RTE_PTYPE_L4_TCP;
+               if (rxd->flags_seq & GVE_RXF_UDP)
+                       rxe->packet_type |= RTE_PTYPE_L4_UDP;
+               if (rxd->flags_seq & GVE_RXF_IPV4)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV4;
+               if (rxd->flags_seq & GVE_RXF_IPV6)
+                       rxe->packet_type |= RTE_PTYPE_L3_IPV6;
+
Quoted 8 lines

drivers/net/gve/gve_rx.c line 128

Quoted 3 lines
drivers/net/gve/gve_rx.c +3 -0
+               if (gve_needs_rss(rxd->flags_seq)) {
+                       rxe->ol_flags |= RTE_MBUF_F_RX_RSS_HASH;
+                       rxe->hash.rss = rte_be_to_cpu_32(rxd->rss_hash);

You are updating "m->hash.rss" anyway, and if this is without and cost you can force enable as done in previous version:
'dev->data->dev_conf.rxmode.offloads |= RTE_ETH_RX_OFFLOAD_RSS_HASH;'

Yes, it seems the RSS is enabled by default with no obvious perf loss. There is no RSS init stage. We will also force enable this in the dev config stage. Thanks!

Quoted 14 lines

<...>

Quoted 11 lines
drivers/net/gve/gve_tx.c +11 -0
+static inline void
+gve_free_bulk_mbuf(struct rte_mbuf **txep, int num)
+{
+       struct rte_mbuf *m, *free[GVE_TX_MAX_FREE_SZ];
+       int nb_free = 0;
+       int i, s;
+
+       if (unlikely(num == 0))
+               return;
+
+       /* Find the 1st mbuf which needs to be free */
Quoted 8 lines

drivers/net/gve/gve_tx.c line 24

Quoted 6 lines
drivers/net/gve/gve_tx.c +6 -0
+       for (s = 0; s < num; s++) {
+               if (txep[s] != NULL) {
+                       m = rte_pktmbuf_prefree_seg(txep[s]);
+                       if (m != NULL)
+                               break;
+                       }

'}' indentation is wrong.

Thanks for the catch! Will update in the coming version.

Quoted 24 lines

<...>

Quoted 21 lines
drivers/net/gve/gve_tx.c +21 -0
+static inline void
+gve_tx_clean_swr_qpl(struct gve_tx_queue *txq)
+{
+       uint32_t start = txq->sw_ntc;
+       uint32_t ntc, nb_clean;
+
+       ntc = txq->sw_tail;
+
+       if (ntc == start)
+               return;
+
+       /* if wrap around, free twice. */
+       if (ntc < start) {
+               nb_clean = txq->nb_tx_desc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               if (start == txq->nb_tx_desc)
Quoted 8 lines

drivers/net/gve/gve_tx.c line 130

Quoted 5 lines
drivers/net/gve/gve_tx.c +5 -0
+                       start = 0;
+               txq->sw_ntc = start;
+       }
+
+       if (ntc > start) {

may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.

Sure, will drop this 'if' in the coming version. Thanks!

Quoted 20 lines

<...>

Review

Quoted 2 lines
(quoted diff) +2 -0
+uint16_t
+gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t

nb_pkts)

drivers/net/gve/gve_tx.c line 462

Quoted 9 lines
drivers/net/gve/gve_tx.c +9 -0
+{
+       struct gve_tx_queue *txq = tx_queue;
+
+       if (txq->is_gqi_qpl)
+               return gve_tx_burst_qpl(tx_queue, tx_pkts, nb_pkts);
+
+       return gve_tx_burst_ra(tx_queue, tx_pkts, nb_pkts);
+}
+

Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check on datapath.

Quoted 2 lines

This is optimization and can be done later, it doesn't have to be in the set.

Maybe not. There are three queue format types, and we can get the real used one via adminq with 'priv->queue_format'. So there may not have mix of the queue types. And currently only GQI_QPL and GQI_RDA queue format are supported in PMD. Also, only GQI_QPL queue format is in use on GCP since GQI_RDA hasn't been released in production.

Will do some refactors for the queue types later. Thanks for the advice!

#75 lore raw

v7 cover introduce GVE PMD

In reply to Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.

Please refer to https://cloud.google.com/compute/docs/networking/using-gvnic
for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6-v7:
improve the code.

Junfeng Guo (8):
net/gve/base: introduce base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  16 +
 doc/guides/nics/gve.rst                |  76 ++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve.h             |  56 ++
 drivers/net/gve/base/gve_adminq.c      | 923 +++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h      | 381 ++++++++++
 drivers/net/gve/base/gve_desc.h        | 138 ++++
 drivers/net/gve/base/gve_desc_dqo.h    | 255 +++++++
 drivers/net/gve/base/gve_osdep.h       | 159 +++++
 drivers/net/gve/base/gve_register.h    |  29 +
 drivers/net/gve/gve_ethdev.c           | 700 +++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 298 ++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/gve_rx.c               | 354 ++++++++++
 drivers/net/gve/gve_tx.c               | 668 ++++++++++++++++++
 drivers/net/gve/meson.build            |  16 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 20 files changed, 4099 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_osdep.h
 create mode 100644 drivers/net/gve/base/gve_register.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.34.1
#76 lore raw

PATCH v7 1/8 net/gve/base: introduce base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

---
 drivers/net/gve/base/gve.h          |  56 ++
 drivers/net/gve/base/gve_adminq.c   | 922 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h   | 379 ++++++++++++
 drivers/net/gve/base/gve_desc.h     | 136 ++++
 drivers/net/gve/base/gve_desc_dqo.h | 253 ++++++++
 drivers/net/gve/base/gve_register.h |  27 +
 6 files changed, 1773 insertions(+)
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_register.h
drivers/net/gve/base/gve.h new file +56 -0
new file mode 100644
index 0000000000..2dc4507acb
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION		"1.3.0"
+#define GVE_VERSION_PREFIX	"GVE-"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR		0
+#define GVE_DB_BAR		2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES		1024
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+#endif /* _GVE_H_ */
drivers/net/gve/base/gve_adminq.c new file +922 -0

922 lines not on this page · raw message

drivers/net/gve/base/gve_adminq.h new file +379 -0

379 lines not on this page · raw message

drivers/net/gve/base/gve_desc.h new file +136 -0

136 lines not on this page · raw message

drivers/net/gve/base/gve_desc_dqo.h new file +253 -0

253 lines not on this page · raw message

drivers/net/gve/base/gve_register.h new file +27 -0
new file mode 100644
index 0000000000..bf7f102cde
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.34.1
#77 lore raw

Re: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 21 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

drivers/net/gve/base/gve_adminq.c line 241

Quoted 7 lines
drivers/net/gve/base/gve_adminq.c new file +7 -0
+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32 prod_cnt)
+{
+	int i;
+
+	for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++) {
+		if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+		    == prod_cnt)

[copy/paste from previous version]

Syntax, why not move second half of the equation in above line? Unless this is coming from google code and updating it brings maintanance cost.

#78 lore raw

RE: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 49 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 11 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/\

Quoted 10 lines

tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

Quoted 1 line

+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32

prod_cnt)

drivers/net/gve/base/gve_adminq.c

Quoted 4 lines
drivers/net/gve/base/gve_adminq.c new file +4 -0
+{
+	int i;
+
+	for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++)

{

Quoted 8 lines

drivers/net/gve/base/gve_adminq.c line 241

Quoted 2 lines
drivers/net/gve/base/gve_adminq.c new file +2 -0
+		if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+		    == prod_cnt)

[copy/paste from previous version]

Syntax, why not move second half of the equation in above line? Unless this is coming from google code and updating it brings maintanance cost.

Yes, this function is just coming from google code without changes. So it would be better to keep this unchanged. Thanks!

Hi Ferruh,
Sorry for the late reply in v6, my mistake.
The replies are edited as drafts but not sent out in the end... Please help review, really thanks!

#79 lore raw

Re: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 54 lines · Junfeng
Quoted 54 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 11 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/\

Quoted 10 lines

tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

Quoted 1 line

+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32

prod_cnt)

drivers/net/gve/base/gve_adminq.c

Quoted 4 lines
drivers/net/gve/base/gve_adminq.c new file +4 -0
+{
+	int i;
+
+	for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++)

{

drivers/net/gve/base/gve_adminq.c line 241

Quoted 2 lines
drivers/net/gve/base/gve_adminq.c new file +2 -0
+		if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+		    == prod_cnt)

[copy/paste from previous version]

Quoted 7 lines
Quoted 3 lines

Syntax, why not move second half of the equation in above line? Unless this is coming from google code and updating it brings maintanance cost.

Yes, this function is just coming from google code without changes. So it would be better to keep this unchanged. Thanks!

ack

#80 lore raw

RE: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 76 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:48
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 26 lines · Junfeng
Quoted 26 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 9 lines · Junfeng Guo

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/\

Quoted 3 lines

tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the

kernel

Quoted 26 lines
Quoted 25 lines
Quoted 7 lines

community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

<...>

Quoted 1 line

+static bool gve_adminq_wait_for_cmd(struct gve_priv *priv, u32

prod_cnt)

drivers/net/gve/base/gve_adminq.c

Quoted 4 lines
drivers/net/gve/base/gve_adminq.c new file +4 -0
+{
+	int i;
+
+	for (i = 0; i < GVE_MAX_ADMINQ_EVENT_COUNTER_CHECK; i++)

{

drivers/net/gve/base/gve_adminq.c line 241

Quoted 2 lines
drivers/net/gve/base/gve_adminq.c new file +2 -0
+		if (ioread32be(&priv->reg_bar0->adminq_event_counter)
+		    == prod_cnt)

[copy/paste from previous version]

Syntax, why not move second half of the equation in above line? Unless this is coming from google code and updating it brings maintanance cost.

Quoted 5 lines
Quoted 3 lines

Yes, this function is just coming from google code without changes. So it would be better to keep this unchanged. Thanks!

ack

Thanks for your review!

#81 lore raw

Re: [PATCH v7 1/8] net/gve/base: introduce base code

In reply to Junfeng Guo
Quoted 71 lines · Junfeng Guo
drivers/net/gve/base/gve_adminq.c new file +71 -0
+int gve_adminq_describe_device(struct gve_priv *priv)
+{
+	struct gve_device_option_jumbo_frames *dev_op_jumbo_frames = NULL;
+	struct gve_device_option_gqi_rda *dev_op_gqi_rda = NULL;
+	struct gve_device_option_gqi_qpl *dev_op_gqi_qpl = NULL;
+	struct gve_device_option_dqo_rda *dev_op_dqo_rda = NULL;
+	struct gve_device_descriptor *descriptor;
+	struct gve_dma_mem descriptor_dma_mem;
+	u32 supported_features_mask = 0;
+	union gve_adminq_command cmd;
+	int err = 0;
+	u8 *mac;
+	u16 mtu;
+
+	memset(&cmd, 0, sizeof(cmd));
+	descriptor = gve_alloc_dma_mem(&descriptor_dma_mem, PAGE_SIZE);
+	if (!descriptor)
+		return -ENOMEM;
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESCRIBE_DEVICE);
+	cmd.describe_device.device_descriptor_addr =
+					cpu_to_be64(descriptor_dma_mem.pa);
+	cmd.describe_device.device_descriptor_version =
+			cpu_to_be32(GVE_ADMINQ_DEVICE_DESCRIPTOR_VERSION);
+	cmd.describe_device.available_length = cpu_to_be32(PAGE_SIZE);
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	if (err)
+		goto free_device_descriptor;
+
+	err = gve_process_device_options(priv, descriptor, &dev_op_gqi_rda,
+					 &dev_op_gqi_qpl, &dev_op_dqo_rda,
+					 &dev_op_jumbo_frames);
+	if (err)
+		goto free_device_descriptor;
+
+	/* If the GQI_RAW_ADDRESSING option is not enabled and the queue format
+	 * is not set to GqiRda, choose the queue format in a priority order:
+	 * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default.
+	 */
+	if (dev_op_dqo_rda) {
+		priv->queue_format = GVE_DQO_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with DQO RDA queue format.");
+		supported_features_mask =
+			be32_to_cpu(dev_op_dqo_rda->supported_features_mask);
+	} else if (dev_op_gqi_rda) {
+		priv->queue_format = GVE_GQI_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+		supported_features_mask =
+			be32_to_cpu(dev_op_gqi_rda->supported_features_mask);
+	} else if (priv->queue_format == GVE_GQI_RDA_FORMAT) {
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA queue format.");
+	} else {
+		priv->queue_format = GVE_GQI_QPL_FORMAT;
+		if (dev_op_gqi_qpl)
+			supported_features_mask =
+				be32_to_cpu(dev_op_gqi_qpl->supported_features_mask);
+		PMD_DRV_LOG(INFO, "Driver is running with GQI QPL queue format.");
+	}
+	if (gve_is_gqi(priv)) {
+		err = gve_set_desc_cnt(priv, descriptor);
+	} else {
+		/* DQO supports LRO. */
+		err = gve_set_desc_cnt_dqo(priv, descriptor, dev_op_dqo_rda);
+	}
+	if (err)
+		goto free_device_descriptor;
+
+	priv->max_registered_pages =
+				be64_to_cpu(descriptor->max_registered_pages);
+	mtu = be16_to_cpu(descriptor->mtu);
+	if (mtu < ETH_MIN_MTU) {

drivers/net/gve/base/gve_adminq.c line 769

Quoted 10 lines
drivers/net/gve/base/gve_adminq.c new file +10 -0
+		PMD_DRV_LOG(ERR, "MTU %d below minimum MTU", mtu);
+		err = -EINVAL;
+		goto free_device_descriptor;
+	}
+	priv->max_mtu = mtu;
+	priv->num_event_counters = be16_to_cpu(descriptor->counters);
+	rte_memcpy(priv->dev_addr.addr_bytes, descriptor->mac, ETH_ALEN);
+	mac = descriptor->mac;
+	PMD_DRV_LOG(INFO, "MAC addr: %02x:%02x:%02x:%02x:%02x:%02x",
+		    mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);

There are 'RTE_ETHER_ADDR_PRT_FMT' & 'RTE_ETHER_ADDR_BYTES' macros for this purpose, you can use it like:

PMD_DRV_LOG(INFO, "MAC addr" RTE_ETHER_ADDR_PRT_FMT, RTE_ETHER_ADDR_BYTES(priv->dev_addr));

So can get rid of 'mac' variable.

#82 lore raw

RE: [PATCH v7 1/8] net/gve/base: introduce base code

Quoted 124 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:51
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 1/8] net/gve/base: introduce base code

drivers/net/gve/base/gve_adminq.c

Quoted 3 lines · Junfeng Guo
drivers/net/gve/base/gve_adminq.c new file +3 -0
+int gve_adminq_describe_device(struct gve_priv *priv)
+{
+	struct gve_device_option_jumbo_frames

*dev_op_jumbo_frames = NULL;

drivers/net/gve/base/gve_adminq.c

Quoted 13 lines
drivers/net/gve/base/gve_adminq.c new file +13 -0
+	struct gve_device_option_gqi_rda *dev_op_gqi_rda = NULL;
+	struct gve_device_option_gqi_qpl *dev_op_gqi_qpl = NULL;
+	struct gve_device_option_dqo_rda *dev_op_dqo_rda = NULL;
+	struct gve_device_descriptor *descriptor;
+	struct gve_dma_mem descriptor_dma_mem;
+	u32 supported_features_mask = 0;
+	union gve_adminq_command cmd;
+	int err = 0;
+	u8 *mac;
+	u16 mtu;
+
+	memset(&cmd, 0, sizeof(cmd));
+	descriptor = gve_alloc_dma_mem(&descriptor_dma_mem,

PAGE_SIZE);

drivers/net/gve/base/gve_adminq.c line 713

Quoted 5 lines
drivers/net/gve/base/gve_adminq.c new file +5 -0
+	if (!descriptor)
+		return -ENOMEM;
+	cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESCRIBE_DEVICE);
+	cmd.describe_device.device_descriptor_addr =
+
	cpu_to_be64(descriptor_dma_mem.pa);

drivers/net/gve/base/gve_adminq.c line 713

Quoted 2 lines
drivers/net/gve/base/gve_adminq.c new file +2 -0
+	cmd.describe_device.device_descriptor_version =
+
	cpu_to_be32(GVE_ADMINQ_DEVICE_DESCRIPTOR_VERSION);

drivers/net/gve/base/gve_adminq.c

Quoted 7 lines
drivers/net/gve/base/gve_adminq.c new file +7 -0
+	cmd.describe_device.available_length = cpu_to_be32(PAGE_SIZE);
+
+	err = gve_adminq_execute_cmd(priv, &cmd);
+	if (err)
+		goto free_device_descriptor;
+
+	err = gve_process_device_options(priv, descriptor,

&dev_op_gqi_rda,

Review

Quoted 1 line
(quoted diff) +1 -0
+					 &dev_op_gqi_qpl,

&dev_op_dqo_rda,

drivers/net/gve/base/gve_adminq.c

Quoted 5 lines
drivers/net/gve/base/gve_adminq.c new file +5 -0
+					 &dev_op_jumbo_frames);
+	if (err)
+		goto free_device_descriptor;
+
+	/* If the GQI_RAW_ADDRESSING option is not enabled and the

queue format

Review

Quoted 1 line
(quoted diff) +1 -0
+	 * is not set to GqiRda, choose the queue format in a priority

order:

drivers/net/gve/base/gve_adminq.c

Quoted 5 lines
drivers/net/gve/base/gve_adminq.c new file +5 -0
+	 * DqoRda, GqiRda, GqiQpl. Use GqiQpl as default.
+	 */
+	if (dev_op_dqo_rda) {
+		priv->queue_format = GVE_DQO_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with DQO RDA

queue format.");

drivers/net/gve/base/gve_adminq.c

Quoted 6 lines
(quoted diff) +2 -0
+		supported_features_mask =
+			be32_to_cpu(dev_op_dqo_rda-

supported_features_mask);

drivers/net/gve/base/gve_adminq.c new file +3 -0
+	} else if (dev_op_gqi_rda) {
+		priv->queue_format = GVE_GQI_RDA_FORMAT;
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA

queue format.");

Review

Quoted 5 lines
(quoted diff) +2 -0
+		supported_features_mask =
+			be32_to_cpu(dev_op_gqi_rda-

supported_features_mask);

(quoted diff) +2 -0
+	} else if (priv->queue_format == GVE_GQI_RDA_FORMAT) {
+		PMD_DRV_LOG(INFO, "Driver is running with GQI RDA

queue format.");

Review

Quoted 7 lines
drivers/net/gve/base/gve_adminq.c new file +5 -0
+	} else {
+		priv->queue_format = GVE_GQI_QPL_FORMAT;
+		if (dev_op_gqi_qpl)
+			supported_features_mask =
+				be32_to_cpu(dev_op_gqi_qpl-

supported_features_mask);

(quoted diff) +1 -0
+		PMD_DRV_LOG(INFO, "Driver is running with GQI QPL

queue format.");

drivers/net/gve/base/gve_adminq.c

Quoted 6 lines
drivers/net/gve/base/gve_adminq.c new file +6 -0
+	}
+	if (gve_is_gqi(priv)) {
+		err = gve_set_desc_cnt(priv, descriptor);
+	} else {
+		/* DQO supports LRO. */
+		err = gve_set_desc_cnt_dqo(priv, descriptor,

dev_op_dqo_rda);

drivers/net/gve/base/gve_adminq.c

Quoted 10 lines
drivers/net/gve/base/gve_adminq.c new file +6 -0
+	}
+	if (err)
+		goto free_device_descriptor;
+
+	priv->max_registered_pages =
+				be64_to_cpu(descriptor-

max_registered_pages);

drivers/net/gve/base/gve_adminq.c new file +3 -0
+	mtu = be16_to_cpu(descriptor->mtu);
+	if (mtu < ETH_MIN_MTU) {
+		PMD_DRV_LOG(ERR, "MTU %d below minimum MTU",

mtu);

drivers/net/gve/base/gve_adminq.c

Quoted 6 lines
drivers/net/gve/base/gve_adminq.c new file +6 -0
+		err = -EINVAL;
+		goto free_device_descriptor;
+	}
+	priv->max_mtu = mtu;
+	priv->num_event_counters = be16_to_cpu(descriptor->counters);
+	rte_memcpy(priv->dev_addr.addr_bytes, descriptor->mac,

ETH_ALEN);

Review

Quoted 2 lines
(quoted diff) +2 -0
+	mac = descriptor->mac;
+	PMD_DRV_LOG(INFO, "MAC

addr: %02x:%02x:%02x:%02x:%02x:%02x",

drivers/net/gve/base/gve_adminq.c line 769

Quoted 1 line
drivers/net/gve/base/gve_adminq.c new file +1 -0
+		    mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);

There are 'RTE_ETHER_ADDR_PRT_FMT' & 'RTE_ETHER_ADDR_BYTES' macros for
this purpose, you can use it like:

Quoted 4 lines

PMD_DRV_LOG(INFO, "MAC addr" RTE_ETHER_ADDR_PRT_FMT, RTE_ETHER_ADDR_BYTES(priv->dev_addr));

So can get rid of 'mac' variable.

Thanks for the advice! Will try this in the coming version.

#83 lore raw

v8 cover introduce GVE PMD

In reply to Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.

Please refer to https://cloud.google.com/compute/docs/networking/using-gvnic
for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6:
improve the code.

v7:
- remove Intel copyright for the google base files.

v8:
- replace ETIME with ETIMEDOUT to pass the build check.
- use RTE_ETHER_ADDR_PRT_FMT/_ADDR_BYTES to get rid of 'mac' variable.
- add limitations in doc for current limited RSS and MTU.

Junfeng Guo (8):
net/gve/base: introduce base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  16 +
 doc/guides/nics/gve.rst                |  82 +++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve.h             |  56 ++
 drivers/net/gve/base/gve_adminq.c      | 921 +++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h      | 381 ++++++++++
 drivers/net/gve/base/gve_desc.h        | 138 ++++
 drivers/net/gve/base/gve_desc_dqo.h    | 255 +++++++
 drivers/net/gve/base/gve_osdep.h       | 159 +++++
 drivers/net/gve/base/gve_register.h    |  29 +
 drivers/net/gve/gve_ethdev.c           | 700 +++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 298 ++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/gve_rx.c               | 354 ++++++++++
 drivers/net/gve/gve_tx.c               | 669 ++++++++++++++++++
 drivers/net/gve/meson.build            |  16 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 20 files changed, 4104 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_osdep.h
 create mode 100644 drivers/net/gve/base/gve_register.h
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
Signature · 1 line
2.34.1
#84 lore raw

PATCH v8 1/8 net/gve/base: introduce base code

The following base code is based on Google Virtual Ethernet (gve) driver v1.3.0 under MIT license.
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h
- gve.h

The original code is in:
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/\
tree/v1.3.0/google/gve

Note that these code are not Intel files and they come from the kernel community. The base code there has the statement of
SPDX-License-Identifier: (GPL-2.0 OR MIT). Here we just follow the required MIT license as an exception to DPDK.

---
 drivers/net/gve/base/gve.h          |  56 ++
 drivers/net/gve/base/gve_adminq.c   | 920 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_adminq.h   | 379 ++++++++++++
 drivers/net/gve/base/gve_desc.h     | 136 ++++
 drivers/net/gve/base/gve_desc_dqo.h | 253 ++++++++
 drivers/net/gve/base/gve_register.h |  27 +
 6 files changed, 1771 insertions(+)
 create mode 100644 drivers/net/gve/base/gve.h
 create mode 100644 drivers/net/gve/base/gve_adminq.c
 create mode 100644 drivers/net/gve/base/gve_adminq.h
 create mode 100644 drivers/net/gve/base/gve_desc.h
 create mode 100644 drivers/net/gve/base/gve_desc_dqo.h
 create mode 100644 drivers/net/gve/base/gve_register.h
drivers/net/gve/base/gve.h new file +56 -0
new file mode 100644
index 0000000000..2dc4507acb
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_H_
+#define _GVE_H_
+
+#include "gve_desc.h"
+
+#define GVE_VERSION		"1.3.0"
+#define GVE_VERSION_PREFIX	"GVE-"
+
+#ifndef GOOGLE_VENDOR_ID
+#define GOOGLE_VENDOR_ID	0x1ae0
+#endif
+
+#define GVE_DEV_ID		0x0042
+
+#define GVE_REG_BAR		0
+#define GVE_DB_BAR		2
+
+/* 1 for management, 1 for rx, 1 for tx */
+#define GVE_MIN_MSIX		3
+
+/* PTYPEs are always 10 bits. */
+#define GVE_NUM_PTYPES		1024
+
+struct gve_irq_db {
+	rte_be32_t id;
+} ____cacheline_aligned;
+
+struct gve_ptype {
+	uint8_t l3_type;  /* `gve_l3_type` in gve_adminq.h */
+	uint8_t l4_type;  /* `gve_l4_type` in gve_adminq.h */
+};
+
+struct gve_ptype_lut {
+	struct gve_ptype ptypes[GVE_NUM_PTYPES];
+};
+
+enum gve_queue_format {
+	GVE_QUEUE_FORMAT_UNSPECIFIED = 0x0, /* default unspecified */
+	GVE_GQI_RDA_FORMAT	     = 0x1, /* GQI Raw Addressing */
+	GVE_GQI_QPL_FORMAT	     = 0x2, /* GQI Queue Page List */
+	GVE_DQO_RDA_FORMAT	     = 0x3, /* DQO Raw Addressing */
+};
+
+enum gve_state_flags_bit {
+	GVE_PRIV_FLAGS_ADMIN_QUEUE_OK		= 1,
+	GVE_PRIV_FLAGS_DEVICE_RESOURCES_OK	= 2,
+	GVE_PRIV_FLAGS_DEVICE_RINGS_OK		= 3,
+	GVE_PRIV_FLAGS_NAPI_ENABLED		= 4,
+};
+
+#endif /* _GVE_H_ */
drivers/net/gve/base/gve_adminq.c new file +920 -0

920 lines not on this page · raw message

drivers/net/gve/base/gve_adminq.h new file +379 -0

379 lines not on this page · raw message

drivers/net/gve/base/gve_desc.h new file +136 -0

136 lines not on this page · raw message

drivers/net/gve/base/gve_desc_dqo.h new file +253 -0

253 lines not on this page · raw message

drivers/net/gve/base/gve_register.h new file +27 -0
new file mode 100644
index 0000000000..bf7f102cde
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: MIT
+ * Google Virtual Ethernet (gve) driver
+ * Copyright (C) 2015-2022 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+	GVE_DEVICE_STATUS_REPORT_STATS_MASK	= BIT(3),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.34.1
#85 lore raw

PATCH v8 2/8 net/gve/base: add OS specific implementation

In reply to Junfeng Guo

Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/base/gve_adminq.h   |   2 +
 drivers/net/gve/base/gve_desc.h     |   2 +
 drivers/net/gve/base/gve_desc_dqo.h |   2 +
 drivers/net/gve/base/gve_osdep.h    | 159 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_register.h |   2 +
 5 files changed, 167 insertions(+)
 create mode 100644 drivers/net/gve/base/gve_osdep.h
drivers/net/gve/base/gve_adminq.h +2 -0
index b2422d7dc8..05550119de 100644
@@ -6,6 +6,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/base/gve_desc.h +2 -0
index e0bbadcfd4..006b36442f 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/base/gve_desc_dqo.h +2 -0
index 9965f190d1..ee1afdecb8 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/base/gve_osdep.h new file +159 -0

159 lines not on this page · raw message

drivers/net/gve/base/gve_register.h +2 -0
index bf7f102cde..c674167f31 100644
@@ -6,6 +6,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
Signature · 1 line
2.34.1
#86 lore raw

PATCH v8 3/8 net/gve: add support for device initialization

In reply to Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

---
 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  10 +
 doc/guides/nics/gve.rst                |  68 +++++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve_adminq.c      |   1 +
 drivers/net/gve/gve_ethdev.c           | 368 +++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 225 +++++++++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/meson.build            |  14 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 12 files changed, 716 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
index 92b381bc30..2d06a76efe 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
new file mode 100644
index 0000000000..44aec28009
@@ -0,0 +1,10 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +68 -0

68 lines not on this page · raw message

doc/guides/nics/index.rst +1 -0
index 32c7544968..4d40ea29a3 100644
@@ -29,6 +29,7 @@ Network Interface Controller Drivers
     enetfec
     enic
     fm10k
+    gve
     hinic
     hns3
     i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
index 1c3daf141d..21c366b0e2 100644
@@ -152,6 +152,11 @@ New Features
   * Added Q-in-CMB feature controlled by devarg ionic_cmb.
   * Added optimized handlers for non-scattered Rx and Tx.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 * **Updated Intel iavf driver.**
 
   * Added flow subscription support.
drivers/net/gve/base/gve_adminq.c +1 -0
index 045d47615d..e745b709b2 100644
@@ -3,6 +3,7 @@
  * Copyright (C) 2015-2022 Google, Inc.
  */
 
+#include "../gve_ethdev.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +368 -0

368 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h new file +225 -0

225 lines not on this page · raw message

drivers/net/gve/gve_logs.h new file +14 -0
new file mode 100644
index 0000000000..0d02da46e1
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+		__func__, ## args)
+
+#endif
drivers/net/gve/meson.build new file +14 -0
new file mode 100644
index 0000000000..d8ec64b3a3
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'base/gve_adminq.c',
+        'gve_ethdev.c',
+)
+includes += include_directories('base')
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..78c3585d7c
@@ -0,0 +1,3 @@
+DPDK_23 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index 35bfa78dee..355dbd07e9 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.34.1
#87 lore raw

PATCH v8 4/8 net/gve: add support for link update

In reply to Junfeng Guo

Support dev_ops link_update.

---
 doc/guides/nics/features/gve.ini |  1 +
 doc/guides/nics/gve.rst          |  3 +++
 drivers/net/gve/gve_ethdev.c     | 30 ++++++++++++++++++++++++++++++
 3 files changed, 34 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index 44aec28009..ae466ad677 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +3 -0
index 703fbcc5de..c42ff23841 100644
@@ -60,6 +60,9 @@ Features and Limitations
 
 In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
+Supported features of the GVE PMD are:
+
+- Link state information
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +30 -0
index acbb412509..34243c1672 100644
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_DRV_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -72,6 +101,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.34.1
#88 lore raw

PATCH v8 5/8 net/gve: add support for MTU setting

In reply to Junfeng Guo

Support dev_ops mtu_set.

---
 doc/guides/nics/features/gve.ini |  1 +
 doc/guides/nics/gve.rst          |  2 ++
 drivers/net/gve/gve_ethdev.c     | 28 ++++++++++++++++++++++++++++
 3 files changed, 31 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index ae466ad677..d1703d8dab 100644
@@ -5,6 +5,7 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +2 -0
index c42ff23841..36a65e4717 100644
@@ -69,3 +69,5 @@ Jumbo Frame is not supported in PMD for now. It'll be added in the future
 DPDK release.
 Also, only GQI_QPL queue format is in use on GCP since GQI_RDA hasn't been
 released in production.
+
+Currently, setting MTU with value larger than 1460 is not supported.
drivers/net/gve/gve_ethdev.c +28 -0
index 34243c1672..554f58640d 100644
@@ -96,12 +96,40 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+			    RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.34.1
#89 lore raw

Re: [PATCH v8 5/8] net/gve: add support for MTU setting

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 109

Quoted 11 lines · On Tue, 25 Oct 2022 17:07:26 +0800 Junfeng Guo <[email protected]> wrote:
drivers/net/gve/gve_ethdev.c +11 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+			    RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}

This check should not be necessary.
In rte_eth_dev_set_mtu it queries device for min/max mtu then calls eth_dev_validate_mtu() to check that the mtu is ok.

#90 lore raw

RE: [PATCH v8 5/8] net/gve: add support for MTU setting

Quoted 17 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Tuesday, October 25, 2022 23:55
To: Guo, Junfeng <[email protected]>
Cc: Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>; [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v8 5/8] net/gve: add support for MTU setting

On Tue, 25 Oct 2022 17:07:26 +0800
Junfeng Guo <[email protected]> wrote:

Quoted 16 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 109

Quoted 11 lines
drivers/net/gve/gve_ethdev.c +11 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+			    RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}

This check should not be necessary.
In rte_eth_dev_set_mtu it queries device for min/max mtu then calls eth_dev_validate_mtu() to check that the mtu is ok.

Thanks for the comment. Yes, this part seems redundant with _validate_mtu() for the same check.
Maybe better to update this as a bugfix later. Thanks!

#91 lore raw

PATCH v8 6/8 net/gve: add support for dev info get and dev configure

In reply to Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads force enabling.

---
 doc/guides/nics/features/gve.ini |  2 ++
 doc/guides/nics/gve.rst          |  5 +++
 drivers/net/gve/gve_ethdev.c     | 59 +++++++++++++++++++++++++++++++-
 drivers/net/gve/gve_ethdev.h     |  3 ++
 4 files changed, 68 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
index d1703d8dab..986df7f94a 100644
@@ -4,8 +4,10 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+RSS hash             = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +5 -0
index 36a65e4717..1051d964f1 100644
@@ -71,3 +71,8 @@ Also, only GQI_QPL queue format is in use on GCP since GQI_RDA hasn't been
 released in production.
 
 Currently, setting MTU with value larger than 1460 is not supported.
+
+Currently, only "RSS hash" is force enabled so that the backend hardware
+device calculated hash values could be shared with applications. But for
+RSS, there is no such API to config RSS hash function or RETA table. So,
+limited RSS is supported only with default config/setting.
drivers/net/gve/gve_ethdev.c +58 -1

78 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h +3 -0
index 2ac2a46ac1..57c29374b5 100644
@@ -18,6 +18,9 @@
 #define GVE_MIN_BUF_SIZE	    1024
 #define GVE_MAX_RX_PKTLEN	    65535
 
+#define GVE_MAX_MTU	RTE_ETHER_MTU
+#define GVE_MIN_MTU	RTE_ETHER_MIN_MTU
+
 /* A list of pages registered with the device during setup and used by a queue
  * as buffers
  */
Signature · 1 line
2.34.1
#92 lore raw

PATCH v8 7/8 net/gve: add support for queue operations

In reply to Junfeng Guo

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve_ethdev.c | 204 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h |  52 +++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 684 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +204 -0
index 7fbe0c78c9..892e7e2e1c 100644
@@ -28,6 +28,68 @@ gve_write_version(uint8_t *driver_version_register)
 	writeb('\n', driver_version_register);
 }
 
+static int
+gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pages)
+{
+	char z_name[RTE_MEMZONE_NAMESIZE];
+	struct gve_queue_page_list *qpl;
+	const struct rte_memzone *mz;
+	dma_addr_t page_bus;
+	uint32_t i;
+
+	if (priv->num_registered_pages + pages >
+	    priv->max_registered_pages) {
+		PMD_DRV_LOG(ERR, "Pages %" PRIu64 " > max registered pages %" PRIu64,
+			    priv->num_registered_pages + pages,
+			    priv->max_registered_pages);
+		return -EINVAL;
+	}
+	qpl = &priv->qpl[id];
+	snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_dev->device.name, id);
+	mz = rte_memzone_reserve_aligned(z_name, pages * PAGE_SIZE,
+					 rte_socket_id(),
+					 RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
+	if (mz == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name);
+		return -ENOMEM;
+	}
+	qpl->page_buses = rte_zmalloc("qpl page buses", pages * sizeof(dma_addr_t), 0);
+	if (qpl->page_buses == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page buses", id);
+		return -ENOMEM;
+	}
+	page_bus = mz->iova;
+	for (i = 0; i < pages; i++) {
+		qpl->page_buses[i] = page_bus;
+		page_bus += PAGE_SIZE;
+	}
+	qpl->id = id;
+	qpl->mz = mz;
+	qpl->num_entries = pages;
+
+	priv->num_registered_pages += pages;
+
+	return 0;
+}
+
+static void
+gve_free_qpls(struct gve_priv *priv)
+{
+	uint16_t nb_txqs = priv->max_nb_txq;
+	uint16_t nb_rxqs = priv->max_nb_rxq;
+	uint32_t i;
+
+	for (i = 0; i < nb_txqs + nb_rxqs; i++) {
+		if (priv->qpl[i].mz != NULL)
+			rte_memzone_free(priv->qpl[i].mz);
+		if (priv->qpl[i].page_buses != NULL)
+			rte_free(priv->qpl[i].page_buses);
+	}
+
+	if (priv->qpl != NULL)
+		rte_free(priv->qpl);
+}
+
 static int
 gve_dev_configure(struct rte_eth_dev *dev)
 {
@@ -42,6 +104,43 @@ gve_dev_configure(struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_refill_pages(struct gve_rx_queue *rxq)
+{
+	struct rte_mbuf *nmb;
+	uint16_t i;
+	int diag;
+
+	diag = rte_pktmbuf_alloc_bulk(rxq->mpool, &rxq->sw_ring[0], rxq->nb_rx_desc);
+	if (diag < 0) {
+		for (i = 0; i < rxq->nb_rx_desc - 1; i++) {
+			nmb = rte_pktmbuf_alloc(rxq->mpool);
+			if (!nmb)
+				break;
+			rxq->sw_ring[i] = nmb;
+		}
+		if (i < rxq->nb_rx_desc - 1)
+			return -ENOMEM;
+	}
+	rxq->nb_avail = 0;
+	rxq->next_avail = rxq->nb_rx_desc - 1;
+
+	for (i = 0; i < rxq->nb_rx_desc; i++) {
+		if (rxq->is_gqi_qpl) {
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(i * PAGE_SIZE);
+		} else {
+			if (i == rxq->nb_rx_desc - 1)
+				break;
+			nmb = rxq->sw_ring[i];
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(rte_mbuf_data_iova(nmb));
+		}
+	}
+
+	rte_write32(rte_cpu_to_be_32(rxq->next_avail), rxq->qrx_tail);
+
+	return 0;
+}
+
 static int
 gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 {
@@ -73,16 +172,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
+	uint16_t num_queues = dev->data->nb_tx_queues;
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
+	uint16_t i;
+	int err;
+
+	priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues;
+	err = gve_adminq_create_tx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u tx queues.", num_queues);
+		return err;
+	}
+	for (i = 0; i < num_queues; i++) {
+		txq = priv->txqs[i];
+		txq->qtx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(txq->qres->db_index)];
+		txq->qtx_head =
+		&priv->cnt_array[rte_be_to_cpu_32(txq->qres->counter_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), txq->ntfy_addr);
+	}
+
+	num_queues = dev->data->nb_rx_queues;
+	priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues;
+	err = gve_adminq_create_rx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u rx queues.", num_queues);
+		goto err_tx;
+	}
+	for (i = 0; i < num_queues; i++) {
+		rxq = priv->rxqs[i];
+		rxq->qrx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(rxq->qres->db_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), rxq->ntfy_addr);
+
+		err = gve_refill_pages(rxq);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to refill for RX");
+			goto err_rx;
+		}
+	}
+
 	dev->data->dev_started = 1;
 	gve_link_update(dev, 0);
 
 	return 0;
+
+err_rx:
+	gve_stop_rx_queues(dev);
+err_tx:
+	gve_stop_tx_queues(dev);
+	return err;
 }
 
 static int
 gve_dev_stop(struct rte_eth_dev *dev)
 {
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+
+	gve_stop_tx_queues(dev);
+	gve_stop_rx_queues(dev);
+
 	dev->data->dev_started = 0;
 
 	return 0;
@@ -91,7 +244,11 @@ gve_dev_stop(struct rte_eth_dev *dev)
 static int
 gve_dev_close(struct rte_eth_dev *dev)
 {
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
 	int err = 0;
+	uint16_t i;
 
 	if (dev->data->dev_started) {
 		err = gve_dev_stop(dev);
@@ -99,6 +256,19 @@ gve_dev_close(struct rte_eth_dev *dev)
 			PMD_DRV_LOG(ERR, "Failed to stop dev.");
 	}
 
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		txq = dev->data->tx_queues[i];
+		gve_tx_queue_release(txq);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		rxq = dev->data->rx_queues[i];
+		gve_rx_queue_release(rxq);
+	}
+
+	gve_free_qpls(priv);
+	rte_free(priv->adminq);
+
 	dev->data->mac_addrs = NULL;
 
 	return err;
@@ -185,6 +355,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.dev_infos_get        = gve_dev_info_get,
+	.rx_queue_setup       = gve_rx_queue_setup,
+	.tx_queue_setup       = gve_tx_queue_setup,
 	.link_update          = gve_link_update,
 	.mtu_set              = gve_dev_mtu_set,
 };
@@ -322,7 +494,9 @@ gve_setup_device_resources(struct gve_priv *priv)
 static int
 gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 {
+	uint16_t pages;
 	int num_ntfy;
+	uint32_t i;
 	int err;
 
 	/* Set up the adminq */
@@ -373,10 +547,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 	PMD_DRV_LOG(INFO, "Max TX queues %d, Max RX queues %d",
 		    priv->max_nb_txq, priv->max_nb_rxq);
 
+	/* In GQI_QPL queue format:
+	 * Allocate queue page lists according to max queue number
+	 * tx qpl id should start from 0 while rx qpl id should start
+	 * from priv->max_nb_txq
+	 */
+	if (priv->queue_format == GVE_GQI_QPL_FORMAT) {
+		priv->qpl = rte_zmalloc("gve_qpl",
+					(priv->max_nb_txq + priv->max_nb_rxq) *
+					sizeof(struct gve_queue_page_list), 0);
+		if (priv->qpl == NULL) {
+			PMD_DRV_LOG(ERR, "Failed to alloc qpl.");
+			err = -ENOMEM;
+			goto free_adminq;
+		}
+
+		for (i = 0; i < priv->max_nb_txq + priv->max_nb_rxq; i++) {
+			if (i < priv->max_nb_txq)
+				pages = priv->tx_pages_per_qpl;
+			else
+				pages = priv->rx_data_slot_cnt;
+			err = gve_alloc_queue_page_list(priv, i, pages);
+			if (err != 0) {
+				PMD_DRV_LOG(ERR, "Failed to alloc qpl %u.", i);
+				goto err_qpl;
+			}
+		}
+	}
+
 setup_device:
 	err = gve_setup_device_resources(priv);
 	if (!err)
 		return 0;
+err_qpl:
+	gve_free_qpls(priv);
 free_adminq:
 	gve_adminq_free(priv);
 	return err;
drivers/net/gve/gve_ethdev.h +52 -0

92 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index d8ec64b3a3..af0010c01c 100644
@@ -9,6 +9,8 @@ endif
 
 sources = files(
         'base/gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
 includes += include_directories('base')
Signature · 1 line
2.34.1
#93 lore raw

PATCH v8 8/8 net/gve: add support for Rx/Tx

In reply to Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 doc/guides/nics/features/gve.ini |   2 +
 doc/guides/nics/gve.rst          |   4 +
 drivers/net/gve/gve_ethdev.c     |  15 +-
 drivers/net/gve/gve_ethdev.h     |  18 ++
 drivers/net/gve/gve_rx.c         | 142 ++++++++++
 drivers/net/gve/gve_tx.c         | 455 +++++++++++++++++++++++++++++++
 6 files changed, 635 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
index 986df7f94a..cdc46b08a3 100644
@@ -7,7 +7,9 @@
 Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+TSO                  = Y
 RSS hash             = Y
+L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +4 -0
index 1051d964f1..92b85e0dbe 100644
@@ -62,7 +62,11 @@ In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
 Supported features of the GVE PMD are:
 
+- Multiple queues for TX and RX
+- TSO offload
 - Link state information
+- TX multi-segments (Scatter TX)
+- Tx UDP/TCP/SCTP Checksum
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +14 -1
index 892e7e2e1c..5c0cd2f2c4 100644
@@ -289,7 +289,13 @@ gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->min_mtu = GVE_MIN_MTU;
 
 	dev_info->rx_offload_capa = 0;
-	dev_info->tx_offload_capa = 0;
+	dev_info->tx_offload_capa =
+		RTE_ETH_TX_OFFLOAD_MULTI_SEGS	|
+		RTE_ETH_TX_OFFLOAD_IPV4_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_UDP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_TCP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_SCTP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_TCP_TSO;
 
 	if (priv->queue_format == GVE_DQO_RDA_FORMAT)
 		dev_info->rx_offload_capa |= RTE_ETH_RX_OFFLOAD_TCP_LRO;
@@ -639,6 +645,13 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	} else {
+		PMD_DRV_LOG(ERR, "DQO_RDA is not implemented and will be added in the future");
+	}
+
 	eth_dev->data->mac_addrs = &priv->dev_addr;
 
 	return 0;
drivers/net/gve/gve_ethdev.h +18 -0
index 00c69d1b88..36b334c36b 100644
@@ -37,6 +37,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -277,4 +289,10 @@ gve_stop_tx_queues(struct rte_eth_dev *dev);
 void
 gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t
+gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t
+gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +142 -0

148 lines not on this page · raw message

drivers/net/gve/gve_tx.c +455 -0

461 lines not on this page · raw message

Signature · 1 line
2.34.1
#94 lore raw
In reply to Junfeng Guo
Quoted 49 lines · Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google Cloud Platform (GCP), which is one of the multiple virtual interfaces from those leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can help those cloud instance consumers with better experience of performance, maintenance who wants to run their own VNFs on GCP.

Please refer tohttps://cloud.google.com/compute/docs/networking/using-gvnic for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-linux/tree/v1.3.0

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6:
improve the code.

v7:
- remove Intel copyright for the google base files.

v8:
- replace ETIME with ETIMEDOUT to pass the build check.
- use RTE_ETHER_ADDR_PRT_FMT/_ADDR_BYTES to get rid of 'mac' variable.
- add limitations in doc for current limited RSS and MTU.

Quoted 9 lines

Junfeng Guo (8):
net/gve/base: introduce base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

Series applied to dpdk-next-net/main, thanks.

Please send a web page to document the new PMD:
https://core.dpdk.org/supported/#nics

#95 lore raw
Quoted 82 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Tuesday, October 25, 2022 20:33
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v8 0/8] introduce GVE PMD

Quoted 3 lines · Junfeng Guo

Introduce a new PMD for Google Virtual Ethernet (GVE).

gve (or gVNIC) is the standard virtual ethernet interface on Google

Cloud

Quoted 1 line

Platform (GCP), which is one of the multiple virtual interfaces from

those

Quoted 3 lines

leading CSP customers in the world.

Having a well maintained/optimized gve PMD on DPDK community can

help those

Quoted 1 line

cloud instance consumers with better experience of performance,

maintenance

Quoted 3 lines

who wants to run their own VNFs on GCP.

Please refer

tohttps://cloud.google.com/compute/docs/networking/using-gvnic

Quoted 10 lines

for the device description.

This patch set requires an exception for MIT license for GVE base code. And the base code includes the following files:
- gve_adminq.c
- gve_adminq.h
- gve_desc.h
- gve_desc_dqo.h
- gve_register.h

Quoted 2 lines

It's based on GVE kernel driver v1.3.0 and the original code is in
https://github.com/GoogleCloudPlatform/compute-virtual-ethernet-

linux/tree/v1.3.0

Quoted 17 lines

v2:
fix some CI check error.

v3:
refactor some code and fix some build error.

v4:
move the Google base code files into DPDK base folder.

v5:
reorder commit sequence and drop the stats feature.

v6:
improve the code.

Quoted 6 lines

v7:
- remove Intel copyright for the google base files.

v8:
- replace ETIME with ETIMEDOUT to pass the build check.
- use RTE_ETHER_ADDR_PRT_FMT/_ADDR_BYTES to get rid of 'mac'

variable.

Quoted 12 lines

- add limitations in doc for current limited RSS and MTU.

Junfeng Guo (8):
net/gve/base: introduce base code net/gve/base: add OS specific implementation net/gve: add support for device initialization net/gve: add support for link update
net/gve: add support for MTU setting
net/gve: add support for dev info get and dev configure net/gve: add support for queue operations net/gve: add support for Rx/Tx

Quoted 4 lines

Series applied to dpdk-next-net/main, thanks.

Please send a web page to document the new PMD:
https://core.dpdk.org/supported/#nics

Sure, thanks for reminding!

#96 lore raw

PATCH v7 2/8 net/gve/base: add OS specific implementation

In reply to Junfeng Guo

Add some MACRO definitions and memory operations which are specific for DPDK.

---
 drivers/net/gve/base/gve_adminq.h   |   2 +
 drivers/net/gve/base/gve_desc.h     |   2 +
 drivers/net/gve/base/gve_desc_dqo.h |   2 +
 drivers/net/gve/base/gve_osdep.h    | 159 ++++++++++++++++++++++++++++
 drivers/net/gve/base/gve_register.h |   2 +
 5 files changed, 167 insertions(+)
 create mode 100644 drivers/net/gve/base/gve_osdep.h
drivers/net/gve/base/gve_adminq.h +2 -0
index b2422d7dc8..05550119de 100644
@@ -6,6 +6,8 @@
 #ifndef _GVE_ADMINQ_H
 #define _GVE_ADMINQ_H
 
+#include "gve_osdep.h"
+
 /* Admin queue opcodes */
 enum gve_adminq_opcodes {
 	GVE_ADMINQ_DESCRIBE_DEVICE		= 0x1,
drivers/net/gve/base/gve_desc.h +2 -0
index e0bbadcfd4..006b36442f 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_DESC_H_
 #define _GVE_DESC_H_
 
+#include "gve_osdep.h"
+
 /* A note on seg_addrs
  *
  * Base addresses encoded in seg_addr are not assumed to be physical
drivers/net/gve/base/gve_desc_dqo.h +2 -0
index 9965f190d1..ee1afdecb8 100644
@@ -8,6 +8,8 @@
 #ifndef _GVE_DESC_DQO_H_
 #define _GVE_DESC_DQO_H_
 
+#include "gve_osdep.h"
+
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
 
drivers/net/gve/base/gve_osdep.h new file +159 -0

159 lines not on this page · raw message

drivers/net/gve/base/gve_register.h +2 -0
index bf7f102cde..c674167f31 100644
@@ -6,6 +6,8 @@
 #ifndef _GVE_REGISTER_H_
 #define _GVE_REGISTER_H_
 
+#include "gve_osdep.h"
+
 /* Fixed Configuration Registers */
 struct gve_registers {
 	__be32	device_status;
Signature · 1 line
2.34.1
#97 lore raw

PATCH v7 3/8 net/gve: add support for device initialization

In reply to Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

---
 MAINTAINERS                            |   6 +
 doc/guides/nics/features/gve.ini       |  10 +
 doc/guides/nics/gve.rst                |  68 +++++
 doc/guides/nics/index.rst              |   1 +
 doc/guides/rel_notes/release_22_11.rst |   5 +
 drivers/net/gve/base/gve_adminq.c      |   1 +
 drivers/net/gve/gve_ethdev.c           | 368 +++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h           | 225 +++++++++++++++
 drivers/net/gve/gve_logs.h             |  14 +
 drivers/net/gve/meson.build            |  14 +
 drivers/net/gve/version.map            |   3 +
 drivers/net/meson.build                |   1 +
 12 files changed, 716 insertions(+)
 create mode 100644 doc/guides/nics/features/gve.ini
 create mode 100644 doc/guides/nics/gve.rst
 create mode 100644 drivers/net/gve/gve_ethdev.c
 create mode 100644 drivers/net/gve/gve_ethdev.h
 create mode 100644 drivers/net/gve/gve_logs.h
 create mode 100644 drivers/net/gve/meson.build
 create mode 100644 drivers/net/gve/version.map
MAINTAINERS +6 -0
index 92b381bc30..2d06a76efe 100644
@@ -697,6 +697,12 @@ F: drivers/net/enic/
 F: doc/guides/nics/enic.rst
 F: doc/guides/nics/features/enic.ini
 
+Google Virtual Ethernet
+M: Junfeng Guo <[email protected]>
+F: drivers/net/gve/
+F: doc/guides/nics/gve.rst
+F: doc/guides/nics/features/gve.ini
+
 Hisilicon hns3
 M: Dongdong Liu <[email protected]>
 M: Yisen Zhuang <[email protected]>
doc/guides/nics/features/gve.ini new file +10 -0
new file mode 100644
index 0000000000..44aec28009
@@ -0,0 +1,10 @@
+;
+; Supported features of the Google Virtual Ethernet 'gve' poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Linux                = Y
+x86-32               = Y
+x86-64               = Y
+Usage doc            = Y
doc/guides/nics/gve.rst new file +68 -0

68 lines not on this page · raw message

doc/guides/nics/index.rst +1 -0
index 32c7544968..4d40ea29a3 100644
@@ -29,6 +29,7 @@ Network Interface Controller Drivers
     enetfec
     enic
     fm10k
+    gve
     hinic
     hns3
     i40e
doc/guides/rel_notes/release_22_11.rst +5 -0
index 1c3daf141d..21c366b0e2 100644
@@ -152,6 +152,11 @@ New Features
   * Added Q-in-CMB feature controlled by devarg ionic_cmb.
   * Added optimized handlers for non-scattered Rx and Tx.
 
+* **Added GVE net PMD**
+
+  * Added the new ``gve`` net driver for Google Virtual Ethernet devices.
+  * See the :doc:`../nics/gve` NIC guide for more details on this new driver.
+
 * **Updated Intel iavf driver.**
 
   * Added flow subscription support.
drivers/net/gve/base/gve_adminq.c +1 -0
index 2eb06a7b68..a550b27e61 100644
@@ -3,6 +3,7 @@
  * Copyright (C) 2015-2022 Google, Inc.
  */
 
+#include "../gve_ethdev.h"
 #include "gve_adminq.h"
 #include "gve_register.h"
 
drivers/net/gve/gve_ethdev.c new file +368 -0

368 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h new file +225 -0

225 lines not on this page · raw message

drivers/net/gve/gve_logs.h new file +14 -0
new file mode 100644
index 0000000000..0d02da46e1
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2022 Intel Corporation
+ */
+
+#ifndef _GVE_LOGS_H_
+#define _GVE_LOGS_H_
+
+extern int gve_logtype_driver;
+
+#define PMD_DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, gve_logtype_driver, "%s(): " fmt "\n", \
+		__func__, ## args)
+
+#endif
drivers/net/gve/meson.build new file +14 -0
new file mode 100644
index 0000000000..d8ec64b3a3
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2022 Intel Corporation
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'base/gve_adminq.c',
+        'gve_ethdev.c',
+)
+includes += include_directories('base')
drivers/net/gve/version.map new file +3 -0
new file mode 100644
index 0000000000..78c3585d7c
@@ -0,0 +1,3 @@
+DPDK_23 {
+	local: *;
+};
drivers/net/meson.build +1 -0
index 35bfa78dee..355dbd07e9 100644
@@ -23,6 +23,7 @@ drivers = [
         'enic',
         'failsafe',
         'fm10k',
+        'gve',
         'hinic',
         'hns3',
         'i40e',
Signature · 1 line
2.34.1
#98 lore raw

Re: [PATCH v7 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 38 lines
drivers/net/gve/gve_ethdev.c new file +38 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;

drivers/net/gve/gve_ethdev.c line 324

Quoted 10 lines
drivers/net/gve/gve_ethdev.c new file +10 -0
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = &priv->dev_addr;
+

[copy/paste from previous version]

What is the value in 'priv->dev_addr'?
Even allocating memory for 'eth_dev->data->mac_addrs' removed or not, as we discussed, independent from it, need to set a valid value to 'priv->dev_addr'.

#99 lore raw

RE: [PATCH v7 3/8] net/gve: add support for device initialization

Quoted 84 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 38 lines
drivers/net/gve/gve_ethdev.c new file +38 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;

drivers/net/gve/gve_ethdev.c line 324

Quoted 10 lines
drivers/net/gve/gve_ethdev.c new file +10 -0
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = &priv->dev_addr;
+

[copy/paste from previous version]

Quoted 5 lines

What is the value in 'priv->dev_addr'?
Even allocating memory for 'eth_dev->data->mac_addrs' removed or not, as
we discussed, independent from it, need to set a valid value to 'priv->dev_addr'.

The value in 'priv->dev_addr' is the 'real' mac address of the gvnic port. So I suppose that there is no need to set a default valid one, since we can get them/it from the backend in gve_adminq_describe_device(priv). Thanks!

Quoted 2 lines
#100 lore raw

Re: [PATCH v7 3/8] net/gve: add support for device initialization

Quoted 90 lines · Junfeng
Quoted 84 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 3/8] net/gve: add support for device initialization

Quoted 12 lines · Junfeng Guo

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 38 lines
drivers/net/gve/gve_ethdev.c new file +38 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;

drivers/net/gve/gve_ethdev.c line 324

Quoted 10 lines
drivers/net/gve/gve_ethdev.c new file +10 -0
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = &priv->dev_addr;
+

[copy/paste from previous version]

Quoted 5 lines

What is the value in 'priv->dev_addr'?
Even allocating memory for 'eth_dev->data->mac_addrs' removed or not, as
we discussed, independent from it, need to set a valid value to 'priv->dev_addr'.

Quoted 3 lines

The value in 'priv->dev_addr' is the 'real' mac address of the gvnic port. So I suppose that there is no need to set a default valid one, since we can get them/it from the backend in gve_adminq_describe_device(priv).

Ack, thanks for clarification.

In 'gve_adminq_describe_device()', RTE_ETHER_ADDR_PRT_FMT & RTE_ETHER_ADDR_BYTES can be used for log, I will comment on patch.

#101 lore raw

RE: [PATCH v7 3/8] net/gve: add support for device initialization

Quoted 111 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:47
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 3/8] net/gve: add support for device initialization

Quoted 15 lines · Junfeng
Quoted 15 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>; Wang, Haiyue <[email protected]>
Subject: Re: [PATCH v7 3/8] net/gve: add support for device

initialization

Quoted 69 lines
Quoted 69 lines · Junfeng Guo
Quoted 13 lines

Support device init and add following devops skeleton:
- dev_configure
- dev_start
- dev_stop
- dev_close

Note that build system (including doc) is also added in this patch.

<...>

Quoted 38 lines
drivers/net/gve/gve_ethdev.c new file +38 -0
+static int
+gve_dev_init(struct rte_eth_dev *eth_dev)
+{
+       struct gve_priv *priv = eth_dev->data->dev_private;
+       int max_tx_queues, max_rx_queues;
+       struct rte_pci_device *pci_dev;
+       struct gve_registers *reg_bar;
+       rte_be32_t *db_bar;
+       int err;
+
+       eth_dev->dev_ops = &gve_eth_dev_ops;
+
+       if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+               return 0;
+
+       pci_dev = RTE_DEV_TO_PCI(eth_dev->device);
+
+       reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
+       if (!reg_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map pci bar!");
+               return -ENOMEM;
+       }
+
+       db_bar = pci_dev->mem_resource[GVE_DB_BAR].addr;
+       if (!db_bar) {
+               PMD_DRV_LOG(ERR, "Failed to map doorbell bar!");
+               return -ENOMEM;
+       }
+
+       gve_write_version(&reg_bar->driver_version);
+       /* Get max queues to alloc etherdev */
+       max_tx_queues = ioread32be(&reg_bar->max_tx_queues);
+       max_rx_queues = ioread32be(&reg_bar->max_rx_queues);
+
+       priv->reg_bar0 = reg_bar;
+       priv->db_bar2 = db_bar;
+       priv->pci_dev = pci_dev;
+       priv->state_flags = 0x0;

drivers/net/gve/gve_ethdev.c line 324

Quoted 10 lines
drivers/net/gve/gve_ethdev.c new file +10 -0
+
+       priv->max_nb_txq = max_tx_queues;
+       priv->max_nb_rxq = max_rx_queues;
+
+       err = gve_init_priv(priv, false);
+       if (err)
+               return err;
+
+       eth_dev->data->mac_addrs = &priv->dev_addr;
+

[copy/paste from previous version]

Quoted 2 lines
Quoted 2 lines

What is the value in 'priv->dev_addr'?
Even allocating memory for 'eth_dev->data->mac_addrs' removed or

not,

Quoted 6 lines
Quoted 3 lines

as
we discussed, independent from it, need to set a valid value to 'priv->dev_addr'.

The value in 'priv->dev_addr' is the 'real' mac address of the gvnic port. So I suppose that there is no need to set a default valid one, since we

Quoted 7 lines

can

Quoted 1 line

get them/it from the backend in gve_adminq_describe_device(priv).

Ack, thanks for clarification.

In 'gve_adminq_describe_device()', RTE_ETHER_ADDR_PRT_FMT & RTE_ETHER_ADDR_BYTES can be used for log, I will comment on patch.

Thanks for the advice! Will try this in the coming version.

#102 lore raw

PATCH v7 4/8 net/gve: add support for link update

In reply to Junfeng Guo

Support dev_ops link_update.

---
 doc/guides/nics/features/gve.ini |  1 +
 doc/guides/nics/gve.rst          |  3 +++
 drivers/net/gve/gve_ethdev.c     | 30 ++++++++++++++++++++++++++++++
 3 files changed, 34 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index 44aec28009..ae466ad677 100644
@@ -4,6 +4,7 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Link status          = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +3 -0
index 703fbcc5de..c42ff23841 100644
@@ -60,6 +60,9 @@ Features and Limitations
 
 In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
+Supported features of the GVE PMD are:
+
+- Link state information
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +30 -0
index acbb412509..34243c1672 100644
@@ -34,10 +34,39 @@ gve_dev_configure(__rte_unused struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	struct rte_eth_link link;
+	int err;
+
+	memset(&link, 0, sizeof(link));
+	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
+	link.link_autoneg = RTE_ETH_LINK_AUTONEG;
+
+	if (!dev->data->dev_started) {
+		link.link_status = RTE_ETH_LINK_DOWN;
+		link.link_speed = RTE_ETH_SPEED_NUM_NONE;
+	} else {
+		link.link_status = RTE_ETH_LINK_UP;
+		PMD_DRV_LOG(DEBUG, "Get link status from hw");
+		err = gve_adminq_report_link_speed(priv);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to get link speed.");
+			priv->link_speed = RTE_ETH_SPEED_NUM_UNKNOWN;
+		}
+		link.link_speed = priv->link_speed;
+	}
+
+	return rte_eth_linkstatus_set(dev, &link);
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	dev->data->dev_started = 1;
+	gve_link_update(dev, 0);
 
 	return 0;
 }
@@ -72,6 +101,7 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
+	.link_update          = gve_link_update,
 };
 
 static void
Signature · 1 line
2.34.1
#103 lore raw

PATCH v7 5/8 net/gve: add support for MTU setting

In reply to Junfeng Guo

Support dev_ops mtu_set.

---
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 28 ++++++++++++++++++++++++++++
 2 files changed, 29 insertions(+)
doc/guides/nics/features/gve.ini +1 -0
index ae466ad677..d1703d8dab 100644
@@ -5,6 +5,7 @@
 ;
 [Features]
 Link status          = Y
+MTU update           = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
drivers/net/gve/gve_ethdev.c +28 -0
index 34243c1672..554f58640d 100644
@@ -96,12 +96,40 @@ gve_dev_close(struct rte_eth_dev *dev)
 	return err;
 }
 
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	int err;
+
+	if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+		PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+			    RTE_ETHER_MIN_MTU, priv->max_mtu);
+		return -EINVAL;
+	}
+
+	/* mtu setting is forbidden if port is start */
+	if (dev->data->dev_started) {
+		PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+		return -EBUSY;
+	}
+
+	err = gve_adminq_set_mtu(priv, mtu);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+		return err;
+	}
+
+	return 0;
+}
+
 static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_configure        = gve_dev_configure,
 	.dev_start            = gve_dev_start,
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.link_update          = gve_link_update,
+	.mtu_set              = gve_dev_mtu_set,
 };
 
 static void
Signature · 1 line
2.34.1
#104 lore raw

Re: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 5 lines · Junfeng Guo

Support dev_ops mtu_set.

<...>

Quoted 17 lines
drivers/net/gve/gve_ethdev.c +17 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+                           RTE_ETHER_MIN_MTU, priv->max_mtu);
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before configuration");
+               return -EBUSY;
+       }

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 125

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+
+       err = gve_adminq_set_mtu(priv, mtu);
+       if (err) {
+               PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);
+               return err;
+       }
+
+       return 0;
+}
+

[copy/paste from previous version]

configure() (gve_dev_configure()) also get 'mtu' as user config ('eth_conf->rxmode.mtu') which is ignored right now,

since there is 'gve_adminq_set_mtu()' command already what do you think to use it within 'gve_dev_configure()'?

#105 lore raw

RE: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 57 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 5 lines · Junfeng Guo

Support dev_ops mtu_set.

<...>

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+                           RTE_ETHER_MIN_MTU, priv->max_mtu);
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before

configuration");

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return -EBUSY;
+       }
+
+       err = gve_adminq_set_mtu(priv, mtu);
+       if (err) {
+               PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu,

err);

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 125

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return err;
+       }
+
+       return 0;
+}
+

[copy/paste from previous version]

Quoted 6 lines

configure() (gve_dev_configure()) also get 'mtu' as user config ('eth_conf->rxmode.mtu') which is ignored right now,

since there is 'gve_adminq_set_mtu()' command already what do you think
to use it within 'gve_dev_configure()'?

There may be issues to set mtu with ('eth_conf->rxmode.mtu'). So better to keep this ignored at this stage.

#106 lore raw

Re: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 60 lines · Junfeng
Quoted 60 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 5 lines · Junfeng Guo

Support dev_ops mtu_set.

<...>

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+                           RTE_ETHER_MIN_MTU, priv->max_mtu);
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before

configuration");

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return -EBUSY;
+       }
+
+       err = gve_adminq_set_mtu(priv, mtu);
+       if (err) {
+               PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu,

err);

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 125

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return err;
+       }
+
+       return 0;
+}
+

[copy/paste from previous version]

configure() (gve_dev_configure()) also get 'mtu' as user config ('eth_conf->rxmode.mtu') which is ignored right now,

Quoted 7 lines
Quoted 3 lines

since there is 'gve_adminq_set_mtu()' command already what do you think
to use it within 'gve_dev_configure()'?

There may be issues to set mtu with ('eth_conf->rxmode.mtu'). So better to keep this ignored at this stage.

What do you mean by issues?

'eth_conf->rxmode.mtu' is user provided config parameter, so user may prefer to provide this value and not call 'rte_eth_dev_set_mtu()' at all and still can expect correct MTU value.

#107 lore raw

RE: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 85 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:48
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 41 lines · Junfeng
Quoted 41 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 5/8] net/gve: add support for MTU setting

Quoted 5 lines · Junfeng Guo

Support dev_ops mtu_set.

<...>

Quoted 5 lines
drivers/net/gve/gve_ethdev.c +5 -0
+static int
+gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
+{
+       struct gve_priv *priv = dev->data->dev_private;
+       int err;

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 10 lines
drivers/net/gve/gve_ethdev.c +10 -0
+
+       if (mtu < RTE_ETHER_MIN_MTU || mtu > priv->max_mtu) {
+               PMD_DRV_LOG(ERR, "MIN MTU is %u, MAX MTU is %u",
+                           RTE_ETHER_MIN_MTU, priv->max_mtu);
+               return -EINVAL;
+       }
+
+       /* mtu setting is forbidden if port is start */
+       if (dev->data->dev_started) {
+               PMD_DRV_LOG(ERR, "Port must be stopped before

configuration");

Quoted 6 lines
Quoted 6 lines

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev)

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return -EBUSY;
+       }
+
+       err = gve_adminq_set_mtu(priv, mtu);
+       if (err) {
+               PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d",

mtu,

Quoted 13 lines
Quoted 13 lines

err);

drivers/net/gve/gve_ethdev.c gve_dev_close(struct rte_eth_dev *dev) · line 125

Quoted 6 lines
drivers/net/gve/gve_ethdev.c +6 -0
+               return err;
+       }
+
+       return 0;
+}
+

[copy/paste from previous version]

configure() (gve_dev_configure()) also get 'mtu' as user config ('eth_conf->rxmode.mtu') which is ignored right now,

Quoted 7 lines
Quoted 3 lines

since there is 'gve_adminq_set_mtu()' command already what do you think
to use it within 'gve_dev_configure()'?

There may be issues to set mtu with ('eth_conf->rxmode.mtu'). So better to keep this ignored at this stage.

Quoted 5 lines

What do you mean by issues?

'eth_conf->rxmode.mtu' is user provided config parameter, so user may prefer to provide this value and not call 'rte_eth_dev_set_mtu()' at all and still can expect correct MTU value.

Yes, it should be like this. But on current GCP, the fetched MTU value is 1460, which is smaller than 1500. And if we set MTU with the value 1500, the backend will return a failed message.
I think it would be better to comment this as an limitation or known issue.

#108 lore raw

PATCH v7 6/8 net/gve: add support for dev info get and dev configure

In reply to Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads force enabling.

---
 doc/guides/nics/features/gve.ini |  2 ++
 doc/guides/nics/gve.rst          |  1 +
 drivers/net/gve/gve_ethdev.c     | 59 +++++++++++++++++++++++++++++++-
 drivers/net/gve/gve_ethdev.h     |  3 ++
 4 files changed, 64 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
index d1703d8dab..986df7f94a 100644
@@ -4,8 +4,10 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+RSS hash             = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +1 -0
index c42ff23841..8c09a5a7fa 100644
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
 Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)
 - Link state information
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
drivers/net/gve/gve_ethdev.c +58 -1

78 lines not on this page · raw message

drivers/net/gve/gve_ethdev.h +3 -0
index 2ac2a46ac1..57c29374b5 100644
@@ -18,6 +18,9 @@
 #define GVE_MIN_BUF_SIZE	    1024
 #define GVE_MAX_RX_PKTLEN	    65535
 
+#define GVE_MAX_MTU	RTE_ETHER_MTU
+#define GVE_MIN_MTU	RTE_ETHER_MIN_MTU
+
 /* A list of pages registered with the device during setup and used by a queue
  * as buffers
  */
Signature · 1 line
2.34.1
#109 lore raw

Re: [PATCH v7 6/8] net/gve: add support for dev info get and dev configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads force enabling.

<...>

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic functionality of packet · line 65

Quoted 7 lines
doc/guides/nics/gve.rst +1 -0
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic functionality of packet
  reception and transmission.
  Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)

[copy/paste from previous version]

I am not sure if driver can claim this, I can see a RSS hash is provided but is it possible to update which hash function to use or update key or RETA table to configure which queue packets goes?

Right now what is RSS calculated on?

Perpaps RSS support can be documented as limited?

And not sure if this update belongs this patch, it should be to the one that has the datapath.

#110 lore raw

RE: [PATCH v7 6/8] net/gve: add support for dev info get and dev configure

Quoted 42 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:51
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 6/8] net/gve: add support for dev info get and dev configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads force enabling.

<...>

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic

Quoted 3 lines
doc/guides/nics/gve.rst +0 -0
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic

functionality of packet

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic functionality of packet · line 65

Quoted 4 lines
doc/guides/nics/gve.rst +1 -0
  reception and transmission.
  Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)

[copy/paste from previous version]

I am not sure if driver can claim this, I can see a RSS hash is provided but is it possible to update which hash function to use or update key or RETA table to configure which queue packets goes?

Quoted 6 lines

Right now what is RSS calculated on?

Perpaps RSS support can be documented as limited?

And not sure if this update belongs this patch, it should be to the one that has the datapath.

Looks that the RSS is enabled by default, and there is no RSS init API. So I just added back the force-enabled RSS offloading code with the corresponding commit message. So the feature list remains unchaged. Thanks!

#111 lore raw

Re: [PATCH v7 6/8] net/gve: add support for dev info get and dev configure

Quoted 44 lines · Junfeng
Quoted 44 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:51
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 6/8] net/gve: add support for dev info get and dev configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads force enabling.

<...>

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic

Quoted 3 lines
doc/guides/nics/gve.rst +0 -0
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic

functionality of packet

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic functionality of packet · line 65

Quoted 4 lines
doc/guides/nics/gve.rst +1 -0
   reception and transmission.
   Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)

[copy/paste from previous version]

I am not sure if driver can claim this, I can see a RSS hash is provided but is it possible to update which hash function to use or update key or RETA table to configure which queue packets goes?

Right now what is RSS calculated on?

Quoted 8 lines
Quoted 4 lines

Perpaps RSS support can be documented as limited?

And not sure if this update belongs this patch, it should be to the one that has the datapath.

Looks that the RSS is enabled by default, and there is no RSS init API. So I just added back the force-enabled RSS offloading code with the corresponding commit message. So the feature list remains unchaged.

There is difference between RSS and RSS hash, what force enabled is "RSS hash" where device calculated hash value is shared with application in case application wants to reuse this value for some reasons.

But for RSS support, there is a set of configuration required by DPDK seems missing, as mentioned above, config RSS hash function (based on which part of the packet is the hash calculated), or RETA table update so app can select which packets goes to which queue, etc...

Is it at least possible to document what existing configuration is?

Because of missing configuration support, I don't think it is correct to document RSS as supported, can you please update it to say something like limited support exist with default config.

#112 lore raw

RE: [PATCH v7 6/8] net/gve: add support for dev info get and dev configure

Quoted 80 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:49
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 6/8] net/gve: add support for dev info get and dev configure

Quoted 14 lines · Junfeng
Quoted 14 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:51
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 6/8] net/gve: add support for dev info get and

dev

Quoted 30 lines
Quoted 30 lines

configure

Quoted 6 lines · Junfeng Guo

Add dev_ops dev_infos_get.
Complete dev_configure with RX offloads force enabling.

<...>

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic

Quoted 3 lines
doc/guides/nics/gve.rst +0 -0
@@ -62,6 +62,7 @@ In this release, the GVE PMD provides the basic

functionality of packet

doc/guides/nics/gve.rst In this release, the GVE PMD provides the basic functionality of packet · line 65

Quoted 4 lines
doc/guides/nics/gve.rst +1 -0
   reception and transmission.
   Supported features of the GVE PMD are:
 
+- Receiver Side Scaling (RSS)

[copy/paste from previous version]

I am not sure if driver can claim this, I can see a RSS hash is provided but is it possible to update which hash function to use or update key or RETA table to configure which queue packets goes?

Right now what is RSS calculated on?

Quoted 8 lines
Quoted 4 lines

Perpaps RSS support can be documented as limited?

And not sure if this update belongs this patch, it should be to the one that has the datapath.

Looks that the RSS is enabled by default, and there is no RSS init API. So I just added back the force-enabled RSS offloading code with the corresponding commit message. So the feature list remains unchaged.

There is difference between RSS and RSS hash, what force enabled is "RSS hash" where device calculated hash value is shared with application in case application wants to reuse this value for some reasons.

But for RSS support, there is a set of configuration required by DPDK seems missing, as mentioned above, config RSS hash function (based on which part of the packet is the hash calculated), or RETA table update so app can select which packets goes to which queue, etc...

Quoted 5 lines

Is it at least possible to document what existing configuration is?

Because of missing configuration support, I don't think it is correct to document RSS as supported, can you please update it to say something like limited support exist with default config.

Thanks for the detailed explanations!
Yes, you are right. The RSS cannot be configured by this user at this time. So it should be commented as an limitation here. Will update this in the coming version. Really thanks for your time and efforts to review!

#113 lore raw

PATCH v7 7/8 net/gve: add support for queue operations

In reply to Junfeng Guo

Add support for queue operations:
- setup rx/tx queue
- release rx/tx queue
- start rx/tx queues
- stop rx/tx queues

---
 drivers/net/gve/gve_ethdev.c | 204 +++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h |  52 +++++++++
 drivers/net/gve/gve_rx.c     | 212 ++++++++++++++++++++++++++++++++++
 drivers/net/gve/gve_tx.c     | 214 +++++++++++++++++++++++++++++++++++
 drivers/net/gve/meson.build  |   2 +
 5 files changed, 684 insertions(+)
 create mode 100644 drivers/net/gve/gve_rx.c
 create mode 100644 drivers/net/gve/gve_tx.c
drivers/net/gve/gve_ethdev.c +204 -0
index 7fbe0c78c9..892e7e2e1c 100644
@@ -28,6 +28,68 @@ gve_write_version(uint8_t *driver_version_register)
 	writeb('\n', driver_version_register);
 }
 
+static int
+gve_alloc_queue_page_list(struct gve_priv *priv, uint32_t id, uint32_t pages)
+{
+	char z_name[RTE_MEMZONE_NAMESIZE];
+	struct gve_queue_page_list *qpl;
+	const struct rte_memzone *mz;
+	dma_addr_t page_bus;
+	uint32_t i;
+
+	if (priv->num_registered_pages + pages >
+	    priv->max_registered_pages) {
+		PMD_DRV_LOG(ERR, "Pages %" PRIu64 " > max registered pages %" PRIu64,
+			    priv->num_registered_pages + pages,
+			    priv->max_registered_pages);
+		return -EINVAL;
+	}
+	qpl = &priv->qpl[id];
+	snprintf(z_name, sizeof(z_name), "gve_%s_qpl%d", priv->pci_dev->device.name, id);
+	mz = rte_memzone_reserve_aligned(z_name, pages * PAGE_SIZE,
+					 rte_socket_id(),
+					 RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE);
+	if (mz == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc %s.", z_name);
+		return -ENOMEM;
+	}
+	qpl->page_buses = rte_zmalloc("qpl page buses", pages * sizeof(dma_addr_t), 0);
+	if (qpl->page_buses == NULL) {
+		PMD_DRV_LOG(ERR, "Failed to alloc qpl %u page buses", id);
+		return -ENOMEM;
+	}
+	page_bus = mz->iova;
+	for (i = 0; i < pages; i++) {
+		qpl->page_buses[i] = page_bus;
+		page_bus += PAGE_SIZE;
+	}
+	qpl->id = id;
+	qpl->mz = mz;
+	qpl->num_entries = pages;
+
+	priv->num_registered_pages += pages;
+
+	return 0;
+}
+
+static void
+gve_free_qpls(struct gve_priv *priv)
+{
+	uint16_t nb_txqs = priv->max_nb_txq;
+	uint16_t nb_rxqs = priv->max_nb_rxq;
+	uint32_t i;
+
+	for (i = 0; i < nb_txqs + nb_rxqs; i++) {
+		if (priv->qpl[i].mz != NULL)
+			rte_memzone_free(priv->qpl[i].mz);
+		if (priv->qpl[i].page_buses != NULL)
+			rte_free(priv->qpl[i].page_buses);
+	}
+
+	if (priv->qpl != NULL)
+		rte_free(priv->qpl);
+}
+
 static int
 gve_dev_configure(struct rte_eth_dev *dev)
 {
@@ -42,6 +104,43 @@ gve_dev_configure(struct rte_eth_dev *dev)
 	return 0;
 }
 
+static int
+gve_refill_pages(struct gve_rx_queue *rxq)
+{
+	struct rte_mbuf *nmb;
+	uint16_t i;
+	int diag;
+
+	diag = rte_pktmbuf_alloc_bulk(rxq->mpool, &rxq->sw_ring[0], rxq->nb_rx_desc);
+	if (diag < 0) {
+		for (i = 0; i < rxq->nb_rx_desc - 1; i++) {
+			nmb = rte_pktmbuf_alloc(rxq->mpool);
+			if (!nmb)
+				break;
+			rxq->sw_ring[i] = nmb;
+		}
+		if (i < rxq->nb_rx_desc - 1)
+			return -ENOMEM;
+	}
+	rxq->nb_avail = 0;
+	rxq->next_avail = rxq->nb_rx_desc - 1;
+
+	for (i = 0; i < rxq->nb_rx_desc; i++) {
+		if (rxq->is_gqi_qpl) {
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(i * PAGE_SIZE);
+		} else {
+			if (i == rxq->nb_rx_desc - 1)
+				break;
+			nmb = rxq->sw_ring[i];
+			rxq->rx_data_ring[i].addr = rte_cpu_to_be_64(rte_mbuf_data_iova(nmb));
+		}
+	}
+
+	rte_write32(rte_cpu_to_be_32(rxq->next_avail), rxq->qrx_tail);
+
+	return 0;
+}
+
 static int
 gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 {
@@ -73,16 +172,70 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete)
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
+	uint16_t num_queues = dev->data->nb_tx_queues;
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
+	uint16_t i;
+	int err;
+
+	priv->txqs = (struct gve_tx_queue **)dev->data->tx_queues;
+	err = gve_adminq_create_tx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u tx queues.", num_queues);
+		return err;
+	}
+	for (i = 0; i < num_queues; i++) {
+		txq = priv->txqs[i];
+		txq->qtx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(txq->qres->db_index)];
+		txq->qtx_head =
+		&priv->cnt_array[rte_be_to_cpu_32(txq->qres->counter_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), txq->ntfy_addr);
+	}
+
+	num_queues = dev->data->nb_rx_queues;
+	priv->rxqs = (struct gve_rx_queue **)dev->data->rx_queues;
+	err = gve_adminq_create_rx_queues(priv, num_queues);
+	if (err) {
+		PMD_DRV_LOG(ERR, "failed to create %u rx queues.", num_queues);
+		goto err_tx;
+	}
+	for (i = 0; i < num_queues; i++) {
+		rxq = priv->rxqs[i];
+		rxq->qrx_tail =
+		&priv->db_bar2[rte_be_to_cpu_32(rxq->qres->db_index)];
+
+		rte_write32(rte_cpu_to_be_32(GVE_IRQ_MASK), rxq->ntfy_addr);
+
+		err = gve_refill_pages(rxq);
+		if (err) {
+			PMD_DRV_LOG(ERR, "Failed to refill for RX");
+			goto err_rx;
+		}
+	}
+
 	dev->data->dev_started = 1;
 	gve_link_update(dev, 0);
 
 	return 0;
+
+err_rx:
+	gve_stop_rx_queues(dev);
+err_tx:
+	gve_stop_tx_queues(dev);
+	return err;
 }
 
 static int
 gve_dev_stop(struct rte_eth_dev *dev)
 {
 	dev->data->dev_link.link_status = RTE_ETH_LINK_DOWN;
+
+	gve_stop_tx_queues(dev);
+	gve_stop_rx_queues(dev);
+
 	dev->data->dev_started = 0;
 
 	return 0;
@@ -91,7 +244,11 @@ gve_dev_stop(struct rte_eth_dev *dev)
 static int
 gve_dev_close(struct rte_eth_dev *dev)
 {
+	struct gve_priv *priv = dev->data->dev_private;
+	struct gve_tx_queue *txq;
+	struct gve_rx_queue *rxq;
 	int err = 0;
+	uint16_t i;
 
 	if (dev->data->dev_started) {
 		err = gve_dev_stop(dev);
@@ -99,6 +256,19 @@ gve_dev_close(struct rte_eth_dev *dev)
 			PMD_DRV_LOG(ERR, "Failed to stop dev.");
 	}
 
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		txq = dev->data->tx_queues[i];
+		gve_tx_queue_release(txq);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		rxq = dev->data->rx_queues[i];
+		gve_rx_queue_release(rxq);
+	}
+
+	gve_free_qpls(priv);
+	rte_free(priv->adminq);
+
 	dev->data->mac_addrs = NULL;
 
 	return err;
@@ -185,6 +355,8 @@ static const struct eth_dev_ops gve_eth_dev_ops = {
 	.dev_stop             = gve_dev_stop,
 	.dev_close            = gve_dev_close,
 	.dev_infos_get        = gve_dev_info_get,
+	.rx_queue_setup       = gve_rx_queue_setup,
+	.tx_queue_setup       = gve_tx_queue_setup,
 	.link_update          = gve_link_update,
 	.mtu_set              = gve_dev_mtu_set,
 };
@@ -322,7 +494,9 @@ gve_setup_device_resources(struct gve_priv *priv)
 static int
 gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 {
+	uint16_t pages;
 	int num_ntfy;
+	uint32_t i;
 	int err;
 
 	/* Set up the adminq */
@@ -373,10 +547,40 @@ gve_init_priv(struct gve_priv *priv, bool skip_describe_device)
 	PMD_DRV_LOG(INFO, "Max TX queues %d, Max RX queues %d",
 		    priv->max_nb_txq, priv->max_nb_rxq);
 
+	/* In GQI_QPL queue format:
+	 * Allocate queue page lists according to max queue number
+	 * tx qpl id should start from 0 while rx qpl id should start
+	 * from priv->max_nb_txq
+	 */
+	if (priv->queue_format == GVE_GQI_QPL_FORMAT) {
+		priv->qpl = rte_zmalloc("gve_qpl",
+					(priv->max_nb_txq + priv->max_nb_rxq) *
+					sizeof(struct gve_queue_page_list), 0);
+		if (priv->qpl == NULL) {
+			PMD_DRV_LOG(ERR, "Failed to alloc qpl.");
+			err = -ENOMEM;
+			goto free_adminq;
+		}
+
+		for (i = 0; i < priv->max_nb_txq + priv->max_nb_rxq; i++) {
+			if (i < priv->max_nb_txq)
+				pages = priv->tx_pages_per_qpl;
+			else
+				pages = priv->rx_data_slot_cnt;
+			err = gve_alloc_queue_page_list(priv, i, pages);
+			if (err != 0) {
+				PMD_DRV_LOG(ERR, "Failed to alloc qpl %u.", i);
+				goto err_qpl;
+			}
+		}
+	}
+
 setup_device:
 	err = gve_setup_device_resources(priv);
 	if (!err)
 		return 0;
+err_qpl:
+	gve_free_qpls(priv);
 free_adminq:
 	gve_adminq_free(priv);
 	return err;
drivers/net/gve/gve_ethdev.h +52 -0

92 lines not on this page · raw message

drivers/net/gve/gve_rx.c new file +212 -0

212 lines not on this page · raw message

drivers/net/gve/gve_tx.c new file +214 -0

214 lines not on this page · raw message

drivers/net/gve/meson.build +2 -0
index d8ec64b3a3..af0010c01c 100644
@@ -9,6 +9,8 @@ endif
 
 sources = files(
         'base/gve_adminq.c',
+        'gve_rx.c',
+        'gve_tx.c',
         'gve_ethdev.c',
 )
 includes += include_directories('base')
Signature · 1 line
2.34.1
#114 lore raw

PATCH v7 8/8 net/gve: add support for Rx/Tx

In reply to Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

---
 doc/guides/nics/features/gve.ini |   2 +
 doc/guides/nics/gve.rst          |   4 +
 drivers/net/gve/gve_ethdev.c     |  15 +-
 drivers/net/gve/gve_ethdev.h     |  18 ++
 drivers/net/gve/gve_rx.c         | 142 ++++++++++
 drivers/net/gve/gve_tx.c         | 454 +++++++++++++++++++++++++++++++
 6 files changed, 634 insertions(+), 1 deletion(-)
doc/guides/nics/features/gve.ini +2 -0
index 986df7f94a..cdc46b08a3 100644
@@ -7,7 +7,9 @@
 Speed capabilities   = Y
 Link status          = Y
 MTU update           = Y
+TSO                  = Y
 RSS hash             = Y
+L4 checksum offload  = Y
 Linux                = Y
 x86-32               = Y
 x86-64               = Y
doc/guides/nics/gve.rst +4 -0
index 8c09a5a7fa..1042852fd6 100644
@@ -62,8 +62,12 @@ In this release, the GVE PMD provides the basic functionality of packet
 reception and transmission.
 Supported features of the GVE PMD are:
 
+- Multiple queues for TX and RX
 - Receiver Side Scaling (RSS)
+- TSO offload
 - Link state information
+- TX multi-segments (Scatter TX)
+- Tx UDP/TCP/SCTP Checksum
 
 Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD.
 Jumbo Frame is not supported in PMD for now. It'll be added in the future
drivers/net/gve/gve_ethdev.c +14 -1
index 892e7e2e1c..5c0cd2f2c4 100644
@@ -289,7 +289,13 @@ gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->min_mtu = GVE_MIN_MTU;
 
 	dev_info->rx_offload_capa = 0;
-	dev_info->tx_offload_capa = 0;
+	dev_info->tx_offload_capa =
+		RTE_ETH_TX_OFFLOAD_MULTI_SEGS	|
+		RTE_ETH_TX_OFFLOAD_IPV4_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_UDP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_TCP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_SCTP_CKSUM	|
+		RTE_ETH_TX_OFFLOAD_TCP_TSO;
 
 	if (priv->queue_format == GVE_DQO_RDA_FORMAT)
 		dev_info->rx_offload_capa |= RTE_ETH_RX_OFFLOAD_TCP_LRO;
@@ -639,6 +645,13 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
 	if (err)
 		return err;
 
+	if (gve_is_gqi(priv)) {
+		eth_dev->rx_pkt_burst = gve_rx_burst;
+		eth_dev->tx_pkt_burst = gve_tx_burst;
+	} else {
+		PMD_DRV_LOG(ERR, "DQO_RDA is not implemented and will be added in the future");
+	}
+
 	eth_dev->data->mac_addrs = &priv->dev_addr;
 
 	return 0;
drivers/net/gve/gve_ethdev.h +18 -0
index 00c69d1b88..36b334c36b 100644
@@ -37,6 +37,18 @@ union gve_tx_desc {
 	struct gve_tx_seg_desc seg; /* subsequent descs for a packet */
 };
 
+/* Offload features */
+union gve_tx_offload {
+	uint64_t data;
+	struct {
+		uint64_t l2_len:7; /* L2 (MAC) Header Length. */
+		uint64_t l3_len:9; /* L3 (IP) Header Length. */
+		uint64_t l4_len:8; /* L4 Header Length. */
+		uint64_t tso_segsz:16; /* TCP TSO segment size */
+		/* uint64_t unused : 24; */
+	};
+};
+
 struct gve_tx_iovec {
 	uint32_t iov_base; /* offset in fifo */
 	uint32_t iov_len;
@@ -277,4 +289,10 @@ gve_stop_tx_queues(struct rte_eth_dev *dev);
 void
 gve_stop_rx_queues(struct rte_eth_dev *dev);
 
+uint16_t
+gve_rx_burst(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+
+uint16_t
+gve_tx_burst(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 #endif /* _GVE_ETHDEV_H_ */
drivers/net/gve/gve_rx.c +142 -0

148 lines not on this page · raw message

drivers/net/gve/gve_tx.c +454 -0

460 lines not on this page · raw message

Signature · 1 line
2.34.1
#115 lore raw

Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 26 lines
drivers/net/gve/gve_tx.c +26 -0
+
+static inline void
+gve_tx_clean_swr_qpl(struct gve_tx_queue *txq)
+{
+       uint32_t start = txq->sw_ntc;
+       uint32_t ntc, nb_clean;
+
+       ntc = txq->sw_tail;
+
+       if (ntc == start)
+               return;
+
+       /* if wrap around, free twice. */
+       if (ntc < start) {
+               nb_clean = txq->nb_tx_desc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               if (start == txq->nb_tx_desc)
+                       start = 0;
+               txq->sw_ntc = start;
+       }
+

drivers/net/gve/gve_tx.c line 138

Quoted 10 lines
drivers/net/gve/gve_tx.c +10 -0
+       if (ntc > start) {
+               nb_clean = ntc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               txq->sw_ntc = start;
+       }
+}

[copy/paste from previous version]

may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.

<...>

drivers/net/gve/gve_tx.c line 461

Quoted 11 lines
drivers/net/gve/gve_tx.c +11 -0
+uint16_t
+gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+       struct gve_tx_queue *txq = tx_queue;
+
+       if (txq->is_gqi_qpl)
+               return gve_tx_burst_qpl(tx_queue, tx_pkts, nb_pkts);
+
+       return gve_tx_burst_ra(tx_queue, tx_pkts, nb_pkts);
+}
+

[copy/paste from previous version]

Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check on datapath.

This is optimization and can be done later, it doesn't have to be in the set.

#116 lore raw

RE: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 64 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:52
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 26 lines
drivers/net/gve/gve_tx.c +26 -0
+
+static inline void
+gve_tx_clean_swr_qpl(struct gve_tx_queue *txq)
+{
+       uint32_t start = txq->sw_ntc;
+       uint32_t ntc, nb_clean;
+
+       ntc = txq->sw_tail;
+
+       if (ntc == start)
+               return;
+
+       /* if wrap around, free twice. */
+       if (ntc < start) {
+               nb_clean = txq->nb_tx_desc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               if (start == txq->nb_tx_desc)
+                       start = 0;
+               txq->sw_ntc = start;
+       }
+

drivers/net/gve/gve_tx.c line 138

Quoted 10 lines
drivers/net/gve/gve_tx.c +10 -0
+       if (ntc > start) {
+               nb_clean = ntc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               txq->sw_ntc = start;
+       }
+}

[copy/paste from previous version]

Quoted 2 lines

may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.

Yes, this 'if' block is dropped in v7 as suggested. Thanks!

Quoted 18 lines

<...>

Review

Quoted 2 lines
(quoted diff) +2 -0
+uint16_t
+gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t

nb_pkts)

drivers/net/gve/gve_tx.c line 461

Quoted 9 lines
drivers/net/gve/gve_tx.c +9 -0
+{
+       struct gve_tx_queue *txq = tx_queue;
+
+       if (txq->is_gqi_qpl)
+               return gve_tx_burst_qpl(tx_queue, tx_pkts, nb_pkts);
+
+       return gve_tx_burst_ra(tx_queue, tx_pkts, nb_pkts);
+}
+

[copy/paste from previous version]

Quoted 6 lines

Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check on datapath.

This is optimization and can be done later, it doesn't have to be in the set.

The exact queue type can be fetched from the backend via adminq in priv->queue_format. So there won't be mix of the queue types. Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD. Also, only GQI_QPL queue format is in use on GCP since GQI_RDA hasn't been released in production.
This part code will be optimized/refactored later when involving the queue type DQO_RDA. Thanks!

#117 lore raw

Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 64 lines · Junfeng
Quoted 64 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:52
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 26 lines
drivers/net/gve/gve_tx.c +26 -0
+
+static inline void
+gve_tx_clean_swr_qpl(struct gve_tx_queue *txq)
+{
+       uint32_t start = txq->sw_ntc;
+       uint32_t ntc, nb_clean;
+
+       ntc = txq->sw_tail;
+
+       if (ntc == start)
+               return;
+
+       /* if wrap around, free twice. */
+       if (ntc < start) {
+               nb_clean = txq->nb_tx_desc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               if (start == txq->nb_tx_desc)
+                       start = 0;
+               txq->sw_ntc = start;
+       }
+

drivers/net/gve/gve_tx.c line 138

Quoted 10 lines
drivers/net/gve/gve_tx.c +10 -0
+       if (ntc > start) {
+               nb_clean = ntc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               txq->sw_ntc = start;
+       }
+}

[copy/paste from previous version]

Quoted 5 lines
Quoted 2 lines

may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.

Yes, this 'if' block is dropped in v7 as suggested. Thanks!

This is v7 and please check above code which has the mentioned 'if' block exist, do you mean in coming v8?

Quoted 33 lines
Quoted 18 lines

<...>

Review

Quoted 2 lines
(quoted diff) +2 -0
+uint16_t
+gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t

nb_pkts)

drivers/net/gve/gve_tx.c line 461

Quoted 9 lines
drivers/net/gve/gve_tx.c +9 -0
+{
+       struct gve_tx_queue *txq = tx_queue;
+
+       if (txq->is_gqi_qpl)
+               return gve_tx_burst_qpl(tx_queue, tx_pkts, nb_pkts);
+
+       return gve_tx_burst_ra(tx_queue, tx_pkts, nb_pkts);
+}
+

[copy/paste from previous version]

Quoted 6 lines

Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check on datapath.

This is optimization and can be done later, it doesn't have to be in the set.

The exact queue type can be fetched from the backend via adminq in priv->queue_format. So there won't be mix of the queue types. Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD. Also, only GQI_QPL queue format is in use on GCP since GQI_RDA hasn't been released in production.
This part code will be optimized/refactored later when involving the queue type DQO_RDA. Thanks!

#118 lore raw

RE: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 80 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Monday, October 24, 2022 18:50
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 64 lines · Junfeng
Quoted 64 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Friday, October 21, 2022 17:52
To: Guo, Junfeng <[email protected]>; Zhang, Qi Z <[email protected]>; Wu, Jingjing <[email protected]>; [email protected]; Xing, Beilei <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Zhang, Helin <[email protected]>
Subject: Re: [PATCH v7 8/8] net/gve: add support for Rx/Tx

Quoted 5 lines · Junfeng Guo

Add Rx/Tx of GQI_QPL queue format and GQI_RDA queue format.

<...>

Quoted 26 lines
drivers/net/gve/gve_tx.c +26 -0
+
+static inline void
+gve_tx_clean_swr_qpl(struct gve_tx_queue *txq)
+{
+       uint32_t start = txq->sw_ntc;
+       uint32_t ntc, nb_clean;
+
+       ntc = txq->sw_tail;
+
+       if (ntc == start)
+               return;
+
+       /* if wrap around, free twice. */
+       if (ntc < start) {
+               nb_clean = txq->nb_tx_desc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               if (start == txq->nb_tx_desc)
+                       start = 0;
+               txq->sw_ntc = start;
+       }
+

drivers/net/gve/gve_tx.c line 138

Quoted 10 lines
drivers/net/gve/gve_tx.c +10 -0
+       if (ntc > start) {
+               nb_clean = ntc - start;
+               if (nb_clean > GVE_TX_MAX_FREE_SZ)
+                       nb_clean = GVE_TX_MAX_FREE_SZ;
+               gve_free_bulk_mbuf(&txq->sw_ring[start], nb_clean);
+               txq->sw_nb_free += nb_clean;
+               start += nb_clean;
+               txq->sw_ntc = start;
+       }
+}

[copy/paste from previous version]

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

may be can drop the 'if' block, since "ntc == start" and "ntc < start" cases already covered.

Yes, this 'if' block is dropped in v7 as suggested. Thanks!

This is v7 and please check above code which has the mentioned 'if' block exist, do you mean in coming v8?

Oh, sorry about this! There is another function with the same issue. I just updated the code there and forgot this. Will update this and check for the rest in the coming version. Thanks!

Quoted 35 lines
Quoted 19 lines
Quoted 19 lines

<...>

Review

Quoted 2 lines
(quoted diff) +2 -0
+uint16_t
+gve_tx_burst(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t

nb_pkts)

drivers/net/gve/gve_tx.c line 461

Quoted 9 lines
drivers/net/gve/gve_tx.c +9 -0
+{
+       struct gve_tx_queue *txq = tx_queue;
+
+       if (txq->is_gqi_qpl)
+               return gve_tx_burst_qpl(tx_queue, tx_pkts, nb_pkts);
+
+       return gve_tx_burst_ra(tx_queue, tx_pkts, nb_pkts);
+}
+

[copy/paste from previous version]

Quoted 3 lines
Quoted 3 lines

Can there be mix of queue types?
If only one queue type is supported in specific config, perhaps burst function can be set during configuration, to prevent if check on

datapath.

Quoted 12 lines
Quoted 3 lines

This is optimization and can be done later, it doesn't have to be in the set.

The exact queue type can be fetched from the backend via adminq in priv->queue_format. So there won't be mix of the queue types. Currently, only GQI_QPL and GQI_RDA queue format are supported in PMD. Also, only GQI_QPL queue format is in use on GCP since GQI_RDA hasn't been released in production.
This part code will be optimized/refactored later when involving the queue type DQO_RDA. Thanks!