Skip to contentSkip to conversation

Add gve driver

PATCH Superseded Catherine Sullivan 3 participants 10 messages Jun 28, 2019 – Jun 29, 2019

Subsystems (4)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover Add gve driver

This patch series adds the gve driver which will support the Compute Engine Virtual NIC that will be available in the future.

v2:
- Patch 1:
- Remove gve_size_assert.h and use static_assert instead.
- Loop forever instead of bugging if the device won't reset
- Use module_pci_driver
- Patch 2:
- Use be16_to_cpu in the RX Seq No define
- Remove unneeded ndo_change_mtu
- Patch 3:
- No Changes
- Patch 4:
- Instead of checking netif_carrier_ok in ethtool stats, just make sure the rings have valid pointers.

Catherine Sullivan (4):
gve: Add basic driver framework for Compute Engine Virtual NIC
gve: Add transmit and receive support
gve: Add workqueue and reset support
gve: Add ethtool support

 .../networking/device_drivers/google/gve.rst  |  123 ++
 .../networking/device_drivers/index.rst       |    1 +
 MAINTAINERS                                   |    9 +
 drivers/net/ethernet/Kconfig                  |    1 +
 drivers/net/ethernet/Makefile                 |    1 +
 drivers/net/ethernet/google/Kconfig           |   27 +
 drivers/net/ethernet/google/Makefile          |    5 +
 drivers/net/ethernet/google/gve/Makefile      |    4 +
 drivers/net/ethernet/google/gve/gve.h         |  456 ++++++
 drivers/net/ethernet/google/gve/gve_adminq.c  |  388 ++++++
 drivers/net/ethernet/google/gve/gve_adminq.h  |  215 +++
 drivers/net/ethernet/google/gve/gve_desc.h    |  113 ++
 drivers/net/ethernet/google/gve/gve_ethtool.c |  239 ++++
 drivers/net/ethernet/google/gve/gve_main.c    | 1217 +++++++++++++++++
 .../net/ethernet/google/gve/gve_register.h    |   27 +
 drivers/net/ethernet/google/gve/gve_rx.c      |  445 ++++++
 drivers/net/ethernet/google/gve/gve_tx.c      |  584 ++++++++
 17 files changed, 3855 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/google/gve.rst
 create mode 100644 drivers/net/ethernet/google/Kconfig
 create mode 100644 drivers/net/ethernet/google/Makefile
 create mode 100644 drivers/net/ethernet/google/gve/Makefile
 create mode 100644 drivers/net/ethernet/google/gve/gve.h
 create mode 100644 drivers/net/ethernet/google/gve/gve_adminq.c
 create mode 100644 drivers/net/ethernet/google/gve/gve_adminq.h
 create mode 100644 drivers/net/ethernet/google/gve/gve_desc.h
 create mode 100644 drivers/net/ethernet/google/gve/gve_ethtool.c
 create mode 100644 drivers/net/ethernet/google/gve/gve_main.c
 create mode 100644 drivers/net/ethernet/google/gve/gve_register.h
 create mode 100644 drivers/net/ethernet/google/gve/gve_rx.c
 create mode 100644 drivers/net/ethernet/google/gve/gve_tx.c
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#1 lore raw

PATCH v2 1/4 gve: Add basic driver framework for Compute Engine Virtual NIC

Add a driver framework for the Compute Engine Virtual NIC that will be available in the future.

At this point the only functionality is loading the driver.

---
 .../networking/device_drivers/google/gve.rst  |  82 ++++
 .../networking/device_drivers/index.rst       |   1 +
 MAINTAINERS                                   |   9 +
 drivers/net/ethernet/Kconfig                  |   1 +
 drivers/net/ethernet/Makefile                 |   1 +
 drivers/net/ethernet/google/Kconfig           |  27 ++
 drivers/net/ethernet/google/Makefile          |   5 +
 drivers/net/ethernet/google/gve/Makefile      |   4 +
 drivers/net/ethernet/google/gve/gve.h         | 135 ++++++
 drivers/net/ethernet/google/gve/gve_adminq.c  | 250 ++++++++++
 drivers/net/ethernet/google/gve/gve_adminq.h  | 133 ++++++
 drivers/net/ethernet/google/gve/gve_main.c    | 446 ++++++++++++++++++
 .../net/ethernet/google/gve/gve_register.h    |  27 ++
 13 files changed, 1121 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/google/gve.rst
 create mode 100644 drivers/net/ethernet/google/Kconfig
 create mode 100644 drivers/net/ethernet/google/Makefile
 create mode 100644 drivers/net/ethernet/google/gve/Makefile
 create mode 100644 drivers/net/ethernet/google/gve/gve.h
 create mode 100644 drivers/net/ethernet/google/gve/gve_adminq.c
 create mode 100644 drivers/net/ethernet/google/gve/gve_adminq.h
 create mode 100644 drivers/net/ethernet/google/gve/gve_main.c
 create mode 100644 drivers/net/ethernet/google/gve/gve_register.h
Documentation/networking/device_drivers/google/gve.rst new file +82 -0
new file mode 100644
index 000000000000..7397c82f4c8f
@@ -0,0 +1,82 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+==============================================================
+Linux kernel driver for Compute Engine Virtual Ethernet (gve):
+==============================================================
+
+Supported Hardware
+===================
+The GVE driver binds to a single PCI device id used by the virtual
+Ethernet device found in some Compute Engine VMs.
+
++--------------+----------+---------+
+|Field         | Value    | Comments|
++==============+==========+=========+
+|Vendor ID     | `0x1AE0` | Google  |
++--------------+----------+---------+
+|Device ID     | `0x0042` |         |
++--------------+----------+---------+
+|Sub-vendor ID | `0x1AE0` | Google  |
++--------------+----------+---------+
+|Sub-device ID | `0x0058` |         |
++--------------+----------+---------+
+|Revision ID   | `0x0`    |         |
++--------------+----------+---------+
+|Device Class  | `0x200`  | Ethernet|
++--------------+----------+---------+
+
+PCI Bars
+========
+The gVNIC PCI device exposes three 32-bit memory BARS:
+- Bar0 - Device configuration and status registers.
+- Bar1 - MSI-X vector table
+- Bar2 - IRQ, RX and TX doorbells
+
+Device Interactions
+===================
+The driver interacts with the device in the following ways:
+ - Registers
+    - A block of MMIO registers
+    - See gve_register.h for more detail
+ - Admin Queue
+    - See description below
+ - Interrupts
+    - See supported interrupts below
+
+Registers
+---------
+All registers are MMIO and big endian.
+
+The registers are used for initializing and configuring the device as well as
+querying device status in response to management interrupts.
+
+Admin Queue (AQ)
+----------------
+The Admin Queue is a PAGE_SIZE memory block, treated as an array of AQ
+commands, used by the driver to issue commands to the device and set up
+resources.The driver and the device maintain a count of how many commands
+have been submitted and executed. To issue AQ commands, the driver must do
+the following (with proper locking):
+
+1)  Copy new commands into next available slots in the AQ array
+2)  Increment its counter by he number of new commands
+3)  Write the counter into the GVE_ADMIN_QUEUE_DOORBELL register
+4)  Poll the ADMIN_QUEUE_EVENT_COUNTER register until it equals
+    the value written to the doorbell, or until a timeout.
+
+The device will update the status field in each AQ command reported as
+executed through the ADMIN_QUEUE_EVENT_COUNTER register.
+
+Interrupts
+----------
+The following interrupts are supported by the driver:
+
+Management Interrupt
+~~~~~~~~~~~~~~~~~~~~
+The management interrupt is used by the device to tell the driver to
+look at the GVE_DEVICE_STATUS register.
+
+Notification Block Interrupts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The notification block interrupts are used to tell the driver to poll
+the queues associated with that interrupt.
Documentation/networking/device_drivers/index.rst +1 -0
index 24598d5f8ffa..2b7fefe72351 100644
@@ -21,6 +21,7 @@ Contents:
    intel/i40e
    intel/iavf
    intel/ice
+   google/gve
    mellanox/mlx5
 
 .. only::  subproject
MAINTAINERS +9 -0
index b4304d10f14e..449e7cdb3303 100644
@@ -6720,6 +6720,15 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/input/touchscreen/goodix.c
 
+GOOGLE ETHERNET DRIVERS
+M:	Catherine Sullivan <[email protected]>
+R:	Sagi Shahar <[email protected]>
+R:	Jon Olson <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	Documentation/networking/device_drivers/google/gve.txt
+F:	drivers/net/ethernet/google
+
 GPD POCKET FAN DRIVER
 M:	Hans de Goede <[email protected]>
 L:	[email protected]
drivers/net/ethernet/Kconfig +1 -0
index fe115b7caba0..93a2d4deb27c 100644
@@ -76,6 +76,7 @@ source "drivers/net/ethernet/ezchip/Kconfig"
 source "drivers/net/ethernet/faraday/Kconfig"
 source "drivers/net/ethernet/freescale/Kconfig"
 source "drivers/net/ethernet/fujitsu/Kconfig"
+source "drivers/net/ethernet/google/Kconfig"
 source "drivers/net/ethernet/hisilicon/Kconfig"
 source "drivers/net/ethernet/hp/Kconfig"
 source "drivers/net/ethernet/huawei/Kconfig"
drivers/net/ethernet/Makefile +1 -0
index 7b5bf9682066..fb9155cffcff 100644
@@ -39,6 +39,7 @@ obj-$(CONFIG_NET_VENDOR_EZCHIP) += ezchip/
 obj-$(CONFIG_NET_VENDOR_FARADAY) += faraday/
 obj-$(CONFIG_NET_VENDOR_FREESCALE) += freescale/
 obj-$(CONFIG_NET_VENDOR_FUJITSU) += fujitsu/
+obj-$(CONFIG_NET_VENDOR_GOOGLE) += google/
 obj-$(CONFIG_NET_VENDOR_HISILICON) += hisilicon/
 obj-$(CONFIG_NET_VENDOR_HP) += hp/
 obj-$(CONFIG_NET_VENDOR_HUAWEI) += huawei/
drivers/net/ethernet/google/Kconfig new file +27 -0
new file mode 100644
index 000000000000..888f08f36101
@@ -0,0 +1,27 @@
+#
+# Google network device configuration
+#
+
+config NET_VENDOR_GOOGLE
+	bool "Google Devices"
+	default y
+	help
+	  If you have a network (Ethernet) device belonging to this class, say Y.
+
+	  Note that the answer to this question doesn't directly affect the
+	  kernel: saying N will just cause the configurator to skip all
+	  the questions about Google devices. If you say Y, you will be asked
+	  for your specific device in the following questions.
+
+if NET_VENDOR_GOOGLE
+
+config GVE
+	tristate "Google Virtual NIC (gVNIC) support"
+	depends on (PCI_MSI && X86)
+	help
+	  This driver supports Google Virtual NIC (gVNIC)"
+
+	  To compile this driver as a module, choose M here.
+	  The module will be called gve.
+
+endif #NET_VENDOR_GOOGLE
drivers/net/ethernet/google/Makefile new file +5 -0
new file mode 100644
index 000000000000..402cc3ba1639
@@ -0,0 +1,5 @@
+#
+# Makefile for the Google network device drivers.
+#
+
+obj-$(CONFIG_GVE) += gve/
drivers/net/ethernet/google/gve/Makefile new file +4 -0
new file mode 100644
index 000000000000..cec03ee6d931
@@ -0,0 +1,4 @@
+# Makefile for the Google virtual Ethernet (gve) driver
+
+obj-$(CONFIG_GVE) += gve.o
+gve-objs := gve_main.o gve_adminq.o
drivers/net/ethernet/google/gve/gve.h new file +135 -0

135 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_adminq.c new file +250 -0

250 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_adminq.h new file +133 -0

133 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_main.c new file +446 -0

446 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_register.h new file +27 -0
new file mode 100644
index 000000000000..84ab8893aadd
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: (GPL-2.0 OR MIT)
+ * Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2019 Google, Inc.
+ */
+
+#ifndef _GVE_REGISTER_H_
+#define _GVE_REGISTER_H_
+
+/* Fixed Configuration Registers */
+struct gve_registers {
+	__be32	device_status;
+	__be32	driver_status;
+	__be32	max_tx_queues;
+	__be32	max_rx_queues;
+	__be32	adminq_pfn;
+	__be32	adminq_doorbell;
+	__be32	adminq_event_counter;
+	u8	reserved[3];
+	u8	driver_version;
+};
+
+enum gve_device_status_flags {
+	GVE_DEVICE_STATUS_RESET_MASK		= BIT(1),
+	GVE_DEVICE_STATUS_LINK_STATUS_MASK	= BIT(2),
+};
+#endif /* _GVE_REGISTER_H_ */
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#2 lore raw

Re: [PATCH net-next v2 1/4] gve: Add basic driver framework for Compute Engine Virtual NIC

I love your patch! Yet something to improve:

Hi Catherine,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url: https://github.com/0day-ci/linux/commits/Catherine-Sullivan/Add-gve-driver/20190629-143743
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.4.0-9) 7.4.0
reproduce:

        # save the attached .config to linux build tree
        make ARCH=i386 

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

In file included from drivers/net/ethernet/google/gve/gve_main.c:16:0:

>> drivers/net/ethernet/google/gve/gve_adminq.h:102:1: error: static assertion failed: "sizeof(struct gve_adminq_set_driver_parameter) == 16"
    static_assert(sizeof(struct gve_adminq_set_driver_parameter) == 16);
    ^~~~~~~~~~~~~~

vim +102 drivers/net/ethernet/google/gve/gve_adminq.h

101

 > 102	static_assert(sizeof(struct gve_adminq_set_driver_parameter) == 16);

103

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#3 lore raw

PATCH v2 2/4 gve: Add transmit and receive support

In reply to Catherine Sullivan

Add support for passing traffic.

---
 .../networking/device_drivers/google/gve.rst  |  30 +
 drivers/net/ethernet/google/gve/Makefile      |   2 +-
 drivers/net/ethernet/google/gve/gve.h         | 257 +++++++-
 drivers/net/ethernet/google/gve/gve_adminq.c  | 138 +++++
 drivers/net/ethernet/google/gve/gve_adminq.h  |  82 +++
 drivers/net/ethernet/google/gve/gve_desc.h    | 113 ++++
 drivers/net/ethernet/google/gve/gve_main.c    | 562 ++++++++++++++++-
 drivers/net/ethernet/google/gve/gve_rx.c      | 442 +++++++++++++
 drivers/net/ethernet/google/gve/gve_tx.c      | 584 ++++++++++++++++++
 9 files changed, 2205 insertions(+), 5 deletions(-)
 create mode 100644 drivers/net/ethernet/google/gve/gve_desc.h
 create mode 100644 drivers/net/ethernet/google/gve/gve_rx.c
 create mode 100644 drivers/net/ethernet/google/gve/gve_tx.c
Documentation/networking/device_drivers/google/gve.rst +30 -0
index 7397c82f4c8f..df8974fb3270 100644
@@ -42,6 +42,8 @@ The driver interacts with the device in the following ways:
     - See description below
  - Interrupts
     - See supported interrupts below
+ - Transmit and Receive Queues
+    - See description below
 
 Registers
 ---------
@@ -80,3 +82,31 @@ Notification Block Interrupts
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The notification block interrupts are used to tell the driver to poll
 the queues associated with that interrupt.
+
+The handler for these irqs schedule the napi for that block to run
+and poll the queues.
+
+Traffic Queues
+--------------
+gVNIC's queues are composed of a descriptor ring and a buffer and are
+assigned to a notification block.
+
+The descriptor rings are power-of-two-sized ring buffers consisting of
+fixed-size descriptors. They advance their head pointer using a __be32
+doorbell located in Bar2. The tail pointers are advanced by consuming
+descriptors in-order and updating a __be32 counter. Both the doorbell
+and the counter overflow to zero.
+
+Each queue's buffers must be registered in advance with the device as a
+queue page list, and packet data can only be put in those pages.
+
+Transmit
+~~~~~~~~
+gve maps the buffers for transmit rings into a FIFO and copies the packets
+into the FIFO before sending them to the NIC.
+
+Receive
+~~~~~~~
+The buffers for receive rings are put into a data ring that is the same
+length as the descriptor ring and the head and tail pointers advance over
+the rings together.
drivers/net/ethernet/google/gve/Makefile +1 -1
index cec03ee6d931..a1890c93705b 100644
@@ -1,4 +1,4 @@
 # Makefile for the Google virtual Ethernet (gve) driver
 
 obj-$(CONFIG_GVE) += gve.o
-gve-objs := gve_main.o gve_adminq.o
+gve-objs := gve_main.o gve_tx.o gve_rx.o gve_adminq.o
drivers/net/ethernet/google/gve/gve.h +256 -1

297 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_adminq.c +138 -0

162 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_adminq.h +82 -0

106 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_desc.h new file +113 -0

113 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_main.c +559 -3

632 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_rx.c new file +442 -0

442 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_tx.c new file +584 -0

584 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#4 lore raw

Re: [PATCH net-next v2 2/4] gve: Add transmit and receive support

I love your patch! Perhaps something to improve:

Hi Catherine,

I love your patch! Perhaps something to improve:

[auto build test WARNING on net-next/master]

url: https://github.com/0day-ci/linux/commits/Catherine-Sullivan/Add-gve-driver/20190629-143743
reproduce:

        # apt-get install sparse
        # sparse version: v0.6.1-rc1-7-g2b96cd8-dirty
        make ARCH=x86_64 allmodconfig
        make C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__'

If you fix the issue, kindly add following tag

sparse warnings: (new ones prefixed by >>)

drivers/net/ethernet/google/gve/gve_rx.c:11:6: sparse: sparse: symbol 'gve_rx_remove_from_block' was not declared. Should it be static?
drivers/net/ethernet/google/gve/gve_rx.c:217:16: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned int val @@ got restricted __be3unsigned int val @@
drivers/net/ethernet/google/gve/gve_rx.c:217:16: sparse: expected unsigned int val
drivers/net/ethernet/google/gve/gve_rx.c:217:16: sparse: got restricted __be32 [usertype]
drivers/net/ethernet/google/gve/gve_rx.c:349:27: sparse: sparse: incorrect type in assignment (different base types) @@ expected restricted __wsum [usertype] csum @@ got restricted __wsum [usertype] csum @@
drivers/net/ethernet/google/gve/gve_rx.c:349:27: sparse: expected restricted __wsum [usertype] csum
drivers/net/ethernet/google/gve/gve_rx.c:349:27: sparse: got restricted __be16 [usertype] csum
drivers/net/ethernet/google/gve/gve_rx.c:374:19: sparse: sparse: incorrect type in assignment (different base types) @@ expected unsigned short [usertype] flags_seq @@ got resunsigned short [usertype] flags_seq @@
drivers/net/ethernet/google/gve/gve_rx.c:374:19: sparse: expected unsigned short [usertype] flags_seq
drivers/net/ethernet/google/gve/gve_rx.c:374:19: sparse: got restricted __be16 [usertype] flags_seq

>> drivers/net/ethernet/google/gve/gve_rx.c:378:17: sparse: sparse: cast to restricted __be16
>> drivers/net/ethernet/google/gve/gve_rx.c:378:17: sparse: sparse: cast to restricted __be16
>> drivers/net/ethernet/google/gve/gve_rx.c:378:17: sparse: sparse: cast to restricted __be16
>> drivers/net/ethernet/google/gve/gve_rx.c:378:17: sparse: sparse: cast to restricted __be16

vim +378 drivers/net/ethernet/google/gve/gve_rx.c

212
213 void gve_rx_write_doorbell(struct gve_priv *priv, struct gve_rx_ring *rx)
214 {
215 u32 db_idx = be32_to_cpu(rx->q_resources->db_index);
216

 > 217		writel(cpu_to_be32(rx->desc.fill_cnt), &priv->db_bar2[db_idx]);
   218	}
   219	
   220	static enum pkt_hash_types gve_rss_type(__be16 pkt_flags)
   221	{
   222		if (likely(pkt_flags & (GVE_RXF_TCP | GVE_RXF_UDP)))
   223			return PKT_HASH_TYPE_L4;
   224		if (pkt_flags & (GVE_RXF_IPV4 | GVE_RXF_IPV6))
   225			return PKT_HASH_TYPE_L3;
   226		return PKT_HASH_TYPE_L2;
   227	}
   228	
   229	static struct sk_buff *gve_rx_copy(struct net_device *dev,
   230					   struct napi_struct *napi,
   231					   struct gve_rx_slot_page_info *page_info,
   232					   u16 len)
   233	{
   234		struct sk_buff *skb = napi_alloc_skb(napi, len);
   235		void *va = page_info->page_address + GVE_RX_PAD +
   236			   page_info->page_offset;
   237	
   238		if (unlikely(!skb))
   239			return NULL;
   240	
   241		__skb_put(skb, len);
   242	
   243		skb_copy_to_linear_data(skb, va, len);
   244	
   245		skb->protocol = eth_type_trans(skb, dev);
   246		return skb;
   247	}
   248	
   249	static struct sk_buff *gve_rx_add_frags(struct net_device *dev,
   250						struct napi_struct *napi,
   251						struct gve_rx_slot_page_info *page_info,
   252						u16 len)
   253	{
   254		struct sk_buff *skb = napi_get_frags(napi);
   255	
   256		if (unlikely(!skb))
   257			return NULL;
   258	
   259		skb_add_rx_frag(skb, 0, page_info->page,
   260				page_info->page_offset +
   261				GVE_RX_PAD, len, PAGE_SIZE / 2);
   262	
   263		return skb;
   264	}
   265	
   266	static void gve_rx_flip_buff(struct gve_rx_slot_page_info *page_info,
   267				     struct gve_rx_data_slot *data_ring)
   268	{
   269		u64 addr = be64_to_cpu(data_ring->qpl_offset);
   270	
   271		page_info->page_offset ^= PAGE_SIZE / 2;
   272		addr ^= PAGE_SIZE / 2;
   273		data_ring->qpl_offset = cpu_to_be64(addr);
   274	}
   275	
   276	static bool gve_rx(struct gve_rx_ring *rx, struct gve_rx_desc *rx_desc,
   277			   netdev_features_t feat)
   278	{
   279		struct gve_rx_slot_page_info *page_info;
   280		struct gve_priv *priv = rx->gve;
   281		struct napi_struct *napi = &priv->ntfy_blocks[rx->ntfy_id].napi;
   282		struct net_device *dev = priv->dev;
   283		struct sk_buff *skb;
   284		int pagecount;
   285		u16 len;
   286		u32 idx;
   287	
   288		/* drop this packet */
   289		if (unlikely(rx_desc->flags_seq & GVE_RXF_ERR))
   290			return true;
   291	
   292		len = be16_to_cpu(rx_desc->len) - GVE_RX_PAD;
   293		idx = rx->data.cnt & rx->data.mask;
   294		page_info = &rx->data.page_info[idx];
   295	
   296		/* gvnic can only receive into registered segments. If the buffer
   297		 * can't be recycled, our only choice is to copy the data out of
   298		 * it so that we can return it to the device.
   299		 */
   300	
   301	#if PAGE_SIZE == 4096
   302		if (len <= priv->rx_copybreak) {
   303			/* Just copy small packets */
   304			skb = gve_rx_copy(dev, napi, page_info, len);
   305			goto have_skb;
   306		}
   307		if (unlikely(!gve_can_recycle_pages(dev))) {
   308			skb = gve_rx_copy(dev, napi, page_info, len);
   309			goto have_skb;
   310		}
   311		pagecount = page_count(page_info->page);
   312		if (pagecount == 1) {
   313			/* No part of this page is used by any SKBs; we attach
   314			 * the page fragment to a new SKB and pass it up the
   315			 * stack.
   316			 */
   317			skb = gve_rx_add_frags(dev, napi, page_info, len);
   318			if (!skb)
   319				return true;
   320			/* Make sure the kernel stack can't release the page */
   321			get_page(page_info->page);
   322			/* "flip" to other packet buffer on this page */
   323			gve_rx_flip_buff(page_info, &rx->data.data_ring[idx]);
   324		} else if (pagecount >= 2) {
   325			/* We have previously passed the other half of this
   326			 * page up the stack, but it has not yet been freed.
   327			 */
   328			skb = gve_rx_copy(dev, napi, page_info, len);
   329		} else {
   330			WARN(pagecount < 1, "Pagecount should never be < 1");
   331			return false;
   332		}
   333	#else
   334		skb = gve_rx_copy(dev, napi, page_info, len);
   335	#endif
   336	
   337	have_skb:
   338		if (!skb)
   339			return true;
   340	
   341		rx->data.cnt++;
   342	
   343		if (likely(feat & NETIF_F_RXCSUM)) {
   344			/* NIC passes up the partial sum */
   345			if (rx_desc->csum)
   346				skb->ip_summed = CHECKSUM_COMPLETE;
   347			else
   348				skb->ip_summed = CHECKSUM_NONE;
   349			skb->csum = rx_desc->csum;
   350		}
   351	
   352		/* parse flags & pass relevant info up */
   353		if (likely(feat & NETIF_F_RXHASH) &&
   354		    gve_needs_rss(rx_desc->flags_seq))
   355			skb_set_hash(skb, be32_to_cpu(rx_desc->rss_hash),
   356				     gve_rss_type(rx_desc->flags_seq));
   357	
   358		if (skb_is_nonlinear(skb))
   359			napi_gro_frags(napi);
   360		else
   361			napi_gro_receive(napi, skb);
   362		return true;
   363	}
   364	
   365	static bool gve_rx_work_pending(struct gve_rx_ring *rx)
   366	{
   367		struct gve_rx_desc *desc;
   368		u16 flags_seq;
   369		u32 next_idx;
   370	
   371		next_idx = rx->desc.cnt & rx->desc.mask;
   372		desc = rx->desc.desc_ring + next_idx;
   373	
   374		flags_seq = desc->flags_seq;
   375		/* Make sure we have synchronized the seq no with the device */
   376		smp_rmb();
   377	
 > 378		return (GVE_SEQNO(flags_seq) == rx->desc.seqno);

379 }
380

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#5 lore raw

Re: [PATCH net-next v2 2/4] gve: Add transmit and receive support

In reply to Catherine Sullivan
I love your patch! Yet something to improve:

Hi Catherine,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url: https://github.com/0day-ci/linux/commits/Catherine-Sullivan/Add-gve-driver/20190629-143743
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.4.0-9) 7.4.0
reproduce:

        # save the attached .config to linux build tree
        make ARCH=i386 

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

In file included from drivers/net/ethernet/google/gve/gve_main.c:16:0:

>> drivers/net/ethernet/google/gve/gve_adminq.h:134:1: error: static assertion failed: "sizeof(struct gve_adminq_create_rx_queue) == 48"
    static_assert(sizeof(struct gve_adminq_create_rx_queue) == 48);
    ^~~~~~~~~~~~~~
   drivers/net/ethernet/google/gve/gve_adminq.h:171:1: error: static assertion failed: "sizeof(struct gve_adminq_set_driver_parameter) == 16"
    static_assert(sizeof(struct gve_adminq_set_driver_parameter) == 16);
    ^~~~~~~~~~~~~~

vim +134 drivers/net/ethernet/google/gve/gve_adminq.h

133

 > 134	static_assert(sizeof(struct gve_adminq_create_rx_queue) == 48);

135

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#6 lore raw

PATCH v2 3/4 gve: Add workqueue and reset support

In reply to Catherine Sullivan

Add support for the workqueue to handle management interrupts and support for resets.

---
 .../networking/device_drivers/google/gve.rst  |  11 +
 drivers/net/ethernet/google/gve/gve.h         |  62 ++++++
 drivers/net/ethernet/google/gve/gve_main.c    | 190 +++++++++++++++++-
 drivers/net/ethernet/google/gve/gve_rx.c      |   5 +-
 4 files changed, 260 insertions(+), 8 deletions(-)
Documentation/networking/device_drivers/google/gve.rst +11 -0
index df8974fb3270..793693cef6e3 100644
@@ -40,6 +40,8 @@ The driver interacts with the device in the following ways:
     - See gve_register.h for more detail
  - Admin Queue
     - See description below
+ - Reset
+    - At any time the device can be reset
  - Interrupts
     - See supported interrupts below
  - Transmit and Receive Queues
@@ -69,6 +71,12 @@ the following (with proper locking):
 The device will update the status field in each AQ command reported as
 executed through the ADMIN_QUEUE_EVENT_COUNTER register.
 
+Device Resets
+-------------
+A device reset is triggered by writing 0x0 to the AQ PFN register.
+This causes the device to release all resources allocated by the
+driver, including the AQ itself.
+
 Interrupts
 ----------
 The following interrupts are supported by the driver:
@@ -78,6 +86,9 @@ Management Interrupt
 The management interrupt is used by the device to tell the driver to
 look at the GVE_DEVICE_STATUS register.
 
+The handler for the management irq simply queues the service task in
+the workqueue to check the register and acks the irq.
+
 Notification Block Interrupts
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The notification block interrupts are used to tell the driver to poll
drivers/net/ethernet/google/gve/gve.h +62 -0

81 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_main.c +183 -7

327 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_rx.c +4 -1
index 5bcf2508d929..c52fca45fe27 100644
@@ -335,6 +335,9 @@ static bool gve_rx(struct gve_rx_ring *rx, struct gve_rx_desc *rx_desc,
 #endif
 
 have_skb:
+	/* We didn't manage to allocate an skb but we haven't had any
+	 * reset worthy failures.
+	 */
 	if (!skb)
 		return true;
 
@@ -400,7 +403,7 @@ bool gve_clean_rx_done(struct gve_rx_ring *rx, int budget,
 			   rx->desc.seqno);
 		bytes += be16_to_cpu(desc->len) - GVE_RX_PAD;
 		if (!gve_rx(rx, desc, feat))
-			return false;
+			gve_schedule_reset(priv);
 		cnt++;
 		idx = cnt & rx->desc.mask;
 		desc = rx->desc.desc_ring + idx;
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#7 lore raw

PATCH v2 4/4 gve: Add ethtool support

In reply to Catherine Sullivan

Add support for the following ethtool commands:

ethtool -s|--change devname [msglvl N] [msglevel type on|off]
ethtool -S|--statistics devname
ethtool -i|--driver devname
ethtool -l|--show-channels devname
ethtool -L|--set-channels devname
ethtool -g|--show-ring devname

ethtool --reset devname

---
 drivers/net/ethernet/google/gve/Makefile      |   2 +-
 drivers/net/ethernet/google/gve/gve.h         |   4 +
 drivers/net/ethernet/google/gve/gve_ethtool.c | 239 ++++++++++++++++++
 drivers/net/ethernet/google/gve/gve_main.c    |  39 +++
 4 files changed, 283 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/google/gve/gve_ethtool.c
drivers/net/ethernet/google/gve/Makefile +1 -1
index a1890c93705b..3354ce40eb97 100644
@@ -1,4 +1,4 @@
 # Makefile for the Google virtual Ethernet (gve) driver
 
 obj-$(CONFIG_GVE) += gve.o
-gve-objs := gve_main.o gve_tx.o gve_rx.o gve_adminq.o
+gve-objs := gve_main.o gve_tx.o gve_rx.o gve_ethtool.o gve_adminq.o
drivers/net/ethernet/google/gve/gve.h +4 -0
index b7cc23b06284..c765f718dc4a 100644
@@ -449,4 +449,8 @@ int gve_reset(struct gve_priv *priv, bool attempt_teardown);
 int gve_adjust_queues(struct gve_priv *priv,
 		      struct gve_queue_config new_rx_config,
 		      struct gve_queue_config new_tx_config);
+/* exported by ethtool.c */
+extern const struct ethtool_ops gve_ethtool_ops;
+/* needed by ethtool */
+extern const char gve_version_str[];
 #endif /* _GVE_H_ */
drivers/net/ethernet/google/gve/gve_ethtool.c new file +239 -0

239 lines not on this page · raw message

drivers/net/ethernet/google/gve/gve_main.c +39 -0
index c1482924a80c..fce9a5f35c05 100644
@@ -735,6 +735,44 @@ static int gve_close(struct net_device *dev)
 	return gve_reset_recovery(priv, false);
 }
 
+int gve_adjust_queues(struct gve_priv *priv,
+		      struct gve_queue_config new_rx_config,
+		      struct gve_queue_config new_tx_config)
+{
+	int err;
+
+	if (netif_carrier_ok(priv->dev)) {
+		/* To make this process as simple as possible we teardown the
+		 * device, set the new configuration, and then bring the device
+		 * up again.
+		 */
+		err = gve_close(priv->dev);
+		/* we have already tried to reset in close,
+		 * just fail at this point
+		 */
+		if (err)
+			return err;
+		priv->tx_cfg = new_tx_config;
+		priv->rx_cfg = new_rx_config;
+
+		err = gve_open(priv->dev);
+		if (err)
+			goto err;
+
+		return 0;
+	}
+	/* Set the config for the next up. */
+	priv->tx_cfg = new_tx_config;
+	priv->rx_cfg = new_rx_config;
+
+	return 0;
+err:
+	netif_err(priv, drv, priv->dev,
+		  "Adjust queues failed! !!! DISABLING ALL QUEUES !!!\n");
+	gve_turndown(priv);
+	return err;
+}
+
 static void gve_turndown(struct gve_priv *priv)
 {
 	int idx;
@@ -1071,6 +1109,7 @@ static int gve_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	}
 	SET_NETDEV_DEV(dev, &pdev->dev);
 	pci_set_drvdata(pdev, dev);
+	dev->ethtool_ops = &gve_ethtool_ops;
 	dev->netdev_ops = &gve_netdev_ops;
 	/* advertise features */
 	dev->hw_features = NETIF_F_HIGHDMA;
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#8 lore raw

Re: [PATCH net-next v2 4/4] gve: Add ethtool support

Quoted 16 lines · On Fri, 28 Jun 2019 10:56:33 -0700 Catherine Sullivan <[email protected]> wrote:
drivers/net/ethernet/google/gve/gve_ethtool.c new file +16 -0
+static void
+gve_get_ethtool_stats(struct net_device *netdev,
+		      struct ethtool_stats *stats, u64 *data)
+{
+	struct gve_priv *priv = netdev_priv(netdev);
+	u64 rx_pkts, rx_bytes, tx_pkts, tx_bytes;
+	int ring;
+	int i;
+
+	ASSERT_RTNL();
+
+	for (rx_pkts = 0, rx_bytes = 0, ring = 0;
+	     ring < priv->rx_cfg.num_queues; ring++) {
+		if (priv->rx) {
+			rx_pkts += priv->rx[ring].rpackets;
+			rx_bytes += priv->rx[ring].rbytes;

drivers/net/ethernet/google/gve/gve_ethtool.c line 114

Quoted 10 lines
drivers/net/ethernet/google/gve/gve_ethtool.c new file +10 -0
+		}
+	}
+	for (tx_pkts = 0, tx_bytes = 0, ring = 0;
+	     ring < priv->tx_cfg.num_queues; ring++) {
+		if (priv->tx) {
+			tx_pkts += priv->tx[ring].pkt_done;
+			tx_bytes += priv->tx[ring].bytes_done;
+		}
+	}
+	memset(data, 0, GVE_MAIN_STATS_LEN * sizeof(*data));

memset here is unnecessary since ethtool_get_stats allocates and zeros the memory already.

#9 lore raw

Re: [PATCH net-next v2 4/4] gve: Add ethtool support

Quoted 25 lines · Stephen Hemminger
Quoted 21 lines · On Fri, 28 Jun 2019 10:56:33 -0700 Catherine Sullivan <[email protected]> wrote:
drivers/net/ethernet/google/gve/gve_ethtool.c new file +21 -0
+static void
+gve_get_ethtool_stats(struct net_device *netdev,
+                   struct ethtool_stats *stats, u64 *data)
+{
+     struct gve_priv *priv = netdev_priv(netdev);
+     u64 rx_pkts, rx_bytes, tx_pkts, tx_bytes;
+     int ring;
+     int i;
+
+     ASSERT_RTNL();
+
+     for (rx_pkts = 0, rx_bytes = 0, ring = 0;
+          ring < priv->rx_cfg.num_queues; ring++) {
+             if (priv->rx) {
+                     rx_pkts += priv->rx[ring].rpackets;
+                     rx_bytes += priv->rx[ring].rbytes;
+             }
+     }
+     for (tx_pkts = 0, tx_bytes = 0, ring = 0;
+          ring < priv->tx_cfg.num_queues; ring++) {
+             if (priv->tx) {
Quoted 8 lines

drivers/net/ethernet/google/gve/gve_ethtool.c line 114

Quoted 5 lines
drivers/net/ethernet/google/gve/gve_ethtool.c new file +5 -0
+                     tx_pkts += priv->tx[ring].pkt_done;
+                     tx_bytes += priv->tx[ring].bytes_done;
+             }
+     }
+     memset(data, 0, GVE_MAIN_STATS_LEN * sizeof(*data));

memset here is unnecessary since ethtool_get_stats allocates and zeros the memory already.

Ah, thanks, removed in v3.