Skip to contentSkip to conversation

Add ionic driver

PATCH Superseded Shannon Nelson 5 participants 69 messages Jul 22, 2019 – Aug 9, 2019

Subsystems (4)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover Add ionic driver

This is a patch series that adds the ionic driver, supporting the Pensando ethernet device.

In this initial patchset we implement basic transmit and receive. Later patchsets will add more advanced features.

Our thanks to Andrew Lunn, Michal Kubecek, Jacub Kicinski, Jiri Pirko, and the ever present kbuild test robots for their comments and suggestions.

New in v4:
- use devlink struct alloc for ionic device specific struct
- add support for devlink_port
- fixup devlink fixed vs running version usage
- use bitmap_copy() instead of memcpy() for link_ksettings
- don't bother to zero out the advertising bits before copying in the support bits
- drop unknown xcvr types (will be expanded on later)
- flap the connection to force auto-negotiation
- use is_power_of_2() rather than open code
- simplify set/get_pauseparam use of pause->autoneg
- add a couple comments about NIC status data updated in DMA spaces

New in v3:
- use le32_to_cpu() on queue_count[] values in debugfs
- dma_free_coherent() can handle NULL pointers
- remove unused SS_TEST from ethtool handlers
- one more case of stop the tx ring if there is no room
- remove a couple of stray // comments

New in v2:
- removed debugfs error checking and cut down on debugfs use
- remove redundant bounds checking on incoming values for mtu and ethtool
- don't alloc rx_filter memory until the match type has been checked
- free the ionic struct on remove
- simplified link_up and netif_carrier_ok comparison
- put stats into ethtool -S, out of debugfs
- moved dev_cmd and dev_info dumping to ethtool -d, out of debugfs
- added devlink support
- used kernel's rss init routines rather than open code
- set the Kbuild dependant on 64BIT
- cut down on some unnecessary log messaging
- cleaned up ionic_get_link_ksettings
- cleaned up other little code bits here and there

Shannon Nelson (19):
ionic: Add basic framework for IONIC Network device driver
ionic: Add hardware init and device commands
ionic: Add port management commands
ionic: Add basic lif support
ionic: Add interrupts and doorbells
ionic: Add basic adminq support
ionic: Add adminq action
ionic: Add notifyq support
ionic: Add the basic NDO callbacks for netdev support
ionic: Add management of rx filters
ionic: Add Rx filter and rx_mode ndo support
ionic: Add async link status check and basic stats
ionic: Add initial ethtool support
ionic: Add Tx and Rx handling
ionic: Add netdev-event handling
ionic: Add driver stats
ionic: Add RSS support
ionic: Add coalesce and other features
ionic: Add basic devlink interface

 .../networking/device_drivers/index.rst       |    1 +
 .../device_drivers/pensando/ionic.rst         |   64 +
 MAINTAINERS                                   |    8 +
 drivers/net/ethernet/Kconfig                  |    1 +
 drivers/net/ethernet/Makefile                 |    1 +
 drivers/net/ethernet/pensando/Kconfig         |   32 +
 drivers/net/ethernet/pensando/Makefile        |    6 +
 drivers/net/ethernet/pensando/ionic/Makefile  |    8 +
 drivers/net/ethernet/pensando/ionic/ionic.h   |   74 +
 .../net/ethernet/pensando/ionic/ionic_bus.h   |   16 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |  290 ++
 .../ethernet/pensando/ionic/ionic_debugfs.c   |  279 ++
 .../ethernet/pensando/ionic/ionic_debugfs.h   |   38 +
 .../net/ethernet/pensando/ionic/ionic_dev.c   |  535 ++++
 .../net/ethernet/pensando/ionic/ionic_dev.h   |  284 ++
 .../ethernet/pensando/ionic/ionic_devlink.c   |   93 +
 .../ethernet/pensando/ionic/ionic_devlink.h   |   14 +
 .../ethernet/pensando/ionic/ionic_ethtool.c   |  774 +++++
 .../ethernet/pensando/ionic/ionic_ethtool.h   |    9 +
 .../net/ethernet/pensando/ionic/ionic_if.h    | 2552 +++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 2262 +++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  270 ++
 .../net/ethernet/pensando/ionic/ionic_main.c  |  553 ++++
 .../net/ethernet/pensando/ionic/ionic_regs.h  |  133 +
 .../ethernet/pensando/ionic/ionic_rx_filter.c |  143 +
 .../ethernet/pensando/ionic/ionic_rx_filter.h |   35 +
 .../net/ethernet/pensando/ionic/ionic_stats.c |  333 +++
 .../net/ethernet/pensando/ionic/ionic_stats.h |   53 +
 .../net/ethernet/pensando/ionic/ionic_txrx.c  |  879 ++++++
 .../net/ethernet/pensando/ionic/ionic_txrx.h  |   15 +
 30 files changed, 9755 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/pensando/ionic.rst
 create mode 100644 drivers/net/ethernet/pensando/Kconfig
 create mode 100644 drivers/net/ethernet/pensando/Makefile
 create mode 100644 drivers/net/ethernet/pensando/ionic/Makefile
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_bus.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_debugfs.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_debugfs.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_devlink.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_devlink.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_ethtool.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_ethtool.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_if.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_lif.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_lif.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_main.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_regs.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_rx_filter.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_rx_filter.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_stats.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_stats.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.h
Signature · 1 line
2.17.1
#1 lore raw

PATCH v4 1/19 ionic: Add basic framework for IONIC Network device driver

This patch adds a basic driver framework for the Pensando IONIC network device. There is no functionality right now other than the ability to load and unload.

---
 .../networking/device_drivers/index.rst       |  1 +
 .../device_drivers/pensando/ionic.rst         | 64 +++++++++++++++++++
 MAINTAINERS                                   |  8 +++
 drivers/net/ethernet/Kconfig                  |  1 +
 drivers/net/ethernet/Makefile                 |  1 +
 drivers/net/ethernet/pensando/Kconfig         | 32 ++++++++++
 drivers/net/ethernet/pensando/Makefile        |  6 ++
 drivers/net/ethernet/pensando/ionic/Makefile  |  6 ++
 drivers/net/ethernet/pensando/ionic/ionic.h   | 25 ++++++++
 .../net/ethernet/pensando/ionic/ionic_bus.h   | 10 +++
 .../ethernet/pensando/ionic/ionic_bus_pci.c   | 58 +++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_main.c  | 29 +++++++++
 12 files changed, 241 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/pensando/ionic.rst
 create mode 100644 drivers/net/ethernet/pensando/Kconfig
 create mode 100644 drivers/net/ethernet/pensando/Makefile
 create mode 100644 drivers/net/ethernet/pensando/ionic/Makefile
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_bus.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_main.c
Documentation/networking/device_drivers/index.rst +1 -0
index 2b7fefe72351..57fec66c5419 100644
@@ -23,6 +23,7 @@ Contents:
    intel/ice
    google/gve
    mellanox/mlx5
+   pensando/ionic
 
 .. only::  subproject
 
Documentation/networking/device_drivers/pensando/ionic.rst new file +64 -0

64 lines not on this page · raw message

MAINTAINERS +8 -0
index 211ea3a199bd..17634eb4ce72 100644
@@ -12502,6 +12502,14 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/platform/x86/peaq-wmi.c
 
+PENSANDO ETHERNET DRIVERS
+M:	Shannon Nelson <[email protected]>
+M:	Pensando Drivers <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	Documentation/networking/device_drivers/pensando/ionic.rst
+F:	drivers/net/ethernet/pensando/
+
 PER-CPU MEMORY ALLOCATOR
 M:	Dennis Zhou <[email protected]>
 M:	Tejun Heo <[email protected]>
drivers/net/ethernet/Kconfig +1 -0
index 93a2d4deb27c..2830dc283ce6 100644
@@ -168,6 +168,7 @@ config ETHOC
 
 source "drivers/net/ethernet/packetengines/Kconfig"
 source "drivers/net/ethernet/pasemi/Kconfig"
+source "drivers/net/ethernet/pensando/Kconfig"
 source "drivers/net/ethernet/qlogic/Kconfig"
 source "drivers/net/ethernet/qualcomm/Kconfig"
 source "drivers/net/ethernet/rdc/Kconfig"
drivers/net/ethernet/Makefile +1 -0
index fb9155cffcff..061edd22f507 100644
@@ -97,3 +97,4 @@ obj-$(CONFIG_NET_VENDOR_WIZNET) += wiznet/
 obj-$(CONFIG_NET_VENDOR_XILINX) += xilinx/
 obj-$(CONFIG_NET_VENDOR_XIRCOM) += xircom/
 obj-$(CONFIG_NET_VENDOR_SYNOPSYS) += synopsys/
+obj-$(CONFIG_NET_VENDOR_PENSANDO) += pensando/
drivers/net/ethernet/pensando/Kconfig new file +32 -0
new file mode 100644
index 000000000000..5ea570be8379
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Pensando Systems, Inc
+#
+# Pensando device configuration
+#
+
+config NET_VENDOR_PENSANDO
+	bool "Pensando devices"
+	default y
+	help
+	  If you have a network (Ethernet) card belonging to this class, say Y.
+
+	  Note that the answer to this question doesn't directly affect the
+	  kernel: saying N will just cause the configurator to skip all
+	  the questions about Pensando cards. If you say Y, you will be asked
+	  for your specific card in the following questions.
+
+if NET_VENDOR_PENSANDO
+
+config IONIC
+	tristate "Pensando Ethernet IONIC Support"
+	depends on 64BIT && PCI
+	help
+	  This enables the support for the Pensando family of Ethernet
+	  adapters.  More specific information on this driver can be
+	  found in
+	  <file:Documentation/networking/device_drivers/pensando/ionic.rst>.
+
+          To compile this driver as a module, choose M here. The module
+          will be called ionic.
+
+endif # NET_VENDOR_PENSANDO
drivers/net/ethernet/pensando/Makefile new file +6 -0
new file mode 100644
index 000000000000..21ce7499c122
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Makefile for the Pensando network device drivers.
+#
+
+obj-$(CONFIG_IONIC) += ionic/
drivers/net/ethernet/pensando/ionic/Makefile new file +6 -0
new file mode 100644
index 000000000000..beb3faeccac1
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright(c) 2017 - 2019 Pensando Systems, Inc
+
+obj-$(CONFIG_IONIC) := ionic.o
+
+ionic-y := ionic_main.o ionic_bus_pci.o
drivers/net/ethernet/pensando/ionic/ionic.h new file +25 -0
new file mode 100644
index 000000000000..18c79adef06c
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_H_
+#define _IONIC_H_
+
+#define DRV_NAME		"ionic"
+#define DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
+#define DRV_VERSION		"0.11.0-k"
+
+#define PCI_VENDOR_ID_PENSANDO			0x1dd8
+
+#define PCI_DEVICE_ID_PENSANDO_IONIC_ETH_PF	0x1002
+#define PCI_DEVICE_ID_PENSANDO_IONIC_ETH_VF	0x1003
+
+#define IONIC_SUBDEV_ID_NAPLES_25	0x4000
+#define IONIC_SUBDEV_ID_NAPLES_100_4	0x4001
+#define IONIC_SUBDEV_ID_NAPLES_100_8	0x4002
+
+struct ionic {
+	struct pci_dev *pdev;
+	struct device *dev;
+};
+
+#endif /* _IONIC_H_ */
drivers/net/ethernet/pensando/ionic/ionic_bus.h new file +10 -0
new file mode 100644
index 000000000000..94ba0afc6f38
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_BUS_H_
+#define _IONIC_BUS_H_
+
+int ionic_bus_register_driver(void);
+void ionic_bus_unregister_driver(void);
+
+#endif /* _IONIC_BUS_H_ */
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c new file +58 -0

58 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_main.c new file +29 -0
new file mode 100644
index 000000000000..fb49c0ca5095
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#include <linux/module.h>
+#include <linux/version.h>
+#include <linux/netdevice.h>
+#include <linux/utsname.h>
+
+#include "ionic.h"
+#include "ionic_bus.h"
+
+MODULE_DESCRIPTION(DRV_DESCRIPTION);
+MODULE_AUTHOR("Pensando Systems, Inc");
+MODULE_LICENSE("GPL");
+MODULE_VERSION(DRV_VERSION);
+
+static int __init ionic_init_module(void)
+{
+	pr_info("%s %s, ver %s\n", DRV_NAME, DRV_DESCRIPTION, DRV_VERSION);
+	return ionic_bus_register_driver();
+}
+
+static void __exit ionic_cleanup_module(void)
+{
+	ionic_bus_unregister_driver();
+}
+
+module_init(ionic_init_module);
+module_exit(ionic_cleanup_module);
Signature · 1 line
2.17.1
#2 lore raw

PATCH v4 2/19 ionic: Add hardware init and device commands

In reply to Shannon Nelson

The ionic device has a small set of PCI registers, including a device control and data space, and a large set of message commands.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |    2 +-
 drivers/net/ethernet/pensando/ionic/ionic.h   |   20 +
 .../net/ethernet/pensando/ionic/ionic_bus.h   |    1 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |  140 +-
 .../ethernet/pensando/ionic/ionic_debugfs.c   |   67 +
 .../ethernet/pensando/ionic/ionic_debugfs.h   |   28 +
 .../net/ethernet/pensando/ionic/ionic_dev.c   |  132 +
 .../net/ethernet/pensando/ionic/ionic_dev.h   |  144 +
 .../net/ethernet/pensando/ionic/ionic_if.h    | 2552 +++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_main.c  |  296 ++
 .../net/ethernet/pensando/ionic/ionic_regs.h  |  133 +
 11 files changed, 3512 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_debugfs.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_debugfs.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_if.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_regs.h
drivers/net/ethernet/pensando/ionic/Makefile +1 -1
index beb3faeccac1..49a8e28a5c84 100644
@@ -3,4 +3,4 @@
 
 obj-$(CONFIG_IONIC) := ionic.o
 
-ionic-y := ionic_main.o ionic_bus_pci.o
+ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_debugfs.o
drivers/net/ethernet/pensando/ionic/ionic.h +20 -0
index 18c79adef06c..4fd0ae86556a 100644
@@ -4,6 +4,10 @@
 #ifndef _IONIC_H_
 #define _IONIC_H_
 
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_debugfs.h"
+
 #define DRV_NAME		"ionic"
 #define DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
 #define DRV_VERSION		"0.11.0-k"
@@ -17,9 +21,25 @@
 #define IONIC_SUBDEV_ID_NAPLES_100_4	0x4001
 #define IONIC_SUBDEV_ID_NAPLES_100_8	0x4002
 
+#define devcmd_timeout  10
+
 struct ionic {
 	struct pci_dev *pdev;
 	struct device *dev;
+	struct ionic_dev idev;
+	struct mutex dev_cmd_lock;	/* lock for dev_cmd operations */
+	struct dentry *dentry;
+	struct ionic_dev_bar bars[IONIC_BARS_MAX];
+	unsigned int num_bars;
+	struct identity ident;
 };
 
+int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_wait);
+int ionic_set_dma_mask(struct ionic *ionic);
+int ionic_setup(struct ionic *ionic);
+
+int ionic_identify(struct ionic *ionic);
+int ionic_init(struct ionic *ionic);
+int ionic_reset(struct ionic *ionic);
+
 #endif /* _IONIC_H_ */
drivers/net/ethernet/pensando/ionic/ionic_bus.h +1 -0
index 94ba0afc6f38..24b4c01ec03f 100644
@@ -4,6 +4,7 @@
 #ifndef _IONIC_BUS_H_
 #define _IONIC_BUS_H_
 
+const char *ionic_bus_info(struct ionic *ionic);
 int ionic_bus_register_driver(void);
 void ionic_bus_unregister_driver(void);
 
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +138 -2

165 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c new file +67 -0

67 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_debugfs.h new file +28 -0
new file mode 100644
index 000000000000..c3958b0c15b1
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_DEBUGFS_H_
+#define _IONIC_DEBUGFS_H_
+
+#include <linux/debugfs.h>
+
+#ifdef CONFIG_DEBUG_FS
+
+void ionic_debugfs_create(void);
+void ionic_debugfs_destroy(void);
+int ionic_debugfs_add_dev(struct ionic *ionic);
+void ionic_debugfs_del_dev(struct ionic *ionic);
+int ionic_debugfs_add_bars(struct ionic *ionic);
+int ionic_debugfs_add_dev_cmd(struct ionic *ionic);
+int ionic_debugfs_add_ident(struct ionic *ionic);
+#else
+static inline void ionic_debugfs_create(void) { }
+static inline void ionic_debugfs_destroy(void) { }
+static inline int ionic_debugfs_add_dev(struct ionic *ionic) { return 0; }
+static inline void ionic_debugfs_del_dev(struct ionic *ionic) { }
+static inline int ionic_debugfs_add_bars(struct ionic *ionic) { return 0; }
+static inline int ionic_debugfs_add_dev_cmd(struct ionic *ionic) { return 0; }
+static inline int ionic_debugfs_add_ident(struct ionic *ionic) { return 0; }
+#endif
+
+#endif /* _IONIC_DEBUGFS_H_ */
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +132 -0

132 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_dev.h new file +144 -0

144 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_if.h new file +2552 -0

2,417 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_main.c +296 -0

296 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_regs.h new file +133 -0

133 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.17.1

Part of the diff is not shown here. View the raw message.

maintainer
#3 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:06 -0700

Quoted 12 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +12 -0
+void ionic_init_devinfo(struct ionic_dev *idev)
+{
+	idev->dev_info.asic_type = ioread8(&idev->dev_info_regs->asic_type);
+	idev->dev_info.asic_rev = ioread8(&idev->dev_info_regs->asic_rev);
+
+	memcpy_fromio(idev->dev_info.fw_version,
+		      idev->dev_info_regs->fw_version,
+		      IONIC_DEVINFO_FWVERS_BUFLEN);
+
+	memcpy_fromio(idev->dev_info.serial_num,
+		      idev->dev_info_regs->serial_num,
+		      IONIC_DEVINFO_SERIAL_BUFLEN);

...

drivers/net/ethernet/pensando/ionic/ionic_dev.c line 55

Quoted 1 line
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +1 -0
+	sig = ioread32(&idev->dev_info_regs->signature);

I think if you are going to use the io{read,write}{8,16,32,64}() interfaces then you should use io{read,write}{8,16,32,64}_rep() instead of memcpy_{to,from}io().

#4 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

Quoted 16 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:06 -0700

Quoted 12 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +12 -0
+void ionic_init_devinfo(struct ionic_dev *idev)
+{
+	idev->dev_info.asic_type = ioread8(&idev->dev_info_regs->asic_type);
+	idev->dev_info.asic_rev = ioread8(&idev->dev_info_regs->asic_rev);
+
+	memcpy_fromio(idev->dev_info.fw_version,
+		      idev->dev_info_regs->fw_version,
+		      IONIC_DEVINFO_FWVERS_BUFLEN);
+
+	memcpy_fromio(idev->dev_info.serial_num,
+		      idev->dev_info_regs->serial_num,
+		      IONIC_DEVINFO_SERIAL_BUFLEN);

...

Quoted 5 lines

drivers/net/ethernet/pensando/ionic/ionic_dev.c line 55

Quoted 1 line
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +1 -0
+	sig = ioread32(&idev->dev_info_regs->signature);

I think if you are going to use the io{read,write}{8,16,32,64}() interfaces then you should use io{read,write}{8,16,32,64}_rep() instead of memcpy_{to,from}io().

Sure.
sln

maintainer
#5 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:22 -0700

Quoted 17 lines · David Miller
Quoted 16 lines

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:06 -0700

Quoted 12 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +12 -0
+void ionic_init_devinfo(struct ionic_dev *idev)
+{
+ idev->dev_info.asic_type = ioread8(&idev->dev_info_regs->asic_type);
+ idev->dev_info.asic_rev = ioread8(&idev->dev_info_regs->asic_rev);
+
+	memcpy_fromio(idev->dev_info.fw_version,
+		      idev->dev_info_regs->fw_version,
+		      IONIC_DEVINFO_FWVERS_BUFLEN);
+
+	memcpy_fromio(idev->dev_info.serial_num,
+		      idev->dev_info_regs->serial_num,
+		      IONIC_DEVINFO_SERIAL_BUFLEN);

...

Quoted 6 lines
Quoted 5 lines

drivers/net/ethernet/pensando/ionic/ionic_dev.c line 55

Quoted 1 line
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +1 -0
+	sig = ioread32(&idev->dev_info_regs->signature);

I think if you are going to use the io{read,write}{8,16,32,64}() interfaces then you should use io{read,write}{8,16,32,64}_rep() instead of memcpy_{to,from}io().

Sure.

Note, I could be wrong. Please test.

I think the operation of the two things might be different.

#6 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

Quoted 25 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:22 -0700

Quoted 22 lines · David Miller
Quoted 21 lines

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:06 -0700

Quoted 12 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +12 -0
+void ionic_init_devinfo(struct ionic_dev *idev)
+{
+ idev->dev_info.asic_type = ioread8(&idev->dev_info_regs->asic_type);
+ idev->dev_info.asic_rev = ioread8(&idev->dev_info_regs->asic_rev);
+
+	memcpy_fromio(idev->dev_info.fw_version,
+		      idev->dev_info_regs->fw_version,
+		      IONIC_DEVINFO_FWVERS_BUFLEN);
+
+	memcpy_fromio(idev->dev_info.serial_num,
+		      idev->dev_info_regs->serial_num,
+		      IONIC_DEVINFO_SERIAL_BUFLEN);

...

drivers/net/ethernet/pensando/ionic/ionic_dev.c line 55

Quoted 1 line
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +1 -0
+	sig = ioread32(&idev->dev_info_regs->signature);

I think if you are going to use the io{read,write}{8,16,32,64}() interfaces then you should use io{read,write}{8,16,32,64}_rep() instead of memcpy_{to,from}io().

Quoted 4 lines
Quoted 1 line

Sure.

Note, I could be wrong. Please test.

I think the operation of the two things might be different.

It seems to me that memcpy() usually just does the right thing in most cases, so that's what I went with.  Looking into some of the definitions, and at how I used memcpy_...(), I think there are some appropriate ways to use ioread32_rep() in a couple of my cases, and another case or two where the memcpy variant may not make much difference with ioread8_rep().  It's also possible that sparse may have an opinion.  I'll look at them.

sln

#7 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

In reply to David Miller
Quoted 25 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:22 -0700

Quoted 22 lines · David Miller
Quoted 21 lines

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:06 -0700

Quoted 12 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +12 -0
+void ionic_init_devinfo(struct ionic_dev *idev)
+{
+ idev->dev_info.asic_type = ioread8(&idev->dev_info_regs->asic_type);
+ idev->dev_info.asic_rev = ioread8(&idev->dev_info_regs->asic_rev);
+
+	memcpy_fromio(idev->dev_info.fw_version,
+		      idev->dev_info_regs->fw_version,
+		      IONIC_DEVINFO_FWVERS_BUFLEN);
+
+	memcpy_fromio(idev->dev_info.serial_num,
+		      idev->dev_info_regs->serial_num,
+		      IONIC_DEVINFO_SERIAL_BUFLEN);

...

drivers/net/ethernet/pensando/ionic/ionic_dev.c line 55

Quoted 1 line
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +1 -0
+	sig = ioread32(&idev->dev_info_regs->signature);

I think if you are going to use the io{read,write}{8,16,32,64}() interfaces then you should use io{read,write}{8,16,32,64}_rep() instead of memcpy_{to,from}io().

Quoted 4 lines
Quoted 1 line

Sure.

Note, I could be wrong. Please test.

I think the operation of the two things might be different.

Yes, they are different things, the iowrite*_rep() functions write each value from a buffer all to the same single address, rather than copy a buffer into another buffer, and the ioread*_rep() functions repeatedly read from the same address to fill a buffer.

For example, the iowrite32_rep() boils down to this:
        do {
            __raw_writel(*buf++, addr);
        } while (--count);

Not quite what I need.

sln

maintainer
#8 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

From: Shannon Nelson <[email protected]>
Date: Wed, 24 Jul 2019 10:49:09 -0700

Quoted 35 lines · David Miller
Quoted 25 lines

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:22 -0700

Quoted 22 lines · David Miller
Quoted 21 lines

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:06 -0700

Quoted 12 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +12 -0
+void ionic_init_devinfo(struct ionic_dev *idev)
+{
+ idev->dev_info.asic_type = ioread8(&idev->dev_info_regs->asic_type);
+ idev->dev_info.asic_rev = ioread8(&idev->dev_info_regs->asic_rev);
+
+	memcpy_fromio(idev->dev_info.fw_version,
+		      idev->dev_info_regs->fw_version,
+		      IONIC_DEVINFO_FWVERS_BUFLEN);
+
+	memcpy_fromio(idev->dev_info.serial_num,
+		      idev->dev_info_regs->serial_num,
+		      IONIC_DEVINFO_SERIAL_BUFLEN);

...

drivers/net/ethernet/pensando/ionic/ionic_dev.c line 55

Quoted 1 line
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +1 -0
+	sig = ioread32(&idev->dev_info_regs->signature);

I think if you are going to use the io{read,write}{8,16,32,64}() interfaces then you should use io{read,write}{8,16,32,64}_rep() instead of memcpy_{to,from}io().

Quoted 4 lines
Quoted 1 line

Sure.

Note, I could be wrong. Please test.

I think the operation of the two things might be different.

Yes, they are different things, the iowrite*_rep() functions write each value from a buffer all to the same single address, rather than copy a buffer into another buffer, and the ioread*_rep() functions repeatedly read from the same address to fill a buffer.

Quoted 6 lines

For example, the iowrite32_rep() boils down to this:
        do {
            __raw_writel(*buf++, addr);
        } while (--count);

Not quite what I need.

That's what I suspected.

#9 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

In reply to Shannon Nelson
Quoted 28 lines · Shannon Nelson

The ionic device has a small set of PCI registers, including a device control and data space, and a large set of message commands.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |    2 +-
 drivers/net/ethernet/pensando/ionic/ionic.h   |   20 +
 .../net/ethernet/pensando/ionic/ionic_bus.h   |    1 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |  140 +-
 .../ethernet/pensando/ionic/ionic_debugfs.c   |   67 +
 .../ethernet/pensando/ionic/ionic_debugfs.h   |   28 +
 .../net/ethernet/pensando/ionic/ionic_dev.c   |  132 +
 .../net/ethernet/pensando/ionic/ionic_dev.h   |  144 +
 .../net/ethernet/pensando/ionic/ionic_if.h    | 2552
+++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_main.c  |  296 ++
 .../net/ethernet/pensando/ionic/ionic_regs.h  |  133 +
 11 files changed, 3512 insertions(+), 3 deletions(-)
 create mode 100644
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c
 create mode 100644
drivers/net/ethernet/pensando/ionic/ionic_debugfs.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_if.h
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_regs.h

[...]

drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c static const struct pci_device_id ionic_id_table[] = { · line 164

Quoted 6 lines

static void ionic_remove(struct pci_dev *pdev)
{

drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +1 -1
 	struct ionic *ionic = pci_get_drvdata(pdev);
 
-	devm_kfree(&pdev->dev, ionic);
+	if (ionic) {

nit, in case you are doing another re-spin maybe early return here:
if (!ionic)

     return;

//do stuff

Quoted 14 lines
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +12 -0
+		ionic_reset(ionic);
+		ionic_dev_teardown(ionic);
+		ionic_unmap_bars(ionic);
+		pci_release_regions(pdev);
+		pci_clear_master(pdev);
+		pci_disable_sriov(pdev);
+		pci_disable_device(pdev);
+		ionic_debugfs_del_dev(ionic);
+		mutex_destroy(&ionic->dev_cmd_lock);
+
+		devm_kfree(&pdev->dev, ionic);
+	}
 }

[...]

Quoted 55 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +13 -0
+
+/* Devcmd Interface */
+u8 ionic_dev_cmd_status(struct ionic_dev *idev)
+{
+	return ioread8(&idev->dev_cmd_regs->comp.comp.status);
+}
+
+bool ionic_dev_cmd_done(struct ionic_dev *idev)
+{
+	return ioread32(&idev->dev_cmd_regs->done) & DEV_CMD_DONE;
+}
+
+void ionic_dev_cmd_comp(struct ionic_dev *idev, union dev_cmd_comp

*comp)

drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +40 -0
+{
+	memcpy_fromio(comp, &idev->dev_cmd_regs->comp, sizeof(*comp));
+}
+
+void ionic_dev_cmd_go(struct ionic_dev *idev, union dev_cmd *cmd)
+{
+	memcpy_toio(&idev->dev_cmd_regs->cmd, cmd, sizeof(*cmd));
+	iowrite32(0, &idev->dev_cmd_regs->done);
+	iowrite32(1, &idev->dev_cmd_regs->doorbell);
+}
+
+/* Device commands */
+void ionic_dev_cmd_identify(struct ionic_dev *idev, u8 ver)
+{
+	union dev_cmd cmd = {
+		.identify.opcode = CMD_OPCODE_IDENTIFY,
+		.identify.ver = ver,
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}
+
+void ionic_dev_cmd_init(struct ionic_dev *idev)
+{
+	union dev_cmd cmd = {
+		.init.opcode = CMD_OPCODE_INIT,
+		.init.type = 0,
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}
+
+void ionic_dev_cmd_reset(struct ionic_dev *idev)
+{
+	union dev_cmd cmd = {
+		.reset.opcode = CMD_OPCODE_RESET,
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}

[...]

Quoted 9 lines

+int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_seconds)

drivers/net/ethernet/pensando/ionic/ionic_main.c +7 -0
+{
+	struct ionic_dev *idev = &ionic->idev;
+	unsigned long max_wait, start_time, duration;
+	int opcode;
+	int done;
+	int err;
+

drivers/net/ethernet/pensando/ionic/ionic_main.c line 188

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_main.c +10 -0
+	WARN_ON(in_interrupt());
+
+	/* Wait for dev cmd to complete, retrying if we get EAGAIN,
+	 * but don't wait any longer than max_seconds.
+	 */
+	max_wait = jiffies + (max_seconds * HZ);
+try_again:
+	start_time = jiffies;
+	do {
+		done = ionic_dev_cmd_done(idev);

READ_ONCE required here ? to read from coherent memory modified by the device and read by the driver ?

drivers/net/ethernet/pensando/ionic/ionic_main.c line 192

Quoted 4 lines
drivers/net/ethernet/pensando/ionic/ionic_main.c +4 -0
+		if (done)
+			break;
+		msleep(20);
+	} while (!done && time_before(jiffies, max_wait));

so your command interface is busy polling based, i am relating here to Dave's comment regarding async command completion, is it possible to have interrupt (MSIX?) based command completion in this hw ?

Quoted 43 lines
drivers/net/ethernet/pensando/ionic/ionic_main.c +4 -0
+	duration = jiffies - start_time;
+
+	opcode = idev->dev_cmd_regs->cmd.cmd.opcode;
+	dev_dbg(ionic->dev, "DEVCMD %s (%d) done=%d took %ld secs (%ld

jiffies)\n",

drivers/net/ethernet/pensando/ionic/ionic_main.c +5 -0
+		ionic_opcode_to_str(opcode), opcode,
+		done, duration / HZ, duration);
+
+	if (!done && !time_before(jiffies, max_wait)) {
+		dev_warn(ionic->dev, "DEVCMD %s (%d) timeout after %ld

secs\n",

(quoted diff) +1 -0
+			 ionic_opcode_to_str(opcode), opcode,

max_seconds);

drivers/net/ethernet/pensando/ionic/ionic_main.c +6 -0
+		return -ETIMEDOUT;
+	}
+
+	err = ionic_dev_cmd_status(&ionic->idev);
+	if (err) {
+		if (err == IONIC_RC_EAGAIN && !time_after(jiffies,

max_wait)) {

(quoted diff) +1 -0
+			dev_err(ionic->dev, "DEV_CMD %s (%d) error, %s

(%d) retrying...\n",

drivers/net/ethernet/pensando/ionic/ionic_main.c +10 -0
+				ionic_opcode_to_str(opcode), opcode,
+				ionic_error_to_str(err), err);
+
+			msleep(1000);
+			iowrite32(0, &idev->dev_cmd_regs->done);
+			iowrite32(1, &idev->dev_cmd_regs->doorbell);
+			goto try_again;
+		}
+
+		dev_err(ionic->dev, "DEV_CMD %s (%d) error, %s (%d)

failed\n",

drivers/net/ethernet/pensando/ionic/ionic_main.c +9 -0
+			ionic_opcode_to_str(opcode), opcode,
+			ionic_error_to_str(err), err);
+
+		return ionic_error_to_errno(err);
+	}
+
+	return 0;
+}
+

[...]

#10 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

Quoted 35 lines · Saeed Mahameed
Quoted 28 lines · Shannon Nelson

The ionic device has a small set of PCI registers, including a device control and data space, and a large set of message commands.

---
  drivers/net/ethernet/pensando/ionic/Makefile  |    2 +-
  drivers/net/ethernet/pensando/ionic/ionic.h   |   20 +
  .../net/ethernet/pensando/ionic/ionic_bus.h   |    1 +
  .../ethernet/pensando/ionic/ionic_bus_pci.c   |  140 +-
  .../ethernet/pensando/ionic/ionic_debugfs.c   |   67 +
  .../ethernet/pensando/ionic/ionic_debugfs.h   |   28 +
  .../net/ethernet/pensando/ionic/ionic_dev.c   |  132 +
  .../net/ethernet/pensando/ionic/ionic_dev.h   |  144 +
  .../net/ethernet/pensando/ionic/ionic_if.h    | 2552
+++++++++++++++++
  .../net/ethernet/pensando/ionic/ionic_main.c  |  296 ++
  .../net/ethernet/pensando/ionic/ionic_regs.h  |  133 +
  11 files changed, 3512 insertions(+), 3 deletions(-)

create mode 100644
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c
create mode 100644
drivers/net/ethernet/pensando/ionic/ionic_debugfs.h
create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.c create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_dev.h create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_if.h create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_regs.h

[...]

Quoted 4 lines
  static void ionic_remove(struct pci_dev *pdev)
  {
  	struct ionic *ionic = pci_get_drvdata(pdev);
Quoted 6 lines

drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c static const struct pci_device_id ionic_id_table[] = { · line 164

Quoted 2 lines
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +1 -1
-	devm_kfree(&pdev->dev, ionic);
+	if (ionic) {

nit, in case you are doing another re-spin maybe early return here:
if (!ionic)

      return;

//do stuff

Sure

Quoted 87 lines
Quoted 15 lines
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +12 -0
+		ionic_reset(ionic);
+		ionic_dev_teardown(ionic);
+		ionic_unmap_bars(ionic);
+		pci_release_regions(pdev);
+		pci_clear_master(pdev);
+		pci_disable_sriov(pdev);
+		pci_disable_device(pdev);
+		ionic_debugfs_del_dev(ionic);
+		mutex_destroy(&ionic->dev_cmd_lock);
+
+		devm_kfree(&pdev->dev, ionic);
+	}
  }

[...]

Quoted 55 lines
drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +13 -0
+
+/* Devcmd Interface */
+u8 ionic_dev_cmd_status(struct ionic_dev *idev)
+{
+	return ioread8(&idev->dev_cmd_regs->comp.comp.status);
+}
+
+bool ionic_dev_cmd_done(struct ionic_dev *idev)
+{
+	return ioread32(&idev->dev_cmd_regs->done) & DEV_CMD_DONE;
+}
+
+void ionic_dev_cmd_comp(struct ionic_dev *idev, union dev_cmd_comp

*comp)

drivers/net/ethernet/pensando/ionic/ionic_dev.c new file +40 -0
+{
+	memcpy_fromio(comp, &idev->dev_cmd_regs->comp, sizeof(*comp));
+}
+
+void ionic_dev_cmd_go(struct ionic_dev *idev, union dev_cmd *cmd)
+{
+	memcpy_toio(&idev->dev_cmd_regs->cmd, cmd, sizeof(*cmd));
+	iowrite32(0, &idev->dev_cmd_regs->done);
+	iowrite32(1, &idev->dev_cmd_regs->doorbell);
+}
+
+/* Device commands */
+void ionic_dev_cmd_identify(struct ionic_dev *idev, u8 ver)
+{
+	union dev_cmd cmd = {
+		.identify.opcode = CMD_OPCODE_IDENTIFY,
+		.identify.ver = ver,
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}
+
+void ionic_dev_cmd_init(struct ionic_dev *idev)
+{
+	union dev_cmd cmd = {
+		.init.opcode = CMD_OPCODE_INIT,
+		.init.type = 0,
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}
+
+void ionic_dev_cmd_reset(struct ionic_dev *idev)
+{
+	union dev_cmd cmd = {
+		.reset.opcode = CMD_OPCODE_RESET,
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}

[...]

Quoted 13 lines

+int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_seconds)

drivers/net/ethernet/pensando/ionic/ionic_main.c +11 -0
+{
+	struct ionic_dev *idev = &ionic->idev;
+	unsigned long max_wait, start_time, duration;
+	int opcode;
+	int done;
+	int err;
+
+	WARN_ON(in_interrupt());
+
+	/* Wait for dev cmd to complete, retrying if we get EAGAIN,
+	 * but don't wait any longer than max_seconds.
Quoted 8 lines

drivers/net/ethernet/pensando/ionic/ionic_main.c line 188

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_main.c +6 -0
+	 */
+	max_wait = jiffies + (max_seconds * HZ);
+try_again:
+	start_time = jiffies;
+	do {
+		done = ionic_dev_cmd_done(idev);

READ_ONCE required here ? to read from coherent memory modified by the device and read by the driver ?

Good idea, I'll add that in.

Quoted 8 lines

drivers/net/ethernet/pensando/ionic/ionic_main.c line 192

Quoted 5 lines
drivers/net/ethernet/pensando/ionic/ionic_main.c +4 -0
+		if (done)
+			break;
+		msleep(20);
+	} while (!done && time_before(jiffies, max_wait));

so your command interface is busy polling based, i am relating here to Dave's comment regarding async command completion, is it possible to have interrupt (MSIX?) based command completion in this hw ?

As I wrote elsewhere, this is only the low-level dev_cmd that does polling; the adminq does a wait that is completed by an MSI-x handler.

sln

#11 lore raw

Re: [PATCH v4 net-next 02/19] ionic: Add hardware init and device commands

Quoted 16 lines · Shannon Nelson
Quoted 11 lines · Saeed Mahameed

On Mon, 2019-07-22 at 14:40 -0700, Shannon Nelson wrote:

drivers/net/ethernet/pensando/ionic/ionic_main.c +9 -0
+
+    /* Wait for dev cmd to complete, retrying if we get EAGAIN,
+     * but don't wait any longer than max_seconds.
+     */
+    max_wait = jiffies + (max_seconds * HZ);
+try_again:
+    start_time = jiffies;
+    do {
+        done = ionic_dev_cmd_done(idev);
Quoted 2 lines

READ_ONCE required here ? to read from coherent memory modified by the device and read by the driver ?

Good idea, I'll add that in.

Looking closer at this, it is more for coordinating memory reads between threads and irq handlers.  This is polling a PCI register, which is already marked as volatile and in at least some definitions (e.g. x86) has a barrier.

sln

#12 lore raw

PATCH v4 3/19 ionic: Add port management commands

In reply to Shannon Nelson

The port management commands apply to the physical port associated with the PCI device, which might be shared among several logical interfaces.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |   4 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |  16 +++
 .../net/ethernet/pensando/ionic/ionic_dev.c   | 116 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_dev.h   |  15 +++
 .../net/ethernet/pensando/ionic/ionic_main.c  |  95 ++++++++++++++
 5 files changed, 246 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic.h +4 -0
index 4fd0ae86556a..f77712feef5c 100644
@@ -42,4 +42,8 @@ int ionic_identify(struct ionic *ionic);
 int ionic_init(struct ionic *ionic);
 int ionic_reset(struct ionic *ionic);
 
+int ionic_port_identify(struct ionic *ionic);
+int ionic_port_init(struct ionic *ionic);
+int ionic_port_reset(struct ionic *ionic);
+
 #endif /* _IONIC_H_ */
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +16 -0
index 7eb23f0e87ac..7a41f8995c35 100644
@@ -137,8 +137,23 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		goto err_out_teardown;
 	}
 
+	/* Configure the ports */
+	err = ionic_port_identify(ionic);
+	if (err) {
+		dev_err(dev, "Cannot identify port: %d, aborting\n", err);
+		goto err_out_reset;
+	}
+
+	err = ionic_port_init(ionic);
+	if (err) {
+		dev_err(dev, "Cannot init port: %d, aborting\n", err);
+		goto err_out_reset;
+	}
+
 	return 0;
 
+err_out_reset:
+	ionic_reset(ionic);
 err_out_teardown:
 	ionic_dev_teardown(ionic);
 err_out_unmap_bars:
@@ -162,6 +177,7 @@ static void ionic_remove(struct pci_dev *pdev)
 	struct ionic *ionic = pci_get_drvdata(pdev);
 
 	if (ionic) {
+		ionic_port_reset(ionic);
 		ionic_reset(ionic);
 		ionic_dev_teardown(ionic);
 		ionic_unmap_bars(ionic);
drivers/net/ethernet/pensando/ionic/ionic_dev.c +116 -0

119 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_dev.h +15 -0
index 66afab3ee396..fe5e1b0e8d55 100644
@@ -123,6 +123,10 @@ struct ionic_dev {
 	struct ionic_intr __iomem *intr_ctrl;
 	u64 __iomem *intr_status;
 
+	struct port_info *port_info;
+	dma_addr_t port_info_pa;
+	u32 port_info_sz;
+
 	struct ionic_devinfo dev_info;
 };
 
@@ -141,4 +145,15 @@ void ionic_dev_cmd_identify(struct ionic_dev *idev, u8 ver);
 void ionic_dev_cmd_init(struct ionic_dev *idev);
 void ionic_dev_cmd_reset(struct ionic_dev *idev);
 
+void ionic_dev_cmd_port_identify(struct ionic_dev *idev);
+void ionic_dev_cmd_port_init(struct ionic_dev *idev);
+void ionic_dev_cmd_port_reset(struct ionic_dev *idev);
+void ionic_dev_cmd_port_state(struct ionic_dev *idev, u8 state);
+void ionic_dev_cmd_port_speed(struct ionic_dev *idev, u32 speed);
+void ionic_dev_cmd_port_mtu(struct ionic_dev *idev, u32 mtu);
+void ionic_dev_cmd_port_autoneg(struct ionic_dev *idev, u8 an_enable);
+void ionic_dev_cmd_port_fec(struct ionic_dev *idev, u8 fec_type);
+void ionic_dev_cmd_port_pause(struct ionic_dev *idev, u8 pause_type);
+void ionic_dev_cmd_port_loopback(struct ionic_dev *idev, u8 loopback_mode);
+
 #endif /* _IONIC_DEV_H_ */
drivers/net/ethernet/pensando/ionic/ionic_main.c +95 -0

101 lines not on this page · raw message

Signature · 1 line
2.17.1
#13 lore raw

PATCH v4 4/19 ionic: Add basic lif support

In reply to Shannon Nelson

The LIF is the Logical Interface, which represents the external connections. The NIC can multiplex many LIFs to a single port, but in most setups, LIF0 is the primary control for the port.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   3 +-
 drivers/net/ethernet/pensando/ionic/ionic.h   |   8 +-
 .../net/ethernet/pensando/ionic/ionic_bus.h   |   2 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |  48 +++
 .../ethernet/pensando/ionic/ionic_debugfs.c   |  47 +++
 .../ethernet/pensando/ionic/ionic_debugfs.h   |   6 +
 .../net/ethernet/pensando/ionic/ionic_dev.c   |  34 ++
 .../net/ethernet/pensando/ionic/ionic_dev.h   |   7 +
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 292 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  70 +++++
 .../net/ethernet/pensando/ionic/ionic_main.c  |   1 +
 11 files changed, 516 insertions(+), 2 deletions(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_lif.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_lif.h
drivers/net/ethernet/pensando/ionic/Makefile +2 -1
index 49a8e28a5c84..04d519d00be6 100644
@@ -3,4 +3,5 @@
 
 obj-$(CONFIG_IONIC) := ionic.o
 
-ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_debugfs.o
+ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_debugfs.o \
+	   ionic_lif.o
drivers/net/ethernet/pensando/ionic/ionic.h +7 -1
index f77712feef5c..afff2e074f1a 100644
@@ -6,7 +6,6 @@
 
 #include "ionic_if.h"
 #include "ionic_dev.h"
-#include "ionic_debugfs.h"
 
 #define DRV_NAME		"ionic"
 #define DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
@@ -32,6 +31,13 @@ struct ionic {
 	struct ionic_dev_bar bars[IONIC_BARS_MAX];
 	unsigned int num_bars;
 	struct identity ident;
+	struct list_head lifs;
+	unsigned int nnqs_per_lif;
+	unsigned int neqs_per_lif;
+	unsigned int ntxqs_per_lif;
+	unsigned int nrxqs_per_lif;
+	DECLARE_BITMAP(lifbits, IONIC_LIFS_MAX);
+	unsigned int nintrs;
 };
 
 int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_wait);
drivers/net/ethernet/pensando/ionic/ionic_bus.h +2 -0
index 24b4c01ec03f..3b1e2d0ebf8f 100644
@@ -5,6 +5,8 @@
 #define _IONIC_BUS_H_
 
 const char *ionic_bus_info(struct ionic *ionic);
+int ionic_bus_alloc_irq_vectors(struct ionic *ionic, unsigned int nintrs);
+void ionic_bus_free_irq_vectors(struct ionic *ionic);
 int ionic_bus_register_driver(void);
 void ionic_bus_unregister_driver(void);
 
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +48 -0

74 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +47 -0

61 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_debugfs.h +6 -0
index c3958b0c15b1..cb00166e7c30 100644
@@ -15,6 +15,9 @@ void ionic_debugfs_del_dev(struct ionic *ionic);
 int ionic_debugfs_add_bars(struct ionic *ionic);
 int ionic_debugfs_add_dev_cmd(struct ionic *ionic);
 int ionic_debugfs_add_ident(struct ionic *ionic);
+int ionic_debugfs_add_sizes(struct ionic *ionic);
+int ionic_debugfs_add_lif(struct lif *lif);
+void ionic_debugfs_del_lif(struct lif *lif);
 #else
 static inline void ionic_debugfs_create(void) { }
 static inline void ionic_debugfs_destroy(void) { }
@@ -23,6 +26,9 @@ static inline void ionic_debugfs_del_dev(struct ionic *ionic) { }
 static inline int ionic_debugfs_add_bars(struct ionic *ionic) { return 0; }
 static inline int ionic_debugfs_add_dev_cmd(struct ionic *ionic) { return 0; }
 static inline int ionic_debugfs_add_ident(struct ionic *ionic) { return 0; }
+static inline int ionic_debugfs_add_sizes(struct ionic *ionic) { return 0; }
+static inline int ionic_debugfs_add_lif(struct lif *lif) { return 0; }
+static inline void ionic_debugfs_del_lif(struct lif *lif) { return 0; }
 #endif
 
 #endif /* _IONIC_DEBUGFS_H_ */
drivers/net/ethernet/pensando/ionic/ionic_dev.c +34 -0
index e2eda2bedac6..59d863e45072 100644
@@ -246,3 +246,37 @@ void ionic_dev_cmd_port_loopback(struct ionic_dev *idev, u8 loopback_mode)
 
 	ionic_dev_cmd_go(idev, &cmd);
 }
+
+/* LIF commands */
+void ionic_dev_cmd_lif_identify(struct ionic_dev *idev, u8 type, u8 ver)
+{
+	union dev_cmd cmd = {
+		.lif_identify.opcode = CMD_OPCODE_LIF_IDENTIFY,
+		.lif_identify.type = type,
+		.lif_identify.ver = ver,
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}
+
+void ionic_dev_cmd_lif_init(struct ionic_dev *idev, u16 lif_index,
+			    dma_addr_t info_pa)
+{
+	union dev_cmd cmd = {
+		.lif_init.opcode = CMD_OPCODE_LIF_INIT,
+		.lif_init.index = cpu_to_le16(lif_index),
+		.lif_init.info_pa = cpu_to_le64(info_pa),
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}
+
+void ionic_dev_cmd_lif_reset(struct ionic_dev *idev, u16 lif_index)
+{
+	union dev_cmd cmd = {
+		.lif_init.opcode = CMD_OPCODE_LIF_RESET,
+		.lif_init.index = cpu_to_le16(lif_index),
+	};
+
+	ionic_dev_cmd_go(idev, &cmd);
+}
drivers/net/ethernet/pensando/ionic/ionic_dev.h +7 -0
index fe5e1b0e8d55..a8a84734b433 100644
@@ -10,6 +10,8 @@
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
+#define IONIC_LIFS_MAX			1024
+
 struct ionic_dev_bar {
 	void __iomem *vaddr;
 	phys_addr_t bus_addr;
@@ -156,4 +158,9 @@ void ionic_dev_cmd_port_fec(struct ionic_dev *idev, u8 fec_type);
 void ionic_dev_cmd_port_pause(struct ionic_dev *idev, u8 pause_type);
 void ionic_dev_cmd_port_loopback(struct ionic_dev *idev, u8 loopback_mode);
 
+void ionic_dev_cmd_lif_identify(struct ionic_dev *idev, u8 type, u8 ver);
+void ionic_dev_cmd_lif_init(struct ionic_dev *idev, u16 lif_index,
+			    dma_addr_t addr);
+void ionic_dev_cmd_lif_reset(struct ionic_dev *idev, u16 lif_index);
+
 #endif /* _IONIC_DEV_H_ */
drivers/net/ethernet/pensando/ionic/ionic_lif.c new file +292 -0

292 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h new file +70 -0

70 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_main.c +1 -0
index 78eab326b067..b732e7345550 100644
@@ -8,6 +8,7 @@
 
 #include "ionic.h"
 #include "ionic_bus.h"
+#include "ionic_lif.h"
 #include "ionic_debugfs.h"
 
 MODULE_DESCRIPTION(DRV_DESCRIPTION);
Signature · 1 line
2.17.1
#14 lore raw

PATCH v4 5/19 ionic: Add interrupts and doorbells

In reply to Shannon Nelson

The ionic interrupt model is based on interrupt control blocks accessed through the PCI BAR. Doorbell registers are used by the driver to signal to the NIC that requests are waiting on the message queues. Interrupts are used by the NIC to signal to the driver that answers are waiting on the completion queues.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |  3 +
 .../net/ethernet/pensando/ionic/ionic_bus.h   |  2 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   | 12 ++++
 .../net/ethernet/pensando/ionic/ionic_dev.c   |  6 ++
 .../net/ethernet/pensando/ionic/ionic_dev.h   | 21 ++++++
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 64 +++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  4 ++
 7 files changed, 112 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic.h +3 -0
index afff2e074f1a..c86a08752b3b 100644
@@ -4,6 +4,8 @@
 #ifndef _IONIC_H_
 #define _IONIC_H_
 
+struct lif;
+
 #include "ionic_if.h"
 #include "ionic_dev.h"
 
@@ -38,6 +40,7 @@ struct ionic {
 	unsigned int nrxqs_per_lif;
 	DECLARE_BITMAP(lifbits, IONIC_LIFS_MAX);
 	unsigned int nintrs;
+	DECLARE_BITMAP(intrs, INTR_CTRL_REGS_MAX);
 };
 
 int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_wait);
drivers/net/ethernet/pensando/ionic/ionic_bus.h +2 -0
index 3b1e2d0ebf8f..6b29e94f81d6 100644
@@ -9,5 +9,7 @@ int ionic_bus_alloc_irq_vectors(struct ionic *ionic, unsigned int nintrs);
 void ionic_bus_free_irq_vectors(struct ionic *ionic);
 int ionic_bus_register_driver(void);
 void ionic_bus_unregister_driver(void);
+void __iomem *ionic_bus_map_dbpage(struct ionic *ionic, int page_num);
+void ionic_bus_unmap_dbpage(struct ionic *ionic, void __iomem *page);
 
 #endif /* _IONIC_BUS_H_ */
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +12 -0
index 0ed724efe117..838d7c423b2c 100644
@@ -87,6 +87,18 @@ static void ionic_unmap_bars(struct ionic *ionic)
 	}
 }
 
+void __iomem *ionic_bus_map_dbpage(struct ionic *ionic, int page_num)
+{
+	return pci_iomap_range(ionic->pdev,
+			       ionic->bars[IONIC_PCI_BAR_DBELL].res_index,
+			       (u64)page_num << PAGE_SHIFT, PAGE_SIZE);
+}
+
+void ionic_bus_unmap_dbpage(struct ionic *ionic, void __iomem *page)
+{
+	iounmap(page);
+}
+
 static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 {
 	struct device *dev = &pdev->dev;
drivers/net/ethernet/pensando/ionic/ionic_dev.c +6 -0
index 59d863e45072..c0710bf200a5 100644
@@ -9,6 +9,7 @@
 #include <linux/etherdevice.h>
 #include "ionic.h"
 #include "ionic_dev.h"
+#include "ionic_lif.h"
 
 void ionic_init_devinfo(struct ionic_dev *idev)
 {
@@ -280,3 +281,8 @@ void ionic_dev_cmd_lif_reset(struct ionic_dev *idev, u16 lif_index)
 
 	ionic_dev_cmd_go(idev, &cmd);
 }
+
+int ionic_db_page_num(struct lif *lif, int pid)
+{
+	return (lif->hw_index * lif->dbid_count) + pid;
+}
drivers/net/ethernet/pensando/ionic/ionic_dev.h +21 -0
index a8a84734b433..6d30adeab8c5 100644
@@ -132,8 +132,27 @@ struct ionic_dev {
 	struct ionic_devinfo dev_info;
 };
 
+#define INTR_INDEX_NOT_ASSIGNED		-1
+#define INTR_NAME_MAX_SZ		32
+
+struct intr {
+	char name[INTR_NAME_MAX_SZ];
+	unsigned int index;
+	unsigned int vector;
+	u64 rearm_count;
+	unsigned int cpu;
+	cpumask_t affinity_mask;
+};
+
 struct ionic;
 
+static inline void ionic_intr_init(struct ionic_dev *idev, struct intr *intr,
+				   unsigned long index)
+{
+	ionic_intr_clean(idev->intr_ctrl, index);
+	intr->index = index;
+}
+
 void ionic_init_devinfo(struct ionic_dev *idev);
 int ionic_dev_setup(struct ionic *ionic);
 void ionic_dev_teardown(struct ionic *ionic);
@@ -163,4 +182,6 @@ void ionic_dev_cmd_lif_init(struct ionic_dev *idev, u16 lif_index,
 			    dma_addr_t addr);
 void ionic_dev_cmd_lif_reset(struct ionic_dev *idev, u16 lif_index);
 
+int ionic_db_page_num(struct lif *lif, int pid);
+
 #endif /* _IONIC_DEV_H_ */
drivers/net/ethernet/pensando/ionic/ionic_lif.c +64 -0

92 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h +4 -0
index 37f79d36744a..98d0699234e9 100644
@@ -23,6 +23,8 @@ struct lif {
 	bool registered;
 	unsigned int index;
 	unsigned int hw_index;
+	unsigned int kern_pid;
+	u64 __iomem *kern_dbpage;
 	unsigned int neqs;
 	unsigned int nxqs;
 
@@ -30,6 +32,8 @@ struct lif {
 	dma_addr_t info_pa;
 	u32 info_sz;
 
+	unsigned long *dbid_inuse;
+	unsigned int dbid_count;
 	struct dentry *dentry;
 	u32 flags;
 };
Signature · 1 line
2.17.1
maintainer
#15 lore raw

Re: [PATCH v4 net-next 05/19] ionic: Add interrupts and doorbells

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:09 -0700

Quoted 7 lines

The ionic interrupt model is based on interrupt control blocks accessed through the PCI BAR. Doorbell registers are used by the driver to signal to the NIC that requests are waiting on the message queues. Interrupts are used by the NIC to signal to the driver that answers are waiting on the completion queues.

After applying this patch we get a warning:

drivers/net/ethernet/pensando/ionic/ionic_lif.c:33:13: warning: ‘ionic_intr_free’ defined but not used [-Wunused-function]
static void ionic_intr_free(struct lif *lif, int index)

             ^~~~~~~~~~~~~~~
drivers/net/ethernet/pensando/ionic/ionic_lif.c:15:12: warning: ‘ionic_intr_alloc’ defined but not used [-Wunused-function]
 static int ionic_intr_alloc(struct lif *lif, struct intr *intr)
            ^~~~~~~~~~~~~~~~

Also:

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_lif_init(struct lif *lif) · line 196

Quoted 2 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+	lif->dbid_inuse = kzalloc(BITS_TO_LONGS(lif->dbid_count) * sizeof(long),
+				  GFP_KERNEL);

You can use bitmap_alloc() and friends from linux/bitmap.h for this kind of stuff.

#16 lore raw

Re: [PATCH v4 net-next 05/19] ionic: Add interrupts and doorbells

Quoted 10 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:09 -0700

Quoted 7 lines

The ionic interrupt model is based on interrupt control blocks accessed through the PCI BAR. Doorbell registers are used by the driver to signal to the NIC that requests are waiting on the message queues. Interrupts are used by the NIC to signal to the driver that answers are waiting on the completion queues.

Quoted 8 lines

After applying this patch we get a warning:

drivers/net/ethernet/pensando/ionic/ionic_lif.c:33:13: warning: ‘ionic_intr_free’ defined but not used [-Wunused-function]
static void ionic_intr_free(struct lif *lif, int index)

              ^~~~~~~~~~~~~~~
drivers/net/ethernet/pensando/ionic/ionic_lif.c:15:12: warning: ‘ionic_intr_alloc’ defined but not used [-Wunused-function]
  static int ionic_intr_alloc(struct lif *lif, struct intr *intr)
             ^~~~~~~~~~~~~~~~

Yes, and they get used in the next patch.  This is so I can keep similar bits of functionality together in one patch and to keep the next patch a little smaller.

Quoted 5 lines

Also:

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_lif_init(struct lif *lif) · line 196

Quoted 2 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+	lif->dbid_inuse = kzalloc(BITS_TO_LONGS(lif->dbid_count) * sizeof(long),
+				  GFP_KERNEL);

You can use bitmap_alloc() and friends from linux/bitmap.h for this kind of stuff.

Sure.
sln

#17 lore raw

PATCH v4 6/19 ionic: Add basic adminq support

In reply to Shannon Nelson

Most of the NIC configuration happens through the AdminQ message queue. NAPI is used for basic interrupt handling and message queue management. These routines are set up to be shared among different types of queues when used in slow-path handling.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |   3 +
 .../net/ethernet/pensando/ionic/ionic_bus.h   |   1 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |   5 +
 .../ethernet/pensando/ionic/ionic_debugfs.c   | 149 ++++++++
 .../ethernet/pensando/ionic/ionic_debugfs.h   |   4 +
 .../net/ethernet/pensando/ionic/ionic_dev.c   | 247 +++++++++++++
 .../net/ethernet/pensando/ionic/ionic_dev.h   |  92 +++++
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 328 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  72 ++++
 .../net/ethernet/pensando/ionic/ionic_main.c  |  26 ++
 10 files changed, 927 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic.h +3 -0
index c86a08752b3b..c8c93e9efb39 100644
@@ -43,6 +43,9 @@ struct ionic {
 	DECLARE_BITMAP(intrs, INTR_CTRL_REGS_MAX);
 };
 
+int ionic_napi(struct napi_struct *napi, int budget, ionic_cq_cb cb,
+	       ionic_cq_done_cb done_cb, void *done_arg);
+
 int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_wait);
 int ionic_set_dma_mask(struct ionic *ionic);
 int ionic_setup(struct ionic *ionic);
drivers/net/ethernet/pensando/ionic/ionic_bus.h +1 -0
index 6b29e94f81d6..2f4d08c64910 100644
@@ -4,6 +4,7 @@
 #ifndef _IONIC_BUS_H_
 #define _IONIC_BUS_H_
 
+int ionic_bus_get_irq(struct ionic *ionic, unsigned int num);
 const char *ionic_bus_info(struct ionic *ionic);
 int ionic_bus_alloc_irq_vectors(struct ionic *ionic, unsigned int nintrs);
 void ionic_bus_free_irq_vectors(struct ionic *ionic);
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +5 -0
index 838d7c423b2c..59d1ae7ce532 100644
@@ -19,6 +19,11 @@ static const struct pci_device_id ionic_id_table[] = {
 };
 MODULE_DEVICE_TABLE(pci, ionic_id_table);
 
+int ionic_bus_get_irq(struct ionic *ionic, unsigned int num)
+{
+	return pci_irq_vector(ionic->pdev, num);
+}
+
 const char *ionic_bus_info(struct ionic *ionic)
 {
 	return pci_name(ionic->pdev);
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +149 -0

159 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_debugfs.h +4 -0
index cb00166e7c30..fa5030d48763 100644
@@ -17,7 +17,9 @@ int ionic_debugfs_add_dev_cmd(struct ionic *ionic);
 int ionic_debugfs_add_ident(struct ionic *ionic);
 int ionic_debugfs_add_sizes(struct ionic *ionic);
 int ionic_debugfs_add_lif(struct lif *lif);
+int ionic_debugfs_add_qcq(struct lif *lif, struct qcq *qcq);
 void ionic_debugfs_del_lif(struct lif *lif);
+void ionic_debugfs_del_qcq(struct qcq *qcq);
 #else
 static inline void ionic_debugfs_create(void) { }
 static inline void ionic_debugfs_destroy(void) { }
@@ -28,7 +30,9 @@ static inline int ionic_debugfs_add_dev_cmd(struct ionic *ionic) { return 0; }
 static inline int ionic_debugfs_add_ident(struct ionic *ionic) { return 0; }
 static inline int ionic_debugfs_add_sizes(struct ionic *ionic) { return 0; }
 static inline int ionic_debugfs_add_lif(struct lif *lif) { return 0; }
+static inline int ionic_debugfs_add_qcq(struct lif *lif, struct qcq *qcq) { return 0; }
 static inline void ionic_debugfs_del_lif(struct lif *lif) { return 0; }
+static inline void ionic_debugfs_del_qcq(struct qcq *qcq) { return 0; }
 #endif
 
 #endif /* _IONIC_DEBUGFS_H_ */
drivers/net/ethernet/pensando/ionic/ionic_dev.c +247 -0

254 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_dev.h +92 -0

111 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.c +328 -0

390 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h +72 -0

88 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_main.c +26 -0
index b732e7345550..aa73ea81a4b2 100644
@@ -169,6 +169,32 @@ static const char *ionic_opcode_to_str(enum cmd_opcode opcode)
 	}
 }
 
+int ionic_napi(struct napi_struct *napi, int budget, ionic_cq_cb cb,
+	       ionic_cq_done_cb done_cb, void *done_arg)
+{
+	struct qcq *qcq = napi_to_qcq(napi);
+	struct cq *cq = &qcq->cq;
+	u32 work_done, flags = 0;
+
+	work_done = ionic_cq_service(cq, budget, cb, done_cb, done_arg);
+
+	if (work_done < budget && napi_complete_done(napi, work_done)) {
+		flags |= IONIC_INTR_CRED_UNMASK;
+		DEBUG_STATS_INTR_REARM(cq->bound_intr);
+	}
+
+	if (work_done || flags) {
+		flags |= IONIC_INTR_CRED_RESET_COALESCE;
+		ionic_intr_credits(cq->lif->ionic->idev.intr_ctrl,
+				   cq->bound_intr->index,
+				   work_done, flags);
+	}
+
+	DEBUG_STATS_NAPI_POLL(qcq, work_done);
+
+	return work_done;
+}
+
 int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_seconds)
 {
 	struct ionic_dev *idev = &ionic->idev;
Signature · 1 line
2.17.1
maintainer
#18 lore raw

Re: [PATCH v4 net-next 06/19] ionic: Add basic adminq support

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:10 -0700

Quoted 1 line

+struct queue {

...

Quoted 1 line

+struct cq {

...

Quoted 1 line

+struct napi_stats {

...

Quoted 1 line

+struct q_stats {

...

Quoted 1 line

+struct qcq {

Using names like these and "dev_queue" are just asking for conflicts with the global datastructure namespace both now and in the future.

Please put ionic_ or similar as a prefix to these data structure names.

Thank you.

#19 lore raw

Re: [PATCH v4 net-next 06/19] ionic: Add basic adminq support

Quoted 10 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:10 -0700

Quoted 1 line

+struct queue {

...

Quoted 1 line

+struct cq {

...

Quoted 1 line

+struct napi_stats {

...

Quoted 1 line

+struct q_stats {

Quoted 8 lines

...

Quoted 1 line

+struct qcq {

Using names like these and "dev_queue" are just asking for conflicts with the global datastructure namespace both now and in the future.

Please put ionic_ or similar as a prefix to these data structure names.

Thank you.

Sure
sln

#20 lore raw

PATCH v4 7/19 ionic: Add adminq action

In reply to Shannon Nelson

Add AdminQ specific message requests and completion handling.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |   7 ++
 .../net/ethernet/pensando/ionic/ionic_main.c  | 106 ++++++++++++++++++
 2 files changed, 113 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic.h +7 -0
index c8c93e9efb39..87ab13aee89e 100644
@@ -43,9 +43,16 @@ struct ionic {
 	DECLARE_BITMAP(intrs, INTR_CTRL_REGS_MAX);
 };
 
+struct ionic_admin_ctx {
+	struct completion work;
+	union adminq_cmd cmd;
+	union adminq_comp comp;
+};
+
 int ionic_napi(struct napi_struct *napi, int budget, ionic_cq_cb cb,
 	       ionic_cq_done_cb done_cb, void *done_arg);
 
+int ionic_adminq_post_wait(struct lif *lif, struct ionic_admin_ctx *ctx);
 int ionic_dev_cmd_wait(struct ionic *ionic, unsigned long max_wait);
 int ionic_set_dma_mask(struct ionic *ionic);
 int ionic_setup(struct ionic *ionic);
drivers/net/ethernet/pensando/ionic/ionic_main.c +106 -0

112 lines not on this page · raw message

Signature · 1 line
2.17.1
#21 lore raw

PATCH v4 8/19 ionic: Add notifyq support

In reply to Shannon Nelson

The AdminQ is fine for sending messages and requests to the NIC, but we also need to have events published from the NIC to the driver. The NotifyQ handles this for us, using the same interrupt as AdminQ.

---
 .../ethernet/pensando/ionic/ionic_debugfs.c   |  16 ++
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 181 +++++++++++++++++-
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   4 +
 3 files changed, 200 insertions(+), 1 deletion(-)
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +16 -0
index 9af15c69b2a6..1d05b23de303 100644
@@ -126,6 +126,7 @@ int ionic_debugfs_add_qcq(struct lif *lif, struct qcq *qcq)
 	struct debugfs_blob_wrapper *desc_blob;
 	struct device *dev = lif->ionic->dev;
 	struct intr *intr = &qcq->intr;
+	struct dentry *stats_dentry;
 	struct queue *q = &qcq->q;
 	struct cq *cq = &qcq->cq;
 
@@ -219,6 +220,21 @@ int ionic_debugfs_add_qcq(struct lif *lif, struct qcq *qcq)
 					intr_ctrl_regset);
 	}
 
+	if (qcq->flags & QCQ_F_NOTIFYQ) {
+		stats_dentry = debugfs_create_dir("notifyblock", qcq_dentry);
+		if (IS_ERR_OR_NULL(stats_dentry))
+			return PTR_ERR(stats_dentry);
+
+		debugfs_create_u64("eid", 0400, stats_dentry,
+				   (u64 *)&lif->info->status.eid);
+		debugfs_create_u16("link_status", 0400, stats_dentry,
+				   (u16 *)&lif->info->status.link_status);
+		debugfs_create_u32("link_speed", 0400, stats_dentry,
+				   (u32 *)&lif->info->status.link_speed);
+		debugfs_create_u16("link_down_count", 0400, stats_dentry,
+				   (u16 *)&lif->info->status.link_down_count);
+	}
+
 	return 0;
 }
 
drivers/net/ethernet/pensando/ionic/ionic_lif.c +180 -1

232 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h +4 -0
index 28ab92b43a64..80eec0778f40 100644
@@ -7,6 +7,7 @@
 #include <linux/pci.h>
 
 #define IONIC_ADMINQ_LENGTH	16	/* must be a power of two */
+#define IONIC_NOTIFYQ_LENGTH	64	/* must be a power of two */
 
 #define GET_NAPI_CNTR_IDX(work_done)	(work_done)
 #define MAX_NUM_NAPI_CNTR	(NAPI_POLL_WEIGHT + 1)
@@ -26,6 +27,7 @@ struct rx_stats {
 #define QCQ_F_INITED		BIT(0)
 #define QCQ_F_SG		BIT(1)
 #define QCQ_F_INTR		BIT(2)
+#define QCQ_F_NOTIFYQ		BIT(5)
 
 struct napi_stats {
 	u64 poll_count;
@@ -78,6 +80,8 @@ struct lif {
 	u64 __iomem *kern_dbpage;
 	spinlock_t adminq_lock;		/* lock for AdminQ operations */
 	struct qcq *adminqcq;
+	struct qcq *notifyqcq;
+	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
 
Signature · 1 line
2.17.1
#22 lore raw

Re: [PATCH v4 net-next 08/19] ionic: Add notifyq support

Quoted 50 lines · Shannon Nelson

The AdminQ is fine for sending messages and requests to the NIC, but we also need to have events published from the NIC to the driver. The NotifyQ handles this for us, using the same interrupt as AdminQ.

---
 .../ethernet/pensando/ionic/ionic_debugfs.c   |  16 ++
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 181
+++++++++++++++++-
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   4 +
 3 files changed, 200 insertions(+), 1 deletion(-)
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +12 -0
index 9af15c69b2a6..1d05b23de303 100644
@@ -126,6 +126,7 @@ int ionic_debugfs_add_qcq(struct lif *lif, struct
 qcq *qcq)
 	struct debugfs_blob_wrapper *desc_blob;
 	struct device *dev = lif->ionic->dev;
 	struct intr *intr = &qcq->intr;
+	struct dentry *stats_dentry;
 	struct queue *q = &qcq->q;
 	struct cq *cq = &qcq->cq;
@@ -219,6 +220,21 @@ int ionic_debugfs_add_qcq(struct lif *lif,
 struct qcq *qcq)
 					intr_ctrl_regset);
 	}
 
+	if (qcq->flags & QCQ_F_NOTIFYQ) {
+		stats_dentry = debugfs_create_dir("notifyblock",
 qcq_dentry);
+		if (IS_ERR_OR_NULL(stats_dentry))
+			return PTR_ERR(stats_dentry);
+
+		debugfs_create_u64("eid", 0400, stats_dentry,
+				   (u64 *)&lif->info->status.eid);
+		debugfs_create_u16("link_status", 0400, stats_dentry,
+				   (u16 *)&lif->info-
 >status.link_status);
+		debugfs_create_u32("link_speed", 0400, stats_dentry,
+				   (u32 *)&lif->info-
Quoted 1 line

status.link_speed);

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +1 -0
+		debugfs_create_u16("link_down_count", 0400,

stats_dentry,

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +1 -0
+				   (u16 *)&lif->info-
Quoted 1 line

status.link_down_count);

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c

Quoted 2 lines
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +2 -0
+	}
+

you never write to these lif->info->status.xyz ..
and link state and speed are/should be available in "ethtool <ifname>" so this looks redundant to me. you can also use ethtool -S to report linkdown count.

Quoted 86 lines
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +0 -0
 	return 0;
 }
drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
index 19c046502a26..01f9665611d4 100644
@@ -12,6 +12,8 @@
 #include "ionic_lif.h"
 #include "ionic_debugfs.h"
 
+static int ionic_notifyq_clean(struct lif *lif, int budget);
+
 static bool ionic_adminq_service(struct cq *cq, struct cq_info
 *cq_info)
 {
drivers/net/ethernet/pensando/ionic/ionic_lif.c +7 -1
 	struct admin_comp *comp = cq_info->cq_desc;
@@ -26,7 +28,90 @@ static bool ionic_adminq_service(struct cq *cq,
 struct cq_info *cq_info)
 
 static int ionic_adminq_napi(struct napi_struct *napi, int budget)
 {
-	return ionic_napi(napi, budget, ionic_adminq_service, NULL,
 NULL);
+	struct lif *lif = napi_to_cq(napi)->lif;
+	int n_work = 0;
+	int a_work = 0;
+
+	if (likely(lif->notifyqcq && lif->notifyqcq->flags &
 QCQ_F_INITED))
+		n_work = ionic_notifyq_clean(lif, budget);
+	a_work = ionic_napi(napi, budget, ionic_adminq_service, NULL,

NULL);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +5 -0
+
+	return max(n_work, a_work);
+}
+
+static bool ionic_notifyq_service(struct cq *cq, struct cq_info

*cq_info)

drivers/net/ethernet/pensando/ionic/ionic_lif.c +25 -0
+{
+	union notifyq_comp *comp = cq_info->cq_desc;
+	struct net_device *netdev;
+	struct queue *q;
+	struct lif *lif;
+	u64 eid;
+
+	q = cq->bound_q;
+	lif = q->info[0].cb_arg;
+	netdev = lif->netdev;
+	eid = le64_to_cpu(comp->event.eid);
+
+	/* Have we run out of new completions to process? */
+	if (eid <= lif->last_eid)
+		return false;
+
+	lif->last_eid = eid;
+
+	dev_dbg(lif->ionic->dev, "notifyq event:\n");
+	dynamic_hex_dump("event ", DUMP_PREFIX_OFFSET, 16, 1,
+			 comp, sizeof(*comp), true);
+
+	switch (le16_to_cpu(comp->event.ecode)) {
+	case EVENT_OPCODE_LINK_CHANGE:
+		netdev_info(netdev, "Notifyq EVENT_OPCODE_LINK_CHANGE

eid=%lld\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c +8 -0
+			    eid);
+		netdev_info(netdev,
+			    "  link_status=%d link_speed=%d\n",
+			    le16_to_cpu(comp->link_change.link_status),
+			    le32_to_cpu(comp->link_change.link_speed));
+		break;
+	case EVENT_OPCODE_RESET:
+		netdev_info(netdev, "Notifyq EVENT_OPCODE_RESET

eid=%lld\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c +7 -0
+			    eid);
+		netdev_info(netdev, "  reset_code=%d state=%d\n",
+			    comp->reset.reset_code,
+			    comp->reset.state);
+		break;
+	case EVENT_OPCODE_LOG:
+		netdev_info(netdev, "Notifyq EVENT_OPCODE_LOG

eid=%lld\n", eid);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+		print_hex_dump(KERN_INFO, "notifyq ",

DUMP_PREFIX_OFFSET, 16, 1,

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+			       comp->log.data, sizeof(comp->log.data),
Quoted 1 line

true);

So your device can generate log buffer dump into the kernel log .. I am not sure how acceptable this is, maybe trace buffer is more appropriate for this.

Quoted 214 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
+		break;
+	default:
+		netdev_warn(netdev, "Notifyq unknown event ecode=%d

eid=%lld\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c +17 -0
+			    comp->event.ecode, eid);
+		break;
+	}
+
+	return true;
+}
+
+static int ionic_notifyq_clean(struct lif *lif, int budget)
+{
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct cq *cq = &lif->notifyqcq->cq;
+	u32 work_done;
+
+	work_done = ionic_cq_service(cq, budget, ionic_notifyq_service,
+				     NULL, NULL);
+	if (work_done)
+		ionic_intr_credits(idev->intr_ctrl, cq->bound_intr-
Quoted 1 line

index,

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+				   work_done,

IONIC_INTR_CRED_RESET_COALESCE);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +20 -0
+
+	/* If we ran out of budget, there are more events
+	 * to process and napi will reschedule us soon
+	 */
+	if (work_done == budget)
+		goto return_to_napi;
+
+return_to_napi:
+	return work_done;
 }
 
 static irqreturn_t ionic_isr(int irq, void *data)
@@ -62,6 +147,17 @@ static void ionic_intr_free(struct lif *lif, int
 index)
 		clear_bit(index, lif->ionic->intrs);
 }
 
+static void ionic_link_qcq_interrupts(struct qcq *src_qcq, struct
 qcq *n_qcq)
+{
+	if (WARN_ON(n_qcq->flags & QCQ_F_INTR)) {
+		ionic_intr_free(n_qcq->cq.lif, n_qcq->intr.index);
+		n_qcq->flags &= ~QCQ_F_INTR;
+	}
+
+	n_qcq->intr.vector = src_qcq->intr.vector;
+	n_qcq->intr.index = src_qcq->intr.index;
+}
+
 static int ionic_qcq_alloc(struct lif *lif, unsigned int type,
drivers/net/ethernet/pensando/ionic/ionic_lif.c +25 -0
 			   unsigned int index,
 			   const char *name, unsigned int flags,
@@ -236,11 +332,36 @@ static int ionic_qcqs_alloc(struct lif *lif)
 	if (err)
 		return err;
 
+	if (lif->ionic->nnqs_per_lif) {
+		flags = QCQ_F_NOTIFYQ;
+		err = ionic_qcq_alloc(lif, IONIC_QTYPE_NOTIFYQ, 0,
 "notifyq",
+				      flags, IONIC_NOTIFYQ_LENGTH,
+				      sizeof(struct notifyq_cmd),
+				      sizeof(union notifyq_comp),
+				      0, lif->kern_pid, &lif-
 >notifyqcq);
+		if (err)
+			goto err_out_free_adminqcq;
+
+		/* Let the notifyq ride on the adminq interrupt */
+		ionic_link_qcq_interrupts(lif->adminqcq, lif-
 >notifyqcq);
+	}
+
 	return 0;
+
+err_out_free_adminqcq:
+	ionic_qcq_free(lif, lif->adminqcq);
+	lif->adminqcq = NULL;
+
+	return err;
 }
 
 static void ionic_qcqs_free(struct lif *lif)
 {
+	if (lif->notifyqcq) {
+		ionic_qcq_free(lif, lif->notifyqcq);
+		lif->notifyqcq = NULL;
+	}
+
drivers/net/ethernet/pensando/ionic/ionic_lif.c +28 -0
 	if (lif->adminqcq) {
 		ionic_qcq_free(lif, lif->adminqcq);
 		lif->adminqcq = NULL;
@@ -400,6 +521,7 @@ static void ionic_lif_deinit(struct lif *lif)
 	clear_bit(LIF_INITED, lif->state);
 
 	napi_disable(&lif->adminqcq->napi);
+	ionic_lif_qcq_deinit(lif, lif->notifyqcq);
 	ionic_lif_qcq_deinit(lif, lif->adminqcq);
 
 	ionic_lif_reset(lif);
@@ -484,6 +606,55 @@ static int ionic_lif_adminq_init(struct lif
 *lif)
 	return 0;
 }
 
+static int ionic_lif_notifyq_init(struct lif *lif)
+{
+	struct device *dev = lif->ionic->dev;
+	struct qcq *qcq = lif->notifyqcq;
+	struct queue *q = &qcq->q;
+	int err;
+
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.q_init = {
+			.opcode = CMD_OPCODE_Q_INIT,
+			.lif_index = cpu_to_le16(lif->index),
+			.type = q->type,
+			.index = cpu_to_le32(q->index),
+			.flags = cpu_to_le16(IONIC_QINIT_F_IRQ |
+					     IONIC_QINIT_F_ENA),
+			.intr_index = cpu_to_le16(lif->adminqcq-
 >intr.index),
+			.pid = cpu_to_le16(q->pid),
+			.ring_size = ilog2(q->num_descs),
+			.ring_base = cpu_to_le64(q->base_pa),
+		}
+	};
+
+	dev_dbg(dev, "notifyq_init.pid %d\n", ctx.cmd.q_init.pid);
+	dev_dbg(dev, "notifyq_init.index %d\n", ctx.cmd.q_init.index);
+	dev_dbg(dev, "notifyq_init.ring_base 0x%llx\n",
 ctx.cmd.q_init.ring_base);
+	dev_dbg(dev, "notifyq_init.ring_size %d\n",

ctx.cmd.q_init.ring_size);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +30 -0
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	q->hw_type = ctx.comp.q_init.hw_type;
+	q->hw_index = le32_to_cpu(ctx.comp.q_init.hw_index);
+	q->dbval = IONIC_DBELL_QID(q->hw_index);
+
+	dev_dbg(dev, "notifyq->hw_type %d\n", q->hw_type);
+	dev_dbg(dev, "notifyq->hw_index %d\n", q->hw_index);
+
+	/* preset the callback info */
+	q->info[0].cb_arg = lif;
+
+	qcq->flags |= QCQ_F_INITED;
+
+	ionic_debugfs_add_qcq(lif, qcq);
+
+	return 0;
+}
+
 static int ionic_lif_init(struct lif *lif)
 {
 	struct ionic_dev *idev = &lif->ionic->idev;
@@ -534,10 +705,18 @@ static int ionic_lif_init(struct lif *lif)
 	if (err)
 		goto err_out_adminq_deinit;
 
+	if (lif->ionic->nnqs_per_lif) {
+		err = ionic_lif_notifyq_init(lif);
+		if (err)
+			goto err_out_notifyq_deinit;
+	}
+
 	set_bit(LIF_INITED, lif->state);
 
 	return 0;
 
+err_out_notifyq_deinit:
+	ionic_lif_qcq_deinit(lif, lif->notifyqcq);
 err_out_adminq_deinit:
 	ionic_lif_qcq_deinit(lif, lif->adminqcq);
 	ionic_lif_reset(lif);
drivers/net/ethernet/pensando/ionic/ionic_lif.h +4 -0
index 28ab92b43a64..80eec0778f40 100644
@@ -7,6 +7,7 @@
 #include <linux/pci.h>
 
 #define IONIC_ADMINQ_LENGTH	16	/* must be a power of two */
+#define IONIC_NOTIFYQ_LENGTH	64	/* must be a power of two */
 
 #define GET_NAPI_CNTR_IDX(work_done)	(work_done)
 #define MAX_NUM_NAPI_CNTR	(NAPI_POLL_WEIGHT + 1)
@@ -26,6 +27,7 @@ struct rx_stats {
 #define QCQ_F_INITED		BIT(0)
 #define QCQ_F_SG		BIT(1)
 #define QCQ_F_INTR		BIT(2)
+#define QCQ_F_NOTIFYQ		BIT(5)
 
 struct napi_stats {
 	u64 poll_count;
@@ -78,6 +80,8 @@ struct lif {
 	u64 __iomem *kern_dbpage;
 	spinlock_t adminq_lock;		/* lock for AdminQ operations
 */
 	struct qcq *adminqcq;
+	struct qcq *notifyqcq;
+	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
#23 lore raw

Re: [PATCH v4 net-next 08/19] ionic: Add notifyq support

Quoted 46 lines · Saeed Mahameed
Quoted 45 lines · Shannon Nelson

The AdminQ is fine for sending messages and requests to the NIC, but we also need to have events published from the NIC to the driver. The NotifyQ handles this for us, using the same interrupt as AdminQ.

---
  .../ethernet/pensando/ionic/ionic_debugfs.c   |  16 ++
  .../net/ethernet/pensando/ionic/ionic_lif.c   | 181
+++++++++++++++++-
  .../net/ethernet/pensando/ionic/ionic_lif.h   |   4 +
  3 files changed, 200 insertions(+), 1 deletion(-)
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +12 -0
index 9af15c69b2a6..1d05b23de303 100644
@@ -126,6 +126,7 @@ int ionic_debugfs_add_qcq(struct lif *lif, struct
 qcq *qcq)
  	struct debugfs_blob_wrapper *desc_blob;
  	struct device *dev = lif->ionic->dev;
  	struct intr *intr = &qcq->intr;
+	struct dentry *stats_dentry;
  	struct queue *q = &qcq->q;
  	struct cq *cq = &qcq->cq;
@@ -219,6 +220,21 @@ int ionic_debugfs_add_qcq(struct lif *lif,
 struct qcq *qcq)
  					intr_ctrl_regset);
  	}
 
+	if (qcq->flags & QCQ_F_NOTIFYQ) {
+		stats_dentry = debugfs_create_dir("notifyblock",
 qcq_dentry);
+		if (IS_ERR_OR_NULL(stats_dentry))
+			return PTR_ERR(stats_dentry);
+
+		debugfs_create_u64("eid", 0400, stats_dentry,
+				   (u64 *)&lif->info->status.eid);
+		debugfs_create_u16("link_status", 0400, stats_dentry,
+				   (u16 *)&lif->info-
 > status.link_status);
+		debugfs_create_u32("link_speed", 0400, stats_dentry,
+				   (u32 *)&lif->info-
Quoted 8 lines

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c

Quoted 7 lines
Quoted 1 line

status.link_speed);

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +1 -0
+		debugfs_create_u16("link_down_count", 0400,

stats_dentry,

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +1 -0
+				   (u16 *)&lif->info-
Quoted 1 line

status.link_down_count);

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +2 -0
+	}
+

you never write to these lif->info->status.xyz ..

This is data coming out of DMA memory from the nic, the driver only reads it.

Quoted 3 lines

and link state and speed are/should be available in "ethtool <ifname>" so this looks redundant to me. you can also use ethtool -S to report linkdown count.

The notifyblock is a chunk of data that usually stays together, but isn't really something for ethtool -S, I'd prefer to leave this here.

[...]

Quoted 10 lines
Quoted 7 lines
drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +2 -0
+	case EVENT_OPCODE_LOG:
+		netdev_info(netdev, "Notifyq EVENT_OPCODE_LOG

eid=%lld\n", eid);

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +1 -0
+		print_hex_dump(KERN_INFO, "notifyq ",

DUMP_PREFIX_OFFSET, 16, 1,

drivers/net/ethernet/pensando/ionic/ionic_debugfs.c +1 -0
+			       comp->log.data, sizeof(comp->log.data),

true);

So your device can generate log buffer dump into the kernel log .. I am not sure how acceptable this is, maybe trace buffer is more appropriate for this.

It turns out that this early design feature has gone unused, so I'll drop it out of here for now.

Thanks,
sln

#24 lore raw

PATCH v4 9/19 ionic: Add the basic NDO callbacks for netdev support

In reply to Shannon Nelson

Set up the initial NDO structure and callbacks for netdev to use, and register the netdev. This will allow us to do a few basic operations on the device, but no traffic yet.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |   1 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |   9 +
 .../net/ethernet/pensando/ionic/ionic_dev.h   |   2 +
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 348 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   5 +
 5 files changed, 365 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic.h +1 -0
index 87ab13aee89e..d7eee79b2a10 100644
@@ -34,6 +34,7 @@ struct ionic {
 	unsigned int num_bars;
 	struct identity ident;
 	struct list_head lifs;
+	struct lif *master_lif;
 	unsigned int nnqs_per_lif;
 	unsigned int neqs_per_lif;
 	unsigned int ntxqs_per_lif;
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +9 -0
index 59d1ae7ce532..98c12b770c7f 100644
@@ -206,8 +206,16 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		goto err_out_free_lifs;
 	}
 
+	err = ionic_lifs_register(ionic);
+	if (err) {
+		dev_err(dev, "Cannot register LIFs: %d, aborting\n", err);
+		goto err_out_deinit_lifs;
+	}
+
 	return 0;
 
+err_out_deinit_lifs:
+	ionic_lifs_deinit(ionic);
 err_out_free_lifs:
 	ionic_lifs_free(ionic);
 err_out_free_irqs:
@@ -239,6 +247,7 @@ static void ionic_remove(struct pci_dev *pdev)
 	struct ionic *ionic = pci_get_drvdata(pdev);
 
 	if (ionic) {
+		ionic_lifs_unregister(ionic);
 		ionic_lifs_deinit(ionic);
 		ionic_lifs_free(ionic);
 		ionic_bus_free_irq_vectors(ionic);
drivers/net/ethernet/pensando/ionic/ionic_dev.h +2 -0
index 8bd1501dd639..523927566925 100644
@@ -10,6 +10,8 @@
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
+#define IONIC_MIN_MTU			ETH_MIN_MTU
+#define IONIC_MAX_MTU			9194
 #define IONIC_LIFS_MAX			1024
 
 struct ionic_dev_bar {
drivers/net/ethernet/pensando/ionic/ionic_lif.c +348 -0

386 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h +5 -0
index 80eec0778f40..ef3f7340a277 100644
@@ -61,6 +61,8 @@ struct qcq {
 
 enum lif_state_flags {
 	LIF_INITED,
+	LIF_UP,
+	LIF_QUEUE_RESET,
 
 	/* leave this as last */
 	LIF_STATE_SIZE
@@ -84,6 +86,7 @@ struct lif {
 	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
+	u64 hw_features;
 
 	struct lif_info *info;
 	dma_addr_t info_pa;
@@ -124,6 +127,8 @@ int ionic_lifs_alloc(struct ionic *ionic);
 void ionic_lifs_free(struct ionic *ionic);
 void ionic_lifs_deinit(struct ionic *ionic);
 int ionic_lifs_init(struct ionic *ionic);
+int ionic_lifs_register(struct ionic *ionic);
+void ionic_lifs_unregister(struct ionic *ionic);
 int ionic_lif_identify(struct ionic *ionic, u8 lif_type,
 		       union lif_identity *lif_ident);
 int ionic_lifs_size(struct ionic *ionic);
Signature · 1 line
2.17.1
#25 lore raw

Re: [PATCH v4 net-next 09/19] ionic: Add the basic NDO callbacks for netdev support

Quoted 85 lines · Shannon Nelson

Set up the initial NDO structure and callbacks for netdev to use, and register the netdev. This will allow us to do a few basic operations on the device, but no traffic yet.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |   1 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   |   9 +
 .../net/ethernet/pensando/ionic/ionic_dev.h   |   2 +
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 348
++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   5 +
 5 files changed, 365 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic.h +1 -0
index 87ab13aee89e..d7eee79b2a10 100644
@@ -34,6 +34,7 @@ struct ionic {
 	unsigned int num_bars;
 	struct identity ident;
 	struct list_head lifs;
+	struct lif *master_lif;
 	unsigned int nnqs_per_lif;
 	unsigned int neqs_per_lif;
 	unsigned int ntxqs_per_lif;
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +8 -0
index 59d1ae7ce532..98c12b770c7f 100644
@@ -206,8 +206,16 @@ static int ionic_probe(struct pci_dev *pdev,
 const struct pci_device_id *ent)
 		goto err_out_free_lifs;
 	}
 
+	err = ionic_lifs_register(ionic);
+	if (err) {
+		dev_err(dev, "Cannot register LIFs: %d, aborting\n",
 err);
+		goto err_out_deinit_lifs;
+	}
+
 	return 0;
 
+err_out_deinit_lifs:
+	ionic_lifs_deinit(ionic);
 err_out_free_lifs:
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +1 -0
 	ionic_lifs_free(ionic);
 err_out_free_irqs:
@@ -239,6 +247,7 @@ static void ionic_remove(struct pci_dev *pdev)
 	struct ionic *ionic = pci_get_drvdata(pdev);
 
 	if (ionic) {
+		ionic_lifs_unregister(ionic);
 		ionic_lifs_deinit(ionic);
 		ionic_lifs_free(ionic);
 		ionic_bus_free_irq_vectors(ionic);
drivers/net/ethernet/pensando/ionic/ionic_dev.h +2 -0
index 8bd1501dd639..523927566925 100644
@@ -10,6 +10,8 @@
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
+#define IONIC_MIN_MTU			ETH_MIN_MTU
+#define IONIC_MAX_MTU			9194
 #define IONIC_LIFS_MAX			1024
 
 struct ionic_dev_bar {
drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
index 01f9665611d4..005b1d908fa1 100644
@@ -12,8 +12,74 @@
 #include "ionic_lif.h"
 #include "ionic_debugfs.h"
 
+static int ionic_set_nic_features(struct lif *lif, netdev_features_t
 features);
 static int ionic_notifyq_clean(struct lif *lif, int budget);

drivers/net/ethernet/pensando/ionic/ionic_lif.c line 27

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +9 -0
 
+int ionic_open(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	netif_carrier_off(netdev);
+
+	set_bit(LIF_UP, lif->state);
+
+	if (netif_carrier_ok(netdev))

always false ? you just invoked netif_carrier_off two lines ago..

Quoted 21 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +19 -0
+		netif_tx_wake_all_queues(netdev);
+
+	return 0;
+}
+
+static int ionic_lif_stop(struct lif *lif)
+{
+	struct net_device *ndev = lif->netdev;
+	int err = 0;
+
+	if (!test_bit(LIF_UP, lif->state)) {
+		dev_dbg(lif->ionic->dev, "%s: %s state=DOWN\n",
+			__func__, lif->name);
+		return 0;
+	}
+	dev_dbg(lif->ionic->dev, "%s: %s state=UP\n", __func__, lif-
 >name);
+	clear_bit(LIF_UP, lif->state);
+
+	/* carrier off before disabling queues to avoid watchdog

timeout */

drivers/net/ethernet/pensando/ionic/ionic_lif.c line 49

Quoted 4 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +4 -0
+	netif_carrier_off(ndev);
+	netif_tx_stop_all_queues(ndev);
+	netif_tx_disable(ndev);
+	synchronize_rcu();

why synchronize_rcu ?

Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +8 -0
+
+	return err;
+}
+
+int ionic_stop(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+

drivers/net/ethernet/pensando/ionic/ionic_lif.c line 67

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+	return ionic_lif_stop(lif);
+}
+
+int ionic_reset_queues(struct lif *lif)
+{
+	bool running;
+	int err = 0;
+
+	/* Put off the next watchdog timeout */
+	netif_trans_update(lif->netdev);

this doesn't seem right to me also this won't help you if the next while loop takes too long.. also netif_trans_update is marked to be only used for legacy drivers.

Quoted 380 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +15 -0
+
+	while (test_and_set_bit(LIF_QUEUE_RESET, lif->state))
+		usleep_range(100, 200);
+
+	running = netif_running(lif->netdev);
+	if (running)
+		err = ionic_stop(lif->netdev);
+	if (!err && running)
+		ionic_open(lif->netdev);
+
+	clear_bit(LIF_QUEUE_RESET, lif->state);
+
+	return err;
+}
+
 static bool ionic_adminq_service(struct cq *cq, struct cq_info

*cq_info)
{

drivers/net/ethernet/pensando/ionic/ionic_lif.c +15 -0
 	struct admin_comp *comp = cq_info->cq_desc;
@@ -114,6 +180,81 @@ static int ionic_notifyq_clean(struct lif *lif,
 int budget)
 	return work_done;
 }
 
+static int ionic_set_features(struct net_device *netdev,
+			      netdev_features_t features)
+{
+	struct lif *lif = netdev_priv(netdev);
+	int err;
+
+	netdev_dbg(netdev, "%s: lif->features=0x%08llx
 new_features=0x%08llx\n",
+		   __func__, (u64)lif->netdev->features,
 (u64)features);
+
+	err = ionic_set_nic_features(lif, features);
+
+	return err;
+}
+
+static int ionic_set_mac_address(struct net_device *netdev, void

*sa)

drivers/net/ethernet/pensando/ionic/ionic_lif.c +35 -0
+{
+	netdev_info(netdev, "%s: stubbed\n", __func__);
+	return 0;
+}
+
+static int ionic_change_mtu(struct net_device *netdev, int new_mtu)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.lif_setattr = {
+			.opcode = CMD_OPCODE_LIF_SETATTR,
+			.index = cpu_to_le16(lif->index),
+			.attr = IONIC_LIF_ATTR_MTU,
+			.mtu = cpu_to_le32(new_mtu),
+		},
+	};
+	int err;
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	netdev->mtu = new_mtu;
+	err = ionic_reset_queues(lif);
+
+	return err;
+}
+
+static void ionic_tx_timeout(struct net_device *netdev)
+{
+	netdev_info(netdev, "%s: stubbed\n", __func__);
+}
+
+static int ionic_vlan_rx_add_vid(struct net_device *netdev, __be16

proto,

drivers/net/ethernet/pensando/ionic/ionic_lif.c +7 -0
+				 u16 vid)
+{
+	netdev_info(netdev, "%s: stubbed\n", __func__);
+	return 0;
+}
+
+static int ionic_vlan_rx_kill_vid(struct net_device *netdev, __be16

proto,

drivers/net/ethernet/pensando/ionic/ionic_lif.c +24 -0
+				  u16 vid)
+{
+	netdev_info(netdev, "%s: stubbed\n", __func__);
+	return 0;
+}
+
+static const struct net_device_ops ionic_netdev_ops = {
+	.ndo_open               = ionic_open,
+	.ndo_stop               = ionic_stop,
+	.ndo_set_features	= ionic_set_features,
+	.ndo_set_mac_address	= ionic_set_mac_address,
+	.ndo_validate_addr	= eth_validate_addr,
+	.ndo_tx_timeout         = ionic_tx_timeout,
+	.ndo_change_mtu         = ionic_change_mtu,
+	.ndo_vlan_rx_add_vid    = ionic_vlan_rx_add_vid,
+	.ndo_vlan_rx_kill_vid   = ionic_vlan_rx_kill_vid,
+};
+
 static irqreturn_t ionic_isr(int irq, void *data)
 {
 	struct napi_struct *napi = data;
@@ -388,6 +529,12 @@ static struct lif *ionic_lif_alloc(struct ionic
 *ionic, unsigned int index)
 
 	lif = netdev_priv(netdev);
 	lif->netdev = netdev;
+	ionic->master_lif = lif;
+	netdev->netdev_ops = &ionic_netdev_ops;
+
+	netdev->watchdog_timeo = 2 * HZ;
+	netdev->min_mtu = IONIC_MIN_MTU;
+	netdev->max_mtu = IONIC_MAX_MTU;
 
 	lif->neqs = ionic->neqs_per_lif;
drivers/net/ethernet/pensando/ionic/ionic_lif.c +57 -0
 	lif->nxqs = ionic->ntxqs_per_lif;
@@ -655,6 +802,177 @@ static int ionic_lif_notifyq_init(struct lif
 *lif)
 	return 0;
 }
 
+static __le64 ionic_netdev_features_to_nic(netdev_features_t
 features)
+{
+	u64 wanted = 0;
+
+	if (features & NETIF_F_HW_VLAN_CTAG_TX)
+		wanted |= ETH_HW_VLAN_TX_TAG;
+	if (features & NETIF_F_HW_VLAN_CTAG_RX)
+		wanted |= ETH_HW_VLAN_RX_STRIP;
+	if (features & NETIF_F_HW_VLAN_CTAG_FILTER)
+		wanted |= ETH_HW_VLAN_RX_FILTER;
+	if (features & NETIF_F_RXHASH)
+		wanted |= ETH_HW_RX_HASH;
+	if (features & NETIF_F_RXCSUM)
+		wanted |= ETH_HW_RX_CSUM;
+	if (features & NETIF_F_SG)
+		wanted |= ETH_HW_TX_SG;
+	if (features & NETIF_F_HW_CSUM)
+		wanted |= ETH_HW_TX_CSUM;
+	if (features & NETIF_F_TSO)
+		wanted |= ETH_HW_TSO;
+	if (features & NETIF_F_TSO6)
+		wanted |= ETH_HW_TSO_IPV6;
+	if (features & NETIF_F_TSO_ECN)
+		wanted |= ETH_HW_TSO_ECN;
+	if (features & NETIF_F_GSO_GRE)
+		wanted |= ETH_HW_TSO_GRE;
+	if (features & NETIF_F_GSO_GRE_CSUM)
+		wanted |= ETH_HW_TSO_GRE_CSUM;
+	if (features & NETIF_F_GSO_IPXIP4)
+		wanted |= ETH_HW_TSO_IPXIP4;
+	if (features & NETIF_F_GSO_IPXIP6)
+		wanted |= ETH_HW_TSO_IPXIP6;
+	if (features & NETIF_F_GSO_UDP_TUNNEL)
+		wanted |= ETH_HW_TSO_UDP;
+	if (features & NETIF_F_GSO_UDP_TUNNEL_CSUM)
+		wanted |= ETH_HW_TSO_UDP_CSUM;
+
+	return cpu_to_le64(wanted);
+}
+
+static int ionic_set_nic_features(struct lif *lif, netdev_features_t
 features)
+{
+	struct device *dev = lif->ionic->dev;
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.lif_setattr = {
+			.opcode = CMD_OPCODE_LIF_SETATTR,
+			.index = cpu_to_le16(lif->index),
+			.attr = IONIC_LIF_ATTR_FEATURES,
+		},
+	};
+	u64 vlan_flags = ETH_HW_VLAN_TX_TAG |
+			 ETH_HW_VLAN_RX_STRIP |
+			 ETH_HW_VLAN_RX_FILTER;
+	int err;
+
+	ctx.cmd.lif_setattr.features =

ionic_netdev_features_to_nic(features);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	lif->hw_features = le64_to_cpu(ctx.cmd.lif_setattr.features &
+				       ctx.comp.lif_setattr.features);
+
+	if ((vlan_flags & features) &&
+	    !(vlan_flags & le64_to_cpu(ctx.comp.lif_setattr.features)))
+		dev_info_once(lif->ionic->dev, "NIC is not supporting

vlan offload, likely in SmartNIC mode\n");

drivers/net/ethernet/pensando/ionic/ionic_lif.c +134 -0
+
+	if (lif->hw_features & ETH_HW_VLAN_TX_TAG)
+		dev_dbg(dev, "feature ETH_HW_VLAN_TX_TAG\n");
+	if (lif->hw_features & ETH_HW_VLAN_RX_STRIP)
+		dev_dbg(dev, "feature ETH_HW_VLAN_RX_STRIP\n");
+	if (lif->hw_features & ETH_HW_VLAN_RX_FILTER)
+		dev_dbg(dev, "feature ETH_HW_VLAN_RX_FILTER\n");
+	if (lif->hw_features & ETH_HW_RX_HASH)
+		dev_dbg(dev, "feature ETH_HW_RX_HASH\n");
+	if (lif->hw_features & ETH_HW_TX_SG)
+		dev_dbg(dev, "feature ETH_HW_TX_SG\n");
+	if (lif->hw_features & ETH_HW_TX_CSUM)
+		dev_dbg(dev, "feature ETH_HW_TX_CSUM\n");
+	if (lif->hw_features & ETH_HW_RX_CSUM)
+		dev_dbg(dev, "feature ETH_HW_RX_CSUM\n");
+	if (lif->hw_features & ETH_HW_TSO)
+		dev_dbg(dev, "feature ETH_HW_TSO\n");
+	if (lif->hw_features & ETH_HW_TSO_IPV6)
+		dev_dbg(dev, "feature ETH_HW_TSO_IPV6\n");
+	if (lif->hw_features & ETH_HW_TSO_ECN)
+		dev_dbg(dev, "feature ETH_HW_TSO_ECN\n");
+	if (lif->hw_features & ETH_HW_TSO_GRE)
+		dev_dbg(dev, "feature ETH_HW_TSO_GRE\n");
+	if (lif->hw_features & ETH_HW_TSO_GRE_CSUM)
+		dev_dbg(dev, "feature ETH_HW_TSO_GRE_CSUM\n");
+	if (lif->hw_features & ETH_HW_TSO_IPXIP4)
+		dev_dbg(dev, "feature ETH_HW_TSO_IPXIP4\n");
+	if (lif->hw_features & ETH_HW_TSO_IPXIP6)
+		dev_dbg(dev, "feature ETH_HW_TSO_IPXIP6\n");
+	if (lif->hw_features & ETH_HW_TSO_UDP)
+		dev_dbg(dev, "feature ETH_HW_TSO_UDP\n");
+	if (lif->hw_features & ETH_HW_TSO_UDP_CSUM)
+		dev_dbg(dev, "feature ETH_HW_TSO_UDP_CSUM\n");
+
+	return 0;
+}
+
+static int ionic_init_nic_features(struct lif *lif)
+{
+	struct net_device *netdev = lif->netdev;
+	netdev_features_t features;
+	int err;
+
+	/* set up what we expect to support by default */
+	features = NETIF_F_HW_VLAN_CTAG_TX |
+		   NETIF_F_HW_VLAN_CTAG_RX |
+		   NETIF_F_HW_VLAN_CTAG_FILTER |
+		   NETIF_F_RXHASH |
+		   NETIF_F_SG |
+		   NETIF_F_HW_CSUM |
+		   NETIF_F_RXCSUM |
+		   NETIF_F_TSO |
+		   NETIF_F_TSO6 |
+		   NETIF_F_TSO_ECN;
+
+	err = ionic_set_nic_features(lif, features);
+	if (err)
+		return err;
+
+	/* tell the netdev what we actually can support */
+	netdev->features |= NETIF_F_HIGHDMA;
+
+	if (lif->hw_features & ETH_HW_VLAN_TX_TAG)
+		netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_TX;
+	if (lif->hw_features & ETH_HW_VLAN_RX_STRIP)
+		netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_RX;
+	if (lif->hw_features & ETH_HW_VLAN_RX_FILTER)
+		netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_FILTER;
+	if (lif->hw_features & ETH_HW_RX_HASH)
+		netdev->hw_features |= NETIF_F_RXHASH;
+	if (lif->hw_features & ETH_HW_TX_SG)
+		netdev->hw_features |= NETIF_F_SG;
+
+	if (lif->hw_features & ETH_HW_TX_CSUM)
+		netdev->hw_enc_features |= NETIF_F_HW_CSUM;
+	if (lif->hw_features & ETH_HW_RX_CSUM)
+		netdev->hw_enc_features |= NETIF_F_RXCSUM;
+	if (lif->hw_features & ETH_HW_TSO)
+		netdev->hw_enc_features |= NETIF_F_TSO;
+	if (lif->hw_features & ETH_HW_TSO_IPV6)
+		netdev->hw_enc_features |= NETIF_F_TSO6;
+	if (lif->hw_features & ETH_HW_TSO_ECN)
+		netdev->hw_enc_features |= NETIF_F_TSO_ECN;
+	if (lif->hw_features & ETH_HW_TSO_GRE)
+		netdev->hw_enc_features |= NETIF_F_GSO_GRE;
+	if (lif->hw_features & ETH_HW_TSO_GRE_CSUM)
+		netdev->hw_enc_features |= NETIF_F_GSO_GRE_CSUM;
+	if (lif->hw_features & ETH_HW_TSO_IPXIP4)
+		netdev->hw_enc_features |= NETIF_F_GSO_IPXIP4;
+	if (lif->hw_features & ETH_HW_TSO_IPXIP6)
+		netdev->hw_enc_features |= NETIF_F_GSO_IPXIP6;
+	if (lif->hw_features & ETH_HW_TSO_UDP)
+		netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
+	if (lif->hw_features & ETH_HW_TSO_UDP_CSUM)
+		netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
+
+	netdev->hw_features |= netdev->hw_enc_features;
+	netdev->features |= netdev->hw_features;
+
+	netdev->priv_flags |= IFF_UNICAST_FLT;
+
+	return 0;
+}
+
 static int ionic_lif_init(struct lif *lif)
 {
 	struct ionic_dev *idev = &lif->ionic->idev;
@@ -711,6 +1029,10 @@ static int ionic_lif_init(struct lif *lif)
 			goto err_out_notifyq_deinit;
 	}
 
+	err = ionic_init_nic_features(lif);
+	if (err)
+		goto err_out_notifyq_deinit;
+
 	set_bit(LIF_INITED, lif->state);
 
 	return 0;
@@ -745,6 +1067,32 @@ int ionic_lifs_init(struct ionic *ionic)
 	return 0;
 }
 
+int ionic_lifs_register(struct ionic *ionic)
+{
+	int err;
+
+	/* only register LIF0 for now */
+	err = register_netdev(ionic->master_lif->netdev);
+	if (err) {
+		dev_err(ionic->dev, "Cannot register net device,
 aborting\n");
+		return err;
+	}
+
+	ionic->master_lif->registered = true;
+
+	return 0;
+}
+
+void ionic_lifs_unregister(struct ionic *ionic)
+{
+	/* There is only one lif ever registered in the
+	 * current model, so don't bother searching the
+	 * ionic->lif for candidates to unregister
+	 */
+	if (ionic->master_lif->netdev->reg_state == NETREG_REGISTERED)
+		unregister_netdev(ionic->master_lif->netdev);
+}
+
 int ionic_lif_identify(struct ionic *ionic, u8 lif_type,
 		       union lif_identity *lid)

{

drivers/net/ethernet/pensando/ionic/ionic_lif.h +5 -0
index 80eec0778f40..ef3f7340a277 100644
@@ -61,6 +61,8 @@ struct qcq {
 
 enum lif_state_flags {
 	LIF_INITED,
+	LIF_UP,
+	LIF_QUEUE_RESET,
 
 	/* leave this as last */
 	LIF_STATE_SIZE
@@ -84,6 +86,7 @@ struct lif {
 	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
+	u64 hw_features;
 
 	struct lif_info *info;
 	dma_addr_t info_pa;
@@ -124,6 +127,8 @@ int ionic_lifs_alloc(struct ionic *ionic);
 void ionic_lifs_free(struct ionic *ionic);
 void ionic_lifs_deinit(struct ionic *ionic);
 int ionic_lifs_init(struct ionic *ionic);
+int ionic_lifs_register(struct ionic *ionic);
+void ionic_lifs_unregister(struct ionic *ionic);
 int ionic_lif_identify(struct ionic *ionic, u8 lif_type,
 		       union lif_identity *lif_ident);
 int ionic_lifs_size(struct ionic *ionic);
#26 lore raw

Re: [PATCH v4 net-next 09/19] ionic: Add the basic NDO callbacks for netdev support

Quoted 15 lines · Saeed Mahameed
Quoted 14 lines · Shannon Nelson

Set up the initial NDO structure and callbacks for netdev to use, and register the netdev. This will allow us to do a few basic operations on the device, but no traffic yet.

---
  drivers/net/ethernet/pensando/ionic/ionic.h   |   1 +
  .../ethernet/pensando/ionic/ionic_bus_pci.c   |   9 +
  .../net/ethernet/pensando/ionic/ionic_dev.h   |   2 +
  .../net/ethernet/pensando/ionic/ionic_lif.c   | 348
++++++++++++++++++
  .../net/ethernet/pensando/ionic/ionic_lif.h   |   5 +
  5 files changed, 365 insertions(+)

[...]

Quoted 5 lines
Quoted 5 lines

+static int ionic_set_nic_features(struct lif *lif, netdev_features_t features);
static int ionic_notifyq_clean(struct lif *lif, int budget);

Quoted 10 lines

drivers/net/ethernet/pensando/ionic/ionic_lif.c line 26

Quoted 9 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +9 -0
+int ionic_open(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	netif_carrier_off(netdev);
+
+	set_bit(LIF_UP, lif->state);
+
+	if (netif_carrier_ok(netdev))

always false ? you just invoked netif_carrier_off two lines ago..

Hmmm... an artifact of splitting up an existing driver.  This makes more sense when there's a link status check in between these, which comes in about 3 patches later.  Unless this really causes someone significant heartburn, I'm going to leave this as is for now.

Quoted 17 lines
Quoted 17 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +16 -0
+		netif_tx_wake_all_queues(netdev);
+
+	return 0;
+}
+
+static int ionic_lif_stop(struct lif *lif)
+{
+	struct net_device *ndev = lif->netdev;
+	int err = 0;
+
+	if (!test_bit(LIF_UP, lif->state)) {
+		dev_dbg(lif->ionic->dev, "%s: %s state=DOWN\n",
+			__func__, lif->name);
+		return 0;
+	}
+	dev_dbg(lif->ionic->dev, "%s: %s state=UP\n", __func__, lif-
Quoted 1 line

name);

Quoted 9 lines

drivers/net/ethernet/pensando/ionic/ionic_lif.c line 49

Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
+	clear_bit(LIF_UP, lif->state);
+
+	/* carrier off before disabling queues to avoid watchdog

timeout */

drivers/net/ethernet/pensando/ionic/ionic_lif.c +4 -0
+	netif_carrier_off(ndev);
+	netif_tx_stop_all_queues(ndev);
+	netif_tx_disable(ndev);
+	synchronize_rcu();

why synchronize_rcu ?

Looks like a little leakage from a feature in the internal driver, I'll remove it.

Quoted 13 lines
Quoted 13 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +13 -0
+
+	return err;
+}
+
+int ionic_stop(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	return ionic_lif_stop(lif);
+}
+
+int ionic_reset_queues(struct lif *lif)
+{
Quoted 8 lines

drivers/net/ethernet/pensando/ionic/ionic_lif.c line 67

Quoted 5 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +5 -0
+	bool running;
+	int err = 0;
+
+	/* Put off the next watchdog timeout */
+	netif_trans_update(lif->netdev);

this doesn't seem right to me also this won't help you if the next while loop takes too long.. also netif_trans_update is marked to be only used for legacy drivers.

If the loop takes too long, then I don't mind if the watchdog goes off.

Yes, its primary use is now handled by netdev_start_xmit(), but it is still a handy gizmo to give a little more cushion until the carrier is set off in ionic_lif_stop().

sln

#27 lore raw

PATCH v4 10/19 ionic: Add management of rx filters

In reply to Shannon Nelson

Set up the infrastructure for managing Rx filters. We can't ask the hardware for what filters it has, so we keep a local list of filters that we've pushed into the HW.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   4 +-
 .../net/ethernet/pensando/ionic/ionic_lif.c   |   6 +
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   2 +
 .../ethernet/pensando/ionic/ionic_rx_filter.c | 143 ++++++++++++++++++
 .../ethernet/pensando/ionic/ionic_rx_filter.h |  35 +++++
 5 files changed, 188 insertions(+), 2 deletions(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_rx_filter.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_rx_filter.h
drivers/net/ethernet/pensando/ionic/Makefile +2 -2
index 04d519d00be6..7d9cdc5f02a1 100644
@@ -3,5 +3,5 @@
 
 obj-$(CONFIG_IONIC) := ionic.o
 
-ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_debugfs.o \
-	   ionic_lif.o
+ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o \
+	   ionic_lif.o ionic_rx_filter.o ionic_debugfs.o
drivers/net/ethernet/pensando/ionic/ionic_lif.c +6 -0
index 005b1d908fa1..72ac0fd56b69 100644
@@ -667,6 +667,8 @@ static void ionic_lif_deinit(struct lif *lif)
 
 	clear_bit(LIF_INITED, lif->state);
 
+	ionic_rx_filters_deinit(lif);
+
 	napi_disable(&lif->adminqcq->napi);
 	ionic_lif_qcq_deinit(lif, lif->notifyqcq);
 	ionic_lif_qcq_deinit(lif, lif->adminqcq);
@@ -1033,6 +1035,10 @@ static int ionic_lif_init(struct lif *lif)
 	if (err)
 		goto err_out_notifyq_deinit;
 
+	err = ionic_rx_filters_init(lif);
+	if (err)
+		goto err_out_notifyq_deinit;
+
 	set_bit(LIF_INITED, lif->state);
 
 	return 0;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +2 -0
index ef3f7340a277..9f112aa69033 100644
@@ -5,6 +5,7 @@
 #define _IONIC_LIF_H_
 
 #include <linux/pci.h>
+#include "ionic_rx_filter.h"
 
 #define IONIC_ADMINQ_LENGTH	16	/* must be a power of two */
 #define IONIC_NOTIFYQ_LENGTH	64	/* must be a power of two */
@@ -92,6 +93,7 @@ struct lif {
 	dma_addr_t info_pa;
 	u32 info_sz;
 
+	struct rx_filters rx_filters;
 	unsigned long *dbid_inuse;
 	unsigned int dbid_count;
 	struct dentry *dentry;
drivers/net/ethernet/pensando/ionic/ionic_rx_filter.c new file +143 -0

143 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_rx_filter.h new file +35 -0
new file mode 100644
index 000000000000..67ae279a4cb6
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_RX_FILTER_H_
+#define _IONIC_RX_FILTER_H_
+
+#define RXQ_INDEX_ANY		(0xFFFF)
+struct rx_filter {
+	u32 flow_id;
+	u32 filter_id;
+	u16 rxq_index;
+	struct rx_filter_add_cmd cmd;
+	struct hlist_node by_hash;
+	struct hlist_node by_id;
+};
+
+#define RX_FILTER_HASH_BITS	10
+#define RX_FILTER_HLISTS	BIT(RX_FILTER_HASH_BITS)
+#define RX_FILTER_HLISTS_MASK	(RX_FILTER_HLISTS - 1)
+struct rx_filters {
+	spinlock_t lock;				/* filter list lock */
+	struct hlist_head by_hash[RX_FILTER_HLISTS];	/* by skb hash */
+	struct hlist_head by_id[RX_FILTER_HLISTS];	/* by filter_id */
+};
+
+void ionic_rx_filter_free(struct lif *lif, struct rx_filter *f);
+int ionic_rx_filter_del(struct lif *lif, struct rx_filter *f);
+int ionic_rx_filters_init(struct lif *lif);
+void ionic_rx_filters_deinit(struct lif *lif);
+int ionic_rx_filter_save(struct lif *lif, u32 flow_id, u16 rxq_index,
+			 u32 hash, struct ionic_admin_ctx *ctx);
+struct rx_filter *ionic_rx_filter_by_vlan(struct lif *lif, u16 vid);
+struct rx_filter *ionic_rx_filter_by_addr(struct lif *lif, const u8 *addr);
+
+#endif /* _IONIC_RX_FILTER_H_ */
Signature · 1 line
2.17.1
#28 lore raw

Re: [PATCH v4 net-next 10/19] ionic: Add management of rx filters

Quoted 19 lines · Shannon Nelson

Set up the infrastructure for managing Rx filters. We can't ask the hardware for what filters it has, so we keep a local list of filters that we've pushed into the HW.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   4 +-
 .../net/ethernet/pensando/ionic/ionic_lif.c   |   6 +
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   2 +
 .../ethernet/pensando/ionic/ionic_rx_filter.c | 143
++++++++++++++++++
 .../ethernet/pensando/ionic/ionic_rx_filter.h |  35 +++++
 5 files changed, 188 insertions(+), 2 deletions(-)
 create mode 100644
drivers/net/ethernet/pensando/ionic/ionic_rx_filter.c
 create mode 100644

drivers/net/ethernet/pensando/ionic/ionic_rx_filter.h

[...]

Quoted 21 lines
drivers/net/ethernet/pensando/ionic/ionic_rx_filter.h new file +15 -0
+#define RXQ_INDEX_ANY		(0xFFFF)
+struct rx_filter {
+	u32 flow_id;
+	u32 filter_id;
+	u16 rxq_index;
+	struct rx_filter_add_cmd cmd;
+	struct hlist_node by_hash;
+	struct hlist_node by_id;
+};
+
+#define RX_FILTER_HASH_BITS	10
+#define RX_FILTER_HLISTS	BIT(RX_FILTER_HASH_BITS)
+#define RX_FILTER_HLISTS_MASK	(RX_FILTER_HLISTS - 1)
+struct rx_filters {
+	spinlock_t lock;				/* filter list lock

*/

(quoted diff) +1 -0
+	struct hlist_head by_hash[RX_FILTER_HLISTS];	/* by skb

hash */

(quoted diff) +1 -0
+	struct hlist_head by_id[RX_FILTER_HLISTS];	/* by

filter_id */

Review

Quoted 2 lines
(quoted diff) +2 -0
+};
+

Following Dave's comment on this, you use too generic struct and macro /define names, i strongly recommend to add a unique prefix to this driver.

#29 lore raw

PATCH v4 11/19 ionic: Add Rx filter and rx_mode ndo support

In reply to Shannon Nelson

Add the Rx filtering and rx_mode NDO callbacks. Also add the deferred work thread handling needed to manage the filter requests otuside of the netif_addr_lock spinlock.

---
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 389 +++++++++++++++++-
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  29 ++
 2 files changed, 412 insertions(+), 6 deletions(-)
drivers/net/ethernet/pensando/ionic/ionic_lif.c +383 -6
index 72ac0fd56b69..efcda1337f91 100644
@@ -12,9 +12,55 @@
 #include "ionic_lif.h"
 #include "ionic_debugfs.h"
 
+static void ionic_lif_rx_mode(struct lif *lif, unsigned int rx_mode);
+static int ionic_lif_addr_add(struct lif *lif, const u8 *addr);
+static int ionic_lif_addr_del(struct lif *lif, const u8 *addr);
+
 static int ionic_set_nic_features(struct lif *lif, netdev_features_t features);
 static int ionic_notifyq_clean(struct lif *lif, int budget);
 
+static void ionic_lif_deferred_work(struct work_struct *work)
+{
+	struct lif *lif = container_of(work, struct lif, deferred.work);
+	struct ionic_deferred *def = &lif->deferred;
+	struct ionic_deferred_work *w = NULL;
+
+	spin_lock_bh(&def->lock);
+	if (!list_empty(&def->list)) {
+		w = list_first_entry(&def->list,
+				     struct ionic_deferred_work, list);
+		list_del(&w->list);
+	}
+	spin_unlock_bh(&def->lock);
+
+	if (w) {
+		switch (w->type) {
+		case DW_TYPE_RX_MODE:
+			ionic_lif_rx_mode(lif, w->rx_mode);
+			break;
+		case DW_TYPE_RX_ADDR_ADD:
+			ionic_lif_addr_add(lif, w->addr);
+			break;
+		case DW_TYPE_RX_ADDR_DEL:
+			ionic_lif_addr_del(lif, w->addr);
+			break;
+		default:
+			break;
+		}
+		kfree(w);
+		schedule_work(&def->work);
+	}
+}
+
+static void ionic_lif_deferred_enqueue(struct ionic_deferred *def,
+				       struct ionic_deferred_work *work)
+{
+	spin_lock_bh(&def->lock);
+	list_add_tail(&work->list, &def->list);
+	spin_unlock_bh(&def->lock);
+	schedule_work(&def->work);
+}
+
 int ionic_open(struct net_device *netdev)
 {
 	struct lif *lif = netdev_priv(netdev);
@@ -180,6 +226,235 @@ static int ionic_notifyq_clean(struct lif *lif, int budget)
 	return work_done;
 }
 
+static int ionic_lif_addr_add(struct lif *lif, const u8 *addr)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_add = {
+			.opcode = CMD_OPCODE_RX_FILTER_ADD,
+			.lif_index = cpu_to_le16(lif->index),
+			.match = cpu_to_le16(RX_FILTER_MATCH_MAC),
+		},
+	};
+	struct rx_filter *f;
+	int err;
+
+	/* don't bother if we already have it */
+	spin_lock_bh(&lif->rx_filters.lock);
+	f = ionic_rx_filter_by_addr(lif, addr);
+	spin_unlock_bh(&lif->rx_filters.lock);
+	if (f)
+		return 0;
+
+	netdev_dbg(lif->netdev, "rx_filter add ADDR %pM (id %d)\n", addr,
+		   ctx.comp.rx_filter_add.filter_id);
+
+	memcpy(ctx.cmd.rx_filter_add.mac.addr, addr, ETH_ALEN);
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	return ionic_rx_filter_save(lif, 0, RXQ_INDEX_ANY, 0, &ctx);
+}
+
+static int ionic_lif_addr_del(struct lif *lif, const u8 *addr)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_del = {
+			.opcode = CMD_OPCODE_RX_FILTER_DEL,
+			.lif_index = cpu_to_le16(lif->index),
+		},
+	};
+	struct rx_filter *f;
+	int err;
+
+	spin_lock_bh(&lif->rx_filters.lock);
+	f = ionic_rx_filter_by_addr(lif, addr);
+	if (!f) {
+		spin_unlock_bh(&lif->rx_filters.lock);
+		return -ENOENT;
+	}
+
+	ctx.cmd.rx_filter_del.filter_id = cpu_to_le32(f->filter_id);
+	ionic_rx_filter_free(lif, f);
+	spin_unlock_bh(&lif->rx_filters.lock);
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	netdev_dbg(lif->netdev, "rx_filter del ADDR %pM (id %d)\n", addr,
+		   ctx.cmd.rx_filter_del.filter_id);
+
+	return 0;
+}
+
+static int ionic_lif_addr(struct lif *lif, const u8 *addr, bool add)
+{
+	struct ionic *ionic = lif->ionic;
+	struct ionic_deferred_work *work;
+	unsigned int nmfilters;
+	unsigned int nufilters;
+
+	if (add) {
+		/* Do we have space for this filter?  We test the counters
+		 * here before checking the need for deferral so that we
+		 * can return an overflow error to the stack.
+		 */
+		nmfilters = le32_to_cpu(ionic->ident.lif.eth.max_mcast_filters);
+		nufilters = le32_to_cpu(ionic->ident.lif.eth.max_ucast_filters);
+
+		if ((is_multicast_ether_addr(addr) && lif->nmcast < nmfilters))
+			lif->nmcast++;
+		else if (!is_multicast_ether_addr(addr) &&
+			 lif->nucast < nufilters)
+			lif->nucast++;
+		else
+			return -ENOSPC;
+	} else {
+		if (is_multicast_ether_addr(addr) && lif->nmcast)
+			lif->nmcast--;
+		else if (!is_multicast_ether_addr(addr) && lif->nucast)
+			lif->nucast--;
+	}
+
+	if (in_interrupt()) {
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+			return -ENOMEM;
+		}
+		work->type = add ? DW_TYPE_RX_ADDR_ADD : DW_TYPE_RX_ADDR_DEL;
+		memcpy(work->addr, addr, ETH_ALEN);
+		netdev_dbg(lif->netdev, "deferred: rx_filter %s %pM\n",
+			   add ? "add" : "del", addr);
+		ionic_lif_deferred_enqueue(&lif->deferred, work);
+	} else {
+		netdev_dbg(lif->netdev, "rx_filter %s %pM\n",
+			   add ? "add" : "del", addr);
+		if (add)
+			return ionic_lif_addr_add(lif, addr);
+		else
+			return ionic_lif_addr_del(lif, addr);
+	}
+
+	return 0;
+}
+
+static int ionic_addr_add(struct net_device *netdev, const u8 *addr)
+{
+	return ionic_lif_addr(netdev_priv(netdev), addr, true);
+}
+
+static int ionic_addr_del(struct net_device *netdev, const u8 *addr)
+{
+	return ionic_lif_addr(netdev_priv(netdev), addr, false);
+}
+
+static void ionic_lif_rx_mode(struct lif *lif, unsigned int rx_mode)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_mode_set = {
+			.opcode = CMD_OPCODE_RX_MODE_SET,
+			.lif_index = cpu_to_le16(lif->index),
+			.rx_mode = cpu_to_le16(rx_mode),
+		},
+	};
+	char buf[128];
+	int err;
+	int i;
+#define REMAIN(__x) (sizeof(buf) - (__x))
+
+	i = snprintf(buf, sizeof(buf), "rx_mode 0x%04x -> 0x%04x:",
+		     lif->rx_mode, rx_mode);
+	if (rx_mode & RX_MODE_F_UNICAST)
+		i += snprintf(&buf[i], REMAIN(i), " RX_MODE_F_UNICAST");
+	if (rx_mode & RX_MODE_F_MULTICAST)
+		i += snprintf(&buf[i], REMAIN(i), " RX_MODE_F_MULTICAST");
+	if (rx_mode & RX_MODE_F_BROADCAST)
+		i += snprintf(&buf[i], REMAIN(i), " RX_MODE_F_BROADCAST");
+	if (rx_mode & RX_MODE_F_PROMISC)
+		i += snprintf(&buf[i], REMAIN(i), " RX_MODE_F_PROMISC");
+	if (rx_mode & RX_MODE_F_ALLMULTI)
+		i += snprintf(&buf[i], REMAIN(i), " RX_MODE_F_ALLMULTI");
+	netdev_dbg(lif->netdev, "lif%d %s\n", lif->index, buf);
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		netdev_warn(lif->netdev, "set rx_mode 0x%04x failed: %d\n",
+			    rx_mode, err);
+	else
+		lif->rx_mode = rx_mode;
+}
+
+static void _ionic_lif_rx_mode(struct lif *lif, unsigned int rx_mode)
+{
+	struct ionic_deferred_work *work;
+
+	if (in_interrupt()) {
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+			return;
+		}
+		work->type = DW_TYPE_RX_MODE;
+		work->rx_mode = rx_mode;
+		netdev_dbg(lif->netdev, "deferred: rx_mode\n");
+		ionic_lif_deferred_enqueue(&lif->deferred, work);
+	} else {
+		ionic_lif_rx_mode(lif, rx_mode);
+	}
+}
+
+static void ionic_set_rx_mode(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct identity *ident = &lif->ionic->ident;
+	unsigned int nfilters;
+	unsigned int rx_mode;
+
+	rx_mode = RX_MODE_F_UNICAST;
+	rx_mode |= (netdev->flags & IFF_MULTICAST) ? RX_MODE_F_MULTICAST : 0;
+	rx_mode |= (netdev->flags & IFF_BROADCAST) ? RX_MODE_F_BROADCAST : 0;
+	rx_mode |= (netdev->flags & IFF_PROMISC) ? RX_MODE_F_PROMISC : 0;
+	rx_mode |= (netdev->flags & IFF_ALLMULTI) ? RX_MODE_F_ALLMULTI : 0;
+
+	/* sync unicast addresses
+	 * next check to see if we're in an overflow state
+	 *    if so, we track that we overflowed and enable NIC PROMISC
+	 *    else if the overflow is set and not needed
+	 *       we remove our overflow flag and check the netdev flags
+	 *       to see if we can disable NIC PROMISC
+	 */
+	__dev_uc_sync(netdev, ionic_addr_add, ionic_addr_del);
+	nfilters = le32_to_cpu(ident->lif.eth.max_ucast_filters);
+	if (netdev_uc_count(netdev) + 1 > nfilters) {
+		rx_mode |= RX_MODE_F_PROMISC;
+		lif->uc_overflow = true;
+	} else if (lif->uc_overflow) {
+		lif->uc_overflow = false;
+		if (!(netdev->flags & IFF_PROMISC))
+			rx_mode &= ~RX_MODE_F_PROMISC;
+	}
+
+	/* same for multicast */
+	__dev_mc_sync(netdev, ionic_addr_add, ionic_addr_del);
+	nfilters = le32_to_cpu(ident->lif.eth.max_mcast_filters);
+	if (netdev_mc_count(netdev) > nfilters) {
+		rx_mode |= RX_MODE_F_ALLMULTI;
+		lif->mc_overflow = true;
+	} else if (lif->mc_overflow) {
+		lif->mc_overflow = false;
+		if (!(netdev->flags & IFF_ALLMULTI))
+			rx_mode &= ~RX_MODE_F_ALLMULTI;
+	}
+
+	if (lif->rx_mode != rx_mode)
+		_ionic_lif_rx_mode(lif, rx_mode);
+}
+
 static int ionic_set_features(struct net_device *netdev,
 			      netdev_features_t features)
 {
@@ -196,8 +471,26 @@ static int ionic_set_features(struct net_device *netdev,
 
 static int ionic_set_mac_address(struct net_device *netdev, void *sa)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
-	return 0;
+	struct sockaddr *addr = sa;
+	u8 *mac;
+
+	mac = (u8 *)addr->sa_data;
+	if (ether_addr_equal(netdev->dev_addr, mac))
+		return 0;
+
+	if (!is_valid_ether_addr(mac))
+		return -EADDRNOTAVAIL;
+
+	if (!is_zero_ether_addr(netdev->dev_addr)) {
+		netdev_info(netdev, "deleting mac addr %pM\n",
+			    netdev->dev_addr);
+		ionic_addr_del(netdev, netdev->dev_addr);
+	}
+
+	memcpy(netdev->dev_addr, mac, netdev->addr_len);
+	netdev_info(netdev, "updating mac addr %pM\n", mac);
+
+	return ionic_addr_add(netdev, mac);
 }
 
 static int ionic_change_mtu(struct net_device *netdev, int new_mtu)
@@ -232,20 +525,63 @@ static void ionic_tx_timeout(struct net_device *netdev)
 static int ionic_vlan_rx_add_vid(struct net_device *netdev, __be16 proto,
 				 u16 vid)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
-	return 0;
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_add = {
+			.opcode = CMD_OPCODE_RX_FILTER_ADD,
+			.lif_index = cpu_to_le16(lif->index),
+			.match = cpu_to_le16(RX_FILTER_MATCH_VLAN),
+			.vlan.vlan = cpu_to_le16(vid),
+		},
+	};
+	int err;
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	netdev_dbg(netdev, "rx_filter add VLAN %d (id %d)\n", vid,
+		   ctx.comp.rx_filter_add.filter_id);
+
+	return ionic_rx_filter_save(lif, 0, RXQ_INDEX_ANY, 0, &ctx);
 }
 
 static int ionic_vlan_rx_kill_vid(struct net_device *netdev, __be16 proto,
 				  u16 vid)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
-	return 0;
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_del = {
+			.opcode = CMD_OPCODE_RX_FILTER_DEL,
+			.lif_index = cpu_to_le16(lif->index),
+		},
+	};
+	struct rx_filter *f;
+
+	spin_lock_bh(&lif->rx_filters.lock);
+
+	f = ionic_rx_filter_by_vlan(lif, vid);
+	if (!f) {
+		spin_unlock_bh(&lif->rx_filters.lock);
+		return -ENOENT;
+	}
+
+	netdev_dbg(netdev, "rx_filter del VLAN %d (id %d)\n", vid,
+		   le32_to_cpu(ctx.cmd.rx_filter_del.filter_id));
+
+	ctx.cmd.rx_filter_del.filter_id = cpu_to_le32(f->filter_id);
+	ionic_rx_filter_free(lif, f);
+	spin_unlock_bh(&lif->rx_filters.lock);
+
+	return ionic_adminq_post_wait(lif, &ctx);
 }
 
 static const struct net_device_ops ionic_netdev_ops = {
 	.ndo_open               = ionic_open,
 	.ndo_stop               = ionic_stop,
+	.ndo_set_rx_mode	= ionic_set_rx_mode,
 	.ndo_set_features	= ionic_set_features,
 	.ndo_set_mac_address	= ionic_set_mac_address,
 	.ndo_validate_addr	= eth_validate_addr,
@@ -546,6 +882,10 @@ static struct lif *ionic_lif_alloc(struct ionic *ionic, unsigned int index)
 
 	spin_lock_init(&lif->adminq_lock);
 
+	spin_lock_init(&lif->deferred.lock);
+	INIT_LIST_HEAD(&lif->deferred.list);
+	INIT_WORK(&lif->deferred.work, ionic_lif_deferred_work);
+
 	/* allocate lif info */
 	lif->info_sz = ALIGN(sizeof(*lif->info), PAGE_SIZE);
 	lif->info = dma_alloc_coherent(dev, lif->info_sz,
@@ -607,6 +947,8 @@ static void ionic_lif_free(struct lif *lif)
 	ionic_qcqs_free(lif);
 	ionic_lif_reset(lif);
 
+	cancel_work_sync(&lif->deferred.work);
+
 	/* free lif info */
 	dma_free_coherent(dev, lif->info_sz, lif->info, lif->info_pa);
 	lif->info = NULL;
@@ -975,6 +1317,37 @@ static int ionic_init_nic_features(struct lif *lif)
 	return 0;
 }
 
+static int ionic_station_set(struct lif *lif)
+{
+	struct net_device *netdev = lif->netdev;
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.lif_getattr = {
+			.opcode = CMD_OPCODE_LIF_GETATTR,
+			.index = cpu_to_le16(lif->index),
+			.attr = IONIC_LIF_ATTR_MAC,
+		},
+	};
+	int err;
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	if (!is_zero_ether_addr(netdev->dev_addr)) {
+		netdev_dbg(lif->netdev, "deleting station MAC addr %pM\n",
+			   netdev->dev_addr);
+		ionic_lif_addr(lif, netdev->dev_addr, false);
+	}
+	memcpy(netdev->dev_addr, ctx.comp.lif_getattr.mac,
+	       netdev->addr_len);
+	netdev_dbg(lif->netdev, "adding station MAC addr %pM\n",
+		   netdev->dev_addr);
+	ionic_lif_addr(lif, netdev->dev_addr, true);
+
+	return 0;
+}
+
 static int ionic_lif_init(struct lif *lif)
 {
 	struct ionic_dev *idev = &lif->ionic->idev;
@@ -1039,6 +1412,10 @@ static int ionic_lif_init(struct lif *lif)
 	if (err)
 		goto err_out_notifyq_deinit;
 
+	err = ionic_station_set(lif);
+	if (err)
+		goto err_out_notifyq_deinit;
+
 	set_bit(LIF_INITED, lif->state);
 
 	return 0;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +29 -0
index 9f112aa69033..20b4fa573f77 100644
@@ -60,6 +60,29 @@ struct qcq {
 #define napi_to_qcq(napi)	container_of(napi, struct qcq, napi)
 #define napi_to_cq(napi)	(&napi_to_qcq(napi)->cq)
 
+enum ionic_deferred_work_type {
+	DW_TYPE_RX_MODE,
+	DW_TYPE_RX_ADDR_ADD,
+	DW_TYPE_RX_ADDR_DEL,
+	DW_TYPE_LINK_STATUS,
+	DW_TYPE_LIF_RESET,
+};
+
+struct ionic_deferred_work {
+	struct list_head list;
+	enum ionic_deferred_work_type type;
+	union {
+		unsigned int rx_mode;
+		u8 addr[ETH_ALEN];
+	};
+};
+
+struct ionic_deferred {
+	spinlock_t lock;		/* lock for deferred work list */
+	struct list_head list;
+	struct work_struct work;
+};
+
 enum lif_state_flags {
 	LIF_INITED,
 	LIF_UP,
@@ -87,13 +110,19 @@ struct lif {
 	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
+	unsigned int rx_mode;
 	u64 hw_features;
+	bool mc_overflow;
+	unsigned int nmcast;
+	bool uc_overflow;
+	unsigned int nucast;
 
 	struct lif_info *info;
 	dma_addr_t info_pa;
 	u32 info_sz;
 
 	struct rx_filters rx_filters;
+	struct ionic_deferred deferred;
 	unsigned long *dbid_inuse;
 	unsigned int dbid_count;
 	struct dentry *dentry;
Signature · 1 line
2.17.1
maintainer
#30 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:15 -0700

Quoted 9 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +9 -0
+	if (in_interrupt()) {
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+			return -ENOMEM;
+		}
+		work->type = add ? DW_TYPE_RX_ADDR_ADD : DW_TYPE_RX_ADDR_DEL;
+		memcpy(work->addr, addr, ETH_ALEN);
+		netdev_dbg(lif->netdev, "deferred: rx_filter %s %pM\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_notifyq_clean(struct lif *lif, int budget) · line 340

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+			   add ? "add" : "del", addr);
+		ionic_lif_deferred_enqueue(&lif->deferred, work);
+	} else {
+		netdev_dbg(lif->netdev, "rx_filter %s %pM\n",
+			   add ? "add" : "del", addr);
+		if (add)
+			return ionic_lif_addr_add(lif, addr);
+		else
+			return ionic_lif_addr_del(lif, addr);
+	}

I don't know about this.

Generally interface address changes are expected to be synchronous.

#31 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

Quoted 24 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:15 -0700

Quoted 9 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +9 -0
+	if (in_interrupt()) {
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+			return -ENOMEM;
+		}
+		work->type = add ? DW_TYPE_RX_ADDR_ADD : DW_TYPE_RX_ADDR_DEL;
+		memcpy(work->addr, addr, ETH_ALEN);
+		netdev_dbg(lif->netdev, "deferred: rx_filter %s %pM\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_notifyq_clean(struct lif *lif, int budget) · line 340

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+			   add ? "add" : "del", addr);
+		ionic_lif_deferred_enqueue(&lif->deferred, work);
+	} else {
+		netdev_dbg(lif->netdev, "rx_filter %s %pM\n",
+			   add ? "add" : "del", addr);
+		if (add)
+			return ionic_lif_addr_add(lif, addr);
+		else
+			return ionic_lif_addr_del(lif, addr);
+	}

I don't know about this.

Quoted 1 line

Generally interface address changes are expected to be synchronous.

Yeah, this bothers me a bit as well, but the address change calls come in under spin_lock_bh(), and I'm reluctant to make an AdminQ call under the _bh that could block for a few seconds.

sln

maintainer
#32 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:43 -0700

Quoted 5 lines · David Miller
Quoted 1 line

Generally interface address changes are expected to be synchronous.

Yeah, this bothers me a bit as well, but the address change calls come in under spin_lock_bh(), and I'm reluctant to make an AdminQ call under the _bh that could block for a few seconds.

So it's not about memory allocation but rather the fact that the device might take a while to complete?

Can you start the operation synchronously yet complete it async?

#33 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

Quoted 11 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:43 -0700

Quoted 7 lines · David Miller
Quoted 2 lines

Generally interface address changes are expected to be synchronous.

Yeah, this bothers me a bit as well, but the address change calls come
in under spin_lock_bh(), and I'm reluctant to make an AdminQ call under the _bh that could block for a few seconds.

Quoted 5 lines

So it's not about memory allocation but rather the fact that the device
might take a while to complete?

Can you start the operation synchronously yet complete it async?

The driver is doing busy polling on command completion, doing only busy polling on completion status in the deferred work will not be much different than what they have now..

async completion will only make since if the hardware supports interrupt based (MSI-x) command completion.

#34 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

Quoted 15 lines · Saeed Mahameed
Quoted 14 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:43 -0700

Quoted 7 lines · David Miller
Quoted 2 lines

Generally interface address changes are expected to be synchronous.

Yeah, this bothers me a bit as well, but the address change calls come
in under spin_lock_bh(), and I'm reluctant to make an AdminQ call under the _bh that could block for a few seconds.

So it's not about memory allocation but rather the fact that the device
might take a while to complete?

Quoted 7 lines
Quoted 1 line

Can you start the operation synchronously yet complete it async?

The driver is doing busy polling on command completion, doing only busy polling on completion status in the deferred work will not be much different than what they have now..

async completion will only make since if the hardware supports interrupt based (MSI-x) command completion.

Actually, there are two different command paths - one for basic low-level setup, and one for more advanced work:
- dev_cmd does indeed do polling on PCI register space, and we try to keep this to a minimum
- adminq does a wait_for_completion_timeout() which gets completed in an MSI-x handler

The rx_mode related work goes through adminq.

Yes, it could be made async, but would the extra logic needed buy us that much?  We already know this model works in other drivers.  We wouldn't get the address into the hardware any quicker, and we still wouldn't get any errors from the request back to the original call.

sln

#35 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

In reply to David Miller
Quoted 10 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Tue, 23 Jul 2019 15:50:43 -0700

Quoted 5 lines · David Miller
Quoted 1 line

Generally interface address changes are expected to be synchronous.

Yeah, this bothers me a bit as well, but the address change calls come in under spin_lock_bh(), and I'm reluctant to make an AdminQ call under the _bh that could block for a few seconds.

So it's not about memory allocation but rather the fact that the device might take a while to complete?

Memory allocation may or may not be involved, but yes, mainly we're doing another spin_lock on a firmware command that waits for an ACK or ERROR answer, and in extreme cases could possibly timeout on a dead firmware.  I know that i40e and ice do much the same thing, and I believe mlx5 as well, for the same reasons.  I suspect others do as well.

Quoted 1 line

Can you start the operation synchronously yet complete it async?

This could be possible, but would likely require a bunch more messy logic to track async AdminQ requests, that otherwise is unnecessary.

sln

#36 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

In reply to David Miller
Quoted 26 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:15 -0700

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +7 -0
+	if (in_interrupt()) {
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+			return -ENOMEM;
+		}
+		work->type = add ? DW_TYPE_RX_ADDR_ADD :

DW_TYPE_RX_ADDR_DEL;

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+		memcpy(work->addr, addr, ETH_ALEN);
+		netdev_dbg(lif->netdev, "deferred: rx_filter %s %pM\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_notifyq_clean(struct lif *lif, int budget) · line 340

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+			   add ? "add" : "del", addr);
+		ionic_lif_deferred_enqueue(&lif->deferred, work);
+	} else {
+		netdev_dbg(lif->netdev, "rx_filter %s %pM\n",
+			   add ? "add" : "del", addr);
+		if (add)
+			return ionic_lif_addr_add(lif, addr);
+		else
+			return ionic_lif_addr_del(lif, addr);
+	}

I don't know about this.

Quoted 1 line

Generally interface address changes are expected to be synchronous.

Well, drivers can't sleep on set_rx_mode ndo, dev_set_rx_mode is holding netif_addr_lock_bh.

Some drivers need to wait for firmware/device completion to program the new address and they have to do this asynchronously to avoid atomic context. I assume this driver is having the same issue due to "ionic_adminq_post_wait()" ..

#37 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

In reply to Shannon Nelson
Quoted 447 lines · Shannon Nelson

Add the Rx filtering and rx_mode NDO callbacks. Also add the deferred work thread handling needed to manage the filter requests otuside of the netif_addr_lock spinlock.

---
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 389
+++++++++++++++++-
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  29 ++
 2 files changed, 412 insertions(+), 6 deletions(-)
drivers/net/ethernet/pensando/ionic/ionic_lif.c +39 -0
index 72ac0fd56b69..efcda1337f91 100644
@@ -12,9 +12,55 @@
 #include "ionic_lif.h"
 #include "ionic_debugfs.h"
 
+static void ionic_lif_rx_mode(struct lif *lif, unsigned int
 rx_mode);
+static int ionic_lif_addr_add(struct lif *lif, const u8 *addr);
+static int ionic_lif_addr_del(struct lif *lif, const u8 *addr);
+
 static int ionic_set_nic_features(struct lif *lif, netdev_features_t
 features);
 static int ionic_notifyq_clean(struct lif *lif, int budget);
 
+static void ionic_lif_deferred_work(struct work_struct *work)
+{
+	struct lif *lif = container_of(work, struct lif,
 deferred.work);
+	struct ionic_deferred *def = &lif->deferred;
+	struct ionic_deferred_work *w = NULL;
+
+	spin_lock_bh(&def->lock);
+	if (!list_empty(&def->list)) {
+		w = list_first_entry(&def->list,
+				     struct ionic_deferred_work, list);
+		list_del(&w->list);
+	}
+	spin_unlock_bh(&def->lock);
+
+	if (w) {
+		switch (w->type) {
+		case DW_TYPE_RX_MODE:
+			ionic_lif_rx_mode(lif, w->rx_mode);
+			break;
+		case DW_TYPE_RX_ADDR_ADD:
+			ionic_lif_addr_add(lif, w->addr);
+			break;
+		case DW_TYPE_RX_ADDR_DEL:
+			ionic_lif_addr_del(lif, w->addr);
+			break;
+		default:
+			break;
+		}
+		kfree(w);
+		schedule_work(&def->work);
+	}
+}
+
+static void ionic_lif_deferred_enqueue(struct ionic_deferred *def,
+				       struct ionic_deferred_work

*work)

drivers/net/ethernet/pensando/ionic/ionic_lif.c +80 -0
+{
+	spin_lock_bh(&def->lock);
+	list_add_tail(&work->list, &def->list);
+	spin_unlock_bh(&def->lock);
+	schedule_work(&def->work);
+}
+
 int ionic_open(struct net_device *netdev)
 {
 	struct lif *lif = netdev_priv(netdev);
@@ -180,6 +226,235 @@ static int ionic_notifyq_clean(struct lif *lif,
 int budget)
 	return work_done;
 }
 
+static int ionic_lif_addr_add(struct lif *lif, const u8 *addr)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_add = {
+			.opcode = CMD_OPCODE_RX_FILTER_ADD,
+			.lif_index = cpu_to_le16(lif->index),
+			.match = cpu_to_le16(RX_FILTER_MATCH_MAC),
+		},
+	};
+	struct rx_filter *f;
+	int err;
+
+	/* don't bother if we already have it */
+	spin_lock_bh(&lif->rx_filters.lock);
+	f = ionic_rx_filter_by_addr(lif, addr);
+	spin_unlock_bh(&lif->rx_filters.lock);
+	if (f)
+		return 0;
+
+	netdev_dbg(lif->netdev, "rx_filter add ADDR %pM (id %d)\n",
 addr,
+		   ctx.comp.rx_filter_add.filter_id);
+
+	memcpy(ctx.cmd.rx_filter_add.mac.addr, addr, ETH_ALEN);
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	return ionic_rx_filter_save(lif, 0, RXQ_INDEX_ANY, 0, &ctx);
+}
+
+static int ionic_lif_addr_del(struct lif *lif, const u8 *addr)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_del = {
+			.opcode = CMD_OPCODE_RX_FILTER_DEL,
+			.lif_index = cpu_to_le16(lif->index),
+		},
+	};
+	struct rx_filter *f;
+	int err;
+
+	spin_lock_bh(&lif->rx_filters.lock);
+	f = ionic_rx_filter_by_addr(lif, addr);
+	if (!f) {
+		spin_unlock_bh(&lif->rx_filters.lock);
+		return -ENOENT;
+	}
+
+	ctx.cmd.rx_filter_del.filter_id = cpu_to_le32(f->filter_id);
+	ionic_rx_filter_free(lif, f);
+	spin_unlock_bh(&lif->rx_filters.lock);
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	netdev_dbg(lif->netdev, "rx_filter del ADDR %pM (id %d)\n",
 addr,
+		   ctx.cmd.rx_filter_del.filter_id);
+
+	return 0;
+}
+
+static int ionic_lif_addr(struct lif *lif, const u8 *addr, bool add)
+{
+	struct ionic *ionic = lif->ionic;
+	struct ionic_deferred_work *work;
+	unsigned int nmfilters;
+	unsigned int nufilters;
+
+	if (add) {
+		/* Do we have space for this filter?  We test the

counters

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+		 * here before checking the need for deferral so that

we

drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
+		 * can return an overflow error to the stack.
+		 */
+		nmfilters = le32_to_cpu(ionic-
Quoted 1 line

ident.lif.eth.max_mcast_filters);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+		nufilters = le32_to_cpu(ionic-
Quoted 1 line

ident.lif.eth.max_ucast_filters);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+
+		if ((is_multicast_ether_addr(addr) && lif->nmcast <

nmfilters))

drivers/net/ethernet/pensando/ionic/ionic_lif.c +20 -0
+			lif->nmcast++;
+		else if (!is_multicast_ether_addr(addr) &&
+			 lif->nucast < nufilters)
+			lif->nucast++;
+		else
+			return -ENOSPC;
+	} else {
+		if (is_multicast_ether_addr(addr) && lif->nmcast)
+			lif->nmcast--;
+		else if (!is_multicast_ether_addr(addr) && lif->nucast)
+			lif->nucast--;
+	}
+
+	if (in_interrupt()) {
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+			return -ENOMEM;
+		}
+		work->type = add ? DW_TYPE_RX_ADDR_ADD :

DW_TYPE_RX_ADDR_DEL;

drivers/net/ethernet/pensando/ionic/ionic_lif.c +45 -0
+		memcpy(work->addr, addr, ETH_ALEN);
+		netdev_dbg(lif->netdev, "deferred: rx_filter %s %pM\n",
+			   add ? "add" : "del", addr);
+		ionic_lif_deferred_enqueue(&lif->deferred, work);
+	} else {
+		netdev_dbg(lif->netdev, "rx_filter %s %pM\n",
+			   add ? "add" : "del", addr);
+		if (add)
+			return ionic_lif_addr_add(lif, addr);
+		else
+			return ionic_lif_addr_del(lif, addr);
+	}
+
+	return 0;
+}
+
+static int ionic_addr_add(struct net_device *netdev, const u8 *addr)
+{
+	return ionic_lif_addr(netdev_priv(netdev), addr, true);
+}
+
+static int ionic_addr_del(struct net_device *netdev, const u8 *addr)
+{
+	return ionic_lif_addr(netdev_priv(netdev), addr, false);
+}
+
+static void ionic_lif_rx_mode(struct lif *lif, unsigned int rx_mode)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_mode_set = {
+			.opcode = CMD_OPCODE_RX_MODE_SET,
+			.lif_index = cpu_to_le16(lif->index),
+			.rx_mode = cpu_to_le16(rx_mode),
+		},
+	};
+	char buf[128];
+	int err;
+	int i;
+#define REMAIN(__x) (sizeof(buf) - (__x))
+
+	i = snprintf(buf, sizeof(buf), "rx_mode 0x%04x -> 0x%04x:",
+		     lif->rx_mode, rx_mode);
+	if (rx_mode & RX_MODE_F_UNICAST)
+		i += snprintf(&buf[i], REMAIN(i), "

RX_MODE_F_UNICAST");

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+	if (rx_mode & RX_MODE_F_MULTICAST)
+		i += snprintf(&buf[i], REMAIN(i), "

RX_MODE_F_MULTICAST");

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+	if (rx_mode & RX_MODE_F_BROADCAST)
+		i += snprintf(&buf[i], REMAIN(i), "

RX_MODE_F_BROADCAST");

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+	if (rx_mode & RX_MODE_F_PROMISC)
+		i += snprintf(&buf[i], REMAIN(i), "

RX_MODE_F_PROMISC");

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
+	if (rx_mode & RX_MODE_F_ALLMULTI)
+		i += snprintf(&buf[i], REMAIN(i), "

RX_MODE_F_ALLMULTI");

drivers/net/ethernet/pensando/ionic/ionic_lif.c +5 -0
+	netdev_dbg(lif->netdev, "lif%d %s\n", lif->index, buf);
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		netdev_warn(lif->netdev, "set rx_mode 0x%04x failed:

%d\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c +6 -0
+			    rx_mode, err);
+	else
+		lif->rx_mode = rx_mode;
+}
+
+static void _ionic_lif_rx_mode(struct lif *lif, unsigned int

rx_mode)

drivers/net/ethernet/pensando/ionic/ionic_lif.c +27 -0
+{
+	struct ionic_deferred_work *work;
+
+	if (in_interrupt()) {
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+			return;
+		}
+		work->type = DW_TYPE_RX_MODE;
+		work->rx_mode = rx_mode;
+		netdev_dbg(lif->netdev, "deferred: rx_mode\n");
+		ionic_lif_deferred_enqueue(&lif->deferred, work);
+	} else {
+		ionic_lif_rx_mode(lif, rx_mode);
+	}
+}
+
+static void ionic_set_rx_mode(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct identity *ident = &lif->ionic->ident;
+	unsigned int nfilters;
+	unsigned int rx_mode;
+
+	rx_mode = RX_MODE_F_UNICAST;
+	rx_mode |= (netdev->flags & IFF_MULTICAST) ?

RX_MODE_F_MULTICAST : 0;

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+	rx_mode |= (netdev->flags & IFF_BROADCAST) ?

RX_MODE_F_BROADCAST : 0;

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+	rx_mode |= (netdev->flags & IFF_PROMISC) ? RX_MODE_F_PROMISC :

0;

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+	rx_mode |= (netdev->flags & IFF_ALLMULTI) ? RX_MODE_F_ALLMULTI

: 0;

drivers/net/ethernet/pensando/ionic/ionic_lif.c +55 -2
+
+	/* sync unicast addresses
+	 * next check to see if we're in an overflow state
+	 *    if so, we track that we overflowed and enable NIC PROMISC
+	 *    else if the overflow is set and not needed
+	 *       we remove our overflow flag and check the netdev flags
+	 *       to see if we can disable NIC PROMISC
+	 */
+	__dev_uc_sync(netdev, ionic_addr_add, ionic_addr_del);
+	nfilters = le32_to_cpu(ident->lif.eth.max_ucast_filters);
+	if (netdev_uc_count(netdev) + 1 > nfilters) {
+		rx_mode |= RX_MODE_F_PROMISC;
+		lif->uc_overflow = true;
+	} else if (lif->uc_overflow) {
+		lif->uc_overflow = false;
+		if (!(netdev->flags & IFF_PROMISC))
+			rx_mode &= ~RX_MODE_F_PROMISC;
+	}
+
+	/* same for multicast */
+	__dev_mc_sync(netdev, ionic_addr_add, ionic_addr_del);
+	nfilters = le32_to_cpu(ident->lif.eth.max_mcast_filters);
+	if (netdev_mc_count(netdev) > nfilters) {
+		rx_mode |= RX_MODE_F_ALLMULTI;
+		lif->mc_overflow = true;
+	} else if (lif->mc_overflow) {
+		lif->mc_overflow = false;
+		if (!(netdev->flags & IFF_ALLMULTI))
+			rx_mode &= ~RX_MODE_F_ALLMULTI;
+	}
+
+	if (lif->rx_mode != rx_mode)
+		_ionic_lif_rx_mode(lif, rx_mode);
+}
+
 static int ionic_set_features(struct net_device *netdev,
 			      netdev_features_t features)
 {
@@ -196,8 +471,26 @@ static int ionic_set_features(struct net_device
 *netdev,
 
 static int ionic_set_mac_address(struct net_device *netdev, void
 *sa)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
-	return 0;
+	struct sockaddr *addr = sa;
+	u8 *mac;
+
+	mac = (u8 *)addr->sa_data;
+	if (ether_addr_equal(netdev->dev_addr, mac))
+		return 0;
+
+	if (!is_valid_ether_addr(mac))
+		return -EADDRNOTAVAIL;
+
+	if (!is_zero_ether_addr(netdev->dev_addr)) {
+		netdev_info(netdev, "deleting mac addr %pM\n",
+			    netdev->dev_addr);
+		ionic_addr_del(netdev, netdev->dev_addr);
+	}
+
+	memcpy(netdev->dev_addr, mac, netdev->addr_len);
+	netdev_info(netdev, "updating mac addr %pM\n", mac);
+
+	return ionic_addr_add(netdev, mac);
 }

static int ionic_change_mtu(struct net_device *netdev, int new_mtu)

drivers/net/ethernet/pensando/ionic/ionic_lif.c +47 -4
@@ -232,20 +525,63 @@ static void ionic_tx_timeout(struct net_device
 *netdev)
 static int ionic_vlan_rx_add_vid(struct net_device *netdev, __be16
 proto,
 				 u16 vid)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
-	return 0;
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_add = {
+			.opcode = CMD_OPCODE_RX_FILTER_ADD,
+			.lif_index = cpu_to_le16(lif->index),
+			.match = cpu_to_le16(RX_FILTER_MATCH_VLAN),
+			.vlan.vlan = cpu_to_le16(vid),
+		},
+	};
+	int err;
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	netdev_dbg(netdev, "rx_filter add VLAN %d (id %d)\n", vid,
+		   ctx.comp.rx_filter_add.filter_id);
+
+	return ionic_rx_filter_save(lif, 0, RXQ_INDEX_ANY, 0, &ctx);
 }
 
 static int ionic_vlan_rx_kill_vid(struct net_device *netdev, __be16
 proto,
 				  u16 vid)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
-	return 0;
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.rx_filter_del = {
+			.opcode = CMD_OPCODE_RX_FILTER_DEL,
+			.lif_index = cpu_to_le16(lif->index),
+		},
+	};
+	struct rx_filter *f;
+
+	spin_lock_bh(&lif->rx_filters.lock);
+
+	f = ionic_rx_filter_by_vlan(lif, vid);
+	if (!f) {
+		spin_unlock_bh(&lif->rx_filters.lock);
+		return -ENOENT;
+	}
+
+	netdev_dbg(netdev, "rx_filter del VLAN %d (id %d)\n", vid,
+		   le32_to_cpu(ctx.cmd.rx_filter_del.filter_id));
+
+	ctx.cmd.rx_filter_del.filter_id = cpu_to_le32(f->filter_id);
+	ionic_rx_filter_free(lif, f);
+	spin_unlock_bh(&lif->rx_filters.lock);
+
+	return ionic_adminq_post_wait(lif, &ctx);
 }
 
 static const struct net_device_ops ionic_netdev_ops = {
 	.ndo_open               = ionic_open,
 	.ndo_stop               = ionic_stop,
+	.ndo_set_rx_mode	= ionic_set_rx_mode,
drivers/net/ethernet/pensando/ionic/ionic_lif.c +4 -0
 	.ndo_set_features	= ionic_set_features,
 	.ndo_set_mac_address	= ionic_set_mac_address,
 	.ndo_validate_addr	= eth_validate_addr,
@@ -546,6 +882,10 @@ static struct lif *ionic_lif_alloc(struct ionic
 *ionic, unsigned int index)
 
 	spin_lock_init(&lif->adminq_lock);
 
+	spin_lock_init(&lif->deferred.lock);
+	INIT_LIST_HEAD(&lif->deferred.list);
+	INIT_WORK(&lif->deferred.work, ionic_lif_deferred_work);
+
 	/* allocate lif info */
 	lif->info_sz = ALIGN(sizeof(*lif->info), PAGE_SIZE);
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
 	lif->info = dma_alloc_coherent(dev, lif->info_sz,

drivers/net/ethernet/pensando/ionic/ionic_lif.c static void ionic_lif_free(struct lif *lif) · line 949

Quoted 3 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
@@ -607,6 +947,8 @@ static void ionic_lif_free(struct lif *lif)
 	ionic_qcqs_free(lif);
 	ionic_lif_reset(lif);
 

I don't think you want deferred.work running while reset is executing.. cancel_work_sync should happen as early as you close the netdevice.

I assume ionic_lif_reset will flush all configurations and you don't need to cleanup anything manually? or any data structure stored in driver ?

Quoted 111 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +33 -0
+	cancel_work_sync(&lif->deferred.work);
+
 	/* free lif info */
 	dma_free_coherent(dev, lif->info_sz, lif->info, lif->info_pa);
 	lif->info = NULL;
@@ -975,6 +1317,37 @@ static int ionic_init_nic_features(struct lif
 *lif)
 	return 0;
 }
 
+static int ionic_station_set(struct lif *lif)
+{
+	struct net_device *netdev = lif->netdev;
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.lif_getattr = {
+			.opcode = CMD_OPCODE_LIF_GETATTR,
+			.index = cpu_to_le16(lif->index),
+			.attr = IONIC_LIF_ATTR_MAC,
+		},
+	};
+	int err;
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	if (!is_zero_ether_addr(netdev->dev_addr)) {
+		netdev_dbg(lif->netdev, "deleting station MAC addr
 %pM\n",
+			   netdev->dev_addr);
+		ionic_lif_addr(lif, netdev->dev_addr, false);
+	}
+	memcpy(netdev->dev_addr, ctx.comp.lif_getattr.mac,
+	       netdev->addr_len);
+	netdev_dbg(lif->netdev, "adding station MAC addr %pM\n",
+		   netdev->dev_addr);
+	ionic_lif_addr(lif, netdev->dev_addr, true);
+
+	return 0;
+}
+
 static int ionic_lif_init(struct lif *lif)

{

drivers/net/ethernet/pensando/ionic/ionic_lif.c +4 -0
 	struct ionic_dev *idev = &lif->ionic->idev;
@@ -1039,6 +1412,10 @@ static int ionic_lif_init(struct lif *lif)
 	if (err)
 		goto err_out_notifyq_deinit;
 
+	err = ionic_station_set(lif);
+	if (err)
+		goto err_out_notifyq_deinit;
+
 	set_bit(LIF_INITED, lif->state);
 
 	return 0;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +29 -0
index 9f112aa69033..20b4fa573f77 100644
@@ -60,6 +60,29 @@ struct qcq {
 #define napi_to_qcq(napi)	container_of(napi, struct qcq, napi)
 #define napi_to_cq(napi)	(&napi_to_qcq(napi)->cq)
 
+enum ionic_deferred_work_type {
+	DW_TYPE_RX_MODE,
+	DW_TYPE_RX_ADDR_ADD,
+	DW_TYPE_RX_ADDR_DEL,
+	DW_TYPE_LINK_STATUS,
+	DW_TYPE_LIF_RESET,
+};
+
+struct ionic_deferred_work {
+	struct list_head list;
+	enum ionic_deferred_work_type type;
+	union {
+		unsigned int rx_mode;
+		u8 addr[ETH_ALEN];
+	};
+};
+
+struct ionic_deferred {
+	spinlock_t lock;		/* lock for deferred work list */
+	struct list_head list;
+	struct work_struct work;
+};
+
 enum lif_state_flags {
 	LIF_INITED,
 	LIF_UP,
@@ -87,13 +110,19 @@ struct lif {
 	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
+	unsigned int rx_mode;
 	u64 hw_features;
+	bool mc_overflow;
+	unsigned int nmcast;
+	bool uc_overflow;
+	unsigned int nucast;
 
 	struct lif_info *info;
 	dma_addr_t info_pa;
 	u32 info_sz;
 
 	struct rx_filters rx_filters;
+	struct ionic_deferred deferred;
 	unsigned long *dbid_inuse;
 	unsigned int dbid_count;
 	struct dentry *dentry;
#38 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

Quoted 9 lines · Saeed Mahameed

drivers/net/ethernet/pensando/ionic/ionic_lif.c static void ionic_lif_free(struct lif *lif) · line 949

Quoted 6 lines · Shannon Nelson
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
@@ -607,6 +947,8 @@ static void ionic_lif_free(struct lif *lif)
  	ionic_qcqs_free(lif);
  	ionic_lif_reset(lif);
 

I don't think you want deferred.work running while reset is executing.. cancel_work_sync should happen as early as you close the netdevice.

Given the current implementation, it doesn't actually hurt anything, but yes it makes sense to move it up in the sequence.

Quoted 3 lines

I assume ionic_lif_reset will flush all configurations and you don't need to cleanup anything manually? or any data structure stored in driver ?

Most of the driver data structure cleaning has happened in ionic_lif_deinit() before getting here.

sln

#39 lore raw

Re: [PATCH v4 net-next 11/19] ionic: Add Rx filter and rx_mode ndo support

Quoted 14 lines · Shannon Nelson
Quoted 8 lines · Saeed Mahameed

drivers/net/ethernet/pensando/ionic/ionic_lif.c static void ionic_lif_free(struct lif *lif) · line 949

Quoted 4 lines · Shannon Nelson
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
@@ -607,6 +947,8 @@ static void ionic_lif_free(struct lif *lif)
  	ionic_qcqs_free(lif);
  	ionic_lif_reset(lif);
 

I don't think you want deferred.work running while reset is executing..
cancel_work_sync should happen as early as you close the netdevice.

Given the current implementation, it doesn't actually hurt anything, but
yes it makes sense to move it up in the sequence.

Quoted 7 lines
Quoted 4 lines

I assume ionic_lif_reset will flush all configurations and you don't
need to cleanup anything manually? or any data structure stored in driver ?

Most of the driver data structure cleaning has happened in ionic_lif_deinit() before getting here.

This means that yo do have a problem keeping the deferred.work running after ionic_lif_deinit

Quoted 3 lines

sln

#40 lore raw

PATCH v4 12/19 ionic: Add async link status check and basic stats

In reply to Shannon Nelson

Add code to handle the link status event, and wire up the basic netdev hardware stats.

---
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 116 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   1 +
 2 files changed, 117 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic_lif.c +116 -0
index efcda1337f91..f52af9cb6264 100644
@@ -15,6 +15,7 @@
 static void ionic_lif_rx_mode(struct lif *lif, unsigned int rx_mode);
 static int ionic_lif_addr_add(struct lif *lif, const u8 *addr);
 static int ionic_lif_addr_del(struct lif *lif, const u8 *addr);
+static void ionic_link_status_check(struct lif *lif);
 
 static int ionic_set_nic_features(struct lif *lif, netdev_features_t features);
 static int ionic_notifyq_clean(struct lif *lif, int budget);
@@ -44,6 +45,9 @@ static void ionic_lif_deferred_work(struct work_struct *work)
 		case DW_TYPE_RX_ADDR_DEL:
 			ionic_lif_addr_del(lif, w->addr);
 			break;
+		case DW_TYPE_LINK_STATUS:
+			ionic_link_status_check(lif);
+			break;
 		default:
 			break;
 		}
@@ -69,6 +73,7 @@ int ionic_open(struct net_device *netdev)
 
 	set_bit(LIF_UP, lif->state);
 
+	ionic_link_status_check(lif);
 	if (netif_carrier_ok(netdev))
 		netif_tx_wake_all_queues(netdev);
 
@@ -151,6 +156,39 @@ static int ionic_adminq_napi(struct napi_struct *napi, int budget)
 	return max(n_work, a_work);
 }
 
+static void ionic_link_status_check(struct lif *lif)
+{
+	struct net_device *netdev = lif->netdev;
+	u16 link_status;
+	bool link_up;
+
+	clear_bit(LIF_LINK_CHECK_NEEDED, lif->state);
+
+	link_status = le16_to_cpu(lif->info->status.link_status);
+	link_up = link_status == PORT_OPER_STATUS_UP;
+
+	/* filter out the no-change cases */
+	if (link_up == netif_carrier_ok(netdev))
+		return;
+
+	if (link_up) {
+		netdev_info(netdev, "Link up - %d Gbps\n",
+			    le32_to_cpu(lif->info->status.link_speed) / 1000);
+
+		if (test_bit(LIF_UP, lif->state)) {
+			netif_tx_wake_all_queues(lif->netdev);
+			netif_carrier_on(netdev);
+		}
+	} else {
+		netdev_info(netdev, "Link down\n");
+
+		/* carrier off first to avoid watchdog timeout */
+		netif_carrier_off(netdev);
+		if (test_bit(LIF_UP, lif->state))
+			netif_tx_stop_all_queues(netdev);
+	}
+}
+
 static bool ionic_notifyq_service(struct cq *cq, struct cq_info *cq_info)
 {
 	union notifyq_comp *comp = cq_info->cq_desc;
@@ -182,6 +220,9 @@ static bool ionic_notifyq_service(struct cq *cq, struct cq_info *cq_info)
 			    "  link_status=%d link_speed=%d\n",
 			    le16_to_cpu(comp->link_change.link_status),
 			    le32_to_cpu(comp->link_change.link_speed));
+
+		set_bit(LIF_LINK_CHECK_NEEDED, lif->state);
+
 		break;
 	case EVENT_OPCODE_RESET:
 		netdev_info(netdev, "Notifyq EVENT_OPCODE_RESET eid=%lld\n",
@@ -222,10 +263,81 @@ static int ionic_notifyq_clean(struct lif *lif, int budget)
 	if (work_done == budget)
 		goto return_to_napi;
 
+	/* After outstanding events are processed we can check on
+	 * the link status and any outstanding interrupt credits.
+	 *
+	 * We wait until here to check on the link status in case
+	 * there was a long list of link events from a flap episode.
+	 */
+	if (test_bit(LIF_LINK_CHECK_NEEDED, lif->state)) {
+		struct ionic_deferred_work *work;
+
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);
+		} else {
+			work->type = DW_TYPE_LINK_STATUS;
+			ionic_lif_deferred_enqueue(&lif->deferred, work);
+		}
+	}
+
 return_to_napi:
 	return work_done;
 }
 
+static void ionic_get_stats64(struct net_device *netdev,
+			      struct rtnl_link_stats64 *ns)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct lif_stats *ls;
+
+	memset(ns, 0, sizeof(*ns));
+	ls = &lif->info->stats;
+
+	ns->rx_packets = le64_to_cpu(ls->rx_ucast_packets) +
+			 le64_to_cpu(ls->rx_mcast_packets) +
+			 le64_to_cpu(ls->rx_bcast_packets);
+
+	ns->tx_packets = le64_to_cpu(ls->tx_ucast_packets) +
+			 le64_to_cpu(ls->tx_mcast_packets) +
+			 le64_to_cpu(ls->tx_bcast_packets);
+
+	ns->rx_bytes = le64_to_cpu(ls->rx_ucast_bytes) +
+		       le64_to_cpu(ls->rx_mcast_bytes) +
+		       le64_to_cpu(ls->rx_bcast_bytes);
+
+	ns->tx_bytes = le64_to_cpu(ls->tx_ucast_bytes) +
+		       le64_to_cpu(ls->tx_mcast_bytes) +
+		       le64_to_cpu(ls->tx_bcast_bytes);
+
+	ns->rx_dropped = le64_to_cpu(ls->rx_ucast_drop_packets) +
+			 le64_to_cpu(ls->rx_mcast_drop_packets) +
+			 le64_to_cpu(ls->rx_bcast_drop_packets);
+
+	ns->tx_dropped = le64_to_cpu(ls->tx_ucast_drop_packets) +
+			 le64_to_cpu(ls->tx_mcast_drop_packets) +
+			 le64_to_cpu(ls->tx_bcast_drop_packets);
+
+	ns->multicast = le64_to_cpu(ls->rx_mcast_packets);
+
+	ns->rx_over_errors = le64_to_cpu(ls->rx_queue_empty);
+
+	ns->rx_missed_errors = le64_to_cpu(ls->rx_dma_error) +
+			       le64_to_cpu(ls->rx_queue_disabled) +
+			       le64_to_cpu(ls->rx_desc_fetch_error) +
+			       le64_to_cpu(ls->rx_desc_data_error);
+
+	ns->tx_aborted_errors = le64_to_cpu(ls->tx_dma_error) +
+				le64_to_cpu(ls->tx_queue_disabled) +
+				le64_to_cpu(ls->tx_desc_fetch_error) +
+				le64_to_cpu(ls->tx_desc_data_error);
+
+	ns->rx_errors = ns->rx_over_errors +
+			ns->rx_missed_errors;
+
+	ns->tx_errors = ns->tx_aborted_errors;
+}
+
 static int ionic_lif_addr_add(struct lif *lif, const u8 *addr)
 {
 	struct ionic_admin_ctx ctx = {
@@ -581,6 +693,7 @@ static int ionic_vlan_rx_kill_vid(struct net_device *netdev, __be16 proto,
 static const struct net_device_ops ionic_netdev_ops = {
 	.ndo_open               = ionic_open,
 	.ndo_stop               = ionic_stop,
+	.ndo_get_stats64	= ionic_get_stats64,
 	.ndo_set_rx_mode	= ionic_set_rx_mode,
 	.ndo_set_features	= ionic_set_features,
 	.ndo_set_mac_address	= ionic_set_mac_address,
@@ -1418,6 +1531,8 @@ static int ionic_lif_init(struct lif *lif)
 
 	set_bit(LIF_INITED, lif->state);
 
+	ionic_link_status_check(lif);
+
 	return 0;
 
 err_out_notifyq_deinit:
@@ -1461,6 +1576,7 @@ int ionic_lifs_register(struct ionic *ionic)
 		return err;
 	}
 
+	ionic_link_status_check(ionic->master_lif);
 	ionic->master_lif->registered = true;
 
 	return 0;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +1 -0
index 20b4fa573f77..9930b9390c8a 100644
@@ -86,6 +86,7 @@ struct ionic_deferred {
 enum lif_state_flags {
 	LIF_INITED,
 	LIF_UP,
+	LIF_LINK_CHECK_NEEDED,
 	LIF_QUEUE_RESET,
 
 	/* leave this as last */
Signature · 1 line
2.17.1
#41 lore raw

Re: [PATCH v4 net-next 12/19] ionic: Add async link status check and basic stats

Quoted 106 lines · Shannon Nelson

Add code to handle the link status event, and wire up the basic netdev hardware stats.

---
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 116
++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   1 +
 2 files changed, 117 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
index efcda1337f91..f52af9cb6264 100644
@@ -15,6 +15,7 @@
 static void ionic_lif_rx_mode(struct lif *lif, unsigned int
 rx_mode);
 static int ionic_lif_addr_add(struct lif *lif, const u8 *addr);
 static int ionic_lif_addr_del(struct lif *lif, const u8 *addr);
+static void ionic_link_status_check(struct lif *lif);
 
 static int ionic_set_nic_features(struct lif *lif, netdev_features_t

features);
static int ionic_notifyq_clean(struct lif *lif, int budget);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
@@ -44,6 +45,9 @@ static void ionic_lif_deferred_work(struct
 work_struct *work)
 		case DW_TYPE_RX_ADDR_DEL:
 			ionic_lif_addr_del(lif, w->addr);
 			break;
+		case DW_TYPE_LINK_STATUS:
+			ionic_link_status_check(lif);
+			break;
 		default:
 			break;
drivers/net/ethernet/pensando/ionic/ionic_lif.c +34 -0
 		}
@@ -69,6 +73,7 @@ int ionic_open(struct net_device *netdev)
 
 	set_bit(LIF_UP, lif->state);
 
+	ionic_link_status_check(lif);
 	if (netif_carrier_ok(netdev))
 		netif_tx_wake_all_queues(netdev);
 
@@ -151,6 +156,39 @@ static int ionic_adminq_napi(struct napi_struct
 *napi, int budget)
 	return max(n_work, a_work);
 }
 
+static void ionic_link_status_check(struct lif *lif)
+{
+	struct net_device *netdev = lif->netdev;
+	u16 link_status;
+	bool link_up;
+
+	clear_bit(LIF_LINK_CHECK_NEEDED, lif->state);
+
+	link_status = le16_to_cpu(lif->info->status.link_status);
+	link_up = link_status == PORT_OPER_STATUS_UP;
+
+	/* filter out the no-change cases */
+	if (link_up == netif_carrier_ok(netdev))
+		return;
+
+	if (link_up) {
+		netdev_info(netdev, "Link up - %d Gbps\n",
+			    le32_to_cpu(lif->info->status.link_speed) /
 1000);
+
+		if (test_bit(LIF_UP, lif->state)) {
+			netif_tx_wake_all_queues(lif->netdev);
+			netif_carrier_on(netdev);
+		}
+	} else {
+		netdev_info(netdev, "Link down\n");
+
+		/* carrier off first to avoid watchdog timeout */
+		netif_carrier_off(netdev);
+		if (test_bit(LIF_UP, lif->state))
+			netif_tx_stop_all_queues(netdev);
+	}
+}
+
 static bool ionic_notifyq_service(struct cq *cq, struct cq_info

*cq_info)
{

drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
 	union notifyq_comp *comp = cq_info->cq_desc;
@@ -182,6 +220,9 @@ static bool ionic_notifyq_service(struct cq *cq,
 struct cq_info *cq_info)
 			    "  link_status=%d link_speed=%d\n",
 			    le16_to_cpu(comp->link_change.link_status),
 			    le32_to_cpu(comp->link_change.link_speed));
+
+		set_bit(LIF_LINK_CHECK_NEEDED, lif->state);
+
 		break;
 	case EVENT_OPCODE_RESET:

netdev_info(netdev, "Notifyq EVENT_OPCODE_RESET
eid=%lld\n",

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
@@ -222,10 +263,81 @@ static int ionic_notifyq_clean(struct lif *lif,
 int budget)
 	if (work_done == budget)
 		goto return_to_napi;
 
+	/* After outstanding events are processed we can check on
+	 * the link status and any outstanding interrupt credits.

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_notifyq_clean(struct lif *lif, · line 278

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+	 *
+	 * We wait until here to check on the link status in case
+	 * there was a long list of link events from a flap episode.
+	 */
+	if (test_bit(LIF_LINK_CHECK_NEEDED, lif->state)) {
+		struct ionic_deferred_work *work;
+
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);

why not having a pre allocated dedicated lif->link_check_work, instead of allocating in atomic context on every link check event ?

Quoted 86 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +59 -0
+		} else {
+			work->type = DW_TYPE_LINK_STATUS;
+			ionic_lif_deferred_enqueue(&lif->deferred,
 work);
+		}
+	}
+
 return_to_napi:
 	return work_done;
 }
 
+static void ionic_get_stats64(struct net_device *netdev,
+			      struct rtnl_link_stats64 *ns)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct lif_stats *ls;
+
+	memset(ns, 0, sizeof(*ns));
+	ls = &lif->info->stats;
+
+	ns->rx_packets = le64_to_cpu(ls->rx_ucast_packets) +
+			 le64_to_cpu(ls->rx_mcast_packets) +
+			 le64_to_cpu(ls->rx_bcast_packets);
+
+	ns->tx_packets = le64_to_cpu(ls->tx_ucast_packets) +
+			 le64_to_cpu(ls->tx_mcast_packets) +
+			 le64_to_cpu(ls->tx_bcast_packets);
+
+	ns->rx_bytes = le64_to_cpu(ls->rx_ucast_bytes) +
+		       le64_to_cpu(ls->rx_mcast_bytes) +
+		       le64_to_cpu(ls->rx_bcast_bytes);
+
+	ns->tx_bytes = le64_to_cpu(ls->tx_ucast_bytes) +
+		       le64_to_cpu(ls->tx_mcast_bytes) +
+		       le64_to_cpu(ls->tx_bcast_bytes);
+
+	ns->rx_dropped = le64_to_cpu(ls->rx_ucast_drop_packets) +
+			 le64_to_cpu(ls->rx_mcast_drop_packets) +
+			 le64_to_cpu(ls->rx_bcast_drop_packets);
+
+	ns->tx_dropped = le64_to_cpu(ls->tx_ucast_drop_packets) +
+			 le64_to_cpu(ls->tx_mcast_drop_packets) +
+			 le64_to_cpu(ls->tx_bcast_drop_packets);
+
+	ns->multicast = le64_to_cpu(ls->rx_mcast_packets);
+
+	ns->rx_over_errors = le64_to_cpu(ls->rx_queue_empty);
+
+	ns->rx_missed_errors = le64_to_cpu(ls->rx_dma_error) +
+			       le64_to_cpu(ls->rx_queue_disabled) +
+			       le64_to_cpu(ls->rx_desc_fetch_error) +
+			       le64_to_cpu(ls->rx_desc_data_error);
+
+	ns->tx_aborted_errors = le64_to_cpu(ls->tx_dma_error) +
+				le64_to_cpu(ls->tx_queue_disabled) +
+				le64_to_cpu(ls->tx_desc_fetch_error) +
+				le64_to_cpu(ls->tx_desc_data_error);
+
+	ns->rx_errors = ns->rx_over_errors +
+			ns->rx_missed_errors;
+
+	ns->tx_errors = ns->tx_aborted_errors;
+}
+
 static int ionic_lif_addr_add(struct lif *lif, const u8 *addr)

{

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
 	struct ionic_admin_ctx ctx = {
@@ -581,6 +693,7 @@ static int ionic_vlan_rx_kill_vid(struct
 net_device *netdev, __be16 proto,
 static const struct net_device_ops ionic_netdev_ops = {
 	.ndo_open               = ionic_open,
 	.ndo_stop               = ionic_stop,
+	.ndo_get_stats64	= ionic_get_stats64,
 	.ndo_set_rx_mode	= ionic_set_rx_mode,
 	.ndo_set_features	= ionic_set_features,
drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
 	.ndo_set_mac_address	= ionic_set_mac_address,
@@ -1418,6 +1531,8 @@ static int ionic_lif_init(struct lif *lif)
 
 	set_bit(LIF_INITED, lif->state);
 
+	ionic_link_status_check(lif);
+
 	return 0;
 
 err_out_notifyq_deinit:

drivers/net/ethernet/pensando/ionic/ionic_lif.c int ionic_lifs_register(struct ionic *ionic) · line 1578

Quoted 3 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
@@ -1461,6 +1576,7 @@ int ionic_lifs_register(struct ionic *ionic)
 		return err;
 	}
 

are events (NotifyQ) enabled at this stage ? if so then you might endup racing ionic_link_status_check with itself.

Quoted 17 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+	ionic_link_status_check(ionic->master_lif);
 	ionic->master_lif->registered = true;
 
 	return 0;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +1 -0
index 20b4fa573f77..9930b9390c8a 100644
@@ -86,6 +86,7 @@ struct ionic_deferred {
 enum lif_state_flags {
 	LIF_INITED,
 	LIF_UP,
+	LIF_LINK_CHECK_NEEDED,
 	LIF_QUEUE_RESET,
 
 	/* leave this as last */
#42 lore raw

Re: [PATCH v4 net-next 12/19] ionic: Add async link status check and basic stats

Quoted 10 lines · Saeed Mahameed
Quoted 9 lines · Shannon Nelson

Add code to handle the link status event, and wire up the basic netdev hardware stats.

---
  .../net/ethernet/pensando/ionic/ionic_lif.c   | 116
++++++++++++++++++
  .../net/ethernet/pensando/ionic/ionic_lif.h   |   1 +
  2 files changed, 117 insertions(+)

[...]

Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +6 -0
+	/* After outstanding events are processed we can check on
+	 * the link status and any outstanding interrupt credits.
+	 *
+	 * We wait until here to check on the link status in case
+	 * there was a long list of link events from a flap episode.
+	 */
Quoted 8 lines

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_notifyq_clean(struct lif *lif, int budget) · line 277

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +6 -0
+	if (test_bit(LIF_LINK_CHECK_NEEDED, lif->state)) {
+		struct ionic_deferred_work *work;
+
+		work = kzalloc(sizeof(*work), GFP_ATOMIC);
+		if (!work) {
+			netdev_err(lif->netdev, "%s OOM\n", __func__);

why not having a pre allocated dedicated lif->link_check_work, instead of allocating in atomic context on every link check event ?

I don't want to worry about the possibility of additional requests driven from other threads using the same struct.

Quoted 84 lines
Quoted 84 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
+		} else {
+			work->type = DW_TYPE_LINK_STATUS;
+			ionic_lif_deferred_enqueue(&lif->deferred,

work);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +57 -0
+		}
+	}
+
  return_to_napi:
  	return work_done;
  }
 
+static void ionic_get_stats64(struct net_device *netdev,
+			      struct rtnl_link_stats64 *ns)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct lif_stats *ls;
+
+	memset(ns, 0, sizeof(*ns));
+	ls = &lif->info->stats;
+
+	ns->rx_packets = le64_to_cpu(ls->rx_ucast_packets) +
+			 le64_to_cpu(ls->rx_mcast_packets) +
+			 le64_to_cpu(ls->rx_bcast_packets);
+
+	ns->tx_packets = le64_to_cpu(ls->tx_ucast_packets) +
+			 le64_to_cpu(ls->tx_mcast_packets) +
+			 le64_to_cpu(ls->tx_bcast_packets);
+
+	ns->rx_bytes = le64_to_cpu(ls->rx_ucast_bytes) +
+		       le64_to_cpu(ls->rx_mcast_bytes) +
+		       le64_to_cpu(ls->rx_bcast_bytes);
+
+	ns->tx_bytes = le64_to_cpu(ls->tx_ucast_bytes) +
+		       le64_to_cpu(ls->tx_mcast_bytes) +
+		       le64_to_cpu(ls->tx_bcast_bytes);
+
+	ns->rx_dropped = le64_to_cpu(ls->rx_ucast_drop_packets) +
+			 le64_to_cpu(ls->rx_mcast_drop_packets) +
+			 le64_to_cpu(ls->rx_bcast_drop_packets);
+
+	ns->tx_dropped = le64_to_cpu(ls->tx_ucast_drop_packets) +
+			 le64_to_cpu(ls->tx_mcast_drop_packets) +
+			 le64_to_cpu(ls->tx_bcast_drop_packets);
+
+	ns->multicast = le64_to_cpu(ls->rx_mcast_packets);
+
+	ns->rx_over_errors = le64_to_cpu(ls->rx_queue_empty);
+
+	ns->rx_missed_errors = le64_to_cpu(ls->rx_dma_error) +
+			       le64_to_cpu(ls->rx_queue_disabled) +
+			       le64_to_cpu(ls->rx_desc_fetch_error) +
+			       le64_to_cpu(ls->rx_desc_data_error);
+
+	ns->tx_aborted_errors = le64_to_cpu(ls->tx_dma_error) +
+				le64_to_cpu(ls->tx_queue_disabled) +
+				le64_to_cpu(ls->tx_desc_fetch_error) +
+				le64_to_cpu(ls->tx_desc_data_error);
+
+	ns->rx_errors = ns->rx_over_errors +
+			ns->rx_missed_errors;
+
+	ns->tx_errors = ns->tx_aborted_errors;
+}
+
  static int ionic_lif_addr_add(struct lif *lif, const u8 *addr)
  {
  	struct ionic_admin_ctx ctx = {
@@ -581,6 +693,7 @@ static int ionic_vlan_rx_kill_vid(struct
 net_device *netdev, __be16 proto,
  static const struct net_device_ops ionic_netdev_ops = {
  	.ndo_open               = ionic_open,
  	.ndo_stop               = ionic_stop,
+	.ndo_get_stats64	= ionic_get_stats64,
  	.ndo_set_rx_mode	= ionic_set_rx_mode,
  	.ndo_set_features	= ionic_set_features,

.ndo_set_mac_address = ionic_set_mac_address,

drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
@@ -1418,6 +1531,8 @@ static int ionic_lif_init(struct lif *lif)
 
  	set_bit(LIF_INITED, lif->state);
 
+	ionic_link_status_check(lif);
+
  	return 0;
 
Quoted 7 lines

drivers/net/ethernet/pensando/ionic/ionic_lif.c int ionic_lifs_register(struct ionic *ionic) · line 1578

Quoted 5 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
  err_out_notifyq_deinit:
@@ -1461,6 +1576,7 @@ int ionic_lifs_register(struct ionic *ionic)
  		return err;
  	}
 

are events (NotifyQ) enabled at this stage ? if so then you might endup racing ionic_link_status_check with itself.

I'll look at that again to see what such a race might do.  I probably should add a test here and in a couple other spots to see if the link status check has already been requested.

sln

#43 lore raw

PATCH v4 13/19 ionic: Add initial ethtool support

In reply to Shannon Nelson

Add in the basic ethtool callbacks for device information and control.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   2 +-
 .../net/ethernet/pensando/ionic/ionic_dev.h   |   3 +
 .../ethernet/pensando/ionic/ionic_ethtool.c   | 495 ++++++++++++++++++
 .../ethernet/pensando/ionic/ionic_ethtool.h   |   9 +
 .../net/ethernet/pensando/ionic/ionic_lif.c   |   2 +
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   8 +
 6 files changed, 518 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_ethtool.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_ethtool.h
drivers/net/ethernet/pensando/ionic/Makefile +1 -1
index 7d9cdc5f02a1..9b19bf57a489 100644
@@ -3,5 +3,5 @@
 
 obj-$(CONFIG_IONIC) := ionic.o
 
-ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o \
+ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_ethtool.o \
 	   ionic_lif.o ionic_rx_filter.o ionic_debugfs.o
drivers/net/ethernet/pensando/ionic/ionic_dev.h +3 -0
index 523927566925..bacc9c557329 100644
@@ -12,6 +12,9 @@
 
 #define IONIC_MIN_MTU			ETH_MIN_MTU
 #define IONIC_MAX_MTU			9194
+#define IONIC_MAX_TXRX_DESC		16384
+#define IONIC_MIN_TXRX_DESC		16
+#define IONIC_DEF_TXRX_DESC		4096
 #define IONIC_LIFS_MAX			1024
 
 struct ionic_dev_bar {
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +495 -0

495 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_ethtool.h new file +9 -0
new file mode 100644
index 000000000000..38b91b1d70ae
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_ETHTOOL_H_
+#define _IONIC_ETHTOOL_H_
+
+void ionic_ethtool_set_ops(struct net_device *netdev);
+
+#endif /* _IONIC_ETHTOOL_H_ */
drivers/net/ethernet/pensando/ionic/ionic_lif.c +2 -0
index f52af9cb6264..2bd8ce61c4a0 100644
@@ -10,6 +10,7 @@
 #include "ionic.h"
 #include "ionic_bus.h"
 #include "ionic_lif.h"
+#include "ionic_ethtool.h"
 #include "ionic_debugfs.h"
 
 static void ionic_lif_rx_mode(struct lif *lif, unsigned int rx_mode);
@@ -980,6 +981,7 @@ static struct lif *ionic_lif_alloc(struct ionic *ionic, unsigned int index)
 	lif->netdev = netdev;
 	ionic->master_lif = lif;
 	netdev->netdev_ops = &ionic_netdev_ops;
+	ionic_ethtool_set_ops(netdev);
 
 	netdev->watchdog_timeo = 2 * HZ;
 	netdev->min_mtu = IONIC_MIN_MTU;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +8 -0
index 9930b9390c8a..d8589a306aa5 100644
@@ -111,6 +111,8 @@ struct lif {
 	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
+	unsigned int ntxq_descs;
+	unsigned int nrxq_descs;
 	unsigned int rx_mode;
 	u64 hw_features;
 	bool mc_overflow;
@@ -124,6 +126,8 @@ struct lif {
 
 	struct rx_filters rx_filters;
 	struct ionic_deferred deferred;
+	u32 tx_coalesce_usecs;
+	u32 rx_coalesce_usecs;
 	unsigned long *dbid_inuse;
 	unsigned int dbid_count;
 	struct dentry *dentry;
@@ -165,6 +169,10 @@ int ionic_lif_identify(struct ionic *ionic, u8 lif_type,
 		       union lif_identity *lif_ident);
 int ionic_lifs_size(struct ionic *ionic);
 
+int ionic_open(struct net_device *netdev);
+int ionic_stop(struct net_device *netdev);
+int ionic_reset_queues(struct lif *lif);
+
 static inline void debug_stats_napi_poll(struct qcq *qcq,
 					 unsigned int work_done)
 {
Signature · 1 line
2.17.1
maintainer
#44 lore raw

Re: [PATCH v4 net-next 13/19] ionic: Add initial ethtool support

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:17 -0700

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 55

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +6 -0
+static int ionic_get_link_ksettings(struct net_device *netdev,
+				    struct ethtool_link_ksettings *ks)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	int copper_seen = 0;

Reverse christmas tree ordering here please.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 195

Quoted 9 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +9 -0
+static int ionic_set_link_ksettings(struct net_device *netdev,
+				    const struct ethtool_link_ksettings *ks)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u8 fec_type = PORT_FEC_TYPE_NONE;
+	u32 req_rs, req_fc;
+	int err = 0;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 253

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +6 -0
+static void ionic_get_pauseparam(struct net_device *netdev,
+				 struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	uint8_t pause_type = idev->port_info->config.pause_type;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 270

Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+static int ionic_set_pauseparam(struct net_device *netdev,
+				struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u32 requested_pause;
+	int err;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 407

Quoted 7 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +7 -0
+static int ionic_get_module_info(struct net_device *netdev,
+				 struct ethtool_modinfo *modinfo)
+
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 438

Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+static int ionic_get_module_eeprom(struct net_device *netdev,
+				   struct ethtool_eeprom *ee,
+				   u8 *data)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;
+	u32 len;

Likewise.

#45 lore raw

Re: [PATCH v4 net-next 13/19] ionic: Add initial ethtool support

Quoted 10 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:17 -0700

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 55

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +6 -0
+static int ionic_get_link_ksettings(struct net_device *netdev,
+				    struct ethtool_link_ksettings *ks)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	int copper_seen = 0;

Reverse christmas tree ordering here please.

Sure.  Most of these are because I wanted to do the one assignment first, from which the others are based.  I'll rework these to add the init lines after the declarations.  Same in the other files.

sln

Quoted 45 lines

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 195

Quoted 9 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +9 -0
+static int ionic_set_link_ksettings(struct net_device *netdev,
+				    const struct ethtool_link_ksettings *ks)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u8 fec_type = PORT_FEC_TYPE_NONE;
+	u32 req_rs, req_fc;
+	int err = 0;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 253

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +6 -0
+static void ionic_get_pauseparam(struct net_device *netdev,
+				 struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	uint8_t pause_type = idev->port_info->config.pause_type;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 270

Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+static int ionic_set_pauseparam(struct net_device *netdev,
+				struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u32 requested_pause;
+	int err;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 407

Quoted 7 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +7 -0
+static int ionic_get_module_info(struct net_device *netdev,
+				 struct ethtool_modinfo *modinfo)
+
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;

Likewise.

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 438

Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+static int ionic_get_module_eeprom(struct net_device *netdev,
+				   struct ethtool_eeprom *ee,
+				   u8 *data)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;
+	u32 len;

Likewise.

#46 lore raw

Re: [PATCH v4 net-next 13/19] ionic: Add initial ethtool support

In reply to Shannon Nelson
Quoted 463 lines · Shannon Nelson

Add in the basic ethtool callbacks for device information and control.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   2 +-
 .../net/ethernet/pensando/ionic/ionic_dev.h   |   3 +
 .../ethernet/pensando/ionic/ionic_ethtool.c   | 495
++++++++++++++++++
 .../ethernet/pensando/ionic/ionic_ethtool.h   |   9 +
 .../net/ethernet/pensando/ionic/ionic_lif.c   |   2 +
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   8 +
 6 files changed, 518 insertions(+), 1 deletion(-)
 create mode 100644
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c
 create mode 100644

drivers/net/ethernet/pensando/ionic/ionic_ethtool.h

drivers/net/ethernet/pensando/ionic/Makefile +1 -1
index 7d9cdc5f02a1..9b19bf57a489 100644
@@ -3,5 +3,5 @@
 
 obj-$(CONFIG_IONIC) := ionic.o
 
-ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o \
+ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_ethtool.o
 \
drivers/net/ethernet/pensando/ionic/Makefile +0 -0
 	   ionic_lif.o ionic_rx_filter.o ionic_debugfs.o
drivers/net/ethernet/pensando/ionic/ionic_dev.h +3 -0
index 523927566925..bacc9c557329 100644
@@ -12,6 +12,9 @@
 
 #define IONIC_MIN_MTU			ETH_MIN_MTU
 #define IONIC_MAX_MTU			9194
+#define IONIC_MAX_TXRX_DESC		16384
+#define IONIC_MIN_TXRX_DESC		16
+#define IONIC_DEF_TXRX_DESC		4096
 #define IONIC_LIFS_MAX			1024
 
 struct ionic_dev_bar {
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +20 -0
new file mode 100644
index 000000000000..f7899be547c3
@@ -0,0 +1,495 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#include <linux/module.h>
+#include <linux/netdevice.h>
+
+#include "ionic.h"
+#include "ionic_bus.h"
+#include "ionic_lif.h"
+#include "ionic_ethtool.h"
+
+static void ionic_get_drvinfo(struct net_device *netdev,
+			      struct ethtool_drvinfo *drvinfo)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &ionic->idev;
+
+	strlcpy(drvinfo->driver, DRV_NAME, sizeof(drvinfo->driver));
+	strlcpy(drvinfo->version, DRV_VERSION, sizeof(drvinfo-
Quoted 1 line

version));

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +15 -0
+	strlcpy(drvinfo->fw_version, idev->dev_info.fw_version,
+		sizeof(drvinfo->fw_version));
+	strlcpy(drvinfo->bus_info, ionic_bus_info(ionic),
+		sizeof(drvinfo->bus_info));
+}
+
+#define DEV_CMD_REG_VERSION 1
+#define DEV_INFO_REG_COUNT  32
+#define DEV_CMD_REG_COUNT   32
+static int ionic_get_regs_len(struct net_device *netdev)
+{
+	return (DEV_INFO_REG_COUNT + DEV_CMD_REG_COUNT) * sizeof(u32);
+}
+
+static void ionic_get_regs(struct net_device *netdev, struct

ethtool_regs *regs,

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +24 -0
+			   void *p)
+{
+	struct lif *lif = netdev_priv(netdev);
+	unsigned int size;
+
+	regs->version = DEV_CMD_REG_VERSION;
+
+	size = DEV_INFO_REG_COUNT * sizeof(u32);
+	memcpy_fromio(p, lif->ionic->idev.dev_info_regs->words, size);
+
+	size = DEV_CMD_REG_COUNT * sizeof(u32);
+	memcpy_fromio(p, lif->ionic->idev.dev_cmd_regs->words, size);
+}
+
+static int ionic_get_link_ksettings(struct net_device *netdev,
+				    struct ethtool_link_ksettings *ks)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	int copper_seen = 0;
+
+	ethtool_link_ksettings_zero_link_mode(ks, supported);
+
+	/* The port_info data is found in a DMA space that the NIC

keeps

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +9 -0
+	 * up-to-date, so there's no need to request the data from the
+	 * NIC, we already have it in our memory space.
+	 */
+
+	switch (le16_to_cpu(idev->port_info->status.xcvr.pid)) {
+		/* Copper */
+	case XCVR_PID_QSFP_100G_CR4:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     100000baseCR4_Full

);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +5 -0
+		copper_seen++;
+		break;
+	case XCVR_PID_QSFP_40GBASE_CR4:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     40000baseCR4_Full)

;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +21 -0
+		copper_seen++;
+		break;
+	case XCVR_PID_SFP_25GBASE_CR_S:
+	case XCVR_PID_SFP_25GBASE_CR_L:
+	case XCVR_PID_SFP_25GBASE_CR_N:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     25000baseCR_Full);
+		copper_seen++;
+		break;
+	case XCVR_PID_SFP_10GBASE_AOC:
+	case XCVR_PID_SFP_10GBASE_CU:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     10000baseCR_Full);
+		copper_seen++;
+		break;
+
+		/* Fibre */
+	case XCVR_PID_QSFP_100G_SR4:
+	case XCVR_PID_QSFP_100G_AOC:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     100000baseSR4_Full

);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +4 -0
+		break;
+	case XCVR_PID_QSFP_100G_LR4:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     100000baseLR4_ER4_

Full);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +4 -0
+		break;
+	case XCVR_PID_QSFP_100G_ER4:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     100000baseLR4_ER4_

Full);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +5 -0
+		break;
+	case XCVR_PID_QSFP_40GBASE_SR4:
+	case XCVR_PID_QSFP_40GBASE_AOC:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     40000baseSR4_Full)

;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +4 -0
+		break;
+	case XCVR_PID_QSFP_40GBASE_LR4:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     40000baseLR4_Full)

;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +17 -0
+		break;
+	case XCVR_PID_SFP_25GBASE_SR:
+	case XCVR_PID_SFP_25GBASE_AOC:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     25000baseSR_Full);
+		break;
+	case XCVR_PID_SFP_10GBASE_SR:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     10000baseSR_Full);
+		break;
+	case XCVR_PID_SFP_10GBASE_LR:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     10000baseLR_Full);
+		break;
+	case XCVR_PID_SFP_10GBASE_LRM:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     10000baseLRM_Full)

;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +9 -0
+		break;
+	case XCVR_PID_SFP_10GBASE_ER:
+		ethtool_link_ksettings_add_link_mode(ks, supported,
+						     10000baseER_Full);
+		break;
+	case XCVR_PID_UNKNOWN:
+		break;
+	default:
+		dev_info(lif->ionic->dev, "unknown xcvr type pid=%d /

0x%x\n",

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+			 idev->port_info->status.xcvr.pid,
+			 idev->port_info->status.xcvr.pid);
+		break;
+	}
+	ethtool_link_ksettings_add_link_mode(ks, supported, FIBRE);
+
+	if (ionic_is_pf(lif->ionic))
+		ethtool_link_ksettings_add_link_mode(ks, supported,

Autoneg);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +2 -0
+
+	bitmap_copy(ks->link_modes.advertising, ks-
Quoted 1 line

link_modes.supported,

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+		    __ETHTOOL_LINK_MODE_MASK_NBITS);
+
+	ethtool_link_ksettings_add_link_mode(ks, supported, FEC_NONE);
+	ethtool_link_ksettings_add_link_mode(ks, supported, FEC_RS);
+	ethtool_link_ksettings_add_link_mode(ks, supported, FEC_BASER);
+
+	if (idev->port_info->config.fec_type == PORT_FEC_TYPE_FC)
+		ethtool_link_ksettings_add_link_mode(ks, advertising,

FEC_BASER);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +2 -0
+	else if (idev->port_info->config.fec_type == PORT_FEC_TYPE_RS)
+		ethtool_link_ksettings_add_link_mode(ks, advertising,

FEC_RS);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +1 -0
+	else if (idev->port_info->config.fec_type ==

PORT_FEC_TYPE_NONE)

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +1 -0
+		ethtool_link_ksettings_add_link_mode(ks, advertising,

FEC_NONE);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +4 -0
+
+	ethtool_link_ksettings_add_link_mode(ks, supported, Pause);
+	if (idev->port_info->config.pause_type)
+		ethtool_link_ksettings_add_link_mode(ks, advertising,

Pause);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +5 -0
+
+	if (idev->port_info->status.xcvr.phy == PHY_TYPE_COPPER ||
+	    copper_seen) {
+		ks->base.port = PORT_DA;
+	} else if (idev->port_info->status.xcvr.phy == PHY_TYPE_FIBER)

{

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +20 -0
+		ks->base.port = PORT_FIBRE;
+	} else {
+		ks->base.port = PORT_OTHER;
+	}
+
+	ks->base.speed = le32_to_cpu(lif->info->status.link_speed);
+
+	if (idev->port_info->config.an_enable)
+		ks->base.autoneg = AUTONEG_ENABLE;
+
+	if (le16_to_cpu(lif->info->status.link_status))
+		ks->base.duplex = DUPLEX_FULL;
+	else
+		ks->base.duplex = DUPLEX_UNKNOWN;
+
+	return 0;
+}
+
+static int ionic_set_link_ksettings(struct net_device *netdev,
+				    const struct ethtool_link_ksettings

*ks)

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +20 -0
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u8 fec_type = PORT_FEC_TYPE_NONE;
+	u32 req_rs, req_fc;
+	int err = 0;
+
+	/* set autoneg */
+	if (ks->base.autoneg != idev->port_info->config.an_enable) {
+		mutex_lock(&ionic->dev_cmd_lock);
+		ionic_dev_cmd_port_autoneg(idev, ks->base.autoneg);
+		err = ionic_dev_cmd_wait(ionic, devcmd_timeout);
+		mutex_unlock(&ionic->dev_cmd_lock);
+		if (err)
+			return err;
+	}
+
+	/* set speed */
+	if (ks->base.speed != le32_to_cpu(idev->port_info-
Quoted 1 line

config.speed)) {

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +10 -0
+		mutex_lock(&ionic->dev_cmd_lock);
+		ionic_dev_cmd_port_speed(idev, ks->base.speed);
+		err = ionic_dev_cmd_wait(ionic, devcmd_timeout);
+		mutex_unlock(&ionic->dev_cmd_lock);
+		if (err)
+			return err;
+	}
+
+	/* set FEC */
+	req_rs = ethtool_link_ksettings_test_link_mode(ks, advertising,

FEC_RS);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +1 -0
+	req_fc = ethtool_link_ksettings_test_link_mode(ks, advertising,

FEC_BASER);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +2 -0
+	if (req_rs && req_fc) {
+		netdev_info(netdev, "Only select one FEC mode at a

time\n");

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +3 -0
+		return -EINVAL;
+	} else if (req_fc &&
+		   idev->port_info->config.fec_type !=

PORT_FEC_TYPE_FC) {

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +3 -0
+		fec_type = PORT_FEC_TYPE_FC;
+	} else if (req_rs &&
+		   idev->port_info->config.fec_type !=

PORT_FEC_TYPE_RS) {

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +3 -0
+		fec_type = PORT_FEC_TYPE_RS;
+	} else if (!(req_rs | req_fc) &&
+		   idev->port_info->config.fec_type !=

PORT_FEC_TYPE_NONE) {

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +28 -0
+		fec_type = PORT_FEC_TYPE_NONE;
+	}
+
+	if (fec_type != idev->port_info->config.fec_type) {
+		mutex_lock(&ionic->dev_cmd_lock);
+		ionic_dev_cmd_port_fec(idev, fec_type);
+		err = ionic_dev_cmd_wait(ionic, devcmd_timeout);
+		mutex_unlock(&ionic->dev_cmd_lock);
+		if (err)
+			return err;
+
+		idev->port_info->config.fec_type = fec_type;
+	}
+
+	return 0;
+}
+
+static void ionic_get_pauseparam(struct net_device *netdev,
+				 struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	uint8_t pause_type = idev->port_info->config.pause_type;
+
+	pause->autoneg = 0;
+
+	if (pause_type) {
+		pause->rx_pause = pause_type & IONIC_PAUSE_F_RX ? 1 :

0;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +1 -0
+		pause->tx_pause = pause_type & IONIC_PAUSE_F_TX ? 1 :

0;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +14 -0
+	}
+}
+
+static int ionic_set_pauseparam(struct net_device *netdev,
+				struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u32 requested_pause;
+	int err;
+
+	if (pause->autoneg == AUTONEG_ENABLE) {
+		netdev_info(netdev, "Please use 'ethtool -s ...' to

change autoneg\n");

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +55 -0
+		return -EOPNOTSUPP;
+	}
+
+	/* change both at the same time */
+	requested_pause = PORT_PAUSE_TYPE_LINK;
+	if (pause->rx_pause)
+		requested_pause |= IONIC_PAUSE_F_RX;
+	if (pause->tx_pause)
+		requested_pause |= IONIC_PAUSE_F_TX;
+
+	if (requested_pause == idev->port_info->config.pause_type)
+		return 0;
+
+	idev->port_info->config.pause_type = requested_pause;
+
+	mutex_lock(&ionic->dev_cmd_lock);
+	ionic_dev_cmd_port_pause(idev, requested_pause);
+	err = ionic_dev_cmd_wait(ionic, devcmd_timeout);
+	mutex_unlock(&ionic->dev_cmd_lock);
+	if (err)
+		return err;
+
+	return 0;
+}
+
+static int ionic_get_coalesce(struct net_device *netdev,
+			      struct ethtool_coalesce *coalesce)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	coalesce->tx_coalesce_usecs = lif->tx_coalesce_usecs;
+	coalesce->rx_coalesce_usecs = lif->rx_coalesce_usecs;
+
+	return 0;
+}
+
+static void ionic_get_ringparam(struct net_device *netdev,
+				struct ethtool_ringparam *ring)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	ring->tx_max_pending = IONIC_MAX_TXRX_DESC;
+	ring->tx_pending = lif->ntxq_descs;
+	ring->rx_max_pending = IONIC_MAX_TXRX_DESC;
+	ring->rx_pending = lif->nrxq_descs;
+}
+
+static int ionic_set_ringparam(struct net_device *netdev,
+			       struct ethtool_ringparam *ring)
+{
+	struct lif *lif = netdev_priv(netdev);
+	bool running;
+
+	if (ring->rx_mini_pending || ring->rx_jumbo_pending) {
+		netdev_info(netdev, "Changing jumbo or mini descriptors

not supported\n");

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +6 -0
+		return -EINVAL;
+	}
+
+	if (!is_power_of_2(ring->tx_pending) ||
+	    !is_power_of_2(ring->rx_pending)) {
+		netdev_info(netdev, "Descriptor count must be a power

of 2\n");

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +43 -0
+		return -EINVAL;
+	}
+
+	/* if nothing to do return success */
+	if (ring->tx_pending == lif->ntxq_descs &&
+	    ring->rx_pending == lif->nrxq_descs)
+		return 0;
+
+	while (test_and_set_bit(LIF_QUEUE_RESET, lif->state))
+		usleep_range(200, 400);
+
+	running = test_bit(LIF_UP, lif->state);
+	if (running)
+		ionic_stop(netdev);
+
+	lif->ntxq_descs = ring->tx_pending;
+	lif->nrxq_descs = ring->rx_pending;
+
+	if (running)
+		ionic_open(netdev);
+	clear_bit(LIF_QUEUE_RESET, lif->state);
+
+	return 0;
+}
+
+static void ionic_get_channels(struct net_device *netdev,
+			       struct ethtool_channels *ch)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	/* report maximum channels */
+	ch->max_combined = lif->ionic->ntxqs_per_lif;
+
+	/* report current channels */
+	ch->combined_count = lif->nxqs;
+}
+
+static int ionic_set_channels(struct net_device *netdev,
+			      struct ethtool_channels *ch)
+{
+	struct lif *lif = netdev_priv(netdev);
+	bool running;
+

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 387

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +10 -0
+	if (!ch->combined_count || ch->other_count ||
+	    ch->rx_count || ch->tx_count)
+		return -EINVAL;
+
+	if (ch->combined_count == lif->nxqs)
+		return 0;
+
+	while (test_and_set_bit(LIF_QUEUE_RESET, lif->state))
+		usleep_range(200, 400);
+

I see this is recurring a lot in the driver, i suggest to have a helper function (wait_pending_reset_timeout) and make it return with timeout errno after a reasonable amount of time, especially on user context flows.

Quoted 183 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +77 -0
+	running = test_bit(LIF_UP, lif->state);
+	if (running)
+		ionic_stop(netdev);
+
+	lif->nxqs = ch->combined_count;
+
+	if (running)
+		ionic_open(netdev);
+	clear_bit(LIF_QUEUE_RESET, lif->state);
+
+	return 0;
+}
+
+static int ionic_get_module_info(struct net_device *netdev,
+				 struct ethtool_modinfo *modinfo)
+
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;
+
+	xcvr = &idev->port_info->status.xcvr;
+
+	/* report the module data type and length */
+	switch (xcvr->sprom[0]) {
+	case 0x03: /* SFP */
+		modinfo->type = ETH_MODULE_SFF_8079;
+		modinfo->eeprom_len = ETH_MODULE_SFF_8079_LEN;
+		break;
+	case 0x0D: /* QSFP */
+	case 0x11: /* QSFP28 */
+		modinfo->type = ETH_MODULE_SFF_8436;
+		modinfo->eeprom_len = ETH_MODULE_SFF_8436_LEN;
+		break;
+	default:
+		netdev_info(netdev, "unknown xcvr type 0x%02x\n",
+			    xcvr->sprom[0]);
+		break;
+	}
+
+	return 0;
+}
+
+static int ionic_get_module_eeprom(struct net_device *netdev,
+				   struct ethtool_eeprom *ee,
+				   u8 *data)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;
+	u32 len;
+
+	/* The NIC keeps the module prom up-to-date in the DMA space
+	 * so we can simply copy the module bytes into the data buffer.
+	 */
+	xcvr = &idev->port_info->status.xcvr;
+	len = min_t(u32, sizeof(xcvr->sprom), ee->len);
+	memcpy(data, xcvr->sprom, len);
+
+	return 0;
+}
+
+static int ionic_nway_reset(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	int err = 0;
+
+	/* flap the link to force auto-negotiation */
+
+	mutex_lock(&ionic->dev_cmd_lock);
+
+	ionic_dev_cmd_port_state(&ionic->idev, PORT_ADMIN_STATE_DOWN);
+	err = ionic_dev_cmd_wait(ionic, devcmd_timeout);
+
+	if (!err) {
+		ionic_dev_cmd_port_state(&ionic->idev,

PORT_ADMIN_STATE_UP);

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +31 -0
+		err = ionic_dev_cmd_wait(ionic, devcmd_timeout);
+	}
+
+	mutex_unlock(&ionic->dev_cmd_lock);
+
+	return err;
+}
+
+static const struct ethtool_ops ionic_ethtool_ops = {
+	.get_drvinfo		= ionic_get_drvinfo,
+	.get_regs_len		= ionic_get_regs_len,
+	.get_regs		= ionic_get_regs,
+	.get_link		= ethtool_op_get_link,
+	.get_link_ksettings	= ionic_get_link_ksettings,
+	.get_coalesce		= ionic_get_coalesce,
+	.get_ringparam		= ionic_get_ringparam,
+	.set_ringparam		= ionic_set_ringparam,
+	.get_channels		= ionic_get_channels,
+	.set_channels		= ionic_set_channels,
+	.get_module_info	= ionic_get_module_info,
+	.get_module_eeprom	= ionic_get_module_eeprom,
+	.get_pauseparam		= ionic_get_pauseparam,
+	.set_pauseparam		= ionic_set_pauseparam,
+	.set_link_ksettings	= ionic_set_link_ksettings,
+	.nway_reset		= ionic_nway_reset,
+};
+
+void ionic_ethtool_set_ops(struct net_device *netdev)
+{
+	netdev->ethtool_ops = &ionic_ethtool_ops;
+}
drivers/net/ethernet/pensando/ionic/ionic_ethtool.h new file +9 -0
new file mode 100644
index 000000000000..38b91b1d70ae
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_ETHTOOL_H_
+#define _IONIC_ETHTOOL_H_
+
+void ionic_ethtool_set_ops(struct net_device *netdev);
+
+#endif /* _IONIC_ETHTOOL_H_ */
drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
index f52af9cb6264..2bd8ce61c4a0 100644
@@ -10,6 +10,7 @@
 #include "ionic.h"
 #include "ionic_bus.h"
 #include "ionic_lif.h"
+#include "ionic_ethtool.h"
 #include "ionic_debugfs.h"
 
 static void ionic_lif_rx_mode(struct lif *lif, unsigned int

rx_mode);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
@@ -980,6 +981,7 @@ static struct lif *ionic_lif_alloc(struct ionic
 *ionic, unsigned int index)
 	lif->netdev = netdev;
 	ionic->master_lif = lif;
 	netdev->netdev_ops = &ionic_netdev_ops;
+	ionic_ethtool_set_ops(netdev);
 
 	netdev->watchdog_timeo = 2 * HZ;
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
 	netdev->min_mtu = IONIC_MIN_MTU;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +8 -0
index 9930b9390c8a..d8589a306aa5 100644
@@ -111,6 +111,8 @@ struct lif {
 	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
+	unsigned int ntxq_descs;
+	unsigned int nrxq_descs;
 	unsigned int rx_mode;
 	u64 hw_features;
 	bool mc_overflow;
@@ -124,6 +126,8 @@ struct lif {
 
 	struct rx_filters rx_filters;
 	struct ionic_deferred deferred;
+	u32 tx_coalesce_usecs;
+	u32 rx_coalesce_usecs;
 	unsigned long *dbid_inuse;
 	unsigned int dbid_count;
 	struct dentry *dentry;
@@ -165,6 +169,10 @@ int ionic_lif_identify(struct ionic *ionic, u8
 lif_type,
 		       union lif_identity *lif_ident);
 int ionic_lifs_size(struct ionic *ionic);
 
+int ionic_open(struct net_device *netdev);
+int ionic_stop(struct net_device *netdev);
+int ionic_reset_queues(struct lif *lif);
+
 static inline void debug_stats_napi_poll(struct qcq *qcq,
 					 unsigned int work_done)

{

#47 lore raw

Re: [PATCH v4 net-next 13/19] ionic: Add initial ethtool support

Quoted 7 lines · Saeed Mahameed
Quoted 5 lines · Shannon Nelson

Add in the basic ethtool callbacks for device information and control.

---

Quoted 10 lines

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 345

Quoted 4 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +4 -0
+
+	while (test_and_set_bit(LIF_QUEUE_RESET, lif->state))
+		usleep_range(200, 400);
+

I see this is recurring a lot in the driver, i suggest to have a helper function (wait_pending_reset_timeout) and make it return with timeout errno after a reasonable amount of time, especially on user context flows.

Sure.
sln

#48 lore raw

Re: [PATCH v4 net-next 13/19] ionic: Add initial ethtool support

In reply to Shannon Nelson
Quoted 25 lines · Shannon Nelson
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +25 -0
+static void ionic_get_pauseparam(struct net_device *netdev,
+				 struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	uint8_t pause_type = idev->port_info->config.pause_type;
+
+	pause->autoneg = 0;
+
+	if (pause_type) {
+		pause->rx_pause = pause_type & IONIC_PAUSE_F_RX ? 1 : 0;
+		pause->tx_pause = pause_type & IONIC_PAUSE_F_TX ? 1 : 0;
+	}
+}
+
+static int ionic_set_pauseparam(struct net_device *netdev,
+				struct ethtool_pauseparam *pause)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic *ionic = lif->ionic;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u32 requested_pause;
+	int err;
+
+	if (pause->autoneg == AUTONEG_ENABLE) {

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 282

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +10 -0
+		netdev_info(netdev, "Please use 'ethtool -s ...' to change autoneg\n");
+		return -EOPNOTSUPP;
+	}
+
+	/* change both at the same time */
+	requested_pause = PORT_PAUSE_TYPE_LINK;
+	if (pause->rx_pause)
+		requested_pause |= IONIC_PAUSE_F_RX;
+	if (pause->tx_pause)
+		requested_pause |= IONIC_PAUSE_F_TX;

Hi Shannon

This not what i was expecting, and i'm guessing what you have here is not right.

So you don't support the case of pause->autoneg == AUTONEG_ENABLE. That implies that setting pause values directly configures the MAC. The values from auto-neg are always ignored. Then why did you set PAUSE in the get ksettings function?

If you always ignore the values from auto-neg, then your info message also makes no sense.

What really does the firmware do?

Andrew

#49 lore raw

Re: [PATCH v4 net-next 13/19] ionic: Add initial ethtool support

In reply to Shannon Nelson
Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+static int ionic_get_module_eeprom(struct net_device *netdev,
+				   struct ethtool_eeprom *ee,
+				   u8 *data)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;
+	u32 len;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 448

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +10 -0
+
+	/* The NIC keeps the module prom up-to-date in the DMA space
+	 * so we can simply copy the module bytes into the data buffer.
+	 */
+	xcvr = &idev->port_info->status.xcvr;
+	len = min_t(u32, sizeof(xcvr->sprom), ee->len);
+	memcpy(data, xcvr->sprom, len);
+
+	return 0;
+}

Is the firmware doing this DMA update atomically? The diagnostic values are u16s. Is there any chance we do this memcpy at the same time the DMA is active and we get a mix of old and new data?

Often in cases like this you do the copy twice and ensure you get the same values each time. If not, keep repeating the copy until you do get the same values twice.

Andrew

#50 lore raw

Re: [PATCH v4 net-next 13/19] ionic: Add initial ethtool support

Quoted 22 lines · Andrew Lunn
Quoted 8 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +8 -0
+static int ionic_get_module_eeprom(struct net_device *netdev,
+				   struct ethtool_eeprom *ee,
+				   u8 *data)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct xcvr_status *xcvr;
+	u32 len;

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c line 448

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c new file +10 -0
+
+	/* The NIC keeps the module prom up-to-date in the DMA space
+	 * so we can simply copy the module bytes into the data buffer.
+	 */
+	xcvr = &idev->port_info->status.xcvr;
+	len = min_t(u32, sizeof(xcvr->sprom), ee->len);
+	memcpy(data, xcvr->sprom, len);
+
+	return 0;
+}

Is the firmware doing this DMA update atomically? The diagnostic values are u16s. Is there any chance we do this memcpy at the same time the DMA is active and we get a mix of old and new data?

Quoted 3 lines

Often in cases like this you do the copy twice and ensure you get the same values each time. If not, keep repeating the copy until you do get the same values twice.

Regardless of how the structs are all aligned and our PCI block does large writes, I can see how an unoptimized memcpy() that is doing byte-by-byte copy rather than by words might result in a mangled value.  I think this is the only buffer that may be susceptible to this.  Sure, doing a double copy should work here.

sln

#51 lore raw

PATCH v4 14/19 ionic: Add Tx and Rx handling

In reply to Shannon Nelson

Add both the Tx and Rx queue setup and handling. The related stats display comes later. Instead of using the generic napi routines used by the slow-path commands, the Tx and Rx paths are simplified and inlined in one file in order to get better compiler optimizations.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   2 +-
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 387 ++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  52 ++
 .../net/ethernet/pensando/ionic/ionic_txrx.c  | 879 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_txrx.h  |  15 +
 5 files changed, 1334 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.h
drivers/net/ethernet/pensando/ionic/Makefile +1 -1
index 9b19bf57a489..0e2dc53f08d4 100644
@@ -4,4 +4,4 @@
 obj-$(CONFIG_IONIC) := ionic.o
 
 ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_ethtool.o \
-	   ionic_lif.o ionic_rx_filter.o ionic_debugfs.o
+	   ionic_lif.o ionic_rx_filter.o ionic_txrx.o ionic_debugfs.o
drivers/net/ethernet/pensando/ionic/ionic_lif.c +387 -0

482 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h +52 -0

109 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +879 -0

879 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_txrx.h new file +15 -0
new file mode 100644
index 000000000000..2391a0eec65a
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_TXRX_H_
+#define _IONIC_TXRX_H_
+
+void ionic_rx_flush(struct cq *cq);
+void ionic_tx_flush(struct cq *cq);
+
+void ionic_rx_fill(struct queue *q);
+void ionic_rx_empty(struct queue *q);
+int ionic_rx_napi(struct napi_struct *napi, int budget);
+netdev_tx_t ionic_start_xmit(struct sk_buff *skb, struct net_device *netdev);
+
+#endif /* _IONIC_TXRX_H_ */
Signature · 1 line
2.17.1
maintainer
#52 lore raw

Re: [PATCH v4 net-next 14/19] ionic: Add Tx and Rx handling

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:18 -0700

Quoted 7 lines

Add both the Tx and Rx queue setup and handling. The related stats display comes later. Instead of using the generic napi routines used by the slow-path commands, the Tx and Rx paths are simplified and inlined in one file in order to get better compiler optimizations.

A lot of reverse christmas tree ordering violations here, please fix all of them.

#53 lore raw

Re: [PATCH v4 net-next 14/19] ionic: Add Tx and Rx handling

In reply to Shannon Nelson
Quoted 60 lines · Shannon Nelson

Add both the Tx and Rx queue setup and handling. The related stats display comes later. Instead of using the generic napi routines used by the slow-path commands, the Tx and Rx paths are simplified and inlined in one file in order to get better compiler optimizations.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   2 +-
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 387 ++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  52 ++
 .../net/ethernet/pensando/ionic/ionic_txrx.c  | 879
++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_txrx.h  |  15 +
 5 files changed, 1334 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.h
drivers/net/ethernet/pensando/ionic/Makefile +1 -1
index 9b19bf57a489..0e2dc53f08d4 100644
@@ -4,4 +4,4 @@
 obj-$(CONFIG_IONIC) := ionic.o
 
 ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_ethtool.o
 \
-	   ionic_lif.o ionic_rx_filter.o ionic_debugfs.o
+	   ionic_lif.o ionic_rx_filter.o ionic_txrx.o ionic_debugfs.o
drivers/net/ethernet/pensando/ionic/ionic_lif.c +8 -0
index 2bd8ce61c4a0..40d3b1cb362a 100644
@@ -10,6 +10,7 @@
 #include "ionic.h"
 #include "ionic_bus.h"
 #include "ionic_lif.h"
+#include "ionic_txrx.h"
 #include "ionic_ethtool.h"
 #include "ionic_debugfs.h"
 
@@ -18,6 +19,13 @@ static int ionic_lif_addr_add(struct lif *lif,
 const u8 *addr);
 static int ionic_lif_addr_del(struct lif *lif, const u8 *addr);
 static void ionic_link_status_check(struct lif *lif);
 
+static int ionic_lif_stop(struct lif *lif);
+static int ionic_txrx_alloc(struct lif *lif);
+static int ionic_txrx_init(struct lif *lif);
+static void ionic_qcq_free(struct lif *lif, struct qcq *qcq);
+static int ionic_lif_txqs_init(struct lif *lif);
+static int ionic_lif_rxqs_init(struct lif *lif);
+static void ionic_lif_qcq_deinit(struct lif *lif, struct qcq *qcq);
 static int ionic_set_nic_features(struct lif *lif, netdev_features_t
 features);

static int ionic_notifyq_clean(struct lif *lif, int budget);

drivers/net/ethernet/pensando/ionic/ionic_lif.c static void ionic_lif_deferred_enqueue(struct · line 76

Quoted 3 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
@@ -66,12 +74,96 @@ static void ionic_lif_deferred_enqueue(struct
 ionic_deferred *def,
 	schedule_work(&def->work);
 }

Bottom up or top down ? your current design is very mixed and I had to to scroll down and up too often just to understand what a function is doing, i strongly suggest to pick an use one approach.

[1]
https://en.wikipedia.org/wiki/Top-down_and_bottom-up_design#Programming

Quoted 95 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +84 -0
+static int ionic_qcq_enable(struct qcq *qcq)
+{
+	struct queue *q = &qcq->q;
+	struct lif *lif = q->lif;
+	struct device *dev = lif->ionic->dev;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.q_control = {
+			.opcode = CMD_OPCODE_Q_CONTROL,
+			.lif_index = cpu_to_le16(lif->index),
+			.type = q->type,
+			.index = cpu_to_le32(q->index),
+			.oper = IONIC_Q_ENABLE,
+		},
+	};
+
+	dev_dbg(dev, "q_enable.index %d q_enable.qtype %d\n",
+		ctx.cmd.q_control.index, ctx.cmd.q_control.type);
+
+	if (qcq->flags & QCQ_F_INTR) {
+		irq_set_affinity_hint(qcq->intr.vector,
+				      &qcq->intr.affinity_mask);
+		napi_enable(&qcq->napi);
+		ionic_intr_clean(idev->intr_ctrl, qcq->intr.index);
+		ionic_intr_mask(idev->intr_ctrl, qcq->intr.index,
+				IONIC_INTR_MASK_CLEAR);
+	}
+
+	return ionic_adminq_post_wait(lif, &ctx);
+}
+
+static int ionic_qcq_disable(struct qcq *qcq)
+{
+	struct queue *q = &qcq->q;
+	struct lif *lif = q->lif;
+	struct device *dev = lif->ionic->dev;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.q_control = {
+			.opcode = CMD_OPCODE_Q_CONTROL,
+			.lif_index = cpu_to_le16(lif->index),
+			.type = q->type,
+			.index = cpu_to_le32(q->index),
+			.oper = IONIC_Q_DISABLE,
+		},
+	};
+
+	dev_dbg(dev, "q_disable.index %d q_disable.qtype %d\n",
+		ctx.cmd.q_control.index, ctx.cmd.q_control.type);
+
+	if (qcq->flags & QCQ_F_INTR) {
+		ionic_intr_mask(idev->intr_ctrl, qcq->intr.index,
+				IONIC_INTR_MASK_SET);
+		synchronize_irq(qcq->intr.vector);
+		irq_set_affinity_hint(qcq->intr.vector, NULL);
+		napi_disable(&qcq->napi);
+	}
+
+	return ionic_adminq_post_wait(lif, &ctx);
+}
+
 int ionic_open(struct net_device *netdev)
 {
 	struct lif *lif = netdev_priv(netdev);
+	unsigned int i;
+	int err;
 
 	netif_carrier_off(netdev);
 
+	err = ionic_txrx_alloc(lif);
+	if (err)
+		return err;
+
+	err = ionic_txrx_init(lif);
+	if (err)
+		goto err_out;
+
+	for (i = 0; i < lif->nxqs; i++) {
+		err = ionic_qcq_enable(lif->txqcqs[i].qcq);
+		if (err)
+			goto err_out;
+
+		ionic_rx_fill(&lif->rxqcqs[i].qcq->q);
+		err = ionic_qcq_enable(lif->rxqcqs[i].qcq);
+		if (err)
+			goto err_out;
+	}
+
 	set_bit(LIF_UP, lif->state);
 
 	ionic_link_status_check(lif);

drivers/net/ethernet/pensando/ionic/ionic_lif.c int ionic_open(struct net_device *netdev) · line 176

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
@@ -79,11 +171,16 @@ int ionic_open(struct net_device *netdev)
 		netif_tx_wake_all_queues(netdev);
 
 	return 0;
+
+err_out:
+	ionic_lif_stop(lif);

This is dangerous, maybe now it is ok to stop a partially open driver, but future development might break this assumption, and to avoid this i strongly recommend to have a symmetrical cleanup flow, or have a documentation that says ionic_lif_stop might be called with partially open resources and proper checks must be done.

Quoted 19 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +7 -0
+	return err;
 }
 
 static int ionic_lif_stop(struct lif *lif)
 {
 	struct net_device *ndev = lif->netdev;
+	unsigned int i;
 	int err = 0;
 
 	if (!test_bit(LIF_UP, lif->state)) {
@@ -100,6 +197,21 @@ static int ionic_lif_stop(struct lif *lif)
 	netif_tx_disable(ndev);
 	synchronize_rcu();
 
+	for (i = 0; i < lif->nxqs; i++) {
+		(void)ionic_qcq_disable(lif->txqcqs[i].qcq);
+		ionic_tx_flush(&lif->txqcqs[i].qcq->cq);
+		ionic_lif_qcq_deinit(lif, lif->txqcqs[i].qcq);
+		ionic_qcq_free(lif, lif->txqcqs[i].qcq);

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_lif_stop(struct lif *lif) · line 214

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+		lif->txqcqs[i].qcq = NULL;
+
+		(void)ionic_qcq_disable(lif->rxqcqs[i].qcq);
+		ionic_rx_flush(&lif->rxqcqs[i].qcq->cq);
+		ionic_lif_qcq_deinit(lif, lif->rxqcqs[i].qcq);
+		ionic_rx_empty(&lif->rxqcqs[i].qcq->q);
+		ionic_qcq_free(lif, lif->rxqcqs[i].qcq);
+		lif->rxqcqs[i].qcq = NULL;
+	}
+

Why don't you break this down to stages/functions (disbale/uninit/free) as you did on open (alloc/ini/enable) ? will be easier to review and to spot inconsistencies

Quoted 77 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
 	return err;
 }
 
@@ -694,6 +806,7 @@ static int ionic_vlan_rx_kill_vid(struct
 net_device *netdev, __be16 proto,
 static const struct net_device_ops ionic_netdev_ops = {
 	.ndo_open               = ionic_open,
 	.ndo_stop               = ionic_stop,
+	.ndo_start_xmit		= ionic_start_xmit,
 	.ndo_get_stats64	= ionic_get_stats64,
 	.ndo_set_rx_mode	= ionic_set_rx_mode,
drivers/net/ethernet/pensando/ionic/ionic_lif.c +56 -0
 	.ndo_set_features	= ionic_set_features,
@@ -909,10 +1022,83 @@ static void ionic_qcq_free(struct lif *lif,
 struct qcq *qcq)
 	devm_kfree(dev, qcq);
 }
 
+static int ionic_txrx_alloc(struct lif *lif)
+{
+	unsigned int flags;
+	unsigned int i;
+	int err = 0;
+
+	flags = QCQ_F_TX_STATS | QCQ_F_SG;
+	for (i = 0; i < lif->nxqs; i++) {
+		err = ionic_qcq_alloc(lif, IONIC_QTYPE_TXQ, i, "tx",
 flags,
+				      lif->ntxq_descs,
+				      sizeof(struct txq_desc),
+				      sizeof(struct txq_comp),
+				      sizeof(struct txq_sg_desc),
+				      lif->kern_pid, &lif-
 >txqcqs[i].qcq);
+		if (err)
+			goto err_out_free_txqcqs;
+
+		lif->txqcqs[i].qcq->stats = lif->txqcqs[i].stats;
+	}
+
+	flags = QCQ_F_RX_STATS | QCQ_F_INTR;
+	for (i = 0; i < lif->nxqs; i++) {
+		err = ionic_qcq_alloc(lif, IONIC_QTYPE_RXQ, i, "rx",
 flags,
+				      lif->nrxq_descs,
+				      sizeof(struct rxq_desc),
+				      sizeof(struct rxq_comp),
+				      0, lif->kern_pid, &lif-
 >rxqcqs[i].qcq);
+		if (err)
+			goto err_out_free_rxqcqs;
+
+		lif->rxqcqs[i].qcq->stats = lif->rxqcqs[i].stats;
+
+		ionic_link_qcq_interrupts(lif->rxqcqs[i].qcq,
+					  lif->txqcqs[i].qcq);
+	}
+
+	return 0;
+
+err_out_free_rxqcqs:
+	for (i = 0; i < lif->nxqs; i++)
+		ionic_qcq_free(lif, lif->rxqcqs[i].qcq);
+err_out_free_txqcqs:
+	for (i = 0; i < lif->nxqs; i++)
+		ionic_qcq_free(lif, lif->txqcqs[i].qcq);
+
+	return err;
+}
+
+static int ionic_txrx_init(struct lif *lif)
+{
+	int err;
+
+	err = ionic_lif_txqs_init(lif);
+	if (err)
+		return err;
+

drivers/net/ethernet/pensando/ionic/ionic_lif.c static void ionic_qcq_free(struct lif *lif, · line 1095

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+	err = ionic_lif_rxqs_init(lif);
+	if (err)
+		goto err_out;
+
+	ionic_set_rx_mode(lif->netdev);
+
+	return 0;
+
+err_out:
+	ionic_stop(lif->netdev);

I would move error handling to the caller, keep the code flow and err path symmetrical with good path.

Quoted 19 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +7 -0
+
+	return err;
+}
+
 static int ionic_qcqs_alloc(struct lif *lif)
 {
+	struct device *dev = lif->ionic->dev;
+	unsigned int q_list_size;
 	unsigned int flags;
 	int err;
+	int i;
 
 	flags = QCQ_F_INTR;
 	err = ionic_qcq_alloc(lif, IONIC_QTYPE_ADMINQ, 0, "admin",

flags,

drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
@@ -937,8 +1123,47 @@ static int ionic_qcqs_alloc(struct lif *lif)
 		ionic_link_qcq_interrupts(lif->adminqcq, lif-
 >notifyqcq);
 	}

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_qcqs_alloc(struct lif *lif) · line 1135

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +8 -0
 
+	q_list_size = sizeof(*lif->txqcqs) * lif->nxqs;
+	err = -ENOMEM;
+	lif->txqcqs = devm_kzalloc(dev, q_list_size, GFP_KERNEL);
+	if (!lif->txqcqs)
+		goto err_out_free_notifyqcq;
+	for (i = 0; i < lif->nxqs; i++) {
+		lif->txqcqs[i].stats = devm_kzalloc(dev, sizeof(struct
 q_stats),
+						    GFP_KERNEL);

why not inlineing stats into txqcq struct and avoid this kzalloc ?

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_qcqs_alloc(struct lif *lif) · line 1145

Quoted 13 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +12 -0
+		if (!lif->txqcqs[i].stats)
+			goto err_out_free_tx_stats;
+	}
+
+	lif->rxqcqs = devm_kzalloc(dev, q_list_size, GFP_KERNEL);
+	if (!lif->rxqcqs)
+		goto err_out_free_tx_stats;
+	for (i = 0; i < lif->nxqs; i++) {
+		lif->rxqcqs[i].stats = devm_kzalloc(dev, sizeof(struct
 q_stats),
+						    GFP_KERNEL);
+		if (!lif->rxqcqs[i].stats)
+			goto err_out_free_rx_stats;

same

Quoted 176 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +19 -0
+	}
+
 	return 0;
 
+err_out_free_rx_stats:
+	for (i = 0; i < lif->nxqs; i++)
+		if (lif->rxqcqs[i].stats)
+			devm_kfree(dev, lif->rxqcqs[i].stats);
+	devm_kfree(dev, lif->rxqcqs);
+	lif->rxqcqs = NULL;
+err_out_free_tx_stats:
+	for (i = 0; i < lif->nxqs; i++)
+		if (lif->txqcqs[i].stats)
+			devm_kfree(dev, lif->txqcqs[i].stats);
+	devm_kfree(dev, lif->txqcqs);
+	lif->txqcqs = NULL;
+err_out_free_notifyqcq:
+	if (lif->notifyqcq) {
+		ionic_qcq_free(lif, lif->notifyqcq);
+		lif->notifyqcq = NULL;
+	}

err_out_free_adminqcq:

drivers/net/ethernet/pensando/ionic/ionic_lif.c +17 -0
 	ionic_qcq_free(lif, lif->adminqcq);
 	lif->adminqcq = NULL;
@@ -948,6 +1173,9 @@ static int ionic_qcqs_alloc(struct lif *lif)
 
 static void ionic_qcqs_free(struct lif *lif)
 {
+	struct device *dev = lif->ionic->dev;
+	unsigned int i;
+
 	if (lif->notifyqcq) {
 		ionic_qcq_free(lif, lif->notifyqcq);
 		lif->notifyqcq = NULL;
@@ -957,6 +1185,20 @@ static void ionic_qcqs_free(struct lif *lif)
 		ionic_qcq_free(lif, lif->adminqcq);
 		lif->adminqcq = NULL;
 	}
+
+	for (i = 0; i < lif->nxqs; i++)
+		if (lif->rxqcqs[i].stats)
+			devm_kfree(dev, lif->rxqcqs[i].stats);
+
+	devm_kfree(dev, lif->rxqcqs);
+	lif->rxqcqs = NULL;
+
+	for (i = 0; i < lif->nxqs; i++)
+		if (lif->txqcqs[i].stats)
+			devm_kfree(dev, lif->txqcqs[i].stats);
+
+	devm_kfree(dev, lif->txqcqs);
+	lif->txqcqs = NULL;
 }
 
 static struct lif *ionic_lif_alloc(struct ionic *ionic, unsigned int

index)

drivers/net/ethernet/pensando/ionic/ionic_lif.c +30 -0
@@ -992,6 +1234,8 @@ static struct lif *ionic_lif_alloc(struct ionic
 *ionic, unsigned int index)
 
 	lif->ionic = ionic;
 	lif->index = index;
+	lif->ntxq_descs = IONIC_DEF_TXRX_DESC;
+	lif->nrxq_descs = IONIC_DEF_TXRX_DESC;
 
 	snprintf(lif->name, sizeof(lif->name), "lif%u", index);
@@ -1432,6 +1676,147 @@ static int ionic_init_nic_features(struct lif
 *lif)
 	return 0;
 }
 
+static int ionic_lif_txq_init(struct lif *lif, struct qcq *qcq)
+{
+	struct device *dev = lif->ionic->dev;
+	struct queue *q = &qcq->q;
+	struct cq *cq = &qcq->cq;
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.q_init = {
+			.opcode = CMD_OPCODE_Q_INIT,
+			.lif_index = cpu_to_le16(lif->index),
+			.type = q->type,
+			.index = cpu_to_le32(q->index),
+			.flags = cpu_to_le16(IONIC_QINIT_F_IRQ |
+					     IONIC_QINIT_F_SG),
+			.intr_index = cpu_to_le16(lif->rxqcqs[q-
 >index].qcq->intr.index),
+			.pid = cpu_to_le16(q->pid),
+			.ring_size = ilog2(q->num_descs),
+			.ring_base = cpu_to_le64(q->base_pa),
+			.cq_ring_base = cpu_to_le64(cq->base_pa),
+			.sg_ring_base = cpu_to_le64(q->sg_base_pa),
+		},
+	};
+	int err;
+
+	dev_dbg(dev, "txq_init.pid %d\n", ctx.cmd.q_init.pid);
+	dev_dbg(dev, "txq_init.index %d\n", ctx.cmd.q_init.index);
+	dev_dbg(dev, "txq_init.ring_base 0x%llx\n",
 ctx.cmd.q_init.ring_base);
+	dev_dbg(dev, "txq_init.ring_size %d\n",

ctx.cmd.q_init.ring_size);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +53 -0
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	q->hw_type = ctx.comp.q_init.hw_type;
+	q->hw_index = le32_to_cpu(ctx.comp.q_init.hw_index);
+	q->dbval = IONIC_DBELL_QID(q->hw_index);
+
+	dev_dbg(dev, "txq->hw_type %d\n", q->hw_type);
+	dev_dbg(dev, "txq->hw_index %d\n", q->hw_index);
+
+	qcq->flags |= QCQ_F_INITED;
+
+	ionic_debugfs_add_qcq(lif, qcq);
+
+	return 0;
+}
+
+static int ionic_lif_txqs_init(struct lif *lif)
+{
+	unsigned int i;
+	int err;
+
+	for (i = 0; i < lif->nxqs; i++) {
+		err = ionic_lif_txq_init(lif, lif->txqcqs[i].qcq);
+		if (err)
+			goto err_out;
+	}
+
+	return 0;
+
+err_out:
+	for (; i > 0; i--)
+		ionic_lif_qcq_deinit(lif, lif->txqcqs[i-1].qcq);
+
+	return err;
+}
+
+static int ionic_lif_rxq_init(struct lif *lif, struct qcq *qcq)
+{
+	struct device *dev = lif->ionic->dev;
+	struct queue *q = &qcq->q;
+	struct cq *cq = &qcq->cq;
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.q_init = {
+			.opcode = CMD_OPCODE_Q_INIT,
+			.lif_index = cpu_to_le16(lif->index),
+			.type = q->type,
+			.index = cpu_to_le32(q->index),
+			.flags = cpu_to_le16(IONIC_QINIT_F_IRQ),
+			.intr_index = cpu_to_le16(cq->bound_intr-
Quoted 1 line

index),

drivers/net/ethernet/pensando/ionic/ionic_lif.c +11 -0
+			.pid = cpu_to_le16(q->pid),
+			.ring_size = ilog2(q->num_descs),
+			.ring_base = cpu_to_le64(q->base_pa),
+			.cq_ring_base = cpu_to_le64(cq->base_pa),
+		},
+	};
+	int err;
+
+	dev_dbg(dev, "rxq_init.pid %d\n", ctx.cmd.q_init.pid);
+	dev_dbg(dev, "rxq_init.index %d\n", ctx.cmd.q_init.index);
+	dev_dbg(dev, "rxq_init.ring_base 0x%llx\n",

ctx.cmd.q_init.ring_base);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
+	dev_dbg(dev, "rxq_init.ring_size %d\n",

ctx.cmd.q_init.ring_size);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +6 -0
+
+	err = ionic_adminq_post_wait(lif, &ctx);
+	if (err)
+		return err;
+
+	q->hw_type = ctx.comp.q_init.hw_type;

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_init_nic_features(struct lif *lif) · line 1787

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+	q->hw_index = le32_to_cpu(ctx.comp.q_init.hw_index);
+	q->dbval = IONIC_DBELL_QID(q->hw_index);
+
+	dev_dbg(dev, "rxq->hw_type %d\n", q->hw_type);
+	dev_dbg(dev, "rxq->hw_index %d\n", q->hw_index);
+
+	netif_napi_add(lif->netdev, &qcq->napi, ionic_rx_napi,
+		       NAPI_POLL_WEIGHT);
+
+	err = ionic_request_irq(lif, qcq);

Again, your init/deinit objects code paths is very asymmetrical I was looking for where you do free_irq, and eventually found it in ionic_lif_qcq_deinit which is shared between rx/tx queues, but it is very hard to follow who sets the QCQ_F_INTR flag so ionic_lif_qcq_deinit would free the irq. so i gave up !

I strongly suggest for you to make the code symmetrical as much as possible by following two simple rules:
1) each function "do_foo" must have a reverse function "undo_foo"..
2) each function "do_foo" should handle its own error path rollback and NOT by blindly calling "undo_foo"

This is a very well defined code structure and useful technique to help reviewers and developers keep track of what is happening, and most importantly a very good practice that reduces the number of current bugs and future bugs by an order of magnitude!

example:

open()
alloc_txrx_queues()
init_txrx_queues()
request_irqs()
enable_txrx_queues()

close()
disable_txrx_queues()
free_irqs()
deinit_txrx_queues()
dealloc_txrxqs()

this will remove the need for some flags and unnecessary state tracking for when a resources is open/close/inited, if each function is symmetrical and handles its own rollback, so at any point in the code you would know what to do without any extra if statement branches.

Quoted 399 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +34 -0
+	if (err) {
+		netif_napi_del(&qcq->napi);
+		return err;
+	}
+
+	qcq->flags |= QCQ_F_INITED;
+
+	ionic_debugfs_add_qcq(lif, qcq);
+
+	return 0;
+}
+
+static int ionic_lif_rxqs_init(struct lif *lif)
+{
+	unsigned int i;
+	int err;
+
+	for (i = 0; i < lif->nxqs; i++) {
+		err = ionic_lif_rxq_init(lif, lif->rxqcqs[i].qcq);
+		if (err)
+			goto err_out;
+	}
+
+	return 0;
+
+err_out:
+	for (; i > 0; i--)
+		ionic_lif_qcq_deinit(lif, lif->rxqcqs[i-1].qcq);
+
+	return err;
+}
+
 static int ionic_station_set(struct lif *lif)
 {
 	struct net_device *netdev = lif->netdev;
@@ -1531,6 +1916,8 @@ static int ionic_lif_init(struct lif *lif)
 	if (err)
 		goto err_out_notifyq_deinit;
 
+	lif->rx_copybreak = IONIC_RX_COPYBREAK_DEFAULT;
+
 	set_bit(LIF_INITED, lif->state);
 
 	ionic_link_status_check(lif);
drivers/net/ethernet/pensando/ionic/ionic_lif.h +28 -0
index d8589a306aa5..88d5bf8f58a1 100644
@@ -14,20 +14,38 @@
 #define MAX_NUM_NAPI_CNTR	(NAPI_POLL_WEIGHT + 1)
 #define GET_SG_CNTR_IDX(num_sg_elems)	(num_sg_elems)
 #define MAX_NUM_SG_CNTR		(IONIC_TX_MAX_SG_ELEMS + 1)
+#define IONIC_RX_COPYBREAK_DEFAULT		256
 
 struct tx_stats {
+	u64 dma_map_err;
 	u64 pkts;
 	u64 bytes;
+	u64 clean;
+	u64 linearize;
+	u64 no_csum;
+	u64 csum;
+	u64 crc32_csum;
+	u64 tso;
+	u64 frags;
+	u64 sg_cntr[MAX_NUM_SG_CNTR];
 };
 
 struct rx_stats {
+	u64 dma_map_err;
+	u64 alloc_err;
 	u64 pkts;
 	u64 bytes;
+	u64 csum_none;
+	u64 csum_complete;
+	u64 csum_error;
+	u64 buffers_posted;
 };
 
 #define QCQ_F_INITED		BIT(0)
 #define QCQ_F_SG		BIT(1)
 #define QCQ_F_INTR		BIT(2)
+#define QCQ_F_TX_STATS		BIT(3)
+#define QCQ_F_RX_STATS		BIT(4)
 #define QCQ_F_NOTIFYQ		BIT(5)
 
 struct napi_stats {
@@ -56,7 +74,14 @@ struct qcq {
 	struct dentry *dentry;
 };
 
+struct qcqst {
+	struct qcq *qcq;
+	struct q_stats *stats;
+};
+
 #define q_to_qcq(q)		container_of(q, struct qcq, q)
+#define q_to_tx_stats(q)	(&q_to_qcq(q)->stats->tx)
+#define q_to_rx_stats(q)	(&q_to_qcq(q)->stats->rx)
 #define napi_to_qcq(napi)	container_of(napi, struct qcq, napi)
 #define napi_to_cq(napi)	(&napi_to_qcq(napi)->cq)
 
@@ -108,11 +133,14 @@ struct lif {
 	spinlock_t adminq_lock;		/* lock for AdminQ operations
 */
 	struct qcq *adminqcq;
 	struct qcq *notifyqcq;
+	struct qcqst *txqcqs;
+	struct qcqst *rxqcqs;
 	u64 last_eid;
 	unsigned int neqs;
 	unsigned int nxqs;
 	unsigned int ntxq_descs;
 	unsigned int nrxq_descs;
+	u32 rx_copybreak;
 	unsigned int rx_mode;
 	u64 hw_features;
drivers/net/ethernet/pensando/ionic/ionic_lif.h +21 -0
 	bool mc_overflow;
@@ -134,6 +162,11 @@ struct lif {
 	u32 flags;
 };
 
+#define lif_to_txqcq(lif, i)	((lif)->txqcqs[i].qcq)
+#define lif_to_rxqcq(lif, i)	((lif)->rxqcqs[i].qcq)
+#define lif_to_txq(lif, i)	(&lif_to_txqcq((lif), i)->q)
+#define lif_to_rxq(lif, i)	(&lif_to_txqcq((lif), i)->q)
+
 static inline bool ionic_is_pf(struct ionic *ionic)
 {
 	return ionic->pdev &&
@@ -173,6 +206,22 @@ int ionic_open(struct net_device *netdev);
 int ionic_stop(struct net_device *netdev);
 int ionic_reset_queues(struct lif *lif);
 
+static inline void debug_stats_txq_post(struct qcq *qcq,
+					struct txq_desc *desc, bool
 dbell)
+{
+	u8 num_sg_elems = ((le64_to_cpu(desc->cmd) >>
 IONIC_TXQ_DESC_NSGE_SHIFT)
+						&
 IONIC_TXQ_DESC_NSGE_MASK);
+	u8 sg_cntr_idx;
+
+	qcq->q.dbell_count += dbell;
+
+	sg_cntr_idx = GET_SG_CNTR_IDX(num_sg_elems);
+	if (sg_cntr_idx > (MAX_NUM_SG_CNTR - 1))
+		sg_cntr_idx = MAX_NUM_SG_CNTR - 1;
+
+	qcq->stats->tx.sg_cntr[sg_cntr_idx]++;
+}
+

static inline void debug_stats_napi_poll(struct qcq *qcq,

drivers/net/ethernet/pensando/ionic/ionic_lif.h +3 -0
 					 unsigned int work_done)
 {
@@ -188,7 +237,10 @@ static inline void debug_stats_napi_poll(struct
 qcq *qcq,
 }
 
 #define DEBUG_STATS_CQE_CNT(cq)		((cq)->compl_count++)
+#define DEBUG_STATS_RX_BUFF_CNT(qcq)	((qcq)->stats-
 >rx.buffers_posted++)
 #define DEBUG_STATS_INTR_REARM(intr)	((intr)->rearm_count++)
+#define DEBUG_STATS_TXQ_POST(qcq, txdesc, dbell) \
+	debug_stats_txq_post(qcq, txdesc, dbell)
 #define DEBUG_STATS_NAPI_POLL(qcq, work_done) \
drivers/net/ethernet/pensando/ionic/ionic_lif.h +0 -0
 	debug_stats_napi_poll(qcq, work_done)
drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +13 -0
new file mode 100644
index 000000000000..7787e7dd5504
@@ -0,0 +1,879 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/if_vlan.h>
+#include <net/ip6_checksum.h>
+
+#include "ionic.h"
+#include "ionic_lif.h"
+#include "ionic_txrx.h"
+
+static void ionic_tx_clean(struct queue *q, struct desc_info

*desc_info,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +2 -0
+			   struct cq_info *cq_info, void *cb_arg);
+static void ionic_rx_clean(struct queue *q, struct desc_info

*desc_info,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +19 -0
+			   struct cq_info *cq_info, void *cb_arg);
+
+static inline void ionic_txq_post(struct queue *q, bool ring_dbell,
+				  desc_cb cb_func, void *cb_arg)
+{
+	DEBUG_STATS_TXQ_POST(q_to_qcq(q), q->head->desc, ring_dbell);
+
+	ionic_q_post(q, ring_dbell, cb_func, cb_arg);
+}
+
+static inline void ionic_rxq_post(struct queue *q, bool ring_dbell,
+				  desc_cb cb_func, void *cb_arg)
+{
+	ionic_q_post(q, ring_dbell, cb_func, cb_arg);
+
+	DEBUG_STATS_RX_BUFF_CNT(q_to_qcq(q));
+}
+
+static void ionic_rx_recycle(struct queue *q, struct desc_info

*desc_info,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +12 -0
+			     struct sk_buff *skb)
+{
+	struct rxq_desc *old = desc_info->desc;
+	struct rxq_desc *new = q->head->desc;
+
+	new->addr = old->addr;
+	new->len = old->len;
+
+	ionic_rxq_post(q, true, ionic_rx_clean, skb);
+}
+
+static bool ionic_rx_copybreak(struct queue *q, struct desc_info

*desc_info,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +1 -0
+			       struct cq_info *cq_info, struct sk_buff

**skb)

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +12 -0
+{
+	struct net_device *netdev = q->lif->netdev;
+	struct device *dev = q->lif->ionic->dev;
+	struct rxq_desc *desc = desc_info->desc;
+	struct rxq_comp *comp = cq_info->cq_desc;
+	struct sk_buff *new_skb;
+	u16 clen, dlen;
+
+	clen = le16_to_cpu(comp->len);
+	dlen = le16_to_cpu(desc->len);
+	if (clen > q->lif->rx_copybreak) {
+		dma_unmap_single(dev, (dma_addr_t)le64_to_cpu(desc-
Quoted 1 line

addr),

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +7 -0
+				 dlen, DMA_FROM_DEVICE);
+		return false;
+	}
+
+	new_skb = netdev_alloc_skb_ip_align(netdev, clen);
+	if (!new_skb) {
+		dma_unmap_single(dev, (dma_addr_t)le64_to_cpu(desc-
Quoted 1 line

addr),

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +5 -0
+				 dlen, DMA_FROM_DEVICE);
+		return false;
+	}
+
+	dma_sync_single_for_cpu(dev, (dma_addr_t)le64_to_cpu(desc-
Quoted 1 line

addr),

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +11 -0
+				clen, DMA_FROM_DEVICE);
+
+	memcpy(new_skb->data, (*skb)->data, clen);
+
+	ionic_rx_recycle(q, desc_info, *skb);
+	*skb = new_skb;
+
+	return true;
+}
+
+static void ionic_rx_clean(struct queue *q, struct desc_info

*desc_info,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +34 -0
+			   struct cq_info *cq_info, void *cb_arg)
+{
+	struct rxq_comp *comp = cq_info->cq_desc;
+	struct sk_buff *skb = cb_arg;
+	struct qcq *qcq = q_to_qcq(q);
+	struct net_device *netdev;
+	struct rx_stats *stats;
+
+	stats = q_to_rx_stats(q);
+	netdev = q->lif->netdev;
+
+	if (comp->status) {
+		ionic_rx_recycle(q, desc_info, skb);
+		return;
+	}
+
+	if (unlikely(test_bit(LIF_QUEUE_RESET, q->lif->state))) {
+		/* no packet processing while resetting */
+		ionic_rx_recycle(q, desc_info, skb);
+		return;
+	}
+
+	stats->pkts++;
+	stats->bytes += le16_to_cpu(comp->len);
+
+	ionic_rx_copybreak(q, desc_info, cq_info, &skb);
+
+	skb_put(skb, le16_to_cpu(comp->len));
+	skb->protocol = eth_type_trans(skb, netdev);
+
+	skb_record_rx_queue(skb, q->index);
+
+	if (netdev->features & NETIF_F_RXHASH) {
+		switch (comp->pkt_type_color &

IONIC_RXQ_COMP_PKT_TYPE_MASK) {

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +34 -0
+		case PKT_TYPE_IPV4:
+		case PKT_TYPE_IPV6:
+			skb_set_hash(skb, le32_to_cpu(comp->rss_hash),
+				     PKT_HASH_TYPE_L3);
+			break;
+		case PKT_TYPE_IPV4_TCP:
+		case PKT_TYPE_IPV6_TCP:
+		case PKT_TYPE_IPV4_UDP:
+		case PKT_TYPE_IPV6_UDP:
+			skb_set_hash(skb, le32_to_cpu(comp->rss_hash),
+				     PKT_HASH_TYPE_L4);
+			break;
+		}
+	}
+
+	if (netdev->features & NETIF_F_RXCSUM) {
+		if (comp->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
+			skb->ip_summed = CHECKSUM_COMPLETE;
+			skb->csum = (__wsum)le16_to_cpu(comp->csum);
+			stats->csum_complete++;
+		}
+	} else {
+		stats->csum_none++;
+	}
+
+	if ((comp->csum_flags & IONIC_RXQ_COMP_CSUM_F_TCP_BAD) ||
+	    (comp->csum_flags & IONIC_RXQ_COMP_CSUM_F_UDP_BAD) ||
+	    (comp->csum_flags & IONIC_RXQ_COMP_CSUM_F_IP_BAD))
+		stats->csum_error++;
+
+	if (netdev->features & NETIF_F_HW_VLAN_CTAG_RX) {
+		if (comp->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN)
+			__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q),
+					       le16_to_cpu(comp-
Quoted 1 line

vlan_tci));

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +59 -0
+	}
+
+	napi_gro_receive(&qcq->napi, skb);
+}
+
+static bool ionic_rx_service(struct cq *cq, struct cq_info *cq_info)
+{
+	struct rxq_comp *comp = cq_info->cq_desc;
+	struct queue *q = cq->bound_q;
+	struct desc_info *desc_info;
+
+	if (!color_match(comp->pkt_type_color, cq->done_color))
+		return false;
+
+	/* check for empty queue */
+	if (q->tail->index == q->head->index)
+		return false;
+
+	desc_info = q->tail;
+	if (desc_info->index != le16_to_cpu(comp->comp_index))
+		return false;
+
+	q->tail = desc_info->next;
+
+	/* clean the related q entry, only one per qc completion */
+	ionic_rx_clean(q, desc_info, cq_info, desc_info->cb_arg);
+
+	desc_info->cb = NULL;
+	desc_info->cb_arg = NULL;
+
+	return true;
+}
+
+static u32 ionic_rx_walk_cq(struct cq *rxcq, u32 limit)
+{
+	u32 work_done = 0;
+
+	while (ionic_rx_service(rxcq, rxcq->tail)) {
+		if (rxcq->tail->last)
+			rxcq->done_color = !rxcq->done_color;
+		rxcq->tail = rxcq->tail->next;
+		DEBUG_STATS_CQE_CNT(rxcq);
+
+		if (++work_done >= limit)
+			break;
+	}
+
+	return work_done;
+}
+
+void ionic_rx_flush(struct cq *cq)
+{
+	struct ionic_dev *idev = &cq->lif->ionic->idev;
+	u32 work_done;
+
+	work_done = ionic_rx_walk_cq(cq, cq->num_descs);
+
+	if (work_done)
+		ionic_intr_credits(idev->intr_ctrl, cq->bound_intr-
Quoted 1 line

index,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +1 -0
+				   work_done,
Quoted 2 lines

IONIC_INTR_CRED_RESET_COALESCE);
+}

mixing between rx and tx unrelated flows, very hard to review.

Quoted 234 lines
drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +24 -0
+
+void ionic_tx_flush(struct cq *cq)
+{
+	struct ionic_dev *idev = &cq->lif->ionic->idev;
+	struct txq_comp *comp = cq->tail->cq_desc;
+	struct queue *q = cq->bound_q;
+	struct desc_info *desc_info;
+	unsigned int work_done = 0;
+
+	/* walk the completed cq entries */
+	while (work_done < cq->num_descs &&
+	       color_match(comp->color, cq->done_color)) {
+
+		/* clean the related q entries, there could be
+		 * several q entries completed for each cq completion
+		 */
+		do {
+			desc_info = q->tail;
+			q->tail = desc_info->next;
+			ionic_tx_clean(q, desc_info, cq->tail,
+				       desc_info->cb_arg);
+			desc_info->cb = NULL;
+			desc_info->cb_arg = NULL;
+		} while (desc_info->index != le16_to_cpu(comp-
Quoted 1 line

comp_index));

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +13 -0
+
+		if (cq->tail->last)
+			cq->done_color = !cq->done_color;
+
+		cq->tail = cq->tail->next;
+		comp = cq->tail->cq_desc;
+		DEBUG_STATS_CQE_CNT(cq);
+
+		work_done++;
+	}
+
+	if (work_done)
+		ionic_intr_credits(idev->intr_ctrl, cq->bound_intr-
Quoted 1 line

index,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +4 -0
+				   work_done, 0);
+}
+
+static struct sk_buff *ionic_rx_skb_alloc(struct queue *q, unsigned

int len,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +18 -0
+					  dma_addr_t *dma_addr)
+{
+	struct lif *lif = q->lif;
+	struct net_device *netdev = lif->netdev;
+	struct device *dev = lif->ionic->dev;
+	struct rx_stats *stats;
+	struct sk_buff *skb;
+
+	stats = q_to_rx_stats(q);
+	skb = netdev_alloc_skb_ip_align(netdev, len);
+	if (!skb) {
+		net_warn_ratelimited("%s: SKB alloc failed on %s!\n",
+				     netdev->name, q->name);
+		stats->alloc_err++;
+		return NULL;
+	}
+
+	*dma_addr = dma_map_single(dev, skb->data, len,

DMA_FROM_DEVICE);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +3 -0
+	if (dma_mapping_error(dev, *dma_addr)) {
+		dev_kfree_skb(skb);
+		net_warn_ratelimited("%s: DMA single map failed on

%s!\n",

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +62 -0
+				     netdev->name, q->name);
+		stats->dma_map_err++;
+		return NULL;
+	}
+
+	return skb;
+}
+
+static void ionic_rx_skb_free(struct queue *q, struct sk_buff *skb,
+			      unsigned int len, dma_addr_t dma_addr)
+{
+	struct device *dev = q->lif->ionic->dev;
+
+	dma_unmap_single(dev, dma_addr, len, DMA_FROM_DEVICE);
+	dev_kfree_skb(skb);
+}
+
+#define RX_RING_DOORBELL_STRIDE		((1 << 2) - 1)
+
+void ionic_rx_fill(struct queue *q)
+{
+	struct net_device *netdev = q->lif->netdev;
+	struct rxq_desc *desc;
+	struct sk_buff *skb;
+	dma_addr_t dma_addr;
+	bool ring_doorbell;
+	unsigned int len;
+	unsigned int i;
+
+	len = netdev->mtu + ETH_HLEN;
+
+	for (i = ionic_q_space_avail(q); i; i--) {
+		skb = ionic_rx_skb_alloc(q, len, &dma_addr);
+		if (!skb)
+			return;
+
+		desc = q->head->desc;
+		desc->addr = cpu_to_le64(dma_addr);
+		desc->len = cpu_to_le16(len);
+		desc->opcode = RXQ_DESC_OPCODE_SIMPLE;
+
+		ring_doorbell = ((q->head->index + 1) &
+				RX_RING_DOORBELL_STRIDE) == 0;
+
+		ionic_rxq_post(q, ring_doorbell, ionic_rx_clean, skb);
+	}
+}
+
+static void ionic_rx_fill_cb(void *arg)
+{
+	ionic_rx_fill(arg);
+}
+
+void ionic_rx_empty(struct queue *q)
+{
+	struct desc_info *cur = q->tail;
+	struct rxq_desc *desc;
+
+	while (cur != q->head) {
+		desc = cur->desc;
+
+		ionic_rx_skb_free(q, cur->cb_arg, le16_to_cpu(desc-
Quoted 1 line

len),

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +26 -0
+				  le64_to_cpu(desc->addr));
+		cur->cb_arg = NULL;
+
+		cur = cur->next;
+	}
+}
+
+int ionic_rx_napi(struct napi_struct *napi, int budget)
+{
+	struct qcq *qcq = napi_to_qcq(napi);
+	struct cq *rxcq = napi_to_cq(napi);
+	unsigned int qi = rxcq->bound_q->index;
+	struct lif *lif = rxcq->bound_q->lif;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	struct cq *txcq = &lif->txqcqs[qi].qcq->cq;
+	u32 work_done = 0;
+	u32 flags = 0;
+
+	ionic_tx_flush(txcq);
+
+	work_done = ionic_rx_walk_cq(rxcq, budget);
+
+	if (work_done)
+		ionic_rx_fill_cb(rxcq->bound_q);
+
+	if (work_done < budget && napi_complete_done(napi, work_done))

{

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +7 -0
+		flags |= IONIC_INTR_CRED_UNMASK;
+		DEBUG_STATS_INTR_REARM(rxcq->bound_intr);
+	}
+
+	if (work_done || flags) {
+		flags |= IONIC_INTR_CRED_RESET_COALESCE;
+		ionic_intr_credits(idev->intr_ctrl, rxcq->bound_intr-
Quoted 1 line

index,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +9 -0
+				   work_done, flags);
+	}
+
+	DEBUG_STATS_NAPI_POLL(qcq, work_done);
+
+	return work_done;
+}
+
+static dma_addr_t ionic_tx_map_single(struct queue *q, void *data,

size_t len)

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +8 -0
+{
+	struct tx_stats *stats = q_to_tx_stats(q);
+	struct device *dev = q->lif->ionic->dev;
+	dma_addr_t dma_addr;
+
+	dma_addr = dma_map_single(dev, data, len, DMA_TO_DEVICE);
+	if (dma_mapping_error(dev, dma_addr)) {
+		net_warn_ratelimited("%s: DMA single map failed on

%s!\n",

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +8 -0
+				     q->lif->netdev->name, q->name);
+		stats->dma_map_err++;
+		return 0;
+	}
+	return dma_addr;
+}
+
+static dma_addr_t ionic_tx_map_frag(struct queue *q, const

skb_frag_t *frag,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +7 -0
+				    size_t offset, size_t len)
+{
+	struct tx_stats *stats = q_to_tx_stats(q);
+	struct device *dev = q->lif->ionic->dev;
+	dma_addr_t dma_addr;
+
+	dma_addr = skb_frag_dma_map(dev, frag, offset, len,

DMA_TO_DEVICE);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +2 -0
+	if (dma_mapping_error(dev, dma_addr)) {
+		net_warn_ratelimited("%s: DMA frag map failed on

%s!\n",

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +8 -0
+				     q->lif->netdev->name, q->name);
+		stats->dma_map_err++;
+		return 0;
+	}
+	return dma_addr;
+}
+
+static void ionic_tx_clean(struct queue *q, struct desc_info

*desc_info,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +20 -0
+			   struct cq_info *cq_info, void *cb_arg)
+{
+	struct txq_sg_desc *sg_desc = desc_info->sg_desc;
+	struct txq_sg_elem *elem = sg_desc->elems;
+	struct tx_stats *stats = q_to_tx_stats(q);
+	struct txq_desc *desc = desc_info->desc;
+	struct device *dev = q->lif->ionic->dev;
+	struct sk_buff *skb = cb_arg;
+	u8 opcode, flags, nsge;
+	u16 queue_index;
+	unsigned int i;
+	u64 addr;
+
+	decode_txq_desc_cmd(le64_to_cpu(desc->cmd),
+			    &opcode, &flags, &nsge, &addr);
+
+	dma_unmap_page(dev, (dma_addr_t)addr,
+		       le16_to_cpu(desc->len), DMA_TO_DEVICE);
+	for (i = 0; i < nsge; i++, elem++)
+		dma_unmap_page(dev, (dma_addr_t)le64_to_cpu(elem-
Quoted 1 line

addr),

drivers/net/ethernet/pensando/ionic/ionic_txrx.c line 433

Quoted 3 lines
drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +3 -0
+			       le16_to_cpu(elem->len), DMA_TO_DEVICE);
+
+	if (skb) {

when skb is null and queue is stopped what do you do ?
also you should do this at the end of napi cycle.. and not per packet.

Quoted 439 lines
drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +4 -0
+		queue_index = skb_get_queue_mapping(skb);
+		if (unlikely(__netif_subqueue_stopped(q->lif->netdev,
+						      queue_index))) {
+			netif_wake_subqueue(q->lif->netdev,

queue_index);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +44 -0
+			q->wake++;
+		}
+		dev_kfree_skb_any(skb);
+		stats->clean++;
+	}
+}
+
+static void ionic_tx_tcp_inner_pseudo_csum(struct sk_buff *skb)
+{
+	skb_cow_head(skb, 0);
+
+	if (skb->protocol == cpu_to_be16(ETH_P_IP)) {
+		inner_ip_hdr(skb)->check = 0;
+		inner_tcp_hdr(skb)->check =
+			~csum_tcpudp_magic(inner_ip_hdr(skb)->saddr,
+					   inner_ip_hdr(skb)->daddr,
+					   0, IPPROTO_TCP, 0);
+	} else if (skb->protocol == cpu_to_be16(ETH_P_IPV6)) {
+		inner_tcp_hdr(skb)->check =
+			~csum_ipv6_magic(&inner_ipv6_hdr(skb)->saddr,
+					 &inner_ipv6_hdr(skb)->daddr,
+					 0, IPPROTO_TCP, 0);
+	}
+}
+
+static void ionic_tx_tcp_pseudo_csum(struct sk_buff *skb)
+{
+	skb_cow_head(skb, 0);
+
+	if (skb->protocol == cpu_to_be16(ETH_P_IP)) {
+		ip_hdr(skb)->check = 0;
+		tcp_hdr(skb)->check =
+			~csum_tcpudp_magic(ip_hdr(skb)->saddr,
+					   ip_hdr(skb)->daddr,
+					   0, IPPROTO_TCP, 0);
+	} else if (skb->protocol == cpu_to_be16(ETH_P_IPV6)) {
+		tcp_hdr(skb)->check =
+			~csum_ipv6_magic(&ipv6_hdr(skb)->saddr,
+					 &ipv6_hdr(skb)->daddr,
+					 0, IPPROTO_TCP, 0);
+	}
+}
+
+static void ionic_tx_tso_post(struct queue *q, struct txq_desc

*desc,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +16 -0
+			      struct sk_buff *skb,
+			      dma_addr_t addr, u8 nsge, u16 len,
+			      unsigned int hdrlen, unsigned int mss,
+			      bool outer_csum,
+			      u16 vlan_tci, bool has_vlan,
+			      bool start, bool done)
+{
+	u8 flags = 0;
+	u64 cmd;
+
+	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
+	flags |= outer_csum ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
+	flags |= start ? IONIC_TXQ_DESC_FLAG_TSO_SOT : 0;
+	flags |= done ? IONIC_TXQ_DESC_FLAG_TSO_EOT : 0;
+
+	cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_TSO, flags,

nsge, addr);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +9 -0
+	desc->cmd = cpu_to_le64(cmd);
+	desc->len = cpu_to_le16(len);
+	desc->vlan_tci = cpu_to_le16(vlan_tci);
+	desc->hdr_len = cpu_to_le16(hdrlen);
+	desc->mss = cpu_to_le16(mss);
+
+	if (done) {
+		skb_tx_timestamp(skb);
+		ionic_txq_post(q, !netdev_xmit_more(), ionic_tx_clean,

skb);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +48 -0
+	} else {
+		ionic_txq_post(q, false, ionic_tx_clean, NULL);
+	}
+}
+
+static struct txq_desc *ionic_tx_tso_next(struct queue *q,
+					  struct txq_sg_elem **elem)
+{
+	struct txq_sg_desc *sg_desc = q->head->sg_desc;
+	struct txq_desc *desc = q->head->desc;
+
+	*elem = sg_desc->elems;
+	return desc;
+}
+
+static int ionic_tx_tso(struct queue *q, struct sk_buff *skb)
+{
+	struct tx_stats *stats = q_to_tx_stats(q);
+	struct desc_info *abort = q->head;
+	struct desc_info *rewind = abort;
+	unsigned int frag_left = 0;
+	struct txq_sg_elem *elem;
+	unsigned int offset = 0;
+	unsigned int len_left;
+	struct txq_desc *desc;
+	dma_addr_t desc_addr;
+	unsigned int hdrlen;
+	unsigned int nfrags;
+	unsigned int seglen;
+	u64 total_bytes = 0;
+	u64 total_pkts = 0;
+	unsigned int left;
+	unsigned int len;
+	unsigned int mss;
+	skb_frag_t *frag;
+	bool start, done;
+	bool outer_csum;
+	bool has_vlan;
+	u16 desc_len;
+	u8 desc_nsge;
+	u16 vlan_tci;
+	bool encap;
+
+	mss = skb_shinfo(skb)->gso_size;
+	nfrags = skb_shinfo(skb)->nr_frags;
+	len_left = skb->len - skb_headlen(skb);
+	outer_csum = (skb_shinfo(skb)->gso_type & SKB_GSO_GRE_CSUM) ||
+		     (skb_shinfo(skb)->gso_type &

SKB_GSO_UDP_TUNNEL_CSUM);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +32 -0
+	has_vlan = !!skb_vlan_tag_present(skb);
+	vlan_tci = skb_vlan_tag_get(skb);
+	encap = skb->encapsulation;
+
+	/* Preload inner-most TCP csum field with IP pseudo hdr
+	 * calculated with IP length set to zero.  HW will later
+	 * add in length to each TCP segment resulting from the TSO.
+	 */
+
+	if (encap)
+		ionic_tx_tcp_inner_pseudo_csum(skb);
+	else
+		ionic_tx_tcp_pseudo_csum(skb);
+
+	if (encap)
+		hdrlen = skb_inner_transport_header(skb) - skb->data +
+			 inner_tcp_hdrlen(skb);
+	else
+		hdrlen = skb_transport_offset(skb) + tcp_hdrlen(skb);
+
+	seglen = hdrlen + mss;
+	left = skb_headlen(skb);
+
+	desc = ionic_tx_tso_next(q, &elem);
+	start = true;
+
+	/* Chop skb->data up into desc segments */
+
+	while (left > 0) {
+		len = min(seglen, left);
+		frag_left = seglen - len;
+		desc_addr = ionic_tx_map_single(q, skb->data + offset,

len);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +37 -0
+		if (!desc_addr)
+			goto err_out_abort;
+		desc_len = len;
+		desc_nsge = 0;
+		left -= len;
+		offset += len;
+		if (nfrags > 0 && frag_left > 0)
+			continue;
+		done = (nfrags == 0 && left == 0);
+		ionic_tx_tso_post(q, desc, skb,
+				  desc_addr, desc_nsge, desc_len,
+				  hdrlen, mss,
+				  outer_csum,
+				  vlan_tci, has_vlan,
+				  start, done);
+		total_pkts++;
+		total_bytes += start ? len : len + hdrlen;
+		desc = ionic_tx_tso_next(q, &elem);
+		start = false;
+		seglen = mss;
+	}
+
+	/* Chop skb frags into desc segments */
+
+	for (frag = skb_shinfo(skb)->frags; len_left; frag++) {
+		offset = 0;
+		left = skb_frag_size(frag);
+		len_left -= left;
+		nfrags--;
+		stats->frags++;
+
+		while (left > 0) {
+			if (frag_left > 0) {
+				len = min(frag_left, left);
+				frag_left -= len;
+				elem->addr =
+				    cpu_to_le64(ionic_tx_map_frag(q,

frag,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +1 -0
+								  offse

t, len));

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +11 -0
+				if (!elem->addr)
+					goto err_out_abort;
+				elem->len = cpu_to_le16(len);
+				elem++;
+				desc_nsge++;
+				left -= len;
+				offset += len;
+				if (nfrags > 0 && frag_left > 0)
+					continue;
+				done = (nfrags == 0 && left == 0);
+				ionic_tx_tso_post(q, desc, skb,

desc_addr,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +2 -0
+						  desc_nsge, desc_len,
+						  hdrlen, mss,

outer_csum,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +4 -0
+						  vlan_tci, has_vlan,
+						  start, done);
+				total_pkts++;
+				total_bytes += start ? len : len +

hdrlen;

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +7 -0
+				desc = ionic_tx_tso_next(q, &elem);
+				start = false;
+			} else {
+				len = min(mss, left);
+				frag_left = mss - len;
+				desc_addr = ionic_tx_map_frag(q, frag,
+							      offset,

len);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +10 -0
+				if (!desc_addr)
+					goto err_out_abort;
+				desc_len = len;
+				desc_nsge = 0;
+				left -= len;
+				offset += len;
+				if (nfrags > 0 && frag_left > 0)
+					continue;
+				done = (nfrags == 0 && left == 0);
+				ionic_tx_tso_post(q, desc, skb,

desc_addr,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +2 -0
+						  desc_nsge, desc_len,
+						  hdrlen, mss,

outer_csum,

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +4 -0
+						  vlan_tci, has_vlan,
+						  start, done);
+				total_pkts++;
+				total_bytes += start ? len : len +

hdrlen;

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +44 -0
+				desc = ionic_tx_tso_next(q, &elem);
+				start = false;
+			}
+		}
+	}
+
+	stats->pkts += total_pkts;
+	stats->bytes += total_bytes;
+	stats->tso++;
+
+	return 0;
+
+err_out_abort:
+	while (rewind->desc != q->head->desc) {
+		ionic_tx_clean(q, rewind, NULL, NULL);
+		rewind = rewind->next;
+	}
+	q->head = abort;
+
+	return -ENOMEM;
+}
+
+static int ionic_tx_calc_csum(struct queue *q, struct sk_buff *skb)
+{
+	struct tx_stats *stats = q_to_tx_stats(q);
+	struct txq_desc *desc = q->head->desc;
+	dma_addr_t addr;
+	bool has_vlan;
+	u8 flags = 0;
+	bool encap;
+	u64 cmd;
+
+	has_vlan = !!skb_vlan_tag_present(skb);
+	encap = skb->encapsulation;
+
+	addr = ionic_tx_map_single(q, skb->data, skb_headlen(skb));
+	if (!addr)
+		return -ENOMEM;
+
+	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
+	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
+
+	cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_CSUM_PARTIAL,
+				  flags, skb_shinfo(skb)->nr_frags,

addr);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +15 -0
+	desc->cmd = cpu_to_le64(cmd);
+	desc->len = cpu_to_le16(skb_headlen(skb));
+	desc->vlan_tci = cpu_to_le16(skb_vlan_tag_get(skb));
+	desc->csum_start = cpu_to_le16(skb_checksum_start_offset(skb));
+	desc->csum_offset = cpu_to_le16(skb->csum_offset);
+
+	if (skb->csum_not_inet)
+		stats->crc32_csum++;
+	else
+		stats->csum++;
+
+	return 0;
+}
+
+static int ionic_tx_calc_no_csum(struct queue *q, struct sk_buff

*skb)

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +21 -0
+{
+	struct tx_stats *stats = q_to_tx_stats(q);
+	struct txq_desc *desc = q->head->desc;
+	dma_addr_t addr;
+	bool has_vlan;
+	u8 flags = 0;
+	bool encap;
+	u64 cmd;
+
+	has_vlan = !!skb_vlan_tag_present(skb);
+	encap = skb->encapsulation;
+
+	addr = ionic_tx_map_single(q, skb->data, skb_headlen(skb));
+	if (!addr)
+		return -ENOMEM;
+
+	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
+	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
+
+	cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_CSUM_NONE,
+				  flags, skb_shinfo(skb)->nr_frags,

addr);

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +59 -0
+	desc->cmd = cpu_to_le64(cmd);
+	desc->len = cpu_to_le16(skb_headlen(skb));
+	desc->vlan_tci = cpu_to_le16(skb_vlan_tag_get(skb));
+
+	stats->no_csum++;
+
+	return 0;
+}
+
+static int ionic_tx_skb_frags(struct queue *q, struct sk_buff *skb)
+{
+	unsigned int len_left = skb->len - skb_headlen(skb);
+	struct txq_sg_desc *sg_desc = q->head->sg_desc;
+	struct txq_sg_elem *elem = sg_desc->elems;
+	struct tx_stats *stats = q_to_tx_stats(q);
+	dma_addr_t dma_addr;
+	skb_frag_t *frag;
+	u16 len;
+
+	for (frag = skb_shinfo(skb)->frags; len_left; frag++, elem++) {
+		len = skb_frag_size(frag);
+		elem->len = cpu_to_le16(len);
+		dma_addr = ionic_tx_map_frag(q, frag, 0, len);
+		if (!dma_addr)
+			return -ENOMEM;
+		elem->addr = cpu_to_le64(dma_addr);
+		len_left -= len;
+		stats->frags++;
+	}
+
+	return 0;
+}
+
+static int ionic_tx(struct queue *q, struct sk_buff *skb)
+{
+	struct tx_stats *stats = q_to_tx_stats(q);
+	int err;
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL)
+		err = ionic_tx_calc_csum(q, skb);
+	else
+		err = ionic_tx_calc_no_csum(q, skb);
+	if (err)
+		return err;
+
+	err = ionic_tx_skb_frags(q, skb);
+	if (err)
+		return err;
+
+	skb_tx_timestamp(skb);
+	stats->pkts++;
+	stats->bytes += skb->len;
+
+	ionic_txq_post(q, !netdev_xmit_more(), ionic_tx_clean, skb);
+
+	return 0;
+}
+
+static int ionic_tx_descs_needed(struct queue *q, struct sk_buff

*skb)

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +24 -0
+{
+	struct tx_stats *stats = q_to_tx_stats(q);
+	int err;
+
+	/* If TSO, need roundup(skb->len/mss) descs */
+	if (skb_is_gso(skb))
+		return (skb->len / skb_shinfo(skb)->gso_size) + 1;
+
+	/* If non-TSO, just need 1 desc and nr_frags sg elems */
+	if (skb_shinfo(skb)->nr_frags <= IONIC_TX_MAX_SG_ELEMS)
+		return 1;
+
+	/* Too many frags, so linearize */
+	err = skb_linearize(skb);
+	if (err)
+		return err;
+
+	stats->linearize++;
+
+	/* Need 1 desc and zero sg elems */
+	return 1;
+}
+
+netdev_tx_t ionic_start_xmit(struct sk_buff *skb, struct net_device

*netdev)

drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +25 -0
+{
+	u16 queue_index = skb_get_queue_mapping(skb);
+	struct lif *lif = netdev_priv(netdev);
+	struct queue *q;
+	int ndescs;
+	int err;
+
+	if (unlikely(!test_bit(LIF_UP, lif->state))) {
+		dev_kfree_skb(skb);
+		return NETDEV_TX_OK;
+	}
+
+	if (likely(lif_to_txqcq(lif, queue_index)))
+		q = lif_to_txq(lif, queue_index);
+	else
+		q = lif_to_txq(lif, 0);
+
+	ndescs = ionic_tx_descs_needed(q, skb);
+	if (ndescs < 0)
+		goto err_out_drop;
+
+	if (!ionic_q_has_space(q, ndescs)) {
+		netif_stop_subqueue(netdev, queue_index);
+		q->stop++;
+

drivers/net/ethernet/pensando/ionic/ionic_txrx.c line 862

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +10 -0
+		/* Might race with ionic_tx_clean, check again */
+		smp_rmb();
+		if (ionic_q_has_space(q, ndescs)) {
+			netif_wake_subqueue(netdev, queue_index);
+			q->wake++;
+		} else {
+			return NETDEV_TX_BUSY;
+		}
+	}
+

You are missing netdev_tx_sent_queue() here
and corresponding netdev_tx_completed_queue() on completion.

Quoted 40 lines
drivers/net/ethernet/pensando/ionic/ionic_txrx.c new file +17 -0
+	if (skb_is_gso(skb))
+		err = ionic_tx_tso(q, skb);
+	else
+		err = ionic_tx(q, skb);
+
+	if (err)
+		goto err_out_drop;
+
+	return NETDEV_TX_OK;
+
+err_out_drop:
+	netif_stop_subqueue(netdev, queue_index);
+	q->stop++;
+	q->drop++;
+	dev_kfree_skb(skb);
+	return NETDEV_TX_OK;
+}
drivers/net/ethernet/pensando/ionic/ionic_txrx.h new file +13 -0
new file mode 100644
index 000000000000..2391a0eec65a
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_TXRX_H_
+#define _IONIC_TXRX_H_
+
+void ionic_rx_flush(struct cq *cq);
+void ionic_tx_flush(struct cq *cq);
+
+void ionic_rx_fill(struct queue *q);
+void ionic_rx_empty(struct queue *q);
+int ionic_rx_napi(struct napi_struct *napi, int budget);
+netdev_tx_t ionic_start_xmit(struct sk_buff *skb, struct net_device

*netdev);

drivers/net/ethernet/pensando/ionic/ionic_txrx.h new file +2 -0
+
+#endif /* _IONIC_TXRX_H_ */
#54 lore raw

Re: [PATCH v4 net-next 14/19] ionic: Add Tx and Rx handling

Quoted 68 lines · Saeed Mahameed
Quoted 60 lines · Shannon Nelson

Add both the Tx and Rx queue setup and handling. The related stats display comes later. Instead of using the generic napi routines used by the slow-path commands, the Tx and Rx paths are simplified and inlined in one file in order to get better compiler optimizations.

---
  drivers/net/ethernet/pensando/ionic/Makefile  |   2 +-
  .../net/ethernet/pensando/ionic/ionic_lif.c   | 387 ++++++++
  .../net/ethernet/pensando/ionic/ionic_lif.h   |  52 ++
  .../net/ethernet/pensando/ionic/ionic_txrx.c  | 879
++++++++++++++++++
  .../net/ethernet/pensando/ionic/ionic_txrx.h  |  15 +
  5 files changed, 1334 insertions(+), 1 deletion(-)

create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.c create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_txrx.h

drivers/net/ethernet/pensando/ionic/Makefile +1 -1
index 9b19bf57a489..0e2dc53f08d4 100644
@@ -4,4 +4,4 @@
  obj-$(CONFIG_IONIC) := ionic.o
 
  ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_ethtool.o
 \
-	   ionic_lif.o ionic_rx_filter.o ionic_debugfs.o
+	   ionic_lif.o ionic_rx_filter.o ionic_txrx.o ionic_debugfs.o
drivers/net/ethernet/pensando/ionic/ionic_lif.c +8 -0
index 2bd8ce61c4a0..40d3b1cb362a 100644
@@ -10,6 +10,7 @@
  #include "ionic.h"
  #include "ionic_bus.h"
  #include "ionic_lif.h"
+#include "ionic_txrx.h"
  #include "ionic_ethtool.h"
  #include "ionic_debugfs.h"
 
@@ -18,6 +19,13 @@ static int ionic_lif_addr_add(struct lif *lif,
 const u8 *addr);
  static int ionic_lif_addr_del(struct lif *lif, const u8 *addr);
  static void ionic_link_status_check(struct lif *lif);
 
+static int ionic_lif_stop(struct lif *lif);
+static int ionic_txrx_alloc(struct lif *lif);
+static int ionic_txrx_init(struct lif *lif);
+static void ionic_qcq_free(struct lif *lif, struct qcq *qcq);
+static int ionic_lif_txqs_init(struct lif *lif);
+static int ionic_lif_rxqs_init(struct lif *lif);
+static void ionic_lif_qcq_deinit(struct lif *lif, struct qcq *qcq);
  static int ionic_set_nic_features(struct lif *lif, netdev_features_t
 features);

static int ionic_notifyq_clean(struct lif *lif, int budget);

drivers/net/ethernet/pensando/ionic/ionic_lif.c static void ionic_lif_deferred_enqueue(struct · line 76

Quoted 3 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
@@ -66,12 +74,96 @@ static void ionic_lif_deferred_enqueue(struct
 ionic_deferred *def,
  	schedule_work(&def->work);
  }

Bottom up or top down ? your current design is very mixed and I had to to scroll down and up too often just to understand what a function is doing, i strongly suggest to pick an use one approach.

Quoted 2 lines

[1]
https://en.wikipedia.org/wiki/Top-down_and_bottom-up_design#Programming

Hi Saeed,

Sorry it has taken me so long to address the comments you had on this patch: I've been contemplating them while working some internal issues.  Part of my hesitance is also because, while correct, these are suggesting some significant rework and juggling of stable code. I've been aware of these issues since I inherited the code, and have been addressing them over time while trying to keep on top of stability, and was planning to continue doing so.  It sounds like I need to evolve the driver a little further before the next patchset version.  I think I'll have time in the next week to do this, so I'm hoping to have a new patchset out in another week or so.

Thanks for the inputs,
sln

#55 lore raw

PATCH v4 15/19 ionic: Add netdev-event handling

In reply to Shannon Nelson

When the netdev gets a new name from userland, pass that name down to the NIC for internal tracking.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |  2 +
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 65 +++++++++++++++++++
 2 files changed, 67 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic.h +2 -0
index d7eee79b2a10..9b720187b549 100644
@@ -42,6 +42,8 @@ struct ionic {
 	DECLARE_BITMAP(lifbits, IONIC_LIFS_MAX);
 	unsigned int nintrs;
 	DECLARE_BITMAP(intrs, INTR_CTRL_REGS_MAX);
+	struct work_struct nb_work;
+	struct notifier_block nb;
 };
 
 struct ionic_admin_ctx {
drivers/net/ethernet/pensando/ionic/ionic_lif.c +65 -0

81 lines not on this page · raw message

Signature · 1 line
2.17.1
#56 lore raw

Re: [PATCH v4 net-next 15/19] ionic: Add netdev-event handling

Quoted 3 lines · Shannon Nelson

When the netdev gets a new name from userland, pass that name down to the NIC for internal tracking.

Just out of curiosity, why your NIC internal device/firmware need to keep tracking of the netdev name ?

#57 lore raw

Re: [PATCH v4 net-next 15/19] ionic: Add netdev-event handling

Quoted 8 lines · Saeed Mahameed
Quoted 3 lines · Shannon Nelson

When the netdev gets a new name from userland, pass that name down to the NIC for internal tracking.

Just out of curiosity, why your NIC internal device/firmware need to keep tracking of the netdev name ?

It is helpful in a debugging method inside the NIC firmware.

sln

#58 lore raw

Re: [PATCH v4 net-next 15/19] ionic: Add netdev-event handling

In reply to Shannon Nelson
Quoted 8 lines · On Mon, 22 Jul 2019 14:40:19 -0700 Shannon Nelson <[email protected]> wrote:
drivers/net/ethernet/pensando/ionic/ionic_lif.c +8 -0
+
+static void ionic_lif_set_netdev_info(struct lif *lif)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.lif_setattr = {
+			.opcode = CMD_OPCODE_LIF_SETATTR,
+			.index = cpu_to_le16(lif->index),

drivers/net/ethernet/pensando/ionic/ionic_lif.c int ionic_lifs_init(struct ionic *ionic) · line 1977

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +10 -0
+			.attr = IONIC_LIF_ATTR_NAME,
+		},
+	};
+
+	strlcpy(ctx.cmd.lif_setattr.name, lif->netdev->name,
+		sizeof(ctx.cmd.lif_setattr.name));
+
+	dev_info(lif->ionic->dev, "NETDEV_CHANGENAME %s %s\n",
+		 lif->name, ctx.cmd.lif_setattr.name);
+

There is already a kernel message for this. Repeating the same thing in the driver is redundant.

#59 lore raw

Re: [PATCH v4 net-next 15/19] ionic: Add netdev-event handling

Quoted 15 lines · Stephen Hemminger
Quoted 12 lines · On Mon, 22 Jul 2019 14:40:19 -0700 Shannon Nelson <[email protected]> wrote:
drivers/net/ethernet/pensando/ionic/ionic_lif.c +12 -0
+
+static void ionic_lif_set_netdev_info(struct lif *lif)
+{
+	struct ionic_admin_ctx ctx = {
+		.work = COMPLETION_INITIALIZER_ONSTACK(ctx.work),
+		.cmd.lif_setattr = {
+			.opcode = CMD_OPCODE_LIF_SETATTR,
+			.index = cpu_to_le16(lif->index),
+			.attr = IONIC_LIF_ATTR_NAME,
+		},
+	};
+
Quoted 8 lines

drivers/net/ethernet/pensando/ionic/ionic_lif.c int ionic_lifs_init(struct ionic *ionic) · line 1977

Quoted 6 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +6 -0
+	strlcpy(ctx.cmd.lif_setattr.name, lif->netdev->name,
+		sizeof(ctx.cmd.lif_setattr.name));
+
+	dev_info(lif->ionic->dev, "NETDEV_CHANGENAME %s %s\n",
+		 lif->name, ctx.cmd.lif_setattr.name);
+

There is already a kernel message for this. Repeating the same thing in the driver is redundant.

True, except for the lif name information.  But since that really is debugging information, and I was getting tired of seeing those redundant messages, I was planning to remove them soon anyway. Thanks for the extra nudge.

sln

#60 lore raw

PATCH v4 16/19 ionic: Add driver stats

In reply to Shannon Nelson

Add in the detailed statistics for ethtool -S that the driver keeps as it processes packets. Display of the additional debug statistics can be enabled through the ethtool priv-flags feature.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |   3 +-
 .../ethernet/pensando/ionic/ionic_ethtool.c   | 101 ++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  14 +
 .../net/ethernet/pensando/ionic/ionic_stats.c | 333 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_stats.h |  53 +++
 5 files changed, 503 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_stats.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_stats.h
drivers/net/ethernet/pensando/ionic/Makefile +2 -1
index 0e2dc53f08d4..4f3cfbf36c23 100644
@@ -4,4 +4,5 @@
 obj-$(CONFIG_IONIC) := ionic.o
 
 ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_ethtool.o \
-	   ionic_lif.o ionic_rx_filter.o ionic_txrx.o ionic_debugfs.o
+	   ionic_lif.o ionic_rx_filter.o ionic_txrx.o ionic_debugfs.o \
+	   ionic_stats.o
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +101 -0

119 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h +14 -0
index 88d5bf8f58a1..f93bfa2b4393 100644
@@ -108,8 +108,22 @@ struct ionic_deferred {
 	struct work_struct work;
 };
 
+struct lif_sw_stats {
+	u64 tx_packets;
+	u64 tx_bytes;
+	u64 rx_packets;
+	u64 rx_bytes;
+	u64 tx_tso;
+	u64 tx_no_csum;
+	u64 tx_csum;
+	u64 rx_csum_none;
+	u64 rx_csum_complete;
+	u64 rx_csum_error;
+};
+
 enum lif_state_flags {
 	LIF_INITED,
+	LIF_SW_DEBUG_STATS,
 	LIF_UP,
 	LIF_LINK_CHECK_NEEDED,
 	LIF_QUEUE_RESET,
drivers/net/ethernet/pensando/ionic/ionic_stats.c new file +333 -0

333 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_stats.h new file +53 -0

53 lines not on this page · raw message

Signature · 1 line
2.17.1
#61 lore raw

PATCH v4 17/19 ionic: Add RSS support

In reply to Shannon Nelson

Add code to manipulate through ethtool the RSS configuration used by the NIC.

---
 .../ethernet/pensando/ionic/ionic_ethtool.c   | 73 ++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 87 +++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.h   |  8 ++
 3 files changed, 168 insertions(+)
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +73 -0
index d7503a3dafb7..742d7d47f4d8 100644
@@ -468,6 +468,74 @@ static int ionic_set_channels(struct net_device *netdev,
 	return 0;
 }
 
+static int ionic_get_rxnfc(struct net_device *netdev,
+			   struct ethtool_rxnfc *info, u32 *rules)
+{
+	struct lif *lif = netdev_priv(netdev);
+	int err = 0;
+
+	switch (info->cmd) {
+	case ETHTOOL_GRXRINGS:
+		info->data = lif->nxqs;
+		break;
+	default:
+		netdev_err(netdev, "Command parameter %d is not supported\n",
+			   info->cmd);
+		err = -EOPNOTSUPP;
+	}
+
+	return err;
+}
+
+static u32 ionic_get_rxfh_indir_size(struct net_device *netdev)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	return le16_to_cpu(lif->ionic->ident.lif.eth.rss_ind_tbl_sz);
+}
+
+static u32 ionic_get_rxfh_key_size(struct net_device *netdev)
+{
+	return IONIC_RSS_HASH_KEY_SIZE;
+}
+
+static int ionic_get_rxfh(struct net_device *netdev, u32 *indir, u8 *key,
+			  u8 *hfunc)
+{
+	struct lif *lif = netdev_priv(netdev);
+	unsigned int i, tbl_sz;
+
+	if (indir) {
+		tbl_sz = le16_to_cpu(lif->ionic->ident.lif.eth.rss_ind_tbl_sz);
+		for (i = 0; i < tbl_sz; i++)
+			indir[i] = lif->rss_ind_tbl[i];
+	}
+
+	if (key)
+		memcpy(key, lif->rss_hash_key, IONIC_RSS_HASH_KEY_SIZE);
+
+	if (hfunc)
+		*hfunc = ETH_RSS_HASH_TOP;
+
+	return 0;
+}
+
+static int ionic_set_rxfh(struct net_device *netdev, const u32 *indir,
+			  const u8 *key, const u8 hfunc)
+{
+	struct lif *lif = netdev_priv(netdev);
+	int err;
+
+	if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP)
+		return -EOPNOTSUPP;
+
+	err = ionic_lif_rss_config(lif, lif->rss_types, key, indir);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 static u32 ionic_get_priv_flags(struct net_device *netdev)
 {
 	struct lif *lif = netdev_priv(netdev);
@@ -580,6 +648,11 @@ static const struct ethtool_ops ionic_ethtool_ops = {
 	.get_strings		= ionic_get_strings,
 	.get_ethtool_stats	= ionic_get_stats,
 	.get_sset_count		= ionic_get_sset_count,
+	.get_rxnfc		= ionic_get_rxnfc,
+	.get_rxfh_indir_size	= ionic_get_rxfh_indir_size,
+	.get_rxfh_key_size	= ionic_get_rxfh_key_size,
+	.get_rxfh		= ionic_get_rxfh,
+	.set_rxfh		= ionic_set_rxfh,
 	.get_priv_flags		= ionic_get_priv_flags,
 	.set_priv_flags		= ionic_set_priv_flags,
 	.get_module_info	= ionic_get_module_info,
drivers/net/ethernet/pensando/ionic/ionic_lif.c +87 -0

133 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.h +8 -0
index f93bfa2b4393..0e6908f959f2 100644
@@ -166,6 +166,12 @@ struct lif {
 	dma_addr_t info_pa;
 	u32 info_sz;
 
+	u16 rss_types;
+	u8 rss_hash_key[IONIC_RSS_HASH_KEY_SIZE];
+	u8 *rss_ind_tbl;
+	dma_addr_t rss_ind_tbl_pa;
+	u32 rss_ind_tbl_sz;
+
 	struct rx_filters rx_filters;
 	struct ionic_deferred deferred;
 	u32 tx_coalesce_usecs;
@@ -215,6 +221,8 @@ void ionic_lifs_unregister(struct ionic *ionic);
 int ionic_lif_identify(struct ionic *ionic, u8 lif_type,
 		       union lif_identity *lif_ident);
 int ionic_lifs_size(struct ionic *ionic);
+int ionic_lif_rss_config(struct lif *lif, u16 types,
+			 const u8 *key, const u32 *indir);
 
 int ionic_open(struct net_device *netdev);
 int ionic_stop(struct net_device *netdev);
Signature · 1 line
2.17.1
#62 lore raw

PATCH v4 18/19 ionic: Add coalesce and other features

In reply to Shannon Nelson

Interrupt coalescing, tunable copybreak value, and tx timeout.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |   2 +-
 .../ethernet/pensando/ionic/ionic_ethtool.c   | 105 ++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.c   |  13 ++-
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   1 +
 4 files changed, 119 insertions(+), 2 deletions(-)
drivers/net/ethernet/pensando/ionic/ionic.h +1 -1
index 9b720187b549..cd08166f73a9 100644
@@ -11,7 +11,7 @@ struct lif;
 
 #define DRV_NAME		"ionic"
 #define DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
-#define DRV_VERSION		"0.11.0-k"
+#define DRV_VERSION		"0.11.0-44-k"
 
 #define PCI_VENDOR_ID_PENSANDO			0x1dd8
 
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +105 -0

129 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_lif.c +12 -1
index 68a9975e34c6..8473b065763b 100644
@@ -744,9 +744,19 @@ static int ionic_change_mtu(struct net_device *netdev, int new_mtu)
 	return err;
 }
 
+static void ionic_tx_timeout_work(struct work_struct *ws)
+{
+	struct lif *lif = container_of(ws, struct lif, tx_timeout_work);
+
+	netdev_info(lif->netdev, "Tx Timeout recovery\n");
+	ionic_reset_queues(lif);
+}
+
 static void ionic_tx_timeout(struct net_device *netdev)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
+	struct lif *lif = netdev_priv(netdev);
+
+	schedule_work(&lif->tx_timeout_work);
 }
 
 static int ionic_vlan_rx_add_vid(struct net_device *netdev, __be16 proto,
@@ -2009,6 +2019,7 @@ static int ionic_lif_init(struct lif *lif)
 
 	ionic_link_status_check(lif);
 
+	INIT_WORK(&lif->tx_timeout_work, ionic_tx_timeout_work);
 	return 0;
 
 err_out_notifyq_deinit:
drivers/net/ethernet/pensando/ionic/ionic_lif.h +1 -0
index 0e6908f959f2..76cc519acd5a 100644
@@ -180,6 +180,7 @@ struct lif {
 	unsigned int dbid_count;
 	struct dentry *dentry;
 	u32 flags;
+	struct work_struct tx_timeout_work;
 };
 
 #define lif_to_txqcq(lif, i)	((lif)->txqcqs[i].qcq)
Signature · 1 line
2.17.1
maintainer
#63 lore raw

Re: [PATCH v4 net-next 18/19] ionic: Add coalesce and other features

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:22 -0700

Quoted 7 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +0 -0
index 742d7d47f4d8..e6b579a40b70 100644
@@ -377,6 +377,75 @@ static int ionic_get_coalesce(struct net_device *netdev,
 	return 0;
 }

drivers/net/ethernet/pensando/ionic/ionic_ethtool.c static int ionic_get_coalesce(struct net_device *netdev, · line 388

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +9 -0
 
+static int ionic_set_coalesce(struct net_device *netdev,
+			      struct ethtool_coalesce *coalesce)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct identity *ident = &lif->ionic->ident;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u32 tx_coal, rx_coal;
+	struct qcq *qcq;
+	unsigned int i;

Reverse christmas tree please.

#64 lore raw

Re: [PATCH v4 net-next 18/19] ionic: Add coalesce and other features

In reply to Shannon Nelson
Quoted 177 lines · Shannon Nelson

Interrupt coalescing, tunable copybreak value, and tx timeout.

---
 drivers/net/ethernet/pensando/ionic/ionic.h   |   2 +-
 .../ethernet/pensando/ionic/ionic_ethtool.c   | 105
++++++++++++++++++
 .../net/ethernet/pensando/ionic/ionic_lif.c   |  13 ++-
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   1 +
 4 files changed, 119 insertions(+), 2 deletions(-)
drivers/net/ethernet/pensando/ionic/ionic.h +1 -1
index 9b720187b549..cd08166f73a9 100644
@@ -11,7 +11,7 @@ struct lif;
 
 #define DRV_NAME		"ionic"
 #define DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
-#define DRV_VERSION		"0.11.0-k"
+#define DRV_VERSION		"0.11.0-44-k"
 
 #define PCI_VENDOR_ID_PENSANDO			0x1dd8
 
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +69 -0
index 742d7d47f4d8..e6b579a40b70 100644
@@ -377,6 +377,75 @@ static int ionic_get_coalesce(struct net_device
 *netdev,
 	return 0;
 }
 
+static int ionic_set_coalesce(struct net_device *netdev,
+			      struct ethtool_coalesce *coalesce)
+{
+	struct lif *lif = netdev_priv(netdev);
+	struct identity *ident = &lif->ionic->ident;
+	struct ionic_dev *idev = &lif->ionic->idev;
+	u32 tx_coal, rx_coal;
+	struct qcq *qcq;
+	unsigned int i;
+
+	if (coalesce->rx_max_coalesced_frames ||
+	    coalesce->rx_coalesce_usecs_irq ||
+	    coalesce->rx_max_coalesced_frames_irq ||
+	    coalesce->tx_max_coalesced_frames ||
+	    coalesce->tx_coalesce_usecs_irq ||
+	    coalesce->tx_max_coalesced_frames_irq ||
+	    coalesce->stats_block_coalesce_usecs ||
+	    coalesce->use_adaptive_rx_coalesce ||
+	    coalesce->use_adaptive_tx_coalesce ||
+	    coalesce->pkt_rate_low ||
+	    coalesce->rx_coalesce_usecs_low ||
+	    coalesce->rx_max_coalesced_frames_low ||
+	    coalesce->tx_coalesce_usecs_low ||
+	    coalesce->tx_max_coalesced_frames_low ||
+	    coalesce->pkt_rate_high ||
+	    coalesce->rx_coalesce_usecs_high ||
+	    coalesce->rx_max_coalesced_frames_high ||
+	    coalesce->tx_coalesce_usecs_high ||
+	    coalesce->tx_max_coalesced_frames_high ||
+	    coalesce->rate_sample_interval)
+		return -EINVAL;
+
+	if (ident->dev.intr_coal_div == 0)
+		return -EIO;
+
+	/* Convert from usecs to device units */
+	tx_coal = coalesce->tx_coalesce_usecs *
+		  le32_to_cpu(ident->dev.intr_coal_mult) /
+		  le32_to_cpu(ident->dev.intr_coal_div);
+	rx_coal = coalesce->rx_coalesce_usecs *
+		  le32_to_cpu(ident->dev.intr_coal_mult) /
+		  le32_to_cpu(ident->dev.intr_coal_div);
+
+	if (tx_coal > INTR_CTRL_COAL_MAX || rx_coal >
 INTR_CTRL_COAL_MAX)
+		return -ERANGE;
+
+	if (coalesce->tx_coalesce_usecs != lif->tx_coalesce_usecs) {
+		for (i = 0; i < lif->nxqs; i++) {
+			qcq = lif->txqcqs[i].qcq;
+			ionic_intr_coal_init(idev->intr_ctrl,
+					     qcq->intr.index,
+					     tx_coal);
+		}
+		lif->tx_coalesce_usecs = coalesce->tx_coalesce_usecs;
+	}
+
+	if (coalesce->rx_coalesce_usecs != lif->rx_coalesce_usecs) {
+		for (i = 0; i < lif->nxqs; i++) {
+			qcq = lif->rxqcqs[i].qcq;
+			ionic_intr_coal_init(idev->intr_ctrl,
+					     qcq->intr.index,
+					     rx_coal);
+		}
+		lif->rx_coalesce_usecs = coalesce->rx_coalesce_usecs;
+	}
+
+	return 0;
+}
+
 static void ionic_get_ringparam(struct net_device *netdev,
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +33 -0
 				struct ethtool_ringparam *ring)
 {
@@ -562,6 +631,39 @@ static int ionic_set_priv_flags(struct
 net_device *netdev, u32 priv_flags)
 	return 0;
 }
 
+static int ionic_set_tunable(struct net_device *dev,
+			     const struct ethtool_tunable *tuna,
+			     const void *data)
+{
+	struct lif *lif = netdev_priv(dev);
+
+	switch (tuna->id) {
+	case ETHTOOL_RX_COPYBREAK:
+		lif->rx_copybreak = *(u32 *)data;
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+
+	return 0;
+}
+
+static int ionic_get_tunable(struct net_device *netdev,
+			     const struct ethtool_tunable *tuna, void
 *data)
+{
+	struct lif *lif = netdev_priv(netdev);
+
+	switch (tuna->id) {
+	case ETHTOOL_RX_COPYBREAK:
+		*(u32 *)data = lif->rx_copybreak;
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+
+	return 0;
+}
+
 static int ionic_get_module_info(struct net_device *netdev,
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +1 -0
 				 struct ethtool_modinfo *modinfo)
@@ -641,6 +743,7 @@ static const struct ethtool_ops ionic_ethtool_ops
 = {
 	.get_link		= ethtool_op_get_link,
 	.get_link_ksettings	= ionic_get_link_ksettings,
 	.get_coalesce		= ionic_get_coalesce,
+	.set_coalesce		= ionic_set_coalesce,
 	.get_ringparam		= ionic_get_ringparam,
 	.set_ringparam		= ionic_set_ringparam,
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +2 -0
 	.get_channels		= ionic_get_channels,
@@ -655,6 +758,8 @@ static const struct ethtool_ops ionic_ethtool_ops
 = {
 	.set_rxfh		= ionic_set_rxfh,
 	.get_priv_flags		= ionic_get_priv_flags,
 	.set_priv_flags		= ionic_set_priv_flags,
+	.get_tunable		= ionic_get_tunable,
+	.set_tunable		= ionic_set_tunable,
 	.get_module_info	= ionic_get_module_info,
 	.get_module_eeprom	= ionic_get_module_eeprom,
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +0 -0
 	.get_pauseparam		= ionic_get_pauseparam,
drivers/net/ethernet/pensando/ionic/ionic_lif.c +0 -0
index 68a9975e34c6..8473b065763b 100644
@@ -744,9 +744,19 @@ static int ionic_change_mtu(struct net_device
 *netdev, int new_mtu)

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_change_mtu(struct net_device · line 754

Quoted 10 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +6 -0
 	return err;
 }
 
+static void ionic_tx_timeout_work(struct work_struct *ws)
+{
+	struct lif *lif = container_of(ws, struct lif,
 tx_timeout_work);
+
+	netdev_info(lif->netdev, "Tx Timeout recovery\n");
+	ionic_reset_queues(lif);

missing rtnl_lock ?

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_change_mtu(struct net_device *netdev, int new_mtu) · line 760

Quoted 9 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +5 -1
+}
+
 static void ionic_tx_timeout(struct net_device *netdev)
 {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
+	struct lif *lif = netdev_priv(netdev);
+
+	schedule_work(&lif->tx_timeout_work);
 }

missing cancel work ? be careful when combined with the rtnl_lockthough ..

Quoted 24 lines

static int ionic_vlan_rx_add_vid(struct net_device *netdev, __be16
proto,

drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
@@ -2009,6 +2019,7 @@ static int ionic_lif_init(struct lif *lif)
 
 	ionic_link_status_check(lif);
 
+	INIT_WORK(&lif->tx_timeout_work, ionic_tx_timeout_work);
 	return 0;
 
 err_out_notifyq_deinit:
drivers/net/ethernet/pensando/ionic/ionic_lif.h +1 -0
index 0e6908f959f2..76cc519acd5a 100644
@@ -180,6 +180,7 @@ struct lif {
 	unsigned int dbid_count;
 	struct dentry *dentry;
 	u32 flags;
+	struct work_struct tx_timeout_work;
 };
 
 #define lif_to_txqcq(lif, i)	((lif)->txqcqs[i].qcq)
#65 lore raw

Re: [PATCH v4 net-next 18/19] ionic: Add coalesce and other features

Quoted 11 lines · Saeed Mahameed

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_change_mtu(struct net_device *netdev, int new_mtu) · line 752

Quoted 7 lines · Shannon Nelson
drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
+static void ionic_tx_timeout_work(struct work_struct *ws)
+{
+	struct lif *lif = container_of(ws, struct lif,

tx_timeout_work);

drivers/net/ethernet/pensando/ionic/ionic_lif.c +3 -0
+
+	netdev_info(lif->netdev, "Tx Timeout recovery\n");
+	ionic_reset_queues(lif);

missing rtnl_lock ?

Quoted 12 lines

drivers/net/ethernet/pensando/ionic/ionic_lif.c static int ionic_change_mtu(struct net_device *netdev, int new_mtu) · line 760

Quoted 9 lines
drivers/net/ethernet/pensando/ionic/ionic_lif.c +5 -1
+}
+
  static void ionic_tx_timeout(struct net_device *netdev)
  {
-	netdev_info(netdev, "%s: stubbed\n", __func__);
+	struct lif *lif = netdev_priv(netdev);
+
+	schedule_work(&lif->tx_timeout_work);
  }

missing cancel work ? be careful when combined with the rtnl_lockthough ..

Yep, good catch on both.  I'll take care of those.

sln

#66 lore raw

PATCH v4 19/19 ionic: Add basic devlink interface

In reply to Shannon Nelson

Add a devlink interface for access to information that isn't normally available through ethtool or the iplink interface.

---
 drivers/net/ethernet/pensando/ionic/Makefile  |  2 +-
 drivers/net/ethernet/pensando/ionic/ionic.h   |  3 +
 .../ethernet/pensando/ionic/ionic_bus_pci.c   | 12 ++-
 .../ethernet/pensando/ionic/ionic_devlink.c   | 93 +++++++++++++++++++
 .../ethernet/pensando/ionic/ionic_devlink.h   | 14 +++
 5 files changed, 120 insertions(+), 4 deletions(-)
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_devlink.c
 create mode 100644 drivers/net/ethernet/pensando/ionic/ionic_devlink.h
drivers/net/ethernet/pensando/ionic/Makefile +1 -1
index 4f3cfbf36c23..ce187c7b33a8 100644
@@ -5,4 +5,4 @@ obj-$(CONFIG_IONIC) := ionic.o
 
 ionic-y := ionic_main.o ionic_bus_pci.o ionic_dev.o ionic_ethtool.o \
 	   ionic_lif.o ionic_rx_filter.o ionic_txrx.o ionic_debugfs.o \
-	   ionic_stats.o
+	   ionic_stats.o ionic_devlink.o
drivers/net/ethernet/pensando/ionic/ionic.h +3 -0
index cd08166f73a9..728dfb42a52a 100644
@@ -8,6 +8,7 @@ struct lif;
 
 #include "ionic_if.h"
 #include "ionic_dev.h"
+#include "ionic_devlink.h"
 
 #define DRV_NAME		"ionic"
 #define DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
@@ -44,6 +45,8 @@ struct ionic {
 	DECLARE_BITMAP(intrs, INTR_CTRL_REGS_MAX);
 	struct work_struct nb_work;
 	struct notifier_block nb;
+	struct devlink *dl;
+	struct devlink_port dl_port;
 };
 
 struct ionic_admin_ctx {
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +9 -3
index 98c12b770c7f..dfc0df2c8c8f 100644
@@ -10,6 +10,7 @@
 #include "ionic_bus.h"
 #include "ionic_lif.h"
 #include "ionic_debugfs.h"
+#include "ionic_devlink.h"
 
 /* Supported devices */
 static const struct pci_device_id ionic_id_table[] = {
@@ -110,7 +111,7 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	struct ionic *ionic;
 	int err;
 
-	ionic = devm_kzalloc(dev, sizeof(*ionic), GFP_KERNEL);
+	ionic = ionic_devlink_alloc(dev);
 	if (!ionic)
 		return -ENOMEM;
 
@@ -212,6 +213,10 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 		goto err_out_deinit_lifs;
 	}
 
+	err = ionic_devlink_register(ionic);
+	if (err)
+		dev_err(dev, "Cannot register devlink: %d\n", err);
+
 	return 0;
 
 err_out_deinit_lifs:
@@ -237,6 +242,7 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	ionic_debugfs_del_dev(ionic);
 err_out_clear_drvdata:
 	mutex_destroy(&ionic->dev_cmd_lock);
+	ionic_devlink_free(ionic);
 	pci_set_drvdata(pdev, NULL);
 
 	return err;
@@ -247,6 +253,7 @@ static void ionic_remove(struct pci_dev *pdev)
 	struct ionic *ionic = pci_get_drvdata(pdev);
 
 	if (ionic) {
+		ionic_devlink_unregister(ionic);
 		ionic_lifs_unregister(ionic);
 		ionic_lifs_deinit(ionic);
 		ionic_lifs_free(ionic);
@@ -261,8 +268,7 @@ static void ionic_remove(struct pci_dev *pdev)
 		pci_disable_device(pdev);
 		ionic_debugfs_del_dev(ionic);
 		mutex_destroy(&ionic->dev_cmd_lock);
-
-		devm_kfree(&pdev->dev, ionic);
+		ionic_devlink_free(ionic);
 	}
 }
 
drivers/net/ethernet/pensando/ionic/ionic_devlink.c new file +93 -0

93 lines not on this page · raw message

drivers/net/ethernet/pensando/ionic/ionic_devlink.h new file +14 -0
new file mode 100644
index 000000000000..0690172fc57a
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2017 - 2019 Pensando Systems, Inc */
+
+#ifndef _IONIC_DEVLINK_H_
+#define _IONIC_DEVLINK_H_
+
+#include <net/devlink.h>
+
+struct ionic *ionic_devlink_alloc(struct device *dev);
+void ionic_devlink_free(struct ionic *ionic);
+int ionic_devlink_register(struct ionic *ionic);
+void ionic_devlink_unregister(struct ionic *ionic);
+
+#endif /* _IONIC_DEVLINK_H_ */
Signature · 1 line
2.17.1
maintainer
#67 lore raw

Re: [PATCH v4 net-next 19/19] ionic: Add basic devlink interface

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:23 -0700

drivers/net/ethernet/pensando/ionic/ionic_devlink.c line 47

Quoted 4 lines
drivers/net/ethernet/pensando/ionic/ionic_devlink.c new file +4 -0
+struct ionic *ionic_devlink_alloc(struct device *dev)
+{
+	struct devlink *dl;
+	struct ionic *ionic;

Reverse christmas tree please.

#68 lore raw

Re: [PATCH v4 net-next 19/19] ionic: Add basic devlink interface

Quoted 8 lines · David Miller

From: Shannon Nelson <[email protected]>
Date: Mon, 22 Jul 2019 14:40:23 -0700

drivers/net/ethernet/pensando/ionic/ionic_devlink.c line 47

Quoted 4 lines
drivers/net/ethernet/pensando/ionic/ionic_devlink.c new file +4 -0
+struct ionic *ionic_devlink_alloc(struct device *dev)
+{
+	struct devlink *dl;
+	struct ionic *ionic;

Reverse christmas tree please.

Yep, I missed this one.

Thanks for your review time.
sln