ldmvsw: Add ldmvsw driver

From: Aaron Young <[email protected]>

This series adds a new Logical Domains vSwitch (ldmvsw) driver.

The ldmvsw driver code will live in the drivers/net/ethernet/sun/ directory and will operate on Oracle systems running SPARC Linux in a Logical Domains environment (typically in the control domain).

The ldmvsw driver is very similar in function to the existing sunvnet driver. Ldmvsw creates a network interface for each "vsw-port" node found in the Machine Description (MD) of a service domain. These nodes correspond to ports on a vswitch created by the logical domains manager. The created network interface(s) can be used by bridge/vswitch software (such as the Linux bridge or Open vSwitch) to provide guest domain(s) with network interconnectivity or connectivity to a physical network.

Here is a example diagram of ldmvsw driver usage in a logical domain environment to provide a guest domain with network connectivity to a physical NIC on the service domain:

   +----------------+             +-----------------
   | Service Domain |             |  Guest domain  |
   |                |             |                |
   |  LinuxBridge   |             |                |
   |    |    |      |             |                |
   |   NIC Ldmvsw   |             |    Sunvnet     |
   +----------------+             +----------------+
        |    |           LDC              |
       LAN   ------------------------------

As stated, the sunvnet and ldmvsw drivers are _very_ similar in function. They both create network interface(s) to receive/transmit network traffic across LDC network channel(s). Since the driver is so similar in function to sunvnet, the approach will be as follows to integrate the driver and take advantage of common code:

Patch #1: Split sunvnet.c driver into sunvnet.c and sunvnet_common.c Patch #2: Modify the sunvnet_common code and data structures to be compatible
with both the sunvnet and ldmvsw drivers.
Patch #3: Add the new ldmvsw.c driver code
Patch #4: Checkpatch cleanup of the sunvnet/sunvnet_common code.

NOTE - Patch#1 renames a file (sunvnet.h -> sunvnet_common.h). When generating the patches (using git format-patch), I had to use the --no-renames option otherwise patch#1 would NOT apply using 'patch -p1' - which as I understand is a requirement for patch acceptance. I wasn't sure if this is proper thing to do. Please advise if not. Thanks.

v2 changes:
* change all EXPORT_SYMBOL declarations to EXPORT_SYMBOL_GPL
* remove inline attribute for external function port_is_up_common()
* Give all exported/global funcs in sunvnet_common.c a 'sunvnet_' prefix to avoid kernel global namespace pollution/collisions
* ldmvsw.c: Order local variable declarations from longest to shortest line
* ldmvsw.c: register the netdevice after all supporting state is ready/setup.
NOTE: The consensus at Oracle is that the following functions
must be done AFTER register_netdev() - this is the same ordering currently used in the sunvnet driver:
1. sunvnet_port_add_txq_common() - needs registered netdev
2. napi_enable() - requires registered netdev
3. vio_port_up() - as soon as this function is called LDC handshake messages will come in which must be handled by the napi code.

Aaron Young (4):
net-next: ldmvsw: Split sunvnet driver into common code
net-next: ldmvsw: Make sunvnet_common compatible with ldmvsw
net-next: ldmvsw: Add ldmvsw.c driver code
net-next: ldmvsw: Checkpatch sunvnet.c and sunvnet_common.c

 arch/sparc/configs/sparc64_defconfig      |    1 +
 drivers/net/ethernet/sun/Kconfig          |   16 +
 drivers/net/ethernet/sun/Makefile         |    2 +
 drivers/net/ethernet/sun/ldmvsw.c         |  468 ++++++++
 drivers/net/ethernet/sun/sunvnet.c        | 1753 +----------------------------
 drivers/net/ethernet/sun/sunvnet.h        |  114 --
 drivers/net/ethernet/sun/sunvnet_common.c | 1732 ++++++++++++++++++++++++++++
 drivers/net/ethernet/sun/sunvnet_common.h |  145 +++
 8 files changed, 2421 insertions(+), 1810 deletions(-)
 create mode 100644 drivers/net/ethernet/sun/ldmvsw.c
 delete mode 100644 drivers/net/ethernet/sun/sunvnet.h
 create mode 100644 drivers/net/ethernet/sun/sunvnet_common.c
 create mode 100644 drivers/net/ethernet/sun/sunvnet_common.h

From: Aaron Young <[email protected]>

Split sunvnet.c into sunvnet.c and sunvnet_common.c.

Details:

Since the sunvnet and ldmvsw drivers will both use common sunvnet code, move the functions (and support functions) anticipated to be common code from sunvnet.c to sunvnet_common.c. Similarly, sunvnet.h was renamed to sunvnet_common.h. The sunvnet_common.c code will be compiled into the kernel and act as a library of functions that are linked by either (or both) drivers when loaded.

Function names for external functions in sunvnet_common.c (to be called by both the sunvnet and ldmvsw drivers) were tagged with a "_common" suffix to clearly designate them as common functions.

No functional changes as of yet... just moved code verbatim to the new sunvnet_common.c/h files.

Makefile/Kconfig support added to build sunvnet_common.c file. The code is included in the kernel if SUN_LDOMS is defined/selected.

NOTE - per the SubmittingPatches documentation, since the code was just moved from one file another, the code was NOT checkpatch'd in this commit to aid in review.

---
 drivers/net/ethernet/sun/Kconfig          |    5 +
 drivers/net/ethernet/sun/Makefile         |    1 +
 drivers/net/ethernet/sun/sunvnet.c        | 1755 +----------------------------
 drivers/net/ethernet/sun/sunvnet.h        |  114 --
 drivers/net/ethernet/sun/sunvnet_common.c | 1753 ++++++++++++++++++++++++++++
 drivers/net/ethernet/sun/sunvnet_common.h |  133 +++
 6 files changed, 1917 insertions(+), 1844 deletions(-)
 delete mode 100644 drivers/net/ethernet/sun/sunvnet.h
 create mode 100644 drivers/net/ethernet/sun/sunvnet_common.c
 create mode 100644 drivers/net/ethernet/sun/sunvnet_common.h

index dee94b6..aa58c11 100644
@@ -69,6 +69,11 @@ config CASSINI
 	  Support for the Sun Cassini chip, aka Sun GigaSwift Ethernet. See also
 	  <http://docs.oracle.com/cd/E19113-01/giga.ether.pci/817-4341-10/817-4341-10.pdf>.
 
+config SUNVNET_COMMON
+	bool
+	depends on SUN_LDOMS
+	default y if SUN_LDOMS
+
 config SUNVNET
 	tristate "Sun Virtual Network support"
 	depends on SUN_LDOMS

index 1e620ff..7b622aa 100644
@@ -7,5 +7,6 @@ obj-$(CONFIG_SUNQE) += sunqe.o
 obj-$(CONFIG_SUNBMAC) += sunbmac.o
 obj-$(CONFIG_SUNGEM) += sungem.o
 obj-$(CONFIG_CASSINI) += cassini.o
+obj-$(CONFIG_SUNVNET_COMMON) += sunvnet_common.o
 obj-$(CONFIG_SUNVNET) += sunvnet.o
 obj-$(CONFIG_NIU) += niu.o

1.7.1

Part of the diff is not shown here. View the raw message.

From: Aaron Young <[email protected]>

Modify sunvnet common code and data structures to be compatible with both sunvnet and ldmvsw drivers.

Details:

Sunvnet operates on "vnet-port" nodes which appear in the Machine Description (MD) in a guest domain. Ldmvsw operates on "vsw-port" nodes which appear in the MD of a service domain.

A difference between the sunvnet driver and the ldmvsw driver is the sunvnet driver creates a network interface (i.e. a struct net_device) for every vnet-port *parent* "network" node. Several vnet-ports may appear under this common parent network node - each corresponding to a common parent network interface. Conversely, since bridge/vswitch software will need to interface with every vsw-port in a system, the ldmvsw driver creates a network interface (i.e. a struct net_device) for every vsw-port - not every parent node as with sunvnet. This difference required some special handling in the common code as explained below.

There are 2 key data structures used by the sunvnet and ldmvsw drivers (which are now found in sunvnet_common.h):

1. struct vnet_port
This structure represents a vnet-port node in sunvnet and a vsw-port in the ldmvsw driver.

2. struct vnet
This structure represents a parent "network" node in sunvnet and a parent "virtual-network-switch" node in ldmvsw.

Since the sunvnet driver allocates a net_device for every parent "network" node, a net_device member appears in the struct vnet. Since the ldmvsw driver allocates a net_device for every port, a net_device member was added to the vnet_port. The common code distinguishes which structure net_device member to use by checking a 'vsw' bit that was added to the vnet_port structure. See the VNET_PORT_TO_NET_DEVICE() marco in sunvnet_common.h.

The netdev_priv() in sunvnet is allocated as a vnet. The netdev_priv() in ldmvsw is a vnet_port. Therefore, any place in the common code where a netdev_priv() call was made, a wrapper function was implemented in each driver to first get the vnet and/or vnet_port (in a driver specific way) and pass them as newly added parameters to the common functions (see wrapper funcs: vnet_set_rx_mode() and vnet_poll_controller()). Since these wrapper functions call __tx_port_find(), __tx_port_find() was moved from the common code back into sunvnet.c. Note - ldmvsw.c does not require this function.

These changes also required that port_is_up() be made into a common function and thus it was given a _common suffix and exported like the other common functions.

A wrapper function was also added for vnet_start_xmit_common() to pass a driver-specific function arg to return the port associated with a given struct sk_buff and struct net_device. This was required because vnet_start_xmit_common() grabs a lock prior to getting the associated port. Using a function pointer arg allowed the code to work unchanged without risking changes to the non-trivial locking logic in vnet_start_xmit_common().

---
 drivers/net/ethernet/sun/sunvnet.c        |   74 ++++++++++++++++++++++-
 drivers/net/ethernet/sun/sunvnet_common.c |   89 ++++++++++-------------------
 drivers/net/ethernet/sun/sunvnet_common.h |   28 +++++++---
 3 files changed, 121 insertions(+), 70 deletions(-)

index 5b91135..98c5f16 100644
@@ -1,6 +1,7 @@
 /* sunvnet.c: Sun LDOM Virtual Network Driver.
  *
  * Copyright (C) 2007, 2008 David S. Miller <[email protected]>
+ * Copyright (C) 2016 Oracle. All rights reserved.
  */
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

@@ -84,18 +85,83 @@ static const struct ethtool_ops vnet_ethtool_ops = {
 static LIST_HEAD(vnet_list);
 static DEFINE_MUTEX(vnet_list_mutex);
 
+static struct vnet_port *__tx_port_find(struct vnet *vp, struct sk_buff *skb)
+{
+	unsigned int hash = vnet_hashfn(skb->data);
+	struct hlist_head *hp = &vp->port_hash[hash];
+	struct vnet_port *port;
+
+	hlist_for_each_entry_rcu(port, hp, hash) {
+		if (!sunvnet_port_is_up_common(port))
+			continue;
+		if (ether_addr_equal(port->raddr, skb->data))
+			return port;
+	}
+	list_for_each_entry_rcu(port, &vp->port_list, list) {
+		if (!port->switch_port)
+			continue;
+		if (!sunvnet_port_is_up_common(port))
+			continue;
+		return port;
+	}
+	return NULL;
+}
+
+/* func arg to vnet_start_xmit_common() to get the proper tx port */
+static struct vnet_port *vnet_tx_port_find(struct sk_buff *skb,
+					   struct net_device *dev)
+{
+	struct vnet *vp = netdev_priv(dev);
+
+	return __tx_port_find(vp, skb);
+}
+
+static u16 vnet_select_queue(struct net_device *dev, struct sk_buff *skb,
+			     void *accel_priv, select_queue_fallback_t fallback)
+{
+	struct vnet *vp = netdev_priv(dev);
+	struct vnet_port *port = __tx_port_find(vp, skb);
+
+	if (!port)
+		return 0;
+
+	return port->q_index;
+}
+
+/* Wrappers to common functions */
+static int vnet_start_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+	return sunvnet_start_xmit_common(skb, dev, vnet_tx_port_find);
+}
+
+static void vnet_set_rx_mode(struct net_device *dev)
+{
+	struct vnet *vp = netdev_priv(dev);
+
+	return sunvnet_set_rx_mode_common(dev, vp);
+}
+
+#ifdef CONFIG_NET_POLL_CONTROLLER
+static void vnet_poll_controller(struct net_device *dev)
+{
+	struct vnet *vp = netdev_priv(dev);
+
+	return sunvnet_poll_controller_common(dev, vp);
+}
+#endif
+
 static const struct net_device_ops vnet_ops = {
 	.ndo_open		= sunvnet_open_common,
 	.ndo_stop		= sunvnet_close_common,
-	.ndo_set_rx_mode	= sunvnet_set_rx_mode_common,
+	.ndo_set_rx_mode	= vnet_set_rx_mode,
 	.ndo_set_mac_address	= sunvnet_set_mac_addr_common,
 	.ndo_validate_addr	= eth_validate_addr,
 	.ndo_tx_timeout		= sunvnet_tx_timeout_common,
 	.ndo_change_mtu		= sunvnet_change_mtu_common,
-	.ndo_start_xmit		= sunvnet_start_xmit_common,
-	.ndo_select_queue	= sunvnet_select_queue_common,
+	.ndo_start_xmit		= vnet_start_xmit,
+	.ndo_select_queue	= vnet_select_queue,
 #ifdef CONFIG_NET_POLL_CONTROLLER
-	.ndo_poll_controller	= sunvnet_poll_controller_common,
+	.ndo_poll_controller	= vnet_poll_controller,
 #endif
 };
 

index c29c526..bd36528 100644
@@ -32,6 +32,13 @@ struct vnet_tx_entry {
 };
 
 struct vnet;
+
+/* Structure to describe a vnet-port or vsw-port in the MD.
+ * If the vsw bit is set, this structure represents a vswitch
+ * port, and the net_device can be found from ->dev. If the
+ * vsw bit is not set, the net_device is available from ->vp->dev.
+ * See the VNET_PORT_TO_NET_DEVICE macro below.
+ */
 struct vnet_port {
 	struct vio_driver_state	vio;
 

@@ -39,9 +46,11 @@ struct vnet_port {
 	u8			raddr[ETH_ALEN];
 	unsigned		switch_port:1;
 	unsigned		tso:1;
-	unsigned		__pad:14;
+	unsigned		vsw:1;
+	unsigned		__pad:13;
 
 	struct vnet		*vp;
+	struct net_device	*dev;
 
 	struct vnet_tx_entry	tx_bufs[VNET_TX_RING_SIZE];
 

@@ -105,21 +114,23 @@ struct vnet {
 	int			nports;
 };
 
+/* Def used by common code to get the net_device from the proper location */
+#define VNET_PORT_TO_NET_DEVICE(__port) \
+	((__port)->vsw ? (__port)->dev : (__port)->vp->dev)
+
 /* Common funcs */
 void sunvnet_clean_timer_expire_common(unsigned long port0);
 int sunvnet_open_common(struct net_device *dev);
 int sunvnet_close_common(struct net_device *dev);
-void sunvnet_set_rx_mode_common(struct net_device *dev);
+void sunvnet_set_rx_mode_common(struct net_device *dev, struct vnet *vp);
 int sunvnet_set_mac_addr_common(struct net_device *dev, void *p);
 void sunvnet_tx_timeout_common(struct net_device *dev);
 int sunvnet_change_mtu_common(struct net_device *dev, int new_mtu);
-int sunvnet_start_xmit_common(struct sk_buff *skb, struct net_device *dev);
-u16 sunvnet_select_queue_common(struct net_device *dev,
-				struct sk_buff *skb,
-				void *accel_priv,
-				select_queue_fallback_t fallback);
+int sunvnet_start_xmit_common(struct sk_buff *skb, struct net_device *dev,
+			   struct vnet_port *(*vnet_tx_port)
+			   (struct sk_buff *, struct net_device *));
 #ifdef CONFIG_NET_POLL_CONTROLLER
-void sunvnet_poll_controller_common(struct net_device *dev);
+void sunvnet_poll_controller_common(struct net_device *dev, struct vnet *vp);
 #endif
 void sunvnet_event_common(void *arg, int event);
 int sunvnet_send_attr_common(struct vio_driver_state *vio);

@@ -127,6 +138,7 @@ int sunvnet_handle_attr_common(struct vio_driver_state *vio, void *arg);
 void sunvnet_handshake_complete_common(struct vio_driver_state *vio);
 int sunvnet_poll_common(struct napi_struct *napi, int budget);
 void sunvnet_port_free_tx_bufs_common(struct vnet_port *port);
+bool sunvnet_port_is_up_common(struct vnet_port *vnet);
 void sunvnet_port_add_txq_common(struct vnet_port *port);
 void sunvnet_port_rm_txq_common(struct vnet_port *port);
 

1.7.1

From: Aaron Young <[email protected]>

Add ldmvsw.c driver

Details:

The ldmvsw driver very closely follows the sunvnet.c code and makes use of the sunvnet_common.c code for core functionality.

A significant difference between sunvnet and ldmvsw driver is sunvnet creates a network interface for each vnet-port *parent* node in the MD while the ldmvsw driver creates a network interface for every vsw-port node in the Machine Description (MD). Therefore the netdev_priv() for sunvnet is a vnet structure while the netdev_priv() for ldmvsw is a vnet_port structure.

Vnet_port structures allocated by ldmvsw have the vsw bit set. When finding the net_device associated with a port, the common code keys off this bit to use either the net_device found in the vnet_port or the net_device in the vnet structure (see the VNET_PORT_TO_NET_DEVICE() macro in sunvnet_common.h). This scheme allows the common code to work with both drivers with minimal changes.

Similar to Xen, network interfaces created by the ldmvsw driver will always have a HW Addr (i.e. mac address) of FE:FF:FF:FF:FF:FF and each will be assigned the devname "vif<cfg_handle>.<port_id>" - where <cfg_handle> and <port_id> are a unique handle/port pair assigned to the associated vsw-port node in the MD.

---
 arch/sparc/configs/sparc64_defconfig |    1 +
 drivers/net/ethernet/sun/Kconfig     |   11 +
 drivers/net/ethernet/sun/Makefile    |    1 +
 drivers/net/ethernet/sun/ldmvsw.c    |  468 ++++++++++++++++++++++++++++++++++
 4 files changed, 481 insertions(+), 0 deletions(-)
 create mode 100644 drivers/net/ethernet/sun/ldmvsw.c

index 6b68f12..04920ab 100644
@@ -102,6 +102,7 @@ CONFIG_SUNLANCE=m
 CONFIG_HAPPYMEAL=m
 CONFIG_SUNGEM=m
 CONFIG_SUNVNET=m
+CONFIG_LDMVSW=m
 CONFIG_NET_PCI=y
 CONFIG_E1000=m
 CONFIG_E1000E=m

index aa58c11..a4b40e3 100644
@@ -80,6 +80,17 @@ config SUNVNET
 	---help---
 	  Support for virtual network devices under Sun Logical Domains.
 
+config LDMVSW
+	tristate "Sun4v LDoms Virtual Switch support"
+	depends on SUN_LDOMS
+	---help---
+	  Support for virtual switch devices under Sun4v Logical Domains.
+	  This driver adds a network interface for every vsw-port node
+	  found in the machine description of a service domain.
+	  Linux bridge/switch software can use these interfaces for
+	  guest domain network interconnectivity or guest domain
+	  connection to a physical network on a service domain.
+
 config NIU
 	tristate "Sun Neptune 10Gbit Ethernet support"
 	depends on PCI

index 7b622aa..3785543 100644
@@ -9,4 +9,5 @@ obj-$(CONFIG_SUNGEM) += sungem.o
 obj-$(CONFIG_CASSINI) += cassini.o
 obj-$(CONFIG_SUNVNET_COMMON) += sunvnet_common.o
 obj-$(CONFIG_SUNVNET) += sunvnet.o
+obj-$(CONFIG_LDMVSW) += ldmvsw.o
 obj-$(CONFIG_NIU) += niu.o

1.7.1

From: Aaron Young <[email protected]>

Checkpatch updates for sunvnet.c and sunvnet_common.c.

---
 drivers/net/ethernet/sun/sunvnet.c        |    6 +-
 drivers/net/ethernet/sun/sunvnet_common.c |  100 +++++++++++++++--------------
 2 files changed, 56 insertions(+), 50 deletions(-)

index 98c5f16..a2f9b47 100644
@@ -66,12 +66,14 @@ static void vnet_get_drvinfo(struct net_device *dev,
 static u32 vnet_get_msglevel(struct net_device *dev)
 {
 	struct vnet *vp = netdev_priv(dev);
+
 	return vp->msg_enable;
 }
 
 static void vnet_set_msglevel(struct net_device *dev, u32 value)
 {
 	struct vnet *vp = netdev_priv(dev);
+
 	vp->msg_enable = value;
 }
 

@@ -359,7 +361,7 @@ static int vnet_port_probe(struct vio_dev *vdev, const struct vio_device_id *id)
 	INIT_LIST_HEAD(&port->list);
 
 	switch_port = 0;
-	if (mdesc_get_property(hp, vdev->mp, "switch-port", NULL) != NULL)
+	if (mdesc_get_property(hp, vdev->mp, "switch-port", NULL))
 		switch_port = 1;
 	port->switch_port = switch_port;
 	port->tso = true;

@@ -403,7 +405,6 @@ static int vnet_port_remove(struct vio_dev *vdev)
 	struct vnet_port *port = dev_get_drvdata(&vdev->dev);
 
 	if (port) {
-
 		del_timer_sync(&port->vio.timer);
 
 		napi_disable(&port->napi);

@@ -421,7 +422,6 @@ static int vnet_port_remove(struct vio_dev *vdev)
 		dev_set_drvdata(&vdev->dev, NULL);
 
 		kfree(port);
-
 	}
 	return 0;
 }

1.7.1

From: Aaron Young <[email protected]>
Date: Tue, 15 Mar 2016 11:35:36 -0700

Series applied, thanks Aaron.