Skip to contentSkip to conversation

Series short description

RFC PATCH In review John Fastabend 5 participants 35 messages Sep 11, 2013 – Oct 14, 2013

Subsystems (12)
lore
Share & tools
lore
#0 lore raw

cover Series short description

This patch series implements virtual station interfaces or VSIs. The VSI term comes from the IEEE std 802.1Qbg-2012 specification which should be merged with 802.1Q proper at some point.

3.18 Virtual Station Interface (VSI): An interface to a
virtual station that is attached to a DRP of an edge edge relay.

A DRP (downlink relay port) is the link between an edge relay and a VSI. An edge relay is basically a simple bridge that does not need to support learning, flooding, xSTP, etc. Which matches up well with the ixgbe and I believe other hardware embedded bridge (ebridge) implementations.

This series adds a new VSI rtnl link type. I chose to do this via a link type because it allows us to reuse a lot of the existing infrastructure to bring up a net device and it lets a VSI look similar to a macvlan. The macvlan link type being the software equivalent of a VSI. In many cases I can simply replace the macvlan type with vsi from the ip command line tool and my existing scripts work but use the ebridge instead of SW.

The usage model looks like this,

# ip link add link p3p2 numtxqueues 2 numrxqueues 2 type vsi
# ip link add link p3p2 numtxqueues 2 numrxqueues 2 type vsi
# ip link add link p3p2 numtxqueues 4 numrxqueues 4 type vsi
# ip link set dev vsi0 addr 00:1b:21:69:9f:15
# ip link set dev vsi1 addr 00:1b:21:69:9f:16
# ip link set dev vsi2 addr 00:1b:21:69:9f:17
# ip link set dev vsi0 up
# ip link set dev vsi1 up
# ip link set dev vsi2 up
# ip link show
16: p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 00:1b:21:69:9f:09 brd ff:ff:ff:ff:ff:ff
17: vsi0@p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 00:1b:21:69:9f:15 brd ff:ff:ff:ff:ff:ff
18: vsi1@p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 00:1b:21:69:9f:16 brd ff:ff:ff:ff:ff:ff
19: vsi2@p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 00:1b:21:69:9f:17 brd ff:ff:ff:ff:ff:ff

And creates this topology,

      vsi0      vsi1       vsi2
       |         |          |
     +------------------------+
     |                        |
     |        ebridge         |
     |                        |
     +------------------------+
                 |
               p3p2

At this point each vsi# will receive their assigned MAC addresses. Using the 'fdb' interfaces additional L2/L3/tunnel entries could be added to the vsi#. And VSIs can be assigned to net name spaces.

The topology of the ebridge is tracked via the upper and lower dev lists. After we get Veaceslav Falico's work to expose this via sysfs then the topology will be visible. Although it can be learned also via iflink:ifindex to some extent.

PATCH DESCRIPTION:

The first extend rtnl link ops priv_size routine so VSI link types can set the private space for the netdev being created. This is required because device drivers will use this space.

The second patch adds some helper routines to traverse the lower dev list.

The third patch is the interface work to support VSI devices.

And the last patch is an implementation for ixgbe. Notice there are still a few items I need to clean up on this patch before submitting but it is working, without DCB/FCoE, now. And I think I can simplify it some to bring down the line count.

My plan would be to submit this as a real patch after net-next opens but I wanted to see if there was any initial feedback especially related to the VSI link type.

phew... sorry that was a bit long winded.

---

John Fastabend (4):
net: rtnetlink: make priv_size a function for devs with dynamic size
net: Add lower dev list helpers
net: VSI: Add virtual station interface support
ixgbe: Adding VSI support to ixgbe

 drivers/infiniband/ulp/ipoib/ipoib_netlink.c     |    4 
 drivers/net/Kconfig                              |    9 
 drivers/net/Makefile                             |    1 
 drivers/net/bonding/bond_main.c                  |    4 
 drivers/net/caif/caif_hsi.c                      |    4 
 drivers/net/ethernet/intel/ixgbe/Makefile        |    3 
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |   32 ++
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    4 
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |   15 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  307 +++++++++++-----
 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c     |  428 ++++++++++++++++++++++
 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h     |   71 ++++
 drivers/net/ifb.c                                |    4 
 drivers/net/macvlan.c                            |    4 
 drivers/net/nlmon.c                              |    4 
 drivers/net/team/team.c                          |    4 
 drivers/net/tun.c                                |    4 
 drivers/net/veth.c                               |    4 
 drivers/net/vsi.c                                |  124 ++++++
 drivers/net/vxlan.c                              |    4 
 include/linux/netdevice.h                        |   35 ++
 include/net/rtnetlink.h                          |   11 +
 include/uapi/linux/if.h                          |    1 
 net/8021q/vlan_netlink.c                         |    4 
 net/bridge/br_netlink.c                          |    4 
 net/core/dev.c                                   |   25 +
 net/core/rtnetlink.c                             |    2 
 net/ieee802154/6lowpan.c                         |    4 
 net/ipv4/ip_gre.c                                |    6 
 net/ipv4/ip_tunnel.c                             |    2 
 net/ipv4/ip_vti.c                                |    4 
 net/ipv4/ipip.c                                  |    4 
 net/ipv6/ip6_tunnel.c                            |    4 
 net/ipv6/sit.c                                   |    4 
 34 files changed, 1028 insertions(+), 116 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c
 create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h
 create mode 100644 drivers/net/vsi.c
Signature · 1 line
Signature
#1 lore raw

RFC PATCH 1/4 net: rtnetlink: make priv_size a function for devs with dynamic size

The priv_size rtnl_link_op today is a fixed size_t value.

For upcoming VSI support via rtnl_link_ops it is useful to allow this size to be configurable. This patch converts the existing static definition into a function that returns the size_t value.

To make this conversion as easy as possible the patch uses a new macro RTNL_LINK_OPS_PRIV_SIZE which existing users can call to generate a function equivalent to the previous static value.

RFC NOTE: I'm not entirely sure the macro makes the code easier
to read or just obfuscates what is really happening. Any opinions?

---
 drivers/infiniband/ulp/ipoib/ipoib_netlink.c |    4 +++-
 drivers/net/bonding/bond_main.c              |    4 +++-
 drivers/net/caif/caif_hsi.c                  |    4 +++-
 drivers/net/ifb.c                            |    4 +++-
 drivers/net/macvlan.c                        |    4 +++-
 drivers/net/nlmon.c                          |    4 +++-
 drivers/net/team/team.c                      |    4 +++-
 drivers/net/tun.c                            |    4 +++-
 drivers/net/veth.c                           |    4 +++-
 drivers/net/vxlan.c                          |    4 +++-
 include/net/rtnetlink.h                      |   11 ++++++++++-
 net/8021q/vlan_netlink.c                     |    4 +++-
 net/bridge/br_netlink.c                      |    4 +++-
 net/core/rtnetlink.c                         |    2 +-
 net/ieee802154/6lowpan.c                     |    4 +++-
 net/ipv4/ip_gre.c                            |    6 ++++--
 net/ipv4/ip_tunnel.c                         |    2 +-
 net/ipv4/ip_vti.c                            |    4 +++-
 net/ipv4/ipip.c                              |    4 +++-
 net/ipv6/ip6_tunnel.c                        |    4 +++-
 net/ipv6/sit.c                               |    4 +++-
 21 files changed, 67 insertions(+), 22 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_netlink.c +3 -1
index f81abe1..8fd93fc 100644
@@ -155,11 +155,13 @@ static size_t ipoib_get_size(const struct net_device *dev)
 		nla_total_size(2);	/* IFLA_IPOIB_UMCAST */
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(ipoib, ipoib_dev_priv);
+
 static struct rtnl_link_ops ipoib_link_ops __read_mostly = {
 	.kind		= "ipoib",
 	.maxtype	= IFLA_IPOIB_MAX,
 	.policy		= ipoib_policy,
-	.priv_size	= sizeof(struct ipoib_dev_priv),
+	.priv_size	= ipoib_priv_size,
 	.setup		= ipoib_setup,
 	.newlink	= ipoib_new_child_link,
 	.changelink	= ipoib_changelink,
drivers/net/bonding/bond_main.c +3 -1
index 39e5b1c..1b3caf4 100644
@@ -4503,9 +4503,11 @@ static unsigned int bond_get_num_tx_queues(void)
 	return tx_queues;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(bond, bonding);
+
 static struct rtnl_link_ops bond_link_ops __read_mostly = {
 	.kind			= "bond",
-	.priv_size		= sizeof(struct bonding),
+	.priv_size		= bond_priv_size,
 	.setup			= bond_setup,
 	.validate		= bond_validate,
 	.get_num_tx_queues	= bond_get_num_tx_queues,
drivers/net/caif/caif_hsi.c +3 -1
index 5e40a8b..bf4f19a 100644
@@ -1445,9 +1445,11 @@ err:
 	return -ENODEV;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(caif, cfhsi);
+
 static struct rtnl_link_ops caif_hsi_link_ops __read_mostly = {
 	.kind		= "cfhsi",
-	.priv_size	= sizeof(struct cfhsi),
+	.priv_size	= caif_priv_size,
 	.setup		= cfhsi_setup,
 	.maxtype	= __IFLA_CAIF_HSI_MAX,
 	.policy	= caif_hsi_policy,
drivers/net/ifb.c +3 -1
index a3bed28..ed8f5fa 100644
@@ -251,9 +251,11 @@ static int ifb_validate(struct nlattr *tb[], struct nlattr *data[])
 	return 0;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(ifb, ifb_private);
+
 static struct rtnl_link_ops ifb_link_ops __read_mostly = {
 	.kind		= "ifb",
-	.priv_size	= sizeof(struct ifb_private),
+	.priv_size	= ifb_priv_size,
 	.setup		= ifb_setup,
 	.validate	= ifb_validate,
 };
drivers/net/macvlan.c +3 -1
index 64dfaa3..6f293f7 100644
@@ -934,10 +934,12 @@ static const struct nla_policy macvlan_policy[IFLA_MACVLAN_MAX + 1] = {
 	[IFLA_MACVLAN_FLAGS] = { .type = NLA_U16 },
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(macvlan, macvlan_dev);
+
 int macvlan_link_register(struct rtnl_link_ops *ops)
 {
 	/* common fields */
-	ops->priv_size		= sizeof(struct macvlan_dev);
+	ops->priv_size		= macvlan_priv_size,
 	ops->validate		= macvlan_validate;
 	ops->maxtype		= IFLA_MACVLAN_MAX;
 	ops->policy		= macvlan_policy;
drivers/net/nlmon.c +3 -1
index b57ce5f..693f357 100644
@@ -154,9 +154,11 @@ static int nlmon_validate(struct nlattr *tb[], struct nlattr *data[])
 	return 0;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(nlmon, nlmon);
+
 static struct rtnl_link_ops nlmon_link_ops __read_mostly = {
 	.kind			= "nlmon",
-	.priv_size		= sizeof(struct nlmon),
+	.priv_size		= nlmon_priv_size,
 	.setup			= nlmon_setup,
 	.validate		= nlmon_validate,
 };
drivers/net/team/team.c +3 -1
index 50e43e6..b5f0526 100644
@@ -2068,9 +2068,11 @@ static unsigned int team_get_num_rx_queues(void)
 	return TEAM_DEFAULT_NUM_RX_QUEUES;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(team, team);
+
 static struct rtnl_link_ops team_link_ops __read_mostly = {
 	.kind			= DRV_NAME,
-	.priv_size		= sizeof(struct team),
+	.priv_size		= team_priv_size,
 	.setup			= team_setup,
 	.newlink		= team_newlink,
 	.validate		= team_validate,
drivers/net/tun.c +3 -1
index a639de8..d5aebec 100644
@@ -1380,9 +1380,11 @@ static int tun_validate(struct nlattr *tb[], struct nlattr *data[])
 	return -EINVAL;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(tun, tun_struct);
+
 static struct rtnl_link_ops tun_link_ops __read_mostly = {
 	.kind		= DRV_NAME,
-	.priv_size	= sizeof(struct tun_struct),
+	.priv_size	= tun_priv_size,
 	.setup		= tun_setup,
 	.validate	= tun_validate,
 };
drivers/net/veth.c +3 -1
index eee1f19..61f7122 100644
@@ -434,9 +434,11 @@ static const struct nla_policy veth_policy[VETH_INFO_MAX + 1] = {
 	[VETH_INFO_PEER]	= { .len = sizeof(struct ifinfomsg) },
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(veth, veth_priv);
+
 static struct rtnl_link_ops veth_link_ops = {
 	.kind		= DRV_NAME,
-	.priv_size	= sizeof(struct veth_priv),
+	.priv_size	= veth_priv_size,
 	.setup		= veth_setup,
 	.validate	= veth_validate,
 	.newlink	= veth_newlink,
drivers/net/vxlan.c +3 -1
index bf64b41..cff4c45 100644
@@ -2616,11 +2616,13 @@ nla_put_failure:
 	return -EMSGSIZE;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(vxlan, vxlan_dev);
+
 static struct rtnl_link_ops vxlan_link_ops __read_mostly = {
 	.kind		= "vxlan",
 	.maxtype	= IFLA_VXLAN_MAX,
 	.policy		= vxlan_policy,
-	.priv_size	= sizeof(struct vxlan_dev),
+	.priv_size	= vxlan_priv_size,
 	.setup		= vxlan_setup,
 	.validate	= vxlan_validate,
 	.newlink	= vxlan_newlink,
include/net/rtnetlink.h +10 -1
index 7026648..b6a2ffb 100644
@@ -4,6 +4,14 @@
 #include <linux/rtnetlink.h>
 #include <net/netlink.h>
 
+/* generate a get_priv function for simple field */
+#define RTNL_LINK_OPS_PRIV_SIZE(device, size_struct)		\
+static size_t device##_priv_size(struct net *src_net,		\
+				  struct nlattr *tb[])		\
+{								\
+	return sizeof(struct size_struct);			\
+}
+
 typedef int (*rtnl_doit_func)(struct sk_buff *, struct nlmsghdr *);
 typedef int (*rtnl_dumpit_func)(struct sk_buff *, struct netlink_callback *);
 typedef u16 (*rtnl_calcit_func)(struct sk_buff *, struct nlmsghdr *);
@@ -54,7 +62,8 @@ struct rtnl_link_ops {
 
 	const char		*kind;
 
-	size_t			priv_size;
+	size_t			(*priv_size)(struct net *src_net,
+					     struct nlattr *tb[]);
 	void			(*setup)(struct net_device *dev);
 
 	int			maxtype;
net/8021q/vlan_netlink.c +3 -1
index 3091297..6052370 100644
@@ -238,11 +238,13 @@ nla_put_failure:
 	return -EMSGSIZE;
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(vlan, vlan_dev_priv);
+
 struct rtnl_link_ops vlan_link_ops __read_mostly = {
 	.kind		= "vlan",
 	.maxtype	= IFLA_VLAN_MAX,
 	.policy		= vlan_policy,
-	.priv_size	= sizeof(struct vlan_dev_priv),
+	.priv_size	= vlan_priv_size,
 	.setup		= vlan_setup,
 	.validate	= vlan_validate,
 	.newlink	= vlan_newlink,
net/bridge/br_netlink.c +3 -1
index b9259ef..3f4a792 100644
@@ -469,9 +469,11 @@ static struct rtnl_af_ops br_af_ops = {
 	.get_link_af_size	= br_get_link_af_size,
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(br, net_bridge);
+
 struct rtnl_link_ops br_link_ops __read_mostly = {
 	.kind		= "bridge",
-	.priv_size	= sizeof(struct net_bridge),
+	.priv_size	= br_priv_size,
 	.setup		= br_dev_setup,
 	.validate	= br_validate,
 	.dellink	= br_dev_delete,
net/core/rtnetlink.c +1 -1
index 2a0e21d..76320fb 100644
@@ -1673,7 +1673,7 @@ struct net_device *rtnl_create_link(struct net *net,
 		num_rx_queues = ops->get_num_rx_queues();
 
 	err = -ENOMEM;
-	dev = alloc_netdev_mqs(ops->priv_size, ifname, ops->setup,
+	dev = alloc_netdev_mqs(ops->priv_size(net, tb), ifname, ops->setup,
 			       num_tx_queues, num_rx_queues);
 	if (!dev)
 		goto err;
net/ieee802154/6lowpan.c +3 -1
index c85e71e..8ee9235 100644
@@ -1420,9 +1420,11 @@ static void lowpan_dellink(struct net_device *dev, struct list_head *head)
 	dev_put(real_dev);
 }
 
+RTNL_LINK_OPS_PRIV_SIZE(lowpan, lowpan_dev_info);
+
 static struct rtnl_link_ops lowpan_link_ops __read_mostly = {
 	.kind		= "lowpan",
-	.priv_size	= sizeof(struct lowpan_dev_info),
+	.priv_size	= lowpan_priv_size,
 	.setup		= lowpan_setup,
 	.newlink	= lowpan_newlink,
 	.dellink	= lowpan_dellink,
net/ipv4/ip_gre.c +4 -2
index d7aea4c..c3cfb7b 100644
@@ -731,11 +731,13 @@ static const struct nla_policy ipgre_policy[IFLA_GRE_MAX + 1] = {
 	[IFLA_GRE_PMTUDISC]	= { .type = NLA_U8 },
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(ipgre, ip_tunnel);
+
 static struct rtnl_link_ops ipgre_link_ops __read_mostly = {
 	.kind		= "gre",
 	.maxtype	= IFLA_GRE_MAX,
 	.policy		= ipgre_policy,
-	.priv_size	= sizeof(struct ip_tunnel),
+	.priv_size	= ipgre_priv_size,
 	.setup		= ipgre_tunnel_setup,
 	.validate	= ipgre_tunnel_validate,
 	.newlink	= ipgre_newlink,
@@ -749,7 +751,7 @@ static struct rtnl_link_ops ipgre_tap_ops __read_mostly = {
 	.kind		= "gretap",
 	.maxtype	= IFLA_GRE_MAX,
 	.policy		= ipgre_policy,
-	.priv_size	= sizeof(struct ip_tunnel),
+	.priv_size	= ipgre_priv_size,
 	.setup		= ipgre_tap_setup,
 	.validate	= ipgre_tap_validate,
 	.newlink	= ipgre_newlink,
net/ipv4/ip_tunnel.c +1 -1
index ac9fabe..562dd12 100644
@@ -293,7 +293,7 @@ static struct net_device *__ip_tunnel_create(struct net *net,
 	}
 
 	ASSERT_RTNL();
-	dev = alloc_netdev(ops->priv_size, name, ops->setup);
+	dev = alloc_netdev(ops->priv_size(net, NULL), name, ops->setup);
 	if (!dev) {
 		err = -ENOMEM;
 		goto failed;
net/ipv4/ip_vti.c +3 -1
index e805e7b..545cc20 100644
@@ -416,11 +416,13 @@ static const struct nla_policy vti_policy[IFLA_VTI_MAX + 1] = {
 	[IFLA_VTI_REMOTE]	= { .len = FIELD_SIZEOF(struct iphdr, daddr) },
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(vti, ip_tunnel);
+
 static struct rtnl_link_ops vti_link_ops __read_mostly = {
 	.kind		= "vti",
 	.maxtype	= IFLA_VTI_MAX,
 	.policy		= vti_policy,
-	.priv_size	= sizeof(struct ip_tunnel),
+	.priv_size	= vti_priv_size,
 	.setup		= vti_tunnel_setup,
 	.validate	= vti_tunnel_validate,
 	.newlink	= vti_newlink,
net/ipv4/ipip.c +3 -1
index 7f80fb4..ee5a926 100644
@@ -408,11 +408,13 @@ static const struct nla_policy ipip_policy[IFLA_IPTUN_MAX + 1] = {
 	[IFLA_IPTUN_PMTUDISC]		= { .type = NLA_U8 },
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(ipip, ip_tunnel);
+
 static struct rtnl_link_ops ipip_link_ops __read_mostly = {
 	.kind		= "ipip",
 	.maxtype	= IFLA_IPTUN_MAX,
 	.policy		= ipip_policy,
-	.priv_size	= sizeof(struct ip_tunnel),
+	.priv_size	= ipip_priv_size,
 	.setup		= ipip_tunnel_setup,
 	.newlink	= ipip_newlink,
 	.changelink	= ipip_changelink,
net/ipv6/ip6_tunnel.c +3 -1
index 61355f7..2ab41b9 100644
@@ -1682,11 +1682,13 @@ static const struct nla_policy ip6_tnl_policy[IFLA_IPTUN_MAX + 1] = {
 	[IFLA_IPTUN_PROTO]		= { .type = NLA_U8 },
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(ip6_tnl, ip6_tnl);
+
 static struct rtnl_link_ops ip6_link_ops __read_mostly = {
 	.kind		= "ip6tnl",
 	.maxtype	= IFLA_IPTUN_MAX,
 	.policy		= ip6_tnl_policy,
-	.priv_size	= sizeof(struct ip6_tnl),
+	.priv_size	= ip6_tnl_priv_size,
 	.setup		= ip6_tnl_dev_setup,
 	.validate	= ip6_tnl_validate,
 	.newlink	= ip6_tnl_newlink,
net/ipv6/sit.c +3 -1
index 7ee5cb9..bd638ba 100644
@@ -1540,11 +1540,13 @@ static const struct nla_policy ipip6_policy[IFLA_IPTUN_MAX + 1] = {
 #endif
 };
 
+RTNL_LINK_OPS_PRIV_SIZE(ipip6, ip_tunnel);
+
 static struct rtnl_link_ops sit_link_ops __read_mostly = {
 	.kind		= "sit",
 	.maxtype	= IFLA_IPTUN_MAX,
 	.policy		= ipip6_policy,
-	.priv_size	= sizeof(struct ip_tunnel),
+	.priv_size	= ipip6_priv_size,
 	.setup		= ipip6_tunnel_setup,
 	.validate	= ipip6_validate,
 	.newlink	= ipip6_newlink,
#2 lore raw

RFC PATCH 2/4 net: Add lower dev list helpers

In reply to John Fastabend

This patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.

VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.

---
 include/linux/netdevice.h |    8 ++++++++
 net/core/dev.c            |   25 +++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
include/linux/netdevice.h +8 -0
index 041b42a..4d24b38 100644
@@ -2813,6 +2813,8 @@ extern int		bpf_jit_enable;
 extern bool netdev_has_upper_dev(struct net_device *dev,
 				 struct net_device *upper_dev);
 extern bool netdev_has_any_upper_dev(struct net_device *dev);
+extern struct net_device *netdev_lower_get_next_dev_rcu(struct net_device *dev,
+							struct list_head **iter);
 extern struct net_device *netdev_upper_get_next_dev_rcu(struct net_device *dev,
 							struct list_head **iter);
 
@@ -2823,6 +2825,12 @@ extern struct net_device *netdev_upper_get_next_dev_rcu(struct net_device *dev,
 	     upper; \
 	     upper = netdev_upper_get_next_dev_rcu(dev, &(iter)))
 
+#define netdev_for_each_lower_dev_rcu(dev, lower, iter) \
+	for (iter = &(dev)->lower_dev_list, \
+	     lower = netdev_lower_get_next_dev_rcu(dev, &(iter)); \
+	     lower; \
+	     lower = netdev_lower_get_next_dev_rcu(dev, &(iter)))
+
 extern struct net_device *netdev_master_upper_dev_get(struct net_device *dev);
 extern struct net_device *netdev_master_upper_dev_get_rcu(struct net_device *dev);
 extern int netdev_upper_dev_link(struct net_device *dev,
net/core/dev.c +25 -0
index 5c713f2..65ed610 100644
@@ -4468,6 +4468,31 @@ struct net_device *netdev_master_upper_dev_get(struct net_device *dev)
 }
 EXPORT_SYMBOL(netdev_master_upper_dev_get);
 
+/* netdev_lower_get_next_dev_rcu - Get the next dev from lower list
+ * @dev: device
+ * @iter: list_head ** of the current position
+ *
+ * Gets the next device from the dev's lower list, starting from iter
+ * position. The caller must hold RTNL/RCU read lock.
+ */
+struct net_device *netdev_lower_get_next_dev_rcu(struct net_device *dev,
+						 struct list_head **iter)
+{
+	struct netdev_adjacent *lower;
+
+	WARN_ON_ONCE(!rcu_read_lock_held());
+
+	lower = list_entry_rcu((*iter)->next, struct netdev_adjacent, list);
+
+	if (&lower->list == &dev->lower_dev_list)
+		return NULL;
+
+	*iter = &lower->list;
+
+	return lower->dev;
+}
+EXPORT_SYMBOL(netdev_lower_get_next_dev_rcu);
+
 /* netdev_upper_get_next_dev_rcu - Get the next dev from upper list
  * @dev: device
  * @iter: list_head ** of the current position
#3 lore raw

Re: [RFC PATCH 2/4] net: Add lower dev list helpers

Quoted 5 lines · John Fastabend

This patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.

VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.

Just as a note (as I have quite no idea how ixgbe works) - you are aware that the upper/lower lists currently include *all* upper/lower devices, and not only the first-connected ones?

I've seen that you usually verify it, however not always, just a heads-up - sorry if misread.

I'm also currently trying to get the new patchset included - which would split the first-tier connected devices from all the 'other' (as in - lower of a lower and upper of an upper) devices, so that way, I think, it would be easier/faster for you to use it. It also has a ->private field, easily accessible, which you could have used instead of/in conjunction with/for struct ixgbe_vsi_adapter.

[PATCH v2 net-next 00/27] bonding: use neighbours instead of own lists

Here's the patchset.

Hope that helps.

Quoted 70 lines

---
include/linux/netdevice.h | 8 ++++++++
net/core/dev.c | 25 +++++++++++++++++++++++++

2 files changed, 33 insertions(+)
include/linux/netdevice.h +8 -0
index 041b42a..4d24b38 100644
@@ -2813,6 +2813,8 @@ extern int		bpf_jit_enable;
 extern bool netdev_has_upper_dev(struct net_device *dev,
 				 struct net_device *upper_dev);
 extern bool netdev_has_any_upper_dev(struct net_device *dev);
+extern struct net_device *netdev_lower_get_next_dev_rcu(struct net_device *dev,
+							struct list_head **iter);
 extern struct net_device *netdev_upper_get_next_dev_rcu(struct net_device *dev,
 							struct list_head **iter);
 
@@ -2823,6 +2825,12 @@ extern struct net_device *netdev_upper_get_next_dev_rcu(struct net_device *dev,
 	     upper; \
 	     upper = netdev_upper_get_next_dev_rcu(dev, &(iter)))
 
+#define netdev_for_each_lower_dev_rcu(dev, lower, iter) \
+	for (iter = &(dev)->lower_dev_list, \
+	     lower = netdev_lower_get_next_dev_rcu(dev, &(iter)); \
+	     lower; \
+	     lower = netdev_lower_get_next_dev_rcu(dev, &(iter)))
+
 extern struct net_device *netdev_master_upper_dev_get(struct net_device *dev);
 extern struct net_device *netdev_master_upper_dev_get_rcu(struct net_device *dev);
 extern int netdev_upper_dev_link(struct net_device *dev,
net/core/dev.c +25 -0
index 5c713f2..65ed610 100644
@@ -4468,6 +4468,31 @@ struct net_device *netdev_master_upper_dev_get(struct net_device *dev)
 }
 EXPORT_SYMBOL(netdev_master_upper_dev_get);
 
+/* netdev_lower_get_next_dev_rcu - Get the next dev from lower list
+ * @dev: device
+ * @iter: list_head ** of the current position
+ *
+ * Gets the next device from the dev's lower list, starting from iter
+ * position. The caller must hold RTNL/RCU read lock.
+ */
+struct net_device *netdev_lower_get_next_dev_rcu(struct net_device *dev,
+						 struct list_head **iter)
+{
+	struct netdev_adjacent *lower;
+
+	WARN_ON_ONCE(!rcu_read_lock_held());
+
+	lower = list_entry_rcu((*iter)->next, struct netdev_adjacent, list);
+
+	if (&lower->list == &dev->lower_dev_list)
+		return NULL;
+
+	*iter = &lower->list;
+
+	return lower->dev;
+}
+EXPORT_SYMBOL(netdev_lower_get_next_dev_rcu);
+
 /* netdev_upper_get_next_dev_rcu - Get the next dev from upper list
 * @dev: device
 * @iter: list_head ** of the current position
#4 lore raw

Re: [RFC PATCH 2/4] net: Add lower dev list helpers

Quoted 10 lines · Veaceslav Falico
Quoted 5 lines · John Fastabend

This patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.

VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.

Just as a note (as I have quite no idea how ixgbe works) - you are aware that the upper/lower lists currently include *all* upper/lower devices, and not only the first-connected ones?

Yep, the netif_is_vsi_port() should hopefully catch all the cases.

Quoted 3 lines

I've seen that you usually verify it, however not always, just a heads-up - sorry if misread.

I'll audit it again, maybe I missed a case.

Quoted 7 lines

I'm also currently trying to get the new patchset included - which would split the first-tier connected devices from all the 'other' (as in - lower of a lower and upper of an upper) devices, so that way, I think, it would be easier/faster for you to use it. It also has a ->private field, easily accessible, which you could have used instead of/in conjunction with/for struct ixgbe_vsi_adapter.

Yeah I saw the patch set I'll use the improved lists once I see the series gets applied. Although in my case the list walking only occurs in the configuration path so its not critical.

Also the net device structure already has private space to hang the device dependent structure off of, so I'm not sure having another private field helps here.

Thanks,
John

Signature · 1 line
John Fastabend         Intel Corporation
#5 lore raw

Re: [RFC PATCH 2/4] net: Add lower dev list helpers

Quoted 12 lines · John Fastabend
Quoted 10 lines · Veaceslav Falico
Quoted 5 lines · John Fastabend

This patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.

VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.

Just as a note (as I have quite no idea how ixgbe works) - you are aware that the upper/lower lists currently include *all* upper/lower devices, and not only the first-connected ones?

Quoted 7 lines

Yep, the netif_is_vsi_port() should hopefully catch all the cases.

Quoted 3 lines

I've seen that you usually verify it, however not always, just a heads-up - sorry if misread.

I'll audit it again, maybe I missed a case.

It was about lower devices, and I think I've just really misread it - indeed they can be only 'your' devices.

Quoted 9 lines
Quoted 8 lines

I'm also currently trying to get the new patchset included - which would split the first-tier connected devices from all the 'other' (as in - lower of a lower and upper of an upper) devices, so that way, I think, it would be easier/faster for you to use it. It also has a ->private field, easily accessible, which you could have used instead of/in conjunction with/for struct ixgbe_vsi_adapter.

Quoted 7 lines

Yeah I saw the patch set I'll use the improved lists once I see the series gets applied. Although in my case the list walking only occurs in the configuration path so its not critical.

Also the net device structure already has private space to hang the device dependent structure off of, so I'm not sure having another private field helps here.

Yep, it should/can be used when linking devices which might already be using their private space for their own structures, which is not your case, I suppose.

Quoted 6 lines

Thanks,
John

Signature · 1 line
John Fastabend         Intel Corporation
#6 lore raw

RFC PATCH 3/4 net: VSI: Add virtual station interface support

In reply to John Fastabend

This patch adds support for a new device type VSI (virtual station interface) this device type exposes additional net devices complete with queues and a MAC/VLAN pair to the host OS that are logically stacked on top of a switching/routing component with the physical link acting as the downlink to the peer switch.

The hardware on receive path will forward packets to the new VSI net device using the forwarding database (FDB) already exposed via the ndo ops ndo_fdb_{add|del|dump}. On transmit the hardware may use either a VEB or VEPA. In the VEB case traffic may be "switched" between VSI net devices by the hardware and in VEPA case all traffic is sent to the adjacent switch. The hardware _should_ expose this functionality via the ndo_bridge_{set|get}link ndo operations.

This net device should be functionally analogous to an offloaded macvlan device with the ebridge component offloaded into hardware.

Also notice that for now the ixgbe implementation accompanying this patch set only supports L2 forwarding the fdb interfaces could push L3/L4 forwarding to the hardware for more advanced usages including vxlan and other tunnel schemes.

---
 drivers/net/Kconfig       |    9 +++
 drivers/net/Makefile      |    1 
 drivers/net/vsi.c         |  124 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/netdevice.h |   27 ++++++++++
 include/uapi/linux/if.h   |    1 
 5 files changed, 162 insertions(+)
 create mode 100644 drivers/net/vsi.c
drivers/net/Kconfig +9 -0
index b45b240..19be0fb 100644
@@ -362,4 +362,13 @@ config VMXNET3
 
 source "drivers/net/hyperv/Kconfig"
 
+config VSI
+	tristate "Virtual Station Interfaces (VSI)"
+	help
+	  This supports chip sets with embedded switching components
+	  and allows creating additional net devices that are
+	  logically slaves of a master net device typically the net
+	  device associated with the physical function. For these
+	  child devices switching occurs in the hardware component.
+
 endif # NETDEVICES
drivers/net/Makefile +1 -0
index 3fef8a8..3ef1d66 100644
@@ -23,6 +23,7 @@ obj-$(CONFIG_VETH) += veth.o
 obj-$(CONFIG_VIRTIO_NET) += virtio_net.o
 obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_NLMON) += nlmon.o
+obj-$(CONFIG_VSI) += vsi.o
 
 #
 # Networking Drivers
drivers/net/vsi.c new file +124 -0

124 lines not on this page · raw message

include/linux/netdevice.h +27 -0
index 4d24b38..9817745 100644
@@ -961,6 +961,24 @@ struct netdev_phys_port_id {
  *	Called by vxlan to notify the driver about a UDP port and socket
  *	address family that vxlan is not listening to anymore. The operation
  *	is protected by the vxlan_net->sock_lock.
+ *
+ * int (*ndo_vsi_add)(struct net_device *lower, struct net_device *dev)
+ *	Called by the virtual station interface (VSI) link type to add a new
+ *	net device 'dev' to an embedded switch where the embedded switch
+ *	management net device is identified by 'lower'. This should return
+ *	0 on success or may return negative error codes. Error codes should
+ *	be used here to signify resource constraints, unsupportable attributes,
+ *	or any other condition which caused the creation to fail.
+ * void (*ndo_vsi_del)(struct net_device *dev)
+ *	Called by the virtual station interface (VSI) link type to remove the
+ *	net device 'dev' from an embedded switch. Drivers may not fail this
+ *	command.
+ * size_t (*ndo_vsi_size)(struct net_device *dev)
+ *	Called by the virtual station interface (VSI) link type to add the
+ *	required private size to a VSI interface that is being created. If
+ *	this routine is not implemented size_t 0 is used. The 'dev' argument
+ *	indicates the embedded switch management interface where the new
+ *	net devices is being attached.
  */
 struct net_device_ops {
 	int			(*ndo_init)(struct net_device *dev);
@@ -1097,6 +1115,10 @@ struct net_device_ops {
 	void			(*ndo_del_vxlan_port)(struct  net_device *dev,
 						      sa_family_t sa_family,
 						      __u16 port);
+	int			(*ndo_vsi_add)(struct net_device *lower,
+					       struct net_device *dev);
+	void			(*ndo_vsi_del)(struct net_device *dev);
+	size_t			(*ndo_vsi_size)(struct net_device *dev);
 };
 
 /*
@@ -2967,6 +2989,11 @@ static inline bool netif_supports_nofcs(struct net_device *dev)
 	return dev->priv_flags & IFF_SUPP_NOFCS;
 }
 
+static inline bool netif_is_vsi_port(struct net_device *dev)
+{
+	return dev->priv_flags & IFF_VSI_PORT;
+}
+
 extern struct pernet_operations __net_initdata loopback_net_ops;
 
 /* Logging, debugging and troubleshooting/diagnostic helpers. */
include/uapi/linux/if.h +1 -0
index 1ec407b..9b8d6a0 100644
@@ -83,6 +83,7 @@
 #define IFF_SUPP_NOFCS	0x80000		/* device supports sending custom FCS */
 #define IFF_LIVE_ADDR_CHANGE 0x100000	/* device supports hardware address
 					 * change when it's running */
+#define IFF_VSI_PORT 0x200000		/* Virtual Station Interface port */
 
 
 #define IF_GET_IFACE	0x0001		/* for querying only */
#7 lore raw

Re: [RFC PATCH 3/4] net: VSI: Add virtual station interface support

John-
Sorry for not copying your orgional patch into your email, but I apparently
erased this email prior to fully reading it. At any rate, as we discussed the other day, I think this idea is great, but it would benefit from being implemented using macvlans rather than a new link type. If we can set a hardware flag in the underlying driver to indicate support for hardware forwarding, we can skip the macvlan soft forwarding, and just transmit directly. It should save us needing to create a whole new link type that is so simmilar to one we already have.

I'm back home now, so I can start looking at this if you like on monday.

Best
Neil

#8 lore raw

Re: [RFC PATCH 3/4] net: VSI: Add virtual station interface support

Quoted 11 lines · Neil Horman

John-
Sorry for not copying your orgional patch into your email, but I apparently
erased this email prior to fully reading it. At any rate, as we discussed the other day, I think this idea is great, but it would benefit from being implemented using macvlans rather than a new link type. If we can set a hardware flag in the underlying driver to indicate support for hardware forwarding, we can skip the macvlan soft forwarding, and just transmit directly. It should save us needing to create a whole new link type that is so simmilar to one we already have.

Quoted 6 lines

I'm back home now, so I can start looking at this if you like on monday.

Best
Neil

Yes definitely take a look making it an extension (flag) to macvlan would be more useful if it can work. The one question I have is would we need to clear the flag when a feature is attached to the PF that can not be done in hardware. For example attaching an ingress qdisc, ebtables, or starting up tcpdump are two that come to mind.

.John

Sorry for the duplication Neil, I sent this mail from a broken email client just a minute ago and vger dropped it.

#9 lore raw

Re: [RFC PATCH 3/4] net: VSI: Add virtual station interface support

Quoted 19 lines · John Fastabend
Quoted 11 lines · Neil Horman

John-
Sorry for not copying your orgional patch into your email, but I apparently
erased this email prior to fully reading it. At any rate, as we discussed the other day, I think this idea is great, but it would benefit from being implemented using macvlans rather than a new link type. If we can set a hardware flag in the underlying driver to indicate support for hardware forwarding, we can skip the macvlan soft forwarding, and just transmit directly. It should save us needing to create a whole new link type that is so simmilar to one we already have.

Quoted 6 lines

I'm back home now, so I can start looking at this if you like on monday.

Best
Neil

Quoted 6 lines

Yes definitely take a look making it an extension (flag) to macvlan would be more useful if it can work. The one question I have is would we need to clear the flag when a feature is attached to the PF that can not be done in hardware. For example attaching an ingress qdisc, ebtables, or starting up tcpdump are two that come to mind.

I think that it probably would, although I expect that this would be the case independent of the implementations (virtual station interfaces vs. macvlans). The only other option I see is to disallow those features that can't be done in hardware, but I think thats unworkable long term.

Neil

Quoted 9 lines

.John

Sorry for the duplication Neil, I sent this mail from a broken email client just a minute ago and vger dropped it.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw

RFC PATCH 4/4 ixgbe: Adding VSI support to ixgbe

In reply to John Fastabend

This adds initial base support for VSI interfaces to the ixgbe driver.

This supports up to 32 VSIs per physical function with each VSI supporting up to 4 TX/RX queue pairs. The hardware has support for modes using 2 queue pairs that enable up to 64 VSIs but they are not enabled here. VSIs can be instantiated with 1,2, or 4 queues unfortunately due to hardware restrictions related to RSS masking across queues 3 TX/RX queues is not supported. If a user inputs a queue parameter of 3 for either TX or RX the vsi_add will fail with EINVAL and a message will be printed to dmesg explaining the error. EBUSY is returned if more than 32 VSIs are added.

The driver uses a bitmask added to the ixgbe_adapter struct to track VSIs. VSIs spawn net devices that can be independently managed via their own ndo and ethtool ops.

Currently, MAC addresses must be added manually via 'ip link set' or similar configuration (netlink) commands before bringing the spawned devices online. I could easily generate random MAC addresses if that is prefered although not required in my use case and seems a bit arbitrary. I expect most users have a pool of known "good" MAC addresses they can use or for testing can simply pick their favorite address.

Additionally VSIs do not support promisc mode. It is not clear to me at least what promiscuous mode means when the VSI is behind a non-learning embedded bridge wich does not do flooding. The hardware does not support flooding on the embedded bridge however it does support mirroring which could be enabled to get something that looks like promiscuous mode. Currently there is no kernel support (netlink cmd?) to configure mirroring.

The embedded bridge that connects VSIs looks like an edge relay in IEEE std terms. And can support both VEB and VEPA modes which are managed independently of VSIs through the {set|get} bridge APIs.

VSIs only support a single MAC addresses in this patch. This is a software limitation and can/will be extended in subsequent patches. For now a single MAC addresses is supported.

NOTE:
The SRIOV and VMDQ flags in ixgbe seem to be a bit overloaded and convoluted. This patch uses these flags. A follow up patch could clean up the ixgbe flag usage scheme.

RFC TODO: I'll complete these before sending the real patch
(1) interop with DCB/FCOE/SR-IOV
(2) vmdq_netdev can be dropped from tx_ring and just use correctly set netdev. In slowpath we can look up management netdev via vadapter->real_adapter->netdev and save the hotpath vmdq_netdev if/else branch.

Otherwise I've been adding/delete VSIs with multiple netperf TCP sessions without any crashes but I still need to do some more testing.

---
 drivers/net/ethernet/intel/ixgbe/Makefile        |    3 
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |   32 ++
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    4 
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |   15 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  307 +++++++++++-----
 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c     |  428 ++++++++++++++++++++++
 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h     |   71 ++++
 7 files changed, 766 insertions(+), 94 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c
 create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h
drivers/net/ethernet/intel/ixgbe/Makefile +2 -1
index be2989e..24136e7 100644
@@ -34,7 +34,8 @@ obj-$(CONFIG_IXGBE) += ixgbe.o
 
 ixgbe-objs := ixgbe_main.o ixgbe_common.o ixgbe_ethtool.o \
               ixgbe_82599.o ixgbe_82598.o ixgbe_phy.o ixgbe_sriov.o \
-              ixgbe_mbx.o ixgbe_x540.o ixgbe_lib.o ixgbe_ptp.o
+              ixgbe_mbx.o ixgbe_x540.o ixgbe_lib.o ixgbe_ptp.o \
+	      ixgbe_vsi.o
 
 ixgbe-$(CONFIG_IXGBE_DCB) +=  ixgbe_dcb.o ixgbe_dcb_82598.o \
                               ixgbe_dcb_82599.o ixgbe_dcb_nl.o
drivers/net/ethernet/intel/ixgbe/ixgbe.h +30 -2
index 0ac6b11..ba2ab14 100644
@@ -244,6 +244,7 @@ struct ixgbe_ring {
 	unsigned long last_rx_timestamp;
 	unsigned long state;
 	u8 __iomem *tail;
+	struct net_device *vmdq_netdev;
 	dma_addr_t dma;			/* phys. address of descriptor ring */
 	unsigned int size;		/* length in bytes */
 
@@ -288,11 +289,15 @@ enum ixgbe_ring_f_enum {
 };
 
 #define IXGBE_MAX_RSS_INDICES  16
-#define IXGBE_MAX_VMDQ_INDICES 64
+#define IXGBE_MAX_VMDQ_INDICES 32
 #define IXGBE_MAX_FDIR_INDICES 63	/* based on q_vector limit */
 #define IXGBE_MAX_FCOE_INDICES  8
 #define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
 #define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
+#define IXGBE_MAX_VSI_QUEUES 4
+#define IXGBE_MAX_VSI_QUEUES 4
+#define IXGBE_BAD_VSI_QUEUE 3
+
 struct ixgbe_ring_feature {
 	u16 limit;	/* upper limit on feature indices */
 	u16 indices;	/* current value of indices */
@@ -738,6 +743,7 @@ struct ixgbe_adapter {
 #endif /*CONFIG_DEBUG_FS*/
 
 	u8 default_up;
+	unsigned long vsi_bitmask; /* Bitmask indicating in use pools */
 };
 
 struct ixgbe_fdir_filter {
@@ -747,6 +753,17 @@ struct ixgbe_fdir_filter {
 	u16 action;
 };
 
+struct ixgbe_vsi_adapter {
+	unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
+	struct net_device *netdev;
+	struct ixgbe_adapter *real_adapter;
+	unsigned int tx_base_queue;
+	unsigned int rx_base_queue;
+	struct net_device_stats net_stats;
+	int pool;
+	bool online;
+};
+
 enum ixgbe_state_t {
 	__IXGBE_TESTING,
 	__IXGBE_RESETTING,
@@ -879,9 +896,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}
 static inline void ixgbe_dbg_init(void) {}
 static inline void ixgbe_dbg_exit(void) {}
 #endif /* CONFIG_DEBUG_FS */
+static inline struct net_device *netdev_ring(const struct ixgbe_ring *ring)
+{
+	return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev;
+}
+
 static inline struct netdev_queue *txring_txq(const struct ixgbe_ring *ring)
 {
-	return netdev_get_tx_queue(ring->netdev, ring->queue_index);
+	return netdev_get_tx_queue(netdev_ring(ring), ring->queue_index);
 }
 
 extern void ixgbe_ptp_init(struct ixgbe_adapter *adapter);
@@ -915,4 +937,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);
 void ixgbe_sriov_reinit(struct ixgbe_adapter *adapter);
 #endif
 
+int ixgbe_get_settings(struct net_device *dev, struct ethtool_cmd *ecmd);
+int ixgbe_write_uc_addr_list(struct net_device *netdev);
+netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb,
+				  struct ixgbe_adapter *adapter,
+				  struct ixgbe_ring *tx_ring);
+void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring);
 #endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
index 0e1b973..48b2d81 100644
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = {
 };
 #define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN
 
-static int ixgbe_get_settings(struct net_device *netdev,
-                              struct ethtool_cmd *ecmd)
+int ixgbe_get_settings(struct net_device *netdev,
+		       struct ethtool_cmd *ecmd)
 {
 	struct ixgbe_adapter *adapter = netdev_priv(netdev);
 	struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
index 90b4e10..e2dd635 100644
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)
 #endif
 
 	/* only proceed if SR-IOV is enabled */
-	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED))
+	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) &&
+	    !(adapter->flags & IXGBE_FLAG_VMDQ_ENABLED))
 		return false;
 
 	/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 
 		/* apply Tx specific ring traits */
 		ring->count = adapter->tx_ring_count;
-		ring->queue_index = txr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				txr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = txr_idx;
 
 		/* assign ring to adapter */
 		adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 #endif /* IXGBE_FCOE */
 		/* apply Rx specific ring traits */
 		ring->count = adapter->rx_ring_count;
-		ring->queue_index = rxr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				rxr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = rxr_idx;
 
 		/* assign ring to adapter */
 		adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +221 -86

589 lines not on this page · raw message

drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c new file +428 -0

428 lines not on this page · raw message

drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h new file +71 -0

71 lines not on this page · raw message

#11 lore raw

cover net: alternate proposal for using macvlans with forwarding acceleration

In reply to John Fastabend

John, et al. -
As promised, heres my (very rough) first pass at an alternate propsal for
what you're trying to do with virtual station interfaces here. Its completely untested, but it builds, and I'll be trying to run it over the next few days (though I'm sure I got part of the hardware manipulation wrong). I wanted to post it early though so you could get a look at it to see what you did and didn't like about it. Some notes:

1) As discussed, the major effort here is to tie in macvlans with l2 forwarding acceleration, rather than creating a new vsi link type. That should make management easier for admins (be it via ovs or some other mechanism). It basically exposes a bit less to the user, which I think is good.

2) I've separated out the l2 forwarding acceleration operations from the net_device_operations structure. I'm not sure I like that yet, but I'm kind on leaning that way. Since a limited set of hardare supports forwarding acceleration, it makes for a nice easy way to group functionality without polluting the net_device_operations structure. It also lets us group simmilar functions together nicely (I can see a future l3_accel_ops structure if we can do l3 flows in hardware). Anywho, its a divergence from what we've been doing so I thought I would call attention to it.

3) I've included a l2_accel_xmit method in the accel_ops structure for fast path forwarding, but I'm not sure I like that. It seems we should be able to use ndo_start_xmit and key off some data to recognize that we should be doing hardware forwarding. I'm not quite sure how to do that yet though. Something to think about.

4) I've borrowed heavily from your vsi work of course just to get this building. I think theres probbaly alot of consolidation that can be done in the code that I added to ixgbe_main.c to make it smaller. Again, I just wanted to post this so you could speak up if you though this was all crap before I wen't too far down the rabbit hole.

Regards
Neil

#12 lore raw

RFC PATCH 1/2 net: Add layer 2 hardware acceleration operations for macvlan devices

Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.

---
 drivers/net/macvlan.c      | 37 +++++++++++++++++++++++++++++++++++++
 include/linux/if_macvlan.h |  1 +
 include/linux/netdevice.h  | 10 ++++++++++
 net/core/dev.c             |  3 +++
 4 files changed, 51 insertions(+)
drivers/net/macvlan.c +37 -0
index 9bf46bd..0c37b30 100644
@@ -296,8 +296,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,
 	unsigned int len = skb->len;
 	int ret;
 	const struct macvlan_dev *vlan = netdev_priv(dev);
+	const struct l2_forwarding_accel_ops *l2a_ops = vlan->lowerdev->l2a_ops;
+
+	if (l2a_ops->l2_accel_xmit) {
+		ret = l2a_ops->l2_accel_xmit(skb, vlan->l2a_priv);
+		if (likely(ret == NETDEV_TX_OK))
+			goto update_stats;
+	}
 
 	ret = macvlan_queue_xmit(skb, dev);
+update_stats:
 	if (likely(ret == NET_XMIT_SUCCESS || ret == NET_XMIT_CN)) {
 		struct macvlan_pcpu_stats *pcpu_stats;
 
@@ -336,6 +344,7 @@ static int macvlan_open(struct net_device *dev)
 {
 	struct macvlan_dev *vlan = netdev_priv(dev);
 	struct net_device *lowerdev = vlan->lowerdev;
+	const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;
 	int err;
 
 	if (vlan->port->passthru) {
@@ -347,6 +356,19 @@ static int macvlan_open(struct net_device *dev)
 		goto hash_add;
 	}
 
+	if (l2a_ops->l2_accel_add_dev) {
+		/* The lowerdev supports l2 switching
+		 * try to add this macvlan to it
+		 */
+		vlan->l2a_priv = kzalloc(l2a_ops->priv_size, GFP_KERNEL);
+		if (!vlan->l2a_priv)
+			return -ENOMEM;
+		err = l2a_ops->l2_accel_add_dev(vlan->lowerdev,
+						dev, vlan->l2a_priv);
+		if (err < 0)
+			return err;
+	}
+
 	err = -EBUSY;
 	if (macvlan_addr_busy(vlan->port, dev->dev_addr))
 		goto out;
@@ -367,6 +389,13 @@ hash_add:
 del_unicast:
 	dev_uc_del(lowerdev, dev->dev_addr);
 out:
+	if (vlan->l2a_priv) {
+		if (l2a_ops->l2_accel_del_dev)
+			l2a_ops->l2_accel_del_dev(vlan->lowerdev,
+						  vlan->l2a_priv);
+		kfree(vlan->l2a_priv);
+		vlan->l2a_priv = NULL;
+	}
 	return err;
 }
 
@@ -374,6 +403,7 @@ static int macvlan_stop(struct net_device *dev)
 {
 	struct macvlan_dev *vlan = netdev_priv(dev);
 	struct net_device *lowerdev = vlan->lowerdev;
+	const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;
 
 	dev_uc_unsync(lowerdev, dev);
 	dev_mc_unsync(lowerdev, dev);
@@ -391,6 +421,12 @@ static int macvlan_stop(struct net_device *dev)
 
 hash_del:
 	macvlan_hash_del(vlan, !dev->dismantle);
+	if (vlan->l2a_priv) {
+		l2a_ops->l2_accel_del_dev(vlan->lowerdev, vlan->l2a_priv);
+		kfree(vlan->l2a_priv);
+		vlan->l2a_priv = NULL;
+	}
+
 	return 0;
 }
 
@@ -801,6 +837,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev,
 		if (err < 0)
 			return err;
 	}
+
 	port = macvlan_port_get_rtnl(lowerdev);
 
 	/* Only 1 macvlan device can be created in passthru mode */
include/linux/if_macvlan.h +1 -0
index ddd33fd..9b16af9 100644
@@ -61,6 +61,7 @@ struct macvlan_dev {
 	struct hlist_node	hlist;
 	struct macvlan_port	*port;
 	struct net_device	*lowerdev;
+	void			*l2a_priv;
 	struct macvlan_pcpu_stats __percpu *pcpu_stats;
 
 	DECLARE_BITMAP(mc_filter, MACVLAN_MC_FILTER_SZ);
include/linux/netdevice.h +10 -0
index 3de49ac..4cbc535 100644
@@ -1099,6 +1099,15 @@ struct net_device_ops {
 						      __be16 port);
 };
 
+struct l2_forwarding_accel_ops {
+	size_t priv_size;
+	int	(*l2_accel_add_dev)(struct net_device *pdev,
+				    struct net_device *vdev, void *priv);
+	void	(*l2_accel_del_dev)(struct net_device *pdev, void *priv);
+	netdev_tx_t             (*l2_accel_xmit)(struct sk_buff *skb,
+						 void *priv);
+};
+
 /*
  *	The DEVICE structure.
  *	Actually, this whole structure is a big mistake.  It mixes I/O
@@ -1183,6 +1192,7 @@ struct net_device {
 	/* Management operations */
 	const struct net_device_ops *netdev_ops;
 	const struct ethtool_ops *ethtool_ops;
+	const struct l2_forwarding_accel_ops *l2a_ops;
 
 	/* Hardware header description */
 	const struct header_ops *header_ops;
net/core/dev.c +3 -0
index 5c713f2..5c5eec2 100644
@@ -5992,6 +5992,7 @@ struct netdev_queue *dev_ingress_queue_create(struct net_device *dev)
 }
 
 static const struct ethtool_ops default_ethtool_ops;
+static const struct l2_forwarding_accel_ops default_l2a_ops;
 
 void netdev_set_default_ethtool_ops(struct net_device *dev,
 				    const struct ethtool_ops *ops)
@@ -6090,6 +6091,8 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name,
 	dev->group = INIT_NETDEV_GROUP;
 	if (!dev->ethtool_ops)
 		dev->ethtool_ops = &default_ethtool_ops;
+	if (!dev->l2a_ops)
+		dev->l2a_ops = &default_l2a_ops;
 	return dev;
 
 free_all:
Signature · 1 line
1.8.3.1
#13 lore raw

Re: [RFC PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

Quoted 5 lines · Neil Horman

Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.

Some additional nits below which maybe you have already thought of.

Quoted 16 lines
---
  drivers/net/macvlan.c      | 37 +++++++++++++++++++++++++++++++++++++
  include/linux/if_macvlan.h |  1 +
  include/linux/netdevice.h  | 10 ++++++++++
  net/core/dev.c             |  3 +++
  4 files changed, 51 insertions(+)
drivers/net/macvlan.c +0 -0
index 9bf46bd..0c37b30 100644

drivers/net/macvlan.c netdev_tx_t macvlan_start_xmit(struct sk_buff *skb, · line 303

Quoted 8 lines
drivers/net/macvlan.c +5 -0
@@ -296,8 +296,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,
  	unsigned int len = skb->len;
  	int ret;
  	const struct macvlan_dev *vlan = netdev_priv(dev);
+	const struct l2_forwarding_accel_ops *l2a_ops = vlan->lowerdev->l2a_ops;
+
+	if (l2a_ops->l2_accel_xmit) {
+		ret = l2a_ops->l2_accel_xmit(skb, vlan->l2a_priv);
+		if (likely(ret == NETDEV_TX_OK))

maybe dev_xmit_complete() would be more appropriate?

Quoted 17 lines
drivers/net/macvlan.c +4 -0
+			goto update_stats;
+	}
 
  	ret = macvlan_queue_xmit(skb, dev);
+update_stats:
  	if (likely(ret == NET_XMIT_SUCCESS || ret == NET_XMIT_CN)) {
  		struct macvlan_pcpu_stats *pcpu_stats;
 
@@ -336,6 +344,7 @@ static int macvlan_open(struct net_device *dev)
  {
  	struct macvlan_dev *vlan = netdev_priv(dev);
  	struct net_device *lowerdev = vlan->lowerdev;
+	const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;
  	int err;
 
  	if (vlan->port->passthru) {

drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 356

Quoted 1 line
drivers/net/macvlan.c +0 -0
@@ -347,6 +356,19 @@ static int macvlan_open(struct net_device *dev)
  		goto hash_add;

Looks like this might break in the passthru case? If you don't call l2_accel_add_dev here but still use the l2_accel_xmit.

drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 359

Quoted 3 lines
drivers/net/macvlan.c +1 -0
  	}
 
+	if (l2a_ops->l2_accel_add_dev) {

In the error cases it might be preferred to fallback to the non-offloaded software path. For example hardware may have a limit to the number of VSIs that can be created but we wouldn't want to push that up the stack.

Quoted 16 lines
drivers/net/macvlan.c +12 -0
+		/* The lowerdev supports l2 switching
+		 * try to add this macvlan to it
+		 */
+		vlan->l2a_priv = kzalloc(l2a_ops->priv_size, GFP_KERNEL);
+		if (!vlan->l2a_priv)
+			return -ENOMEM;
+		err = l2a_ops->l2_accel_add_dev(vlan->lowerdev,
+						dev, vlan->l2a_priv);
+		if (err < 0)
+			return err;
+	}
+
  	err = -EBUSY;
  	if (macvlan_addr_busy(vlan->port, dev->dev_addr))
  		goto out;

drivers/net/macvlan.c hash_add: · line 392

Quoted 4 lines
drivers/net/macvlan.c +1 -0
@@ -367,6 +389,13 @@ hash_add:
  del_unicast:
  	dev_uc_del(lowerdev, dev->dev_addr);
  out:
+	if (vlan->l2a_priv) {

Add a feature flag here so it can be disabled.

drivers/net/macvlan.c hash_add: · line 400

Quoted 8 lines
drivers/net/macvlan.c +6 -0
+		if (l2a_ops->l2_accel_del_dev)
+			l2a_ops->l2_accel_del_dev(vlan->lowerdev,
+						  vlan->l2a_priv);
+		kfree(vlan->l2a_priv);
+		vlan->l2a_priv = NULL;
+	}
  	return err;
  }

[...]

Thanks,
John

Signature · 1 line
John Fastabend         Intel Corporation
#14 lore raw

Re: [RFC PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

Quoted 9 lines · John Fastabend
Quoted 5 lines · Neil Horman

Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.

Some additional nits below which maybe you have already thought of.

Its ok, you can say it, theyre more than nits :), gaping holes is more like it. This pass was completely untested, I'm still ironing out bits, but I think for the most part we're in agreement on the items below. Comments inline.

Quoted 11 lines
Quoted 11 lines
---
 drivers/net/macvlan.c      | 37 +++++++++++++++++++++++++++++++++++++
 include/linux/if_macvlan.h |  1 +
 include/linux/netdevice.h  | 10 ++++++++++
 net/core/dev.c             |  3 +++
 4 files changed, 51 insertions(+)
Quoted 16 lines

drivers/net/macvlan.c netdev_tx_t macvlan_start_xmit(struct sk_buff *skb, · line 303

Quoted 13 lines
drivers/net/macvlan.c +5 -0
index 9bf46bd..0c37b30 100644
@@ -296,8 +296,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,
 	unsigned int len = skb->len;
 	int ret;
 	const struct macvlan_dev *vlan = netdev_priv(dev);
+	const struct l2_forwarding_accel_ops *l2a_ops = vlan->lowerdev->l2a_ops;
+
+	if (l2a_ops->l2_accel_xmit) {
+		ret = l2a_ops->l2_accel_xmit(skb, vlan->l2a_priv);
+		if (likely(ret == NETDEV_TX_OK))

maybe dev_xmit_complete() would be more appropriate?

Yup, I've currently modified this to dev_queue_xmit, but _complete might be a better option.

Quoted 15 lines
Quoted 15 lines
drivers/net/macvlan.c +4 -0
+			goto update_stats;
+	}
 
 	ret = macvlan_queue_xmit(skb, dev);
+update_stats:
 	if (likely(ret == NET_XMIT_SUCCESS || ret == NET_XMIT_CN)) {
 		struct macvlan_pcpu_stats *pcpu_stats;
 
@@ -336,6 +344,7 @@ static int macvlan_open(struct net_device *dev)
 {
 	struct macvlan_dev *vlan = netdev_priv(dev);
 	struct net_device *lowerdev = vlan->lowerdev;
+	const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;
 	int err;
 
Quoted 7 lines

drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 356

Quoted 3 lines
drivers/net/macvlan.c +0 -0
 	if (vlan->port->passthru) {
@@ -347,6 +356,19 @@ static int macvlan_open(struct net_device *dev)
 		goto hash_add;

Looks like this might break in the passthru case? If you don't call l2_accel_add_dev here but still use the l2_accel_xmit.

Yeah, I need to clean this up.

Quoted 9 lines

drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 359

Quoted 3 lines
drivers/net/macvlan.c +1 -0
 	}
 
+	if (l2a_ops->l2_accel_add_dev) {

In the error cases it might be preferred to fallback to the non-offloaded software path. For example hardware may have a limit to the number of VSIs that can be created but we wouldn't want to push that up the stack.

Hadn't thought of that, but yes, I agree, falling back to software switching is definately desireable here

Quoted 14 lines
Quoted 14 lines
drivers/net/macvlan.c +12 -0
+		/* The lowerdev supports l2 switching
+		 * try to add this macvlan to it
+		 */
+		vlan->l2a_priv = kzalloc(l2a_ops->priv_size, GFP_KERNEL);
+		if (!vlan->l2a_priv)
+			return -ENOMEM;
+		err = l2a_ops->l2_accel_add_dev(vlan->lowerdev,
+						dev, vlan->l2a_priv);
+		if (err < 0)
+			return err;
+	}
+
 	err = -EBUSY;
 	if (macvlan_addr_busy(vlan->port, dev->dev_addr))
Quoted 9 lines

drivers/net/macvlan.c hash_add: · line 392

Quoted 6 lines
drivers/net/macvlan.c +1 -0
 		goto out;
@@ -367,6 +389,13 @@ hash_add:
 del_unicast:
 	dev_uc_del(lowerdev, dev->dev_addr);
 out:
+	if (vlan->l2a_priv) {

Add a feature flag here so it can be disabled.

Makes sense.

I'm still working out kinks, but I hope to have a next version later this week/early next. I'll make sure to incorporate these changes.

Thanks!
Neil

Quoted 18 lines

drivers/net/macvlan.c hash_add: · line 400

Quoted 8 lines
drivers/net/macvlan.c +6 -0
+		if (l2a_ops->l2_accel_del_dev)
+			l2a_ops->l2_accel_del_dev(vlan->lowerdev,
+						  vlan->l2a_priv);
+		kfree(vlan->l2a_priv);
+		vlan->l2a_priv = NULL;
+	}
 	return err;
 }

[...]

Thanks,
John

Signature · 1 line
John Fastabend         Intel Corporation
#15 lore raw

RFC PATCH 2/2 ixgbe: enable l2 forwarding acceleration for macvlans

In reply to Neil Horman

Now that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |  32 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   4 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h     |  54 ++++
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |  15 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    | 360 +++++++++++++++++------
 5 files changed, 374 insertions(+), 91 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h
drivers/net/ethernet/intel/ixgbe/ixgbe.h +30 -2
index 0ac6b11..cfa6a8b 100644
@@ -244,6 +244,7 @@ struct ixgbe_ring {
 	unsigned long last_rx_timestamp;
 	unsigned long state;
 	u8 __iomem *tail;
+	struct net_device *vmdq_netdev;
 	dma_addr_t dma;			/* phys. address of descriptor ring */
 	unsigned int size;		/* length in bytes */
 
@@ -288,11 +289,15 @@ enum ixgbe_ring_f_enum {
 };
 
 #define IXGBE_MAX_RSS_INDICES  16
-#define IXGBE_MAX_VMDQ_INDICES 64
+#define IXGBE_MAX_VMDQ_INDICES 32
 #define IXGBE_MAX_FDIR_INDICES 63	/* based on q_vector limit */
 #define IXGBE_MAX_FCOE_INDICES  8
 #define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
 #define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
+#define IXGBE_MAX_L2A_QUEUES 4
+#define IXGBE_MAX_L2A_QUEUES 4
+#define IXGBE_BAD_L2A_QUEUE 3
+
 struct ixgbe_ring_feature {
 	u16 limit;	/* upper limit on feature indices */
 	u16 indices;	/* current value of indices */
@@ -738,6 +743,7 @@ struct ixgbe_adapter {
 #endif /*CONFIG_DEBUG_FS*/
 
 	u8 default_up;
+	unsigned long l2a_bitmask; /* Bitmask indicating in use pools */
 };
 
 struct ixgbe_fdir_filter {
@@ -747,6 +753,17 @@ struct ixgbe_fdir_filter {
 	u16 action;
 };
 
+struct ixgbe_l2a_adapter {
+	unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
+	struct net_device *netdev;
+	struct ixgbe_adapter *real_adapter;
+	unsigned int tx_base_queue;
+	unsigned int rx_base_queue;
+	struct net_device_stats net_stats;
+	int pool;
+	bool online;
+};
+
 enum ixgbe_state_t {
 	__IXGBE_TESTING,
 	__IXGBE_RESETTING,
@@ -879,9 +896,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}
 static inline void ixgbe_dbg_init(void) {}
 static inline void ixgbe_dbg_exit(void) {}
 #endif /* CONFIG_DEBUG_FS */
+static inline struct net_device *netdev_ring(const struct ixgbe_ring *ring)
+{
+	return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev;
+}
+
 static inline struct netdev_queue *txring_txq(const struct ixgbe_ring *ring)
 {
-	return netdev_get_tx_queue(ring->netdev, ring->queue_index);
+	return netdev_get_tx_queue(netdev_ring(ring), ring->queue_index);
 }
 
 extern void ixgbe_ptp_init(struct ixgbe_adapter *adapter);
@@ -915,4 +937,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);
 void ixgbe_sriov_reinit(struct ixgbe_adapter *adapter);
 #endif
 
+int ixgbe_get_settings(struct net_device *dev, struct ethtool_cmd *ecmd);
+int ixgbe_write_uc_addr_list(struct net_device *netdev);
+netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb,
+				  struct ixgbe_adapter *adapter,
+				  struct ixgbe_ring *tx_ring);
+void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring);
 #endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
index e8649ab..277af14 100644
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = {
 };
 #define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN
 
-static int ixgbe_get_settings(struct net_device *netdev,
-                              struct ethtool_cmd *ecmd)
+int ixgbe_get_settings(struct net_device *netdev,
+		       struct ethtool_cmd *ecmd)
 {
 	struct ixgbe_adapter *adapter = netdev_priv(netdev);
 	struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h new file +54 -0

54 lines not on this page · raw message

drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
index 90b4e10..e2dd635 100644
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)
 #endif
 
 	/* only proceed if SR-IOV is enabled */
-	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED))
+	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) &&
+	    !(adapter->flags & IXGBE_FLAG_VMDQ_ENABLED))
 		return false;
 
 	/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 
 		/* apply Tx specific ring traits */
 		ring->count = adapter->tx_ring_count;
-		ring->queue_index = txr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				txr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = txr_idx;
 
 		/* assign ring to adapter */
 		adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 #endif /* IXGBE_FCOE */
 		/* apply Rx specific ring traits */
 		ring->count = adapter->rx_ring_count;
-		ring->queue_index = rxr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				rxr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = rxr_idx;
 
 		/* assign ring to adapter */
 		adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +276 -84

579 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#16 lore raw

Re: [RFC PATCH 0/2] net: alternate proposal for using macvlans with forwarding acceleration

In reply to Neil Horman
Quoted 7 lines · Neil Horman

John, et al. -
As promised, heres my (very rough) first pass at an alternate propsal for
what you're trying to do with virtual station interfaces here. Its completely untested, but it builds, and I'll be trying to run it over the next few days (though I'm sure I got part of the hardware manipulation wrong). I wanted to post it early though so you could get a look at it to see what you did and didn't like about it. Some notes:

Sorry for the delay. I like the idea one nice win here is my macvlan kvm setup would use the offloads without having to reconfigure.

Quoted 6 lines

1) As discussed, the major effort here is to tie in macvlans with l2 forwarding acceleration, rather than creating a new vsi link type. That should make management easier for admins (be it via ovs or some other mechanism). It basically exposes a bit less to the user, which I think is good.

The trick here is the offload path may be functionally different from the non-offload path. The user needs some visibility into this. For example any qdiscs running on the lowerdev will not be visible from the accelerated path.

When a new link type was being used I was able to convince myself that it was a property of the link type. But if we reuse macvlan I think we need some way to either automatically disable the offload path when this occurs or provide the user visibility. Maybe a feature flag and a netif_can_hw_offload() routine is needed?

Quoted 9 lines

2) I've separated out the l2 forwarding acceleration operations from the net_device_operations structure. I'm not sure I like that yet, but I'm kind on leaning that way. Since a limited set of hardare supports forwarding acceleration, it makes for a nice easy way to group functionality without polluting the net_device_operations structure. It also lets us group simmilar functions together nicely (I can see a future l3_accel_ops structure if we can do l3 flows in hardware). Anywho, its a divergence from what we've been doing so I thought I would call attention to it.

Quoted 5 lines

3) I've included a l2_accel_xmit method in the accel_ops structure for fast path forwarding, but I'm not sure I like that. It seems we should be able to use ndo_start_xmit and key off some data to recognize that we should be doing hardware forwarding. I'm not quite sure how to do that yet though. Something to think about.

Without a specific xmit routine though you will be adding operations in the common case for a special case. Having a new op fixes this.

Quoted 6 lines

4) I've borrowed heavily from your vsi work of course just to get this building. I think theres probbaly alot of consolidation that can be done in the code that I added to ixgbe_main.c to make it smaller. Again, I just wanted to post this so you could speak up if you though this was all crap before I wen't too far down the rabbit hole.

There was some consolidation needed in my original RFC as well. I can help clean some of this stuff up if you want.

.John

Signature · 1 line
John Fastabend         Intel Corporation
#17 lore raw

Re: [RFC PATCH 0/2] net: alternate proposal for using macvlans with forwarding acceleration

Quoted 9 lines · John Fastabend
Quoted 7 lines · Neil Horman

John, et al. -
As promised, heres my (very rough) first pass at an alternate propsal for
what you're trying to do with virtual station interfaces here. Its completely untested, but it builds, and I'll be trying to run it over the next few days (though I'm sure I got part of the hardware manipulation wrong). I wanted to post it early though so you could get a look at it to see what you did and didn't like about it. Some notes:

Quoted 3 lines

Sorry for the delay. I like the idea one nice win here is my macvlan kvm setup would use the offloads without having to reconfigure.

Thats ok, I've been swamped, so this has gone slower than I had hoped for me.

Quoted 7 lines
Quoted 6 lines

1) As discussed, the major effort here is to tie in macvlans with l2 forwarding acceleration, rather than creating a new vsi link type. That should make management easier for admins (be it via ovs or some other mechanism). It basically exposes a bit less to the user, which I think is good.

Quoted 5 lines

The trick here is the offload path may be functionally different from the non-offload path. The user needs some visibility into this. For example any qdiscs running on the lowerdev will not be visible from the accelerated path.

Actually, given your suggestion in your other note (using dev_queue_xmit), I think this will be handled, in that we will now pass accelerated traffic through any lowerdev attached qdiscs.

Quoted 6 lines

When a new link type was being used I was able to convince myself that it was a property of the link type. But if we reuse macvlan I think we need some way to either automatically disable the offload path when this occurs or provide the user visibility. Maybe a feature flag and a netif_can_hw_offload() routine is needed?

As noted above, I think we can make macvlans work identially with and without acceleration by reusing the xmit path, and keying off data (what I've done is attach a pointer to acceleration data in the skb so the common ndo_start_xmit path in the driver can differentiate accelerated from non-accelerated skbs). That will allow everything to function the same way. I do think though that a feature flag just to provide some visibility to the admin is warranted.

Quoted 9 lines
Quoted 9 lines

2) I've separated out the l2 forwarding acceleration operations from the net_device_operations structure. I'm not sure I like that yet, but I'm kind on leaning that way. Since a limited set of hardare supports forwarding acceleration, it makes for a nice easy way to group functionality without polluting the net_device_operations structure. It also lets us group simmilar functions together nicely (I can see a future l3_accel_ops structure if we can do l3 flows in hardware). Anywho, its a divergence from what we've been doing so I thought I would call attention to it.

Quoted 9 lines
Quoted 5 lines

3) I've included a l2_accel_xmit method in the accel_ops structure for fast path forwarding, but I'm not sure I like that. It seems we should be able to use ndo_start_xmit and key off some data to recognize that we should be doing hardware forwarding. I'm not quite sure how to do that yet though. Something to think about.

Without a specific xmit routine though you will be adding operations in the common case for a special case. Having a new op fixes this.

Agreed, and thats something to consider. I'm not sure if separating the special case to its own operation is a net win however, given the behavioral differences that arise (the qdisc issue above). My next version will use a common xmit path. We can look it over then for comparison and weight the pros and cons.

Quoted 10 lines
Quoted 6 lines

4) I've borrowed heavily from your vsi work of course just to get this building. I think theres probbaly alot of consolidation that can be done in the code that I added to ixgbe_main.c to make it smaller. Again, I just wanted to post this so you could speak up if you though this was all crap before I wen't too far down the rabbit hole.

There was some consolidation needed in my original RFC as well. I can help clean some of this stuff up if you want.

That would be greatly appreciated. Thanks. Next version should be available in the next few days. I'll get it to you asap.

Best
Neil

Quoted 5 lines

.John

Signature · 1 line
John Fastabend         Intel Corporation
#18 lore raw

v2 cover net: alternate proposal for using macvlans with forwarding acceleration

In reply to Neil Horman

Hey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.

Some change notes:

* Changes to the fowarding ops structure - Removed the priv_size field, and added a flags field. Removal of the priv_size field was accomplished by just having the add method return a void * and using ERR_PTR and PTR_ERR checks, which also allows us to allocate memory for the acceleration path in the driver, which I like. I'm not super happy still with how I'm using the flags (currenly only used to indicate support for feature sets), but at least we have the flags now, and they can be exposed to user space via iproute2 or ethtool if need be

* Changes to the Transmit path - Specifically I'm using dev_queue_xmit to send frames now, which I like as it makes the macvlan subject to the lowerdevs qdisc configuration.

* Changes to the acceleration fail path behavior - Now if we don't/can't use acceleration, we just fall back to using the normal macvlan software switch strategy

* General clenups (some renaming, that I'm not super sure of, but I though forwarding acceleration (fwd) would be a better prefix than l2 acceleration).

Still a long way to go I think, and lots of tweaking to do, but I didn't want to keep you waiting John. Anywho, take a look at what I'm doing and feel free to rip it apart.

Thanks!
Neil

#19 lore raw

PATCH 1/2 net: Add layer 2 hardware acceleration operations for macvlan devices

Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.

CC John Fastabend <[email protected]>
CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>
---
 drivers/net/macvlan.c      | 32 ++++++++++++++++++++++++++++++++
 include/linux/if_macvlan.h |  1 +
 include/linux/netdevice.h  | 22 ++++++++++++++++++++++
 include/linux/skbuff.h     |  9 ++++++---
 net/core/dev.c             |  3 +++
 5 files changed, 64 insertions(+), 3 deletions(-)
drivers/net/macvlan.c +32 -0
index 9bf46bd..38d0fc5 100644
@@ -297,7 +297,17 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,
 	int ret;
 	const struct macvlan_dev *vlan = netdev_priv(dev);
 
+	if (vlan->fwd_priv) {
+		skb->dev = vlan->lowerdev;
+		skb->accel_priv = vlan->fwd_priv;
+		ret = dev_queue_xmit(skb);
+		if (likely(ret == NETDEV_TX_OK))
+			goto update_stats;
+	}
+
+	skb->accel_priv = NULL;
 	ret = macvlan_queue_xmit(skb, dev);
+update_stats:
 	if (likely(ret == NET_XMIT_SUCCESS || ret == NET_XMIT_CN)) {
 		struct macvlan_pcpu_stats *pcpu_stats;
 
@@ -347,6 +357,18 @@ static int macvlan_open(struct net_device *dev)
 		goto hash_add;
 	}
 
+	if (fwd_accel_supports(lowerdev, FA_FLG_STA_SUPPORT)) {
+		vlan->fwd_priv = fwd_accel_add_station(lowerdev, dev);
+		/*
+		 * If we get a NULL pointer back, or if we get an error
+		 * then we should just fall through to the non accelerated path
+		 */
+		if (IS_ERR_OR_NULL(vlan->fwd_priv))
+			vlan->fwd_priv = NULL;
+		else
+			return 0;
+	}
+
 	err = -EBUSY;
 	if (macvlan_addr_busy(vlan->port, dev->dev_addr))
 		goto out;
@@ -367,6 +389,10 @@ hash_add:
 del_unicast:
 	dev_uc_del(lowerdev, dev->dev_addr);
 out:
+	if (vlan->fwd_priv) {
+		fwd_accel_del_station(lowerdev, vlan->fwd_priv);
+		vlan->fwd_priv = NULL;
+	}
 	return err;
 }
 
@@ -391,6 +417,11 @@ static int macvlan_stop(struct net_device *dev)
 
 hash_del:
 	macvlan_hash_del(vlan, !dev->dismantle);
+	if (vlan->fwd_priv) {
+		fwd_accel_del_station(lowerdev, vlan->fwd_priv);
+		vlan->fwd_priv = NULL;
+	}
+
 	return 0;
 }
 
@@ -801,6 +832,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev,
 		if (err < 0)
 			return err;
 	}
+
 	port = macvlan_port_get_rtnl(lowerdev);
 
 	/* Only 1 macvlan device can be created in passthru mode */
include/linux/if_macvlan.h +1 -0
index ddd33fd..c270285 100644
@@ -61,6 +61,7 @@ struct macvlan_dev {
 	struct hlist_node	hlist;
 	struct macvlan_port	*port;
 	struct net_device	*lowerdev;
+	void			*fwd_priv;
 	struct macvlan_pcpu_stats __percpu *pcpu_stats;
 
 	DECLARE_BITMAP(mc_filter, MACVLAN_MC_FILTER_SZ);
include/linux/netdevice.h +22 -0
index 3de49ac..ea18f07 100644
@@ -1100,6 +1100,27 @@ struct net_device_ops {
 };
 
 /*
+ * Flags to ennumerate hardware acceleration support
+ */
+#define FA_FLG_STA_SUPPORT (1 << 1)
+
+#define fwd_accel_supports(dev, feature) (dev->fwd_ops->flags & feature)
+#define fwd_accel_add_station(pdev, vdev) dev->fwd_ops->fwd_accel_add_station(pdev, vdev)
+#define fwd_accel_del_station(pdev, priv) dev->fwd_ops->fwd_accel_del_station(pdev, priv)
+
+struct forwarding_accel_ops {
+	unsigned int flags;
+
+	/*
+	 * fwd_accel_[add|del]_station must be set if
+	 * FA_FLG_STA_SUPPORT is set
+	 */
+	void*	(*fwd_accel_add_station)(struct net_device *pdev,
+					struct net_device *vdev);
+	void	(*fwd_accel_del_station)(struct net_device *pdev, void *priv);
+};
+
+/*
  *	The DEVICE structure.
  *	Actually, this whole structure is a big mistake.  It mixes I/O
  *	data with strictly "high-level" data, and it has to know about
@@ -1183,6 +1204,7 @@ struct net_device {
 	/* Management operations */
 	const struct net_device_ops *netdev_ops;
 	const struct ethtool_ops *ethtool_ops;
+	const struct forwarding_accel_ops *fwd_ops;
 
 	/* Hardware header description */
 	const struct header_ops *header_ops;
include/linux/skbuff.h +6 -3
index 2ddb48d..0be9152 100644
@@ -426,9 +426,12 @@ struct sk_buff {
 	char			cb[48] __aligned(8);
 
 	unsigned long		_skb_refdst;
-#ifdef CONFIG_XFRM
-	struct	sec_path	*sp;
-#endif
+
+	union {
+		struct	sec_path	*sp;
+		void 			*accel_priv;
+	};
+
 	unsigned int		len,
 				data_len;
 	__u16			mac_len,
net/core/dev.c +3 -0
index 5c713f2..5f99382 100644
@@ -5992,6 +5992,7 @@ struct netdev_queue *dev_ingress_queue_create(struct net_device *dev)
 }
 
 static const struct ethtool_ops default_ethtool_ops;
+static const struct forwarding_accel_ops default_fwd_ops;
 
 void netdev_set_default_ethtool_ops(struct net_device *dev,
 				    const struct ethtool_ops *ops)
@@ -6090,6 +6091,8 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name,
 	dev->group = INIT_NETDEV_GROUP;
 	if (!dev->ethtool_ops)
 		dev->ethtool_ops = &default_ethtool_ops;
+	if (!dev->fwd_ops)
+		dev->fwd_ops = &default_fwd_ops;
 	return dev;
 
 free_all:
Signature · 1 line
1.8.3.1
maintainer
#20 lore raw

Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

From: Neil Horman <[email protected]>
Date: Fri, 4 Oct 2013 16:10:04 -0400

include/linux/skbuff.h struct sk_buff { · line 434

Quoted 13 lines
include/linux/skbuff.h +6 -3
@@ -426,9 +426,12 @@ struct sk_buff {
 	char			cb[48] __aligned(8);
 
 	unsigned long		_skb_refdst;
-#ifdef CONFIG_XFRM
-	struct	sec_path	*sp;
-#endif
+
+	union {
+		struct	sec_path	*sp;
+		void 			*accel_priv;
+	};
+

I'm not %100 sure these two things are really mutually exclusive.

What if bridging ebtables does an input route lookup? That can populate the security path.

Also, why have you not added this to the usual netdev_ops and hw_features?

#21 lore raw

Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

Forgive the poor reply format, Dave, I deleted your email (to fast on the trigger apparently), so I have to reconstruct it.

Quoted 16 lines

include/linux/skbuff.h struct sk_buff { · line 434

Quoted 13 lines
include/linux/skbuff.h +6 -3
@@ -426,9 +426,12 @@ struct sk_buff {
 	char			cb[48] __aligned(8);
 
 	unsigned long		_skb_refdst;
-#ifdef CONFIG_XFRM
-	struct	sec_path	*sp;
-#endif
+
+	union {
+		struct	sec_path	*sp;
+		void 			*accel_priv;
+	};
+

I'm not %100 sure these two things are really mutually exclusive.

Quoted 3 lines

What if bridging ebtables does an input route lookup? That can populate the security path.

You are mostly likely right, thats why this is an RFC, I haven't really thought through that bit fully yet, to be perfectly honest. I wanted a place for a pointer to the accelerated data path data to live, and that looked like a reasonably safe place at the time, but as you point out, its not. I'll need to find a better place for it.

Quoted 2 lines

Also, why have you not added this to the usual netdev_ops and hw_features?

Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a size_t variable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....

Regards
Neil

maintainer
#22 lore raw

Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

From: Neil Horman <[email protected]>
Date: Mon, 7 Oct 2013 17:20:00 -0400

Quoted 8 lines

Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a size_t variable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....

Can these dataplane devices still act like a normal networking port and send and receive packets at the host level?

If yes, that would be an extremely strong argument for netdev_ops.

#23 lore raw

Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

Quoted 12 lines · David Miller

From: Neil Horman <[email protected]>
Date: Mon, 7 Oct 2013 17:20:00 -0400

Quoted 8 lines

Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a size_t variable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....

We can do something like the dcbnl ops and add another pointer off the net device structure and then use the skb->dev field to find the correct set of ops? This seems like the simplest option to me and isolates the ops structure.

Is there some information loss from hanging it off the netdevice structure vs the skb? I can't see any.

Quoted 3 lines

Can these dataplane devices still act like a normal networking port and send and receive packets at the host level?

Yes they act like normal networking ports except for there is a switching component in the hardware. These patches are not looking at virtual or multiple physical functions at the moment.

Quoted 1 line

If yes, that would be an extremely strong argument for netdev_ops.

I agree.

Signature · 1 line
John Fastabend         Intel Corporation
#24 lore raw

Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

Quoted 14 lines · John Fastabend
Quoted 12 lines · David Miller

From: Neil Horman <[email protected]>
Date: Mon, 7 Oct 2013 17:20:00 -0400

Quoted 8 lines

Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a size_t variable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....

Quoted 5 lines

We can do something like the dcbnl ops and add another pointer off the net device structure and then use the skb->dev field to find the correct set of ops? This seems like the simplest option to me and isolates the ops structure.

We certainly could do that, or perhaps, for what we're trying to do here, just using standard netdev_ops is sufficient. I kind of like the separation (like the dcbnl_ops), but like I said, experimenting. I'll try the next version with the accel methods added to the netdev structure for comparison.

Quoted 3 lines

Is there some information loss from hanging it off the netdevice structure vs the skb? I can't see any.

No, not that I'm aware of. The only reason I added it to the skb in this version was that, by doing so, I was able to make dual use of the netdev's standard tx path.

Quoted 8 lines
Quoted 3 lines

Can these dataplane devices still act like a normal networking port and send and receive packets at the host level?

Yes they act like normal networking ports except for there is a switching component in the hardware. These patches are not looking at virtual or multiple physical functions at the moment.

To be clear, as John says, these patches aren't addressing any dataplane acceleration devices beyond the internal switching capabilities of the ixgbe cards. That said, other chips will have varying degrees of capabilities, from simple L2 switching, to full content addressable memories that allow for l2/l3 forwarding, as well as higher level routing functions. Again however, these patches are just to integrate macvlans with johns virtual station interface work.

Quoted 3 lines
Quoted 1 line

If yes, that would be an extremely strong argument for netdev_ops.

I agree.

In this specific case, that may well be the case, yes. I'm not so sure of that for more advanced switching/routing accelerators, but we probably should do what makes sense now, and worry about future bridges when we forward over them (pardon the pun :) ).

Neil

Quoted 5 lines
Signature · 2 lines
-- 
John Fastabend         Intel Corporation
#25 lore raw

PATCH 2/2 ixgbe: enable l2 forwarding acceleration for macvlans

In reply to Neil Horman

Now that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.

CC John Fastabend <[email protected]>
CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |  33 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   4 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h     |  54 ++++
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |  15 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    | 373 +++++++++++++++++------
 5 files changed, 387 insertions(+), 92 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h
drivers/net/ethernet/intel/ixgbe/ixgbe.h +31 -2
index 0ac6b11..e924efa 100644
@@ -219,6 +219,17 @@ enum ixgbe_ring_state_t {
 	__IXGBE_RX_FCOE,
 };
 
+struct ixgbe_fwd_adapter {
+	unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
+	struct net_device *netdev;
+	struct ixgbe_adapter *real_adapter;
+	unsigned int tx_base_queue;
+	unsigned int rx_base_queue;
+	struct net_device_stats net_stats;
+	int pool;
+	bool online;
+};
+
 #define check_for_tx_hang(ring) \
 	test_bit(__IXGBE_TX_DETECT_HANG, &(ring)->state)
 #define set_check_for_tx_hang(ring) \
@@ -236,6 +247,7 @@ struct ixgbe_ring {
 	struct ixgbe_q_vector *q_vector; /* backpointer to host q_vector */
 	struct net_device *netdev;	/* netdev ring belongs to */
 	struct device *dev;		/* device for DMA mapping */
+	struct ixgbe_fwd_adapter *l2_accel_priv;
 	void *desc;			/* descriptor ring memory */
 	union {
 		struct ixgbe_tx_buffer *tx_buffer_info;
@@ -244,6 +256,7 @@ struct ixgbe_ring {
 	unsigned long last_rx_timestamp;
 	unsigned long state;
 	u8 __iomem *tail;
+	struct net_device *vmdq_netdev;
 	dma_addr_t dma;			/* phys. address of descriptor ring */
 	unsigned int size;		/* length in bytes */
 
@@ -288,11 +301,15 @@ enum ixgbe_ring_f_enum {
 };
 
 #define IXGBE_MAX_RSS_INDICES  16
-#define IXGBE_MAX_VMDQ_INDICES 64
+#define IXGBE_MAX_VMDQ_INDICES 32
 #define IXGBE_MAX_FDIR_INDICES 63	/* based on q_vector limit */
 #define IXGBE_MAX_FCOE_INDICES  8
 #define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
 #define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
+#define IXGBE_MAX_L2A_QUEUES 4
+#define IXGBE_MAX_L2A_QUEUES 4
+#define IXGBE_BAD_L2A_QUEUE 3
+
 struct ixgbe_ring_feature {
 	u16 limit;	/* upper limit on feature indices */
 	u16 indices;	/* current value of indices */
@@ -738,6 +755,7 @@ struct ixgbe_adapter {
 #endif /*CONFIG_DEBUG_FS*/
 
 	u8 default_up;
+	unsigned long fwd_bitmask; /* Bitmask indicating in use pools */
 };
 
 struct ixgbe_fdir_filter {
@@ -879,9 +897,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}
 static inline void ixgbe_dbg_init(void) {}
 static inline void ixgbe_dbg_exit(void) {}
 #endif /* CONFIG_DEBUG_FS */
+static inline struct net_device *netdev_ring(const struct ixgbe_ring *ring)
+{
+	return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev;
+}
+
 static inline struct netdev_queue *txring_txq(const struct ixgbe_ring *ring)
 {
-	return netdev_get_tx_queue(ring->netdev, ring->queue_index);
+	return netdev_get_tx_queue(netdev_ring(ring), ring->queue_index);
 }
 
 extern void ixgbe_ptp_init(struct ixgbe_adapter *adapter);
@@ -915,4 +938,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);
 void ixgbe_sriov_reinit(struct ixgbe_adapter *adapter);
 #endif
 
+int ixgbe_get_settings(struct net_device *dev, struct ethtool_cmd *ecmd);
+int ixgbe_write_uc_addr_list(struct net_device *netdev);
+netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb,
+				  struct ixgbe_adapter *adapter,
+				  struct ixgbe_ring *tx_ring);
+void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring);
 #endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
index e8649ab..277af14 100644
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = {
 };
 #define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN
 
-static int ixgbe_get_settings(struct net_device *netdev,
-                              struct ethtool_cmd *ecmd)
+int ixgbe_get_settings(struct net_device *netdev,
+		       struct ethtool_cmd *ecmd)
 {
 	struct ixgbe_adapter *adapter = netdev_priv(netdev);
 	struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h new file +54 -0

54 lines not on this page · raw message

drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
index 90b4e10..e2dd635 100644
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)
 #endif
 
 	/* only proceed if SR-IOV is enabled */
-	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED))
+	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) &&
+	    !(adapter->flags & IXGBE_FLAG_VMDQ_ENABLED))
 		return false;
 
 	/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 
 		/* apply Tx specific ring traits */
 		ring->count = adapter->tx_ring_count;
-		ring->queue_index = txr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				txr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = txr_idx;
 
 		/* assign ring to adapter */
 		adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 #endif /* IXGBE_FCOE */
 		/* apply Rx specific ring traits */
 		ring->count = adapter->rx_ring_count;
-		ring->queue_index = rxr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				rxr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = rxr_idx;
 
 		/* assign ring to adapter */
 		adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +288 -85

616 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#26 lore raw

Re: [RFC PATCH 0/2 v2] net: alternate proposal for using macvlans with forwarding acceleration

In reply to Neil Horman
Quoted 6 lines · Neil Horman

Hey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.

Some change notes:

Quoted 7 lines

* Changes to the fowarding ops structure - Removed the priv_size field, and added a flags field. Removal of the priv_size field was accomplished by just having the add method return a void * and using ERR_PTR and PTR_ERR checks, which also allows us to allocate memory for the acceleration path in the driver, which I like. I'm not super happy still with how I'm using the flags (currenly only used to indicate support for feature sets), but at least we have the flags now, and they can be exposed to user space via iproute2 or ethtool if need be

For the flag why not use the existing feature flag namespace? Adding NETIF_F_HW_L2FW_DOFFLOAD or something equivalent to netdev_features.h and also doing the string updates would get the userspace support for free.

The one downside of a global feature flag approach like this is if the user wants to create some offloaded macvlan devices and some SW only macvlans the control sequence is a bit clumsy. But unless we add a new flag to the macvlan netlink message I'm not sure how to avoid this. Further if you have multiple control applications creating/deleting these macvlans the flag mechanism starts to break down. One app sets the flag the other deletes, you get the idea.

It might be worth adding a netlink flag to change the state of the HW offload. I know this goes against the 'it just gets offloaded' line of thought, but if you make the default to offload then it should be OK.

Quoted 4 lines

* Changes to the Transmit path - Specifically I'm using dev_queue_xmit to send frames now, which I like as it makes the macvlan subject to the lowerdevs qdisc configuration.

This creates perhaps another oddity where on TX the packets will be visible to the lowerdev. Think tcpdump and egress qdisc. But on ingress the packets will be delivered directly to macvlan device. Also I was thinking there might be good reasons to skip the lowerdev qdisc, for performance reasons or QOS setups.

So it might be best the way you had it in the previous revision even if it was not functionally equivalent to the SW path. If you skip the lower dev and submit directly to the hardware then you also don't need a sk_buff change. The driver can do a lookup via the skb->dev field and additional hints from the stack are not needed.

If there is a perceived problem with not having the HW and SW completely equivalent functionally we could always default the feature flag to off. Then enabling the feature would be a single 'ethtool' command. This seems like a nice compromise.

I like where this is going though!

Thanks,
John

Signature · 1 line
John Fastabend         Intel Corporation
#27 lore raw

Re: [RFC PATCH 0/2 v2] net: alternate proposal for using macvlans with forwarding acceleration

Quoted 28 lines · John Fastabend
Quoted 6 lines · Neil Horman

Hey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.

Some change notes:

Quoted 7 lines

* Changes to the fowarding ops structure - Removed the priv_size field, and added a flags field. Removal of the priv_size field was accomplished by just having the add method return a void * and using ERR_PTR and PTR_ERR checks, which also allows us to allocate memory for the acceleration path in the driver, which I like. I'm not super happy still with how I'm using the flags (currenly only used to indicate support for feature sets), but at least we have the flags now, and they can be exposed to user space via iproute2 or ethtool if need be

For the flag why not use the existing feature flag namespace? Adding NETIF_F_HW_L2FW_DOFFLOAD or something equivalent to netdev_features.h and also doing the string updates would get the userspace support for free.

The one downside of a global feature flag approach like this is if the user wants to create some offloaded macvlan devices and some SW only macvlans the control sequence is a bit clumsy. But unless we add a new flag to the macvlan netlink message I'm not sure how to avoid this. Further if you have multiple control applications creating/deleting these macvlans the flag mechanism starts to break down. One app sets the flag the other deletes, you get the idea.

Quoted 5 lines

It might be worth adding a netlink flag to change the state of the HW offload. I know this goes against the 'it just gets offloaded' line of thought, but if you make the default to offload then it should be OK.

Good suggestions, I should have thought to use the global flags previously. I got hung up on the notion that for some reason our feature set should be separated like the ops. I really don't need to do that.

Quoted 11 lines
Quoted 4 lines

* Changes to the Transmit path - Specifically I'm using dev_queue_xmit to send frames now, which I like as it makes the macvlan subject to the lowerdevs qdisc configuration.

This creates perhaps another oddity where on TX the packets will be visible to the lowerdev. Think tcpdump and egress qdisc. But on ingress the packets will be delivered directly to macvlan device. Also I was thinking there might be good reasons to skip the lowerdev qdisc, for performance reasons or QOS setups.

Hmm, is that avoidable by doing an extra check in dev_queue_xmit (i.e. checking to see if accel_data or some other pointer is set)? Or do you think its worth separating the tx path to an accelerated or non accelerated case?

Quoted 6 lines

So it might be best the way you had it in the previous revision even if it was not functionally equivalent to the SW path. If you skip the lower dev and submit directly to the hardware then you also don't need a sk_buff change. The driver can do a lookup via the skb->dev field and additional hints from the stack are not needed.

Ok, fair enough, I'll put it back the way I had it previously.

Quoted 5 lines

If there is a perceived problem with not having the HW and SW completely equivalent functionally we could always default the feature flag to off. Then enabling the feature would be a single 'ethtool' command. This seems like a nice compromise.

Yeah, I can agree with that.

Quoted 1 line

I like where this is going though!

I'm glad you think so! I'll have another version out later this week! Best
Neil

Quoted 7 lines

Thanks,
John

Signature · 1 line
John Fastabend         Intel Corporation
#28 lore raw

v3 cover net: alternate proposal for using macvlans with forwarding acceleration

In reply to Neil Horman

Hey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.

Change notes:

* Moved the feature flag to netdev_features.h. No ethtool option for disabling it yet, but its there now, and seems to fit fairly well. I was actually thinking about your comment John, regarding the clumsiness in allowing sw and hw accel vlans on the same lowerdev, and it just occured to me that we could use the same flag on the macvlan device directly - i.e. if we found that a lowerdev supported acceleration, then call ndo_dfwd_station_add, and, if successfull, set the same feature flag in the macvlan device. Then we could use ethtool to control the enabling/disabling of acceleration at the macvlan device directly. Thoughts?

* Moved the acceleration net device methods back into net_device_ops. Looks pretty good to me there.

* Restored the use of a separate xmit routine so we weren't subject to the lowerdevs queue disciplines. I integrated its use with dev_hard_start_xmit, so we could share the use of the linearization code, etc. Let me know what you think.

Best
Neil

#29 lore raw

PATCH 1/2 net: Add layer 2 hardware acceleration operations for macvlan devices

Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.

CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>
---
 drivers/net/macvlan.c           | 31 +++++++++++++++++++++++++++++++
 include/linux/if_macvlan.h      |  1 +
 include/linux/netdev_features.h |  2 ++
 include/linux/netdevice.h       | 11 ++++++++++-
 include/linux/skbuff.h          |  4 ++--
 net/core/dev.c                  | 18 +++++++++++++-----
 net/core/ethtool.c              |  1 +
 net/sched/sch_generic.c         |  2 +-
 8 files changed, 61 insertions(+), 9 deletions(-)
drivers/net/macvlan.c +31 -0
index 9bf46bd..c5a2718 100644
@@ -297,7 +297,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,
 	int ret;
 	const struct macvlan_dev *vlan = netdev_priv(dev);
 
+	if (vlan->fwd_priv) {
+		skb->dev = vlan->lowerdev;
+		ret = dev_hard_start_xmit(skb, skb->dev, NULL, vlan->fwd_priv);
+					  
+		if (likely(ret == NETDEV_TX_OK))
+			goto update_stats;
+	}
+
 	ret = macvlan_queue_xmit(skb, dev);
+update_stats:
 	if (likely(ret == NET_XMIT_SUCCESS || ret == NET_XMIT_CN)) {
 		struct macvlan_pcpu_stats *pcpu_stats;
 
@@ -347,6 +356,18 @@ static int macvlan_open(struct net_device *dev)
 		goto hash_add;
 	}
 
+	if (lowerdev->features & NETIF_F_HW_L2FW_DOFFLOAD) {
+		vlan->fwd_priv = lowerdev->netdev_ops->ndo_dfwd_add_station(lowerdev, dev);
+		/*
+		 * If we get a NULL pointer back, or if we get an error
+		 * then we should just fall through to the non accelerated path
+		 */
+		if (IS_ERR_OR_NULL(vlan->fwd_priv))
+			vlan->fwd_priv = NULL;
+		else
+			return 0;
+	}
+
 	err = -EBUSY;
 	if (macvlan_addr_busy(vlan->port, dev->dev_addr))
 		goto out;
@@ -367,6 +388,10 @@ hash_add:
 del_unicast:
 	dev_uc_del(lowerdev, dev->dev_addr);
 out:
+	if (vlan->fwd_priv) {
+		lowerdev->netdev_ops->ndo_dfwd_del_station(lowerdev, vlan->fwd_priv);
+		vlan->fwd_priv = NULL;
+	}	
 	return err;
 }
 
@@ -391,6 +416,11 @@ static int macvlan_stop(struct net_device *dev)
 
 hash_del:
 	macvlan_hash_del(vlan, !dev->dismantle);
+	if (vlan->fwd_priv) {
+		lowerdev->netdev_ops->ndo_dfwd_del_station(lowerdev, vlan->fwd_priv);
+		vlan->fwd_priv = NULL;
+	}
+
 	return 0;
 }
 
@@ -801,6 +831,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev,
 		if (err < 0)
 			return err;
 	}
+
 	port = macvlan_port_get_rtnl(lowerdev);
 
 	/* Only 1 macvlan device can be created in passthru mode */
include/linux/if_macvlan.h +1 -0
index ddd33fd..c270285 100644
@@ -61,6 +61,7 @@ struct macvlan_dev {
 	struct hlist_node	hlist;
 	struct macvlan_port	*port;
 	struct net_device	*lowerdev;
+	void			*fwd_priv;
 	struct macvlan_pcpu_stats __percpu *pcpu_stats;
 
 	DECLARE_BITMAP(mc_filter, MACVLAN_MC_FILTER_SZ);
include/linux/netdev_features.h +2 -0
index a2a89a5..9d1ee76 100644
@@ -60,6 +60,7 @@ enum {
 	NETIF_F_HW_VLAN_STAG_TX_BIT,	/* Transmit VLAN STAG HW acceleration */
 	NETIF_F_HW_VLAN_STAG_RX_BIT,	/* Receive VLAN STAG HW acceleration */
 	NETIF_F_HW_VLAN_STAG_FILTER_BIT,/* Receive filtering on VLAN STAGs */
+	NETIF_F_HW_L2FW_DOFFLOAD_BIT,	/* Allow L2 Forwarding in Hardware */
 
 	/*
 	 * Add your fresh new feature above and remember to update
@@ -112,6 +113,7 @@ enum {
 #define NETIF_F_HW_VLAN_STAG_FILTER __NETIF_F(HW_VLAN_STAG_FILTER)
 #define NETIF_F_HW_VLAN_STAG_RX	__NETIF_F(HW_VLAN_STAG_RX)
 #define NETIF_F_HW_VLAN_STAG_TX	__NETIF_F(HW_VLAN_STAG_TX)
+#define NETIF_F_HW_L2FW_DOFFLOAD	__NETIF_F(HW_L2FW_DOFFLOAD)
 
 /* Features valid for ethtool to change */
 /* = all defined minus driver/device-class-related */
include/linux/netdevice.h +10 -1
index 3de49ac..0249179 100644
@@ -1097,6 +1097,13 @@ struct net_device_ops {
 	void			(*ndo_del_vxlan_port)(struct  net_device *dev,
 						      sa_family_t sa_family,
 						      __be16 port);
+
+	void*			(*ndo_dfwd_add_station)(struct net_device *pdev,
+							struct net_device *vdev);
+	void			(*ndo_dfwd_del_station)(struct net_device *pdev, void *priv);
+
+	netdev_tx_t		(*ndo_dfwd_start_xmit) (struct sk_buff *skb,
+							struct net_device *dev, void *priv);
 };
 
 /*
@@ -1183,6 +1190,7 @@ struct net_device {
 	/* Management operations */
 	const struct net_device_ops *netdev_ops;
 	const struct ethtool_ops *ethtool_ops;
+	const struct forwarding_accel_ops *fwd_ops;
 
 	/* Hardware header description */
 	const struct header_ops *header_ops;
@@ -2383,7 +2391,8 @@ extern int		dev_get_phys_port_id(struct net_device *dev,
 					     struct netdev_phys_port_id *ppid);
 extern int		dev_hard_start_xmit(struct sk_buff *skb,
 					    struct net_device *dev,
-					    struct netdev_queue *txq);
+					    struct netdev_queue *txq,
+					    void *accel_priv);
 extern int		dev_forward_skb(struct net_device *dev,
 					struct sk_buff *skb);
 
include/linux/skbuff.h +2 -2
index 2ddb48d..1710fdb 100644
@@ -426,9 +426,9 @@ struct sk_buff {
 	char			cb[48] __aligned(8);
 
 	unsigned long		_skb_refdst;
-#ifdef CONFIG_XFRM
+
 	struct	sec_path	*sp;
-#endif
+
 	unsigned int		len,
 				data_len;
 	__u16			mac_len,
net/core/dev.c +13 -5
index 5c713f2..ecad8c2 100644
@@ -2536,7 +2536,7 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 }
 
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
-			struct netdev_queue *txq)
+			struct netdev_queue *txq, void *accel_priv)
 {
 	const struct net_device_ops *ops = dev->netdev_ops;
 	int rc = NETDEV_TX_OK;
@@ -2602,9 +2602,13 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			dev_queue_xmit_nit(skb, dev);
 
 		skb_len = skb->len;
-		rc = ops->ndo_start_xmit(skb, dev);
+		if (accel_priv)
+			rc = ops->ndo_dfwd_start_xmit(skb, dev, accel_priv);
+		else
+			rc = ops->ndo_start_xmit(skb, dev);
+
 		trace_net_dev_xmit(skb, rc, dev, skb_len);
-		if (rc == NETDEV_TX_OK)
+		if (rc == NETDEV_TX_OK && txq)
 			txq_trans_update(txq);
 		return rc;
 	}
@@ -2620,7 +2624,10 @@ gso:
 			dev_queue_xmit_nit(nskb, dev);
 
 		skb_len = nskb->len;
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (accel_priv)
+			rc = ops->ndo_dfwd_start_xmit(nskb, dev, accel_priv);
+		else
+			rc = ops->ndo_start_xmit(nskb, dev);
 		trace_net_dev_xmit(nskb, rc, dev, skb_len);
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
@@ -2645,6 +2652,7 @@ out_kfree_skb:
 out:
 	return rc;
 }
+EXPORT_SYMBOL_GPL(dev_hard_start_xmit);
 
 static void qdisc_pkt_len_init(struct sk_buff *skb)
 {
@@ -2852,7 +2860,7 @@ int dev_queue_xmit(struct sk_buff *skb)
 
 			if (!netif_xmit_stopped(txq)) {
 				__this_cpu_inc(xmit_recursion);
-				rc = dev_hard_start_xmit(skb, dev, txq);
+				rc = dev_hard_start_xmit(skb, dev, txq, NULL);
 				__this_cpu_dec(xmit_recursion);
 				if (dev_xmit_complete(rc)) {
 					HARD_TX_UNLOCK(dev, txq);
net/core/ethtool.c +1 -0
index 78e9d92..9f0c599b 100644
@@ -94,6 +94,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN]
 	[NETIF_F_LOOPBACK_BIT] =         "loopback",
 	[NETIF_F_RXFCS_BIT] =            "rx-fcs",
 	[NETIF_F_RXALL_BIT] =            "rx-all",
+	[NETIF_F_HW_L2FW_DOFFLOAD_BIT] = "l2-fwd-offload",
 };
 
 static int ethtool_get_features(struct net_device *dev, void __user *useraddr)
net/sched/sch_generic.c +1 -1
index e7121d2..8c44b1b 100644
@@ -126,7 +126,7 @@ int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
 
 	HARD_TX_LOCK(dev, txq, smp_processor_id());
 	if (!netif_xmit_frozen_or_stopped(txq))
-		ret = dev_hard_start_xmit(skb, dev, txq);
+		ret = dev_hard_start_xmit(skb, dev, txq, NULL);
 
 	HARD_TX_UNLOCK(dev, txq);
 
Signature · 1 line
1.8.3.1
#30 lore raw

Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

Quoted 11 lines · Neil Horman

Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.

CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>

---

[...]

include/linux/skbuff.h struct sk_buff { · line 429

Quoted 11 lines
include/linux/skbuff.h +1 -1
index 2ddb48d..1710fdb 100644
@@ -426,9 +426,9 @@ struct sk_buff {
  	char			cb[48] __aligned(8);
 
  	unsigned long		_skb_refdst;
-#ifdef CONFIG_XFRM
+

Is this a hold-over from the previous patches? 'sp' isn't touched anywhere else so put the ifdef/endif back.

Quoted 8 lines
include/linux/skbuff.h +1 -1
  	struct	sec_path	*sp;
-#endif
+
  	unsigned int		len,
  				data_len;
  	__u16			mac_len,
net/core/dev.c +0 -0
index 5c713f2..ecad8c2 100644
Signature · 1 line
John Fastabend         Intel Corporation
#31 lore raw

Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices

Quoted 23 lines · John Fastabend
Quoted 11 lines · Neil Horman

Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.

CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>

---

[...]

Quoted 8 lines
include/linux/skbuff.h +0 -0
index 2ddb48d..1710fdb 100644
@@ -426,9 +426,9 @@ struct sk_buff {
 	char			cb[48] __aligned(8);
 
Quoted 7 lines

include/linux/skbuff.h struct sk_buff { · line 429

Quoted 3 lines
include/linux/skbuff.h +1 -1
 	unsigned long		_skb_refdst;
-#ifdef CONFIG_XFRM
+

Is this a hold-over from the previous patches? 'sp' isn't touched anywhere else so put the ifdef/endif back.

Yeah, my screw up, I wanted to get this out before the weekend and missed that screw up. Sorry.

Neil

Quoted 12 lines
Quoted 8 lines
include/linux/skbuff.h +1 -1
 	struct	sec_path	*sp;
-#endif
+
 	unsigned int		len,
 				data_len;
 	__u16			mac_len,
net/core/dev.c +0 -0
index 5c713f2..ecad8c2 100644
Signature · 1 line
John Fastabend         Intel Corporation
#32 lore raw

PATCH 2/2 ixgbe: enable l2 forwarding acceleration for macvlans

In reply to Neil Horman

Now that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.

CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |  33 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   4 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h     |  54 ++++
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |  15 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    | 388 ++++++++++++++++++-----
 5 files changed, 400 insertions(+), 94 deletions(-)
 create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h
drivers/net/ethernet/intel/ixgbe/ixgbe.h +31 -2
index 0ac6b11..e924efa 100644
@@ -219,6 +219,17 @@ enum ixgbe_ring_state_t {
 	__IXGBE_RX_FCOE,
 };
 
+struct ixgbe_fwd_adapter {
+	unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
+	struct net_device *netdev;
+	struct ixgbe_adapter *real_adapter;
+	unsigned int tx_base_queue;
+	unsigned int rx_base_queue;
+	struct net_device_stats net_stats;
+	int pool;
+	bool online;
+};
+
 #define check_for_tx_hang(ring) \
 	test_bit(__IXGBE_TX_DETECT_HANG, &(ring)->state)
 #define set_check_for_tx_hang(ring) \
@@ -236,6 +247,7 @@ struct ixgbe_ring {
 	struct ixgbe_q_vector *q_vector; /* backpointer to host q_vector */
 	struct net_device *netdev;	/* netdev ring belongs to */
 	struct device *dev;		/* device for DMA mapping */
+	struct ixgbe_fwd_adapter *l2_accel_priv;
 	void *desc;			/* descriptor ring memory */
 	union {
 		struct ixgbe_tx_buffer *tx_buffer_info;
@@ -244,6 +256,7 @@ struct ixgbe_ring {
 	unsigned long last_rx_timestamp;
 	unsigned long state;
 	u8 __iomem *tail;
+	struct net_device *vmdq_netdev;
 	dma_addr_t dma;			/* phys. address of descriptor ring */
 	unsigned int size;		/* length in bytes */
 
@@ -288,11 +301,15 @@ enum ixgbe_ring_f_enum {
 };
 
 #define IXGBE_MAX_RSS_INDICES  16
-#define IXGBE_MAX_VMDQ_INDICES 64
+#define IXGBE_MAX_VMDQ_INDICES 32
 #define IXGBE_MAX_FDIR_INDICES 63	/* based on q_vector limit */
 #define IXGBE_MAX_FCOE_INDICES  8
 #define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
 #define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
+#define IXGBE_MAX_L2A_QUEUES 4
+#define IXGBE_MAX_L2A_QUEUES 4
+#define IXGBE_BAD_L2A_QUEUE 3
+
 struct ixgbe_ring_feature {
 	u16 limit;	/* upper limit on feature indices */
 	u16 indices;	/* current value of indices */
@@ -738,6 +755,7 @@ struct ixgbe_adapter {
 #endif /*CONFIG_DEBUG_FS*/
 
 	u8 default_up;
+	unsigned long fwd_bitmask; /* Bitmask indicating in use pools */
 };
 
 struct ixgbe_fdir_filter {
@@ -879,9 +897,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}
 static inline void ixgbe_dbg_init(void) {}
 static inline void ixgbe_dbg_exit(void) {}
 #endif /* CONFIG_DEBUG_FS */
+static inline struct net_device *netdev_ring(const struct ixgbe_ring *ring)
+{
+	return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev;
+}
+
 static inline struct netdev_queue *txring_txq(const struct ixgbe_ring *ring)
 {
-	return netdev_get_tx_queue(ring->netdev, ring->queue_index);
+	return netdev_get_tx_queue(netdev_ring(ring), ring->queue_index);
 }
 
 extern void ixgbe_ptp_init(struct ixgbe_adapter *adapter);
@@ -915,4 +938,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);
 void ixgbe_sriov_reinit(struct ixgbe_adapter *adapter);
 #endif
 
+int ixgbe_get_settings(struct net_device *dev, struct ethtool_cmd *ecmd);
+int ixgbe_write_uc_addr_list(struct net_device *netdev);
+netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb,
+				  struct ixgbe_adapter *adapter,
+				  struct ixgbe_ring *tx_ring);
+void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring);
 #endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
index e8649ab..277af14 100644
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = {
 };
 #define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN
 
-static int ixgbe_get_settings(struct net_device *netdev,
-                              struct ethtool_cmd *ecmd)
+int ixgbe_get_settings(struct net_device *netdev,
+		       struct ethtool_cmd *ecmd)
 {
 	struct ixgbe_adapter *adapter = netdev_priv(netdev);
 	struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h new file +54 -0

54 lines not on this page · raw message

drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
index 90b4e10..e2dd635 100644
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)
 #endif
 
 	/* only proceed if SR-IOV is enabled */
-	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED))
+	if (!(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED) &&
+	    !(adapter->flags & IXGBE_FLAG_VMDQ_ENABLED))
 		return false;
 
 	/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 
 		/* apply Tx specific ring traits */
 		ring->count = adapter->tx_ring_count;
-		ring->queue_index = txr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				txr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = txr_idx;
 
 		/* assign ring to adapter */
 		adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
 #endif /* IXGBE_FCOE */
 		/* apply Rx specific ring traits */
 		ring->count = adapter->rx_ring_count;
-		ring->queue_index = rxr_idx;
+		if (adapter->num_rx_pools > 1)
+			ring->queue_index =
+				rxr_idx % adapter->num_rx_queues_per_pool;
+		else
+			ring->queue_index = rxr_idx;
 
 		/* assign ring to adapter */
 		adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +301 -87

634 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#33 lore raw

Re: [PATCH 2/2] ixgbe: enable l2 forwarding acceleration for macvlans

Quoted 10 lines · Neil Horman

Now that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.

CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>

---

Neil, I'm fairly sure I can simplify this patch further. I'll be able to take a shot at it Tuesday if you don't mind waiting.

I can resubmit the series then and preserve your signed-off on at least the first patch. Seem reasonable?

.John

Signature · 1 line
John Fastabend         Intel Corporation
#34 lore raw

Re: [PATCH 2/2] ixgbe: enable l2 forwarding acceleration for macvlans

Quoted 12 lines · John Fastabend
Quoted 10 lines · Neil Horman

Now that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.

CC Andy Gospodarek <[email protected]>
CC "David S. Miller" <[email protected]>

---

Quoted 6 lines

Neil, I'm fairly sure I can simplify this patch further. I'll be able to take a shot at it Tuesday if you don't mind waiting.

I can resubmit the series then and preserve your signed-off on at least the first patch. Seem reasonable?

That sounds fine.
Thanks!
Neil

Quoted 5 lines

.John

Signature · 1 line
John Fastabend         Intel Corporation