Series short description
Subsystems (12)
cover Series short description
This patch series implements virtual station interfaces or VSIs. The VSI term comes from the IEEE std 802.1Qbg-2012 specification which should be merged with 802.1Q proper at some point.
3.18 Virtual Station Interface (VSI): An interface to a
virtual station that is attached to a DRP of an edge edge relay.
A DRP (downlink relay port) is the link between an edge relay and a VSI. An edge relay is basically a simple bridge that does not need to support learning, flooding, xSTP, etc. Which matches up well with the ixgbe and I believe other hardware embedded bridge (ebridge) implementations.
This series adds a new VSI rtnl link type. I chose to do this via a link type because it allows us to reuse a lot of the existing infrastructure to bring up a net device and it lets a VSI look similar to a macvlan. The macvlan link type being the software equivalent of a VSI. In many cases I can simply replace the macvlan type with vsi from the ip command line tool and my existing scripts work but use the ebridge instead of SW.
The usage model looks like this,
# ip link add link p3p2 numtxqueues 2 numrxqueues 2 type vsi
# ip link add link p3p2 numtxqueues 2 numrxqueues 2 type vsi
# ip link add link p3p2 numtxqueues 4 numrxqueues 4 type vsi
# ip link set dev vsi0 addr 00:1b:21:69:9f:15
# ip link set dev vsi1 addr 00:1b:21:69:9f:16
# ip link set dev vsi2 addr 00:1b:21:69:9f:17
# ip link set dev vsi0 up
# ip link set dev vsi1 up
# ip link set dev vsi2 up
# ip link show
16: p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 00:1b:21:69:9f:09 brd ff:ff:ff:ff:ff:ff
17: vsi0@p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 00:1b:21:69:9f:15 brd ff:ff:ff:ff:ff:ff
18: vsi1@p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 00:1b:21:69:9f:16 brd ff:ff:ff:ff:ff:ff
19: vsi2@p3p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 00:1b:21:69:9f:17 brd ff:ff:ff:ff:ff:ffAnd creates this topology,
vsi0 vsi1 vsi2
| | |
+------------------------+
| |
| ebridge |
| |
+------------------------+
|
p3p2At this point each vsi# will receive their assigned MAC addresses. Using the 'fdb' interfaces additional L2/L3/tunnel entries could be added to the vsi#. And VSIs can be assigned to net name spaces.
The topology of the ebridge is tracked via the upper and lower dev lists. After we get Veaceslav Falico's work to expose this via sysfs then the topology will be visible. Although it can be learned also via iflink:ifindex to some extent.
PATCH DESCRIPTION:
The first extend rtnl link ops priv_size routine so VSI link types can set the private space for the netdev being created. This is required because device drivers will use this space.
The second patch adds some helper routines to traverse the lower dev list.
The third patch is the interface work to support VSI devices.
And the last patch is an implementation for ixgbe. Notice there are still a few items I need to clean up on this patch before submitting but it is working, without DCB/FCoE, now. And I think I can simplify it some to bring down the line count.
My plan would be to submit this as a real patch after net-next opens but I wanted to see if there was any initial feedback especially related to the VSI link type.
phew... sorry that was a bit long winded.
---
John Fastabend (4):
net: rtnetlink: make priv_size a function for devs with dynamic size
net: Add lower dev list helpers
net: VSI: Add virtual station interface support
ixgbe: Adding VSI support to ixgbe
drivers/infiniband/ulp/ipoib/ipoib_netlink.c | 4 drivers/net/Kconfig | 9 drivers/net/Makefile | 1 drivers/net/bonding/bond_main.c | 4 drivers/net/caif/caif_hsi.c | 4 drivers/net/ethernet/intel/ixgbe/Makefile | 3 drivers/net/ethernet/intel/ixgbe/ixgbe.h | 32 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 15 + drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 307 +++++++++++----- drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c | 428 ++++++++++++++++++++++ drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h | 71 ++++ drivers/net/ifb.c | 4 drivers/net/macvlan.c | 4 drivers/net/nlmon.c | 4 drivers/net/team/team.c | 4 drivers/net/tun.c | 4 drivers/net/veth.c | 4 drivers/net/vsi.c | 124 ++++++ drivers/net/vxlan.c | 4 include/linux/netdevice.h | 35 ++ include/net/rtnetlink.h | 11 + include/uapi/linux/if.h | 1 net/8021q/vlan_netlink.c | 4 net/bridge/br_netlink.c | 4 net/core/dev.c | 25 + net/core/rtnetlink.c | 2 net/ieee802154/6lowpan.c | 4 net/ipv4/ip_gre.c | 6 net/ipv4/ip_tunnel.c | 2 net/ipv4/ip_vti.c | 4 net/ipv4/ipip.c | 4 net/ipv6/ip6_tunnel.c | 4 net/ipv6/sit.c | 4 34 files changed, 1028 insertions(+), 116 deletions(-) create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h create mode 100644 drivers/net/vsi.c
Signature · 1 line
Signature
RFC PATCH 1/4 net: rtnetlink: make priv_size a function for devs with dynamic size
The priv_size rtnl_link_op today is a fixed size_t value.
For upcoming VSI support via rtnl_link_ops it is useful to allow this size to be configurable. This patch converts the existing static definition into a function that returns the size_t value.
To make this conversion as easy as possible the patch uses a new macro RTNL_LINK_OPS_PRIV_SIZE which existing users can call to generate a function equivalent to the previous static value.
RFC NOTE: I'm not entirely sure the macro makes the code easier
to read or just obfuscates what is really happening. Any opinions?
--- drivers/infiniband/ulp/ipoib/ipoib_netlink.c | 4 +++- drivers/net/bonding/bond_main.c | 4 +++- drivers/net/caif/caif_hsi.c | 4 +++- drivers/net/ifb.c | 4 +++- drivers/net/macvlan.c | 4 +++- drivers/net/nlmon.c | 4 +++- drivers/net/team/team.c | 4 +++- drivers/net/tun.c | 4 +++- drivers/net/veth.c | 4 +++- drivers/net/vxlan.c | 4 +++- include/net/rtnetlink.h | 11 ++++++++++- net/8021q/vlan_netlink.c | 4 +++- net/bridge/br_netlink.c | 4 +++- net/core/rtnetlink.c | 2 +- net/ieee802154/6lowpan.c | 4 +++- net/ipv4/ip_gre.c | 6 ++++-- net/ipv4/ip_tunnel.c | 2 +- net/ipv4/ip_vti.c | 4 +++- net/ipv4/ipip.c | 4 +++- net/ipv6/ip6_tunnel.c | 4 +++- net/ipv6/sit.c | 4 +++- 21 files changed, 67 insertions(+), 22 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_netlink.c +3 -1
@@ -155,11 +155,13 @@ static size_t ipoib_get_size(const struct net_device *dev)nla_total_size (2 );/* IFLA_IPOIB_UMCAST */ } +RTNL_LINK_OPS_PRIV_SIZE (ipoib, ipoib_dev_priv); +static struct rtnl_link_ops ipoib_link_ops__read_mostly = { .kind ="ipoib" , .maxtype =IFLA_IPOIB_MAX , .policy = ipoib_policy, - .priv_size =sizeof (struct ipoib_dev_priv ), + .priv_size = ipoib_priv_size, .setup = ipoib_setup, .newlink = ipoib_new_child_link, .changelink = ipoib_changelink,
drivers/net/bonding/bond_main.c +3 -1
@@ -4503,9 +4503,11 @@ static unsigned int bond_get_num_tx_queues(void)return tx_queues; } +RTNL_LINK_OPS_PRIV_SIZE (bond, bonding); +static struct rtnl_link_ops bond_link_ops__read_mostly = { .kind ="bond" , - .priv_size =sizeof (struct bonding ), + .priv_size = bond_priv_size, .setup = bond_setup, .validate = bond_validate, .get_num_tx_queues = bond_get_num_tx_queues,
drivers/net/caif/caif_hsi.c +3 -1
@@ -1445,9 +1445,11 @@ err:return -ENODEV ; } +RTNL_LINK_OPS_PRIV_SIZE (caif, cfhsi); +static struct rtnl_link_ops caif_hsi_link_ops__read_mostly = { .kind ="cfhsi" , - .priv_size =sizeof (struct cfhsi ), + .priv_size = caif_priv_size, .setup = cfhsi_setup, .maxtype = __IFLA_CAIF_HSI_MAX, .policy = caif_hsi_policy,
drivers/net/ifb.c +3 -1
@@ -251,9 +251,11 @@ static int ifb_validate(struct nlattr *tb[], struct nlattr *data[])return 0 ; } +RTNL_LINK_OPS_PRIV_SIZE (ifb, ifb_private); +static struct rtnl_link_ops ifb_link_ops__read_mostly = { .kind ="ifb" , - .priv_size =sizeof (struct ifb_private ), + .priv_size = ifb_priv_size, .setup = ifb_setup, .validate = ifb_validate, };
drivers/net/macvlan.c +3 -1
@@ -934,10 +934,12 @@ static const struct nla_policy macvlan_policy[IFLA_MACVLAN_MAX + 1] = { [IFLA_MACVLAN_FLAGS ] = { .type =NLA_U16 }, }; +RTNL_LINK_OPS_PRIV_SIZE (macvlan ,macvlan_dev ); +int macvlan_link_register (struct rtnl_link_ops *ops) {/* common fields */ - ops->priv_size =sizeof (struct macvlan_dev ); + ops->priv_size = macvlan_priv_size, ops->validate = macvlan_validate; ops->maxtype =IFLA_MACVLAN_MAX ; ops->policy = macvlan_policy;
drivers/net/nlmon.c +3 -1
@@ -154,9 +154,11 @@ static int nlmon_validate(struct nlattr *tb[], struct nlattr *data[])return 0 ; } +RTNL_LINK_OPS_PRIV_SIZE (nlmon, nlmon); +static struct rtnl_link_ops nlmon_link_ops__read_mostly = { .kind ="nlmon" , - .priv_size =sizeof (struct nlmon ), + .priv_size = nlmon_priv_size, .setup = nlmon_setup, .validate = nlmon_validate, };
drivers/net/team/team.c +3 -1
@@ -2068,9 +2068,11 @@ static unsigned int team_get_num_rx_queues(void)return TEAM_DEFAULT_NUM_RX_QUEUES ; } +RTNL_LINK_OPS_PRIV_SIZE (team, team); +static struct rtnl_link_ops team_link_ops__read_mostly = { .kind =DRV_NAME , - .priv_size =sizeof (struct team ), + .priv_size = team_priv_size, .setup = team_setup, .newlink = team_newlink, .validate = team_validate,
drivers/net/tun.c +3 -1
@@ -1380,9 +1380,11 @@ static int tun_validate(struct nlattr *tb[], struct nlattr *data[])return -EINVAL ; } +RTNL_LINK_OPS_PRIV_SIZE (tun, tun_struct); +static struct rtnl_link_ops tun_link_ops__read_mostly = { .kind =DRV_NAME , - .priv_size =sizeof (struct tun_struct ), + .priv_size = tun_priv_size, .setup = tun_setup, .validate = tun_validate, };
drivers/net/veth.c +3 -1
@@ -434,9 +434,11 @@ static const struct nla_policy veth_policy[VETH_INFO_MAX + 1] = { [VETH_INFO_PEER ] = { .len =sizeof (struct ifinfomsg ) }, }; +RTNL_LINK_OPS_PRIV_SIZE (veth ,veth_priv ); +static struct rtnl_link_ops veth_link_ops = { .kind =DRV_NAME , - .priv_size =sizeof (struct veth_priv ), + .priv_size = veth_priv_size, .setup = veth_setup, .validate = veth_validate, .newlink = veth_newlink,
drivers/net/vxlan.c +3 -1
@@ -2616,11 +2616,13 @@ nla_put_failure:return -EMSGSIZE ; } +RTNL_LINK_OPS_PRIV_SIZE (vxlan, vxlan_dev); +static struct rtnl_link_ops vxlan_link_ops__read_mostly = { .kind ="vxlan" , .maxtype =IFLA_VXLAN_MAX , .policy = vxlan_policy, - .priv_size =sizeof (struct vxlan_dev ), + .priv_size = vxlan_priv_size, .setup = vxlan_setup, .validate = vxlan_validate, .newlink = vxlan_newlink,
include/net/rtnetlink.h +10 -1
@@ -4,6 +4,14 @@#include <linux/rtnetlink.h> #include <net/netlink.h> +/* generate a get_priv function for simple field */ +#define RTNL_LINK_OPS_PRIV_SIZE (device, size_struct) \ +static size_t device##_priv_size(struct net *src_net, \ + struct nlattr *tb[]) \ +{ \ + return sizeof(struct size_struct); \ +} +typedef int (*rtnl_doit_func )(struct sk_buff *,struct nlmsghdr *);typedef int (*rtnl_dumpit_func )(struct sk_buff *,struct netlink_callback *);typedef u16 (*rtnl_calcit_func )(struct sk_buff *,struct nlmsghdr *);
@@ -54,7 +62,8 @@ struct rtnl_link_ops {const char *kind; -size_t priv_size; +size_t (*priv_size)(struct net *src_net, +struct nlattr *tb[]);void (*setup)(struct net_device *dev);int maxtype;
net/8021q/vlan_netlink.c +3 -1
@@ -238,11 +238,13 @@ nla_put_failure:return -EMSGSIZE ; } +RTNL_LINK_OPS_PRIV_SIZE (vlan, vlan_dev_priv); +struct rtnl_link_ops vlan_link_ops__read_mostly = { .kind ="vlan" , .maxtype =IFLA_VLAN_MAX , .policy = vlan_policy, - .priv_size =sizeof (struct vlan_dev_priv ), + .priv_size = vlan_priv_size, .setup = vlan_setup, .validate = vlan_validate, .newlink = vlan_newlink,
net/bridge/br_netlink.c +3 -1
@@ -469,9 +469,11 @@ static struct rtnl_af_ops br_af_ops = { .get_link_af_size = br_get_link_af_size, }; +RTNL_LINK_OPS_PRIV_SIZE (br, net_bridge); +struct rtnl_link_ops br_link_ops__read_mostly = { .kind ="bridge" , - .priv_size =sizeof (struct net_bridge ), + .priv_size = br_priv_size, .setup = br_dev_setup, .validate = br_validate, .dellink = br_dev_delete,
net/core/rtnetlink.c +1 -1
@@ -1673,7 +1673,7 @@ struct net_device *rtnl_create_link(struct net *net, num_rx_queues = ops->get_num_rx_queues (); err = -ENOMEM ; - dev =alloc_netdev_mqs (ops->priv_size, ifname, ops->setup, + dev =alloc_netdev_mqs (ops->priv_size (net, tb), ifname, ops->setup, num_tx_queues, num_rx_queues);if (!dev) goto err;
net/ieee802154/6lowpan.c +3 -1
@@ -1420,9 +1420,11 @@ static void lowpan_dellink(struct net_device *dev, struct list_head *head)dev_put (real_dev); } +RTNL_LINK_OPS_PRIV_SIZE (lowpan, lowpan_dev_info); +static struct rtnl_link_ops lowpan_link_ops__read_mostly = { .kind ="lowpan" , - .priv_size =sizeof (struct lowpan_dev_info ), + .priv_size = lowpan_priv_size, .setup = lowpan_setup, .newlink = lowpan_newlink, .dellink = lowpan_dellink,
net/ipv4/ip_gre.c +4 -2
@@ -731,11 +731,13 @@ static const struct nla_policy ipgre_policy[IFLA_GRE_MAX + 1] = { [IFLA_GRE_PMTUDISC ] = { .type =NLA_U8 }, }; +RTNL_LINK_OPS_PRIV_SIZE (ipgre ,ip_tunnel ); +static struct rtnl_link_ops ipgre_link_ops__read_mostly = { .kind ="gre" , .maxtype =IFLA_GRE_MAX , .policy = ipgre_policy, - .priv_size =sizeof (struct ip_tunnel ), + .priv_size = ipgre_priv_size, .setup = ipgre_tunnel_setup, .validate = ipgre_tunnel_validate, .newlink = ipgre_newlink,
@@ -749,7 +751,7 @@ static struct rtnl_link_ops ipgre_tap_ops __read_mostly = { .kind = "gretap", .maxtype = IFLA_GRE_MAX, .policy = ipgre_policy, - .priv_size = sizeof(struct ip_tunnel), + .priv_size = ipgre_priv_size, .setup = ipgre_tap_setup, .validate = ipgre_tap_validate, .newlink = ipgre_newlink,
net/ipv4/ip_tunnel.c +1 -1
@@ -293,7 +293,7 @@ static struct net_device *__ip_tunnel_create(struct net *net, }ASSERT_RTNL (); - dev =alloc_netdev (ops->priv_size, name, ops->setup); + dev =alloc_netdev (ops->priv_size (net,NULL ), name, ops->setup);if (!dev) { err = -ENOMEM ; goto failed;
net/ipv4/ip_vti.c +3 -1
@@ -416,11 +416,13 @@ static const struct nla_policy vti_policy[IFLA_VTI_MAX + 1] = { [IFLA_VTI_REMOTE] = { .len = FIELD_SIZEOF(struct iphdr, daddr) }, }; +RTNL_LINK_OPS_PRIV_SIZE (vti ,ip_tunnel ); +static struct rtnl_link_ops vti_link_ops__read_mostly = { .kind ="vti" , .maxtype =IFLA_VTI_MAX , .policy = vti_policy, - .priv_size =sizeof (struct ip_tunnel ), + .priv_size = vti_priv_size, .setup = vti_tunnel_setup, .validate = vti_tunnel_validate, .newlink = vti_newlink,
net/ipv4/ipip.c +3 -1
@@ -408,11 +408,13 @@ static const struct nla_policy ipip_policy[IFLA_IPTUN_MAX + 1] = { [IFLA_IPTUN_PMTUDISC ] = { .type =NLA_U8 }, }; +RTNL_LINK_OPS_PRIV_SIZE (ipip ,ip_tunnel ); +static struct rtnl_link_ops ipip_link_ops__read_mostly = { .kind ="ipip" , .maxtype =IFLA_IPTUN_MAX , .policy = ipip_policy, - .priv_size =sizeof (struct ip_tunnel ), + .priv_size = ipip_priv_size, .setup = ipip_tunnel_setup, .newlink = ipip_newlink, .changelink = ipip_changelink,
net/ipv6/ip6_tunnel.c +3 -1
@@ -1682,11 +1682,13 @@ static const struct nla_policy ip6_tnl_policy[IFLA_IPTUN_MAX + 1] = { [IFLA_IPTUN_PROTO ] = { .type =NLA_U8 }, }; +RTNL_LINK_OPS_PRIV_SIZE (ip6_tnl ,ip6_tnl ); +static struct rtnl_link_ops ip6_link_ops__read_mostly = { .kind ="ip6tnl" , .maxtype =IFLA_IPTUN_MAX , .policy = ip6_tnl_policy, - .priv_size =sizeof (struct ip6_tnl ), + .priv_size = ip6_tnl_priv_size, .setup = ip6_tnl_dev_setup, .validate = ip6_tnl_validate, .newlink = ip6_tnl_newlink,
net/ipv6/sit.c +3 -1
@@ -1540,11 +1540,13 @@ static const struct nla_policy ipip6_policy[IFLA_IPTUN_MAX + 1] = {#endif }; +RTNL_LINK_OPS_PRIV_SIZE (ipip6, ip_tunnel); +static struct rtnl_link_ops sit_link_ops__read_mostly = { .kind ="sit" , .maxtype =IFLA_IPTUN_MAX , .policy = ipip6_policy, - .priv_size =sizeof (struct ip_tunnel ), + .priv_size = ipip6_priv_size, .setup = ipip6_tunnel_setup, .validate = ipip6_validate, .newlink = ipip6_newlink,
RFC PATCH 2/4 net: Add lower dev list helpers
In reply to John FastabendThis patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.
VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.
--- include/linux/netdevice.h | 8 ++++++++ net/core/dev.c | 25 +++++++++++++++++++++++++ 2 files changed, 33 insertions(+)
include/linux/netdevice.h +8 -0
@@ -2813,6 +2813,8 @@ extern int bpf_jit_enable;extern bool netdev_has_upper_dev (struct net_device *dev,struct net_device *upper_dev);extern bool netdev_has_any_upper_dev (struct net_device *dev); +extern struct net_device *netdev_lower_get_next_dev_rcu (struct net_device *dev, +struct list_head **iter);extern struct net_device *netdev_upper_get_next_dev_rcu (struct net_device *dev,struct list_head **iter);
@@ -2823,6 +2825,12 @@ extern struct net_device *netdev_upper_get_next_dev_rcu(struct net_device *dev,upper ; \ upper = netdev_upper_get_next_dev_rcu(dev, &(iter))) +#define netdev_for_each_lower_dev_rcu (dev, lower, iter) \ + for (iter = &(dev)->lower_dev_list, \ + lower = netdev_lower_get_next_dev_rcu(dev, &(iter)); \ + lower; \ + lower = netdev_lower_get_next_dev_rcu(dev, &(iter))) +extern struct net_device *netdev_master_upper_dev_get (struct net_device *dev);extern struct net_device *netdev_master_upper_dev_get_rcu (struct net_device *dev); extern int netdev_upper_dev_link(struct net_device *dev,
net/core/dev.c +25 -0
@@ -4468,6 +4468,31 @@ struct net_device *netdev_master_upper_dev_get(struct net_device *dev) }EXPORT_SYMBOL (netdev_master_upper_dev_get); +/* netdev_lower_get_next_dev_rcu - Get the next dev from lower list +* @dev: device +* @iter: list_head ** of the current position +* +* Gets the next device from the dev's lower list, starting from iter +* position. The caller must hold RTNL/RCU read lock. +*/ +struct net_device *netdev_lower_get_next_dev_rcu (struct net_device *dev, +struct list_head **iter) +{ +struct netdev_adjacent *lower; + +WARN_ON_ONCE (!rcu_read_lock_held ()); + + lower = list_entry_rcu((*iter)->next, struct netdev_adjacent, list); + +if (&lower->list == &dev->lower_dev_list) +return NULL ; + + *iter = &lower->list; + +return lower->dev; +} +EXPORT_SYMBOL (netdev_lower_get_next_dev_rcu); +/* netdev_upper_get_next_dev_rcu - Get the next dev from upper list * @dev: device * @iter: list_head ** of the current position
Re: [RFC PATCH 2/4] net: Add lower dev list helpers
Quoted 5 lines · John Fastabend
This patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.
VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.
Just as a note (as I have quite no idea how ixgbe works) - you are aware that the upper/lower lists currently include *all* upper/lower devices, and not only the first-connected ones?
I've seen that you usually verify it, however not always, just a heads-up - sorry if misread.
I'm also currently trying to get the new patchset included - which would split the first-tier connected devices from all the 'other' (as in - lower of a lower and upper of an upper) devices, so that way, I think, it would be easier/faster for you to use it. It also has a ->private field, easily accessible, which you could have used instead of/in conjunction with/for struct ixgbe_vsi_adapter.
[PATCH v2 net-next 00/27] bonding: use neighbours instead of own lists
Here's the patchset.
Hope that helps.
Quoted 70 lines
Signed-off-by John Fastabend <[email protected]>---
include/linux/netdevice.h| 8 ++++++++net/core/dev.c| 25 +++++++++++++++++++++++++2 files changed, 33 insertions(+)include/linux/netdevice.h +8
-0@@ -2813,6 +2813,8 @@ extern int bpf_jit_enable;extern bool netdev_has_upper_dev (struct net_device *dev,struct net_device *upper_dev);extern bool netdev_has_any_upper_dev (struct net_device *dev); +extern struct net_device *netdev_lower_get_next_dev_rcu (struct net_device *dev, +struct list_head **iter);extern struct net_device *netdev_upper_get_next_dev_rcu (struct net_device *dev,struct list_head **iter);@@ -2823,6 +2825,12 @@ extern struct net_device *netdev_upper_get_next_dev_rcu(struct net_device *dev,upper ; \ upper = netdev_upper_get_next_dev_rcu(dev, &(iter))) +#define netdev_for_each_lower_dev_rcu (dev, lower, iter) \ + for (iter = &(dev)->lower_dev_list, \ + lower = netdev_lower_get_next_dev_rcu(dev, &(iter)); \ + lower; \ + lower = netdev_lower_get_next_dev_rcu(dev, &(iter))) +extern struct net_device *netdev_master_upper_dev_get (struct net_device *dev);extern struct net_device *netdev_master_upper_dev_get_rcu (struct net_device *dev); extern int netdev_upper_dev_link(struct net_device *dev,net/core/dev.c +25
-0@@ -4468,6 +4468,31 @@ struct net_device *netdev_master_upper_dev_get(struct net_device *dev) }EXPORT_SYMBOL (netdev_master_upper_dev_get); +/* netdev_lower_get_next_dev_rcu - Get the next dev from lower list +* @dev: device +* @iter: list_head ** of the current position +* +* Gets the next device from the dev's lower list, starting from iter +* position. The caller must hold RTNL/RCU read lock. +*/ +struct net_device *netdev_lower_get_next_dev_rcu (struct net_device *dev, +struct list_head **iter) +{ +struct netdev_adjacent *lower; + +WARN_ON_ONCE (!rcu_read_lock_held ()); + + lower = list_entry_rcu((*iter)->next, struct netdev_adjacent, list); + +if (&lower->list == &dev->lower_dev_list) +return NULL ; + + *iter = &lower->list; + +return lower->dev; +} +EXPORT_SYMBOL (netdev_lower_get_next_dev_rcu); +/* netdev_upper_get_next_dev_rcu - Get the next dev from upper list * @dev: device * @iter: list_head ** of the current position
Re: [RFC PATCH 2/4] net: Add lower dev list helpers
Quoted 10 lines · Veaceslav Falico
Quoted 5 lines · John Fastabend
This patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.
VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.
Just as a note (as I have quite no idea how ixgbe works) - you are aware that the upper/lower lists currently include *all* upper/lower devices, and not only the first-connected ones?
Yep, the netif_is_vsi_port() should hopefully catch all the cases.
Quoted 3 lines
I've seen that you usually verify it, however not always, just a heads-up - sorry if misread.
I'll audit it again, maybe I missed a case.
Quoted 7 lines
I'm also currently trying to get the new patchset included - which would split the first-tier connected devices from all the 'other' (as in - lower of a lower and upper of an upper) devices, so that way, I think, it would be easier/faster for you to use it. It also has a ->private field, easily accessible, which you could have used instead of/in conjunction with/for struct
ixgbe_vsi_adapter.
Yeah I saw the patch set I'll use the improved lists once I see the series gets applied. Although in my case the list walking only occurs in the configuration path so its not critical.
Also the net device structure already has private space to hang the device dependent structure off of, so I'm not sure having another private field helps here.
Thanks,
John
Signature · 1 line
John Fastabend Intel Corporation
Re: [RFC PATCH 2/4] net: Add lower dev list helpers
Quoted 12 lines · John Fastabend
Quoted 10 lines · Veaceslav Falico
Quoted 5 lines · John Fastabend
This patch adds helpers to traverse the lower dev lists, these helpers match the upper dev list implementation.
VSI implementers may use these to track a list of connected netdevs. This is easier then having drivers do their own accounting.
Just as a note (as I have quite no idea how ixgbe works) - you are aware that the upper/lower lists currently include *all* upper/lower devices, and not only the first-connected ones?
Quoted 7 lines
Yep, the
netif_is_vsi_port()should hopefully catch all the cases.Quoted 3 lines
I've seen that you usually verify it, however not always, just a heads-up - sorry if misread.
I'll audit it again, maybe I missed a case.
It was about lower devices, and I think I've just really misread it - indeed they can be only 'your' devices.
Quoted 9 lines
Quoted 8 lines
I'm also currently trying to get the new patchset included - which would split the first-tier connected devices from all the 'other' (as in - lower of a lower and upper of an upper) devices, so that way, I think, it would be easier/faster for you to use it. It also has a ->private field, easily accessible, which you could have used instead of/in conjunction with/for struct
ixgbe_vsi_adapter.
Quoted 7 lines
Yeah I saw the patch set I'll use the improved lists once I see the series gets applied. Although in my case the list walking only occurs in the configuration path so its not critical.
Also the net device structure already has private space to hang the device dependent structure off of, so I'm not sure having another private field helps here.
Yep, it should/can be used when linking devices which might already be using their private space for their own structures, which is not your case, I suppose.
Quoted 6 lines
Thanks,
JohnSignature · 1 line
John Fastabend Intel Corporation
RFC PATCH 3/4 net: VSI: Add virtual station interface support
In reply to John FastabendThis patch adds support for a new device type VSI (virtual station interface) this device type exposes additional net devices complete with queues and a MAC/VLAN pair to the host OS that are logically stacked on top of a switching/routing component with the physical link acting as the downlink to the peer switch.
The hardware on receive path will forward packets to the new VSI net device using the forwarding database (FDB) already exposed via the ndo ops ndo_fdb_{add|del|dump}. On transmit the hardware may use either a VEB or VEPA. In the VEB case traffic may be "switched" between VSI net devices by the hardware and in VEPA case all traffic is sent to the adjacent switch. The hardware _should_ expose this functionality via the ndo_bridge_{set|get}link ndo operations.
This net device should be functionally analogous to an offloaded macvlan device with the ebridge component offloaded into hardware.
Also notice that for now the ixgbe implementation accompanying this patch set only supports L2 forwarding the fdb interfaces could push L3/L4 forwarding to the hardware for more advanced usages including vxlan and other tunnel schemes.
--- drivers/net/Kconfig | 9 +++ drivers/net/Makefile | 1 drivers/net/vsi.c | 124 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/netdevice.h | 27 ++++++++++ include/uapi/linux/if.h | 1 5 files changed, 162 insertions(+) create mode 100644 drivers/net/vsi.c
drivers/net/Kconfig +9 -0
@@ -362,4 +362,13 @@ config VMXNET3 source "drivers/net/hyperv/Kconfig" +config VSI + tristate "Virtual Station Interfaces (VSI)" + help + This supports chip sets with embedded switching components + and allows creating additional net devices that are + logically slaves of a master net device typically the net + device associated with the physical function. For these + child devices switching occurs in the hardware component. + endif # NETDEVICES
drivers/net/Makefile +1 -0
@@ -23,6 +23,7 @@ obj-$(CONFIG_VETH) += veth.o obj-$(CONFIG_VIRTIO_NET) += virtio_net.o obj-$(CONFIG_VXLAN) += vxlan.o obj-$(CONFIG_NLMON) += nlmon.o +obj-$(CONFIG_VSI) += vsi.o # # Networking Drivers
drivers/net/vsi.c new file +124 -0
124 lines not on this page · raw message
include/linux/netdevice.h +27 -0
@@ -961,6 +961,24 @@ struct netdev_phys_port_id {* Called by vxlan to notify the driver about a UDP port and socket * address family that vxlan is not listening to anymore. The operation * is protected by the vxlan_net->sock_lock. +* +* int (*ndo_vsi_add)(struct net_device *lower, struct net_device *dev) +* Called by the virtual station interface (VSI) link type to add a new +* net device 'dev' to an embedded switch where the embedded switch +* management net device is identified by 'lower'. This should return +* 0 on success or may return negative error codes. Error codes should +* be used here to signify resource constraints, unsupportable attributes, +* or any other condition which caused the creation to fail. +* void (*ndo_vsi_del)(struct net_device *dev) +* Called by the virtual station interface (VSI) link type to remove the +* net device 'dev' from an embedded switch. Drivers may not fail this +* command. +* size_t (*ndo_vsi_size)(struct net_device *dev) +* Called by the virtual station interface (VSI) link type to add the +* required private size to a VSI interface that is being created. If +* this routine is not implemented size_t 0 is used. The 'dev' argument +* indicates the embedded switch management interface where the new +* net devices is being attached. */ struct net_device_ops {int (*ndo_init)(struct net_device *dev);
@@ -1097,6 +1115,10 @@ struct net_device_ops {void (*ndo_del_vxlan_port)(struct net_device *dev,sa_family_t sa_family,__u16 port); +int (*ndo_vsi_add)(struct net_device *lower, +struct net_device *dev); +void (*ndo_vsi_del)(struct net_device *dev); +size_t (*ndo_vsi_size)(struct net_device *dev); };/*
@@ -2967,6 +2989,11 @@ static inline bool netif_supports_nofcs(struct net_device *dev)return dev->priv_flags &IFF_SUPP_NOFCS ; } +static inline bool netif_is_vsi_port (struct net_device *dev) +{ +return dev->priv_flags &IFF_VSI_PORT ; +} +extern struct pernet_operations __net_initdata loopback_net_ops;/* Logging, debugging and troubleshooting/diagnostic helpers. */
include/uapi/linux/if.h +1 -0
@@ -83,6 +83,7 @@#define IFF_SUPP_NOFCS 0x80000/* device supports sending custom FCS */ #define IFF_LIVE_ADDR_CHANGE 0x100000/* device supports hardware address * change when it's running */ +#define IFF_VSI_PORT 0x200000/* Virtual Station Interface port */ #define IF_GET_IFACE 0x0001/* for querying only */
Re: [RFC PATCH 3/4] net: VSI: Add virtual station interface support
John-
Sorry for not copying your orgional patch into your email, but I apparently
erased this email prior to fully reading it. At any rate, as we discussed the other day, I think this idea is great, but it would benefit from being implemented using macvlans rather than a new link type. If we can set a hardware flag in the underlying driver to indicate support for hardware forwarding, we can skip the macvlan soft forwarding, and just transmit directly. It should save us needing to create a whole new link type that is so simmilar to one we already have.
I'm back home now, so I can start looking at this if you like on monday.
Best
Neil
Re: [RFC PATCH 3/4] net: VSI: Add virtual station interface support
Quoted 11 lines · Neil Horman
John-
Sorry for not copying your orgional patch into your email, but I apparently
erased this email prior to fully reading it. At any rate, as we discussed the other day, I think this idea is great, but it would benefit from being implemented using macvlans rather than a new link type. If we can set a hardware flag in the underlying driver to indicate support for hardware forwarding, we can skip the macvlan soft forwarding, and just transmit directly. It should save us needing to create a whole new link type that is so simmilar to one we already have.
Quoted 6 lines
I'm back home now, so I can start looking at this if you like on monday.
Best
Neil
Yes definitely take a look making it an extension (flag) to macvlan would be more useful if it can work. The one question I have is would we need to clear the flag when a feature is attached to the PF that can not be done in hardware. For example attaching an ingress qdisc, ebtables, or starting up tcpdump are two that come to mind.
.John
Sorry for the duplication Neil, I sent this mail from a broken email client just a minute ago and vger dropped it.
Re: [RFC PATCH 3/4] net: VSI: Add virtual station interface support
Quoted 19 lines · John Fastabend
Quoted 11 lines · Neil Horman
John-
Sorry for not copying your orgional patch into your email, but I apparently
erased this email prior to fully reading it. At any rate, as we discussed the other day, I think this idea is great, but it would benefit from being implemented using macvlans rather than a new link type. If we can set a hardware flag in the underlying driver to indicate support for hardware forwarding, we can skip the macvlan soft forwarding, and just transmit directly. It should save us needing to create a whole new link type that is so simmilar to one we already have.Quoted 6 lines
I'm back home now, so I can start looking at this if you like on monday.
Best
Neil
Quoted 6 lines
Yes definitely take a look making it an extension (flag) to macvlan would be more useful if it can work. The one question I have is would we need to clear the flag when a feature is attached to the PF that can not be done in hardware. For example attaching an ingress qdisc, ebtables, or starting up tcpdump are two that come to mind.
I think that it probably would, although I expect that this would be the case independent of the implementations (virtual station interfaces vs. macvlans). The only other option I see is to disallow those features that can't be done in hardware, but I think thats unworkable long term.
Neil
Quoted 9 lines
.John
Sorry for the duplication Neil, I sent this mail from a broken email client just a minute ago and vger dropped it.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
RFC PATCH 4/4 ixgbe: Adding VSI support to ixgbe
In reply to John FastabendThis adds initial base support for VSI interfaces to the ixgbe driver.
This supports up to 32 VSIs per physical function with each VSI supporting up to 4 TX/RX queue pairs. The hardware has support for modes using 2 queue pairs that enable up to 64 VSIs but they are not enabled here. VSIs can be instantiated with 1,2, or 4 queues unfortunately due to hardware restrictions related to RSS masking across queues 3 TX/RX queues is not supported. If a user inputs a queue parameter of 3 for either TX or RX the vsi_add will fail with EINVAL and a message will be printed to dmesg explaining the error. EBUSY is returned if more than 32 VSIs are added.
The driver uses a bitmask added to the ixgbe_adapter struct to track VSIs. VSIs spawn net devices that can be independently managed via their own ndo and ethtool ops.
Currently, MAC addresses must be added manually via 'ip link set' or similar configuration (netlink) commands before bringing the spawned devices online. I could easily generate random MAC addresses if that is prefered although not required in my use case and seems a bit arbitrary. I expect most users have a pool of known "good" MAC addresses they can use or for testing can simply pick their favorite address.
Additionally VSIs do not support promisc mode. It is not clear to me at least what promiscuous mode means when the VSI is behind a non-learning embedded bridge wich does not do flooding. The hardware does not support flooding on the embedded bridge however it does support mirroring which could be enabled to get something that looks like promiscuous mode. Currently there is no kernel support (netlink cmd?) to configure mirroring.
The embedded bridge that connects VSIs looks like an edge relay in IEEE std terms. And can support both VEB and VEPA modes which are managed independently of VSIs through the {set|get} bridge APIs.
VSIs only support a single MAC addresses in this patch. This is a software limitation and can/will be extended in subsequent patches. For now a single MAC addresses is supported.
NOTE:
The SRIOV and VMDQ flags in ixgbe seem to be a bit overloaded and convoluted. This patch uses these flags. A follow up patch could clean up the ixgbe flag usage scheme.
RFC TODO: I'll complete these before sending the real patch
(1) interop with DCB/FCOE/SR-IOV
(2) vmdq_netdev can be dropped from tx_ring and just use correctly set netdev. In slowpath we can look up management netdev via vadapter->real_adapter->netdev and save the hotpath vmdq_netdev if/else branch.
Otherwise I've been adding/delete VSIs with multiple netperf TCP sessions without any crashes but I still need to do some more testing.
--- drivers/net/ethernet/intel/ixgbe/Makefile | 3 drivers/net/ethernet/intel/ixgbe/ixgbe.h | 32 ++ drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 15 + drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 307 +++++++++++----- drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c | 428 ++++++++++++++++++++++ drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h | 71 ++++ 7 files changed, 766 insertions(+), 94 deletions(-) create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h
drivers/net/ethernet/intel/ixgbe/Makefile +2 -1
@@ -34,7 +34,8 @@ obj-$(CONFIG_IXGBE) += ixgbe.o ixgbe-objs := ixgbe_main.o ixgbe_common.o ixgbe_ethtool.o \ ixgbe_82599.o ixgbe_82598.o ixgbe_phy.o ixgbe_sriov.o \ - ixgbe_mbx.o ixgbe_x540.o ixgbe_lib.o ixgbe_ptp.o + ixgbe_mbx.o ixgbe_x540.o ixgbe_lib.o ixgbe_ptp.o \ + ixgbe_vsi.o ixgbe-$(CONFIG_IXGBE_DCB) += ixgbe_dcb.o ixgbe_dcb_82598.o \ ixgbe_dcb_82599.o ixgbe_dcb_nl.o
drivers/net/ethernet/intel/ixgbe/ixgbe.h +30 -2
@@ -244,6 +244,7 @@ struct ixgbe_ring {unsigned long last_rx_timestamp;unsigned long state;u8 __iomem *tail; +struct net_device *vmdq_netdev;dma_addr_t dma;/* phys. address of descriptor ring */ unsigned int size;/* length in bytes */
@@ -288,11 +289,15 @@ enum ixgbe_ring_f_enum { };#define IXGBE_MAX_RSS_INDICES 16 -#define IXGBE_MAX_VMDQ_INDICES 64 +#define IXGBE_MAX_VMDQ_INDICES 32#define IXGBE_MAX_FDIR_INDICES 63/* based on q_vector limit */ #define IXGBE_MAX_FCOE_INDICES 8#define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)#define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1) +#define IXGBE_MAX_VSI_QUEUES 4 +#define IXGBE_MAX_VSI_QUEUES 4 +#define IXGBE_BAD_VSI_QUEUE 3 +struct ixgbe_ring_feature {u16 limit;/* upper limit on feature indices */ u16 indices;/* current value of indices */
@@ -738,6 +743,7 @@ struct ixgbe_adapter {#endif /*CONFIG_DEBUG_FS*/ u8 default_up; +unsigned long vsi_bitmask;/* Bitmask indicating in use pools */ };struct ixgbe_fdir_filter {
@@ -747,6 +753,17 @@ struct ixgbe_fdir_filter {u16 action; }; +struct ixgbe_vsi_adapter { +unsigned long active_vlans[BITS_TO_LONGS (VLAN_N_VID )]; +struct net_device *netdev; +struct ixgbe_adapter *real_adapter; +unsigned int tx_base_queue; +unsigned int rx_base_queue; +struct net_device_stats net_stats; +int pool; +bool online; +}; +enum ixgbe_state_t { __IXGBE_TESTING, __IXGBE_RESETTING,
@@ -879,9 +896,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}static inline void ixgbe_dbg_init (void ) {}static inline void ixgbe_dbg_exit (void ) {}#endif /* CONFIG_DEBUG_FS */ +static inline struct net_device *netdev_ring (const struct ixgbe_ring *ring) +{ +return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev; +} +static inline struct netdev_queue *txring_txq (const struct ixgbe_ring *ring) { -return netdev_get_tx_queue (ring->netdev, ring->queue_index); +return netdev_get_tx_queue (netdev_ring (ring), ring->queue_index); }extern void ixgbe_ptp_init (struct ixgbe_adapter *adapter);
@@ -915,4 +937,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);void ixgbe_sriov_reinit (struct ixgbe_adapter *adapter);#endif +int ixgbe_get_settings (struct net_device *dev,struct ethtool_cmd *ecmd); +int ixgbe_write_uc_addr_list (struct net_device *netdev); +netdev_tx_t ixgbe_xmit_frame_ring (struct sk_buff *skb, +struct ixgbe_adapter *adapter, +struct ixgbe_ring *tx_ring); +void ixgbe_clean_rx_ring (struct ixgbe_ring *rx_ring);#endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = { };#define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN -static int ixgbe_get_settings (struct net_device *netdev, -struct ethtool_cmd *ecmd) +int ixgbe_get_settings (struct net_device *netdev, +struct ethtool_cmd *ecmd) {struct ixgbe_adapter *adapter =netdev_priv (netdev);struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)#endif /* only proceed if SR-IOV is enabled */ -if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED )) +if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && + !(adapter->flags &IXGBE_FLAG_VMDQ_ENABLED ))return false;/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,/* apply Tx specific ring traits */ ring->count = adapter->tx_ring_count; - ring->queue_index = txr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + txr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = txr_idx;/* assign ring to adapter */ adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,#endif /* IXGBE_FCOE */ /* apply Rx specific ring traits */ ring->count = adapter->rx_ring_count; - ring->queue_index = rxr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + rxr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = rxr_idx;/* assign ring to adapter */ adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +221 -86
589 lines not on this page · raw message
drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.c new file +428 -0
428 lines not on this page · raw message
drivers/net/ethernet/intel/ixgbe/ixgbe_vsi.h new file +71 -0
71 lines not on this page · raw message
cover net: alternate proposal for using macvlans with forwarding acceleration
In reply to John FastabendJohn, et al. -
As promised, heres my (very rough) first pass at an alternate propsal for
what you're trying to do with virtual station interfaces here. Its completely untested, but it builds, and I'll be trying to run it over the next few days (though I'm sure I got part of the hardware manipulation wrong). I wanted to post it early though so you could get a look at it to see what you did and didn't like about it. Some notes:
1) As discussed, the major effort here is to tie in macvlans with l2 forwarding acceleration, rather than creating a new vsi link type. That should make management easier for admins (be it via ovs or some other mechanism). It basically exposes a bit less to the user, which I think is good.
2) I've separated out the l2 forwarding acceleration operations from the net_device_operations structure. I'm not sure I like that yet, but I'm kind on leaning that way. Since a limited set of hardare supports forwarding acceleration, it makes for a nice easy way to group functionality without polluting the net_device_operations structure. It also lets us group simmilar functions together nicely (I can see a future l3_accel_ops structure if we can do l3 flows in hardware). Anywho, its a divergence from what we've been doing so I thought I would call attention to it.
3) I've included a l2_accel_xmit method in the accel_ops structure for fast path forwarding, but I'm not sure I like that. It seems we should be able to use ndo_start_xmit and key off some data to recognize that we should be doing hardware forwarding. I'm not quite sure how to do that yet though. Something to think about.
4) I've borrowed heavily from your vsi work of course just to get this building. I think theres probbaly alot of consolidation that can be done in the code that I added to ixgbe_main.c to make it smaller. Again, I just wanted to post this so you could speak up if you though this was all crap before I wen't too far down the rabbit hole.
Regards
Neil
RFC PATCH 1/2 net: Add layer 2 hardware acceleration operations for macvlan devices
Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.
--- drivers/net/macvlan.c | 37 +++++++++++++++++++++++++++++++++++++ include/linux/if_macvlan.h | 1 + include/linux/netdevice.h | 10 ++++++++++ net/core/dev.c | 3 +++ 4 files changed, 51 insertions(+)
drivers/net/macvlan.c +37 -0
@@ -296,8 +296,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,unsigned int len = skb->len;int ret;const struct macvlan_dev *vlan =netdev_priv (dev); +const struct l2_forwarding_accel_ops *l2a_ops = vlan->lowerdev->l2a_ops; + +if (l2a_ops->l2_accel_xmit) { + ret = l2a_ops->l2_accel_xmit (skb, vlan->l2a_priv); +if (likely (ret ==NETDEV_TX_OK )) + goto update_stats; + } ret =macvlan_queue_xmit (skb, dev); +update_stats:if (likely (ret ==NET_XMIT_SUCCESS || ret ==NET_XMIT_CN )) {struct macvlan_pcpu_stats *pcpu_stats;
@@ -336,6 +344,7 @@ static int macvlan_open(struct net_device *dev) {struct macvlan_dev *vlan =netdev_priv (dev);struct net_device *lowerdev = vlan->lowerdev; +const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;int err;if (vlan->port->passthru) {
@@ -347,6 +356,19 @@ static int macvlan_open(struct net_device *dev) goto hash_add; } +if (l2a_ops->l2_accel_add_dev) { +/* The lowerdev supports l2 switching +* try to add this macvlan to it +*/ + vlan->l2a_priv =kzalloc (l2a_ops->priv_size,GFP_KERNEL ); +if (!vlan->l2a_priv) +return -ENOMEM ; + err = l2a_ops->l2_accel_add_dev (vlan->lowerdev, + dev, vlan->l2a_priv); +if (err <0 ) +return err; + } + err = -EBUSY ;if (macvlan_addr_busy (vlan->port, dev->dev_addr)) goto out;
@@ -367,6 +389,13 @@ hash_add: del_unicast:dev_uc_del (lowerdev, dev->dev_addr); out: +if (vlan->l2a_priv) { +if (l2a_ops->l2_accel_del_dev) + l2a_ops->l2_accel_del_dev (vlan->lowerdev, + vlan->l2a_priv); +kfree (vlan->l2a_priv); + vlan->l2a_priv =NULL ; + }return err; }
@@ -374,6 +403,7 @@ static int macvlan_stop(struct net_device *dev) {struct macvlan_dev *vlan =netdev_priv (dev);struct net_device *lowerdev = vlan->lowerdev; +const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;dev_uc_unsync (lowerdev, dev);dev_mc_unsync (lowerdev, dev);
@@ -391,6 +421,12 @@ static int macvlan_stop(struct net_device *dev) hash_del:macvlan_hash_del (vlan, !dev->dismantle); +if (vlan->l2a_priv) { + l2a_ops->l2_accel_del_dev (vlan->lowerdev, vlan->l2a_priv); +kfree (vlan->l2a_priv); + vlan->l2a_priv =NULL ; + } +return 0 ; }
@@ -801,6 +837,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev,if (err <0 )return err; } + port =macvlan_port_get_rtnl (lowerdev);/* Only 1 macvlan device can be created in passthru mode */
include/linux/if_macvlan.h +1 -0
@@ -61,6 +61,7 @@ struct macvlan_dev {struct hlist_node hlist;struct macvlan_port *port;struct net_device *lowerdev; +void *l2a_priv;struct macvlan_pcpu_stats __percpu *pcpu_stats;DECLARE_BITMAP (mc_filter,MACVLAN_MC_FILTER_SZ );
include/linux/netdevice.h +10 -0
@@ -1099,6 +1099,15 @@ struct net_device_ops { __be16 port); }; +struct l2_forwarding_accel_ops { +size_t priv_size; +int (*l2_accel_add_dev)(struct net_device *pdev, +struct net_device *vdev,void *priv); +void (*l2_accel_del_dev)(struct net_device *pdev,void *priv); +netdev_tx_t (*l2_accel_xmit)(struct sk_buff *skb, +void *priv); +}; +/* * The DEVICE structure. * Actually, this whole structure is a big mistake. It mixes I/O
@@ -1183,6 +1192,7 @@ struct net_device {/* Management operations */ const struct net_device_ops *netdev_ops;const struct ethtool_ops *ethtool_ops; +const struct l2_forwarding_accel_ops *l2a_ops;/* Hardware header description */ const struct header_ops *header_ops;
net/core/dev.c +3 -0
@@ -5992,6 +5992,7 @@ struct netdev_queue *dev_ingress_queue_create(struct net_device *dev) }static const struct ethtool_ops default_ethtool_ops; +static const struct l2_forwarding_accel_ops default_l2a_ops;void netdev_set_default_ethtool_ops (struct net_device *dev,const struct ethtool_ops *ops)
@@ -6090,6 +6091,8 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name, dev->group =INIT_NETDEV_GROUP ;if (!dev->ethtool_ops) dev->ethtool_ops = &default_ethtool_ops; +if (!dev->l2a_ops) + dev->l2a_ops = &default_l2a_ops;return dev; free_all:
Signature · 1 line
1.8.3.1
Re: [RFC PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
Quoted 5 lines · Neil Horman
Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.
Some additional nits below which maybe you have already thought of.
Quoted 16 lines
Signed-off-by Neil Horman <[email protected]>CC "David S. Miller" <[email protected]>--- drivers/net/macvlan.c | 37 +++++++++++++++++++++++++++++++++++++ include/linux/if_macvlan.h | 1 + include/linux/netdevice.h | 10 ++++++++++ net/core/dev.c | 3 +++ 4 files changed, 51 insertions(+)drivers/net/macvlan.c +0
-0
drivers/net/macvlan.c netdev_tx_t macvlan_start_xmit(struct sk_buff *skb, · line 303
Quoted 8 lines
drivers/net/macvlan.c +5
-0@@ -296,8 +296,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,unsigned int len = skb->len;int ret;const struct macvlan_dev *vlan =netdev_priv (dev); +const struct l2_forwarding_accel_ops *l2a_ops = vlan->lowerdev->l2a_ops; + +if (l2a_ops->l2_accel_xmit) { + ret = l2a_ops->l2_accel_xmit (skb, vlan->l2a_priv); +if (likely (ret ==NETDEV_TX_OK ))
maybe dev_xmit_complete() would be more appropriate?
Quoted 17 lines
drivers/net/macvlan.c +4
-0+ goto update_stats; + } ret =macvlan_queue_xmit (skb, dev); +update_stats:if (likely (ret ==NET_XMIT_SUCCESS || ret ==NET_XMIT_CN )) {struct macvlan_pcpu_stats *pcpu_stats;@@ -336,6 +344,7 @@ static int macvlan_open(struct net_device *dev) {struct macvlan_dev *vlan =netdev_priv (dev);struct net_device *lowerdev = vlan->lowerdev; +const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;int err;if (vlan->port->passthru) {
drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 356
Quoted 1 line
drivers/net/macvlan.c +0
-0@@ -347,6 +356,19 @@ static int macvlan_open(struct net_device *dev) goto hash_add;
Looks like this might break in the passthru case? If you don't call l2_accel_add_dev here but still use the l2_accel_xmit.
drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 359
Quoted 3 lines
drivers/net/macvlan.c +1
-0} +if (l2a_ops->l2_accel_add_dev) {
In the error cases it might be preferred to fallback to the non-offloaded software path. For example hardware may have a limit to the number of VSIs that can be created but we wouldn't want to push that up the stack.
Quoted 16 lines
drivers/net/macvlan.c +12
-0+/* The lowerdev supports l2 switching +* try to add this macvlan to it +*/ + vlan->l2a_priv =kzalloc (l2a_ops->priv_size,GFP_KERNEL ); +if (!vlan->l2a_priv) +return -ENOMEM ; + err = l2a_ops->l2_accel_add_dev (vlan->lowerdev, + dev, vlan->l2a_priv); +if (err <0 ) +return err; + } + err = -EBUSY ;if (macvlan_addr_busy (vlan->port, dev->dev_addr)) goto out;
drivers/net/macvlan.c hash_add: · line 392
Quoted 4 lines
drivers/net/macvlan.c +1
-0@@ -367,6 +389,13 @@ hash_add: del_unicast:dev_uc_del (lowerdev, dev->dev_addr); out: +if (vlan->l2a_priv) {
Add a feature flag here so it can be disabled.
drivers/net/macvlan.c hash_add: · line 400
Quoted 8 lines
drivers/net/macvlan.c +6
-0+if (l2a_ops->l2_accel_del_dev) + l2a_ops->l2_accel_del_dev (vlan->lowerdev, + vlan->l2a_priv); +kfree (vlan->l2a_priv); + vlan->l2a_priv =NULL ; + }return err; }
[...]
Thanks,
John
Signature · 1 line
John Fastabend Intel Corporation
Re: [RFC PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
Quoted 9 lines · John Fastabend
Quoted 5 lines · Neil Horman
Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.
Some additional nits below which maybe you have already thought of.
Its ok, you can say it, theyre more than nits :), gaping holes is more like it. This pass was completely untested, I'm still ironing out bits, but I think for the most part we're in agreement on the items below. Comments inline.
Quoted 11 lines
Quoted 11 lines
Signed-off-by Neil Horman <[email protected]>CC "David S. Miller" <[email protected]>--- drivers/net/macvlan.c | 37 +++++++++++++++++++++++++++++++++++++ include/linux/if_macvlan.h | 1 + include/linux/netdevice.h | 10 ++++++++++ net/core/dev.c | 3 +++ 4 files changed, 51 insertions(+)
Quoted 16 lines
drivers/net/macvlan.c netdev_tx_t macvlan_start_xmit(struct sk_buff *skb, · line 303
Quoted 13 lines
drivers/net/macvlan.c +5
-0@@ -296,8 +296,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,unsigned int len = skb->len;int ret;const struct macvlan_dev *vlan =netdev_priv (dev); +const struct l2_forwarding_accel_ops *l2a_ops = vlan->lowerdev->l2a_ops; + +if (l2a_ops->l2_accel_xmit) { + ret = l2a_ops->l2_accel_xmit (skb, vlan->l2a_priv); +if (likely (ret ==NETDEV_TX_OK ))maybe
dev_xmit_complete()would be more appropriate?
Yup, I've currently modified this to dev_queue_xmit, but _complete might be a better option.
Quoted 15 lines
Quoted 15 lines
drivers/net/macvlan.c +4
-0+ goto update_stats; + } ret =macvlan_queue_xmit (skb, dev); +update_stats:if (likely (ret ==NET_XMIT_SUCCESS || ret ==NET_XMIT_CN )) {struct macvlan_pcpu_stats *pcpu_stats;@@ -336,6 +344,7 @@ static int macvlan_open(struct net_device *dev) {struct macvlan_dev *vlan =netdev_priv (dev);struct net_device *lowerdev = vlan->lowerdev; +const struct l2_forwarding_accel_ops *l2a_ops = lowerdev->l2a_ops;int err;
Quoted 7 lines
drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 356
Quoted 3 lines
drivers/net/macvlan.c +0
-0if (vlan->port->passthru) {@@ -347,6 +356,19 @@ static int macvlan_open(struct net_device *dev) goto hash_add;Looks like this might break in the passthru case? If you don't call
l2_accel_add_devhere but still use thel2_accel_xmit.
Yeah, I need to clean this up.
Quoted 9 lines
drivers/net/macvlan.c static int macvlan_open(struct net_device *dev) · line 359
Quoted 3 lines
drivers/net/macvlan.c +1
-0} +if (l2a_ops->l2_accel_add_dev) {In the error cases it might be preferred to fallback to the non-offloaded software path. For example hardware may have a limit to the number of VSIs that can be created but we wouldn't want to push that up the stack.
Hadn't thought of that, but yes, I agree, falling back to software switching is definately desireable here
Quoted 14 lines
Quoted 14 lines
drivers/net/macvlan.c +12
-0+/* The lowerdev supports l2 switching +* try to add this macvlan to it +*/ + vlan->l2a_priv =kzalloc (l2a_ops->priv_size,GFP_KERNEL ); +if (!vlan->l2a_priv) +return -ENOMEM ; + err = l2a_ops->l2_accel_add_dev (vlan->lowerdev, + dev, vlan->l2a_priv); +if (err <0 ) +return err; + } + err = -EBUSY ;if (macvlan_addr_busy (vlan->port, dev->dev_addr))
Quoted 9 lines
drivers/net/macvlan.c hash_add: · line 392
Quoted 6 lines
drivers/net/macvlan.c +1
-0goto out;@@ -367,6 +389,13 @@ hash_add: del_unicast:dev_uc_del (lowerdev, dev->dev_addr); out: +if (vlan->l2a_priv) {Add a feature flag here so it can be disabled.
Makes sense.
I'm still working out kinks, but I hope to have a next version later this week/early next. I'll make sure to incorporate these changes.
Thanks!
Neil
Quoted 18 lines
drivers/net/macvlan.c hash_add: · line 400
Quoted 8 lines
drivers/net/macvlan.c +6
-0+if (l2a_ops->l2_accel_del_dev) + l2a_ops->l2_accel_del_dev (vlan->lowerdev, + vlan->l2a_priv); +kfree (vlan->l2a_priv); + vlan->l2a_priv =NULL ; + }return err; }[...]
Thanks,
JohnSignature · 1 line
John Fastabend Intel Corporation
RFC PATCH 2/2 ixgbe: enable l2 forwarding acceleration for macvlans
In reply to Neil HormanNow that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 32 +- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 +- drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h | 54 ++++ drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 15 +- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 360 +++++++++++++++++------ 5 files changed, 374 insertions(+), 91 deletions(-) create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h
drivers/net/ethernet/intel/ixgbe/ixgbe.h +30 -2
@@ -244,6 +244,7 @@ struct ixgbe_ring {unsigned long last_rx_timestamp;unsigned long state;u8 __iomem *tail; +struct net_device *vmdq_netdev;dma_addr_t dma;/* phys. address of descriptor ring */ unsigned int size;/* length in bytes */
@@ -288,11 +289,15 @@ enum ixgbe_ring_f_enum { };#define IXGBE_MAX_RSS_INDICES 16 -#define IXGBE_MAX_VMDQ_INDICES 64 +#define IXGBE_MAX_VMDQ_INDICES 32#define IXGBE_MAX_FDIR_INDICES 63/* based on q_vector limit */ #define IXGBE_MAX_FCOE_INDICES 8#define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)#define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1) +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_BAD_L2A_QUEUE 3 +struct ixgbe_ring_feature {u16 limit;/* upper limit on feature indices */ u16 indices;/* current value of indices */
@@ -738,6 +743,7 @@ struct ixgbe_adapter {#endif /*CONFIG_DEBUG_FS*/ u8 default_up; +unsigned long l2a_bitmask;/* Bitmask indicating in use pools */ };struct ixgbe_fdir_filter {
@@ -747,6 +753,17 @@ struct ixgbe_fdir_filter {u16 action; }; +struct ixgbe_l2a_adapter { +unsigned long active_vlans[BITS_TO_LONGS (VLAN_N_VID )]; +struct net_device *netdev; +struct ixgbe_adapter *real_adapter; +unsigned int tx_base_queue; +unsigned int rx_base_queue; +struct net_device_stats net_stats; +int pool; +bool online; +}; +enum ixgbe_state_t { __IXGBE_TESTING, __IXGBE_RESETTING,
@@ -879,9 +896,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}static inline void ixgbe_dbg_init (void ) {}static inline void ixgbe_dbg_exit (void ) {}#endif /* CONFIG_DEBUG_FS */ +static inline struct net_device *netdev_ring (const struct ixgbe_ring *ring) +{ +return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev; +} +static inline struct netdev_queue *txring_txq (const struct ixgbe_ring *ring) { -return netdev_get_tx_queue (ring->netdev, ring->queue_index); +return netdev_get_tx_queue (netdev_ring (ring), ring->queue_index); }extern void ixgbe_ptp_init (struct ixgbe_adapter *adapter);
@@ -915,4 +937,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);void ixgbe_sriov_reinit (struct ixgbe_adapter *adapter);#endif +int ixgbe_get_settings (struct net_device *dev,struct ethtool_cmd *ecmd); +int ixgbe_write_uc_addr_list (struct net_device *netdev); +netdev_tx_t ixgbe_xmit_frame_ring (struct sk_buff *skb, +struct ixgbe_adapter *adapter, +struct ixgbe_ring *tx_ring); +void ixgbe_clean_rx_ring (struct ixgbe_ring *rx_ring);#endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = { };#define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN -static int ixgbe_get_settings (struct net_device *netdev, -struct ethtool_cmd *ecmd) +int ixgbe_get_settings (struct net_device *netdev, +struct ethtool_cmd *ecmd) {struct ixgbe_adapter *adapter =netdev_priv (netdev);struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h new file +54 -0
54 lines not on this page · raw message
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)#endif /* only proceed if SR-IOV is enabled */ -if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED )) +if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && + !(adapter->flags &IXGBE_FLAG_VMDQ_ENABLED ))return false;/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,/* apply Tx specific ring traits */ ring->count = adapter->tx_ring_count; - ring->queue_index = txr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + txr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = txr_idx;/* assign ring to adapter */ adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,#endif /* IXGBE_FCOE */ /* apply Rx specific ring traits */ ring->count = adapter->rx_ring_count; - ring->queue_index = rxr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + rxr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = rxr_idx;/* assign ring to adapter */ adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +276 -84
579 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [RFC PATCH 0/2] net: alternate proposal for using macvlans with forwarding acceleration
In reply to Neil HormanQuoted 7 lines · Neil Horman
John, et al. -
As promised, heres my (very rough) first pass at an alternate propsal for
what you're trying to do with virtual station interfaces here. Its completely untested, but it builds, and I'll be trying to run it over the next few days (though I'm sure I got part of the hardware manipulation wrong). I wanted to post it early though so you could get a look at it to see what you did and didn't like about it. Some notes:
Sorry for the delay. I like the idea one nice win here is my macvlan kvm setup would use the offloads without having to reconfigure.
Quoted 6 lines
1) As discussed, the major effort here is to tie in macvlans with l2 forwarding acceleration, rather than creating a new vsi link type. That should make management easier for admins (be it via ovs or some other mechanism). It basically exposes a bit less to the user, which I think is good.
The trick here is the offload path may be functionally different from the non-offload path. The user needs some visibility into this. For example any qdiscs running on the lowerdev will not be visible from the accelerated path.
When a new link type was being used I was able to convince myself that it was a property of the link type. But if we reuse macvlan I think we need some way to either automatically disable the offload path when this occurs or provide the user visibility. Maybe a feature flag and a netif_can_hw_offload() routine is needed?
Quoted 9 lines
2) I've separated out the l2 forwarding acceleration operations from the
net_device_operationsstructure. I'm not sure I like that yet, but I'm kind on leaning that way. Since a limited set of hardare supports forwarding acceleration, it makes for a nice easy way to group functionality without polluting thenet_device_operationsstructure. It also lets us group simmilar functions together nicely (I can see a futurel3_accel_opsstructure if we can do l3 flows in hardware). Anywho, its a divergence from what we've been doing so I thought I would call attention to it.
Quoted 5 lines
3) I've included a
l2_accel_xmitmethod in theaccel_opsstructure for fast path forwarding, but I'm not sure I like that. It seems we should be able to usendo_start_xmitand key off some data to recognize that we should be doing hardware forwarding. I'm not quite sure how to do that yet though. Something to think about.
Without a specific xmit routine though you will be adding operations in the common case for a special case. Having a new op fixes this.
Quoted 6 lines
4) I've borrowed heavily from your vsi work of course just to get this building. I think theres probbaly alot of consolidation that can be done in the code that I added to
ixgbe_main.cto make it smaller. Again, I just wanted to post this so you could speak up if you though this was all crap before I wen't too far down the rabbit hole.
There was some consolidation needed in my original RFC as well. I can help clean some of this stuff up if you want.
.John
Signature · 1 line
John Fastabend Intel Corporation
Re: [RFC PATCH 0/2] net: alternate proposal for using macvlans with forwarding acceleration
Quoted 9 lines · John Fastabend
Quoted 7 lines · Neil Horman
John, et al. -
As promised, heres my (very rough) first pass at an alternate propsal for
what you're trying to do with virtual station interfaces here. Its completely untested, but it builds, and I'll be trying to run it over the next few days (though I'm sure I got part of the hardware manipulation wrong). I wanted to post it early though so you could get a look at it to see what you did and didn't like about it. Some notes:
Quoted 3 lines
Sorry for the delay. I like the idea one nice win here is my macvlan kvm setup would use the offloads without having to reconfigure.
Thats ok, I've been swamped, so this has gone slower than I had hoped for me.
Quoted 7 lines
Quoted 6 lines
1) As discussed, the major effort here is to tie in macvlans with l2 forwarding acceleration, rather than creating a new vsi link type. That should make management easier for admins (be it via ovs or some other mechanism). It basically exposes a bit less to the user, which I think is good.
Quoted 5 lines
The trick here is the offload path may be functionally different from the non-offload path. The user needs some visibility into this. For example any qdiscs running on the lowerdev will not be visible from the accelerated path.
Actually, given your suggestion in your other note (using dev_queue_xmit), I think this will be handled, in that we will now pass accelerated traffic through any lowerdev attached qdiscs.
Quoted 6 lines
When a new link type was being used I was able to convince myself that it was a property of the link type. But if we reuse macvlan I think we need some way to either automatically disable the offload path when this occurs or provide the user visibility. Maybe a feature flag and a
netif_can_hw_offload()routine is needed?
As noted above, I think we can make macvlans work identially with and without acceleration by reusing the xmit path, and keying off data (what I've done is attach a pointer to acceleration data in the skb so the common ndo_start_xmit path in the driver can differentiate accelerated from non-accelerated skbs). That will allow everything to function the same way. I do think though that a feature flag just to provide some visibility to the admin is warranted.
Quoted 9 lines
Quoted 9 lines
2) I've separated out the l2 forwarding acceleration operations from the
net_device_operationsstructure. I'm not sure I like that yet, but I'm kind on leaning that way. Since a limited set of hardare supports forwarding acceleration, it makes for a nice easy way to group functionality without polluting thenet_device_operationsstructure. It also lets us group simmilar functions together nicely (I can see a futurel3_accel_opsstructure if we can do l3 flows in hardware). Anywho, its a divergence from what we've been doing so I thought I would call attention to it.
Quoted 9 lines
Quoted 5 lines
3) I've included a
l2_accel_xmitmethod in theaccel_opsstructure for fast path forwarding, but I'm not sure I like that. It seems we should be able to usendo_start_xmitand key off some data to recognize that we should be doing hardware forwarding. I'm not quite sure how to do that yet though. Something to think about.Without a specific xmit routine though you will be adding operations in the common case for a special case. Having a new op fixes this.
Agreed, and thats something to consider. I'm not sure if separating the special case to its own operation is a net win however, given the behavioral differences that arise (the qdisc issue above). My next version will use a common xmit path. We can look it over then for comparison and weight the pros and cons.
Quoted 10 lines
Quoted 6 lines
4) I've borrowed heavily from your vsi work of course just to get this building. I think theres probbaly alot of consolidation that can be done in the code that I added to
ixgbe_main.cto make it smaller. Again, I just wanted to post this so you could speak up if you though this was all crap before I wen't too far down the rabbit hole.There was some consolidation needed in my original RFC as well. I can help clean some of this stuff up if you want.
That would be greatly appreciated. Thanks. Next version should be available in the next few days. I'll get it to you asap.
Best
Neil
Quoted 5 lines
.John
Signature · 1 line
John Fastabend Intel Corporation
v2 cover net: alternate proposal for using macvlans with forwarding acceleration
In reply to Neil HormanHey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.
Some change notes:
* Changes to the fowarding ops structure - Removed the priv_size field, and added a flags field. Removal of the priv_size field was accomplished by just having the add method return a void * and using ERR_PTR and PTR_ERR checks, which also allows us to allocate memory for the acceleration path in the driver, which I like. I'm not super happy still with how I'm using the flags (currenly only used to indicate support for feature sets), but at least we have the flags now, and they can be exposed to user space via iproute2 or ethtool if need be
* Changes to the Transmit path - Specifically I'm using dev_queue_xmit to send frames now, which I like as it makes the macvlan subject to the lowerdevs qdisc configuration.
* Changes to the acceleration fail path behavior - Now if we don't/can't use acceleration, we just fall back to using the normal macvlan software switch strategy
* General clenups (some renaming, that I'm not super sure of, but I though forwarding acceleration (fwd) would be a better prefix than l2 acceleration).
Still a long way to go I think, and lots of tweaking to do, but I didn't want to keep you waiting John. Anywho, take a look at what I'm doing and feel free to rip it apart.
Thanks!
Neil
PATCH 1/2 net: Add layer 2 hardware acceleration operations for macvlan devices
Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.
--- drivers/net/macvlan.c | 32 ++++++++++++++++++++++++++++++++ include/linux/if_macvlan.h | 1 + include/linux/netdevice.h | 22 ++++++++++++++++++++++ include/linux/skbuff.h | 9 ++++++--- net/core/dev.c | 3 +++ 5 files changed, 64 insertions(+), 3 deletions(-)
drivers/net/macvlan.c +32 -0
@@ -297,7 +297,17 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,int ret;const struct macvlan_dev *vlan =netdev_priv (dev); +if (vlan->fwd_priv) { + skb->dev = vlan->lowerdev; + skb->accel_priv = vlan->fwd_priv; + ret =dev_queue_xmit (skb); +if (likely (ret ==NETDEV_TX_OK )) + goto update_stats; + } + + skb->accel_priv =NULL ; ret =macvlan_queue_xmit (skb, dev); +update_stats:if (likely (ret ==NET_XMIT_SUCCESS || ret ==NET_XMIT_CN )) {struct macvlan_pcpu_stats *pcpu_stats;
@@ -347,6 +357,18 @@ static int macvlan_open(struct net_device *dev) goto hash_add; } +if (fwd_accel_supports (lowerdev,FA_FLG_STA_SUPPORT )) { + vlan->fwd_priv =fwd_accel_add_station (lowerdev, dev); +/* +* If we get a NULL pointer back, or if we get an error +* then we should just fall through to the non accelerated path +*/ +if (IS_ERR_OR_NULL (vlan->fwd_priv)) + vlan->fwd_priv =NULL ; +else +return 0 ; + } + err = -EBUSY ;if (macvlan_addr_busy (vlan->port, dev->dev_addr)) goto out;
@@ -367,6 +389,10 @@ hash_add: del_unicast:dev_uc_del (lowerdev, dev->dev_addr); out: +if (vlan->fwd_priv) { +fwd_accel_del_station (lowerdev, vlan->fwd_priv); + vlan->fwd_priv =NULL ; + }return err; }
@@ -391,6 +417,11 @@ static int macvlan_stop(struct net_device *dev) hash_del:macvlan_hash_del (vlan, !dev->dismantle); +if (vlan->fwd_priv) { +fwd_accel_del_station (lowerdev, vlan->fwd_priv); + vlan->fwd_priv =NULL ; + } +return 0 ; }
@@ -801,6 +832,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev,if (err <0 )return err; } + port =macvlan_port_get_rtnl (lowerdev);/* Only 1 macvlan device can be created in passthru mode */
include/linux/if_macvlan.h +1 -0
@@ -61,6 +61,7 @@ struct macvlan_dev {struct hlist_node hlist;struct macvlan_port *port;struct net_device *lowerdev; +void *fwd_priv;struct macvlan_pcpu_stats __percpu *pcpu_stats;DECLARE_BITMAP (mc_filter,MACVLAN_MC_FILTER_SZ );
include/linux/netdevice.h +22 -0
@@ -1100,6 +1100,27 @@ struct net_device_ops { };/* +* Flags to ennumerate hardware acceleration support +*/ +#define FA_FLG_STA_SUPPORT (1 << 1) + +#define fwd_accel_supports (dev, feature) (dev->fwd_ops->flags & feature) +#define fwd_accel_add_station (pdev, vdev) dev->fwd_ops->fwd_accel_add_station(pdev, vdev) +#define fwd_accel_del_station (pdev, priv) dev->fwd_ops->fwd_accel_del_station(pdev, priv) + +struct forwarding_accel_ops { +unsigned int flags; + +/* +* fwd_accel_[add|del]_station must be set if +* FA_FLG_STA_SUPPORT is set +*/ +void * (*fwd_accel_add_station)(struct net_device *pdev, +struct net_device *vdev); +void (*fwd_accel_del_station)(struct net_device *pdev,void *priv); +}; + +/* * The DEVICE structure. * Actually, this whole structure is a big mistake. It mixes I/O * data with strictly "high-level" data, and it has to know about
@@ -1183,6 +1204,7 @@ struct net_device {/* Management operations */ const struct net_device_ops *netdev_ops;const struct ethtool_ops *ethtool_ops; +const struct forwarding_accel_ops *fwd_ops;/* Hardware header description */ const struct header_ops *header_ops;
include/linux/skbuff.h +6 -3
@@ -426,9 +426,12 @@ struct sk_buff {char cb[48 ]__aligned(8) ;unsigned long _skb_refdst; -#ifdef CONFIG_XFRM -struct sec_path *sp; -#endif + +union { +struct sec_path *sp; +void *accel_priv; + }; +unsigned int len, data_len; __u16 mac_len,
net/core/dev.c +3 -0
@@ -5992,6 +5992,7 @@ struct netdev_queue *dev_ingress_queue_create(struct net_device *dev) }static const struct ethtool_ops default_ethtool_ops; +static const struct forwarding_accel_ops default_fwd_ops;void netdev_set_default_ethtool_ops (struct net_device *dev,const struct ethtool_ops *ops)
@@ -6090,6 +6091,8 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name, dev->group =INIT_NETDEV_GROUP ;if (!dev->ethtool_ops) dev->ethtool_ops = &default_ethtool_ops; +if (!dev->fwd_ops) + dev->fwd_ops = &default_fwd_ops;return dev; free_all:
Signature · 1 line
1.8.3.1
David Miller
maintainerRe: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
From: Neil Horman <[email protected]>
Date: Fri, 4 Oct 2013 16:10:04 -0400
include/linux/skbuff.h struct sk_buff { · line 434
Quoted 13 lines
include/linux/skbuff.h +6
-3@@ -426,9 +426,12 @@ struct sk_buff {char cb[48 ]__aligned(8) ;unsigned long _skb_refdst; -#ifdef CONFIG_XFRM -struct sec_path *sp; -#endif + +union { +struct sec_path *sp; +void *accel_priv; + }; +
I'm not %100 sure these two things are really mutually exclusive.
What if bridging ebtables does an input route lookup? That can populate the security path.
Also, why have you not added this to the usual netdev_ops and hw_features?
Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
Forgive the poor reply format, Dave, I deleted your email (to fast on the trigger apparently), so I have to reconstruct it.
Quoted 16 lines
include/linux/skbuff.h struct sk_buff { · line 434
Quoted 13 lines
include/linux/skbuff.h +6
-3@@ -426,9 +426,12 @@ struct sk_buff {char cb[48 ]__aligned(8) ;unsigned long _skb_refdst; -#ifdef CONFIG_XFRM -struct sec_path *sp; -#endif + +union { +struct sec_path *sp; +void *accel_priv; + }; +I'm not %100 sure these two things are really mutually exclusive.
Quoted 3 lines
What if bridging ebtables does an input route lookup? That can populate the security path.
You are mostly likely right, thats why this is an RFC, I haven't really thought through that bit fully yet, to be perfectly honest. I wanted a place for a pointer to the accelerated data path data to live, and that looked like a reasonably safe place at the time, but as you point out, its not. I'll need to find a better place for it.
Quoted 2 lines
Also, why have you not added this to the usual
netdev_opsandhw_features?
Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a size_t variable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....
Regards
Neil
David Miller
maintainerRe: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
From: Neil Horman <[email protected]>
Date: Mon, 7 Oct 2013 17:20:00 -0400
Quoted 8 lines
Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a
size_tvariable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....
Can these dataplane devices still act like a normal networking port and send and receive packets at the host level?
If yes, that would be an extremely strong argument for netdev_ops.
Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
Quoted 12 lines · David Miller
From: Neil Horman <[email protected]>
Date: Mon, 7 Oct 2013 17:20:00 -0400Quoted 8 lines
Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a
size_tvariable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....
We can do something like the dcbnl ops and add another pointer off the net device structure and then use the skb->dev field to find the correct set of ops? This seems like the simplest option to me and isolates the ops structure.
Is there some information loss from hanging it off the netdevice structure vs the skb? I can't see any.
Quoted 3 lines
Can these dataplane devices still act like a normal networking port and send and receive packets at the host level?
Yes they act like normal networking ports except for there is a switching component in the hardware. These patches are not looking at virtual or multiple physical functions at the moment.
Quoted 1 line
If yes, that would be an extremely strong argument for
netdev_ops.
I agree.
Signature · 1 line
John Fastabend Intel Corporation
Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
Quoted 14 lines · John Fastabend
Quoted 12 lines · David Miller
From: Neil Horman <[email protected]>
Date: Mon, 7 Oct 2013 17:20:00 -0400Quoted 8 lines
Thats me experimenting. I was thinking that origionally this functionality might be grouped separately, so that we could handle it independently of the standard network device operations (you might have noticed in v1 of my patch I had a
size_tvariable in there, so I thought the separation might be organizationally nice). It was also something I was tinkering with for potential future work to support other data plane accelerators (like the FM6000 switch chip from intel) in a manner that didn't pollute the more typical host network devices. Like I said though, just experimenting at the moment....
Quoted 5 lines
We can do something like the dcbnl ops and add another pointer off the net device structure and then use the
skb->devfield to find the correct set of ops? This seems like the simplest option to me and isolates the ops structure.
We certainly could do that, or perhaps, for what we're trying to do here, just using standard netdev_ops is sufficient. I kind of like the separation (like the dcbnl_ops), but like I said, experimenting. I'll try the next version with the accel methods added to the netdev structure for comparison.
Quoted 3 lines
Is there some information loss from hanging it off the netdevice structure vs the skb? I can't see any.
No, not that I'm aware of. The only reason I added it to the skb in this version was that, by doing so, I was able to make dual use of the netdev's standard tx path.
Quoted 8 lines
Quoted 3 lines
Can these dataplane devices still act like a normal networking port and send and receive packets at the host level?
Yes they act like normal networking ports except for there is a switching component in the hardware. These patches are not looking at virtual or multiple physical functions at the moment.
To be clear, as John says, these patches aren't addressing any dataplane acceleration devices beyond the internal switching capabilities of the ixgbe cards. That said, other chips will have varying degrees of capabilities, from simple L2 switching, to full content addressable memories that allow for l2/l3 forwarding, as well as higher level routing functions. Again however, these patches are just to integrate macvlans with johns virtual station interface work.
Quoted 3 lines
Quoted 1 line
If yes, that would be an extremely strong argument for
netdev_ops.I agree.
In this specific case, that may well be the case, yes. I'm not so sure of that for more advanced switching/routing accelerators, but we probably should do what makes sense now, and worry about future bridges when we forward over them (pardon the pun :) ).
Neil
Quoted 5 lines
Signature · 2 lines
-- John Fastabend Intel Corporation
PATCH 2/2 ixgbe: enable l2 forwarding acceleration for macvlans
In reply to Neil HormanNow that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 33 +- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 +- drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h | 54 ++++ drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 15 +- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 373 +++++++++++++++++------ 5 files changed, 387 insertions(+), 92 deletions(-) create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h
drivers/net/ethernet/intel/ixgbe/ixgbe.h +31 -2
@@ -219,6 +219,17 @@ enum ixgbe_ring_state_t { __IXGBE_RX_FCOE, }; +struct ixgbe_fwd_adapter { +unsigned long active_vlans[BITS_TO_LONGS (VLAN_N_VID )]; +struct net_device *netdev; +struct ixgbe_adapter *real_adapter; +unsigned int tx_base_queue; +unsigned int rx_base_queue; +struct net_device_stats net_stats; +int pool; +bool online; +}; +#define check_for_tx_hang (ring) \ test_bit(__IXGBE_TX_DETECT_HANG, &(ring)->state)#define set_check_for_tx_hang (ring) \
@@ -236,6 +247,7 @@ struct ixgbe_ring {struct ixgbe_q_vector *q_vector;/* backpointer to host q_vector */ struct net_device *netdev;/* netdev ring belongs to */ struct device *dev;/* device for DMA mapping */ +struct ixgbe_fwd_adapter *l2_accel_priv;void *desc;/* descriptor ring memory */ union {struct ixgbe_tx_buffer *tx_buffer_info;
@@ -244,6 +256,7 @@ struct ixgbe_ring {unsigned long last_rx_timestamp;unsigned long state;u8 __iomem *tail; +struct net_device *vmdq_netdev;dma_addr_t dma;/* phys. address of descriptor ring */ unsigned int size;/* length in bytes */
@@ -288,11 +301,15 @@ enum ixgbe_ring_f_enum { };#define IXGBE_MAX_RSS_INDICES 16 -#define IXGBE_MAX_VMDQ_INDICES 64 +#define IXGBE_MAX_VMDQ_INDICES 32#define IXGBE_MAX_FDIR_INDICES 63/* based on q_vector limit */ #define IXGBE_MAX_FCOE_INDICES 8#define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)#define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1) +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_BAD_L2A_QUEUE 3 +struct ixgbe_ring_feature {u16 limit;/* upper limit on feature indices */ u16 indices;/* current value of indices */
@@ -738,6 +755,7 @@ struct ixgbe_adapter {#endif /*CONFIG_DEBUG_FS*/ u8 default_up; +unsigned long fwd_bitmask;/* Bitmask indicating in use pools */ };struct ixgbe_fdir_filter {
@@ -879,9 +897,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}static inline void ixgbe_dbg_init (void ) {}static inline void ixgbe_dbg_exit (void ) {}#endif /* CONFIG_DEBUG_FS */ +static inline struct net_device *netdev_ring (const struct ixgbe_ring *ring) +{ +return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev; +} +static inline struct netdev_queue *txring_txq (const struct ixgbe_ring *ring) { -return netdev_get_tx_queue (ring->netdev, ring->queue_index); +return netdev_get_tx_queue (netdev_ring (ring), ring->queue_index); }extern void ixgbe_ptp_init (struct ixgbe_adapter *adapter);
@@ -915,4 +938,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);void ixgbe_sriov_reinit (struct ixgbe_adapter *adapter);#endif +int ixgbe_get_settings (struct net_device *dev,struct ethtool_cmd *ecmd); +int ixgbe_write_uc_addr_list (struct net_device *netdev); +netdev_tx_t ixgbe_xmit_frame_ring (struct sk_buff *skb, +struct ixgbe_adapter *adapter, +struct ixgbe_ring *tx_ring); +void ixgbe_clean_rx_ring (struct ixgbe_ring *rx_ring);#endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = { };#define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN -static int ixgbe_get_settings (struct net_device *netdev, -struct ethtool_cmd *ecmd) +int ixgbe_get_settings (struct net_device *netdev, +struct ethtool_cmd *ecmd) {struct ixgbe_adapter *adapter =netdev_priv (netdev);struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h new file +54 -0
54 lines not on this page · raw message
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)#endif /* only proceed if SR-IOV is enabled */ -if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED )) +if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && + !(adapter->flags &IXGBE_FLAG_VMDQ_ENABLED ))return false;/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,/* apply Tx specific ring traits */ ring->count = adapter->tx_ring_count; - ring->queue_index = txr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + txr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = txr_idx;/* assign ring to adapter */ adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,#endif /* IXGBE_FCOE */ /* apply Rx specific ring traits */ ring->count = adapter->rx_ring_count; - ring->queue_index = rxr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + rxr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = rxr_idx;/* assign ring to adapter */ adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +288 -85
616 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [RFC PATCH 0/2 v2] net: alternate proposal for using macvlans with forwarding acceleration
In reply to Neil HormanQuoted 6 lines · Neil Horman
Hey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.Some change notes:
Quoted 7 lines
* Changes to the fowarding ops structure - Removed the
priv_sizefield, and added a flags field. Removal of thepriv_sizefield was accomplished by just having the add method return a void * and usingERR_PTRandPTR_ERRchecks, which also allows us to allocate memory for the acceleration path in the driver, which I like. I'm not super happy still with how I'm using the flags (currenly only used to indicate support for feature sets), but at least we have the flags now, and they can be exposed to user space via iproute2 or ethtool if need be
For the flag why not use the existing feature flag namespace? Adding NETIF_F_HW_L2FW_DOFFLOAD or something equivalent to netdev_features.h and also doing the string updates would get the userspace support for free.
The one downside of a global feature flag approach like this is if the user wants to create some offloaded macvlan devices and some SW only macvlans the control sequence is a bit clumsy. But unless we add a new flag to the macvlan netlink message I'm not sure how to avoid this. Further if you have multiple control applications creating/deleting these macvlans the flag mechanism starts to break down. One app sets the flag the other deletes, you get the idea.
It might be worth adding a netlink flag to change the state of the HW offload. I know this goes against the 'it just gets offloaded' line of thought, but if you make the default to offload then it should be OK.
Quoted 4 lines
* Changes to the Transmit path - Specifically I'm using
dev_queue_xmitto send frames now, which I like as it makes the macvlan subject to the lowerdevs qdisc configuration.
This creates perhaps another oddity where on TX the packets will be visible to the lowerdev. Think tcpdump and egress qdisc. But on ingress the packets will be delivered directly to macvlan device. Also I was thinking there might be good reasons to skip the lowerdev qdisc, for performance reasons or QOS setups.
So it might be best the way you had it in the previous revision even if it was not functionally equivalent to the SW path. If you skip the lower dev and submit directly to the hardware then you also don't need a sk_buff change. The driver can do a lookup via the skb->dev field and additional hints from the stack are not needed.
If there is a perceived problem with not having the HW and SW completely equivalent functionally we could always default the feature flag to off. Then enabling the feature would be a single 'ethtool' command. This seems like a nice compromise.
I like where this is going though!
Thanks,
John
Signature · 1 line
John Fastabend Intel Corporation
Re: [RFC PATCH 0/2 v2] net: alternate proposal for using macvlans with forwarding acceleration
Quoted 28 lines · John Fastabend
Quoted 6 lines · Neil Horman
Hey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.Some change notes:
Quoted 7 lines
* Changes to the fowarding ops structure - Removed the
priv_sizefield, and added a flags field. Removal of thepriv_sizefield was accomplished by just having the add method return a void * and usingERR_PTRandPTR_ERRchecks, which also allows us to allocate memory for the acceleration path in the driver, which I like. I'm not super happy still with how I'm using the flags (currenly only used to indicate support for feature sets), but at least we have the flags now, and they can be exposed to user space via iproute2 or ethtool if need beFor the flag why not use the existing feature flag namespace? Adding
NETIF_F_HW_L2FW_DOFFLOADor something equivalent tonetdev_features.hand also doing the string updates would get the userspace support for free.The one downside of a global feature flag approach like this is if the user wants to create some offloaded macvlan devices and some SW only macvlans the control sequence is a bit clumsy. But unless we add a new flag to the macvlan netlink message I'm not sure how to avoid this. Further if you have multiple control applications creating/deleting these macvlans the flag mechanism starts to break down. One app sets the flag the other deletes, you get the idea.
Quoted 5 lines
It might be worth adding a netlink flag to change the state of the HW offload. I know this goes against the 'it just gets offloaded' line of thought, but if you make the default to offload then it should be OK.
Good suggestions, I should have thought to use the global flags previously. I got hung up on the notion that for some reason our feature set should be separated like the ops. I really don't need to do that.
Quoted 11 lines
Quoted 4 lines
* Changes to the Transmit path - Specifically I'm using
dev_queue_xmitto send frames now, which I like as it makes the macvlan subject to the lowerdevs qdisc configuration.This creates perhaps another oddity where on TX the packets will be visible to the lowerdev. Think tcpdump and egress qdisc. But on ingress the packets will be delivered directly to macvlan device. Also I was thinking there might be good reasons to skip the lowerdev qdisc, for performance reasons or QOS setups.
Hmm, is that avoidable by doing an extra check in dev_queue_xmit (i.e. checking to see if accel_data or some other pointer is set)? Or do you think its worth separating the tx path to an accelerated or non accelerated case?
Quoted 6 lines
So it might be best the way you had it in the previous revision even if it was not functionally equivalent to the SW path. If you skip the lower dev and submit directly to the hardware then you also don't need a
sk_buffchange. The driver can do a lookup via theskb->devfield and additional hints from the stack are not needed.
Ok, fair enough, I'll put it back the way I had it previously.
Quoted 5 lines
If there is a perceived problem with not having the HW and SW completely equivalent functionally we could always default the feature flag to off. Then enabling the feature would be a single 'ethtool' command. This seems like a nice compromise.
Yeah, I can agree with that.
Quoted 1 line
I like where this is going though!
I'm glad you think so! I'll have another version out later this week! Best
Neil
Quoted 7 lines
Thanks,
JohnSignature · 1 line
John Fastabend Intel Corporation
v3 cover net: alternate proposal for using macvlans with forwarding acceleration
In reply to Neil HormanHey all-
heres the next, updated version of the vsi/macvlan integration that we've
been discussing.
Change notes:
* Moved the feature flag to netdev_features.h. No ethtool option for disabling it yet, but its there now, and seems to fit fairly well. I was actually thinking about your comment John, regarding the clumsiness in allowing sw and hw accel vlans on the same lowerdev, and it just occured to me that we could use the same flag on the macvlan device directly - i.e. if we found that a lowerdev supported acceleration, then call ndo_dfwd_station_add, and, if successfull, set the same feature flag in the macvlan device. Then we could use ethtool to control the enabling/disabling of acceleration at the macvlan device directly. Thoughts?
* Moved the acceleration net device methods back into net_device_ops. Looks pretty good to me there.
* Restored the use of a separate xmit routine so we weren't subject to the lowerdevs queue disciplines. I integrated its use with dev_hard_start_xmit, so we could share the use of the linearization code, etc. Let me know what you think.
Best
Neil
PATCH 1/2 net: Add layer 2 hardware acceleration operations for macvlan devices
Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.
--- drivers/net/macvlan.c | 31 +++++++++++++++++++++++++++++++ include/linux/if_macvlan.h | 1 + include/linux/netdev_features.h | 2 ++ include/linux/netdevice.h | 11 ++++++++++- include/linux/skbuff.h | 4 ++-- net/core/dev.c | 18 +++++++++++++----- net/core/ethtool.c | 1 + net/sched/sch_generic.c | 2 +- 8 files changed, 61 insertions(+), 9 deletions(-)
drivers/net/macvlan.c +31 -0
@@ -297,7 +297,16 @@ netdev_tx_t macvlan_start_xmit(struct sk_buff *skb,int ret;const struct macvlan_dev *vlan =netdev_priv (dev); +if (vlan->fwd_priv) { + skb->dev = vlan->lowerdev; + ret =dev_hard_start_xmit (skb, skb->dev,NULL , vlan->fwd_priv); + +if (likely (ret ==NETDEV_TX_OK )) + goto update_stats; + } + ret =macvlan_queue_xmit (skb, dev); +update_stats:if (likely (ret ==NET_XMIT_SUCCESS || ret ==NET_XMIT_CN )) {struct macvlan_pcpu_stats *pcpu_stats;
@@ -347,6 +356,18 @@ static int macvlan_open(struct net_device *dev) goto hash_add; } +if (lowerdev->features &NETIF_F_HW_L2FW_DOFFLOAD ) { + vlan->fwd_priv = lowerdev->netdev_ops->ndo_dfwd_add_station (lowerdev, dev); +/* +* If we get a NULL pointer back, or if we get an error +* then we should just fall through to the non accelerated path +*/ +if (IS_ERR_OR_NULL (vlan->fwd_priv)) + vlan->fwd_priv =NULL ; +else +return 0 ; + } + err = -EBUSY ;if (macvlan_addr_busy (vlan->port, dev->dev_addr)) goto out;
@@ -367,6 +388,10 @@ hash_add: del_unicast:dev_uc_del (lowerdev, dev->dev_addr); out: +if (vlan->fwd_priv) { + lowerdev->netdev_ops->ndo_dfwd_del_station (lowerdev, vlan->fwd_priv); + vlan->fwd_priv =NULL ; + }return err; }
@@ -391,6 +416,11 @@ static int macvlan_stop(struct net_device *dev) hash_del:macvlan_hash_del (vlan, !dev->dismantle); +if (vlan->fwd_priv) { + lowerdev->netdev_ops->ndo_dfwd_del_station (lowerdev, vlan->fwd_priv); + vlan->fwd_priv =NULL ; + } +return 0 ; }
@@ -801,6 +831,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev,if (err <0 )return err; } + port =macvlan_port_get_rtnl (lowerdev);/* Only 1 macvlan device can be created in passthru mode */
include/linux/if_macvlan.h +1 -0
@@ -61,6 +61,7 @@ struct macvlan_dev {struct hlist_node hlist;struct macvlan_port *port;struct net_device *lowerdev; +void *fwd_priv;struct macvlan_pcpu_stats __percpu *pcpu_stats;DECLARE_BITMAP (mc_filter,MACVLAN_MC_FILTER_SZ );
include/linux/netdev_features.h +2 -0
@@ -60,6 +60,7 @@ enum {NETIF_F_HW_VLAN_STAG_TX_BIT ,/* Transmit VLAN STAG HW acceleration */ NETIF_F_HW_VLAN_STAG_RX_BIT ,/* Receive VLAN STAG HW acceleration */ NETIF_F_HW_VLAN_STAG_FILTER_BIT ,/* Receive filtering on VLAN STAGs */ +NETIF_F_HW_L2FW_DOFFLOAD_BIT ,/* Allow L2 Forwarding in Hardware */ /* * Add your fresh new feature above and remember to update
@@ -112,6 +113,7 @@ enum {#define NETIF_F_HW_VLAN_STAG_FILTER __NETIF_F(HW_VLAN_STAG_FILTER)#define NETIF_F_HW_VLAN_STAG_RX __NETIF_F(HW_VLAN_STAG_RX)#define NETIF_F_HW_VLAN_STAG_TX __NETIF_F(HW_VLAN_STAG_TX) +#define NETIF_F_HW_L2FW_DOFFLOAD __NETIF_F(HW_L2FW_DOFFLOAD)/* Features valid for ethtool to change */ /* = all defined minus driver/device-class-related */
include/linux/netdevice.h +10 -1
@@ -1097,6 +1097,13 @@ struct net_device_ops {void (*ndo_del_vxlan_port)(struct net_device *dev,sa_family_t sa_family,__be16 port); + +void * (*ndo_dfwd_add_station)(struct net_device *pdev, +struct net_device *vdev); +void (*ndo_dfwd_del_station)(struct net_device *pdev,void *priv); + +netdev_tx_t (*ndo_dfwd_start_xmit) (struct sk_buff *skb, +struct net_device *dev,void *priv); };/*
@@ -1183,6 +1190,7 @@ struct net_device {/* Management operations */ const struct net_device_ops *netdev_ops;const struct ethtool_ops *ethtool_ops; +const struct forwarding_accel_ops *fwd_ops;/* Hardware header description */ const struct header_ops *header_ops;
@@ -2383,7 +2391,8 @@ extern int dev_get_phys_port_id(struct net_device *dev, struct netdev_phys_port_id *ppid);extern int dev_hard_start_xmit (struct sk_buff *skb,struct net_device *dev, -struct netdev_queue *txq); +struct netdev_queue *txq, +void *accel_priv);extern int dev_forward_skb (struct net_device *dev,struct sk_buff *skb);
include/linux/skbuff.h +2 -2
@@ -426,9 +426,9 @@ struct sk_buff {char cb[48 ]__aligned(8) ;unsigned long _skb_refdst; -#ifdef CONFIG_XFRM +struct sec_path *sp; -#endif +unsigned int len, data_len; __u16 mac_len,
net/core/dev.c +13 -5
@@ -2536,7 +2536,7 @@ static inline int skb_needs_linearize(struct sk_buff *skb, }int dev_hard_start_xmit (struct sk_buff *skb,struct net_device *dev, -struct netdev_queue *txq) +struct netdev_queue *txq,void *accel_priv) {const struct net_device_ops *ops = dev->netdev_ops;int rc =NETDEV_TX_OK ;
@@ -2602,9 +2602,13 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,dev_queue_xmit_nit (skb, dev); skb_len = skb->len; - rc = ops->ndo_start_xmit (skb, dev); +if (accel_priv) + rc = ops->ndo_dfwd_start_xmit (skb, dev, accel_priv); +else + rc = ops->ndo_start_xmit (skb, dev); +trace_net_dev_xmit (skb, rc, dev, skb_len); -if (rc ==NETDEV_TX_OK ) +if (rc ==NETDEV_TX_OK && txq)txq_trans_update (txq);return rc; }
@@ -2620,7 +2624,10 @@ gso:dev_queue_xmit_nit (nskb, dev); skb_len = nskb->len; - rc = ops->ndo_start_xmit (nskb, dev); +if (accel_priv) + rc = ops->ndo_dfwd_start_xmit (nskb, dev, accel_priv); +else + rc = ops->ndo_start_xmit (nskb, dev);trace_net_dev_xmit (nskb, rc, dev, skb_len);if (unlikely (rc !=NETDEV_TX_OK )) {if (rc & ~NETDEV_TX_MASK )
@@ -2645,6 +2652,7 @@ out_kfree_skb: out:return rc; } +EXPORT_SYMBOL_GPL (dev_hard_start_xmit);static void qdisc_pkt_len_init (struct sk_buff *skb) {
@@ -2852,7 +2860,7 @@ int dev_queue_xmit(struct sk_buff *skb)if (!netif_xmit_stopped (txq)) {__this_cpu_inc (xmit_recursion); - rc =dev_hard_start_xmit (skb, dev, txq); + rc =dev_hard_start_xmit (skb, dev, txq,NULL );__this_cpu_dec (xmit_recursion);if (dev_xmit_complete (rc)) {HARD_TX_UNLOCK (dev, txq);
net/core/ethtool.c +1 -0
@@ -94,6 +94,7 @@ static const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] [NETIF_F_LOOPBACK_BIT] = "loopback", [NETIF_F_RXFCS_BIT] = "rx-fcs", [NETIF_F_RXALL_BIT] = "rx-all", + [NETIF_F_HW_L2FW_DOFFLOAD_BIT] = "l2-fwd-offload", };static int ethtool_get_features (struct net_device *dev,void __user *useraddr)
net/sched/sch_generic.c +1 -1
@@ -126,7 +126,7 @@ int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,HARD_TX_LOCK (dev, txq,smp_processor_id ());if (!netif_xmit_frozen_or_stopped (txq)) - ret =dev_hard_start_xmit (skb, dev, txq); + ret =dev_hard_start_xmit (skb, dev, txq,NULL );HARD_TX_UNLOCK (dev, txq);
Signature · 1 line
1.8.3.1
Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
Quoted 11 lines · Neil Horman
Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.
Signed-off-by Neil Horman <[email protected]>CC Andy Gospodarek <[email protected]>CC "David S. Miller" <[email protected]>---
[...]
include/linux/skbuff.h struct sk_buff { · line 429
Quoted 11 lines
include/linux/skbuff.h +1
-1@@ -426,9 +426,9 @@ struct sk_buff {char cb[48 ]__aligned(8) ;unsigned long _skb_refdst; -#ifdef CONFIG_XFRM +
Is this a hold-over from the previous patches? 'sp' isn't touched anywhere else so put the ifdef/endif back.
Quoted 8 lines
include/linux/skbuff.h +1
-1struct sec_path *sp; -#endif +unsigned int len, data_len; __u16 mac_len,net/core/dev.c +0
-0
Signature · 1 line
John Fastabend Intel Corporation
Re: [PATCH 1/2] net: Add layer 2 hardware acceleration operations for macvlan devices
Quoted 23 lines · John Fastabend
Quoted 11 lines · Neil Horman
Add a operations structure that allows a network interface to export the fact that it supports package forwarding in hardware between physical interfaces and other mac layer devices assigned to it (such as macvlans). this operaions structure can be used by virtual mac devices to bypass software switching so that forwarding can be done in hardware more efficiently.
Signed-off-by Neil Horman <[email protected]>CC Andy Gospodarek <[email protected]>CC "David S. Miller" <[email protected]>---
[...]
Quoted 8 lines
include/linux/skbuff.h +0
-0@@ -426,9 +426,9 @@ struct sk_buff {char cb[48 ]__aligned(8) ;
Quoted 7 lines
include/linux/skbuff.h struct sk_buff { · line 429
Quoted 3 lines
include/linux/skbuff.h +1
-1unsigned long _skb_refdst; -#ifdef CONFIG_XFRM +Is this a hold-over from the previous patches? 'sp' isn't touched anywhere else so put the ifdef/endif back.
Yeah, my screw up, I wanted to get this out before the weekend and missed that screw up. Sorry.
Neil
Quoted 12 lines
Quoted 8 lines
include/linux/skbuff.h +1
-1struct sec_path *sp; -#endif +unsigned int len, data_len; __u16 mac_len,net/core/dev.c +0
-0Signature · 1 line
John Fastabend Intel Corporation
PATCH 2/2 ixgbe: enable l2 forwarding acceleration for macvlans
In reply to Neil HormanNow that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.
--- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 33 +- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 4 +- drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h | 54 ++++ drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 15 +- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 388 ++++++++++++++++++----- 5 files changed, 400 insertions(+), 94 deletions(-) create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h
drivers/net/ethernet/intel/ixgbe/ixgbe.h +31 -2
@@ -219,6 +219,17 @@ enum ixgbe_ring_state_t { __IXGBE_RX_FCOE, }; +struct ixgbe_fwd_adapter { +unsigned long active_vlans[BITS_TO_LONGS (VLAN_N_VID )]; +struct net_device *netdev; +struct ixgbe_adapter *real_adapter; +unsigned int tx_base_queue; +unsigned int rx_base_queue; +struct net_device_stats net_stats; +int pool; +bool online; +}; +#define check_for_tx_hang (ring) \ test_bit(__IXGBE_TX_DETECT_HANG, &(ring)->state)#define set_check_for_tx_hang (ring) \
@@ -236,6 +247,7 @@ struct ixgbe_ring {struct ixgbe_q_vector *q_vector;/* backpointer to host q_vector */ struct net_device *netdev;/* netdev ring belongs to */ struct device *dev;/* device for DMA mapping */ +struct ixgbe_fwd_adapter *l2_accel_priv;void *desc;/* descriptor ring memory */ union {struct ixgbe_tx_buffer *tx_buffer_info;
@@ -244,6 +256,7 @@ struct ixgbe_ring {unsigned long last_rx_timestamp;unsigned long state;u8 __iomem *tail; +struct net_device *vmdq_netdev;dma_addr_t dma;/* phys. address of descriptor ring */ unsigned int size;/* length in bytes */
@@ -288,11 +301,15 @@ enum ixgbe_ring_f_enum { };#define IXGBE_MAX_RSS_INDICES 16 -#define IXGBE_MAX_VMDQ_INDICES 64 +#define IXGBE_MAX_VMDQ_INDICES 32#define IXGBE_MAX_FDIR_INDICES 63/* based on q_vector limit */ #define IXGBE_MAX_FCOE_INDICES 8#define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)#define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1) +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_MAX_L2A_QUEUES 4 +#define IXGBE_BAD_L2A_QUEUE 3 +struct ixgbe_ring_feature {u16 limit;/* upper limit on feature indices */ u16 indices;/* current value of indices */
@@ -738,6 +755,7 @@ struct ixgbe_adapter {#endif /*CONFIG_DEBUG_FS*/ u8 default_up; +unsigned long fwd_bitmask;/* Bitmask indicating in use pools */ };struct ixgbe_fdir_filter {
@@ -879,9 +897,14 @@ static inline void ixgbe_dbg_adapter_exit(struct ixgbe_adapter *adapter) {}static inline void ixgbe_dbg_init (void ) {}static inline void ixgbe_dbg_exit (void ) {}#endif /* CONFIG_DEBUG_FS */ +static inline struct net_device *netdev_ring (const struct ixgbe_ring *ring) +{ +return ring->vmdq_netdev ? ring->vmdq_netdev : ring->netdev; +} +static inline struct netdev_queue *txring_txq (const struct ixgbe_ring *ring) { -return netdev_get_tx_queue (ring->netdev, ring->queue_index); +return netdev_get_tx_queue (netdev_ring (ring), ring->queue_index); }extern void ixgbe_ptp_init (struct ixgbe_adapter *adapter);
@@ -915,4 +938,10 @@ extern void ixgbe_ptp_check_pps_event(struct ixgbe_adapter *adapter, u32 eicr);void ixgbe_sriov_reinit (struct ixgbe_adapter *adapter);#endif +int ixgbe_get_settings (struct net_device *dev,struct ethtool_cmd *ecmd); +int ixgbe_write_uc_addr_list (struct net_device *netdev); +netdev_tx_t ixgbe_xmit_frame_ring (struct sk_buff *skb, +struct ixgbe_adapter *adapter, +struct ixgbe_ring *tx_ring); +void ixgbe_clean_rx_ring (struct ixgbe_ring *rx_ring);#endif /* _IXGBE_H_ */
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +2 -2
@@ -150,8 +150,8 @@ static const char ixgbe_gstrings_test[][ETH_GSTRING_LEN] = { };#define IXGBE_TEST_LEN sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN -static int ixgbe_get_settings (struct net_device *netdev, -struct ethtool_cmd *ecmd) +int ixgbe_get_settings (struct net_device *netdev, +struct ethtool_cmd *ecmd) {struct ixgbe_adapter *adapter =netdev_priv (netdev);struct ixgbe_hw *hw = &adapter->hw;
drivers/net/ethernet/intel/ixgbe/ixgbe_l2a.h new file +54 -0
54 lines not on this page · raw message
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c +12 -3
@@ -500,7 +500,8 @@ static bool ixgbe_set_sriov_queues(struct ixgbe_adapter *adapter)#endif /* only proceed if SR-IOV is enabled */ -if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED )) +if (!(adapter->flags &IXGBE_FLAG_SRIOV_ENABLED ) && + !(adapter->flags &IXGBE_FLAG_VMDQ_ENABLED ))return false;/* Add starting offset to total pool count */
@@ -852,7 +853,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,/* apply Tx specific ring traits */ ring->count = adapter->tx_ring_count; - ring->queue_index = txr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + txr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = txr_idx;/* assign ring to adapter */ adapter->tx_ring[txr_idx] = ring;
@@ -895,7 +900,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,#endif /* IXGBE_FCOE */ /* apply Rx specific ring traits */ ring->count = adapter->rx_ring_count; - ring->queue_index = rxr_idx; +if (adapter->num_rx_pools >1 ) + ring->queue_index = + rxr_idx % adapter->num_rx_queues_per_pool; +else + ring->queue_index = rxr_idx;/* assign ring to adapter */ adapter->rx_ring[rxr_idx] = ring;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +301 -87
634 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH 2/2] ixgbe: enable l2 forwarding acceleration for macvlans
Quoted 10 lines · Neil Horman
Now that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.
Signed-off-by Neil Horman <[email protected]>CC Andy Gospodarek <[email protected]>CC "David S. Miller" <[email protected]>---
Neil, I'm fairly sure I can simplify this patch further. I'll be able to take a shot at it Tuesday if you don't mind waiting.
I can resubmit the series then and preserve your signed-off on at least the first patch. Seem reasonable?
.John
Signature · 1 line
John Fastabend Intel Corporation
Re: [PATCH 2/2] ixgbe: enable l2 forwarding acceleration for macvlans
Quoted 12 lines · John Fastabend
Quoted 10 lines · Neil Horman
Now that l2 acceleration ops are in place from the prior patch, enable ixgbe to take advantage of these operations. Allow it to allocate queues for a macvlan so that when we transmit a frame, we can do the switching in hardware inside the ixgbe card, rather than in software.
Signed-off-by Neil Horman <[email protected]>CC Andy Gospodarek <[email protected]>CC "David S. Miller" <[email protected]>---
Quoted 6 lines
Neil, I'm fairly sure I can simplify this patch further. I'll be able to take a shot at it Tuesday if you don't mind waiting.
I can resubmit the series then and preserve your signed-off on at least the first patch. Seem reasonable?
That sounds fine.
Thanks!
Neil
Quoted 5 lines
.John
Signature · 1 line
John Fastabend Intel Corporation