Proposal for VRF-lite

From: Shrijeet Mukherjee <[email protected]>

This change is needed for the following VRF driver which creates a routing domain and allows applications to bind to the domain.

No active code path changes.

Signed-off-by Shrijeet Mukherjee <[email protected]>

---
  net/ipv4/fib_frontend.c |    1 +
  net/ipv4/fib_trie.c     |    2 ++
  2 files changed, 3 insertions(+)

net/ipv4/fib_frontend.c +0 -0

index 872494e..9d4cef4 100644

net/ipv4/fib_trie.c +2 -0

index 01bce15..97fa62d 100644
@@ -1247,6 +1247,7 @@ out:
  err:
  	return err;
  }
+EXPORT_SYMBOL_GPL(fib_table_insert);
 
  static inline t_key prefix_mismatch(t_key key, struct key_vector *n)
  {

@@ -1535,6 +1536,7 @@ int fib_table_delete(struct fib_table *tb, struct fib_config *cfg)
  	alias_free_mem_rcu(fa_to_delete);
  	return 0;
  }
+EXPORT_SYMBOL_GPL(fib_table_delete);
 
  /* Scan for the next leaf starting at the provided key value */
  static struct key_vector *leaf_walk_rcu(struct key_vector **tn, t_key key)

drivers/net/Kconfig +0 -0

index 019fcef..27a333c 100644
@@ -283,6 +283,12 @@ config NLMON
  	  diagnostics, etc. This is mostly intended for developers or support
  	  to debug netlink issues. If unsure, say N.

From: Shrijeet Mukherjee <[email protected]>

This driver borrows heavily from IPvlan and teaming drivers.

Routing domains (VRF-lite) are created by instantiating a device and enslaving all routed interfaces that participate in the domain. As part of the enslavement, all local routes pointing to enslaved devices are re-pointed to the vrf device, thus forcing outgoing sockets to bind to the vrf to function.

Standard FIB rules can then bind the VRF device to tables and regular fib rule processing is followed.

Routed traffic through the box, is fwded by using the VRF device as the IIF and following the IIF rule to a table which is mated with the VRF.

Locally originated traffic is directed at the VRF device using SO_BINDTODEVICE or cmsg headers. This in turn drops the packet into the xmit function of the vrf driver, which then completes the ip lookup and output.

This solution is completely orthogonal to namespaces and allow the L3 equivalent of vlans to exist allowing the routing space to be partitioned.

Example use is
ip link add vrf0 type vrf table 5
ip link set eth1 master vrf0
ip link set vrf0 up

ip rule add iif vrf0 table 5
ip rule add oif vrf0 table 5

TODO:
This changeset is for IPv4 only
Connected route management can be made much better, but is deferred to user space for now.

From: Shrijeet Mukherjee <[email protected]>

drivers/net/vrf.c new file +0 -0

From: Shrijeet Mukherjee <[email protected]>

This driver borrows heavily from IPvlan and teaming drivers.

Routing domains (VRF-lite) are created by instantiating a device and enslaving all routed interfaces that participate in the domain. As part of the enslavement, all local routes pointing to enslaved devices are re-pointed to the vrf device, thus forcing outgoing sockets to bind to the vrf to function.

Standard FIB rules can then bind the VRF device to tables and regular fib rule processing is followed.

Routed traffic through the box, is fwded by using the VRF device as the IIF and following the IIF rule to a table which is mated with the VRF.

Locally originated traffic is directed at the VRF device using SO_BINDTODEVICE or cmsg headers. This in turn drops the packet into the xmit function of the vrf driver, which then completes the ip lookup and output.

This solution is completely orthogonal to namespaces and allow the L3 equivalent of vlans to exist allowing the routing space to be partitioned.

Example use is
ip link add vrf0 type vrf table 5
ip link set eth1 master vrf0
ip link set vrf0 up

ip rule add iif vrf0 table 5
ip rule add oif vrf0 table 5

TODO:
This changeset is for IPv4 only
Connected route management can be made much better, but is deferred to user space for now.

Signed-off-by Shrijeet Mukherjee <[email protected]>

---
 drivers/net/Kconfig          |    6 +
 drivers/net/Makefile         |    1 +
 drivers/net/vrf.c            |  654 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/netdevice.h    |   10 +
 include/net/flow.h           |    1 +
 include/net/vrf.h            |   19 ++
 include/uapi/linux/if_link.h |    9 +
 7 files changed, 700 insertions(+)
 create mode 100644 drivers/net/vrf.c
 create mode 100644 include/net/vrf.h

drivers/net/Kconfig +6 -0

index 019fcef..27a333c 100644
@@ -283,6 +283,12 @@ config NLMON
          diagnostics, etc. This is mostly intended for developers or support
          to debug netlink issues. If unsure, say N.
 
+config NET_VRF
+       tristate "Virtual Routing and Forwarding (Lite)"
+       ---help---
+          This option enables the support for mapping interfaces into VRF's. The
+          support enables VRF devices
+
 endif # NET_CORE
 
 config SUNGEM_PHY

drivers/net/Makefile +1 -0

index c12cb22..ca16dd6 100644
@@ -25,6 +25,7 @@ obj-$(CONFIG_VIRTIO_NET) += virtio_net.o
 obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_GENEVE) += geneve.o
 obj-$(CONFIG_NLMON) += nlmon.o
+obj-$(CONFIG_NET_VRF) += vrf.o
 
 #
 # Networking Drivers

drivers/net/vrf.c new file +30 -0

new file mode 100644
index 0000000..08b3e79
@@ -0,0 +1,654 @@
+/*
+ * vrf.c: device driver to encapsulate a VRF space
+ *
+ * Copyright (c) 2015 Cumulus Networks
+ *
+ * Based on dummy, team and ipvlan drivers
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/netdevice.h>
+#include <linux/etherdevice.h>
+#include <linux/ip.h>
+#include <linux/init.h>
+#include <linux/moduleparam.h>
+#include <linux/rtnetlink.h>
+#include <net/rtnetlink.h>
+#include <net/arp.h>
+#include <linux/u64_stats_sync.h>
+#include <linux/hashtable.h>
+
+#include <linux/inetdevice.h>
+#include <net/ip.h>
+#include <net/ip_fib.h>
+#include <net/ip6_route.h>

drivers/net/vrf.c new file +237 -0

+
+#define vrf_master_get_rcu(dev) \
+       ((struct net_device *)rcu_dereference(dev->rx_handler_data))
+
+struct pcpu_dstats {
+       u64                     tx_pkts;
+       u64                     tx_bytes;
+       u64                     tx_drps;
+       u64                     rx_pkts;
+       u64                     rx_bytes;
+       struct u64_stats_sync   syncp;
+};
+
+struct slave {
+       struct list_head        list;
+       struct net_device       *dev;
+       long                    priority;
+};
+
+struct slave_queue {
+       spinlock_t              lock; /* lock for slave insert/delete */
+       struct list_head        all_slaves;
+       int                     num_slaves;
+       struct net_device       *master_dev;
+};
+
+struct net_vrf {
+       struct slave_queue      queue;
+       struct fib_table        *tb;
+       u32                     tb_id;
+};
+
+static int is_ip_rx_frame(struct sk_buff *skb)
+{
+       switch (skb->protocol) {
+       case htons(ETH_P_IP):
+       case htons(ETH_P_IPV6):
+               return 1;
+       }
+       return 0;
+}
+
+/* note: already called with rcu_read_lock */
+static rx_handler_result_t vrf_handle_frame(struct sk_buff **pskb)
+{
+       struct sk_buff *skb = *pskb;
+
+       if (is_ip_rx_frame(skb)) {
+               struct net_device *dev = vrf_master_get_rcu(skb->dev);
+               struct pcpu_dstats *dstats = this_cpu_ptr(dev->dstats);
+
+               u64_stats_update_begin(&dstats->syncp);
+               dstats->rx_pkts++;
+               dstats->rx_bytes += skb->len;
+               u64_stats_update_end(&dstats->syncp);
+       }
+       return RX_HANDLER_PASS;
+}
+
+static struct rtnl_link_stats64 *vrf_get_stats64(
+       struct net_device *dev, struct rtnl_link_stats64 *stats)
+{
+       int i;
+
+       for_each_possible_cpu(i) {
+               const struct pcpu_dstats *dstats;
+               u64 tbytes, tpkts, tdrops, rbytes, rpkts;
+               unsigned int start;
+
+               dstats = per_cpu_ptr(dev->dstats, i);
+               do {
+                       start = u64_stats_fetch_begin_irq(&dstats->syncp);
+                       tbytes = dstats->tx_bytes;
+                       tpkts = dstats->tx_pkts;
+                       tdrops = dstats->tx_drps;
+                       rbytes = dstats->rx_bytes;
+                       rpkts = dstats->rx_pkts;
+               } while (u64_stats_fetch_retry_irq(&dstats->syncp, start));
+               stats->tx_bytes += tbytes;
+               stats->tx_packets += tpkts;
+               stats->tx_dropped += tdrops;
+               stats->rx_bytes += rbytes;
+               stats->rx_packets += rpkts;
+       }
+       return stats;
+}
+
+static netdev_tx_t vrf_process_v6_outbound(struct sk_buff *skb,
+                                          struct net_device *dev)
+{
+       return 0;
+}
+
+static int _vrf_send_v4_prep(struct sk_buff *skb, struct flowi4 *fl4,
+                            struct net_device *vrf_dev)
+{
+       struct rtable *rt;
+       struct net_device *dev = skb->dev;
+
+       rt = ip_route_output_flow(dev_net(dev), fl4, NULL);
+
+       if (IS_ERR(rt))
+               goto err;
+
+       if ((rt->rt_type != RTN_UNICAST && rt->rt_type != RTN_LOCAL) ||
+           !rt->dst.dev->vrf_ptr) {
+               ip_rt_put(rt);
+               goto err;
+       }
+
+       /* prevent slave cross reference */
+       if (rt->dst.dev->vrf_ptr->ifindex != vrf_dev->ifindex) {
+               ip_rt_put(rt);
+               goto err;
+       }
+
+       skb_dst_drop(skb);
+       skb_dst_set(skb, &rt->dst);
+
+       return 0;
+err:
+       return 1;
+}
+
+static netdev_tx_t vrf_process_v4_outbound(struct sk_buff *skb,
+                                          struct net_device *vrf_dev)
+{
+       struct iphdr *ip4h = ip_hdr(skb);
+       int ret = NET_XMIT_DROP;
+       struct net_device *dev = skb->dev;
+       struct flowi4 fl4 = {
+               /* needed to match OIF rule */
+               .flowi4_oif = vrf_dev->ifindex,
+               .flowi4_iif = LOOPBACK_IFINDEX,
+               .flowi4_tos = RT_TOS(ip4h->tos),
+               .flowi4_flags = FLOWI_FLAG_ANYSRC | FLOWI_FLAG_VRFSRC,
+               .daddr = ip4h->daddr,
+               .saddr = 0,
+       };
+
+       if (_vrf_send_v4_prep(skb, &fl4, vrf_dev))
+               goto err;
+
+       dev = skb_dst(skb)->dev;
+       ip4h->saddr = inet_select_addr(dev, 0, RT_SCOPE_LINK);
+       ret = ip_local_out(skb);
+
+       if (unlikely(net_xmit_eval(ret)))
+               vrf_dev->stats.tx_errors++;
+       else
+               ret = NET_XMIT_SUCCESS;
+
+       goto out;
+err:
+       vrf_dev->stats.tx_errors++;
+       kfree_skb(skb);
+out:
+       return ret;
+}
+
+static netdev_tx_t is_ip_tx_frame(struct sk_buff *skb, struct net_device *dev)
+{
+       /* the frames we recv have full L2 headers, strip */
+       if (skb_mac_header_was_set(skb)) {
+               skb_pull(skb, sizeof(struct ethhdr));
+               skb->mac_header = (typeof(skb->mac_header))~0U;
+               skb_reset_network_header(skb);
+       }
+
+       switch (skb->protocol) {
+       case htons(ETH_P_IP):
+               return vrf_process_v4_outbound(skb, dev);
+       case htons(ETH_P_IPV6):
+               return vrf_process_v6_outbound(skb, dev);
+       default:
+               return NET_XMIT_DROP;
+       }
+}
+
+static netdev_tx_t vrf_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+       netdev_tx_t ret = is_ip_tx_frame(skb, dev);
+
+       if (likely(ret == NET_XMIT_SUCCESS || ret == NET_XMIT_CN)) {
+               struct pcpu_dstats *dstats = this_cpu_ptr(dev->dstats);
+
+               u64_stats_update_begin(&dstats->syncp);
+               dstats->tx_pkts++;
+               dstats->tx_bytes += skb->len;
+               u64_stats_update_end(&dstats->syncp);
+       } else {
+               this_cpu_inc(dev->dstats->tx_drps);
+       }
+
+       return ret;
+}
+
+int vrf_output(struct sock *sk, struct sk_buff *skb)
+{
+       struct net_device *dev = skb_dst(skb)->dev;
+
+       return vrf_xmit(skb, dev);
+}
+
+/**************************** device handling ********************/
+
+/* queue->lock must be held */
+static struct slave *__vrf_find_slave_dev(struct slave_queue *queue,
+                                         struct net_device *dev)
+{
+       struct list_head *this, *head;
+
+       head = &queue->all_slaves;
+       list_for_each(this, head) {
+               struct slave *slave = list_entry(this, struct slave, list);
+
+               if (slave->dev == dev)
+                       return slave;
+       }
+
+       return NULL;
+}
+
+static void vrf_kill_one_slave(struct slave_queue *queue, struct slave *slave)
+{
+       list_del(&slave->list);
+       queue->num_slaves--;
+       slave->dev->flags &= ~IFF_SLAVE;
+       netdev_rx_handler_unregister(slave->dev);
+       kfree(slave->dev->vrf_ptr);
+       slave->dev->vrf_ptr = NULL;
+       dev_put(slave->dev);
+       kfree(slave);
+}
+
+/* queue->lock must be held */
+static int __vrf_insert_slave(struct slave_queue *queue, struct slave *slave,

drivers/net/vrf.c new file +122 -0

+       if (!slave->dev->vrf_ptr)
+               return -ENODEV;
+       slave->dev->vrf_ptr->ifindex = master_ifindex;
+       slave->dev->vrf_ptr->tb_id = vrf->tb_id;
+
+       /* register the packet handler for slave ports */
+       err = netdev_rx_handler_register(slave->dev, vrf_handle_frame,
+                                        (void *)master);
+       if (err) {
+               netdev_err(slave->dev,
+                          "Device %s failed to register rx_handler\n",
+                          slave->dev->name);
+               return err;
+       }
+
+       return 0;
+}
+
+static void vrf_fib_magic(int cmd, int type, __be32 dst, int dst_len,
+                         struct in_ifaddr *ifa, struct net_device *vrf_dev)
+{
+       struct net_vrf *vrf = netdev_priv(vrf_dev);
+       struct net *net = dev_net(ifa->ifa_dev->dev);
+       struct net *vrf_net = dev_net(vrf_dev);
+       struct fib_table *tb, *lc_tb;
+       struct fib_config cfg = {
+               .fc_protocol = RTPROT_KERNEL,
+               .fc_type = type,
+               .fc_dst = dst,
+               .fc_dst_len = dst_len,
+               .fc_prefsrc = ifa->ifa_local,
+               .fc_nlflags = NLM_F_CREATE | NLM_F_APPEND,
+               .fc_nlinfo = {
+                       .nl_net = net,
+               },
+       };
+
+       lc_tb = fib_new_table(dev_net(vrf_dev), RT_TABLE_LOCAL);
+       tb = vrf->tb;
+       if (!tb || !lc_tb)
+               return;
+
+       if (net != vrf_net)
+               return;
+
+       if (type != RTN_LOCAL)
+               cfg.fc_scope = RT_SCOPE_LINK;
+       else
+               cfg.fc_scope = RT_SCOPE_HOST;
+
+       if (cmd == RTM_NEWROUTE) {
+               cfg.fc_table = lc_tb->tb_id;
+               cfg.fc_oif = ifa->ifa_dev->dev->ifindex;
+               fib_table_delete(lc_tb, &cfg);
+               cfg.fc_table = tb->tb_id;
+               cfg.fc_oif = vrf_dev->ifindex;
+               fib_table_insert(tb, &cfg);
+       } else {
+               cfg.fc_table = tb->tb_id;
+               cfg.fc_oif = vrf_dev->ifindex;
+               fib_table_delete(tb, &cfg);
+               cfg.fc_table = lc_tb->tb_id;
+               cfg.fc_oif = ifa->ifa_dev->dev->ifindex;
+               fib_table_insert(lc_tb, &cfg);
+       }
+}
+
+static void vrf_move_local_routes(int cmd, struct net_device *dev,
+                                 struct net_device *port_dev)
+{
+       struct in_device *in_dev = __in_dev_get_rcu(port_dev);
+
+       if (!in_dev)
+               return;
+
+       for_ifa(in_dev) {
+               __be32 addr;
+
+               addr = ifa->ifa_local;
+               vrf_fib_magic(cmd, RTN_LOCAL, addr, 32, ifa, dev);
+       } endfor_ifa(in_dev);
+}
+
+static int vrf_inetaddr_event(struct notifier_block *this, unsigned long event,
+                             void *ptr)
+{
+       struct in_ifaddr *ifa = (struct in_ifaddr *)ptr;
+       struct net_device *port_dev = ifa->ifa_dev->dev;
+       struct net *net = dev_net(port_dev);
+       int master = 0;
+
+       if (port_dev->vrf_ptr)
+               master = port_dev->vrf_ptr->ifindex;
+
+       if (master) {
+               struct net_device *dev = dev_get_by_index(net, master);
+
+               switch (event) {
+               case NETDEV_UP:
+                       vrf_move_local_routes(RTM_NEWROUTE, dev, port_dev);
+                       break;
+               case NETDEV_DOWN:
+                       vrf_move_local_routes(RTM_DELROUTE, dev, port_dev);
+                       break;
+               }
+       }
+       return NOTIFY_DONE;
+}
+
+static struct notifier_block vrf_inetaddr_notifier = {
+       .notifier_call = vrf_inetaddr_event,
+};
+
+/* netlink lock is assumed here */
+static int vrf_add_slave(struct net_device *dev,
+                        struct net_device *port_dev)
+{
+       if (!dev || !port_dev)
+               return -ENODEV;
+
+       if (dev_net(dev) != dev_net(port_dev))
+               return -ENODEV;

drivers/net/vrf.c new file +174 -0

+               s->dev = port_dev;
+
+               spin_lock_bh(&vrf->queue.lock);
+               ret = __vrf_insert_slave(&vrf->queue, s, dev);
+               if (ret)
+                       kfree(s);
+
+               spin_unlock_bh(&vrf->queue.lock);
+
+               vrf_move_local_routes(RTM_NEWROUTE, dev, port_dev);
+               ret = netdev_master_upper_dev_link(port_dev, dev);
+               return ret;
+       }
+
+       return -EINVAL;
+}
+
+static int vrf_del_slave(struct net_device *dev,
+                        struct net_device *port_dev)
+{
+       struct net_vrf *vrf = netdev_priv(dev);
+       struct slave_queue *queue = &vrf->queue;
+       struct slave *slave = __vrf_find_slave_dev(queue, port_dev);
+
+       if (!slave)
+               return -EINVAL;
+
+       vrf_kill_one_slave(queue, slave);
+       vrf_move_local_routes(RTM_DELROUTE, dev, port_dev);
+       netdev_upper_dev_unlink(port_dev, dev);
+
+       return 0;
+}
+
+static int vrf_dev_init(struct net_device *dev)
+{
+       struct net_vrf *vrf = netdev_priv(dev);
+
+       spin_lock_init(&vrf->queue.lock);
+       INIT_LIST_HEAD(&vrf->queue.all_slaves);
+       vrf->queue.master_dev = dev;
+
+       dev->dstats = netdev_alloc_pcpu_stats(struct pcpu_dstats);
+       dev->flags  =  IFF_MASTER | IFF_NOARP;
+       if (!dev->dstats)
+               return -ENOMEM;
+
+       return 0;
+}
+
+static void vrf_dev_uninit(struct net_device *dev)
+{
+       free_percpu(dev->dstats);
+}
+
+static int vrf_dev_close(struct net_device *dev)
+{
+       struct net_vrf *vrf = netdev_priv(dev);
+       struct slave_queue *queue = &vrf->queue;
+       struct list_head *this, *head;
+
+       head = &queue->all_slaves;
+       list_for_each(this, head) {
+               struct slave *slave = list_entry(this, struct slave, list);
+
+               slave->dev->vrf_ptr->ifindex = 0;
+               slave->dev->vrf_ptr->tb_id = 0;
+       }
+
+       if (dev->flags & IFF_MASTER)
+               dev->flags &= ~IFF_UP;
+/* XXX does the table not need a free
+ * fib_table_delete(vrf->tb, cfg);
+ */
+       return 0;
+}
+
+static int vrf_dev_open(struct net_device *dev)
+{
+       struct net_vrf *vrf = netdev_priv(dev);
+       struct slave_queue *queue = &vrf->queue;
+       struct list_head *this, *head;
+       int err = 0;
+
+       head = &queue->all_slaves;
+       list_for_each(this, head) {
+               struct slave *slave = list_entry(this, struct slave, list);
+
+               slave->dev->vrf_ptr->ifindex = dev->ifindex;
+               slave->dev->vrf_ptr->tb_id = vrf->tb_id;
+       }
+
+       if (dev->flags & IFF_MASTER)
+               dev->flags |= IFF_UP;
+
+       if (!vrf->tb)
+               return -EINVAL;
+
+       return err;
+}
+
+static int vrf_neigh_create(struct neighbour *n)
+{
+       n->nud_state = NUD_REACHABLE;
+       n->dead      = 0;
+
+       return 0;
+}
+
+static const struct net_device_ops vrf_netdev_ops = {
+       .ndo_init               = vrf_dev_init,
+       .ndo_uninit             = vrf_dev_uninit,
+       .ndo_open               = vrf_dev_open,
+       .ndo_stop               = vrf_dev_close,
+       .ndo_start_xmit         = vrf_xmit,
+       .ndo_get_stats64        = vrf_get_stats64,
+       .ndo_add_slave          = vrf_add_slave,
+       .ndo_del_slave          = vrf_del_slave,
+       .ndo_neigh_construct    = vrf_neigh_create,
+};
+
+static void vrf_get_drvinfo(struct net_device *dev,
+                           struct ethtool_drvinfo *info)
+{
+       strlcpy(info->driver, DRV_NAME, sizeof(info->driver));
+       strlcpy(info->version, DRV_VERSION, sizeof(info->version));
+}
+
+static const struct ethtool_ops vrf_ethtool_ops = {
+       .get_drvinfo            = vrf_get_drvinfo,
+};
+
+static void vrf_setup(struct net_device *dev)
+{
+       ether_setup(dev);
+
+       /* Initialize the device structure. */
+       dev->netdev_ops = &vrf_netdev_ops;
+       dev->ethtool_ops = &vrf_ethtool_ops;
+       dev->destructor = free_netdev;
+
+       /* Fill in device structure with ethernet-generic values. */
+       dev->tx_queue_len = 0;
+       eth_hw_addr_random(dev);
+}
+
+static int vrf_validate(struct nlattr *tb[], struct nlattr *data[])
+{
+       if (tb[IFLA_ADDRESS]) {
+               if (nla_len(tb[IFLA_ADDRESS]) != ETH_ALEN)
+                       return -EINVAL;
+               if (!is_valid_ether_addr(nla_data(tb[IFLA_ADDRESS])))
+                       return -EADDRNOTAVAIL;
+       }
+       return 0;
+}
+
+static int vrf_newlink(struct net *src_net, struct net_device *dev,
+                      struct nlattr *tb[], struct nlattr *data[])
+{
+       int err;
+       struct net_vrf *vrf = netdev_priv(dev);
+
+       if (data && data[IFLA_VRF_TABLE]) {
+               vrf->tb_id = nla_get_u32(data[IFLA_VRF_TABLE]);
+               /* reserve a table for this VRF device */
+               vrf->tb = fib_new_table(dev_net(dev), vrf->tb_id);
+               if (!vrf->tb)
+                       return -ERANGE;
+
+               dev->priv_flags |= IFF_VRF_MASTER;
+               err = register_netdevice(dev);
+               if (err) {
+                       free_netdev(dev);

drivers/net/vrf.c new file +10 -0

+}
+
+static const struct nla_policy vrf_nl_policy[IFLA_VRF_MAX + 1] = {
+       [IFLA_VRF_TABLE] = { .type = NLA_U32 },
+};
+
+static struct rtnl_link_ops vrf_link_ops __read_mostly = {
+       .kind           = DRV_NAME,
+       .priv_size      = sizeof(struct net_vrf),
+       .policy         = vrf_nl_policy,

drivers/net/vrf.c new file +24 -0

+
+       rtnl_lock();
+       err = __rtnl_link_register(&vrf_link_ops);
+       if (err < 0)
+               goto out;
+
+       register_inetaddr_notifier(&vrf_inetaddr_notifier);
+
+out:
+       rtnl_unlock();
+       return err;
+}
+
+static void __exit vrf_cleanup_module(void)
+{
+       unregister_inetaddr_notifier(&vrf_inetaddr_notifier);
+       rtnl_link_unregister(&vrf_link_ops);
+}
+
+module_init(vrf_init_module);
+module_exit(vrf_cleanup_module);
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_RTNL_LINK(DRV_NAME);
+MODULE_VERSION(DRV_VERSION);

include/linux/netdevice.h +10 -0

index 51f8d2f..29febf3 100644
@@ -51,6 +51,7 @@
 #include <linux/neighbour.h>
 #include <uapi/linux/netdevice.h>
 #include <uapi/linux/if_bonding.h>
+#include <net/vrf.h>
 
 struct netpoll_info;
 struct device;

@@ -1270,6 +1271,7 @@ enum netdev_priv_flags {
        IFF_XMIT_DST_RELEASE_PERM       = 1<<22,
        IFF_IPVLAN_MASTER               = 1<<23,
        IFF_IPVLAN_SLAVE                = 1<<24,
+       IFF_VRF_MASTER                  = 1<<25,
 };
 
 #define IFF_802_1Q_VLAN                        IFF_802_1Q_VLAN

@@ -1297,6 +1299,7 @@ enum netdev_priv_flags {
 #define IFF_XMIT_DST_RELEASE_PERM      IFF_XMIT_DST_RELEASE_PERM
 #define IFF_IPVLAN_MASTER              IFF_IPVLAN_MASTER
 #define IFF_IPVLAN_SLAVE               IFF_IPVLAN_SLAVE
+#define IFF_VRF_MASTER                 IFF_VRF_MASTER
 
 /**
  *     struct net_device - The DEVICE structure.

@@ -1413,6 +1416,7 @@ enum netdev_priv_flags {
  *     @dn_ptr:        DECnet specific data
  *     @ip6_ptr:       IPv6 specific data
  *     @ax25_ptr:      AX.25 specific data
+ *     @vrf_ptr:       VRF specific data
  *     @ieee80211_ptr: IEEE 802.11 specific data, assign before registering
  *
  *     @last_rx:       Time of last Rx

@@ -1625,6 +1629,7 @@ struct net_device {
        struct dn_dev __rcu     *dn_ptr;
        struct inet6_dev __rcu  *ip6_ptr;
        void                    *ax25_ptr;
+       struct net_vrf_dev      *vrf_ptr;
        struct wireless_dev     *ieee80211_ptr;
        struct wpan_dev         *ieee802154_ptr;
 #if IS_ENABLED(CONFIG_MPLS_ROUTING)

@@ -3776,6 +3781,11 @@ static inline bool netif_supports_nofcs(struct net_device *dev)
        return dev->priv_flags & IFF_SUPP_NOFCS;
 }
 
+static inline bool netif_is_vrf(struct net_device *dev)
+{
+       return dev->priv_flags & IFF_VRF_MASTER;
+}
+
 /* This device needs to keep skb dst for qdisc enqueue or ndo_start_xmit() */
 static inline void netif_keep_dst(struct net_device *dev)
 {

include/net/flow.h +1 -0

index 8109a15..69aaa99 100644
@@ -29,6 +29,7 @@ struct flowi_common {
        __u8    flowic_flags;
 #define FLOWI_FLAG_ANYSRC              0x01
 #define FLOWI_FLAG_KNOWN_NH            0x02
+#define FLOWI_FLAG_VRFSRC              0x04
        __u32   flowic_secid;
 };
 

include/net/vrf.h new file +19 -0

new file mode 100644
index 0000000..11d7dbf8
@@ -0,0 +1,19 @@
+/*
+ * include/net/net_vrf.h - adds vrf dev structure definitions
+ * Copyright (c) 2015 Cumulus Networks
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef __LINUX_NET_VRF_H
+#define __LINUX_NET_VRF_H
+
+struct net_vrf_dev {
+       int                     ifindex; /* ifindex of master dev */
+       u32                     tb_id;
+};
+
+#endif /* __LINUX_NET_VRF_H */

include/uapi/linux/if_link.h +9 -1

index afccc93..b98a443 100644
@@ -339,6 +339,15 @@ enum macvlan_macaddr_mode {
 
 #define MACVLAN_FLAG_NOPROMISC 1
 
+/* VRF section */
+enum {
+       IFLA_VRF_UNSPEC,
+       IFLA_VRF_TABLE,
+       __IFLA_VRF_MAX
+};
+
+#define IFLA_VRF_MAX (__IFLA_VRF_MAX - 1)
+
 /* IPVLAN section */
 enum {
        IFLA_IPVLAN_UNSPEC,
--

1.7.10.4

On Mon, Jun 8, 2015 at 8:35 PM, Shrijeet Mukherjee <[email protected]> wrote:

vrf_kill_one_slave() calls netdev_rx_handler_unregister() which does synchronize_rcu() and
here you're running with the queue spinlock held and softirqs disabled.

drivers/net/vrf.c line 294

Quoted 7 lines

drivers/net/vrf.c new file +7 -0

+
+       dev_hold(slave->dev);
+       list_add(&slave->list, &queue->all_slaves);
+       queue->num_slaves++;
+       slave->dev->flags |= IFF_SLAVE;
+
+       slave->dev->vrf_ptr = kmalloc(sizeof(*slave->dev->vrf_ptr), GFP_KERNEL);

Again this runs with a spinlock and softirqs disabled, GFP_KERNEL can sleep.

drivers/net/vrf.c line 610

Quoted 4 lines

drivers/net/vrf.c new file +4 -0

+static void vrf_dellink(struct net_device *dev, struct list_head *head)
+{
+       /* Need to free the table ? */
+       unregister_netdev(dev);

I think ->dellink() runs with rtnl held and unregister_netdev() tries to acquire rtnl
so you'll probably deadlock here.

Quoted 1 line

+}

From: Shrijeet Mukherjee <[email protected]>

This driver borrows heavily from IPvlan and teaming drivers.

Routing domains (VRF-lite) are created by instantiating a device and enslaving all routed interfaces that participate in the domain. As part of the enslavement, all local routes pointing to enslaved devices are re-pointed to the vrf device, thus forcing outgoing sockets to bind to the vrf to function.

Standard FIB rules can then bind the VRF device to tables and regular fib rule processing is followed.

Routed traffic through the box, is fwded by using the VRF device as the IIF and following the IIF rule to a table which is mated with the VRF.

Locally originated traffic is directed at the VRF device using SO_BINDTODEVICE or cmsg headers. This in turn drops the packet into the xmit function of the vrf driver, which then completes the ip lookup and output.

This solution is completely orthogonal to namespaces and allow the L3 equivalent of vlans to exist allowing the routing space to be partitioned.

Example use is
ip link add vrf0 type vrf table 5
ip link set eth1 master vrf0
ip link set vrf0 up

ip rule add iif vrf0 table 5
ip rule add oif vrf0 table 5

TODO:
This changeset is for IPv4 only
Connected route management can be made much better, but is deferred to user space for now.

Signed-off-by Shrijeet Mukherjee <[email protected]>

---
  drivers/net/Kconfig          |    6 +
  drivers/net/Makefile         |    1 +
  drivers/net/vrf.c            |  654 ++++++++++++++++++++++++++++++++++++++++++
  include/linux/netdevice.h    |   10 +
  include/net/flow.h           |    1 +
  include/net/vrf.h            |   19 ++
  include/uapi/linux/if_link.h |    9 +
  7 files changed, 700 insertions(+)

create mode 100644 drivers/net/vrf.c
create mode 100644 include/net/vrf.h

drivers/net/Kconfig +6 -0

index 019fcef..27a333c 100644
@@ -283,6 +283,12 @@ config NLMON
  	  diagnostics, etc. This is mostly intended for developers or support
  	  to debug netlink issues. If unsure, say N.
 
+config NET_VRF
+	tristate "Virtual Routing and Forwarding (Lite)"
+	---help---
+          This option enables the support for mapping interfaces into VRF's. The
+          support enables VRF devices
+
  endif # NET_CORE
 
  config SUNGEM_PHY

drivers/net/Makefile +1 -0

index c12cb22..ca16dd6 100644
@@ -25,6 +25,7 @@ obj-$(CONFIG_VIRTIO_NET) += virtio_net.o
  obj-$(CONFIG_VXLAN) += vxlan.o
  obj-$(CONFIG_GENEVE) += geneve.o
  obj-$(CONFIG_NLMON) += nlmon.o
+obj-$(CONFIG_NET_VRF) += vrf.o
 
  #
  # Networking Drivers

drivers/net/vrf.c new file +42 -0

new file mode 100644
index 0000000..08b3e79
@@ -0,0 +1,654 @@
+/*
+ * vrf.c: device driver to encapsulate a VRF space
+ *
+ * Copyright (c) 2015 Cumulus Networks
+ *
+ * Based on dummy, team and ipvlan drivers
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/netdevice.h>
+#include <linux/etherdevice.h>
+#include <linux/ip.h>
+#include <linux/init.h>
+#include <linux/moduleparam.h>
+#include <linux/rtnetlink.h>
+#include <net/rtnetlink.h>
+#include <net/arp.h>
+#include <linux/u64_stats_sync.h>
+#include <linux/hashtable.h>
+
+#include <linux/inetdevice.h>
+#include <net/ip.h>
+#include <net/ip_fib.h>
+#include <net/ip6_route.h>
+#include <net/rtnetlink.h>
+#include <net/route.h>
+#include <net/addrconf.h>
+#include <net/vrf.h>
+
+#define DRV_NAME	"vrf"
+#define DRV_VERSION	"1.0"
+
+#define vrf_is_slave(dev)   ((dev->flags & IFF_SLAVE) == IFF_SLAVE)
+#define vrf_is_master(dev)  ((dev->flags & IFF_MASTER) == IFF_MASTER)
+
+#define vrf_master_get_rcu(dev) \

drivers/net/vrf.c new file +19 -0

+struct slave {
+	struct list_head	list;
+	struct net_device	*dev;
+	long			priority;
+};
+
+struct slave_queue {
+	spinlock_t		lock; /* lock for slave insert/delete */
+	struct list_head	all_slaves;
+	int			num_slaves;
+	struct net_device	*master_dev;
+};
+
+struct net_vrf {
+	struct slave_queue	queue;
+	struct fib_table        *tb;
+	u32                     tb_id;
+};
+

drivers/net/vrf.c new file +145 -0

+/* note: already called with rcu_read_lock */
+static rx_handler_result_t vrf_handle_frame(struct sk_buff **pskb)
+{
+	struct sk_buff *skb = *pskb;
+
+	if (is_ip_rx_frame(skb)) {
+		struct net_device *dev = vrf_master_get_rcu(skb->dev);
+		struct pcpu_dstats *dstats = this_cpu_ptr(dev->dstats);
+
+		u64_stats_update_begin(&dstats->syncp);
+		dstats->rx_pkts++;
+		dstats->rx_bytes += skb->len;
+		u64_stats_update_end(&dstats->syncp);
+	}
+	return RX_HANDLER_PASS;
+}
+
+static struct rtnl_link_stats64 *vrf_get_stats64(
+	struct net_device *dev, struct rtnl_link_stats64 *stats)
+{
+	int i;
+
+	for_each_possible_cpu(i) {
+		const struct pcpu_dstats *dstats;
+		u64 tbytes, tpkts, tdrops, rbytes, rpkts;
+		unsigned int start;
+
+		dstats = per_cpu_ptr(dev->dstats, i);
+		do {
+			start = u64_stats_fetch_begin_irq(&dstats->syncp);
+			tbytes = dstats->tx_bytes;
+			tpkts = dstats->tx_pkts;
+			tdrops = dstats->tx_drps;
+			rbytes = dstats->rx_bytes;
+			rpkts = dstats->rx_pkts;
+		} while (u64_stats_fetch_retry_irq(&dstats->syncp, start));
+		stats->tx_bytes += tbytes;
+		stats->tx_packets += tpkts;
+		stats->tx_dropped += tdrops;
+		stats->rx_bytes += rbytes;
+		stats->rx_packets += rpkts;
+	}
+	return stats;
+}
+
+static netdev_tx_t vrf_process_v6_outbound(struct sk_buff *skb,
+					   struct net_device *dev)
+{
+	return 0;
+}
+
+static int _vrf_send_v4_prep(struct sk_buff *skb, struct flowi4 *fl4,
+			     struct net_device *vrf_dev)
+{
+	struct rtable *rt;
+	struct net_device *dev = skb->dev;
+
+	rt = ip_route_output_flow(dev_net(dev), fl4, NULL);
+
+	if (IS_ERR(rt))
+		goto err;
+
+	if ((rt->rt_type != RTN_UNICAST && rt->rt_type != RTN_LOCAL) ||
+	    !rt->dst.dev->vrf_ptr) {
+		ip_rt_put(rt);
+		goto err;
+	}
+
+	/* prevent slave cross reference */
+	if (rt->dst.dev->vrf_ptr->ifindex != vrf_dev->ifindex) {
+		ip_rt_put(rt);
+		goto err;
+	}
+
+	skb_dst_drop(skb);
+	skb_dst_set(skb, &rt->dst);
+
+	return 0;
+err:
+	return 1;
+}
+
+static netdev_tx_t vrf_process_v4_outbound(struct sk_buff *skb,
+					   struct net_device *vrf_dev)
+{
+	struct iphdr *ip4h = ip_hdr(skb);
+	int ret = NET_XMIT_DROP;
+	struct net_device *dev = skb->dev;
+	struct flowi4 fl4 = {
+		/* needed to match OIF rule */
+		.flowi4_oif = vrf_dev->ifindex,
+		.flowi4_iif = LOOPBACK_IFINDEX,
+		.flowi4_tos = RT_TOS(ip4h->tos),
+		.flowi4_flags = FLOWI_FLAG_ANYSRC | FLOWI_FLAG_VRFSRC,
+		.daddr = ip4h->daddr,
+		.saddr = 0,
+	};
+
+	if (_vrf_send_v4_prep(skb, &fl4, vrf_dev))
+		goto err;
+
+	dev = skb_dst(skb)->dev;
+	ip4h->saddr = inet_select_addr(dev, 0, RT_SCOPE_LINK);
+	ret = ip_local_out(skb);
+
+	if (unlikely(net_xmit_eval(ret)))
+		vrf_dev->stats.tx_errors++;
+	else
+		ret = NET_XMIT_SUCCESS;
+
+	goto out;
+err:
+	vrf_dev->stats.tx_errors++;
+	kfree_skb(skb);
+out:
+	return ret;
+}
+
+static netdev_tx_t is_ip_tx_frame(struct sk_buff *skb, struct net_device *dev)
+{
+	/* the frames we recv have full L2 headers, strip */
+	if (skb_mac_header_was_set(skb)) {
+		skb_pull(skb, sizeof(struct ethhdr));
+		skb->mac_header = (typeof(skb->mac_header))~0U;
+		skb_reset_network_header(skb);
+	}
+
+	switch (skb->protocol) {
+	case htons(ETH_P_IP):
+		return vrf_process_v4_outbound(skb, dev);
+	case htons(ETH_P_IPV6):
+		return vrf_process_v6_outbound(skb, dev);
+	default:
+		return NET_XMIT_DROP;
+	}
+}
+
+static netdev_tx_t vrf_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+	netdev_tx_t ret = is_ip_tx_frame(skb, dev);
+
+	if (likely(ret == NET_XMIT_SUCCESS || ret == NET_XMIT_CN)) {
+		struct pcpu_dstats *dstats = this_cpu_ptr(dev->dstats);
+
+		u64_stats_update_begin(&dstats->syncp);

drivers/net/vrf.c new file +89 -0

+int vrf_output(struct sock *sk, struct sk_buff *skb)
+{
+	struct net_device *dev = skb_dst(skb)->dev;
+
+	return vrf_xmit(skb, dev);
+}
+
+/**************************** device handling ********************/
+
+/* queue->lock must be held */
+static struct slave *__vrf_find_slave_dev(struct slave_queue *queue,
+					  struct net_device *dev)
+{
+	struct list_head *this, *head;
+
+	head = &queue->all_slaves;
+	list_for_each(this, head) {
+		struct slave *slave = list_entry(this, struct slave, list);
+
+		if (slave->dev == dev)
+			return slave;
+	}
+
+	return NULL;
+}
+
+static void vrf_kill_one_slave(struct slave_queue *queue, struct slave *slave)
+{
+	list_del(&slave->list);
+	queue->num_slaves--;
+	slave->dev->flags &= ~IFF_SLAVE;
+	netdev_rx_handler_unregister(slave->dev);
+	kfree(slave->dev->vrf_ptr);
+	slave->dev->vrf_ptr = NULL;
+	dev_put(slave->dev);
+	kfree(slave);
+}
+
+/* queue->lock must be held */
+static int __vrf_insert_slave(struct slave_queue *queue, struct slave *slave,
+			      struct net_device *master)
+{
+	struct net_vrf *vrf = netdev_priv(master);
+	struct slave *duplicate_slave = NULL;
+	int master_ifindex = master->ifindex;
+	int err = 0;
+
+	duplicate_slave = __vrf_find_slave_dev(queue, slave->dev);
+	if (duplicate_slave)
+		vrf_kill_one_slave(queue, duplicate_slave);
+
+	dev_hold(slave->dev);
+	list_add(&slave->list, &queue->all_slaves);
+	queue->num_slaves++;
+	slave->dev->flags |= IFF_SLAVE;
+
+	slave->dev->vrf_ptr = kmalloc(sizeof(*slave->dev->vrf_ptr), GFP_KERNEL);
+	if (!slave->dev->vrf_ptr)
+		return -ENODEV;
+	slave->dev->vrf_ptr->ifindex = master_ifindex;
+	slave->dev->vrf_ptr->tb_id = vrf->tb_id;
+
+	/* register the packet handler for slave ports */
+	err = netdev_rx_handler_register(slave->dev, vrf_handle_frame,
+					 (void *)master);
+	if (err) {
+		netdev_err(slave->dev,
+			   "Device %s failed to register rx_handler\n",
+			   slave->dev->name);
+		return err;
+	}
+
+	return 0;
+}
+
+static void vrf_fib_magic(int cmd, int type, __be32 dst, int dst_len,
+			  struct in_ifaddr *ifa, struct net_device *vrf_dev)
+{
+	struct net_vrf *vrf = netdev_priv(vrf_dev);
+	struct net *net = dev_net(ifa->ifa_dev->dev);
+	struct net *vrf_net = dev_net(vrf_dev);
+	struct fib_table *tb, *lc_tb;
+	struct fib_config cfg = {
+		.fc_protocol = RTPROT_KERNEL,
+		.fc_type = type,
+		.fc_dst = dst,
+		.fc_dst_len = dst_len,
+		.fc_prefsrc = ifa->ifa_local,
+		.fc_nlflags = NLM_F_CREATE | NLM_F_APPEND,

drivers/net/vrf.c new file +157 -0

+	if (net != vrf_net)
+		return;
+
+	if (type != RTN_LOCAL)
+		cfg.fc_scope = RT_SCOPE_LINK;
+	else
+		cfg.fc_scope = RT_SCOPE_HOST;
+
+	if (cmd == RTM_NEWROUTE) {
+		cfg.fc_table = lc_tb->tb_id;
+		cfg.fc_oif = ifa->ifa_dev->dev->ifindex;
+		fib_table_delete(lc_tb, &cfg);
+		cfg.fc_table = tb->tb_id;
+		cfg.fc_oif = vrf_dev->ifindex;
+		fib_table_insert(tb, &cfg);
+	} else {
+		cfg.fc_table = tb->tb_id;
+		cfg.fc_oif = vrf_dev->ifindex;
+		fib_table_delete(tb, &cfg);
+		cfg.fc_table = lc_tb->tb_id;
+		cfg.fc_oif = ifa->ifa_dev->dev->ifindex;
+		fib_table_insert(lc_tb, &cfg);
+	}
+}
+
+static void vrf_move_local_routes(int cmd, struct net_device *dev,
+				  struct net_device *port_dev)
+{
+	struct in_device *in_dev = __in_dev_get_rcu(port_dev);
+
+	if (!in_dev)
+		return;
+
+	for_ifa(in_dev) {
+		__be32 addr;
+
+		addr = ifa->ifa_local;
+		vrf_fib_magic(cmd, RTN_LOCAL, addr, 32, ifa, dev);
+	} endfor_ifa(in_dev);
+}
+
+static int vrf_inetaddr_event(struct notifier_block *this, unsigned long event,
+			      void *ptr)
+{
+	struct in_ifaddr *ifa = (struct in_ifaddr *)ptr;
+	struct net_device *port_dev = ifa->ifa_dev->dev;
+	struct net *net = dev_net(port_dev);
+	int master = 0;
+
+	if (port_dev->vrf_ptr)
+		master = port_dev->vrf_ptr->ifindex;
+
+	if (master) {
+		struct net_device *dev = dev_get_by_index(net, master);
+
+		switch (event) {
+		case NETDEV_UP:
+			vrf_move_local_routes(RTM_NEWROUTE, dev, port_dev);
+			break;
+		case NETDEV_DOWN:
+			vrf_move_local_routes(RTM_DELROUTE, dev, port_dev);
+			break;
+		}
+	}
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block vrf_inetaddr_notifier = {
+	.notifier_call = vrf_inetaddr_event,
+};
+
+/* netlink lock is assumed here */
+static int vrf_add_slave(struct net_device *dev,
+			 struct net_device *port_dev)
+{
+	if (!dev || !port_dev)
+		return -ENODEV;
+
+	if (dev_net(dev) != dev_net(port_dev))
+		return -ENODEV;
+
+	if (!vrf_is_master(port_dev) && !vrf_is_slave(port_dev)) {
+		struct slave *s = kmalloc(sizeof(*s), GFP_KERNEL);
+		struct net_vrf *vrf = netdev_priv(dev);
+		int ret;
+
+		if (!s)
+			return -ENOMEM;
+
+		memset(s, 0, sizeof(*s));
+		s->dev = port_dev;
+
+		spin_lock_bh(&vrf->queue.lock);
+		ret = __vrf_insert_slave(&vrf->queue, s, dev);
+		if (ret)
+			kfree(s);
+
+		spin_unlock_bh(&vrf->queue.lock);
+
+		vrf_move_local_routes(RTM_NEWROUTE, dev, port_dev);
+		ret = netdev_master_upper_dev_link(port_dev, dev);
+		return ret;
+	}
+
+	return -EINVAL;
+}
+
+static int vrf_del_slave(struct net_device *dev,
+			 struct net_device *port_dev)
+{
+	struct net_vrf *vrf = netdev_priv(dev);
+	struct slave_queue *queue = &vrf->queue;
+	struct slave *slave = __vrf_find_slave_dev(queue, port_dev);
+
+	if (!slave)
+		return -EINVAL;
+
+	vrf_kill_one_slave(queue, slave);
+	vrf_move_local_routes(RTM_DELROUTE, dev, port_dev);
+	netdev_upper_dev_unlink(port_dev, dev);
+
+	return 0;
+}
+
+static int vrf_dev_init(struct net_device *dev)
+{
+	struct net_vrf *vrf = netdev_priv(dev);
+
+	spin_lock_init(&vrf->queue.lock);
+	INIT_LIST_HEAD(&vrf->queue.all_slaves);
+	vrf->queue.master_dev = dev;
+
+	dev->dstats = netdev_alloc_pcpu_stats(struct pcpu_dstats);
+	dev->flags  =  IFF_MASTER | IFF_NOARP;
+	if (!dev->dstats)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static void vrf_dev_uninit(struct net_device *dev)
+{
+	free_percpu(dev->dstats);
+}
+
+static int vrf_dev_close(struct net_device *dev)
+{
+	struct net_vrf *vrf = netdev_priv(dev);
+	struct slave_queue *queue = &vrf->queue;
+	struct list_head *this, *head;
+
+	head = &queue->all_slaves;
+	list_for_each(this, head) {
+		struct slave *slave = list_entry(this, struct slave, list);
+
+		slave->dev->vrf_ptr->ifindex = 0;
+		slave->dev->vrf_ptr->tb_id = 0;

drivers/net/vrf.c new file +93 -0

+static int vrf_dev_open(struct net_device *dev)
+{
+	struct net_vrf *vrf = netdev_priv(dev);
+	struct slave_queue *queue = &vrf->queue;
+	struct list_head *this, *head;
+	int err = 0;
+
+	head = &queue->all_slaves;
+	list_for_each(this, head) {
+		struct slave *slave = list_entry(this, struct slave, list);
+
+		slave->dev->vrf_ptr->ifindex = dev->ifindex;
+		slave->dev->vrf_ptr->tb_id = vrf->tb_id;
+	}
+
+	if (dev->flags & IFF_MASTER)
+		dev->flags |= IFF_UP;
+
+	if (!vrf->tb)
+		return -EINVAL;
+
+	return err;
+}
+
+static int vrf_neigh_create(struct neighbour *n)
+{
+	n->nud_state = NUD_REACHABLE;
+	n->dead      = 0;
+
+	return 0;
+}
+
+static const struct net_device_ops vrf_netdev_ops = {
+	.ndo_init		= vrf_dev_init,
+	.ndo_uninit		= vrf_dev_uninit,
+	.ndo_open		= vrf_dev_open,
+	.ndo_stop               = vrf_dev_close,
+	.ndo_start_xmit		= vrf_xmit,
+	.ndo_get_stats64	= vrf_get_stats64,
+	.ndo_add_slave          = vrf_add_slave,
+	.ndo_del_slave          = vrf_del_slave,
+	.ndo_neigh_construct    = vrf_neigh_create,
+};
+
+static void vrf_get_drvinfo(struct net_device *dev,
+			    struct ethtool_drvinfo *info)
+{
+	strlcpy(info->driver, DRV_NAME, sizeof(info->driver));
+	strlcpy(info->version, DRV_VERSION, sizeof(info->version));
+}
+
+static const struct ethtool_ops vrf_ethtool_ops = {
+	.get_drvinfo            = vrf_get_drvinfo,
+};
+
+static void vrf_setup(struct net_device *dev)
+{
+	ether_setup(dev);
+
+	/* Initialize the device structure. */
+	dev->netdev_ops = &vrf_netdev_ops;
+	dev->ethtool_ops = &vrf_ethtool_ops;
+	dev->destructor = free_netdev;
+
+	/* Fill in device structure with ethernet-generic values. */
+	dev->tx_queue_len = 0;
+	eth_hw_addr_random(dev);
+}
+
+static int vrf_validate(struct nlattr *tb[], struct nlattr *data[])
+{
+	if (tb[IFLA_ADDRESS]) {
+		if (nla_len(tb[IFLA_ADDRESS]) != ETH_ALEN)
+			return -EINVAL;
+		if (!is_valid_ether_addr(nla_data(tb[IFLA_ADDRESS])))
+			return -EADDRNOTAVAIL;
+	}
+	return 0;
+}
+
+static int vrf_newlink(struct net *src_net, struct net_device *dev,
+		       struct nlattr *tb[], struct nlattr *data[])
+{
+	int err;
+	struct net_vrf *vrf = netdev_priv(dev);
+
+	if (data && data[IFLA_VRF_TABLE]) {
+		vrf->tb_id = nla_get_u32(data[IFLA_VRF_TABLE]);
+		/* reserve a table for this VRF device */
+		vrf->tb = fib_new_table(dev_net(dev), vrf->tb_id);
+		if (!vrf->tb)
+			return -ERANGE;
+

drivers/net/vrf.c new file +48 -0

+static void vrf_dellink(struct net_device *dev, struct list_head *head)
+{
+	/* Need to free the table ? */
+	unregister_netdev(dev);
+}
+
+static const struct nla_policy vrf_nl_policy[IFLA_VRF_MAX + 1] = {
+	[IFLA_VRF_TABLE] = { .type = NLA_U32 },
+};
+
+static struct rtnl_link_ops vrf_link_ops __read_mostly = {
+	.kind		= DRV_NAME,
+	.priv_size      = sizeof(struct net_vrf),
+	.policy         = vrf_nl_policy,
+	.newlink        = vrf_newlink,
+	.dellink        = vrf_dellink,
+	.setup		= vrf_setup,
+	.validate	= vrf_validate,
+	.maxtype        = IFLA_VRF_MAX,
+};
+
+static int __init vrf_init_module(void)
+{
+	int err = 0;
+
+	rtnl_lock();
+	err = __rtnl_link_register(&vrf_link_ops);
+	if (err < 0)
+		goto out;
+
+	register_inetaddr_notifier(&vrf_inetaddr_notifier);
+
+out:
+	rtnl_unlock();
+	return err;
+}
+
+static void __exit vrf_cleanup_module(void)
+{
+	unregister_inetaddr_notifier(&vrf_inetaddr_notifier);
+	rtnl_link_unregister(&vrf_link_ops);
+}
+
+module_init(vrf_init_module);
+module_exit(vrf_cleanup_module);
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_RTNL_LINK(DRV_NAME);
+MODULE_VERSION(DRV_VERSION);

include/linux/netdevice.h +10 -0

index 51f8d2f..29febf3 100644
@@ -51,6 +51,7 @@
  #include <linux/neighbour.h>
  #include <uapi/linux/netdevice.h>
  #include <uapi/linux/if_bonding.h>
+#include <net/vrf.h>
 
  struct netpoll_info;
  struct device;

@@ -1270,6 +1271,7 @@ enum netdev_priv_flags {
  	IFF_XMIT_DST_RELEASE_PERM	= 1<<22,
  	IFF_IPVLAN_MASTER		= 1<<23,
  	IFF_IPVLAN_SLAVE		= 1<<24,
+	IFF_VRF_MASTER		        = 1<<25,
  };
 
  #define IFF_802_1Q_VLAN			IFF_802_1Q_VLAN

@@ -1297,6 +1299,7 @@ enum netdev_priv_flags {
  #define IFF_XMIT_DST_RELEASE_PERM	IFF_XMIT_DST_RELEASE_PERM
  #define IFF_IPVLAN_MASTER		IFF_IPVLAN_MASTER
  #define IFF_IPVLAN_SLAVE		IFF_IPVLAN_SLAVE
+#define IFF_VRF_MASTER		        IFF_VRF_MASTER
 
  /**
   *	struct net_device - The DEVICE structure.

@@ -1413,6 +1416,7 @@ enum netdev_priv_flags {
   *	@dn_ptr:	DECnet specific data
   *	@ip6_ptr:	IPv6 specific data
   *	@ax25_ptr:	AX.25 specific data
+ *	@vrf_ptr:	VRF specific data
   *	@ieee80211_ptr:	IEEE 802.11 specific data, assign before registering
   *
   *	@last_rx:	Time of last Rx

@@ -1625,6 +1629,7 @@ struct net_device {
  	struct dn_dev __rcu     *dn_ptr;
  	struct inet6_dev __rcu	*ip6_ptr;
  	void			*ax25_ptr;
+	struct net_vrf_dev      *vrf_ptr;
  	struct wireless_dev	*ieee80211_ptr;
  	struct wpan_dev		*ieee802154_ptr;
  #if IS_ENABLED(CONFIG_MPLS_ROUTING)

@@ -3776,6 +3781,11 @@ static inline bool netif_supports_nofcs(struct net_device *dev)
  	return dev->priv_flags & IFF_SUPP_NOFCS;
  }
 
+static inline bool netif_is_vrf(struct net_device *dev)
+{
+	return dev->priv_flags & IFF_VRF_MASTER;
+}
+
  /* This device needs to keep skb dst for qdisc enqueue or ndo_start_xmit() */
  static inline void netif_keep_dst(struct net_device *dev)
  {

include/net/flow.h +1 -0

index 8109a15..69aaa99 100644
@@ -29,6 +29,7 @@ struct flowi_common {
  	__u8	flowic_flags;
  #define FLOWI_FLAG_ANYSRC		0x01
  #define FLOWI_FLAG_KNOWN_NH		0x02
+#define FLOWI_FLAG_VRFSRC		0x04
  	__u32	flowic_secid;
  };
 

include/net/vrf.h new file +19 -0

new file mode 100644
index 0000000..11d7dbf8
@@ -0,0 +1,19 @@
+/*
+ * include/net/net_vrf.h - adds vrf dev structure definitions
+ * Copyright (c) 2015 Cumulus Networks
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef __LINUX_NET_VRF_H
+#define __LINUX_NET_VRF_H
+
+struct net_vrf_dev {
+	int                     ifindex; /* ifindex of master dev */
+	u32                     tb_id;
+};
+
+#endif /* __LINUX_NET_VRF_H */

include/uapi/linux/if_link.h +9 -0

index afccc93..b98a443 100644
@@ -339,6 +339,15 @@ enum macvlan_macaddr_mode {
 
  #define MACVLAN_FLAG_NOPROMISC	1
 
+/* VRF section */
+enum {
+	IFLA_VRF_UNSPEC,
+	IFLA_VRF_TABLE,
+	__IFLA_VRF_MAX
+};
+
+#define IFLA_VRF_MAX (__IFLA_VRF_MAX - 1)
+
  /* IPVLAN section */
  enum {
  	IFLA_IPVLAN_UNSPEC,

From: Shrijeet Mukherjee <[email protected]>

Incoming frames for IP protocol stacks need the IIF to be changed from the actual interface to the VRF device. This allows the IIF rule to be used to select tables (or do regular PBR)

This change selects the iif to be the VRF device if it exists and the incoming iif is enslaved to the VRF device.

Since VRF aware sockets are always bound to the VRF device this system allows return traffic to find the socket of origin.

changes are in the arp_rcv, icmp_rcv and ip_rcv paths

Question : I did not wrap the rcv modifications, in CONFIG_NET_VRF as it would create code variations and the vrf_ptr check is there I can make that whole thing modular.

From an architectural level I think the output path looks good. For the

Hi Shrijeet,

Quoted 9 lines · Shrijeet Mukherjee

From: Shrijeet Mukherjee <[email protected]>

Incoming frames for IP protocol stacks need the IIF to be changed from the actual interface to the VRF device. This allows the IIF rule to be used to select tables (or do regular PBR)

This change selects the iif to be the VRF device if it exists and the incoming iif is enslaved to the VRF device.

Quoted 8 lines

Since VRF aware sockets are always bound to the VRF device this system allows return traffic to find the socket of origin.

changes are in the arp_rcv, icmp_rcv and ip_rcv paths

Question : I did not wrap the rcv modifications, in CONFIG_NET_VRF as it would create code variations and the vrf_ptr check is there I can make that whole thing modular.

From an architectural level I think the output path looks good. For the
input path I would also to propose my (I think) more flexible solution:

For rx layer I want to also propose my try:

[PATCH net-next RFC] net: ipv4: arp: strong end system model semantics by per-interface local table override

By allowing to direct routing table lookups to a specific table based on the incoming interface for IPv4 and ARP, we start to behave like a strong end host system without tweaking arp_* sysctl settings.

The main motivation behind this patch was input and forwarding support
in a VRF like model. Maybe it also helps for hardware offloading by allowing reducing rule complexity.

An example:

$ ip rule flush
$ ip rule del
$ ip rule del
$ ip rule add inherit-table
0:      from all inherit-table

This by default still uses RT_TABLE_LOCAL until we set up per interface
route tables:

$ ip link set dev enp0s25 ipv4-rt-table-id 100
$ ip -d link ls dev enp0s25
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
state UP mode DEFAULT group default qlen 1000
    link/ether e4:7f:b2:1b:4c:61 brd ff:ff:ff:ff:ff:ff promiscuity 0 
ipv4-rt-table-id 100 addrgenmode none

This let's incoming and arp requests use routing table 100. The system
will stop responding to arp requests as we don't have any entries in this routing table.

$ ip address add 192.168.88.223/24 dev enp0s25 table 100
$ ip -d address ls
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
state UP group default qlen 1000
    link/ether e4:7f:b2:1b:4c:61 brd ff:ff:ff:ff:ff:ff promiscuity 0
    inet 192.168.88.223/24 scope global enp0s25 table 100
       valid_lft forever preferred_lft forever
$ ip route add 192.168.88.0/24 dev enp0s25 table 100
$ ip route add default via 192.168.88.1 table 100
$ ip route ls dev table 100
local 192.168.88.223 dev enp0s25  proto kernel  scope host  src 
192.168.88.223
192.168.88.0/24 dev enp0s25  scope link
default via 192.168.88.1 dev enp0s25 proto static metric 600

Those changes direct arp lookups towards table 100 and the input route,
too. The local address is used for icmp source addresses and arp replies. The connected route to steer icmp packets out of that interface.

This patch covers only the forwarding path.

Hi Shrijeet,

From an architectural level I think the output path looks good. For the input path I would also to propose my (I think) more flexible solution:

For rx layer I want to also propose my try:

[PATCH net-next RFC] net: ipv4: arp: strong end system model semantics by per-interface local table override

By allowing to direct routing table lookups to a specific table based on the incoming interface for IPv4 and ARP, we start to behave like a strong end host system without tweaking arp_* sysctl settings.

The main motivation behind this patch was input and forwarding support in a VRF like model. Maybe it also helps for hardware offloading by allowing reducing rule complexity.

An example:

$ ip rule flush
$ ip rule del
$ ip rule del
$ ip rule add inherit-table
0:      from all inherit-table

This by default still uses RT_TABLE_LOCAL until we set up per interface route tables:

$ ip link set dev enp0s25 ipv4-rt-table-id 100
$ ip -d link ls dev enp0s25
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether e4:7f:b2:1b:4c:61 brd ff:ff:ff:ff:ff:ff promiscuity 0 ipv4-rt-table-id 100 addrgenmode none

This let's incoming and arp requests use routing table 100. The system will stop responding to arp requests as we don't have any entries in this routing table.

Quoted 28 lines · Hannes Frederic Sowa

Hi Shrijeet,

From an architectural level I think the output path looks good. For the input path I would also to propose my (I think) more flexible solution:

For rx layer I want to also propose my try:

[PATCH net-next RFC] net: ipv4: arp: strong end system model semantics by per-interface local table override

By allowing to direct routing table lookups to a specific table based on the incoming interface for IPv4 and ARP, we start to behave like a strong end host system without tweaking arp_* sysctl settings.

The main motivation behind this patch was input and forwarding support in a VRF like model. Maybe it also helps for hardware offloading by allowing reducing rule complexity.

An example:

$ ip rule flush
$ ip rule del
$ ip rule del
$ ip rule add inherit-table
0:      from all inherit-table

This by default still uses RT_TABLE_LOCAL until we set up per interface route tables:

Quoted 8 lines

$ ip link set dev enp0s25 ipv4-rt-table-id 100
$ ip -d link ls dev enp0s25
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether e4:7f:b2:1b:4c:61 brd ff:ff:ff:ff:ff:ff promiscuity 0 ipv4-rt-table-id 100 addrgenmode none

This let's incoming and arp requests use routing table 100. The system will stop responding to arp requests as we don't have any entries in this routing table.

I like this model in general, as it addresses the issue that I have not addressed around connected routes.

This would force local and directly connected host routes to be learnt into the correct table.

It does bring the question up then.

1. The driver already knows the vrf device to table map
2. If the device also knows the final device to table map

then do we need to use fib_rules and just lookup the table directly. It does make the configuration a little longer since each component device now needs configuration when you add/del a member from a vrf.

If people generally agree and we want to skip the fib_rule lookup, then I can make it such that enslaving already takes the dev-table id as well, and then the process of enslaving in the nominal VRF case becomes

ip link add vrf-dev type vrf table foo ipv4-rt-table-id bar ip link set eth2 master vrf-dev

Quoted 15 lines · Shrijeet Mukherjee

Quoted 5 lines · Hannes Frederic Sowa

Hi Shrijeet,

This let's incoming and arp requests use routing table 100. The system will stop responding to arp requests as we don't have any entries in this routing table.

I like this model in general, as it addresses the issue that I have not addressed around connected routes.

This would force local and directly connected host routes to be learnt into the correct table.

Quoted 8 lines

It does bring the question up then.

1. The driver already knows the vrf device to table map
2. If the device also knows the final device to table map

then do we need to use fib_rules and just lookup the table directly. It does make the configuration a little longer since each component device now needs configuration when you add/del a member from a vrf.

This model is usable on its own, especially if one does not need routing daemons
or user space software dealing with VRFs and sending out packets.

Quoted 7 lines

If people generally agree and we want to skip the fib_rule lookup, then I can make it such that enslaving already takes the dev-table id as well, and then the process of enslaving in the nominal VRF case becomes

ip link add vrf-dev type vrf table foo ipv4-rt-table-id bar ip link set eth2 master vrf-dev

I think this would be great.

Last time I looked into the patches it was not yet clear if we can do that
without holding strong references to the other interfaces. Hopefully this can
be done by just passing down the table ids to the slaves during initializing
and teardown of the master vrf interface.

Bye,
Hannes

Quoted 47 lines · Hannes Frederic Sowa

Quoted 15 lines · Shrijeet Mukherjee

Quoted 5 lines · Hannes Frederic Sowa

Hi Shrijeet,

This let's incoming and arp requests use routing table 100. The system will stop responding to arp requests as we don't have any entries in this routing table.

I like this model in general, as it addresses the issue that I have not addressed around connected routes.

This would force local and directly connected host routes to be learnt into the correct table.

Quoted 8 lines

It does bring the question up then.

1. The driver already knows the vrf device to table map
2. If the device also knows the final device to table map

then do we need to use fib_rules and just lookup the table directly. It does make the configuration a little longer since each component device now needs configuration when you add/del a member from a vrf.

This model is usable on its own, especially if one does not need routing daemons
or user space software dealing with VRFs and sending out packets.

Quoted 7 lines

If people generally agree and we want to skip the fib_rule lookup, then I can make it such that enslaving already takes the dev-table id as well, and then the process of enslaving in the nominal VRF case becomes

ip link add vrf-dev type vrf table foo ipv4-rt-table-id bar ip link set eth2 master vrf-dev

I think this would be great.

Last time I looked into the patches it was not yet clear if we can do that
without holding strong references to the other interfaces. Hopefully this can
be done by just passing down the table ids to the slaves during initializing
and teardown of the master vrf interface.

Quoted 2 lines

Bye,
Hannes

We can do that, and the hooks are all available. But do we want to cut out the fib_rules ? this would close out the opportunity for someone to insert a fib_rule to override the rule which directs to a VRF device.

Personally don't have a strong opinion, but want to make sure we understand that choice.

From: Hannes Frederic Sowa <[email protected]>
Date: Mon, 08 Jun 2015 21:58:37 +0200

include/linux/inetdevice.h static inline void in_dev_put(struct in_device *idev) · line 251

Quoted 10 lines

include/linux/inetdevice.h +10 -0

+static inline u32 ipv4_idev_rt_table(const struct net_device *dev)
+{
+       u32 table_id;
+
+       rcu_read_lock();
+       table_id = __in_dev_get_rcu(dev)->rt_table_id;
+       rcu_read_unlock();
+
+       return table_id != RT_TABLE_UNSPEC ? table_id : RT_TABLE_LOCAL;
+}

It's a real shame you have to do all of this RCU locking and inetdev deref, because in more than half of the call sites the idev is already available.

It is easy to move the rt_table_id to net_device and use the same one for IPv6. This would force people to build symmetric routing configurations. I was striving for maximum flexibility first but I don't really think this matters here.

Hi Shrijeet,

Quoted 12 lines · Shrijeet Mukherjee

From: Shrijeet Mukherjee <[email protected]>

Incoming frames for IP protocol stacks need the IIF to be changed from the actual interface to the VRF device. This allows the IIF rule to be used to select tables (or do regular PBR)

This change selects the iif to be the VRF device if it exists and the incoming iif is enslaved to the VRF device.

Since VRF aware sockets are always bound to the VRF device this system allows return traffic to find the socket of origin.

Quoted 5 lines

changes are in the arp_rcv, icmp_rcv and ip_rcv paths

Question : I did not wrap the rcv modifications, in CONFIG_NET_VRF as it would create code variations and the vrf_ptr check is there I can make that whole thing modular.

From an architectural level I think the output path looks good. For the
input path I would also to propose my (I think) more flexible solution:

For rx layer I want to also propose my try:

[PATCH net-next RFC] net: ipv4: arp: strong end system model semantics by per-interface local table override

Quoted 4 lines · Hannes Frederic Sowa

For rx layer I want to also propose my try:

[PATCH net-next RFC] net: ipv4: arp: strong end system model semantics by per-interface local table override

I applied only the first 2 patches from Shrijeet and then tried to apply your patch; it doesn't apply. Way too many failures. What branch should it apply too?

From: Shrijeet Mukherjee <[email protected]>

Incoming frames for IP protocol stacks need the IIF to be changed from the actual interface to the VRF device. This allows the IIF rule to be used to select tables (or do regular PBR)

This change selects the iif to be the VRF device if it exists and the incoming iif is enslaved to the VRF device.

Since VRF aware sockets are always bound to the VRF device this system allows return traffic to find the socket of origin.

changes are in the arp_rcv, icmp_rcv and ip_rcv paths

Question : I did not wrap the rcv modifications, in CONFIG_NET_VRF as it would create code variations and the vrf_ptr check is there I can make that whole thing modular.

Signed-off-by Shrijeet Mukherjee <[email protected]>

---
  net/ipv4/fib_frontend.c |   14 +++++++++++---
  net/ipv4/fib_trie.c     |    7 +++++--
  net/ipv4/icmp.c         |    6 ++++++
  net/ipv4/route.c        |    3 ++-
  4 files changed, 24 insertions(+), 6 deletions(-)

net/ipv4/fib_frontend.c +2 -0

index 9d4cef4..cf2d584 100644
@@ -45,6 +45,7 @@
  #include <net/ip_fib.h>
  #include <net/rtnetlink.h>
  #include <net/xfrm.h>
+#include <net/vrf.h>
 
  #ifndef CONFIG_IP_MULTIPLE_TABLES
 

@@ -218,15 +219,18 @@ static inline unsigned int __inet_dev_addr_type(struct net *net,
  	struct fib_result	res;
  	unsigned int ret = RTN_BROADCAST;
  	struct fib_table *local_table;
+	int rt_table = RT_TABLE_LOCAL;
 
  	if (ipv4_is_zeronet(addr) || ipv4_is_lbcast(addr))
  		return RTN_BROADCAST;
  	if (ipv4_is_multicast(addr))
  		return RTN_MULTICAST;
 

net/ipv4/fib_trie.c +0 -0

index 97fa62d..515ff11 100644
@@ -1409,8 +1409,11 @@ found:
 
  			if (nh->nh_flags & RTNH_F_DEAD)
  				continue;

net/ipv4/icmp.c +6 -0

index f5203fb..61b7da3 100644
@@ -96,6 +96,7 @@
  #include <net/xfrm.h>
  #include <net/inet_common.h>
  #include <net/ip_fib.h>
+#include <net/vrf.h>
 
  /*
   *	Build xmit assembly blocks

@@ -420,6 +421,8 @@ static void icmp_reply(struct icmp_bxm *icmp_param, struct sk_buff *skb)
  			daddr = icmp_param->replyopts.opt.opt.faddr;
  	}
  	memset(&fl4, 0, sizeof(fl4));
+	if (skb->dev && skb->dev->vrf_ptr)
+		fl4.flowi4_oif = skb->dev->vrf_ptr->ifindex;
  	fl4.daddr = daddr;
  	fl4.saddr = saddr;
  	fl4.flowi4_mark = mark;

@@ -458,6 +461,9 @@ static struct rtable *icmp_route_lookup(struct net *net,
  	fl4->flowi4_proto = IPPROTO_ICMP;
  	fl4->fl4_icmp_type = type;
  	fl4->fl4_icmp_code = code;
+	if (skb_in->dev && skb_in->dev->vrf_ptr)
+		fl4->flowi4_oif =  skb_in->dev->vrf_ptr->ifindex;
+
  	security_skb_classify_flow(skb_in, flowi4_to_flowi(fl4));
  	rt = __ip_route_output_key(net, fl4);
  	if (IS_ERR(rt))

net/ipv4/route.c +2 -1

index f605598..8c37720 100644
@@ -109,6 +109,7 @@
  #include <linux/kmemleak.h>
  #endif
  #include <net/secure_seq.h>
+#include <net/vrf.h>
 
  #define RT_FL_TOS(oldflp4) \
  	((oldflp4)->flowi4_tos & (IPTOS_RT_MASK | RTO_ONLINK))

@@ -1710,7 +1711,7 @@ static int ip_route_input_slow(struct sk_buff *skb, __be32 daddr, __be32 saddr,
  	 *	Now we are ready to route packet.
  	 */
  	fl4.flowi4_oif = 0;
-	fl4.flowi4_iif = dev->ifindex;
+	fl4.flowi4_iif = dev->vrf_ptr ? dev->vrf_ptr->ifindex : dev->ifindex;
  	fl4.flowi4_mark = skb->mark;
  	fl4.flowi4_tos = tos;
  	fl4.flowi4_scope = RT_SCOPE_UNIVERSE;

5. Debugging is built-in as tcpdump and counters on the VRF device works as is.

Quoted 3 lines · Shrijeet Mukherjee

5. Debugging is built-in as tcpdump and counters on the VRF device works as is.

Is the intent that something like this

tcpdump -i vrf0

can be used to see vrf traffic?

vrf_handle_frame only bumps counters; it does not switch skb->dev to the vrf device so for Rx path tcpdump will not get the packets. ie., tcpdump only shows outbound packets.

Quoted 2 lines · Shrijeet Mukherjee

5. Debugging is built-in as tcpdump and counters on the VRF device works as is.

Is the intent that something like this

tcpdump -i vrf0

can be used to see vrf traffic?

vrf_handle_frame only bumps counters; it does not switch skb->dev to the vrf device so for Rx path tcpdump will not get the packets. ie., tcpdump only shows outbound packets.

From: Shrijeet Mukherjee <[email protected]>

In the context of internet scale routing a requirement that always comes up is the need to partition the available routing tables into disjoint routing planes. A specific use case is the multi-tenancy problem where each tenant has their own unique routing tables and in the very least need different default gateways.

This is an attempt to build the ability to create virtual router domains aka VRF's (VRF-lite to be specific) in the linux packet forwarding stack. The main observation is that through the use of

  drivers/net/vrf.c            |  654 ++++++++++++++++++++++++++++++++++++++++++

I'm not really in favor of the name 'vrf'. This term is very controversial and
having a consensus of what is/contains a 'vrf' is quite impossible. There was already a lot of discussions about this topic on quagga ml that show
that everybody has a different opinion about this term ;-)

Hi Nicolas:

Quoted 6 lines · Nicolas Dichtel

I'm not really in favor of the name 'vrf'. This term is very controversial and
having a consensus of what is/contains a 'vrf' is quite impossible. There was already a lot of discussions about this topic on quagga ml that show
that everybody has a different opinion about this term ;-)

Are you referring to this thread?
https://lists.quagga.net/pipermail/quagga-dev/2014-November/011795.html

Quoted 13 lines · David Ahern

Hi Nicolas:

Quoted 7 lines · Nicolas Dichtel

I'm not really in favor of the name 'vrf'. This term is very controversial and
having a consensus of what is/contains a 'vrf' is quite impossible. There was already a lot of discussions about this topic on quagga ml that show
that everybody has a different opinion about this term ;-)

Quoted 2 lines

Are you referring to this thread?
https://lists.quagga.net/pipermail/quagga-dev/2014-November/011795.html

No, there were recent discussions on quagga about that subject. Here is some non-exhaustive pointers:
https://lists.quagga.net/pipermail/quagga-dev/2015-May/012581.html
https://lists.quagga.net/pipermail/quagga-dev/2015-May/012630.html
https://lists.quagga.net/pipermail/quagga-dev/2015-June/012715.html

Note the last pointer also explains why it was called MRF by Cumulus.

Regards,
Nicolas

model with some performance paths that need optimization. (Specifically the output route selector that Roopa, Robert, Thomas and EricB are currently discussing on the MPLS thread)

High Level points

1. Simple overlay driver (minimal changes to current stack)
* uses the existing fib tables and fib rules infrastructure
2. Modelled closely after the ipvlan driver
3. Uses current API and infrastructure.
* Applications can use SO_BINDTODEVICE or cmsg device indentifiers to pick VRF (ping, traceroute just work)

* Standard IP Rules work, and since they are aggregated against the device, scale is manageable
4. Completely orthogonal to Namespaces and only provides separation in the routing plane (and ARP)
5. Debugging is built-in as tcpdump and counters on the VRF device works as is.

                                                 N2
           N1 (all configs here)          +---------------+
    +--------------+                      |               |
    |swp1 :10.0.1.1+----------------------+swp1 :10.0.1.2 |
    |              |                      |               |
    |swp2 :10.0.2.1+----------------------+swp2 :10.0.2.2 |
    |              |                      +---------------+
    | VRF 0        |
    | table 5      |
    |              |
    +---------------+

    |              |
    | VRF 1        |                             N3
    | table 6      |                      +---------------+
    |              |                      |               |
    |swp3 :10.0.2.1+----------------------+swp1 :10.0.2.2 |
    |              |                      |               |
    |swp4 :10.0.3.1+----------------------+swp2 :10.0.3.2 |
    +--------------+                      +---------------+

# Install the lookup rules that map table to VRF domain ip rule add pref 200 oif vrf0 lookup 5
ip rule add pref 200 iif vrf0 lookup 5
ip rule add pref 200 oif vrf1 lookup 6
ip rule add pref 200 iif vrf1 lookup 6

On 06/08/15 at 11:35am, Shrijeet Mukherjee wrote:
[...]

Quoted 3 lines

model with some performance paths that need optimization. (Specifically the output route selector that Roopa, Robert, Thomas and EricB are currently discussing on the MPLS thread)

Thanks for posting these patches just in time. This explains how you intent to deploy Roopa's patches in a scalable manner.

Quoted 8 lines

High Level points

1. Simple overlay driver (minimal changes to current stack)
* uses the existing fib tables and fib rules infrastructure
2. Modelled closely after the ipvlan driver
3. Uses current API and infrastructure.
* Applications can use SO_BINDTODEVICE or cmsg device indentifiers to pick VRF (ping, traceroute just work)

I like the aspect of reusing existing user interfaces. We might need to introduce a more fine grained capability than CAP_NET_RAW to give containers the privileges to bind to a VRF without allowing them to inject raw frames.

Given I understand this correctly: If my intent was to run a process in multiple VRFs, then I would need to run that process in the host network namespace which contains the VRF devices which would also contain the physical devices. While I might want to grant my process the ability to bind to VRFs, I may not want to give it the privileges to bind to any device. So we could consider introducing CAP_NET_VRF which would allow to bind to VRF devices.

Quoted 23 lines · Thomas Graf

On 06/08/15 at 11:35am, Shrijeet Mukherjee wrote:
[...]

Quoted 3 lines

model with some performance paths that need optimization. (Specifically the output route selector that Roopa, Robert, Thomas and EricB are currently discussing on the MPLS thread)

Thanks for posting these patches just in time. This explains how you intent to deploy Roopa's patches in a scalable manner.

Quoted 8 lines

High Level points

1. Simple overlay driver (minimal changes to current stack)
* uses the existing fib tables and fib rules infrastructure
2. Modelled closely after the ipvlan driver
3. Uses current API and infrastructure.
* Applications can use SO_BINDTODEVICE or cmsg device indentifiers to pick VRF (ping, traceroute just work)

I like the aspect of reusing existing user interfaces. We might need to introduce a more fine grained capability than CAP_NET_RAW to give containers the privileges to bind to a VRF without allowing them to inject raw frames.

Quoted 8 lines

Given I understand this correctly: If my intent was to run a process in multiple VRFs, then I would need to run that process in the host network namespace which contains the VRF devices which would also contain the physical devices. While I might want to grant my process the ability to bind to VRFs, I may not want to give it the privileges to bind to any device. So we could consider introducing CAP_NET_VRF which would allow to bind to VRF devices.

If I understand correctly, all existing applications should also be modified
if I want to run them into a VRF/MRF (see my previous email)?

ssh, dhcp, httpd, etc should be runnable per MRF without modifications of their source code. So, it becomes a netns. What's about an IKE dameon?

It makes sense to have both: netns and MRF ; each can have their own logics
of VRF-like behavior depending on how a VRF is defined by the end users.

Quoted 7 lines · Thomas Graf

Do I understand this correctly that swp* represent veth pairs? Why do you have distinct addresses on each peer of the pair? Are the addresses in N2 and N3 considered private and NATed?

[...]

???These are physical boxes in the picture not veth pairs or NAT's :)???

???Are you worried about ip rule scale ? this reduces the scale to number of L3 domains, which should be not that large. I do think we need to speed up rule lookup from the linear walk we have right now.

A generic classifier seems like a bigger hammer, but if that is the way to replace rules it is a worthy concept.

That said, the patches from Hannes et al, will make it such that the table lookup maybe from the driver directly and thus will skip past the fib rule lookup.