Skip to content

RFC net: VRF support

David Ahern netdev 1 version NAK

Status from this message: Nacked-by: "Eric W. Biederman" <[email protected]>

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest29 patches 43 repliesNAK

Patches in v1

#Subject and filesSizeRepliesApplied
1/29net: Introduce net_ctx and macro for context comparisoninclude/net/net_namespace.h+21 -00 replies
2/29net: Flip net_device to use net_ctxinclude/linux/netdevice.h+14 -32 replies
3/29net: Flip sock_common to net_ctxinclude/net/sock.h+12 -50 replies
4/29net: Add net_ctx macros for skbuffsinclude/linux/skbuff.h+4 -00 replies
5/29net: Flip seq_net_private to net_ctxfs/proc/proc_net.c include/linux/seq_file_net.h+7 -50 replies
6/29net: Flip fib_rules and fib_rules_ops to use net_ctxinclude/net/fib_rules.h+4 -20 replies
7/29net: Flip inet_bind_bucket to net_ctxinclude/net/inet_hashtables.h net/ipv4/inet_hashtables.c+18 -50 replies
8/29net: Flip fib_info to net_ctxinclude/net/ip_fib.h+10 -10 replies
9/29net: Flip ip6_flowlabel to net_ctxinclude/net/ipv6.h+12 -10 replies
10/29net: Flip neigh structs to net_ctxinclude/net/neighbour.h net/core/neighbour.c+39 -110 replies
11/29net: Flip nl_info to net_ctxinclude/net/netlink.h+3 -10 replies
12/29net: Add device lookups by net_ctxinclude/linux/netdevice.h net/core/dev.c+38 -00 replies
13/29net: Convert function arg from struct net to struct net_ctxinclude/linux/inetdevice.h include/net/addrconf.h include/net/arp.h +48 more+684 -5830 replies
14/29net: vrf: Introduce vrf header fileinclude/net/vrf.h+36 -03 replies
15/29net: vrf: Add vrf to net_ctx structinclude/linux/netdevice.h include/net/fib_rules.h include/net/inet_sock.h +8 more+31 -40 replies
16/29net: vrf: Set default vrfnet/core/dev.c net/core/neighbour.c net/ipv4/fib_rules.c +2 more+9 -40 replies
17/29net: vrf: Add vrf context to task structfs/proc/base.c include/linux/init_task.h include/linux/sched.h +1 more+99 -00 replies
18/29net: vrf: Plumbing for vrf context on a socketinclude/uapi/linux/in.h net/core/sock.c net/ipv4/inet_connection_sock.c +5 more+24 -22 replies
19/29net: vrf: Add vrf context to skbinclude/linux/skbuff.h include/net/sock.h include/net/tcp.h +10 more+44 -54 replies
20/29net: vrf: Add vrf context to flow structinclude/net/flow.h include/net/route.h net/ipv4/devinet.c +15 more+46 -220 replies
21/29net: vrf: Add vrf context to genid'sinclude/net/ip_fib.h include/net/net_namespace.h net/ipv4/devinet.c +3 more+34 -150 replies
22/29net: vrf: Set VRF id in various network structsinclude/net/inet_hashtables.h include/net/neighbour.h net/core/fib_rules.c +6 more+13 -00 replies
23/29net: vrf: Enable vrf checksinclude/linux/netdevice.h include/net/inet_hashtables.h include/net/ip_fib.h +14 more+60 -150 replies
24/29net: vrf: Add support to get/set vrf context on a deviceinclude/linux/netdevice.h include/uapi/linux/if_link.h net/core/dev.c +1 more+42 -00 replies
25/29net: vrf: Handle VRF any contextinclude/linux/netdevice.h include/net/inet_hashtables.h include/net/neighbour.h +12 more+99 -162 replies
26/29net: vrf: Change single_open_net to pass net_ctxfs/proc/proc_net.c+18 -20 replies
27/29net: vrf: Add vrf checks and context to ipv4 proc filesnet/ipv4/fib_trie.c net/ipv4/proc.c net/ipv4/raw.c +3 more+39 -250 replies
28/29iproute2: vrf: Add vrf subcommandip/Makefile ip/ip.c ip/ip_common.h +1 more+113 -20 replies
29/29iproute2: Add vrf option to ip link commandbridge/link.c include/linux/if_link.h ip/ipaddress.c +1 more+16 -00 replies

Reviews trailers per patch

None of the 29 patches has a Reviewed-by or Acked-by.

PatchEric W. Biederman
1/29
2/29
3/29
4/29
5/29
6/29
7/29
8/29
9/29
10/29
11/29
12/29
13/29
14/29
15/29
16/29
17/29
18/29
19/29Nacked-by
20/29
21/29
22/29
23/29
24/29
25/29
26/29
27/29
28/29
29/29

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH 00/29] net: VRF support

Kernel patches are also available here:

    https://github.com/dsahern/linux.git vrf-3.19

iproute2 patches are also available here:

    https://github.com/dsahern/iproute2 vrf-3.19

Background
----------
The concept of VRFs (Virtual Routing and Forwarding) has been around for over 15 years. Support for VRFs in the Linux kernel has been an often requested feature for almost as long. For a while support was available via an out of tree patch [1]. Since network namespaces came along, the response to queries about VRF support for Linux was 'use namespaces'. But as mentioned previously
[2] network namespaces are not a good match for VRFs. Of the list of problems noted the big one is that namespaces do not scale efficiently to the number of VRFs supported by networking gear (> 1000 VRFs). Networking vendors that want to use Linux as the OS have to carry custom solutions to this problem -- be it userspace networking stacks, extensive kernel patches (to add VRF support or bend the implementation of namespaces), and/or patches to many open source components. The recent addition of switchdev support in the kernel suggests that people expect the use of Linux as a switch networking OS to increase. Hopefully the time is right to re-open the discussion on a salable VRF implementation for the Linux kernel.

The intent of this RFC is to get feedback on the overall idea - namely VRFs as integer id and the nesting of VRFs within a namespace. This set includes changes only to core IPv4 code which shows the concept; changes to the rest of the network stack are fairly repetitive.

This patch set has a number of similarities to the original VRF patch - most notably VRF ids as an integer index and plumbing through iproute2 and netlink. But this set is really a complete re-implementation of the feature, integrating VRF within a namespace and leveraging existing support for network namespaces.

Design
------
Namespaces provide excellent separation of the networking stack from the netdevices and up. The intent of VRFs is to provide an additional, logical separation at the L3 layer within a namespace.

   +----------------------------------------------------------+
   | Namespace foo                                            |
   |                         +---------------+                |
   |          +------+       | L3/L4 service |                |
   |          | lldp |       |   (VRF any)   |                |
   |          +------+       +---------------+                |
   |                                                          |
   |                             +-------------------------+  |
   |                             | VRF M                   |  |
   |  +---------------------+  +-------------------------+ |  |
   |  | VRF 1 (default)     |  | VRF N                   | |  |
   |  |  +---------------+  |  |    +---------------+    | |  |
   |  |  | L3/L4 service |  |  |    | L3/L4 service |    | |  |
   |  |  | (VRF unaware) |  |  |    | (VRF unaware) |    | |  |
   |  |  +---------------+  |  |    +---------------+    | |  |
   |  |                     |  |                         | |  |
   |  |+-----+ +----------+ |  |  +-----+ +----------+   | |  |
   |  || FIB | | neighbor | |  |  | FIB | | neighbor |   | |  |
   |  |+-----+ +----------+ |  |  +-----+ +----------+   | |  |
   |  |                     |  |                         |-+  |
   |  | {dev 1}  {dev 2}    |  | {dev 3} {dev 4} {dev 5} |    |
   |  +---------------------+  +-------------------------+    |
   +----------------------------------------------------------+

This is accomplished by enhancing the current namespace checks to a broader network context that is both a namepsace and a VRF id. The VRF id is a tag applied to relevant structures, an integer between 1 and 4095 which allows for 4095 VRFs (could have 0 be the default VRF and then the range is 0-4095 = 4096s VRFs). (The limitation is arguably artificial. It is based on the genid scheme for versioning networking data which is a 32-bit integer. The VRF id is the lower 12 bits of the genid's.)

Netdevices, sk_buffs, sockets, and tasks are all tagged with a VRF id. Network lookups (devices, sockets, addresses, routes, neighbors) require a match of both network namespace and VRF id (or the special 'vrf any' tag; more on that later).

Beyond the 4-byte tag in various data structures, there are no resources allocated to a VRF so there is no need to create or destroy a VRF which is in-line with the concept of keeping it lightweight for scalability. The trade-off is that VRFs use the the same sysctl settings as the namespace they are part of and, for example, MIB counters.

The VRF id of tasks defaults to 1 and is inherited parent to child. It can be read via the file '/proc/<pid>/vrf' and can be changed anytime by writing to this file (if preferred this can be made a prctl to change the VRF id). This allows services to be launched in a VRF context using ip, similar to what is done for network namespaces.
e.g., ip vrf exec 99 /usr/sbin/sshd

(or a simpler chvrf alias/command can be used to just write the VRF id to the proc file.)

The task's VRF id also affects viewing and modifying network configuration. For example, 'ip addr show', 'ip route ls', 'ifconfig', 'arp -n', etc, only show network data for the VRF associated with the task's VRF id; devices are at the L2 layer so a command listing devices is not impacted by VRF id.

When a socket is created the VRF id is taken from the task. Socket-vrf association for non-connected sockets can be changed using a setsockopt (e.g., create a socket then change VRF id prior to calling bind or connect).

Network devices belong to a single VRF context which defaults to VRF 1. They can be assigned to another VRF using IFLA_VRF attribute in link messages. Similarly the VRF assignment is returned in the IFLA_VRF attribute. The ip command has been modified to display the VRF id of a device. L2 applications like lldp are not VRF aware and still work through through all network devices within the namespace.

On RX skbs get their VRF context from the netdevice the packet is received on. For TX the VRF context for an skb is taken from the socket. The intention is for L3/raw sockets to be able to set the VRF context for a packet TX using cmsg (not coded in this patch set).

VRF aware apps (e.g., L3 VPNs) can have sockets in multiple VRFs for forwarding packets.

The special 'ANY VRF' context allows a single instance of a daemon to provide a service across all VRFs.
e.g., ip vrf exec any /usr/sbin/sshd

The 'any' context applies to listen sockets only; connected sockets are in a VRF context. Child sockets accepted by the daemon acquire the VRF context of the network device the connection originated on.

The 'ANY VRF' context can also be used to display all addresses, routes or neighbors in the kernel cache. That is, 'ip addr show', 'ip route ls', 'ifconfig', 'arp -n', etc, show all network data for the namespace.

About this Patch Set
--------------------
This is not a complete conversion of the networking stack, only a small sampling to test the waters. Only changes are to core IPv4 code [2] which is sufficient to illustrate the fundamental concept. Changes from struct net to net_ctx are very repetitive.

I'm sure there are a lot of oversights and bugs, but the intent here is to solicit feedback on the overall idea.

Examples
--------
To illustrate the VRF patches consider a system with 18 NICs:
- eth0, eth17 are in default namespace (e.g., management namespace)

- eth1 - eth8 are in group1 namespace
- eth1 - eth4 are in VRF 11
- eth5 - eth8 are in VRF 13

- eth9 - eth16 are in group2 namespace
- eth9 - eth12 are in VRF 21
- eth13 - eth16 are in VRF 23

- Addresses assigned to each interface:
- eth1: 1.1.1.1/24
- eth2: 2.2.2.1/24
- eth3: 3.3.3.1/24
- eth4: 4.4.4.1/24
- eth5: 1.1.1.1/24 (not a typo, duplicate address in different vrfs)
- eth6: 6.6.6.1/24
- eth7: 7.7.7.1/24
- eth8: 8.8.8.1/24

- openlldpd is started in each namespace

1. device list is VRF agnostic
- ifconfig -a, ip link show, /proc/net/dev

     --> default namespace shows only eth0 and eth17
     --> group1 namespace shows only eth1 - eth8
     --> group2 namespace shows only eth9 - eth16
         - ip shows vrf assignment of each link

    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 vrf 11 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 02:ab:cd:02:00:01 brd ff:ff:ff:ff:ff:ff

2. address, route, neighbor list is VRF aware
- ifconfig, ip addr show, ip route ls, /proc/net/route

     --> shows only addresses for VRF id of task unless id is 'any'

in VRF 1:
ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

        ether 02:ab:cd:02:00:01  txqueuelen 1000  (Ethernet)

...

No addresses are shown. But if the command is run in VRF 11 or VRF 'any'
ip vrf exec 11 ip addr show dev eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 vrf 11 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:ab:cd:02:00:01 brd ff:ff:ff:ff:ff:ff
inet 1.1.1.1/24 brd 1.1.1.255 scope global eth1
valid_lft forever preferred_lft forever

3. start ssh in group1 namespace
ip netns exec group1 ip vrf exec 11 /usr/sbin/sshd -d ssh to 1.1.1.1 via eth1

ip netns exec group1 ip vrf exec 13 /usr/sbin/sshd -d ssh to 1.1.1.1 via eth5
--> same namespace but different VRFs

4. One ssh instance handles VRFs in group1 namespace ip netns exec group1 ip vrf exec any /usr/sbin/sshd

--> ssh to any address in the namespace works

References
----------
[1] http://sourceforge.net/projects/linux-vrf

[2] http://www.spinics.net/lists/netdev/msg298368.html

[3] To build only enable core ipv4 code. Disable IPv6, netfilter, ipsec, etc.

David Ahern (29):
net: Introduce net_ctx and macro for context comparison
net: Flip net_device to use net_ctx
net: Flip sock_common to net_ctx
net: Add net_ctx macros for skbuffs
net: Flip seq_net_private to net_ctx
net: Flip fib_rules and fib_rules_ops to use net_ctx
net: Flip inet_bind_bucket to net_ctx
net: Flip fib_info to net_ctx
net: Flip ip6_flowlabel to net_ctx
net: Flip neigh structs to net_ctx
net: Flip nl_info to net_ctx
net: Add device lookups by net_ctx
net: Convert function arg from struct net to struct net_ctx
net: vrf: Introduce vrf header file
net: vrf: Add vrf to net_ctx struct
net: vrf: Set default vrf
net: vrf: Add vrf context to task struct
net: vrf: Plumbing for vrf context on a socket
net: vrf: Add vrf context to skb
net: vrf: Add vrf context to flow struct
net: vrf: Add vrf context to genid's
net: vrf: Set VRF id in various network structs
net: vrf: Enable vrf checks
net: vrf: Add support to get/set vrf context on a device
net: vrf: Handle VRF any context
net: vrf: Change single_open_net to pass net_ctx
net: vrf: Add vrf checks and context to ipv4 proc files
iproute2: vrf: Add vrf subcommand
iproute2: Add vrf option to ip link command

 fs/proc/base.c                   |  94 +++++++++++++++++++++++++
 fs/proc/proc_net.c               |  22 +++++-
 include/linux/inetdevice.h       |  12 ++--
 include/linux/init_task.h        |   1 +
 include/linux/netdevice.h        |  44 +++++++++++-
 include/linux/sched.h            |   2 +
 include/linux/seq_file_net.h     |  10 +--
 include/linux/skbuff.h           |   5 ++
 include/net/addrconf.h           |  22 +++---
 include/net/arp.h                |   2 +-
 include/net/dst.h                |  16 ++---
 include/net/fib_rules.h          |  10 ++-
 include/net/flow.h               |  10 ++-
 include/net/inet6_hashtables.h   |  19 +++---
 include/net/inet_hashtables.h    |  60 ++++++++++------
 include/net/inet_sock.h          |   1 +
 include/net/inet_timewait_sock.h |   1 +
 include/net/ip.h                 |  10 +--
 include/net/ip6_fib.h            |   4 +-
 include/net/ip6_route.h          |  24 +++----
 include/net/ip_fib.h             |  38 +++++++----
 include/net/ipv6.h               |  14 +++-
 include/net/neighbour.h          |  93 +++++++++++++++++++++----
 include/net/net_namespace.h      |  39 +++++++++--
 include/net/netlink.h            |   5 +-
 include/net/route.h              |  46 +++++++------
 include/net/sock.h               |  21 ++++--
 include/net/tcp.h                |   1 +
 include/net/transp_v6.h          |   2 +-
 include/net/udp.h                |   8 +--
 include/net/vrf.h                |  36 ++++++++++
 include/net/xfrm.h               |  28 ++++----
 include/uapi/linux/if_link.h     |   1 +
 include/uapi/linux/in.h          |   1 +
 kernel/fork.c                    |   2 +
 net/core/dev.c                   |  95 +++++++++++++++++++++++---
 net/core/fib_rules.c             |  36 ++++++----
 net/core/flow.c                  |   5 +-
 net/core/neighbour.c             | 106 +++++++++++++++--------------
 net/core/rtnetlink.c             |  12 ++++
 net/core/skbuff.c                |  12 ++++
 net/core/sock.c                  |   2 +
 net/ipv4/af_inet.c               |  20 ++++--
 net/ipv4/arp.c                   |  76 ++++++++++++---------
 net/ipv4/datagram.c              |   6 +-
 net/ipv4/devinet.c               |  64 ++++++++++++------
 net/ipv4/fib_frontend.c          |  83 ++++++++++++++---------
 net/ipv4/fib_rules.c             |  12 ++--
 net/ipv4/fib_semantics.c         |  38 +++++++----
 net/ipv4/fib_trie.c              |  24 +++++--
 net/ipv4/icmp.c                  |  40 ++++++-----
 net/ipv4/igmp.c                  |  53 +++++++++------
 net/ipv4/inet_connection_sock.c  |  23 ++++---
 net/ipv4/inet_diag.c             |  13 ++--
 net/ipv4/inet_hashtables.c       |  42 +++++++-----
 net/ipv4/inet_timewait_sock.c    |   1 +
 net/ipv4/ip_input.c              |   6 +-
 net/ipv4/ip_options.c            |  20 +++---
 net/ipv4/ip_output.c             |  16 +++--
 net/ipv4/ip_sockglue.c           |  32 +++++++--
 net/ipv4/ipconfig.c              |   6 +-
 net/ipv4/ipmr.c                  |  53 +++++++++------
 net/ipv4/netfilter.c             |  13 ++--
 net/ipv4/ping.c                  |  41 +++++------
 net/ipv4/proc.c                  |  10 +--
 net/ipv4/raw.c                   |  48 ++++++++-----
 net/ipv4/route.c                 | 143 +++++++++++++++++++++++----------------
 net/ipv4/syncookies.c            |   6 +-
 net/ipv4/tcp_ipv4.c              |  57 +++++++++-------
 net/ipv4/tcp_minisocks.c         |   1 +
 net/ipv4/udp.c                   | 122 ++++++++++++++++++---------------
 net/ipv4/udp_diag.c              |  11 +--
 net/ipv4/xfrm4_policy.c          |  14 ++--
 net/netlink/af_netlink.c         |  12 ++++
 net/sctp/protocol.c              |  10 +--
 net/xfrm/xfrm_policy.c           |   9 +--
 76 files changed, 1415 insertions(+), 682 deletions(-)
 create mode 100644 include/net/vrf.h
Signature · 1 line
1.9.3 (Apple Git-50)