VRFs and the scalability of namespaces
Subsystems (2)
Hi Eric:
As you suggested [1] I am starting a new thread to discuss scalability problems using namespaces for VRFs.
Background
----------
Consider a single system that wants to provide VRF-based features with support for N VRFs. N could easily be 2048 (e.g., 6Wind, [2]), 4000 (e.g., Cisco, [3]) or even higher.
The single system with support for N VRFs runs M services (e.g., quagga, cdp, lldp, stp, strongswan, some homegrown routing protocol) and includes standard system services like sshd. Furthermore, a system also includes monitoring programs like snmpd and tcollector. In short, M is easily 20 processes that need to have a presence across all VRFs.
Network Namespaces for VRFs
---------------------------
For the past 4 years or so the response to VRF questions is a drum beat of "use network namespaces". But namespaces are not a good match for VRFs.
1. Network namespaces are a complete separation of the networking stack from network devices up. VRFs are an L3 concept. Using namespaces forces an L3 separation concept onto L2 apps -- lldp, cdp, etc.
There are use cases when you want device level separation, use cases where you want only L3 and up separation, and cases where you want both (e.g., divy up the netdevices in a system across some small number of namespaces and then provide VRF based features within a namespace).
2. Scalability of apps providing service as namespaces are created. How do you create the presence for each service in a network namespace?
a. Spawn a new process for each namespace? brute force approach and extremely resource intensive. e.g., the quagga example [4]
b. spawn a thread for each namespace? Better than a full process but still a heavyweight solution
c. create a socket per namespace. Better but still this is a resource intensive solution -- N listen sockets per service and each service needs to be modified for namespace support. For opensource software that means each project has to agree that namespace awareness is relevant and agree to take the patches.
3. Just creating a network namespace consumes non-negligible amount of memory -- ~200kB for the 3.10 kernel. I believe the /proc entries are the bulk of that memory usage. 200kB/namespace is again a lot of wasted memory and overhead.
4. For a single process to straddle multiple namespaces it has to run with full root privileges -- CAP_SYS_ADMIN -- to use setns. Using network sockets does not require a process to run as root at all unless it wants privileged ports in which case CAP_NET_BIND_SERVICE is sufficient, not full root.
The Linux kernel needs proper VRF support -- as an L3 concept. A capability to run a process in a "VRF any" context provides a resource efficient solution where a single process with a single listen socket works across all VRFs in a namespace and then connected sockets have a specific VRF context.
Before droning on even more, does the above provide better context on the general problem?
Thanks,
David
[1] https://lkml.org/lkml/2014/9/26/840
[2] http://www.6wind.com/6windgate-performance/ip-forwarding
[4] https://lists.quagga.net/pipermail/quagga-users/2010-February/011351.html
Quoted 5 lines · On Fri, 26 Sep 2014 16:37:26 -0600 David Ahern <[email protected]> wrote:
Hi Eric:
As you suggested [1] I am starting a new thread to discuss scalability problems using namespaces for VRFs.
Whining without suggesting a solution or providing code will probably result in no action.
Quoted 11 lines · Stephen Hemminger
Quoted 5 lines · On Fri, 26 Sep 2014 16:37:26 -0600 David Ahern <[email protected]> wrote:
Hi Eric:
As you suggested [1] I am starting a new thread to discuss scalability problems using namespaces for VRFs.
Whining without suggesting a solution or providing code will probably result in no action.
I have a suggestion; I was trying to set a context on the problem.
David
Quoted 4 lines · David Ahern
Hi Eric:
As you suggested [1] I am starting a new thread to discuss scalability problems using namespaces for VRFs.
I will accept as a given that using network namespaces at a scale of 1000s and with lots of little applications listening for new connections (but generally not doing anything) is outside the classic networking usage of linux and of namespaces and so does not work out of the box and that very least some fixes are necessary.
However your premise that network namespaces are unsupportable unfixable and fundamentally unscalable for what you want to do is unsupported.
The most difficult problem for high levels of efficiency is that that of modifying applications so that they are VRF/namespace aware. That they look at the appropriate set of dns resolvers for the namespace, that when messages are logged the messages report not just the ip address but the context that ip address came from. There are no magic solutions to make that kinds of deep and fundamental code modifications.
I can totally see it being frustrating to use linux as a switch OS when it doesn't quite do what you want on the hardware you want to use, and definitely not with the efficiencies you want.
I will tell you what network namespaces get you. Network namespaces deliver the full power of the linux network stack, every interesting feature works, and network namespaces provide a path where you can use unmodified linux applications.
When you say "proper VRF support" what I hear is that you think something new needs to be added to the linux network stack (called a VRF) with a new userspace interface that somehow because it lacks features is better.
Quoted 11 lines
Background
----------
Consider a single system that wants to provide VRF-based features with support for N VRFs. N could easily be 2048 (e.g., 6Wind, [2]), 4000 (e.g., Cisco, [3]) or even higher.The single system with support for N VRFs runs M services (e.g., quagga, cdp, lldp, stp, strongswan, some homegrown routing protocol) and includes standard system services like sshd. Furthermore, a system also includes monitoring programs like snmpd and tcollector. In short, M is easily 20 processes that need to have a presence across all VRFs.
And trying to run it all on what would be considered an underpowered in most contexts.
Quoted 2 lines
Before droning on even more, does the above provide better context on the general problem?
It provides a rough context on what you are trying to do. Use linux as the OS to run on a switch.
It doesn't actually provide much in the way of context actual problems that show up when you try to use network namespaces. Which is what I was expecting the discussion would be about, and which would I expect be a productive conversation.
Eric
Hi Eric
Quoted 4 lines · Eric W. Biederman
When you say "proper VRF support" what I hear is that you think something new needs to be added to the linux network stack (called a VRF) with a new userspace interface that somehow because it lacks features is better.
From my perspective the existing mechanisms do not seem to provide a sufficient solution for VRFs.
Quoted 10 lines
Quoted 2 lines
Before droning on even more, does the above provide better context on the general problem?
It provides a rough context on what you are trying to do. Use linux as the OS to run on a switch.
It doesn't actually provide much in the way of context actual problems that show up when you try to use network namespaces. Which is what I was expecting the discussion would be about, and which would I expect be a productive conversation.
I don't know how else to explain it beyond what I said in the first email. I listed several specific examples of how namespaces are not an appropriate model for VRFs. Do you disagree on any of those points? Need clarification on any of them? ie., What more were you expecting?
David
Hi,
Quoted 5 lines · David Ahern
The Linux kernel needs proper VRF support -- as an L3 concept. A capability to run a process in a "VRF any" context provides a resource efficient solution where a single process with a single listen socket works across all VRFs in a namespace and then connected sockets have a specific VRF context.
In case you want to have full blown VRF support as in BGP-MPLS-VRF setups I agree that namespaces seem to be not the right tool for the job, even not closely. Although if those namespaces get split up in a more lightweight routing/forwarding namespace and the normal full blown network namespace it seems to be too cumbersome for user space to manage those.
Did you already did an investigation how maybe the rule and table features could be exploited to suite your needs? Some time back I suggested something like "ip route table foo exec ....", keep an default routing lookup indicator in task_struct which gets implicitly propagated to rtnetlink routing table requests/modification for the requested table. Tables already can be specified via rtnetlink, so no change would be needed here.
For sockets something like SO_BINDTOTABLE might work, maybe even we can by default use the task_struct information to also bind the sockets to the per-process table. We certainly need to preserve the routing information on the socket as we need those in icmp error handling (e.g. where to apply ipv4/ipv6 redirects too). Directing incoming packets to specific table also works via ip-rule-iif match.
Advantage with the ip route table foo exec... method would be, that conversion of some unmodified routing management daemons might be easier, others can either use rtnetlink extended attributes which are already available, and we only need to have per-process context routing table control, which seems not too hard to implement in ip-rule subsystem, but I haven't checked.
The problem I see with rules is that some of those tables already work hand in hand, they already have a implicit semantics, e.g. local, main, default and unspec (this is even worse for IPv6, where addrconf already uses hardcoded tables). Working around this might be very tricky and even more problematic to do from user space.
I think I am not yet sure what features you want from VRFs, some things seem to match the rule/table features but others I think are pretty hard to implement.
I worry a bit about icmp error handling and updates of the routing table, but this is a detail we can look into later if all needed features are known.
Bye,
Hannes
Hi,
Addendum:
Quoted 8 lines · Hannes Frederic Sowa
Did you already did an investigation how maybe the rule and table features could be exploited to suite your needs? Some time back I suggested something like "ip route table foo exec ....", keep an default routing lookup indicator in
task_structwhich gets implicitly propagated to rtnetlink routing table requests/modification for the requested table. Tables already can be specified via rtnetlink, so no change would be needed here.
Quoted 6 lines
For sockets something like
SO_BINDTOTABLEmight work, maybe even we can by default use thetask_structinformation to also bind the sockets to the per-process table. We certainly need to preserve the routing information on the socket as we need those in icmp error handling (e.g. where to apply ipv4/ipv6 redirects too). Directing incoming packets to specific table also works via ip-rule-iif match.
Update and lookup rule ids must be separated, so a process might need to get a tuple of references which table to update and which tables to match in ip rules.
Also some data structures on matching might be change, e.g. an ->action which takes an interface and returns the routing table id in O(1) instead of walking the rules and executing the actions in order.
Quoted 5 lines
The problem I see with rules is that some of those tables already work hand in hand, they already have a implicit semantics, e.g. local, main, default and unspec (this is even worse for IPv6, where addrconf already uses hardcoded tables). Working around this might be very tricky and even more problematic to do from user space.
We might also add an rule reference to net_device so we redirect the route changes during address addition/deletion to a separate table, otherwise user space has to move them non-atomically.
Bye,
Hannes
Hi Hannes:
Quoted 2 lines · Hannes Frederic Sowa
Did you already did an investigation how maybe the rule and table features could be exploited to suite your needs? Some time back I
I did look into the existing multiple table option but not to the extent of creating a POC. It has been on my to-do list for 4+ months now I just have not had time to get to it. Based on a number of Google searches to review the history of VRFs and the kernel, I did see the use of multiple routing tables has been suggested as well and its problems have been delineated. e.g.,
http://www.spinics.net/lists/linux-net/msg17502.html
Quoted 26 lines
suggested something like "ip route table foo exec ....", keep an default routing lookup indicator in
task_structwhich gets implicitly propagated to rtnetlink routing table requests/modification for the requested table. Tables already can be specified via rtnetlink, so no change would be needed here.For sockets something like
SO_BINDTOTABLEmight work, maybe even we can by default use thetask_structinformation to also bind the sockets to the per-process table. We certainly need to preserve the routing information on the socket as we need those in icmp error handling (e.g. where to apply ipv4/ipv6 redirects too). Directing incoming packets to specific table also works via ip-rule-iif match.Advantage with the ip route table foo exec... method would be, that conversion of some unmodified routing management daemons might be easier, others can either use rtnetlink extended attributes which are already available, and we only need to have per-process context routing table control, which seems not too hard to implement in ip-rule subsystem, but I haven't checked.
The problem I see with rules is that some of those tables already work hand in hand, they already have a implicit semantics, e.g. local, main, default and unspec (this is even worse for IPv6, where addrconf already uses hardcoded tables). Working around this might be very tricky and even more problematic to do from user space.
Quoted 3 lines
I think I am not yet sure what features you want from VRFs, some things seem to match the rule/table features but others I think are pretty hard to implement.
The features of note:
- resource efficiency -- not having to create a proces/thread/socket per VRF to have a "presence" in all VRFs. e.g., a VRF any context that allows 1 socket to work across VRFs (L3 raw socket, TCP listen socket, unconnected UDP socket). Daemons run a 'vrf any' context; connected clients run a specific vrf context. For non-connected sockets VRF context can be passed via cmsg.
- same IP address on different interfaces in different vrfs. i.e., VRF specific routing and neighbor tables
- cross VRF routing. ability to receive message on 1 vrf and send it on another. Can be handled by the process itself (e.g., L3 vpns).
Thanks,
David
Quoted 8 lines · David Ahern
The features of note:
- resource efficiency -- not having to create a proces/thread/socket per VRF to have a "presence" in all VRFs. e.g., a VRF any context that allows 1 socket to work across VRFs (L3 raw socket, TCP listen socket, unconnected UDP socket). Daemons run a 'vrf any' context; connected clients run a specific vrf context. For non-connected sockets VRF context can be passed via cmsg.- same IP address on different interfaces in different vrfs. i.e., VRF specific routing and neighbor tables
- cross VRF routing. ability to receive message on 1 vrf and send it on another. Can be handled by the process itself (e.g., L3 vpns).
We have implemented support for at least most of this (excepting duplicate IPs) using routing tables, rules, and (optionally, xorp as the router).
It works ok for our purposes (network simulator), but peformance is not great because you end up with a large number of ip rules and they are effectively evaluated linearly it seems.
A quick way to improve performance in our scenario would be to bind rules to specific interfaces, so that packets process a smaller number of rules when they enter an interface, I think...but I have not looked into it closely.
It is hard to show you an example of this without you installing our software to visualize what we are trying to do, but it our software will work on standard kernels, and we auto-generate a perl script that sets up all of the rules and such. You could compare the network diagram in our GUI with the perl script and I think understand the basics of what we are doing fairly quickly. If you want to take a detailed look, let me know and I'll set you up with a demo license.
Thanks,
Ben
Quoted 8 lines
Thanks,
DavidSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 6 lines · Ben Greear
On 09/29/2014 06:06 AM, David Ahern wrote:
We have implemented support for at least most of this (excepting duplicate IPs) using routing tables, rules, and (optionally, xorp as the router).
My undertanding of multiple routing-tables/rules was that they are closer in semantics to switch/router ACLs than to VRFs, eg., one big difference is that an interface can belong to exactly one VRF at a time, which is not mandated by multiple routing-tables/rules.
Was I mistaken?
--Sowmini
Quoted 8 lines · Sowmini Varadhan
Quoted 6 lines · Ben Greear
On 09/29/2014 06:06 AM, David Ahern wrote:
We have implemented support for at least most of this (excepting duplicate IPs) using routing tables, rules, and (optionally, xorp as the router).
Quoted 6 lines
My undertanding of multiple routing-tables/rules was that they are closer in semantics to switch/router ACLs than to VRFs, eg., one big difference is that an interface can belong to exactly one VRF at a time, which is not mandated by multiple routing-tables/rules.
Was I mistaken?
You can effectively force an interface to belong to a particular virtual router (table). It is not trivial to do, and possibly I have still not covered every possible case. Some rules grow somewhat exponentially as interfaces are added to virtual routers (ie, preference 10 rules).
Here is our setup for a system with a single virtual router, which uses table 10001. vap0, vap1, and eth1 are in this virtual router. There are other interfaces on this system outside of the virtual router, so you can ignore rules related to those.
You have to add CT zones for each virtual router as well.
[root@ath10k-2220 ~]# ip ru show
10: from all to 5.1.1.1 iif eth1 lookup local
10: from all to 4.1.0.1 iif vap0 lookup local
10: from all to 4.2.0.1 iif vap0 lookup local
10: from all to 4.2.0.1 iif vap1 lookup local
10: from all to 5.1.1.1 iif vap0 lookup local
10: from all to 4.1.0.1 iif vap1 lookup local
10: from all to 4.1.0.1 iif vap1 lookup local
10: from all to 5.1.1.1 iif vap1 lookup local
10: from all to 4.1.0.1 iif eth1 lookup local
10: from all to 4.2.0.1 iif vap0 lookup local
10: from all to 4.2.0.1 iif eth1 lookup local
20: from all iif eth1 lookup 10001
20: from all iif vap0 lookup 10001
20: from all iif vap1 lookup 10001
30: from 5.1.1.1 lookup 10001
30: from 4.1.0.1 lookup 10001
30: from 4.2.0.1 lookup 10001
50: from all oif rddVR0 lookup 6
50: from all oif rddVR1 lookup 7
50: from all oif rddVR2 lookup 8
50: from all oif rddVR3 lookup 9
50: from all oif wlan0 lookup 4
50: from all oif wlan1 lookup 5
50: from all oif eth1 lookup 10001
50: from all oif vap0 lookup 10001
50: from all oif vap1 lookup 10001
512: from all lookup local
32766: from all lookup main
32767: from all lookup default
[root@ath10k-2220 ~]# ip -4 route show table all
unreachable default table 10001
4.1.0.0/16 via 4.1.0.1 dev vap0 table 10001
4.2.0.0/16 via 4.2.0.1 dev vap1 table 10001
5.1.1.0/24 dev eth1 table 10001 scope link
default via 192.168.100.1 dev eth0
4.1.0.0/16 dev vap0 proto kernel scope link src 4.1.0.1 4.2.0.0/16 dev vap1 proto kernel scope link src 4.2.0.1 5.1.1.0/24 dev eth1 proto kernel scope link src 5.1.1.1
169.254.0.0/16 dev eth0 scope link metric 1002
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.179 broadcast 4.1.0.0 dev vap0 table local proto kernel scope link src 4.1.0.1 local 4.1.0.1 dev vap0 table local proto kernel scope host src 4.1.0.1 broadcast 4.1.255.255 dev vap0 table local proto kernel scope link src 4.1.0.1 broadcast 4.2.0.0 dev vap1 table local proto kernel scope link src 4.2.0.1 local 4.2.0.1 dev vap1 table local proto kernel scope host src 4.2.0.1 broadcast 4.2.255.255 dev vap1 table local proto kernel scope link src 4.2.0.1 broadcast 5.1.1.0 dev eth1 table local proto kernel scope link src 5.1.1.1 local 5.1.1.1 dev eth1 table local proto kernel scope host src 5.1.1.1 broadcast 5.1.1.255 dev eth1 table local proto kernel scope link src 5.1.1.1 broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 broadcast 192.168.100.0 dev eth0 table local proto kernel scope link src 192.168.100.179
local 192.168.100.179 dev eth0 table local proto kernel scope host src 192.168.100.179 broadcast 192.168.100.255 dev eth0 table local proto kernel scope link src 192.168.100.179
[root@ath10k-2220 ~]# ip route show table 10001
unreachable default
4.1.0.0/16 via 4.1.0.1 dev vap0
4.2.0.0/16 via 4.2.0.1 dev vap1
5.1.1.0/24 dev eth1 scope link
Thanks,
Ben
Quoted 7 lines
--Sowmini
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 14 lines · Ben Greear
Quoted 13 lines · Sowmini Varadhan
Quoted 6 lines · Ben Greear
On 09/29/2014 06:06 AM, David Ahern wrote:
We have implemented support for at least most of this (excepting duplicate IPs) using routing tables, rules, and (optionally, xorp as the router).
My undertanding of multiple routing-tables/rules was that they are closer in semantics to switch/router ACLs than to VRFs, eg., one big difference is that an interface can belong to exactly one VRF at a time, which is not mandated by multiple routing-tables/rules.
Quoted 6 lines
Quoted 1 line
Was I mistaken?
You can effectively force an interface to belong to a particular virtual router (table). It is not trivial to do, and possibly I have still not covered every possible case. Some rules grow somewhat exponentially as interfaces are added to virtual routers (ie, preference 10 rules).
An interesting way of doing it; thanks for the reference point.
Fundamentally the design should be able to assign interfaces to a single VRF, support duplicate IP addresses on different interfaces in different VRFs and be able to scale to 10,000+ netdevices -- devices representing physical ports as well as logical interfaces built on top of them (e.g., sub-interfaces).
David
Quoted 22 lines · David Ahern
Quoted 14 lines · Ben Greear
Quoted 13 lines · Sowmini Varadhan
Quoted 6 lines · Ben Greear
On 09/29/2014 06:06 AM, David Ahern wrote:
We have implemented support for at least most of this (excepting duplicate IPs) using routing tables, rules, and (optionally, xorp as the router).
My undertanding of multiple routing-tables/rules was that they are closer in semantics to switch/router ACLs than to VRFs, eg., one big difference is that an interface can belong to exactly one VRF at a time, which is not mandated by multiple routing-tables/rules.
Quoted 6 lines
Quoted 1 line
Was I mistaken?
You can effectively force an interface to belong to a particular virtual router (table). It is not trivial to do, and possibly I have still not covered every possible case. Some rules grow somewhat exponentially as interfaces are added to virtual routers (ie, preference 10 rules).
Quoted 7 lines
An interesting way of doing it; thanks for the reference point.
Fundamentally the design should be able to assign interfaces to a single VRF, support duplicate IP addresses on different interfaces in different VRFs and be able to scale to 10,000+ netdevices -- devices representing physical ports as well as logical interfaces built on top of them (e.g., sub-interfaces).
Duplicate IP addresses don't go well with current linux stack being a soft end model by default. Current separation is done on arp level today if some kind of strong end model is desired. This calls for some kind of namespaces again. ;)
Bye,
Hannes
Quoted 31 lines · Hannes Frederic Sowa
Quoted 22 lines · David Ahern
Quoted 14 lines · Ben Greear
Quoted 13 lines · Sowmini Varadhan
Quoted 6 lines · Ben Greear
On 09/29/2014 06:06 AM, David Ahern wrote:
We have implemented support for at least most of this (excepting duplicate IPs) using routing tables, rules, and (optionally, xorp as the router).
My undertanding of multiple routing-tables/rules was that they are closer in semantics to switch/router ACLs than to VRFs, eg., one big difference is that an interface can belong to exactly one VRF at a time, which is not mandated by multiple routing-tables/rules.
Quoted 6 lines
Quoted 1 line
Was I mistaken?
You can effectively force an interface to belong to a particular virtual router (table). It is not trivial to do, and possibly I have still not covered every possible case. Some rules grow somewhat exponentially as interfaces are added to virtual routers (ie, preference 10 rules).
Quoted 7 lines
An interesting way of doing it; thanks for the reference point.
Fundamentally the design should be able to assign interfaces to a single VRF, support duplicate IP addresses on different interfaces in different VRFs and be able to scale to 10,000+ netdevices -- devices representing physical ports as well as logical interfaces built on top of them (e.g., sub-interfaces).
Quoted 4 lines
Duplicate IP addresses don't go well with current linux stack being a soft end model by default. Current separation is done on arp level today if some kind of strong end model is desired. This calls for some kind of namespaces again. ;)
arp is per interface as well if you set arp-filter properly, the main problem with duplicate IPs is that you can't (easily?) set up routing rules that match them properly...
Thanks,
Ben
Quoted 4 lines
Bye,
Hannes
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 6 lines · David Ahern
Network Namespaces for VRFs
---------------------------
For the past 4 years or so the response to VRF questions is a drum beat of "use network namespaces". But namespaces are not a good match for VRFs.
Quoted 8 lines
1. Network namespaces are a complete separation of the networking stack from network devices up. VRFs are an L3 concept. Using namespaces forces an L3 separation concept onto L2 apps -- lldp, cdp, etc.
There are use cases when you want device level separation, use cases where you want only L3 and up separation, and cases where you want both (e.g., divy up the netdevices in a system across some small number of namespaces and then provide VRF based features within a namespace).
With regarding to L3 isolation, not limited to VRF, it _does_ sound like a cleaner
solution than the current L2 isolation, given the fact that:
1) Isolating L2 relies on /sys etc. in practice, which means we have to use mount namespace as well. Ideally network isolation should be independent of other namespaces.
2) Some resources still need to be shared, otherwise we have to restart some services in each of network namespaces as you mentioned. The ~200kb memory overhead is not a problem I think, as now most servers have 8Gb+ memory, the real problem is some service, for example udevd, has some dependencies, we would fall back to re-implement a whole init service chain for each
netns, which is the expensive part.
Overall, rather than all or none isolation, we should give the user some flexibility
to specify which network resources we want to isolation, which we don't. Again, I am not saying VRF is the solution, I am looking for some more flexible isolation
solution.
Thanks.