net/ipv4 for Source VIPA support, kernel BK Head
Subsystems (4)
PATCH 1/1 net/ipv4 for Source VIPA support, kernel BK Head
The following small patch (applies to BK head) addresses issues relevant for transparent NIC failover (especially in case of NFS). It allows to configure on a per device basis via sysctl an IP address (Source Virtual IP Address - Source VIPA) that is set as IP source address for all connections for which no bind has been applied. ?To allow for NIC failover one then just needs:
1. A dummy-Device set up with the Source VIPA
2. Outbound routes via both/all redundant NICs for the relevant packets (more precisely: dynamic routing with for example ZEBRA)
3. Routes to the Source VIPA on the relevant router having the IPs of the redundant NICs configured as gateways (more precisely: dynamic routing with for example ZEBRA)
Dynamic routing is mandatory as it is necessary that dead routes (e.g. NIC dead) are removed at the relevant router.
We tested this patch in the desired NFS failover usage scenario and of course without any Source VIPA configured.
The reason for the development of this patch is that the alternatives we thought of have serious limitations for the intended usage scenarios:
1. A User space tool intercepting connects and issuing binds (configuration on a per application basis) (refer to: http://oss.software.ibm.com/linux390/useful_add-ons_vipa.shtml)
This approach does not allow for NFS failover which we consider to be a very important use case because NFS works in kernel.
2. ip route xxx.xxx.xxx.xxx/xx src SourceVIPA
OSPF, etc. do not support automatic setup of and discovery of desired source addresses. As a consequence one would have to configure static routes for all use cases which is not desirable in complex routing scenarios and especially in presence of dynamic routing.
3. netfilter ((S)NAT)
NAT takes place after routing is applied and an IP address (e.g. IP of the output NIC) has been set for a packet. Consequently, returned packets are routed to the original IP address. As a result no failover is possible.
4. NIC bonding
There is a strong dependence on the switches' timeout for the IP/MAC pair. In addition to that, as far as we know not all NICs support bonding with failover.
I hope I described the overall use case comprehensible enough to clarify why we consider this patch as very useful and important.
Einar Lueck
include/linux/inetdevice.h +3 -0
@@ -27,6 +27,9 @@int no_policy;int force_igmp_version;void *sysctl; +#ifdef CONFIG_IP_SOURCEVIPA +__u32 source_vipa; +#endif };extern struct ipv4_devconf ipv4_devconf;
include/linux/sysctl.h +3 -0
@@ -393,6 +393,9 @@NET_IPV4_CONF_FORCE_IGMP_VERSION =17 ,NET_IPV4_CONF_ARP_ANNOUNCE =18 ,NET_IPV4_CONF_ARP_IGNORE =19 , +#ifdef CONFIG_IP_SOURCEVIPA +NET_IPV4_CONF_SOURCE_VIPA =20 +#endif };/* /proc/sys/net/ipv4/netfilter */
net/ipv4/Kconfig +14 -0
@@ -115,6 +115,20 @@ handled by the klogd daemon which is responsible for kernel messages ("man klogd"). +config IP_SOURCEVIPA + bool "IP: Source Virtual IP Address" + help + If you say Y you are able to configure on a per device basis + virtual source ip addresses to be set for not explicitly + bound sockets. Thereby, one may force applications like + FTP, NFS, etc. to implicitly bind to dummy interfaces. + On the basis of dummy interfaces one may decouple applications + from physical interfaces and may as a consequence achieve a higher + degree of fault tolerance. + + If unsure, say N. + + config IP_PNP bool "IP: kernel level autoconfiguration" depends on INET
net/ipv4/devinet.c +195 -1
261 lines not on this page · raw message
Quoted 5 lines · Einar Lueck
The following small patch (applies to BK head) addresses issues relevant for transparent NIC failover (especially in case of NFS). It allows to configure on a per device basis via sysctl an IP address (Source Virtual IP Address - Source VIPA) that is set as IP source address for all connections for which no bind has been applied. ?To allow for NIC failover one then just needs:
1. A dummy-Device set up with the Source VIPA
2. Outbound routes via both/all redundant NICs for the relevant packets (more precisely: dynamic routing with for example ZEBRA)
3. Routes to the Source VIPA on the relevant router having the IPs of the redundant NICs configured as gateways (more precisely: dynamic routing with for example ZEBRA)
Dynamic routing is mandatory as it is necessary that dead routes (e.g. NIC dead) are removed at the relevant router.
Is there anything here that cannot already be done by the ip route command and iptables nat ?
Quoted 3 lines · Einar Lueck
The following small patch (applies to BK head) addresses issues relevant for transparent NIC failover (especially in case of NFS). It allows to configure on a per device basis via sysctl an IP address (Source Virtual IP Address - Source VIPA) that is set as IP source address for all connections for which no bind has been applied. ?To allow for NIC failover one then just needs:
1. A dummy-Device set up with the Source VIPA
Why cant you use loopback as the placeholder interface for the VIPA IP?
Quoted 1 line
2. Outbound routes via both/all redundant NICs for the relevant packets (more precisely: dynamic routing with for example ZEBRA)
# telnet localhost zebra ... host# en host(config)# in lo host(config-if)# ip address <desired address>/32 host(config-if)# end host# wri fi
hey presto, "virtual IP" which you can redistribute connected in ospfd/ripd whatever and publish in DNS.
Quoted 3 lines
The reason for the development of this patch is that the alternatives we thought of have serious limitations for the intended usage scenarios:
1. A User space tool intercepting connects and issuing binds (configuration on a per application basis) (refer to: http://oss.software.ibm.com/linux390/useful_add-ons_vipa.shtml) This approach does not allow for NFS failover which we consider to be a very important use case because NFS works in kernel.
What problems did you ahve?
Quoted 1 line
2. ip route xxx.xxx.xxx.xxx/xx src SourceVIPA OSPF, etc. do not support automatic setup of and discovery of desired source addresses.
ip route add default via <gateway> src <virtual ip>
Having zebra be able to apply route-maps to routes and set src would be useful too.
But why do you need this even? See below about replies. Just publish the virtual IP in DNS, not the interface addresses.
Quoted 1 line
As a consequence one would have to configure static routes for all use cases which is not desirable in complex routing scenarios and especially in presence of dynamic routing. 3. netfilter ((S)NAT) NAT takes place after routing is applied and an IP address (e.g. IP of the output NIC) has been set for a packet. Consequently, returned packets are routed to the original IP address. As a result no failover is possible.
TCP responses to requests that come in addressed to the virtual IP will automatically have source of that virtual IP. For UDP too if app binds to the virtual IP or uses IP_HDRINCL. Not sure about kernel UDP though, but I bet linux-nfs list would be amienable to any changes needed for knfsd to work nicely with virtual/loopback IPs.
Quoted 3 lines
4. NIC bonding
There is a strong dependence on the switches' timeout for the IP/MAC pair. In addition to that, as far as we know not all NICs support bonding with failover.
How does bonding come into it?
Quoted 3 lines
I hope I described the overall use case comprehensible enough to clarify why we consider this patch as very useful and important.
Einar Lueck
regards,
Signature · 3 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: <<<<< EVACUATION ROUTE <<<<<
Please apologize: I fear I have not specified the setups we address precisely enough: I think it helps to imagine the following scenario:
1. SERVER1 mounts some NFS stuff exported by SERVER2.
2. SERVER1 has two NICs.
3. The souce IP address of the related packets is the IP address of the NIC the kernel identifies for the outbound route, or the IP address specified via a static route (ip route .... src SVIPA)
4. In case no static routes are allowed and iptables NAT is no option, the relevant packets thus have the IP of a NIC.
5. In case the corresponding NIC dies, we have a serious problem.
Quoted 8 lines · On Donnerstag, 2. September 2004 18:22 Paul Jakma wrote:
Quoted 3 lines · Einar Lueck
The following small patch (applies to BK head) addresses issues relevant for transparent NIC failover (especially in case of NFS). It allows to configure on a per device basis via sysctl an IP address (Source Virtual IP Address - Source VIPA) that is set as IP source address for all connections for which no bind has been applied. ?To allow for NIC failover one then just needs:
1. A dummy-Device set up with the Source VIPA
Why cant you use loopback as the placeholder interface for the VIPA IP?
This is works of course, too. But it does not solve the problem described above. But, maybe I misunderstood you. Please correct me, if I am wrong.
Quoted 12 lines
Quoted 3 lines
2. Outbound routes via both/all redundant NICs for the relevant packets (more precisely: dynamic routing with for example ZEBRA)
# telnet localhost zebra ... host# en host(config)# in lo host(config-if)# ip address <desired address>/32 host(config-if)# end host# wri fi
Quoted 2 lines
hey presto, "virtual IP" which you can redistribute connected in ospfd/ripd whatever and publish in DNS.
I think this is the same as the first point.
Quoted 5 lines
Quoted 4 lines
The reason for the development of this patch is that the alternatives we thought of have serious limitations for the intended usage scenarios:
Quoted 7 lines
Quoted 5 lines
1. A User space tool intercepting connects and issuing binds (configuration on a per application basis) (refer to:
http://oss.software.ibm.com/linux390/useful_add-ons_vipa.shtml)
This approach does not allow for NFS failover which we consider to be a very important use case because NFS works in kernel.What problems did you ahve?
NFS works in kernel. Therefore we could not intercept the corresponding "connect" calls. As a result, the problem scenario described above could not be solved.
Quoted 6 lines
TCP responses to requests that come in addressed to the virtual IP will automatically have source of that virtual IP. For UDP too if app binds to the virtual IP or uses
IP_HDRINCL. Not sure about kernel UDP though, but I bet linux-nfs list would be amienable to any changes needed for knfsd to work nicely with virtual/loopback IPs.
You are right, but the packets do not come in, they go out, as I tried to illustrate above. Anyway, in the other case you are completely right.
Quoted 8 lines
Quoted 6 lines
4. NIC bonding
There is a strong dependence on the switches' timeout for the IP/MAC pair. In addition to that, as far as we know not all NICs support bonding with failover.
How does bonding come into it?
Bonding offers a failover facility. For more details, please refer to:Documentation/networking/bonding.txt in the kernel tree.
Regards
Einar
Quoted 11 lines · Einar Lueck
Please apologize: I fear I have not specified the setups we address precisely enough: I think it helps to imagine the following scenario:
1. SERVER1 mounts some NFS stuff exported by SERVER2.
2. SERVER1 has two NICs.
3. The souce IP address of the related packets is the IP address of the NIC the kernel identifies for the outbound route, or the IP address specified via a static route (ip route .... src SVIPA)
4. In case no static routes are allowed and iptables NAT is no option, the relevant packets thus have the IP of a NIC.
5. In case the corresponding NIC dies, we have a serious problem.
Quoted 1 line
This is works of course, too. But it does not solve the problem described above. But, maybe I misunderstood you. Please correct me, if I am wrong.
I dont see why it wouldnt work, it almost undoubtedly will work for NFS over TCP. And any problems to cause it to not work would be best taking up on the linux-nfs list in order to have a "bind to address" option added to knfsd.
Why would it not work?
Quoted 4 lines
Quoted 2 lines
hey presto, "virtual IP" which you can redistribute connected in ospfd/ripd whatever and publish in DNS.
I think this is the same as the first point.
But you must describe why it would not work. *Why*?
Quoted 1 line
NFS works in kernel. Therefore we could not intercept the corresponding "connect" calls. As a result, the problem scenario described above could not be solved.
Why could it not be solved? And why is the answer not "ask the knfsd people to provide bind-to-ip option"?
Quoted 1 line
You are right, but the packets do not come in, they go out, as I tried to illustrate above. Anyway, in the other case you are completely right.
But on a server, the packets that go out tend to be replies to requests. Or at least, the only packets of interest are replies. It's a rare server that just off its own bat goes and talks to clients which have not communicated first with the server before.
Anyway, even if the server for some reason initiated traffic, the correct answer surely is "modify the server to bind to a specific address", no?
Quoted 2 lines
Bonding offers a failover facility. For more details, please refer to:
Documentation/networking/bonding.txtin the kernel tree.
Right, but what does bonding (layer 2) have to do with virtual IPs and IP source address?
Quoted 2 lines
Regards
Einar
regards,
Signature · 5 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: People say I live in my own little fantasy world... well, at least they *know* me there! -- D.L. Roth
Quoted 5 lines · On Donnerstag, 2. September 2004 22:59 Paul Jakma wrote:
I dont see why it wouldnt work, it almost undoubtedly will work for NFS over TCP. And any problems to cause it to not work would be best taking up on the linux-nfs list in order to have a "bind to address" option added to knfsd.
I just set up the loopback interface via ZEBRA/OSPF as You described it and checked via tcpdump the source IP address of the related NFS packets. The kernel chooses the IP address of the NIC he routes the packets over as the source IP address and not the Source VIPA configured for loopback.
You are right, it would be one option to have a "bind to address" in KNFSD. But our idea was to implement a feature well known from other operating systems like AIX to Linux because this feature is quite popular and liked especially by large customers. As You have read for sure such a feature adding redundant functionality to the kernel is not desired. So maybe we should continue our discussion privately. Thanks for Your suggestions!
Quoted 4 lines
Why could it not be solved? And why is the answer not "ask the knfsd people to provide bind-to-ip option"?
We would win a facility allowing for a Source VIPA for all kinds of servers not offering an explicit bind option. So: Due to the feature port idea mentioned above.
Quoted 4 lines
But on a server, the packets that go out tend to be replies to requests. Or at least, the only packets of interest are replies. It's a rare server that just off its own bat goes and talks to clients which have not communicated first with the server before.
The enterprise customers we care about have for example servers that utilize other servers (application servers utilizing a database or a NFS server, etc.). So to generate replies these servers need replies of other servers .
Quoted 4 lines
Anyway, even if the server for some reason initiated traffic, the correct answer surely is "modify the server to bind to a specific address", no?
As mentioned above ;)
Quoted 7 lines
Quoted 3 lines
Bonding offers a failover facility. For more details, please refer to:
Documentation/networking/bonding.txtin the kernel tree.Right, but what does bonding (layer 2) have to do with virtual IPs and IP source address?
If we focus for a moment just on the NIC-fail-over issue (not caring about layers, virtual IPs, etc.) then bonding offers the desired failover with some restriction. This is the reason why I mentioned it in this context.
Again, thanks for Your suggestions and maybe we should continue our discussion privately.
Regards
Einar.
Quoted 1 line · Einar Lueck
I just set up the loopback interface via ZEBRA/OSPF as You described it and checked via tcpdump the source IP address of the related NFS packets. The kernel chooses the IP address of the NIC he routes the packets over as the source IP address and not the Source VIPA configured for loopback.
Ah, I didnt say adding an address to loopback would make everything use it. Merely that loopback already exists as an interface to which from which you can 'hang' your VIPA - no need for a new interface.
You could try:
ip route change default via <gateway> src <vipa>
Presuming the NFS clients are behind a gateway. If also onlink, you need to modify the connected routes and change the src there too.
Quoted 1 line
You are right, it would be one option to have a "bind to address" in KNFSD.
It might even already exist.. who knows. ;)
Quoted 1 line
But our idea was to implement a feature well known from other operating systems like AIX to Linux because this feature is quite popular and liked especially by large customers.
Right, but Linux can already do it. The configuration might not be the same as AIX, but that's not a good reason.. if it were, you should also be porting smit to Linux to satisfy your customers ;)
Linux can already do what you want I think. Just a matter of configuring it.
Quoted 1 line
We would win a facility allowing for a Source VIPA for all kinds of servers not offering an explicit bind option. So: Due to the feature port idea mentioned above.
Have you tried playing with ip route?
ip route <destination> ...... src <source address>
Quoted 1 line
If we focus for a moment just on the NIC-fail-over issue (not caring about layers, virtual IPs, etc.) then bonding offers the desired failover with some restriction. This is the reason why I mentioned it in this context.
Ah.
Quoted 2 lines
Again, thanks for Your suggestions and maybe we should continue our discussion privately.
Sure.
Quoted 3 lines
Regards
Einar.
regards,
Signature · 3 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: kernel panic: write-only-memory (/dev/wom0) capacity exceeded.
Quoted 6 lines · Paul Jakma
You could try:
ip route change default via <gateway> src <vipa>
Presuming the NFS clients are behind a gateway. If also onlink, you need to modify the connected routes and change the src there too.
We have tried it this way, of course, and it works. But the corresponding routes are static. In our context, all routes have to be dynamic (for fault tolerance and due to configuration overhead). Zebra and OSPFD do not support this type of route. As a consequence, this approach does not work for us. Anyway, iptables is the way to do it.
Regards,
Einar.
Quoted 1 line · Einar Lueck
We have tried it this way, of course, and it works. But the corresponding routes are static. In our context, all routes have to be dynamic (for fault tolerance and due to configuration overhead).
then change the 'connected' routes. that's all you need to affect source address selection. All your other dynamic routes will still work.
Quoted 1 line
Zebra and OSPFD do not support this type of route.
ospfd never would. You could add the required support to zebra if you wished. ;)
Quoted 1 line
As a consequence, this approach does not work for us. Anyway, iptables is the way to do it.
One way, but quite a kludge to effectively NAT local machines. You'd be much better up configuring the routes which affect source address selection to taste.
anyway..
regards,
Signature · 4 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: All of the animals except man know that the principal business of life is to enjoy it.
Quoted 20 lines · Einar Lueck
Quoted 5 lines · On Donnerstag, 2. September 2004 22:59 Paul Jakma wrote:
I dont see why it wouldnt work, it almost undoubtedly will work for NFS over TCP. And any problems to cause it to not work would be best taking up on the linux-nfs list in order to have a "bind to address" option added to knfsd.
I just set up the loopback interface via ZEBRA/OSPF as You described it and checked via tcpdump the source IP address of the related NFS packets. The kernel chooses the IP address of the NIC he routes the packets over as the source IP address and not the Source VIPA configured for loopback.
You are right, it would be one option to have a "bind to address" in KNFSD. But our idea was to implement a feature well known from other operating systems like AIX to Linux because this feature is quite popular and liked especially by large customers. As You have read for sure such a feature adding redundant functionality to the kernel is not desired. So maybe we should continue our discussion privately. Thanks for Your suggestions!
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Why could it not be solved? And why is the answer not "ask the knfsd people to provide bind-to-ip option"?
We would win a facility allowing for a Source VIPA for all kinds of servers not offering an explicit bind option. So: Due to the feature port idea mentioned above.
btw, something very similar is implemented and used by linux-vserver (it's called chbind) to restrict 0.0.0.0 (IADDR_ANY) binds to specified address(es)
if you need more details, just let me know ...
best,
Herbert
Quoted 40 lines
Quoted 4 lines
But on a server, the packets that go out tend to be replies to requests. Or at least, the only packets of interest are replies. It's a rare server that just off its own bat goes and talks to clients which have not communicated first with the server before.
The enterprise customers we care about have for example servers that utilize other servers (application servers utilizing a database or a NFS server, etc.). So to generate replies these servers need replies of other servers .
Quoted 4 lines
Anyway, even if the server for some reason initiated traffic, the correct answer surely is "modify the server to bind to a specific address", no?
As mentioned above ;)
Quoted 7 lines
Quoted 3 lines
Bonding offers a failover facility. For more details, please refer to:
Documentation/networking/bonding.txtin the kernel tree.Right, but what does bonding (layer 2) have to do with virtual IPs and IP source address?
If we focus for a moment just on the NIC-fail-over issue (not caring about layers, virtual IPs, etc.) then bonding offers the desired failover with some restriction. This is the reason why I mentioned it in this context.
Again, thanks for Your suggestions and maybe we should continue our discussion privately.
Regards
Einar.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · Paul Jakma
ip route add default via <gateway> src <virtual ip>
Sitenote: unfortunately, "src <...>" doesn't work for IPv6 routes, at least in 2.4 -- can someone confirm this problem to still exist in current 2.6?
Copying netdev for the record.
Regards,
Daniel
David S. Miller
maintainerQuoted 6 lines · On Mon, 6 Sep 2004 01:02:19 +0200 Daniel Roesen <[email protected]> wrote:
Quoted 1 line · Paul Jakma
ip route add default via <gateway> src <virtual ip>
Sitenote: unfortunately, "src <...>" doesn't work for IPv6 routes, at least in 2.4 -- can someone confirm this problem to still exist in current 2.6?
That's right, no routing by source in ipv6 yet, the folks working with the USAGI guys on MIPV6 support will add the feature.