Skip to contentSkip to conversation

ipvs: add sysctl to ignore tunneled packets

PATCH Superseded Alex Gartrell 3 participants 7 messages Sep 10, 2015 – Sep 15, 2015

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ipvs: add sysctl to ignore tunneled packets

This is a way to avoid nasty routing loops when multiple ipvs instances can forward to eachother.

---
 include/net/ip_vs.h             | 11 +++++++++++
 net/netfilter/ipvs/ip_vs_core.c | 10 +++++++++-
 net/netfilter/ipvs/ip_vs_ctl.c  |  7 +++++++
 3 files changed, 27 insertions(+), 1 deletion(-)
include/net/ip_vs.h +11 -0
index 47677f0..1096a71 100644
@@ -998,6 +998,7 @@ struct netns_ipvs {
 	int			sysctl_backup_only;
 	int			sysctl_conn_reuse_mode;
 	int			sysctl_schedule_icmp;
+	int			sysctl_ignore_tunneled;
 
 	/* ip_vs_lblc */
 	int			sysctl_lblc_expiration;
@@ -1121,6 +1122,11 @@ static inline int sysctl_schedule_icmp(struct netns_ipvs *ipvs)
 	return ipvs->sysctl_schedule_icmp;
 }
 
+static inline int sysctl_ignore_tunneled(struct netns_ipvs *ipvs)
+{
+	return ipvs->sysctl_ignore_tunneled;
+}
+
 #else
 
 static inline int sysctl_sync_threshold(struct netns_ipvs *ipvs)
@@ -1198,6 +1204,11 @@ static inline int sysctl_schedule_icmp(struct netns_ipvs *ipvs)
 	return 0;
 }
 
+static inline int sysctl_ignore_tunneled(struct netns_ipvs *ipvs)
+{
+	return 0;
+}
+
 #endif
 
 /* IPVS core functions
net/netfilter/ipvs/ip_vs_core.c +9 -1
index 99be680..984cf49 100644
@@ -1760,8 +1760,16 @@ ip_vs_in(unsigned int hooknum, struct sk_buff *skb, int af)
 
 	/* Protocol supported? */
 	pd = ip_vs_proto_data_get(net, iph.protocol);
-	if (unlikely(!pd))
+	if (unlikely(!pd)) {
+		/* The only way we'll see this packet again is if it's
+		 * encapsulated, so mark it with ipvs_property=1 so we
+		 * skip it if we're ignoring tunneled packets
+		 */
+		if (sysctl_ignore_tunneled(net_ipvs(net)))
+			skb->ipvs_property = 1;
+
 		return NF_ACCEPT;
+	}
 	pp = pd->pp;
 	/*
 	 * Check if the packet belongs to an existing connection entry
net/netfilter/ipvs/ip_vs_ctl.c +7 -0
index 31d80e2..7338827 100644
@@ -1850,6 +1850,12 @@ static struct ctl_table vs_vars[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec,
 	},
+	{
+		.procname	= "ignore_tunneled",
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
 #ifdef CONFIG_IP_VS_DEBUG
 	{
 		.procname	= "debug_level",
@@ -3902,6 +3908,7 @@ static int __net_init ip_vs_control_net_init_sysctl(struct net *net)
 	ipvs->sysctl_conn_reuse_mode = 1;
 	tbl[idx++].data = &ipvs->sysctl_conn_reuse_mode;
 	tbl[idx++].data = &ipvs->sysctl_schedule_icmp;
+	tbl[idx++].data = &ipvs->sysctl_ignore_tunneled;
 
 	ipvs->sysctl_hdr = register_net_sysctl(net, "net/ipv4/vs", tbl);
 	if (ipvs->sysctl_hdr == NULL) {
Signature · 1 line
Alex Gartrell <[email protected]>
maintainer
#1 lore raw

Hello,

Quoted 2 lines · Alex Gartrell

This is a way to avoid nasty routing loops when multiple ipvs instances can forward to eachother.

More details? Do we set the flag for IPIP packets
and what happens otherwise?

Regards

--
Julian Anastasov <[email protected]>

#2 lore raw
Quoted 2 lines · Julian Anastasov

More details? Do we set the flag for IPIP packets
and what happens otherwise?

This is for the situation where you are running ipvs on a server that is also one of the real servers. What will happen is that you get an IPIP encapsulated packet sent to you from the other ipvs server, decapsulate it, and it is caught by ipvs and then forwarded again.

This is always extra work, but in the worst case, you can end up forwarding it back to the other ipvs server. For us, this has caused a lot of ping-ponging which burns up a CPU core.

Signature · 1 line
Alex Gartrell <[email protected]>
maintainer
#3 lore raw
In reply to Alex Gartrell

Hello,

Quoted 5 lines · Alex Gartrell
net/netfilter/ipvs/ip_vs_core.c +0 -0
index 99be680..984cf49 100644

net/netfilter/ipvs/ip_vs_core.c ip_vs_in(unsigned int hooknum, struct sk_buff *skb, int af) · line 1768

Quoted 10 lines
net/netfilter/ipvs/ip_vs_core.c +6 -1
@@ -1760,8 +1760,16 @@ ip_vs_in(unsigned int hooknum, struct sk_buff *skb, int af)
 
 	/* Protocol supported? */
 	pd = ip_vs_proto_data_get(net, iph.protocol);
-	if (unlikely(!pd))
+	if (unlikely(!pd)) {
+		/* The only way we'll see this packet again is if it's
+		 * encapsulated, so mark it with ipvs_property=1 so we
+		 * skip it if we're ignoring tunneled packets
+		 */
+		if (sysctl_ignore_tunneled(net_ipvs(net)))

We can use "ipvs" here. I remember people used
matching by src MAC to solve such problem for DR. For TUN fwmark or match by input device can work too. In all cases, a fwmark-based service is needed...

net/netfilter/ipvs/ip_vs_core.c ip_vs_in(unsigned int hooknum, struct sk_buff *skb, int af) · line 1772

Quoted 4 lines
net/netfilter/ipvs/ip_vs_core.c +3 -0
+			skb->ipvs_property = 1;
+
 		return NF_ACCEPT;
+	}

Regards

--
Julian Anastasov <[email protected]>

#4 lore raw
Quoted 4 lines · Julian Anastasov

We can use "ipvs" here. I remember people used
matching by src MAC to solve such problem for DR. For TUN fwmark or match by input device can work too. In all cases, a fwmark-based service is needed...

Yeha, to be honest, this approach isn't my ideal. We've had a much nastier version of this patch (that adds a field to skbuff...) for a long time, and this was just a less awful way of doing this.

The problem for us is that moving the whole of our load balancing to fwmark-based pools is a giant nightmare. On top of the obvious stuff (redeploying the userspace element to our load balancers), we'd also need to find a way to prevent conflict between that and our firewalls. It was more engineering than I had time for, sadly.

Other ideas I had to address this:
* Add some mechanism wherein certain fwmark's are ignored
* Add an iptables target that sets ipvs_property=1

I'm also totally open to ideas

cheers,

Signature · 1 line
Alex Gartrell <[email protected]>
maintainer
#5 lore raw

Hello,

Quoted 16 lines · Alex Gartrell
Quoted 4 lines · Julian Anastasov

We can use "ipvs" here. I remember people used
matching by src MAC to solve such problem for DR. For TUN fwmark or match by input device can work too. In all cases, a fwmark-based service is needed...

Yeha, to be honest, this approach isn't my ideal. We've had a much nastier version of this patch (that adds a field to skbuff...) for a long time, and this was just a less awful way of doing this.

The problem for us is that moving the whole of our load balancing to fwmark-based pools is a giant nightmare. On top of the obvious stuff (redeploying the userspace element to our load balancers), we'd also need to find a way to prevent conflict between that and our firewalls. It was more engineering than I had time for, sadly.

Quoted 5 lines

Other ideas I had to address this:
* Add some mechanism wherein certain fwmark's are ignored
* Add an iptables target that sets ipvs_property=1

I'm also totally open to ideas

For now I do not see fatal problems with
this patch except:

- XT_IPVS_IPVS_PROPERTY in xt_ipvs.c can return wrong match, i.e. someone may think the IPIP traffic from sockets is generated by IPVS. There can be a warning in ipvs-sysctl.txt

- In the future, if one implements IPIP protocol for IPVS, the feature will not work

But as the feature is controlled with flag the patch
looks acceptable to me. Just send v2 with changing "net_ipvs(net)" to "ipvs".

Also, think for adding info for all flags you add
in Documentation/networking/ipvs-sysctl.txt, for this patch and for your previous work.

Regards

--
Julian Anastasov <[email protected]>

#6 lore raw
Quoted 3 lines · Julian Anastasov

Also, think for adding info for all flags you add
in Documentation/networking/ipvs-sysctl.txt, for this patch and for your previous work.

I'll include the change for ignore_tunneled in the sysctls doc and send additional patches for the other sysctls.

Signature · 1 line
Alex Gartrell <[email protected]>