netfilter: xtables: add cluster match
Subsystems (2)
Pablo Neira Ayuso
maintainerPATCH 1/1 netfilter: xtables: add cluster match
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters. The cluster can be composed of 32 nodes maximum (although I have only tested this with two nodes, so I cannot tell what is the real scalability limit of this solution in terms of cluster nodes).
Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:
(jhash(source IP) % total_nodes) & node_mask
For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):
iptables -I PREROUTING -t mangle -i eth1 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth1 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth1 \ -m mark ! --mark 0xffff -j DROP iptables -A PREROUTING -t mangle -i eth2 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth2 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth2 \ -m mark ! --mark 0xffff -j DROP
And the following commands to make all nodes see the same packets:
ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2
arptables -I OUTPUT -o eth1 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:01 arptables -I INPUT -i eth1 --h-length 6 \ --destination-mac 01:00:5e:00:01:01 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27 arptables -I OUTPUT -o eth2 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:02 arptables -I INPUT -i eth2 --h-length 6 \ --destination-mac 01:00:5e:00:01:02 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.
echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose
BTW, some final notes:
* This match mangles the skbuff pkt_type in case that it detectsPACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.
---
include/linux/netfilter/xt_cluster.h | 15 +++ net/netfilter/Kconfig | 16 +++ net/netfilter/Makefile | 1 net/netfilter/xt_cluster.c | 165 ++++++++++++++++++++++++++++++++++ 4 files changed, 197 insertions(+), 0 deletions(-) create mode 100644 include/linux/netfilter/xt_cluster.h create mode 100644 net/netfilter/xt_cluster.c
include/linux/netfilter/xt_cluster.h new file +15 -0
@@ -0,0 +1,15 @@ +#ifndef _XT_CLUSTER_MATCH_H +#define _XT_CLUSTER_MATCH_H + +enum xt_cluster_flags { +XT_CLUSTER_F_INV = (1 <<0 ) +}; + +struct xt_cluster_match_info { +u_int32_t total_nodes; +u_int32_t node_mask; +u_int32_t hash_seed; +u_int32_t flags; +}; + +#endif /* _XT_CLUSTER_MATCH_H */
net/netfilter/Kconfig +16 -0
@@ -488,6 +488,22 @@ config NETFILTER_XT_TARGET_TCPOPTSTRIP This option adds a "TCPOPTSTRIP" target, which allows you to strip TCP options from TCP packets. +config NETFILTER_XT_MATCH_CLUSTER + tristate '"cluster" match support' + depends on NF_CONNTRACK + depends on NETFILTER_ADVANCED + ---help--- + This option allows you to build work-load-sharing clusters of + network servers/stateful firewalls without having a dedicated + load-balancing router/server/switch. Basically, this match returns + true when the packet must be handled by this cluster node. Thus, + all nodes see all packets and this match decides which node handles + what packets. The work-load sharing algorithm is based on source + address hashing. + + If you say Y or M here, try `iptables -m cluster --help` for + more information. + config NETFILTER_XT_MATCH_COMMENT tristate '"comment" match support' depends on NETFILTER_ADVANCED
net/netfilter/Makefile +1 -0
@@ -57,6 +57,7 @@ obj-$(CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP) += xt_TCPOPTSTRIP.o obj-$(CONFIG_NETFILTER_XT_TARGET_TRACE) += xt_TRACE.o # matches +obj-$(CONFIG_NETFILTER_XT_MATCH_CLUSTER) += xt_cluster.o obj-$(CONFIG_NETFILTER_XT_MATCH_COMMENT) += xt_comment.o obj-$(CONFIG_NETFILTER_XT_MATCH_CONNBYTES) += xt_connbytes.o obj-$(CONFIG_NETFILTER_XT_MATCH_CONNLIMIT) += xt_connlimit.o
net/netfilter/xt_cluster.c new file +165 -0
165 lines not on this page · raw message
Quoted 5 lines · Pablo Neira Ayuso
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters. The cluster can be composed of 32 nodes maximum (although I have only tested this with two nodes, so I cannot tell what is the real scalability limit of this solution in terms of cluster nodes).
Thanks Pablo.
net/netfilter/xt_cluster.c line 122
Quoted 11 lines
net/netfilter/xt_cluster.c new file +11
-0+ ct =nf_ct_get (skb, &ctinfo); +if (ct ==NULL ) +return false; + +if (ct == &nf_conntrack_untracked) +return false; + +if (ct->master) + hash =xt_cluster_hash (ct->master, info); +else + hash =xt_cluster_hash (ct, info);
This makes a lot of sense for helpers like SIP, where the expectation can arrive from a different source address. I'm just wondering how this works when not using reliable synchronization - in that case, other nodes might not be aware of the expectation and also accept the packet. I don't have a suggestion besides making sure expectations are synchronized, just thought I'd point it out.
net/netfilter/xt_cluster.c line 134
Quoted 7 lines
net/netfilter/xt_cluster.c new file +7
-0+static bool xt_cluster_mt_checkentry (const struct xt_mtchk_param *par) +{ +struct xt_cluster_match_info *info = par->matchinfo; + +if (info->node_mask > (1 << info->total_nodes)) { +printk (KERN_ERR "xt_cluster: the id of this node cannot be " +"higher than the total number of nodes\n" );
This looks like an off-by-one (warning: still at first coffee :)). It may also not be equal to the mask I'd expect. I can change it to >= when applying if you agree.
Pablo Neira Ayuso
maintainerHi Patrick,
Quoted 11 lines · Patrick McHardy
Quoted 10 lines · Pablo Neira Ayuso
net/netfilter/xt_cluster.c new file +10
-0+ ct =nf_ct_get (skb, &ctinfo); +if (ct ==NULL ) +return false; + +if (ct == &nf_conntrack_untracked) +return false; + +if (ct->master) + hash =xt_cluster_hash (ct->master, info); +else
Quoted 8 lines
Quoted 1 line
+ hash =
xt_cluster_hash(ct, info);This makes a lot of sense for helpers like SIP, where the expectation can arrive from a different source address. I'm just wondering how this works when not using reliable synchronization - in that case, other nodes might not be aware of the expectation and also accept the packet. I don't have a suggestion besides making sure expectations are synchronized, just thought I'd point it out.
Indeed.
This sort of problem is interesting, just in case that you have some spare time to think about other synchronization-related problems (otherwise you can skip the following below :)). Conntrackd does not synchronize expectations (at least, it's not in my plans yet), it synchronizes conntrack entries, and that includes the relationship between master and related conntracks. Thus, after the failover, the new primary node knows that the master connection has a helper (so it can create new expectations) and already existing established-related connections are linked to the master conntracks.
Still I see two possible problematic situations with these approach:
* If expectations are not propagated, this means than a FTP-data
connections that is about to start would not success if that connection happens during a failover as the expectation information is lost.
* If the state information is lost for whatever reason (like not using
conntrackd at all or losing the state information due to netlink unreliability), then the former expected connection would be handled like a normal connection by one cluster node. For example, this would break if destination nat is used in the case of FTP (and similarly for other helpers I think).
For the first problem, I can say that conntrackd can be tuned to reduce the chances of this to happen (at the cost of investing more resources in the synchronization). Moreover, connections that are about to start may retry in short and no data was exchanged indeed.
For the second problem, this is actually the sort of problems that I want to avoid making netlink reliable by dropping packets. By reducing the chances to lose state information for whatever reason.
Quoted 11 lines
net/netfilter/xt_cluster.c line 134
Quoted 7 lines
net/netfilter/xt_cluster.c new file +7
-0+static bool xt_cluster_mt_checkentry (const struct xt_mtchk_param *par) +{ +struct xt_cluster_match_info *info = par->matchinfo; + +if (info->node_mask > (1 << info->total_nodes)) { +printk (KERN_ERR "xt_cluster: the id of this node cannot be " +"higher than the total number of nodes\n" );This looks like an off-by-one (warning: still at first coffee :)). It may also not be equal to the mask I'd expect. I can change it to >= when applying if you agree.
You're right! Please change it.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
Quoted 20 lines · Pablo Neira Ayuso
Quoted 6 lines · Patrick McHardy
net/netfilter/xt_cluster.c line 122
Quoted 4 lines · Pablo Neira Ayuso
net/netfilter/xt_cluster.c new file +4
-0+if (ct->master) + hash =xt_cluster_hash (ct->master, info); +else + hash =xt_cluster_hash (ct, info);Quoted 6 lines
This makes a lot of sense for helpers like SIP, where the expectation can arrive from a different source address. I'm just wondering how this works when not using reliable synchronization - in that case, other nodes might not be aware of the expectation and also accept the packet. I don't have a suggestion besides making sure expectations are synchronized, just thought I'd point it out.
Indeed.
This sort of problem is interesting, just in case that you have some spare time to think about other synchronization-related problems (otherwise you can skip the following below :)). Conntrackd does not synchronize expectations (at least, it's not in my plans yet), it synchronizes conntrack entries, and that includes the relationship between master and related conntracks. Thus, after the failover, the new primary node knows that the master connection has a helper (so it can create new expectations) and already existing established-related connections are linked to the master conntracks.
Still I see two possible problematic situations with these approach:
Quoted 7 lines
* If expectations are not propagated, this means than a FTP-data
connections that is about to start would not success if that connection happens during a failover as the expectation information is lost.* If the state information is lost for whatever reason (like not using
conntrackd at all or losing the state information due to netlink unreliability), then the former expected connection would be handled like a normal connection by one cluster node. For example, this would break if destination nat is used in the case of FTP (and similarly for other helpers I think).For the first problem, I can say that conntrackd can be tuned to reduce the chances of this to happen (at the cost of investing more resources in the synchronization). Moreover, connections that are about to start may retry in short and no data was exchanged indeed.
Good point.
Quoted 1 line
For the second problem, this is actually the sort of problems that I want to avoid making netlink reliable by dropping packets. By reducing the chances to lose state information for whatever reason.
Yes, although the netlink delivery only covers part of it. It might be the path where most events are lost though.
Quoted 12 lines
Quoted 11 lines
net/netfilter/xt_cluster.c line 134
Quoted 7 lines
net/netfilter/xt_cluster.c new file +7
-0+static bool xt_cluster_mt_checkentry (const struct xt_mtchk_param *par) +{ +struct xt_cluster_match_info *info = par->matchinfo; + +if (info->node_mask > (1 << info->total_nodes)) { +printk (KERN_ERR "xt_cluster: the id of this node cannot be " +"higher than the total number of nodes\n" );This looks like an off-by-one (warning: still at first coffee :)). It may also not be equal to the mask I'd expect. I can change it to >= when applying if you agree.
Quoted 1 line
You're right! Please change it.
I noticed another problem during compilation:
net/netfilter/xt_cluster.c: In function 'xt_cluster_mt':net/netfilter/xt_cluster.c:124: warning: passing argument 2 of 'constant_test_bit' from incompatible pointer type net/netfilter/xt_cluster.c:124: warning: passing argument 2 of 'variable_test_bit' from incompatible pointer type
The problem is that is uses a u32 for the mask, but the bitops are only defined for unsigned longs. Which is a bit unfortunate since they're not well suited for ABI structures. I'd suggest to simply open-code the bit tests.
Pablo Neira Ayuso
maintainerQuoted 6 lines · Patrick McHardy
Quoted 3 lines
For the second problem, this is actually the sort of problems that I want to avoid making netlink reliable by dropping packets. By reducing the chances to lose state information for whatever reason.
Yes, although the netlink delivery only covers part of it. It might be the path where most events are lost though.
Right, to extend this comment, during the failure there can also be some events pending to be sent in all the existing kernel queues (netlink queues and device transmission queues) that the daemon did not have time to propagate during the failure, this is part of the asynchronous approach. I've been trying to measure this behaviour but I didn't see any significant amount of lost connections in my testbed, not yet at least.
Quoted 22 lines
Quoted 12 lines
Quoted 11 lines
net/netfilter/xt_cluster.c line 134
Quoted 7 lines
net/netfilter/xt_cluster.c new file +7
-0+static bool xt_cluster_mt_checkentry (const struct xt_mtchk_param *par) +{ +struct xt_cluster_match_info *info = par->matchinfo; + +if (info->node_mask > (1 << info->total_nodes)) { +printk (KERN_ERR "xt_cluster: the id of this node cannot be " +"higher than the total number of nodes\n" );This looks like an off-by-one (warning: still at first coffee :)). It may also not be equal to the mask I'd expect. I can change it to >= when applying if you agree.
Quoted 1 line
You're right! Please change it.
I noticed another problem during compilation:
net/netfilter/xt_cluster.c: In function 'xt_cluster_mt':net/netfilter/xt_cluster.c:124: warning: passing argument 2 of 'constant_test_bit' from incompatible pointer typenet/netfilter/xt_cluster.c:124: warning: passing argument 2 of 'variable_test_bit' from incompatible pointer type
Quoted 4 lines
The problem is that is uses a u32 for the mask, but the bitops are only defined for unsigned longs. Which is a bit unfortunate since they're not well suited for ABI structures. I'd suggest to simply open-code the bit tests.
Agreed. The test_bit was a reminiscent of the /proc interface (which allowed node mask bit setting), so I don't need it. I'm going to resend you the patch in a couple of minutes (including the off-by-one issue resolved).
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers