Skip to contentSkip to conversation

netfilter: xtables: add cluster match

PATCH In review Pablo Neira Ayuso 2 participants 9 messages Feb 23, 2009 – Mar 16, 2009

Subsystems (2)
lore
Share & tools
lore

Pablo Neira Ayuso

author maintainer
#0 lore raw

PATCH 1/1 netfilter: xtables: add cluster match

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters. The cluster can be composed of 32 nodes maximum (although I have only tested this with two nodes, so I cannot tell what is the real scalability limit of this solution in terms of cluster nodes).

Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:

(jhash(source IP) % total_nodes) & node_mask

For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):

iptables -I PREROUTING -t mangle -i eth1 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth1 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth1 \
	-m mark ! --mark 0xffff -j DROP
iptables -A PREROUTING -t mangle -i eth2 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth2 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth2 \
	-m mark ! --mark 0xffff -j DROP

And the following commands to make all nodes see the same packets:

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
	--destination-mac 01:00:5e:00:01:01 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
arptables -I OUTPUT -o eth2 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:02
arptables -I INPUT -i eth2 --h-length 6 \
	--destination-mac 01:00:5e:00:01:02 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27

In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

BTW, some final notes:

* This match mangles the skbuff pkt_type in case that it detects
PACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.

---

 include/linux/netfilter/xt_cluster.h |   15 +++
 net/netfilter/Kconfig                |   16 +++
 net/netfilter/Makefile               |    1 
 net/netfilter/xt_cluster.c           |  164 ++++++++++++++++++++++++++++++++++
 4 files changed, 196 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/netfilter/xt_cluster.h
 create mode 100644 net/netfilter/xt_cluster.c
include/linux/netfilter/xt_cluster.h new file +15 -0
new file mode 100644
index 0000000..5e0a0d0
@@ -0,0 +1,15 @@
+#ifndef _XT_CLUSTER_MATCH_H
+#define _XT_CLUSTER_MATCH_H
+
+enum xt_cluster_flags {
+	XT_CLUSTER_F_INV	= (1 << 0)
+};
+
+struct xt_cluster_match_info {
+	u_int32_t		total_nodes;
+	u_int32_t		node_mask;
+	u_int32_t		hash_seed;
+	u_int32_t		flags;
+};
+
+#endif /* _XT_CLUSTER_MATCH_H */
net/netfilter/Kconfig +16 -0
index c2bac9c..77b6405 100644
@@ -488,6 +488,22 @@ config NETFILTER_XT_TARGET_TCPOPTSTRIP
 	  This option adds a "TCPOPTSTRIP" target, which allows you to strip
 	  TCP options from TCP packets.
 
+config NETFILTER_XT_MATCH_CLUSTER
+	tristate '"cluster" match support'
+	depends on NF_CONNTRACK
+	depends on NETFILTER_ADVANCED
+	---help---
+	  This option allows you to build work-load-sharing clusters of
+	  network servers/stateful firewalls without having a dedicated
+	  load-balancing router/server/switch. Basically, this match returns
+	  true when the packet must be handled by this cluster node. Thus,
+	  all nodes see all packets and this match decides which node handles
+	  what packets. The work-load sharing algorithm is based on source
+	  address hashing.
+
+	  If you say Y or M here, try `iptables -m cluster --help` for
+	  more information.
+
 config NETFILTER_XT_MATCH_COMMENT
 	tristate  '"comment" match support'
 	depends on NETFILTER_ADVANCED
net/netfilter/Makefile +1 -0
index da3d909..960399a 100644
@@ -57,6 +57,7 @@ obj-$(CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP) += xt_TCPOPTSTRIP.o
 obj-$(CONFIG_NETFILTER_XT_TARGET_TRACE) += xt_TRACE.o
 
 # matches
+obj-$(CONFIG_NETFILTER_XT_MATCH_CLUSTER) += xt_cluster.o
 obj-$(CONFIG_NETFILTER_XT_MATCH_COMMENT) += xt_comment.o
 obj-$(CONFIG_NETFILTER_XT_MATCH_CONNBYTES) += xt_connbytes.o
 obj-$(CONFIG_NETFILTER_XT_MATCH_CONNLIMIT) += xt_connlimit.o
net/netfilter/xt_cluster.c new file +164 -0

164 lines not on this page · raw message

#1 lore raw

include/linux/netfilter/xt_cluster.h line 13

Quoted 10 lines · Pablo Neira Ayuso
include/linux/netfilter/xt_cluster.h new file +10 -0
+enum xt_cluster_flags {
+	XT_CLUSTER_F_INV	= (1 << 0)
+};
+
+struct xt_cluster_match_info {
+	u_int32_t		total_nodes;
+	u_int32_t		node_mask;
+	u_int32_t		hash_seed;
+	u_int32_t		flags;
+};

This doesn't seem like such a hot idea. I haven't seen the new userspace patch, but assuming you're interested in the flags and not ignoring them in userspace, the user has to specify the hash seed for rule deletions.

You also have to chose the same seed for all nodes in a cluster. This seems needlessly complicated, I'd suggest to simply use zero.

author maintainer
#2 lore raw
Quoted 16 lines · Patrick McHardy

include/linux/netfilter/xt_cluster.h line 13

Quoted 10 lines · Pablo Neira Ayuso
include/linux/netfilter/xt_cluster.h new file +10 -0
+enum xt_cluster_flags {
+    XT_CLUSTER_F_INV    = (1 << 0)
+};
+
+struct xt_cluster_match_info {
+    u_int32_t        total_nodes;
+    u_int32_t        node_mask;
+    u_int32_t        hash_seed;
+    u_int32_t        flags;
+};

This doesn't seem like such a hot idea. I haven't seen the new userspace patch, but assuming you're interested in the flags and not ignoring them in userspace, the user has to specify the hash seed for rule deletions.

The user has to specify the hash seed to delete the rule if it's non-zero, otherwise it must be specified. The hash seed is optional. I don't quite see the problem.

Quoted 2 lines

You also have to chose the same seed for all nodes in a cluster. This seems needlessly complicated, I'd suggest to simply use zero.

One may want to forge traffic to flood a single node? The hash seed avoids this.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#3 lore raw
Quoted 18 lines · Pablo Neira Ayuso
Quoted 16 lines · Patrick McHardy

include/linux/netfilter/xt_cluster.h line 13

Quoted 10 lines · Pablo Neira Ayuso
include/linux/netfilter/xt_cluster.h new file +10 -0
+enum xt_cluster_flags {
+    XT_CLUSTER_F_INV    = (1 << 0)
+};
+
+struct xt_cluster_match_info {
+    u_int32_t        total_nodes;
+    u_int32_t        node_mask;
+    u_int32_t        hash_seed;
+    u_int32_t        flags;
+};

This doesn't seem like such a hot idea. I haven't seen the new userspace patch, but assuming you're interested in the flags and not ignoring them in userspace, the user has to specify the hash seed for rule deletions.

Quoted 1 line

The user has to specify the hash seed to delete the rule if it's non-zero, otherwise it must be specified. The hash seed is optional. I don't quite see the problem.

Its a parameter without a meaning, the user is needlessly bothered with this.

Quoted 4 lines
Quoted 2 lines

You also have to chose the same seed for all nodes in a cluster. This seems needlessly complicated, I'd suggest to simply use zero.

One may want to forge traffic to flood a single node? The hash seed avoids this.

No, it only makes it easier to shut off since I have to use the same source address to be sure I hit the same node. This seems like a valid argument though.

The fact that you have to specify it for deletion still seems unnecesary though. You would never have two rules differing only in the seed value since that would mean the node is part of two clusters. So we might as well move it to the end and ignore it in userspace. What do you think? In case you agree, I also think "secret" would be a more fitting name.

author maintainer
#4 lore raw
Quoted 19 lines · Patrick McHardy
Quoted 18 lines · Pablo Neira Ayuso
Quoted 16 lines · Patrick McHardy

include/linux/netfilter/xt_cluster.h line 13

Quoted 10 lines · Pablo Neira Ayuso
include/linux/netfilter/xt_cluster.h new file +10 -0
+enum xt_cluster_flags {
+    XT_CLUSTER_F_INV    = (1 << 0)
+};
+
+struct xt_cluster_match_info {
+    u_int32_t        total_nodes;
+    u_int32_t        node_mask;
+    u_int32_t        hash_seed;
+    u_int32_t        flags;
+};

This doesn't seem like such a hot idea. I haven't seen the new userspace patch, but assuming you're interested in the flags and not ignoring them in userspace, the user has to specify the hash seed for rule deletions.

Quoted 8 lines
Quoted 3 lines

The user has to specify the hash seed to delete the rule if it's non-zero, otherwise it must be specified. The hash seed is optional. I don't quite see the problem.

Its a parameter without a meaning, the user is needlessly bothered with this.

From the user view, yes. No matter what value you set as long as it is

the same in all the cluster nodes.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

You also have to chose the same seed for all nodes in a cluster. This seems needlessly complicated, I'd suggest to simply use zero.

One may want to forge traffic to flood a single node? The hash seed avoids this.

Quoted 8 lines

No, it only makes it easier to shut off since I have to use the same source address to be sure I hit the same node. This seems like a valid argument though.

The fact that you have to specify it for deletion still seems unnecesary though. You would never have two rules differing only in the seed value since that would mean the node is part of two clusters. So we might as well move it to the end and ignore it in userspace. What do you think?

But the value has to be the same in all the cluster nodes, so how can it be set to ensure that it is the same value?

Quoted 1 line

In case you agree, I also think "secret" would be a more fitting name.

I can rename the field to "secret" in the structure or change the iptables cluster match option to be "--cluster-secret" instead of "--cluster-hash-seed" if you like.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#5 lore raw
Quoted 8 lines · Pablo Neira Ayuso
Quoted 4 lines · Patrick McHardy

The fact that you have to specify it for deletion still seems unnecesary though. You would never have two rules differing only in the seed value since that would mean the node is part of two clusters. So we might as well move it to the end and ignore it in userspace. What do you think?

But the value has to be the same in all the cluster nodes, so how can it be set to ensure that it is the same value?

I only meant ignoring it on comparisons of course, just as we do with all the private pointer stuff. Anyways, its not that important and it in fact would be slightly different behaviour from what we do in other cases, where we only ignore state. So perhaps not a good idea after all.

Quoted 5 lines
Quoted 1 line

In case you agree, I also think "secret" would be a more fitting name.

I can rename the field to "secret" in the structure or change the iptables cluster match option to be "--cluster-secret" instead of "--cluster-hash-seed" if you like.

Its more fitting in my opinion, but I don't really care.

author maintainer
#6 lore raw
Quoted 8 lines · Patrick McHardy
Quoted 5 lines · Pablo Neira Ayuso
Quoted 1 line

In case you agree, I also think "secret" would be a more fitting name.

I can rename the field to "secret" in the structure or change the iptables cluster match option to be "--cluster-secret" instead of "--cluster-hash-seed" if you like.

Its more fitting in my opinion, but I don't really care.

I don't either, would you apply the patch as is now?

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#7 lore raw
Quoted 6 lines · Pablo Neira Ayuso
Quoted 4 lines · Patrick McHardy
Quoted 2 lines · Pablo Neira Ayuso
Quoted 2 lines

In case you agree, I also think "secret" would be a more fitting name.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

I can rename the field to "secret" in the structure or change the iptables cluster match option to be "--cluster-secret" instead of "--cluster-hash-seed" if you like.

Its more fitting in my opinion, but I don't really care.

I don't either, would you apply the patch as is now?

I will :)

#8 lore raw
In reply to Pablo Neira Ayuso
Quoted 58 lines · Pablo Neira Ayuso

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters. The cluster can be composed of 32 nodes maximum (although I have only tested this with two nodes, so I cannot tell what is the real scalability limit of this solution in terms of cluster nodes).

Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:

(jhash(source IP) % total_nodes) & node_mask

For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):

iptables -I PREROUTING -t mangle -i eth1 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth1 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth1 \
	-m mark ! --mark 0xffff -j DROP
iptables -A PREROUTING -t mangle -i eth2 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth2 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth2 \
	-m mark ! --mark 0xffff -j DROP

And the following commands to make all nodes see the same packets:

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
	--destination-mac 01:00:5e:00:01:01 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
arptables -I OUTPUT -o eth2 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:02
arptables -I INPUT -i eth2 --h-length 6 \
	--destination-mac 01:00:5e:00:01:02 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27

In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

BTW, some final notes:

* This match mangles the skbuff pkt_type in case that it detects
PACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.

---

Quoted 7 lines
 include/linux/netfilter/xt_cluster.h |   15 +++
 net/netfilter/Kconfig                |   16 +++
 net/netfilter/Makefile               |    1 
 net/netfilter/xt_cluster.c           |  164 ++++++++++++++++++++++++++++++++++
 4 files changed, 196 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/netfilter/xt_cluster.h
 create mode 100644 net/netfilter/xt_cluster.c

Applied, thanks. I've also added xt_cluster.h to the Kbuild file so the header will be installed.