Skip to contentSkip to conversation

netfilter: xtables: add cluster match

PATCH In review Pablo Neira Ayuso 4 participants 28 messages Feb 14, 2009 – Feb 18, 2009

Subsystems (2)
lore
Share & tools
lore

Pablo Neira Ayuso

author maintainer
#0 lore raw

PATCH 1/1 netfilter: xtables: add cluster match

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.

Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:

	jhash(source IP) % total_nodes == node_id

For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):

iptables -I PREROUTING -t mangle -i eth1 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth1 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth1 \
	-m mark ! --mark 0xffff -j DROP
iptables -A PREROUTING -t mangle -i eth2 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth2 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth2 \
	-m mark ! --mark 0xffff -j DROP

And the following commands to make all nodes see the same packets:

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
	--destination-mac 01:00:5e:00:01:01 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
arptables -I OUTPUT -o eth2 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:02
arptables -I INPUT -i eth2 --h-length 6 \
	--destination-mac 01:00:5e:00:01:02 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27

In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

The match also provides a /proc entry under:

/proc/sys/net/netfilter/cluster/$PROC_NAME

where PROC_NAME is set via --cluster-proc-name. This is useful to include possible cluster reconfigurations via fail-over scripts. Assuming that this is the node 1, if node 2 is down, you can add node 2 to your node-mask as follows:

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

BTW, some final notes:

* This match mangles the skbuff pkt_type in case that it detects
PACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.

---

 include/linux/netfilter/xt_cluster.h |   21 ++
 net/netfilter/Kconfig                |   15 +
 net/netfilter/Makefile               |    1 
 net/netfilter/xt_cluster.c           |  368 ++++++++++++++++++++++++++++++++++
 4 files changed, 405 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/netfilter/xt_cluster.h
 create mode 100644 net/netfilter/xt_cluster.c
include/linux/netfilter/xt_cluster.h new file +21 -0
new file mode 100644
index 0000000..a06401d
@@ -0,0 +1,21 @@
+#ifndef _XT_CLUSTER_MATCH_H
+#define _XT_CLUSTER_MATCH_H
+
+struct proc_dir_entry;
+
+enum xt_cluster_flags {
+	XT_CLUSTER_F_INV = 0,
+};
+
+struct xt_cluster_match_info {
+	u_int16_t		total_nodes;
+	u_int16_t		node_id;
+	u_int32_t		hash_seed;
+	char			proc_name[16];
+	u_int32_t		flags;
+
+	/* Used internally by the kernel */
+	void			*data __attribute__((aligned(8)));
+};
+
+#endif /* _XT_CLUSTER_MATCH_H */
net/netfilter/Kconfig +15 -0
index c2bac9c..52240cc 100644
@@ -488,6 +488,21 @@ config NETFILTER_XT_TARGET_TCPOPTSTRIP
 	  This option adds a "TCPOPTSTRIP" target, which allows you to strip
 	  TCP options from TCP packets.
 
+config NETFILTER_XT_MATCH_CLUSTER
+	tristate '"cluster" match support'
+	depends on NETFILTER_ADVANCED
+	---help---
+	  This option allows you to build work-load-sharing clusters of
+	  network servers/stateful firewalls without having a dedicated
+	  load-balancing router/server/switch. Basically, this match returns
+	  true when the packet must be handled by this cluster node. Thus,
+	  all nodes see all packets and this match decides which node handles
+	  what packets. The work-load sharing algorithm is based on source
+	  address hashing.
+
+	  If you say Y here, try `iptables -m cluster --help` for
+	  more information.
+
 config NETFILTER_XT_MATCH_COMMENT
 	tristate  '"comment" match support'
 	depends on NETFILTER_ADVANCED
net/netfilter/Makefile +1 -0
index da3d909..960399a 100644
@@ -57,6 +57,7 @@ obj-$(CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP) += xt_TCPOPTSTRIP.o
 obj-$(CONFIG_NETFILTER_XT_TARGET_TRACE) += xt_TRACE.o
 
 # matches
+obj-$(CONFIG_NETFILTER_XT_MATCH_CLUSTER) += xt_cluster.o
 obj-$(CONFIG_NETFILTER_XT_MATCH_COMMENT) += xt_comment.o
 obj-$(CONFIG_NETFILTER_XT_MATCH_CONNBYTES) += xt_connbytes.o
 obj-$(CONFIG_NETFILTER_XT_MATCH_CONNLIMIT) += xt_connlimit.o
net/netfilter/xt_cluster.c new file +368 -0

368 lines not on this page · raw message

#1 lore raw
Quoted 2 lines · Pablo Neira Ayuso

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.

All of this nice text (below) should go into libxt_cluster.man :)

Quoted 12 lines

Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:

	jhash(source IP) % total_nodes == node_id

For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):

iptables -I PREROUTING -t mangle -i eth1 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \

[...]

Quoted 9 lines

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

BTW, some final notes:

* This match mangles the skbuff pkt_type in case that it detects PACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.

The supersedes statement should also go into Kconfig.

include/linux/netfilter/xt_cluster.h line 8

Quoted 9 lines
include/linux/netfilter/xt_cluster.h new file +8 -0
@@ -0,0 +1,21 @@
+#ifndef _XT_CLUSTER_MATCH_H
+#define _XT_CLUSTER_MATCH_H
+
+struct proc_dir_entry;
+
+enum xt_cluster_flags {
+	XT_CLUSTER_F_INV = 0,
+};

Hm, that should be XT_CLUSTER_F_INV = 1 << 0.

net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 493

Quoted 3 lines
net/netfilter/Kconfig +3 -0
+config NETFILTER_XT_MATCH_CLUSTER
+	tristate '"cluster" match support'
+	depends on NETFILTER_ADVANCED

xt_cluster depends on NF_CONNTRACK too.

net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 504

Quoted 11 lines
net/netfilter/Kconfig +11 -0
+	---help---
+	  This option allows you to build work-load-sharing clusters of
+	  network servers/stateful firewalls without having a dedicated
+	  load-balancing router/server/switch. Basically, this match returns
+	  true when the packet must be handled by this cluster node. Thus,
+	  all nodes see all packets and this match decides which node handles
+	  what packets. The work-load sharing algorithm is based on source
+	  address hashing.
+
+	  If you say Y here, try `iptables -m cluster --help` for
+	  more information.

Somehow this gives the impression that Y is the only logical choice, when indeed, M would work too.

net/netfilter/xt_cluster.c line 21

Quoted 2 lines
net/netfilter/xt_cluster.c new file +2 -0
+struct xt_cluster_internal {
+	unsigned long		node_mask;

I think I raised concern about this, but can't remember. Should not this be of type nodemask_t instead? Or ... is this "node" perhaps not describing a NUMA node, but a cluster node? Some comments would be appreciated, along the lines of

/**
* @node_mask: cluster node mask
*/

net/netfilter/xt_cluster.c line 69

Quoted 6 lines
net/netfilter/xt_cluster.c new file +5 -0
+	struct proc_dir_entry	*proc;
+	atomic_t		use;
+};
 
+static inline bool
+xt_cluster_is_multicast_addr(const struct sk_buff *skb, int family)

Cosmetic warrants uint8_t family.

Quoted 5 lines
net/netfilter/xt_cluster.c new file +5 -0
+static bool
+xt_cluster_mt(const struct sk_buff *skb, const struct xt_match_param *par)
+{
+	struct sk_buff *pskb = (struct sk_buff *)skb;
+	const struct xt_cluster_match_info *info = par->matchinfo;

net/netfilter/xt_cluster.c line 119

Quoted 10 lines
net/netfilter/xt_cluster.c new file +10 -0
+	const struct xt_cluster_internal *internal = info->data;
+	const struct nf_conn *ct;
+	enum ip_conntrack_info ctinfo;
+	unsigned long hash;
+	bool inv = !!(info->flags & XT_CLUSTER_F_INV);
+
+	if (!xt_cluster_is_multicast_addr(skb, par->family) &&
+	    skb->pkt_type == PACKET_MULTICAST) {
+	    	pskb->pkt_type = PACKET_HOST;
+	}

{} are redundant here

net/netfilter/xt_cluster.c line 133

Quoted 6 lines
net/netfilter/xt_cluster.c new file +6 -0
+	if (ct->master)
+		hash = xt_cluster_hash(ct->master, info);
+	else
+		hash = xt_cluster_hash(ct, info);
+
+	return test_bit(hash, &internal->node_mask) ^ inv;

Since inv is just once used, I would "inline" it, aka

	return test_bit(hash, &internal->node_mask) ^
	       !!(info->flags & XT_CLUSTER_MATCH_INV);

net/netfilter/xt_cluster.c line 160

Quoted 4 lines
net/netfilter/xt_cluster.c new file +4 -0
+static int xt_cluster_seq_show(struct seq_file *s, void *v)
+{
+	unsigned long *mask = v;
+	seq_printf(s, "0x%.8lx\n", *mask);

'.' does not make sense with non-string,non-floating point numbers (though it is a stdc feature it seems). I'd say "0x%08lx", for clarity.

net/netfilter/xt_cluster.c line 201

Quoted 9 lines
net/netfilter/xt_cluster.c new file +9 -0
+	case '-':
+		new_node_id = simple_strtoul(buffer+1, NULL, 10);
+		if (!new_node_id || new_node_id > sizeof(info->node_mask)*8)
+			return -EIO;
+		printk(KERN_NOTICE "cluster: deleting node %u\n", new_node_id);
+		clear_bit(new_node_id-1, &info->node_mask);
+		break;
+	default:
+		return -EIO;

EINVAL, I'd say.

net/netfilter/xt_cluster.c line 240

Quoted 9 lines
net/netfilter/xt_cluster.c new file +9 -0
+static bool
+xt_cluster_proc_entry_exist(struct proc_dir_entry *dir, const char *name)
+{
+	struct proc_dir_entry *tmp;
+
+	for (tmp = dir->subdir; tmp; tmp = tmp->next) {
+		if (strcmp(tmp->name, name) == 0)
+			return true;
+	}

-{}

Looks good so far.

author maintainer
#2 lore raw
Quoted 6 lines · Jan Engelhardt
Quoted 2 lines · Pablo Neira Ayuso

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.

All of this nice text (below) should go into libxt_cluster.man :)

Indeed, that will go into the manpage. I have kept the iptables part until you finish with Jamal's libxtables renaming, to avoid any clashing. Anyway, I don't think that adding this information to the manpage is a reason to trim the patch description, actually I think that this patch deserves this long description and I have seen longer descriptions in the kernel changelog :)

Quoted 15 lines
Quoted 12 lines

Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:

	jhash(source IP) % total_nodes == node_id

For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):

iptables -I PREROUTING -t mangle -i eth1 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \

[...]

Quoted 2 lines

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

Quoted 8 lines
Quoted 6 lines

BTW, some final notes:

* This match mangles the skbuff pkt_type in case that it detects PACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.

The supersedes statement should also go into Kconfig.

Yes, but I was also waiting for Patrick to tell how to proceed with this. I think that we also have to add CLUSTERIP to the schedule-for-removal list.

Quoted 11 lines

include/linux/netfilter/xt_cluster.h line 8

Quoted 9 lines
include/linux/netfilter/xt_cluster.h new file +8 -0
@@ -0,0 +1,21 @@
+#ifndef _XT_CLUSTER_MATCH_H
+#define _XT_CLUSTER_MATCH_H
+
+struct proc_dir_entry;
+
+enum xt_cluster_flags {
+	XT_CLUSTER_F_INV = 0,
+};

Hm, that should be XT_CLUSTER_F_INV = 1 << 0.

Indeed.

Quoted 5 lines

net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 493

Quoted 3 lines
net/netfilter/Kconfig +3 -0
+config NETFILTER_XT_MATCH_CLUSTER
+	tristate '"cluster" match support'
+	depends on NETFILTER_ADVANCED

xt_cluster depends on NF_CONNTRACK too.

Right.

Quoted 15 lines

net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 504

Quoted 11 lines
net/netfilter/Kconfig +11 -0
+	---help---
+	  This option allows you to build work-load-sharing clusters of
+	  network servers/stateful firewalls without having a dedicated
+	  load-balancing router/server/switch. Basically, this match returns
+	  true when the packet must be handled by this cluster node. Thus,
+	  all nodes see all packets and this match decides which node handles
+	  what packets. The work-load sharing algorithm is based on source
+	  address hashing.
+
+	  If you say Y here, try `iptables -m cluster --help` for
+	  more information.

Somehow this gives the impression that Y is the only logical choice, when indeed, M would work too.

Quoted 7 lines

net/netfilter/xt_cluster.c line 21

Quoted 2 lines
net/netfilter/xt_cluster.c new file +2 -0
+struct xt_cluster_internal {
+	unsigned long		node_mask;

I think I raised concern about this, but can't remember. Should not this be of type nodemask_t instead? Or ... is this "node" perhaps not describing a NUMA node, but a cluster node? Some comments would be appreciated, along the lines of

You mentioned cpumask_t last time, but this is neither related to NUMA nor a CPU mask, so I prefer to keep using a long for this thing, moreover test and set bit operations use long.

Quoted 22 lines

/**
* @node_mask: cluster node mask
*/

net/netfilter/xt_cluster.c line 69

Quoted 6 lines
net/netfilter/xt_cluster.c new file +5 -0
+	struct proc_dir_entry	*proc;
+	atomic_t		use;
+};
 
+static inline bool
+xt_cluster_is_multicast_addr(const struct sk_buff *skb, int family)

Cosmetic warrants uint8_t family.

Quoted 9 lines
net/netfilter/xt_cluster.c new file +9 -0
+static bool
+xt_cluster_mt(const struct sk_buff *skb, const struct xt_match_param *par)
+{
+	struct sk_buff *pskb = (struct sk_buff *)skb;
+	const struct xt_cluster_match_info *info = par->matchinfo;
+	const struct xt_cluster_internal *internal = info->data;
+	const struct nf_conn *ct;
+	enum ip_conntrack_info ctinfo;
+	unsigned long hash;
Quoted 8 lines

net/netfilter/xt_cluster.c line 119

Quoted 6 lines
net/netfilter/xt_cluster.c new file +6 -0
+	bool inv = !!(info->flags & XT_CLUSTER_F_INV);
+
+	if (!xt_cluster_is_multicast_addr(skb, par->family) &&
+	    skb->pkt_type == PACKET_MULTICAST) {
+	    	pskb->pkt_type = PACKET_HOST;
+	}

{} are redundant here

When lines are splitted, I have seen these redundant {} for readability.

Quoted 10 lines

net/netfilter/xt_cluster.c line 133

Quoted 6 lines
net/netfilter/xt_cluster.c new file +6 -0
+	if (ct->master)
+		hash = xt_cluster_hash(ct->master, info);
+	else
+		hash = xt_cluster_hash(ct, info);
+
+	return test_bit(hash, &internal->node_mask) ^ inv;

Since inv is just once used, I would "inline" it, aka

	return test_bit(hash, &internal->node_mask) ^
	       !!(info->flags & XT_CLUSTER_MATCH_INV);

Makes sense.

Quoted 7 lines

net/netfilter/xt_cluster.c line 160

Quoted 4 lines
net/netfilter/xt_cluster.c new file +4 -0
+static int xt_cluster_seq_show(struct seq_file *s, void *v)
+{
+	unsigned long *mask = v;
+	seq_printf(s, "0x%.8lx\n", *mask);

'.' does not make sense with non-string,non-floating point numbers (though it is a stdc feature it seems). I'd say "0x%08lx", for clarity.

OK

Quoted 11 lines

net/netfilter/xt_cluster.c line 201

Quoted 9 lines
net/netfilter/xt_cluster.c new file +9 -0
+	case '-':
+		new_node_id = simple_strtoul(buffer+1, NULL, 10);
+		if (!new_node_id || new_node_id > sizeof(info->node_mask)*8)
+			return -EIO;
+		printk(KERN_NOTICE "cluster: deleting node %u\n", new_node_id);
+		clear_bit(new_node_id-1, &info->node_mask);
+		break;
+	default:
+		return -EIO;

EINVAL, I'd say.

Other /proc-related code uses this error, but indeed I prefer EINVAL.

Quoted 11 lines

net/netfilter/xt_cluster.c line 240

Quoted 9 lines
net/netfilter/xt_cluster.c new file +9 -0
+static bool
+xt_cluster_proc_entry_exist(struct proc_dir_entry *dir, const char *name)
+{
+	struct proc_dir_entry *tmp;
+
+	for (tmp = dir->subdir; tmp; tmp = tmp->next) {
+		if (strcmp(tmp->name, name) == 0)
+			return true;
+	}

-{}

Same comment as above, it's there for readability.

Quoted 1 line

Looks good so far.

Thanks for the review.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#3 lore raw
Quoted 11 lines · Pablo Neira Ayuso
Quoted 6 lines · Jan Engelhardt
Quoted 2 lines · Pablo Neira Ayuso

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.

All of this nice text (below) should go into libxt_cluster.man :)

Indeed, that will go into the manpage. I have kept the iptables part until you finish with Jamal's libxtables renaming, to avoid any clashing.

We are done and ready for release as I understood.

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines

* This match mangles the skbuff pkt_type in case that it detects PACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.

The supersedes statement should also go into Kconfig.

Yes, but I was also waiting for Patrick to tell how to proceed with this. I think that we also have to add CLUSTERIP to the schedule-for-removal list.

We do not have to (there is still xt_state and no mention of deprecation despite xt_conntrack superseding it) - but it would be a most logical step indeed.

Quoted 9 lines
Quoted 2 lines

net/netfilter/xt_cluster.c line 21

Quoted 1 line
net/netfilter/xt_cluster.c new file +1 -0
+	unsigned long		node_mask;

You mentioned cpumask_t last time, but this is neither related to NUMA nor a CPU mask, so I prefer to keep using a long for this thing, moreover test and set bit operations use long.

Quoted 3 lines

/**
* @node_mask: cluster node mask
*/

Hence the need for a comment so that this misunderstanding is resolved early.

#4 lore raw
Quoted 7 lines · Jan Engelhardt
Quoted 6 lines · Pablo Neira Ayuso
Quoted 5 lines · Jan Engelhardt
Quoted 2 lines · Pablo Neira Ayuso

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

All of this nice text (below) should go into libxt_cluster.man :)

Indeed, that will go into the manpage. I have kept the iptables part until you finish with Jamal's libxtables renaming, to avoid any clashing.

We are done and ready for release as I understood.

That is, after grabbing the latest contents of our tree.

#5 lore raw
In reply to Pablo Neira Ayuso
Quoted 2 lines · Pablo Neira Ayuso

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.

I'm mixing comments to the cluster match and the ARP mangle target.

Quoted 23 lines

Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:

	jhash(source IP) % total_nodes == node_id

For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):

iptables -I PREROUTING -t mangle -i eth1 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth1 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth1 \
	-m mark ! --mark 0xffff -j DROP
iptables -A PREROUTING -t mangle -i eth2 -m cluster \
	--cluster-total-nodes 2 --cluster-local-node 1 \
	--cluster-proc-name eth2 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth2 \
	-m mark ! --mark 0xffff -j DROP

And the following commands to make all nodes see the same packets:

Quoted 7 lines

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
	--destination-mac 01:00:5e:00:01:01 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27

Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.

Quoted 11 lines
arptables -I OUTPUT -o eth2 --h-length 6 \
	-j mangle --mangle-mac-s 01:00:5e:00:01:02
arptables -I INPUT -i eth2 --h-length 6 \
	--destination-mac 01:00:5e:00:01:02 \
	-j mangle --mangle-mac-d 00:zz:yy:xx:5a:27

In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

I'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?

Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?

Quoted 10 lines

The match also provides a /proc entry under:

/proc/sys/net/netfilter/cluster/$PROC_NAME

where PROC_NAME is set via --cluster-proc-name. This is useful to include possible cluster reconfigurations via fail-over scripts. Assuming that this is the node 1, if node 2 is down, you can add node 2 to your node-mask as follows:

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

Does this provide anything you can't do by replacing the rule itself?

author maintainer
#6 lore raw
Quoted 37 lines · Patrick McHardy
Quoted 2 lines · Pablo Neira Ayuso

This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.

I'm mixing comments to the cluster match and the ARP mangle target.

Quoted 23 lines

Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:

    jhash(source IP) % total_nodes == node_id

For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):

iptables -I PREROUTING -t mangle -i eth1 -m cluster \
    --cluster-total-nodes 2 --cluster-local-node 1 \
    --cluster-proc-name eth1 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth1 \
    -m mark ! --mark 0xffff -j DROP
iptables -A PREROUTING -t mangle -i eth2 -m cluster \
    --cluster-total-nodes 2 --cluster-local-node 1 \
    --cluster-proc-name eth2 -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth2 \
    -m mark ! --mark 0xffff -j DROP

And the following commands to make all nodes see the same packets:

Quoted 7 lines

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
    -j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
    --destination-mac 01:00:5e:00:01:01 \
    -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 3 lines

Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.

Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.

Quoted 6 lines
Quoted 6 lines
arptables -I OUTPUT -o eth2 --h-length 6 \
    -j mangle --mangle-mac-s 01:00:5e:00:01:02
arptables -I INPUT -i eth2 --h-length 6 \
    --destination-mac 01:00:5e:00:01:02 \
    -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 8 lines
Quoted 5 lines

In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

I'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?

If TCP pickup is enabled, one TCP ACK packet coming in the reply direction enters TCP ESTABLISHED state. Since that's a valid state-transition, the cluster match will consider that this is part of a connection that this node is handling since it's a valid state-transition. The cluster match does not mark packets that trigger invalid state transitions.

Quoted 3 lines

Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?

No idea who has done this, but I can send you a patch to fix this naming without breaking backward.

Quoted 9 lines
Quoted 9 lines

The match also provides a /proc entry under:

/proc/sys/net/netfilter/cluster/$PROC_NAME

where PROC_NAME is set via --cluster-proc-name. This is useful to include possible cluster reconfigurations via fail-over scripts. Assuming that this is the node 1, if node 2 is down, you can add node 2 to your node-mask as follows:

Quoted 4 lines
Quoted 1 line

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

Does this provide anything you can't do by replacing the rule itself?

Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.

Now, I see that there is a possible optimization that consists of checking if one node has its node mask all set with regards to the total number of nodes, so that hashing can be skipped. But that's something that we can add later I think.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#7 lore raw
Quoted 9 lines · Pablo Neira Ayuso
Quoted 8 lines · Patrick McHardy
Quoted 7 lines

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
    -j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
    --destination-mac 01:00:5e:00:01:01 \
    -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 5 lines
Quoted 3 lines

Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.

Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.

Great, I wasn't aware that this already existed in userspace :)

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

I'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?

If TCP pickup is enabled, one TCP ACK packet coming in the reply direction enters TCP ESTABLISHED state. Since that's a valid state-transition, the cluster match will consider that this is part of a connection that this node is handling since it's a valid state-transition. The cluster match does not mark packets that trigger invalid state transitions.

Why use conntrack at all? Shouldn't the cluster match simply filter out all packets not for this cluster and thats it? You stated it needs conntrack to get a constant tuple, but I don't see why the conntrack tuple would differ from the data that you can gather from the packet headers.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

Does this provide anything you can't do by replacing the rule itself?

Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.

I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?

Quoted 1 line

Now, I see that there is a possible optimization that consists of checking if one node has its node mask all set with regards to the total number of nodes, so that hashing can be skipped. But that's something that we can add later I think.

Indeed.

author maintainer
#8 lore raw
Quoted 10 lines · Patrick McHardy
Quoted 9 lines · Pablo Neira Ayuso
Quoted 8 lines · Patrick McHardy
Quoted 7 lines

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
    -j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
    --destination-mac 01:00:5e:00:01:01 \
    -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.

Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.

Great, I wasn't aware that this already existed in userspace :)

Yes, it's hosted by the ebtables projects. That tool really need some care. It works but I don't know if it's actively maintained. Probably we can offer hosting for it in git.netfilter.org. I'll investigate this.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.

echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose

I'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?

Quoted 7 lines
Quoted 1 line

If TCP pickup is enabled, one TCP ACK packet coming in the reply direction enters TCP ESTABLISHED state. Since that's a valid state-transition, the cluster match will consider that this is part of a connection that this node is handling since it's a valid state-transition. The cluster match does not mark packets that trigger invalid state transitions.

Why use conntrack at all? Shouldn't the cluster match simply filter out all packets not for this cluster and thats it? You stated it needs conntrack to get a constant tuple, but I don't see why the conntrack tuple would differ from the data that you can gather from the packet headers.

No, source NAT connections would have different headers. A -> B for original, and B -> FW for reply direction. Thus, I cannot apply the same hashing for packets going in the original and the reply direction. Moreover, if this is packet-based, think also about possible asymmetric filtering: original traffic direction filtered by node 1 but reply traffic filtered by node 2. That will not work for a stateful firewall.

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

Does this provide anything you can't do by replacing the rule itself?

Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.

I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?

Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#9 lore raw
Quoted 8 lines · Pablo Neira Ayuso
Quoted 5 lines · Patrick McHardy

Why use conntrack at all? Shouldn't the cluster match simply filter out all packets not for this cluster and thats it? You stated it needs conntrack to get a constant tuple, but I don't see why the conntrack tuple would differ from the data that you can gather from the packet headers.

No, source NAT connections would have different headers. A -> B for original, and B -> FW for reply direction. Thus, I cannot apply the same hashing for packets going in the original and the reply direction.

Ah I'm beginning to understand the topology I think :) Actually it seems its only combined SNAT+DNAT on one connections thats a problem, with only one of both you could tell the cluster match to look at either source or destination address (the unchanged one) in the opposite direction. Only if the opposite direction is completely unrelated from a non-conntrack view we can't deal with it. Anyways, your way to deal with this seems fine to me.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME

Does this provide anything you can't do by replacing the rule itself?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.

I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?

Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.

Thats not what I meant. By having a single node handle all connections from the one which went down, you have an imbalance in load distribution. The nodes are synchronized, so they could just all replace their cluster match with an updated number of nodes.

author maintainer
#10 lore raw
In reply to Pablo Neira Ayuso
Quoted 4 lines · Pablo Neira Ayuso
Quoted 3 lines · Patrick McHardy

I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?

Hm, I was not replying to your question ;). Right, the renumbering also requires getting the states back to the original node. We can use the same hashing approach in userspace to know which states belong to original node that has come back to life when it requests a resynchronization.

Quoted 1 line

Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.

This is not true. We may have something like this (assuming two nodes):

if no mark set and hash % 2 == 0, accept
if no mark set and hash % 2 == 1, accept
if no mark set, drop

So we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#11 lore raw
Quoted 11 lines · Pablo Neira Ayuso
Quoted 4 lines · Pablo Neira Ayuso
Quoted 3 lines · Patrick McHardy

I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?

Hm, I was not replying to your question ;). Right, the renumbering also requires getting the states back to the original node. We can use the same hashing approach in userspace to know which states belong to original node that has come back to life when it requests a resynchronization.

Quoted 1 line

Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.

This is not true. We may have something like this (assuming two nodes):

To whom are you replying now? :)

Quoted 5 lines

if no mark set and hash % 2 == 0, accept
if no mark set and hash % 2 == 1, accept
if no mark set, drop

So we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.

I'm sure someone would argue that changing TCP port numbers of the tcp match through proc would be a nice and simple interface. The fact though is that we have an interface for handing a configuration to the kernel and this is clearly a configuration parameter. We're missing a proper way to use it in userspace from within programs (well, hopefully not for long anymore), but that needs to be fixed in userspace.

author maintainer
#12 lore raw
Quoted 7 lines · Patrick McHardy
Quoted 6 lines · Pablo Neira Ayuso
Quoted 4 lines · Pablo Neira Ayuso
Quoted 3 lines · Patrick McHardy

I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?

Quoted 7 lines
Quoted 5 lines

Hm, I was not replying to your question ;). Right, the renumbering also requires getting the states back to the original node. We can use the same hashing approach in userspace to know which states belong to original node that has come back to life when it requests a resynchronization.

Quoted 1 line

Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.

This is not true. We may have something like this (assuming two nodes):

To whom are you replying now? :)

To myself, never mind :)

Quoted 6 lines
Quoted 5 lines

if no mark set and hash % 2 == 0, accept
if no mark set and hash % 2 == 1, accept
if no mark set, drop

So we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.

Quoted 7 lines

I'm sure someone would argue that changing TCP port numbers of the tcp match through proc would be a nice and simple interface. The fact though is that we have an interface for handing a configuration to the kernel and this is clearly a configuration parameter. We're missing a proper way to use it in userspace from within programs (well, hopefully not for long anymore), but that needs to be fixed in userspace.

Right right, I have no more arguments to support the /proc interface ;). I'll send you a patch without it late at night.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#25 lore raw
In reply to Pablo Neira Ayuso
Quoted 11 lines · Pablo Neira Ayuso
Quoted 11 lines
Quoted 10 lines

And the following commands to make all nodes see the same packets:

ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2

arptables -I OUTPUT -o eth1 --h-length 6 \
   -j mangle --mangle-mac-s 01:00:5e:00:01:01
arptables -I INPUT -i eth1 --h-length 6 \
   --destination-mac 01:00:5e:00:01:01 \
   -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 5 lines
Quoted 3 lines

Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.

Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"?

In case of --mangle-mac-d, getopt automatically handles abbreviations (the best example I can think of where this works is quilt ref --s --d for --sort --diffstat).

Quoted 5 lines
Quoted 3 lines

Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?

No idea who has done this,

Lowercasing targets was the original author's (Bart De Schuymer's) deed ;-)

I have nothing against lowercase targets, in fact, many a user seems to so not get case-sensitity right — “INPUT” vs. “input” (built-in chain names), “MASQUERADE” vs. “masquerade” (targets), to name some — that I would support lowercasing all targets...— if it were not for the clash between MARK (target) and mark (match), for example. Though, calling one of them “mark_m” or underscore-something to avoid the clash however, is what I dislike in favor of uppercasing target names.

--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#26 lore raw
Quoted 8 lines · Jan Engelhardt
Quoted 4 lines
Quoted 3 lines

Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?

No idea who has done this,

Lowercasing targets was the original author's (Bart De Schuymer's) deed ;-)

Quoted 8 lines

I have nothing against lowercase targets, in fact, many a user seems to so not get case-sensitity right — “INPUT” vs. “input” (built-in chain names), “MASQUERADE” vs. “masquerade” (targets), to name some — that I would support lowercasing all targets...— if it were not for the clash between MARK (target) and mark (match), for example. Though, calling one of them “mark_m” or underscore-something to avoid the clash however, is what I dislike in favor of uppercasing target names.

Its actually just exposing an implementation detail of how match and target modules are located on the filesystem. The meaning (match or target) can be deduced from "-j" and "-m".
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw
In reply to Pablo Neira Ayuso
Quoted 11 lines · Pablo Neira Ayuso
Quoted 8 lines
Quoted 6 lines

Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.

Great, I wasn't aware that this already existed in userspace :)

Yes, it's hosted by the ebtables projects. That tool really need some care.

It would indeed. The problem though is, that I, who originally wanted to unify arptables into iptables (as a start, because it has not diverged as much as ebtables), have thrown the towel on that on the grounds of libiptc being such a beast[1], and that developers' time would be better spent on (fasten seatbelts) duplicating xtables/iptables (once more) with the goal of creating an NFPROTO-agnostic table structure instead that obsoletes ip,ip6,arp and ebtables in one go.

[1] http://marc.info/?t=122633592000011&r=1&w=2

#27 lore raw
In reply to Patrick McHardy
Quoted 4 lines · Patrick McHardy
Quoted 4 lines

Lowercasing targets was the original author's (Bart De Schuymer's) deed ;-)

Quoted 9 lines
Quoted 4 lines

[...]
Though, calling one of them “mark_m” or underscore-something to avoid the clash however, is what I dislike in favor of uppercasing target names.

Its actually just exposing an implementation detail of how match and target modules are located on the filesystem. The meaning (match or target) can be deduced from "-j" and "-m".

Too bad we cannot reasonably change it now.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author maintainer
#13 lore raw
In reply to Patrick McHardy

Hi Patrick,

Quoted 11 lines · Patrick McHardy
Quoted 2 lines · Pablo Neira Ayuso

So we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.

I'm sure someone would argue that changing TCP port numbers of the tcp match through proc would be a nice and simple interface. The fact though is that we have an interface for handing a configuration to the kernel and this is clearly a configuration parameter. We're missing a proper way to use it in userspace from within programs (well, hopefully not for long anymore), but that needs to be fixed in userspace.

While reworking this, I think that I have found one argument to support the /proc interface that looks interesting in terms of resource consumption. Assume that we have three nodes, where two of them are down, thus, the only one active would have the following rule-set:

iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 1 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 2 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 3 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 \
        -m mark ! --mark 0xffff -j DROP

Look at the worst case: if the packet goes to node 3, the hashing must be done to check if the packet belongs to node 1 and node 2. Thus, the hashing is done three times. This makes the cluster hashing O(n) where n is the number of cluster nodes.

A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#14 lore raw
Quoted 22 lines · Pablo Neira Ayuso

While reworking this, I think that I have found one argument to support the /proc interface that looks interesting in terms of resource consumption. Assume that we have three nodes, where two of them are down, thus, the only one active would have the following rule-set:

iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 1 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 2 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 3 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 \
        -m mark ! --mark 0xffff -j DROP

Look at the worst case: if the packet goes to node 3, the hashing must be done to check if the packet belongs to node 1 and node 2. Thus, the hashing is done three times. This makes the cluster hashing O(n) where n is the number of cluster nodes.

Quoted 5 lines

A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.

That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.

author maintainer
#15 lore raw
Quoted 29 lines · Patrick McHardy
Quoted 22 lines · Pablo Neira Ayuso

While reworking this, I think that I have found one argument to support the /proc interface that looks interesting in terms of resource consumption. Assume that we have three nodes, where two of them are down, thus, the only one active would have the following rule-set:

iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 1 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 2 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
        --cluster-total-nodes 3 --cluster-local-node 3 \
        -j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 \
        -m mark ! --mark 0xffff -j DROP

Look at the worst case: if the packet goes to node 3, the hashing must be done to check if the packet belongs to node 1 and node 2. Thus, the hashing is done three times. This makes the cluster hashing O(n) where n is the number of cluster nodes.

Quoted 5 lines

A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.

Quoted 3 lines

That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.

Indeed, but in practise existing failover daemons (at least those free/opensource that I know) doesn't show that "intelligent" behaviour since they initially (according to the configuration file) assign the resources to each node, and if one node fails, it assigns the corresponding resources to another sane node (ie. the daemon runs a script with the corresponding iptables rules).

Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.

Therefore AFAICS, without the /proc interface, I would need one iptables rule per cluster-local-node handled, and so it's still the possible sub-optimal situation when one or several node fails.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#16 lore raw

OoO En ce début d'après-midi nuageux du mardi 17 février 2009, vers 14:50, Pablo Neira Ayuso <[email protected]> disait :

Quoted 7 lines

Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.

Moreover, some of them (the one that are using VRRP for example) don't report the total number of nodes still alive. As a user, I would prefer a simple /proc interface to add/remove a node.

Signature · 6 lines
Make sure all variables are initialised before use.
            - The Elements of Programming Style (Kernighan & Plauger)
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#18 lore raw
In reply to Pablo Neira Ayuso
Quoted 25 lines · Pablo Neira Ayuso
Quoted 8 lines · Patrick McHardy
Quoted 5 lines

A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.

That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.

Indeed, but in practise existing failover daemons (at least those free/opensource that I know) doesn't show that "intelligent" behaviour since they initially (according to the configuration file) assign the resources to each node, and if one node fails, it assigns the corresponding resources to another sane node (ie. the daemon runs a script with the corresponding iptables rules).

Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.

Quoted 3 lines

Therefore AFAICS, without the /proc interface, I would need one iptables rule per cluster-local-node handled, and so it's still the possible sub-optimal situation when one or several node fails.

OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.

BTW, I recently looked into TIPC, its incredibly easy to use since it deals with dead-node dectection etc internally and all you need to do is exchange a few messages. Might be quite easy to write a smarter failover daemon.

#17 lore raw
In reply to Vincent Bernat
Quoted 11 lines · Vincent Bernat

OoO En ce début d'après-midi nuageux du mardi 17 février 2009, vers 14:50, Pablo Neira Ayuso <[email protected]> disait :

Quoted 7 lines

Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.

Quoted 3 lines

Moreover, some of them (the one that are using VRRP for example) don't report the total number of nodes still alive. As a user, I would prefer a simple /proc interface to add/remove a node.

That "simple" argument really doesn't cut it, there's nothing inherently more complicated in executing an iptables command compared to executing an echo command. Most likely some program is going to do it anyways. --
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author maintainer
#19 lore raw
In reply to Patrick McHardy
Quoted 28 lines · Patrick McHardy
Quoted 27 lines · Pablo Neira Ayuso
Quoted 10 lines · Patrick McHardy
Quoted 7 lines

A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result
of the hash in the skbuff mark, but the problem is that it would require
a reserved space for hashmarks since they may clash with other user-defined marks.

That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.

Indeed, but in practise existing failover daemons (at least those free/opensource that I know) doesn't show that "intelligent" behaviour since they initially (according to the configuration file) assign the resources to each node, and if one node fails, it assigns the corresponding resources to another sane node (ie. the daemon runs a script with the corresponding iptables rules).

Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.

Quoted 7 lines
Quoted 3 lines

Therefore AFAICS, without the /proc interface, I would need one iptables rule per cluster-local-node handled, and so it's still the possible sub-optimal situation when one or several node fails.

OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.

:(

Quoted 4 lines

BTW, I recently looked into TIPC, its incredibly easy to use since it deals with dead-node dectection etc internally and all you need to do is exchange a few messages. Might be quite easy to write a smarter failover daemon.

I see, I don't have more convincing arguments that "I would also need time for that but in the meanwhile, please allow this". Well, failover daemons are delicate pieces of software, they have to be stable, well-tested, bug-free, give timely responses. Still TIPC is experimental and I guess that the dead-node detection is only layer 3/4 based on heartbeats. Dead-node detection is a tricky issue, the more you can perform different layer checkings, the more increase chances to make wrong decisions that may lead to inconsistent situations and tons of problems. VRRP is the current standard and this one of his limitations, and so on.

Well, if you are not going to accept the /proc interface, not matter what I can argument, I give up on this ;)

Anyway, probably, this is a premature optimization (but worth?). Some numbers, in my testbed, I get ~1800 TCP connections per second less with eight cluster rules (no /proc interface).

24347 TCP connections per second with one rule. 22580 TCP connections per second with eight rules.

OK, I'll send you another patch without the /proc interface.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
#20 lore raw

[Please trim unrelated content, these mails are getting hard to read]

Quoted 17 lines · Pablo Neira Ayuso
Quoted 4 lines · Patrick McHardy

BTW, I recently looked into TIPC, its incredibly easy to use since it deals with dead-node dectection etc internally and all you need to do is exchange a few messages. Might be quite easy to write a smarter failover daemon.

I see, I don't have more convincing arguments that "I would also need time for that but in the meanwhile, please allow this". Well, failover daemons are delicate pieces of software, they have to be stable, well-tested, bug-free, give timely responses. Still TIPC is experimental and I guess that the dead-node detection is only layer 3/4 based on heartbeats. Dead-node detection is a tricky issue, the more you can perform different layer checkings, the more increase chances to make wrong decisions that may lead to inconsistent situations and tons of problems. VRRP is the current standard and this one of his limitations, and so on.

Quoted 2 lines

Well, if you are not going to accept the /proc interface, not matter what I can argument, I give up on this ;)

I'm afraid I can't be convinced of this. If you want to specify multiple node ids, have the iptables command accept them, but there's no reason to use proc for this.

Quoted 8 lines

Anyway, probably, this is a premature optimization (but worth?). Some numbers, in my testbed, I get ~1800 TCP connections per second less with eight cluster rules (no /proc interface).

24347 TCP connections per second with one rule. 22580 TCP connections per second with eight rules.

OK, I'll send you another patch without the /proc interface.

Thanks. As I said, I don't have anything against handling multiple nodes in one rule, as long as its not done using proc.

#21 lore raw
In reply to Pablo Neira Ayuso

OoO Pendant le temps de midi du mercredi 18 février 2009, vers 12:06, Pablo Neira Ayuso <[email protected]> disait :

Quoted 5 lines
Quoted 3 lines

OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.

:(

A solution would be to give the mask on the command line. The HA daemon will output the assigned nodes to a file and the content of this file will be computed into a mask.

Signature · 8 lines
THE BOYS ROOM IS NOT A WATER PARK
THE BOYS ROOM IS NOT A WATER PARK
THE BOYS ROOM IS NOT A WATER PARK
-+- Bart Simpson on chalkboard in episode 3F03
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#22 lore raw
Quoted 7 lines · Vincent Bernat

OoO Pendant le temps de midi du mercredi 18 février 2009, vers 12:06, Pablo Neira Ayuso <[email protected]> disait :

Quoted 3 lines
Quoted 3 lines

OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.

Quoted 5 lines
Quoted 1 line

:(

A solution would be to give the mask on the command line. The HA daemon will output the assigned nodes to a file and the content of this file will be computed into a mask.

Indeed. Just a bitmask (is 64 nodes max enough?) should be fine. --
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author maintainer
#23 lore raw
Quoted 8 lines · Patrick McHardy
Quoted 7 lines · Vincent Bernat

OoO Pendant le temps de midi du mercredi 18 février 2009, vers 12:06, Pablo Neira Ayuso <[email protected]> disait :

Quoted 3 lines
Quoted 3 lines

OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

:(

A solution would be to give the mask on the command line. The HA daemon will output the assigned nodes to a file and the content of this file will be computed into a mask.

Indeed. Just a bitmask (is 64 nodes max enough?) should be fine.

Good point :), I'll add this to the patch that I'm reworking.

Signature · 5 lines
"Los honestos son inadaptados sociales" -- Les Luthiers
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html