netfilter: xtables: add cluster match
Subsystems (2)
Pablo Neira Ayuso
maintainerPATCH 1/1 netfilter: xtables: add cluster match
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.
Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:
jhash(source IP) % total_nodes == node_id
For related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):
iptables -I PREROUTING -t mangle -i eth1 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth1 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth1 \ -m mark ! --mark 0xffff -j DROP iptables -A PREROUTING -t mangle -i eth2 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth2 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth2 \ -m mark ! --mark 0xffff -j DROP
And the following commands to make all nodes see the same packets:
ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2
arptables -I OUTPUT -o eth1 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:01 arptables -I INPUT -i eth1 --h-length 6 \ --destination-mac 01:00:5e:00:01:01 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27 arptables -I OUTPUT -o eth2 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:02 arptables -I INPUT -i eth2 --h-length 6 \ --destination-mac 01:00:5e:00:01:02 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.
echo 0 > /proc/sys/net/netfilter/nf_conntrack_tcp_loose
The match also provides a /proc entry under:
/proc/sys/net/netfilter/cluster/$PROC_NAME
where PROC_NAME is set via --cluster-proc-name. This is useful to include possible cluster reconfigurations via fail-over scripts. Assuming that this is the node 1, if node 2 is down, you can add node 2 to your node-mask as follows:
echo +2 > /proc/sys/net/netfilter/cluster/$PROC_NAME
BTW, some final notes:
* This match mangles the skbuff pkt_type in case that it detectsPACKET_MULTICAST for a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.
---
include/linux/netfilter/xt_cluster.h | 21 ++ net/netfilter/Kconfig | 15 + net/netfilter/Makefile | 1 net/netfilter/xt_cluster.c | 368 ++++++++++++++++++++++++++++++++++ 4 files changed, 405 insertions(+), 0 deletions(-) create mode 100644 include/linux/netfilter/xt_cluster.h create mode 100644 net/netfilter/xt_cluster.c
include/linux/netfilter/xt_cluster.h new file +21 -0
@@ -0,0 +1,21 @@ +#ifndef _XT_CLUSTER_MATCH_H +#define _XT_CLUSTER_MATCH_H + +struct proc_dir_entry ; + +enum xt_cluster_flags { +XT_CLUSTER_F_INV =0 , +}; + +struct xt_cluster_match_info { +u_int16_t total_nodes; +u_int16_t node_id; +u_int32_t hash_seed; +char proc_name[16 ]; +u_int32_t flags; + +/* Used internally by the kernel */ +void *data __attribute__((aligned (8 ))); +}; + +#endif /* _XT_CLUSTER_MATCH_H */
net/netfilter/Kconfig +15 -0
@@ -488,6 +488,21 @@ config NETFILTER_XT_TARGET_TCPOPTSTRIP This option adds a "TCPOPTSTRIP" target, which allows you to strip TCP options from TCP packets. +config NETFILTER_XT_MATCH_CLUSTER + tristate '"cluster" match support' + depends on NETFILTER_ADVANCED + ---help--- + This option allows you to build work-load-sharing clusters of + network servers/stateful firewalls without having a dedicated + load-balancing router/server/switch. Basically, this match returns + true when the packet must be handled by this cluster node. Thus, + all nodes see all packets and this match decides which node handles + what packets. The work-load sharing algorithm is based on source + address hashing. + + If you say Y here, try `iptables -m cluster --help` for + more information. + config NETFILTER_XT_MATCH_COMMENT tristate '"comment" match support' depends on NETFILTER_ADVANCED
net/netfilter/Makefile +1 -0
@@ -57,6 +57,7 @@ obj-$(CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP) += xt_TCPOPTSTRIP.o obj-$(CONFIG_NETFILTER_XT_TARGET_TRACE) += xt_TRACE.o # matches +obj-$(CONFIG_NETFILTER_XT_MATCH_CLUSTER) += xt_cluster.o obj-$(CONFIG_NETFILTER_XT_MATCH_COMMENT) += xt_comment.o obj-$(CONFIG_NETFILTER_XT_MATCH_CONNBYTES) += xt_connbytes.o obj-$(CONFIG_NETFILTER_XT_MATCH_CONNLIMIT) += xt_connlimit.o
net/netfilter/xt_cluster.c new file +368 -0
368 lines not on this page · raw message
Quoted 2 lines · Pablo Neira Ayuso
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.
All of this nice text (below) should go into libxt_cluster.man :)
Quoted 12 lines
Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:
jhash(source IP) % total_nodes == node_idFor related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):
iptables -I PREROUTING -t mangle -i eth1 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \
[...]
Quoted 9 lines
echo +2 >
/proc/sys/net/netfilter/cluster/$PROC_NAMEBTW, some final notes:
* This match mangles the skbuff
pkt_typein case that it detectsPACKET_MULTICASTfor a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.
The supersedes statement should also go into Kconfig.
include/linux/netfilter/xt_cluster.h line 8
Quoted 9 lines
include/linux/netfilter/xt_cluster.h new file +8
-0@@ -0,0 +1,21 @@ +#ifndef _XT_CLUSTER_MATCH_H +#define _XT_CLUSTER_MATCH_H + +struct proc_dir_entry ; + +enum xt_cluster_flags { +XT_CLUSTER_F_INV =0 , +};
Hm, that should be XT_CLUSTER_F_INV = 1 << 0.
net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 493
Quoted 3 lines
net/netfilter/Kconfig +3
-0+config NETFILTER_XT_MATCH_CLUSTER + tristate '"cluster" match support' + depends on NETFILTER_ADVANCED
xt_cluster depends on NF_CONNTRACK too.
net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 504
Quoted 11 lines
net/netfilter/Kconfig +11
-0+ ---help--- + This option allows you to build work-load-sharing clusters of + network servers/stateful firewalls without having a dedicated + load-balancing router/server/switch. Basically, this match returns + true when the packet must be handled by this cluster node. Thus, + all nodes see all packets and this match decides which node handles + what packets. The work-load sharing algorithm is based on source + address hashing. + + If you say Y here, try `iptables -m cluster --help` for + more information.
Somehow this gives the impression that Y is the only logical choice, when indeed, M would work too.
net/netfilter/xt_cluster.c line 21
Quoted 2 lines
net/netfilter/xt_cluster.c new file +2
-0+struct xt_cluster_internal { +unsigned long node_mask;
I think I raised concern about this, but can't remember. Should not this be of type nodemask_t instead? Or ... is this "node" perhaps not describing a NUMA node, but a cluster node? Some comments would be appreciated, along the lines of
/**
* @node_mask: cluster node mask
*/
net/netfilter/xt_cluster.c line 69
Quoted 6 lines
net/netfilter/xt_cluster.c new file +5
-0+struct proc_dir_entry *proc; +atomic_t use; +}; +static inline bool +xt_cluster_is_multicast_addr (const struct sk_buff *skb,int family)
Cosmetic warrants uint8_t family.
Quoted 5 lines
net/netfilter/xt_cluster.c new file +5
-0+static bool +xt_cluster_mt (const struct sk_buff *skb,const struct xt_match_param *par) +{ +struct sk_buff *pskb = (struct sk_buff *)skb; +const struct xt_cluster_match_info *info = par->matchinfo;
net/netfilter/xt_cluster.c line 119
Quoted 10 lines
net/netfilter/xt_cluster.c new file +10
-0+const struct xt_cluster_internal *internal = info->data; +const struct nf_conn *ct; +enum ip_conntrack_info ctinfo; +unsigned long hash; +bool inv = !!(info->flags &XT_CLUSTER_F_INV ); + +if (!xt_cluster_is_multicast_addr (skb, par->family) && + skb->pkt_type ==PACKET_MULTICAST ) { + pskb->pkt_type =PACKET_HOST ; + }
{} are redundant here
net/netfilter/xt_cluster.c line 133
Quoted 6 lines
net/netfilter/xt_cluster.c new file +6
-0+if (ct->master) + hash =xt_cluster_hash (ct->master, info); +else + hash =xt_cluster_hash (ct, info); + +return test_bit (hash, &internal->node_mask) ^ inv;
Since inv is just once used, I would "inline" it, aka
return test_bit(hash, &internal->node_mask) ^ !!(info->flags & XT_CLUSTER_MATCH_INV);
net/netfilter/xt_cluster.c line 160
Quoted 4 lines
net/netfilter/xt_cluster.c new file +4
-0+static int xt_cluster_seq_show (struct seq_file *s,void *v) +{ +unsigned long *mask = v; +seq_printf (s,"0x%.8lx\n" , *mask);
'.' does not make sense with non-string,non-floating point numbers (though it is a stdc feature it seems). I'd say "0x%08lx", for clarity.
net/netfilter/xt_cluster.c line 201
Quoted 9 lines
net/netfilter/xt_cluster.c new file +9
-0+case '-' : + new_node_id =simple_strtoul (buffer+1 ,NULL ,10 ); +if (!new_node_id || new_node_id >sizeof (info->node_mask)*8 ) +return -EIO ; +printk (KERN_NOTICE "cluster: deleting node %u\n" , new_node_id); +clear_bit (new_node_id-1 , &info->node_mask); +break ; +default : +return -EIO ;
EINVAL, I'd say.
net/netfilter/xt_cluster.c line 240
Quoted 9 lines
net/netfilter/xt_cluster.c new file +9
-0+static bool +xt_cluster_proc_entry_exist (struct proc_dir_entry *dir,const char *name) +{ +struct proc_dir_entry *tmp; + +for (tmp = dir->subdir; tmp; tmp = tmp->next) { +if (strcmp (tmp->name, name) ==0 ) +return true; + }
-{}
Looks good so far.
Pablo Neira Ayuso
maintainerQuoted 6 lines · Jan Engelhardt
Quoted 2 lines · Pablo Neira Ayuso
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.
All of this nice text (below) should go into libxt_cluster.man :)
Indeed, that will go into the manpage. I have kept the iptables part until you finish with Jamal's libxtables renaming, to avoid any clashing. Anyway, I don't think that adding this information to the manpage is a reason to trim the patch description, actually I think that this patch deserves this long description and I have seen longer descriptions in the kernel changelog :)
Quoted 15 lines
Quoted 12 lines
Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:
jhash(source IP) % total_nodes == node_idFor related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):
iptables -I PREROUTING -t mangle -i eth1 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \[...]
Quoted 2 lines
echo +2 >
/proc/sys/net/netfilter/cluster/$PROC_NAME
Quoted 8 lines
Quoted 6 lines
BTW, some final notes:
* This match mangles the skbuff
pkt_typein case that it detectsPACKET_MULTICASTfor a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.The supersedes statement should also go into
Kconfig.
Yes, but I was also waiting for Patrick to tell how to proceed with this. I think that we also have to add CLUSTERIP to the schedule-for-removal list.
Quoted 11 lines
include/linux/netfilter/xt_cluster.h line 8
Quoted 9 lines
include/linux/netfilter/xt_cluster.h new file +8
-0@@ -0,0 +1,21 @@ +#ifndef _XT_CLUSTER_MATCH_H +#define _XT_CLUSTER_MATCH_H + +struct proc_dir_entry ; + +enum xt_cluster_flags { +XT_CLUSTER_F_INV =0 , +};Hm, that should be
XT_CLUSTER_F_INV= 1 << 0.
Indeed.
Quoted 5 lines
net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 493
Quoted 3 lines
net/netfilter/Kconfig +3
-0+config NETFILTER_XT_MATCH_CLUSTER + tristate '"cluster" match support' + depends on NETFILTER_ADVANCED
xt_clusterdepends onNF_CONNTRACKtoo.
Right.
Quoted 15 lines
net/netfilter/Kconfig config NETFILTER_XT_TARGET_TCPOPTSTRIP · line 504
Quoted 11 lines
net/netfilter/Kconfig +11
-0+ ---help--- + This option allows you to build work-load-sharing clusters of + network servers/stateful firewalls without having a dedicated + load-balancing router/server/switch. Basically, this match returns + true when the packet must be handled by this cluster node. Thus, + all nodes see all packets and this match decides which node handles + what packets. The work-load sharing algorithm is based on source + address hashing. + + If you say Y here, try `iptables -m cluster --help` for + more information.Somehow this gives the impression that Y is the only logical choice, when indeed, M would work too.
Quoted 7 lines
net/netfilter/xt_cluster.c line 21
Quoted 2 lines
net/netfilter/xt_cluster.c new file +2
-0+struct xt_cluster_internal { +unsigned long node_mask;I think I raised concern about this, but can't remember. Should not this be of type
nodemask_tinstead? Or ... is this "node" perhaps not describing a NUMA node, but a cluster node? Some comments would be appreciated, along the lines of
You mentioned cpumask_t last time, but this is neither related to NUMA nor a CPU mask, so I prefer to keep using a long for this thing, moreover test and set bit operations use long.
Quoted 22 lines
/**
* @node_mask: cluster node mask
*/net/netfilter/xt_cluster.c line 69
Quoted 6 lines
net/netfilter/xt_cluster.c new file +5
-0+struct proc_dir_entry *proc; +atomic_t use; +}; +static inline bool +xt_cluster_is_multicast_addr (const struct sk_buff *skb,int family)Cosmetic warrants
uint8_tfamily.Quoted 9 lines
net/netfilter/xt_cluster.c new file +9
-0+static bool +xt_cluster_mt (const struct sk_buff *skb,const struct xt_match_param *par) +{ +struct sk_buff *pskb = (struct sk_buff *)skb; +const struct xt_cluster_match_info *info = par->matchinfo; +const struct xt_cluster_internal *internal = info->data; +const struct nf_conn *ct; +enum ip_conntrack_info ctinfo; +unsigned long hash;
Quoted 8 lines
net/netfilter/xt_cluster.c line 119
Quoted 6 lines
net/netfilter/xt_cluster.c new file +6
-0+bool inv = !!(info->flags &XT_CLUSTER_F_INV ); + +if (!xt_cluster_is_multicast_addr (skb, par->family) && + skb->pkt_type ==PACKET_MULTICAST ) { + pskb->pkt_type =PACKET_HOST ; + }{} are redundant here
When lines are splitted, I have seen these redundant {} for readability.
Quoted 10 lines
net/netfilter/xt_cluster.c line 133
Quoted 6 lines
net/netfilter/xt_cluster.c new file +6
-0+if (ct->master) + hash =xt_cluster_hash (ct->master, info); +else + hash =xt_cluster_hash (ct, info); + +return test_bit (hash, &internal->node_mask) ^ inv;Since inv is just once used, I would "inline" it, aka
return test_bit(hash, &internal->node_mask) ^ !!(info->flags & XT_CLUSTER_MATCH_INV);
Makes sense.
Quoted 7 lines
net/netfilter/xt_cluster.c line 160
Quoted 4 lines
net/netfilter/xt_cluster.c new file +4
-0+static int xt_cluster_seq_show (struct seq_file *s,void *v) +{ +unsigned long *mask = v; +seq_printf (s,"0x%.8lx\n" , *mask);'.' does not make sense with non-string,non-floating point numbers (though it is a stdc feature it seems). I'd say "0x%08lx", for clarity.
OK
Quoted 11 lines
net/netfilter/xt_cluster.c line 201
Quoted 9 lines
net/netfilter/xt_cluster.c new file +9
-0+case '-' : + new_node_id =simple_strtoul (buffer+1 ,NULL ,10 ); +if (!new_node_id || new_node_id >sizeof (info->node_mask)*8 ) +return -EIO ; +printk (KERN_NOTICE "cluster: deleting node %u\n" , new_node_id); +clear_bit (new_node_id-1 , &info->node_mask); +break ; +default : +return -EIO ;EINVAL, I'd say.
Other /proc-related code uses this error, but indeed I prefer EINVAL.
Quoted 11 lines
net/netfilter/xt_cluster.c line 240
Quoted 9 lines
net/netfilter/xt_cluster.c new file +9
-0+static bool +xt_cluster_proc_entry_exist (struct proc_dir_entry *dir,const char *name) +{ +struct proc_dir_entry *tmp; + +for (tmp = dir->subdir; tmp; tmp = tmp->next) { +if (strcmp (tmp->name, name) ==0 ) +return true; + }-{}
Same comment as above, it's there for readability.
Quoted 1 line
Looks good so far.
Thanks for the review.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
Quoted 11 lines · Pablo Neira Ayuso
Quoted 6 lines · Jan Engelhardt
Quoted 2 lines · Pablo Neira Ayuso
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.
All of this nice text (below) should go into libxt_cluster.man :)
Indeed, that will go into the manpage. I have kept the iptables part until you finish with Jamal's libxtables renaming, to avoid any clashing.
We are done and ready for release as I understood.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
* This match mangles the skbuff
pkt_typein case that it detectsPACKET_MULTICASTfor a non-multicast address. This may be done in a PKTTYPE target for this sole purpose.
* This match supersedes the CLUSTERIP target.The supersedes statement should also go into
Kconfig.Yes, but I was also waiting for Patrick to tell how to proceed with this. I think that we also have to add CLUSTERIP to the schedule-for-removal list.
We do not have to (there is still xt_state and no mention of deprecation despite xt_conntrack superseding it) - but it would be a most logical step indeed.
Quoted 9 lines
Quoted 2 lines
net/netfilter/xt_cluster.c line 21
Quoted 1 line
net/netfilter/xt_cluster.c new file +1
-0+unsigned long node_mask;You mentioned
cpumask_tlast time, but this is neither related to NUMA nor a CPU mask, so I prefer to keep using a long for this thing, moreover test and set bit operations use long.Quoted 3 lines
/**
* @node_mask: cluster node mask
*/
Hence the need for a comment so that this misunderstanding is resolved early.
Quoted 7 lines · Jan Engelhardt
Quoted 6 lines · Pablo Neira Ayuso
Quoted 5 lines · Jan Engelhardt
Quoted 2 lines · Pablo Neira Ayuso
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
All of this nice text (below) should go into libxt_cluster.man :)
Indeed, that will go into the manpage. I have kept the iptables part until you finish with Jamal's libxtables renaming, to avoid any clashing.
We are done and ready for release as I understood.
That is, after grabbing the latest contents of our tree.
Quoted 2 lines · Pablo Neira Ayuso
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.
I'm mixing comments to the cluster match and the ARP mangle target.
Quoted 23 lines
Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:
jhash(source IP) % total_nodes == node_idFor related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):
iptables -I PREROUTING -t mangle -i eth1 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth1 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth1 \ -m mark ! --mark 0xffff -j DROP iptables -A PREROUTING -t mangle -i eth2 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth2 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth2 \ -m mark ! --mark 0xffff -j DROPAnd the following commands to make all nodes see the same packets:
Quoted 7 lines
ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2arptables -I OUTPUT -o eth1 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:01 arptables -I INPUT -i eth1 --h-length 6 \ --destination-mac 01:00:5e:00:01:01 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.
Quoted 11 lines
arptables -I OUTPUT -o eth2 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:02 arptables -I INPUT -i eth2 --h-length 6 \ --destination-mac 01:00:5e:00:01:02 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.
echo 0 >
/proc/sys/net/netfilter/nf_conntrack_tcp_loose
I'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?
Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?
Quoted 10 lines
The match also provides a /proc entry under:
/proc/sys/net/netfilter/cluster/$PROC_NAMEwhere
PROC_NAMEis set via --cluster-proc-name. This is useful to include possible cluster reconfigurations via fail-over scripts. Assuming that this is the node 1, if node 2 is down, you can add node 2 to your node-mask as follows:echo +2 >
/proc/sys/net/netfilter/cluster/$PROC_NAME
Does this provide anything you can't do by replacing the rule itself?
Pablo Neira Ayuso
maintainerQuoted 37 lines · Patrick McHardy
Quoted 2 lines · Pablo Neira Ayuso
This patch adds the iptables cluster match. This match can be used to deploy gateway and back-end load-sharing clusters.
I'm mixing comments to the cluster match and the ARP mangle target.
Quoted 23 lines
Assuming that all the nodes see all packets (see below for an example on how to do that if your switch does not allow this), the cluster match decides if this node has to handle a packet given:
jhash(source IP) % total_nodes == node_idFor related connections, the master conntrack is used. The following is an example of its use to deploy a gateway cluster composed of two nodes (where this is the node 1):
iptables -I PREROUTING -t mangle -i eth1 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth1 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth1 \ -m mark ! --mark 0xffff -j DROP iptables -A PREROUTING -t mangle -i eth2 -m cluster \ --cluster-total-nodes 2 --cluster-local-node 1 \ --cluster-proc-name eth2 -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth2 \ -m mark ! --mark 0xffff -j DROPAnd the following commands to make all nodes see the same packets:
Quoted 7 lines
ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2arptables -I OUTPUT -o eth1 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:01 arptables -I INPUT -i eth1 --h-length 6 \ --destination-mac 01:00:5e:00:01:01 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 3 lines
Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.
Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.
Quoted 6 lines
Quoted 6 lines
arptables -I OUTPUT -o eth2 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:02 arptables -I INPUT -i eth2 --h-length 6 \ --destination-mac 01:00:5e:00:01:02 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 8 lines
Quoted 5 lines
In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.
echo 0 >
/proc/sys/net/netfilter/nf_conntrack_tcp_looseI'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?
If TCP pickup is enabled, one TCP ACK packet coming in the reply direction enters TCP ESTABLISHED state. Since that's a valid state-transition, the cluster match will consider that this is part of a connection that this node is handling since it's a valid state-transition. The cluster match does not mark packets that trigger invalid state transitions.
Quoted 3 lines
Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?
No idea who has done this, but I can send you a patch to fix this naming without breaking backward.
Quoted 9 lines
Quoted 9 lines
The match also provides a /proc entry under:
/proc/sys/net/netfilter/cluster/$PROC_NAMEwhere
PROC_NAMEis set via --cluster-proc-name. This is useful to include possible cluster reconfigurations via fail-over scripts. Assuming that this is the node 1, if node 2 is down, you can add node 2 to your node-mask as follows:
Quoted 4 lines
Quoted 1 line
echo +2 >
/proc/sys/net/netfilter/cluster/$PROC_NAMEDoes this provide anything you can't do by replacing the rule itself?
Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.
Now, I see that there is a possible optimization that consists of checking if one node has its node mask all set with regards to the total number of nodes, so that hashing can be skipped. But that's something that we can add later I think.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
Quoted 9 lines · Pablo Neira Ayuso
Quoted 8 lines · Patrick McHardy
Quoted 7 lines
ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2arptables -I OUTPUT -o eth1 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:01 arptables -I INPUT -i eth1 --h-length 6 \ --destination-mac 01:00:5e:00:01:01 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 5 lines
Quoted 3 lines
Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.
Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.
Great, I wasn't aware that this already existed in userspace :)
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.
echo 0 >
/proc/sys/net/netfilter/nf_conntrack_tcp_looseI'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?
If TCP pickup is enabled, one TCP ACK packet coming in the reply direction enters TCP ESTABLISHED state. Since that's a valid state-transition, the cluster match will consider that this is part of a connection that this node is handling since it's a valid state-transition. The cluster match does not mark packets that trigger invalid state transitions.
Why use conntrack at all? Shouldn't the cluster match simply filter out all packets not for this cluster and thats it? You stated it needs conntrack to get a constant tuple, but I don't see why the conntrack tuple would differ from the data that you can gather from the packet headers.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
echo +2 >
/proc/sys/net/netfilter/cluster/$PROC_NAMEDoes this provide anything you can't do by replacing the rule itself?
Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.
I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?
Quoted 1 line
Now, I see that there is a possible optimization that consists of checking if one node has its node mask all set with regards to the total number of nodes, so that hashing can be skipped. But that's something that we can add later I think.
Indeed.
Pablo Neira Ayuso
maintainerQuoted 10 lines · Patrick McHardy
Quoted 9 lines · Pablo Neira Ayuso
Quoted 8 lines · Patrick McHardy
Quoted 7 lines
ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2arptables -I OUTPUT -o eth1 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:01 arptables -I INPUT -i eth1 --h-length 6 \ --destination-mac 01:00:5e:00:01:01 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.
Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.
Great, I wasn't aware that this already existed in userspace :)
Yes, it's hosted by the ebtables projects. That tool really need some care. It works but I don't know if it's actively maintained. Probably we can offer hosting for it in git.netfilter.org. I'll investigate this.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
In the case of TCP connections, pickup facility has to be disabled to avoid marking TCP ACK packets coming in the reply direction as valid.
echo 0 >
/proc/sys/net/netfilter/nf_conntrack_tcp_looseI'm not sure I understand this. You *don't* want to mark them as valid, and you need to disable pickup for this?
Quoted 7 lines
Quoted 1 line
If TCP pickup is enabled, one TCP ACK packet coming in the reply direction enters TCP ESTABLISHED state. Since that's a valid state-transition, the cluster match will consider that this is part of a connection that this node is handling since it's a valid state-transition. The cluster match does not mark packets that trigger invalid state transitions.
Why use conntrack at all? Shouldn't the cluster match simply filter out all packets not for this cluster and thats it? You stated it needs conntrack to get a constant tuple, but I don't see why the conntrack tuple would differ from the data that you can gather from the packet headers.
No, source NAT connections would have different headers. A -> B for original, and B -> FW for reply direction. Thus, I cannot apply the same hashing for packets going in the original and the reply direction. Moreover, if this is packet-based, think also about possible asymmetric filtering: original traffic direction filtered by node 1 but reply traffic filtered by node 2. That will not work for a stateful firewall.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
echo +2 >
/proc/sys/net/netfilter/cluster/$PROC_NAMEDoes this provide anything you can't do by replacing the rule itself?
Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.
I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?
Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
Quoted 8 lines · Pablo Neira Ayuso
Quoted 5 lines · Patrick McHardy
Why use conntrack at all? Shouldn't the cluster match simply filter out all packets not for this cluster and thats it? You stated it needs conntrack to get a constant tuple, but I don't see why the conntrack tuple would differ from the data that you can gather from the packet headers.
No, source NAT connections would have different headers. A -> B for original, and B -> FW for reply direction. Thus, I cannot apply the same hashing for packets going in the original and the reply direction.
Ah I'm beginning to understand the topology I think :) Actually it seems its only combined SNAT+DNAT on one connections thats a problem, with only one of both you could tell the cluster match to look at either source or destination address (the unchanged one) in the opposite direction. Only if the opposite direction is completely unrelated from a non-conntrack view we can't deal with it. Anyways, your way to deal with this seems fine to me.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
echo +2 >
/proc/sys/net/netfilter/cluster/$PROC_NAMEDoes this provide anything you can't do by replacing the rule itself?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yes, the nodes in the cluster are identifies by an ID, the rule allows you to specify one ID. Say you have two cluster nodes, one with ID 1, and the other with ID 2. If the cluster node with ID 1 goes down, you can echo +1 to node with ID 2 so that it will handle packets going to node with ID 1 and ID 2. Of course, you need conntrackd to allow node ID 2 recover the filtering.
I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?
Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.
Thats not what I meant. By having a single node handle all connections from the one which went down, you have an imbalance in load distribution. The nodes are synchronized, so they could just all replace their cluster match with an updated number of nodes.
Pablo Neira Ayuso
maintainerQuoted 4 lines · Pablo Neira Ayuso
Quoted 3 lines · Patrick McHardy
I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?
Hm, I was not replying to your question ;). Right, the renumbering also requires getting the states back to the original node. We can use the same hashing approach in userspace to know which states belong to original node that has come back to life when it requests a resynchronization.
Quoted 1 line
Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.
This is not true. We may have something like this (assuming two nodes):
if no mark set and hash % 2 == 0, accept
if no mark set and hash % 2 == 1, accept
if no mark set, drop
So we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
Quoted 11 lines · Pablo Neira Ayuso
Quoted 4 lines · Pablo Neira Ayuso
Quoted 3 lines · Patrick McHardy
I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?
Hm, I was not replying to your question ;). Right, the renumbering also requires getting the states back to the original node. We can use the same hashing approach in userspace to know which states belong to original node that has come back to life when it requests a resynchronization.
Quoted 1 line
Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.
This is not true. We may have something like this (assuming two nodes):
To whom are you replying now? :)
Quoted 5 lines
if no mark set and hash % 2 == 0, accept
if no mark set and hash % 2 == 1, accept
if no mark set, dropSo we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.
I'm sure someone would argue that changing TCP port numbers of the tcp match through proc would be a nice and simple interface. The fact though is that we have an interface for handing a configuration to the kernel and this is clearly a configuration parameter. We're missing a proper way to use it in userspace from within programs (well, hopefully not for long anymore), but that needs to be fixed in userspace.
Pablo Neira Ayuso
maintainerQuoted 7 lines · Patrick McHardy
Quoted 6 lines · Pablo Neira Ayuso
Quoted 4 lines · Pablo Neira Ayuso
Quoted 3 lines · Patrick McHardy
I see. That kind of makes sense, but if you're running a synchronization daemon anyways, you might as well renumber all nodes so you still have proper balancing, right?
Quoted 7 lines
Quoted 5 lines
Hm, I was not replying to your question ;). Right, the renumbering also requires getting the states back to the original node. We can use the same hashing approach in userspace to know which states belong to original node that has come back to life when it requests a resynchronization.
Quoted 1 line
Indeed, the daemon may also add a new rule for the node that has gone down but that results in another extra hash operation to mark it or not (one extra hash per rule) :(.
This is not true. We may have something like this (assuming two nodes):
To whom are you replying now? :)
To myself, never mind :)
Quoted 6 lines
Quoted 5 lines
if no mark set and hash % 2 == 0, accept
if no mark set and hash % 2 == 1, accept
if no mark set, dropSo we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.
Quoted 7 lines
I'm sure someone would argue that changing TCP port numbers of the tcp match through proc would be a nice and simple interface. The fact though is that we have an interface for handing a configuration to the kernel and this is clearly a configuration parameter. We're missing a proper way to use it in userspace from within programs (well, hopefully not for long anymore), but that needs to be fixed in userspace.
Right right, I have no more arguments to support the /proc interface ;). I'll send you a patch without it late at night.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
Quoted 11 lines · Pablo Neira Ayuso
Quoted 11 lines
Quoted 10 lines
And the following commands to make all nodes see the same packets:
ip maddr add 01:00:5e:00:01:01 dev eth1
ip maddr add 01:00:5e:00:01:02 dev eth2arptables -I OUTPUT -o eth1 --h-length 6 \ -j mangle --mangle-mac-s 01:00:5e:00:01:01 arptables -I INPUT -i eth1 --h-length 6 \ --destination-mac 01:00:5e:00:01:01 \ -j mangle --mangle-mac-d 00:zz:yy:xx:5a:27
Quoted 5 lines
Quoted 3 lines
Mhh, is the saving of one or two characters really worth these deviations from the kind-of established naming scheme? Its hard to remember all these minor differences in my opinion.
Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"?
In case of --mangle-mac-d, getopt automatically handles abbreviations (the best example I can think of where this works is quilt ref --s --d for --sort --diffstat).
Quoted 5 lines
Quoted 3 lines
Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?
No idea who has done this,
Lowercasing targets was the original author's (Bart De Schuymer's) deed ;-)
I have nothing against lowercase targets, in fact, many a user seems to so not get case-sensitity right — “INPUT” vs. “input” (built-in chain names), “MASQUERADE” vs. “masquerade” (targets), to name some — that I would support lowercasing all targets...— if it were not for the clash between MARK (target) and mark (match), for example. Though, calling one of them “mark_m” or underscore-something to avoid the clash however, is what I dislike in favor of uppercasing target names.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Jan Engelhardt
Quoted 4 lines
Quoted 3 lines
Unrelated to this patch, but maybe the target would also be better named "NAT" instead of the much more generic term "mangle". Why is it using lower case letters btw?
No idea who has done this,
Lowercasing targets was the original author's (Bart De Schuymer's) deed ;-)
Quoted 8 lines
I have nothing against lowercase targets, in fact, many a user seems to so not get case-sensitity right — “INPUT” vs. “input” (built-in chain names), “MASQUERADE” vs. “masquerade” (targets), to name some — that I would support lowercasing all targets...— if it were not for the clash between MARK (target) and mark (match), for example. Though, calling one of them “mark_m” or underscore-something to avoid the clash however, is what I dislike in favor of uppercasing target names.
Its actually just exposing an implementation detail of how match and target modules are located on the filesystem. The meaning (match or target) can be deduced from "-j" and "-m".
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Pablo Neira Ayuso
Quoted 8 lines
Quoted 6 lines
Hm, you mean the name "mangle" or the name of the option "--mangle-mac-d"? This is what we currently have in kernel mainline and arptables userspace, it's not my fault :). I can send you a patch to fix it with a consistent naming without breaking backward compatibility both in kernel and user-space.
Great, I wasn't aware that this already existed in userspace :)
Yes, it's hosted by the ebtables projects. That tool really need some care.
It would indeed. The problem though is, that I, who originally wanted to unify arptables into iptables (as a start, because it has not diverged as much as ebtables), have thrown the towel on that on the grounds of libiptc being such a beast[1], and that developers' time would be better spent on (fasten seatbelts) duplicating xtables/iptables (once more) with the goal of creating an NFPROTO-agnostic table structure instead that obsoletes ip,ip6,arp and ebtables in one go.
Quoted 4 lines · Patrick McHardy
Quoted 4 lines
Lowercasing targets was the original author's (Bart De Schuymer's) deed ;-)
Quoted 9 lines
Quoted 4 lines
[...]
Though, calling one of them “mark_m” or underscore-something to avoid the clash however, is what I dislike in favor of uppercasing target names.Its actually just exposing an implementation detail of how match and target modules are located on the filesystem. The meaning (match or target) can be deduced from "-j" and "-m".
Too bad we cannot reasonably change it now.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Pablo Neira Ayuso
maintainerHi Patrick,
Quoted 11 lines · Patrick McHardy
Quoted 2 lines · Pablo Neira Ayuso
So we can still do this adding rules with the iptables interface. But still having the /proc looks like a simple interface for this.
I'm sure someone would argue that changing TCP port numbers of the tcp match through proc would be a nice and simple interface. The fact though is that we have an interface for handing a configuration to the kernel and this is clearly a configuration parameter. We're missing a proper way to use it in userspace from within programs (well, hopefully not for long anymore), but that needs to be fixed in userspace.
While reworking this, I think that I have found one argument to support the /proc interface that looks interesting in terms of resource consumption. Assume that we have three nodes, where two of them are down, thus, the only one active would have the following rule-set:
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
--cluster-total-nodes 3 --cluster-local-node 1 \
-j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
--cluster-total-nodes 3 --cluster-local-node 2 \
-j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 -m cluster \
--cluster-total-nodes 3 --cluster-local-node 3 \
-j MARK --set-mark 0xffff
iptables -A PREROUTING -t mangle -i eth0 \
-m mark ! --mark 0xffff -j DROPLook at the worst case: if the packet goes to node 3, the hashing must be done to check if the packet belongs to node 1 and node 2. Thus, the hashing is done three times. This makes the cluster hashing O(n) where n is the number of cluster nodes.
A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
Quoted 22 lines · Pablo Neira Ayuso
While reworking this, I think that I have found one argument to support the /proc interface that looks interesting in terms of resource consumption. Assume that we have three nodes, where two of them are down, thus, the only one active would have the following rule-set:
iptables -A PREROUTING -t mangle -i eth0 -m cluster \ --cluster-total-nodes 3 --cluster-local-node 1 \ -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth0 -m cluster \ --cluster-total-nodes 3 --cluster-local-node 2 \ -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth0 -m cluster \ --cluster-total-nodes 3 --cluster-local-node 3 \ -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth0 \ -m mark ! --mark 0xffff -j DROPLook at the worst case: if the packet goes to node 3, the hashing must be done to check if the packet belongs to node 1 and node 2. Thus, the hashing is done three times. This makes the cluster hashing O(n) where n is the number of cluster nodes.
Quoted 5 lines
A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.
That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.
Pablo Neira Ayuso
maintainerQuoted 29 lines · Patrick McHardy
Quoted 22 lines · Pablo Neira Ayuso
While reworking this, I think that I have found one argument to support the /proc interface that looks interesting in terms of resource consumption. Assume that we have three nodes, where two of them are down, thus, the only one active would have the following rule-set:
iptables -A PREROUTING -t mangle -i eth0 -m cluster \ --cluster-total-nodes 3 --cluster-local-node 1 \ -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth0 -m cluster \ --cluster-total-nodes 3 --cluster-local-node 2 \ -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth0 -m cluster \ --cluster-total-nodes 3 --cluster-local-node 3 \ -j MARK --set-mark 0xffff iptables -A PREROUTING -t mangle -i eth0 \ -m mark ! --mark 0xffff -j DROPLook at the worst case: if the packet goes to node 3, the hashing must be done to check if the packet belongs to node 1 and node 2. Thus, the hashing is done three times. This makes the cluster hashing O(n) where n is the number of cluster nodes.
Quoted 5 lines
A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.
Quoted 3 lines
That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.
Indeed, but in practise existing failover daemons (at least those free/opensource that I know) doesn't show that "intelligent" behaviour since they initially (according to the configuration file) assign the resources to each node, and if one node fails, it assigns the corresponding resources to another sane node (ie. the daemon runs a script with the corresponding iptables rules).
Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.
Therefore AFAICS, without the /proc interface, I would need one iptables rule per cluster-local-node handled, and so it's still the possible sub-optimal situation when one or several node fails.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
OoO En ce début d'après-midi nuageux du mardi 17 février 2009, vers 14:50, Pablo Neira Ayuso <[email protected]> disait :
Quoted 7 lines
Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.
Moreover, some of them (the one that are using VRRP for example) don't report the total number of nodes still alive. As a user, I would prefer a simple /proc interface to add/remove a node.
Signature · 6 lines
Make sure all variables are initialised before use.
- The Elements of Programming Style (Kernighan & Plauger)
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 25 lines · Pablo Neira Ayuso
Quoted 8 lines · Patrick McHardy
Quoted 5 lines
A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result of the hash in the skbuff mark, but the problem is that it would require a reserved space for hashmarks since they may clash with other user-defined marks.
That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.
Indeed, but in practise existing failover daemons (at least those free/opensource that I know) doesn't show that "intelligent" behaviour since they initially (according to the configuration file) assign the resources to each node, and if one node fails, it assigns the corresponding resources to another sane node (ie. the daemon runs a script with the corresponding iptables rules).
Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.
Quoted 3 lines
Therefore AFAICS, without the /proc interface, I would need one iptables rule per cluster-local-node handled, and so it's still the possible sub-optimal situation when one or several node fails.
OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.
BTW, I recently looked into TIPC, its incredibly easy to use since it deals with dead-node dectection etc internally and all you need to do is exchange a few messages. Might be quite easy to write a smarter failover daemon.
Quoted 11 lines · Vincent Bernat
OoO En ce début d'après-midi nuageux du mardi 17 février 2009, vers 14:50, Pablo Neira Ayuso <[email protected]> disait :
Quoted 7 lines
Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.
Quoted 3 lines
Moreover, some of them (the one that are using VRRP for example) don't report the total number of nodes still alive. As a user, I would prefer a simple /proc interface to add/remove a node.
That "simple" argument really doesn't cut it, there's nothing inherently more complicated in executing an iptables command compared to executing an echo command. Most likely some program is going to do it anyways. --
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Pablo Neira Ayuso
maintainerQuoted 28 lines · Patrick McHardy
Quoted 27 lines · Pablo Neira Ayuso
Quoted 10 lines · Patrick McHardy
Quoted 7 lines
A possible solution (that thinking it well, I don't like too much yet) would be to convert this to a HASHMARK target that will store the result
of the hash in the skbuff mark, but the problem is that it would require
a reserved space for hashmarks since they may clash with other user-defined marks.That sounds a bit like a premature optimization. What I don't get is why you don't simply set cluster-total-nodes to one when two are down or remove the rule entirely.
Indeed, but in practise existing failover daemons (at least those free/opensource that I know) doesn't show that "intelligent" behaviour since they initially (according to the configuration file) assign the resources to each node, and if one node fails, it assigns the corresponding resources to another sane node (ie. the daemon runs a script with the corresponding iptables rules).
Re-adjusting cluster-total-nodes and cluster-local-nodes options (eg. if one cluster node goes down and there are only two nodes alive, change the rule-set to have only two nodes) seems indeed the natural way to go since the alive cluster nodes would share the workload that the failing node has left. However, as said, existing failover daemons only select one new master to recover what a failing node was doing, thus, only one runs the script to inject the states into the kernel.
Quoted 7 lines
Quoted 3 lines
Therefore AFAICS, without the /proc interface, I would need one iptables rule per cluster-local-node handled, and so it's still the possible sub-optimal situation when one or several node fails.
OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.
:(
Quoted 4 lines
BTW, I recently looked into TIPC, its incredibly easy to use since it deals with dead-node dectection etc internally and all you need to do is exchange a few messages. Might be quite easy to write a smarter failover daemon.
I see, I don't have more convincing arguments that "I would also need time for that but in the meanwhile, please allow this". Well, failover daemons are delicate pieces of software, they have to be stable, well-tested, bug-free, give timely responses. Still TIPC is experimental and I guess that the dead-node detection is only layer 3/4 based on heartbeats. Dead-node detection is a tricky issue, the more you can perform different layer checkings, the more increase chances to make wrong decisions that may lead to inconsistent situations and tons of problems. VRRP is the current standard and this one of his limitations, and so on.
Well, if you are not going to accept the /proc interface, not matter what I can argument, I give up on this ;)
Anyway, probably, this is a premature optimization (but worth?). Some numbers, in my testbed, I get ~1800 TCP connections per second less with eight cluster rules (no /proc interface).
24347 TCP connections per second with one rule. 22580 TCP connections per second with eight rules.
OK, I'll send you another patch without the /proc interface.
Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
[Please trim unrelated content, these mails are getting hard to read]
Quoted 17 lines · Pablo Neira Ayuso
Quoted 4 lines · Patrick McHardy
BTW, I recently looked into TIPC, its incredibly easy to use since it deals with dead-node dectection etc internally and all you need to do is exchange a few messages. Might be quite easy to write a smarter failover daemon.
I see, I don't have more convincing arguments that "I would also need time for that but in the meanwhile, please allow this". Well, failover daemons are delicate pieces of software, they have to be stable, well-tested, bug-free, give timely responses. Still TIPC is experimental and I guess that the dead-node detection is only layer 3/4 based on heartbeats. Dead-node detection is a tricky issue, the more you can perform different layer checkings, the more increase chances to make wrong decisions that may lead to inconsistent situations and tons of problems. VRRP is the current standard and this one of his limitations, and so on.
Quoted 2 lines
Well, if you are not going to accept the /proc interface, not matter what I can argument, I give up on this ;)
I'm afraid I can't be convinced of this. If you want to specify multiple node ids, have the iptables command accept them, but there's no reason to use proc for this.
Quoted 8 lines
Anyway, probably, this is a premature optimization (but worth?). Some numbers, in my testbed, I get ~1800 TCP connections per second less with eight cluster rules (no /proc interface).
24347 TCP connections per second with one rule. 22580 TCP connections per second with eight rules.
OK, I'll send you another patch without the /proc interface.
Thanks. As I said, I don't have anything against handling multiple nodes in one rule, as long as its not done using proc.
OoO Pendant le temps de midi du mercredi 18 février 2009, vers 12:06, Pablo Neira Ayuso <[email protected]> disait :
Quoted 5 lines
Quoted 3 lines
OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.
:(
A solution would be to give the mask on the command line. The HA daemon will output the assigned nodes to a file and the content of this file will be computed into a mask.
Signature · 8 lines
THE BOYS ROOM IS NOT A WATER PARK THE BOYS ROOM IS NOT A WATER PARK THE BOYS ROOM IS NOT A WATER PARK -+- Bart Simpson on chalkboard in episode 3F03 -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Vincent Bernat
OoO Pendant le temps de midi du mercredi 18 février 2009, vers 12:06, Pablo Neira Ayuso <[email protected]> disait :
Quoted 3 lines
Quoted 3 lines
OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.
Quoted 5 lines
Quoted 1 line
:(
A solution would be to give the mask on the command line. The HA daemon will output the assigned nodes to a file and the content of this file will be computed into a mask.
Indeed. Just a bitmask (is 64 nodes max enough?) should be fine. --
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Pablo Neira Ayuso
maintainerQuoted 8 lines · Patrick McHardy
Quoted 7 lines · Vincent Bernat
OoO Pendant le temps de midi du mercredi 18 février 2009, vers 12:06, Pablo Neira Ayuso <[email protected]> disait :
Quoted 3 lines
Quoted 3 lines
OK, that explains why you want to handle it this way. I don't want to merge the proc file part though, so until the daemons get smarter, people will have to use multiple rules.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
:(
A solution would be to give the mask on the command line. The HA daemon will output the assigned nodes to a file and the content of this file will be computed into a mask.
Indeed. Just a bitmask (is 64 nodes max enough?) should be fine.
Good point :), I'll add this to the patch that I'm reworking.
Signature · 5 lines
"Los honestos son inadaptados sociales" -- Les Luthiers -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html