Skip to contentSkip to conversation

xtables: Add locking to prevent concurrent instances

PATCH Superseded Phil Oester 2 participants 4 messages May 22, 2013 netfilter

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xtables: Add locking to prevent concurrent instances

There have been numerous complaints and bug reports over the years when admins attempt to run more than one instance of iptables simultaneously. Currently open bug reports which are related:

325: Parallel execution of the iptables is impossible 758: Retry iptables command on transient failure 764: Doing -Z twice in parallel breaks counters 822: iptables shows negative or other bad packet/byte counts

As Patrick notes in 325: "Since this has been a problem people keep running into, I'd suggest to simply add some locking to iptables to catch the most common case."

I started looking into alternatives to add locking, and of course the most common/obvious solution is to use a pidfile. But this has various downsides, such as if the application is terminated abnormally and the pidfile isn't cleaned up. And this also requires a writable filesystem. Using a UNIX domain socket file (e.g. in /var/run) has similar issues.

Starting in 2.2, Linux added support for abstract sockets. These sockets require no filesystem, and automatically disappear once the application terminates. This is the locking solution I chose to implement in xtables-multi. As an added bonus, since each network namespace has its own socket pool, an iptables instance running in one namespace will not lock out an iptables instance running in another namespace. A filesystem approach would have to recognize and handle multiple network namespaces.

As long as I was adding locking, I also chose to add a retry loop, with 3 attempts made to grab the lock before giving up.

Phil

iptables/xtables-multi.c +29 -1
index 8014d5f..9c22a82 100644
@@ -1,8 +1,12 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <sys/socket.h>
+#include <sys/un.h>
+#include <unistd.h>
 #include "xshared.h"
 
+#include "xtables.h"
 #include "xtables-multi.h"
 
 #ifdef ENABLE_IPV4
@@ -35,7 +39,31 @@ static const struct subcommand multi_subcommands[] = {
 	{NULL},
 };
 
+#define XTMSOCKET_NAME "xtables_multi"
+#define XTMSOCKET_LEN 14
+
 int main(int argc, char **argv)
 {
-	return subcmd_main(argc, argv, multi_subcommands);
+	int i = 0, ret, xtm_socket;
+	struct sockaddr_un xtm_addr;
+
+	memset(&xtm_addr, 0, sizeof(xtm_addr));
+	xtm_addr.sun_family = AF_UNIX;
+	strcpy(xtm_addr.sun_path+1, XTMSOCKET_NAME);
+	xtm_socket = socket(AF_UNIX, SOCK_STREAM, 0);
+	/* If we can't even create a socket, just revert to prior (lockless) behavior */
+	if (xtm_socket < 0)
+		return subcmd_main(argc, argv, multi_subcommands);
+
+	do {
+		ret = bind(xtm_socket, (struct sockaddr*)&xtm_addr,
+			   offsetof(struct sockaddr_un, sun_path)+XTMSOCKET_LEN);
+		if (ret == 0)
+			return subcmd_main(argc, argv, multi_subcommands);
+		i++;
+		sleep(1);
+	} while (i <= 2);
+
+	fprintf(stderr, "ERROR: unable to obtain lock - another instance is already running\n");
+	exit(RESOURCE_PROBLEM);
 }
#1 lore raw
Quoted 40 lines · Phil Oester

There have been numerous complaints and bug reports over the years when admins
attempt to run more than one instance of iptables simultaneously. Currently
open bug reports which are related:

325: Parallel execution of the iptables is impossible 758: Retry iptables command on transient failure 764: Doing -Z twice in parallel breaks counters 822: iptables shows negative or other bad packet/byte counts

As Patrick notes in 325: "Since this has been a problem people keep running
into, I'd suggest to simply add some locking to iptables to catch the most
common case."

I started looking into alternatives to add locking, and of course the most
common/obvious solution is to use a pidfile. But this has various downsides,
such as if the application is terminated abnormally and the pidfile isn't
cleaned up. And this also requires a writable filesystem. Using a UNIX domain
socket file (e.g. in /var/run) has similar issues.

Starting in 2.2, Linux added support for abstract sockets. These sockets
require no filesystem, and automatically disappear once the application terminates. This is the locking solution I chose to implement in xtables-multi.
As an added bonus, since each network namespace has its own socket pool, an
iptables instance running in one namespace will not lock out an iptables
instance running in another namespace. A filesystem approach would have
to recognize and handle multiple network namespaces.

Quoted 3 lines

As long as I was adding locking, I also chose to add a retry loop, with 3
attempts made to grab the lock before giving up.

If I'm not mistaken, we retry failed operations since a few years now, so is this actually still a problem?

If so, why abort after three tries instead of waiting until the lock can be acquired?

#2 lore raw
Quoted 3 lines · Patrick McHardy

If I'm not mistaken, we retry failed operations since a few years now, so is this actually still a problem?

If so, why abort after three tries instead of waiting until the lock can be acquired?

Not sure what you are referring to with retrying operations. The race condition between multiple simultaneous instances of iptables running still exists in latest git. For example, fire up a few simultanous instances of this:

# while : ;  do iptables -A INPUT -j ACCEPT ; iptables -D INPUT -j ACCEPT ; done
iptables: Resource temporarily unavailable.
iptables: Resource temporarily unavailable.
iptables: Invalid argument. Run `dmesg' for more information.
iptables: Resource temporarily unavailable.
iptables: Resource temporarily unavailable.

Bug 764 is another race problem where two simultaneous "iptables -Z" causes counters to get reset to invalid numbers (and is reproducible with current git).

As to your second question re: why only three tries (in 3 seconds): I suppose we could just wait "forever". In most normal cases I wouldn't expect a wait > 1 second for an instance to complete. But there could be pathological corner cases such as dumping a huge ruleset with DNS resolution (i.e. without -n) which could take an eternity to complete. Should we spit out (to stderr) some kind of message in these cases, or just sit silently? Perhaps every 5 seconds?

Phil

#3 lore raw
Quoted 21 lines · Phil Oester
Quoted 1 line · Patrick McHardy

If I'm not mistaken, we retry failed operations since a few years

now, so is this actually still a problem?

Quoted 2 lines

If so, why abort after three tries instead of waiting until the lock

can be acquired?

Not sure what you are referring to with retrying operations. The race condition
between multiple simultaneous instances of iptables running still exists in
latest git. For example, fire up a few simultanous instances of this:

# while : ;  do iptables -A INPUT -j ACCEPT ; iptables -D INPUT -j
ACCEPT ; done
iptables: Resource temporarily unavailable.
iptables: Resource temporarily unavailable.
iptables: Invalid argument. Run `dmesg' for more information.
iptables: Resource temporarily unavailable.
iptables: Resource temporarily unavailable.
Quoted 4 lines

Bug 764 is another race problem where two simultaneous "iptables -Z" causes
counters to get reset to invalid numbers (and is reproducible with current git).

I see, thanks.

Quoted 11 lines

As to your second question re: why only three tries (in 3 seconds): I suppose we
could just wait "forever". In most normal cases I wouldn't expect a wait > 1
second for an instance to complete. But there could be pathological corner cases
such as dumping a huge ruleset with DNS resolution (i.e. without -n) which could
take an eternity to complete. Should we spit out (to stderr) some kind of
message in these cases, or just sit silently? Perhaps every 5 seconds?

Printing a message sounds fine. But I think the main point of using locking is to provide a guarantee that a correct operation won't randomly fail. So I think we should retry forever.