Skip to contentSkip to conversation

iptables: introduce cache evaluation phase

PATCH New Pablo Neira Ayuso 1 participant 8 messages Jan 6, 2020 netfilter

lore
Share & tools
lore

Pablo Neira Ayuso

author maintainer
#0 lore raw

cover iptables: introduce cache evaluation phase

Hi Phil,

Happy new year.

This patchset introduces a new step to parse rules into a list of commands:

input -> parser -> list of commands -> list of jobs -> netlink -> kernel

This allows us to estimate the cache requirements from the list of commands and to avoid in-transit cache cancelations while handling a batch.

This batch is likely disabling your optimizations where selective chain/set netlink dump speeds up things. From reading the code, I'm not sure what paths are specifically benefiting from this optimization since any of the existing nftnl_chain_list_get() and nftnl_set_list_get() calls might exercise this selective netlink dump. I think it should not be too hard to restore those, I remember you have a few tests to evaluate the speed up. Note that this patch also disables the pre-parsing from xtables-restore.

Downside is that there's an extra memory allocation, that could be consolidated in the future, by having one single list of commands/jobs. But would probably make this batch even larger, but I tried to reduce complexity as much as possible. Debugging cache issue is hard in my opinion, if we can avoid fetch/cancel/fetch cache scenario, the better.

The approach that 'among' follows to use the set infrastructure is interesting, probably there's a way to simplify this, but I understand there might be more urgent stuff to have a look.

Let me know,
Thanks.

P.S: I started this patchset by Dec 25 while traveling, tests look
OK here, I'm just releasing this because I'm not sure I can come back to this anytime soon.

Pablo Neira Ayuso (7):
nft: do not check for existing chain from parser
nft: split parsing from netlink commands
nft: calculate cache requirements from list of commands
nft: restore among support
nft: remove cache build calls
nft: skip table list release if uninitialized
nft: missing nft_fini() call in bridge family

 iptables/Makefile.am                               |   2 +-
 iptables/nft-arp.c                                 |   5 +-
 iptables/nft-bridge.c                              |  18 +-
 iptables/nft-cache.c                               |  72 ++--
 iptables/nft-cache.h                               |   1 +
 iptables/nft-cmd.c                                 | 389 +++++++++++++++++++++
 iptables/nft-cmd.h                                 |  79 +++++
 iptables/nft-shared.c                              |   6 +-
 iptables/nft-shared.h                              |   4 +-
 iptables/nft.c                                     | 310 +++++++++++-----
 iptables/nft.h                                     |  49 ++-
 .../shell/testcases/ip6tables/0004-return-codes_0  |   2 +-
 .../shell/testcases/iptables/0004-return-codes_0   |   2 +-
 iptables/xtables-arp.c                             |  26 +-
 iptables/xtables-eb-standalone.c                   |   2 +
 iptables/xtables-eb.c                              |  26 +-
 iptables/xtables-restore.c                         |  35 +-
 iptables/xtables-save.c                            |   3 +
 iptables/xtables.c                                 |  57 ++-
 19 files changed, 863 insertions(+), 225 deletions(-)
 create mode 100644 iptables/nft-cmd.c
 create mode 100644 iptables/nft-cmd.h
Signature · 1 line
2.11.0
author maintainer
#1 lore raw

PATCH 1/7 nft: do not check for existing chain from parser

Follow up patches split the parser from the cache calculation.

---
 iptables/tests/shell/testcases/ip6tables/0004-return-codes_0 | 2 +-
 iptables/tests/shell/testcases/iptables/0004-return-codes_0  | 2 +-
 iptables/xtables.c                                           | 5 -----
 3 files changed, 2 insertions(+), 7 deletions(-)
iptables/tests/shell/testcases/ip6tables/0004-return-codes_0 +1 -1
index f023b7915498..3124cfd86317 100755
@@ -31,7 +31,7 @@ cmd 1 ip6tables -A noexist -j ACCEPT
 cmd 0 ip6tables -C INPUT -j ACCEPT
 cmd 1 ip6tables -C FORWARD -j ACCEPT
 cmd 1 ip6tables -C nonexist -j ACCEPT
-cmd 2 ip6tables -C INPUT -j foobar
+cmd 1 ip6tables -C INPUT -j foobar
 cmd 2 ip6tables -C INPUT -m foobar -j ACCEPT
 cmd 3 ip6tables -t foobar -C INPUT -j ACCEPT
 
iptables/tests/shell/testcases/iptables/0004-return-codes_0 +1 -1
index ce02e0bcb128..136eab83a679 100755
@@ -75,7 +75,7 @@ cmd 2 "$ENOMTH" iptables -C INPUT -m foobar -j ACCEPT
 # messages of those don't match, but iptables-nft ones are actually nicer.
 #cmd 2 "$ENOTGT" iptables -C INPUT -j foobar
 #cmd 3 "$ENOTBL" iptables -t foobar -C INPUT -j ACCEPT
-cmd 2 "" iptables -C INPUT -j foobar
+cmd 1 "" iptables -C INPUT -j foobar
 cmd 3 "" iptables -t foobar -C INPUT -j ACCEPT
 
 exit $global_rc
iptables/xtables.c +0 -5
index 8f9dc628d002..260fb97b3b11 100644
@@ -1031,11 +1031,6 @@ void do_parse(struct nft_handle *h, int argc, char *argv[],
 					   opt2char(OPT_VIANAMEIN),
 					   p->chain);
 		}
-
-		if (!p->xlate && !cs->target && strlen(cs->jumpto) > 0 &&
-		    !nft_chain_exists(h, p->table, cs->jumpto))
-			xtables_error(PARAMETER_PROBLEM,
-				      "Chain '%s' does not exist", cs->jumpto);
 	}
 }
 
Signature · 1 line
2.11.0
author maintainer
#2 lore raw

PATCH 2/7 nft: split parsing from netlink commands

In reply to Pablo Neira Ayuso

This patch updates the parser to generate a list of command objects. This list of commands is then transformed to a list of netlink jobs. This new command object stores the rule using the nftnl representation via nft_rule_new().

To reduce the number of updates in this patch, the nft_*_rule_find() functions have been updated to restore the native representation to skip the update of the rule comparison code.

---
 iptables/Makefile.am       |   2 +-
 iptables/nft-arp.c         |   5 +-
 iptables/nft-bridge.c      |   5 +-
 iptables/nft-cmd.c         | 315 +++++++++++++++++++++++++++++++++++++++++++++
 iptables/nft-cmd.h         |  78 +++++++++++
 iptables/nft-shared.c      |   6 +-
 iptables/nft-shared.h      |   4 +-
 iptables/nft.c             | 259 +++++++++++++++++++++++++++----------
 iptables/nft.h             |  41 +++++-
 iptables/xtables-arp.c     |  26 ++--
 iptables/xtables-eb.c      |  26 ++--
 iptables/xtables-restore.c |  32 ++---
 iptables/xtables.c         |  52 ++++----
 13 files changed, 702 insertions(+), 149 deletions(-)
 create mode 100644 iptables/nft-cmd.c
 create mode 100644 iptables/nft-cmd.h
iptables/Makefile.am +1 -1
index 71b1b1d48d4b..dc66b3cc09c0 100644
@@ -38,7 +38,7 @@ xtables_nft_multi_SOURCES += xtables-save.c xtables-restore.c \
 				nft-shared.c nft-ipv4.c nft-ipv6.c nft-arp.c \
 				xtables-monitor.c nft-cache.c \
 				xtables-arp-standalone.c xtables-arp.c \
-				nft-bridge.c \
+				nft-bridge.c nft-cmd.c \
 				xtables-eb-standalone.c xtables-eb.c \
 				xtables-eb-translate.c \
 				xtables-translate.c
iptables/nft-arp.c +3 -2
index d4a86610ec21..748784bc4904 100644
@@ -634,14 +634,15 @@ static bool nft_arp_is_same(const void *data_a,
 }
 
 static bool nft_arp_rule_find(struct nft_handle *h, struct nftnl_rule *r,
-			      void *data)
+			      struct nftnl_rule *rule)
 {
-	const struct iptables_command_state *cs = data;
+	struct iptables_command_state _cs = {}, *cs = &_cs;
 	struct iptables_command_state this = {};
 	bool ret = false;
 
 	/* Delete by matching rule case */
 	nft_rule_to_iptables_command_state(h, r, &this);
+	nft_rule_to_iptables_command_state(h, rule, cs);
 
 	if (!nft_arp_is_same(&cs->arp, &this.arp))
 		goto out;
iptables/nft-bridge.c +3 -2
index 3f85cbbf5e4c..a5aaa3f87187 100644
@@ -748,13 +748,14 @@ static bool nft_bridge_is_same(const void *data_a, const void *data_b)
 }
 
 static bool nft_bridge_rule_find(struct nft_handle *h, struct nftnl_rule *r,
-				 void *data)
+				 struct nftnl_rule *rule)
 {
-	struct iptables_command_state *cs = data;
+	struct iptables_command_state _cs = {}, *cs = &_cs;
 	struct iptables_command_state this = {};
 	bool ret = false;
 
 	nft_rule_to_ebtables_command_state(h, r, &this);
+	nft_rule_to_ebtables_command_state(h, rule, cs);
 
 	DEBUGP("comparing with... ");
 
iptables/nft-cmd.c new file +315 -0

315 lines not on this page · raw message

iptables/nft-cmd.h new file +78 -0

78 lines not on this page · raw message

iptables/nft-shared.c +4 -2
index 426765641cff..0a2a53f8198a 100644
@@ -989,12 +989,14 @@ void nft_ipv46_parse_target(struct xtables_target *t, void *data)
 	cs->target = t;
 }
 
-bool nft_ipv46_rule_find(struct nft_handle *h, struct nftnl_rule *r, void *data)
+bool nft_ipv46_rule_find(struct nft_handle *h, struct nftnl_rule *r,
+			 struct nftnl_rule *rule)
 {
-	struct iptables_command_state *cs = data, this = {};
+	struct iptables_command_state _cs = {}, this = {}, *cs = &_cs;
 	bool ret = false;
 
 	nft_rule_to_iptables_command_state(h, r, &this);
+	nft_rule_to_iptables_command_state(h, rule, cs);
 
 	DEBUGP("comparing with... ");
 #ifdef DEBUG_DEL
iptables/nft-shared.h +2 -2
index bee99a7dd0c9..89e9d0b9be33 100644
@@ -110,7 +110,7 @@ struct nft_family_ops {
 			   struct iptables_command_state *cs);
 	void (*clear_cs)(struct iptables_command_state *cs);
 	bool (*rule_find)(struct nft_handle *h, struct nftnl_rule *r,
-			  void *data);
+			  struct nftnl_rule *rule);
 	int (*xlate)(const void *data, struct xt_xlate *xl);
 };
 
@@ -172,7 +172,7 @@ struct nft_family_ops *nft_family_ops_lookup(int family);
 
 void nft_ipv46_parse_target(struct xtables_target *t, void *data);
 bool nft_ipv46_rule_find(struct nft_handle *h, struct nftnl_rule *r,
-			 void *data);
+			 struct nftnl_rule *rule);
 
 bool compare_matches(struct xtables_rule_match *mt1, struct xtables_rule_match *mt2);
 bool compare_targets(struct xtables_target *tg1, struct xtables_target *tg2);
iptables/nft.c +193 -66

485 lines not on this page · raw message

iptables/nft.h +35 -6

72 lines not on this page · raw message

iptables/xtables-arp.c +13 -13
index 9cfad76263d3..477f41746b5b 100644
@@ -400,7 +400,7 @@ list_entries(struct nft_handle *h, const char *chain, const char *table,
 	if (linenumbers)
 		format |= FMT_LINENUMBERS;
 
-	return nft_rule_list(h, chain, table, rulenum, format);
+	return nft_cmd_rule_list(h, chain, table, rulenum, format);
 }
 
 static int
@@ -427,10 +427,10 @@ append_entry(struct nft_handle *h,
 			cs->arp.arp.tgt.s_addr = daddrs[j].s_addr;
 			cs->arp.arp.tmsk.s_addr = dmasks[j].s_addr;
 			if (append) {
-				ret = nft_rule_append(h, chain, table, cs, NULL,
+				ret = nft_cmd_rule_append(h, chain, table, cs, NULL,
 						      verbose);
 			} else {
-				ret = nft_rule_insert(h, chain, table, cs,
+				ret = nft_cmd_rule_insert(h, chain, table, cs,
 						      rulenum, verbose);
 			}
 		}
@@ -455,7 +455,7 @@ replace_entry(const char *chain,
 	cs->arp.arp.smsk.s_addr = smask->s_addr;
 	cs->arp.arp.tmsk.s_addr = dmask->s_addr;
 
-	return nft_rule_replace(h, chain, table, cs, rulenum, verbose);
+	return nft_cmd_rule_replace(h, chain, table, cs, rulenum, verbose);
 }
 
 static int
@@ -479,7 +479,7 @@ delete_entry(const char *chain,
 		for (j = 0; j < ndaddrs; j++) {
 			cs->arp.arp.tgt.s_addr = daddrs[j].s_addr;
 			cs->arp.arp.tmsk.s_addr = dmasks[j].s_addr;
-			ret = nft_rule_delete(h, chain, table, cs, verbose);
+			ret = nft_cmd_rule_delete(h, chain, table, cs, verbose);
 		}
 	}
 
@@ -962,7 +962,7 @@ int do_commandarp(struct nft_handle *h, int argc, char *argv[], char **table,
 				   options&OPT_VERBOSE, h);
 		break;
 	case CMD_DELETE_NUM:
-		ret = nft_rule_delete_num(h, chain, *table, rulenum - 1, verbose);
+		ret = nft_cmd_rule_delete_num(h, chain, *table, rulenum - 1, verbose);
 		break;
 	case CMD_REPLACE:
 		ret = replace_entry(chain, *table, &cs, rulenum - 1,
@@ -984,10 +984,10 @@ int do_commandarp(struct nft_handle *h, int argc, char *argv[], char **table,
 				   options&OPT_LINENUMBERS);
 		break;
 	case CMD_FLUSH:
-		ret = nft_rule_flush(h, chain, *table, options & OPT_VERBOSE);
+		ret = nft_cmd_rule_flush(h, chain, *table, options & OPT_VERBOSE);
 		break;
 	case CMD_ZERO:
-		ret = nft_chain_zero_counters(h, chain, *table,
+		ret = nft_cmd_chain_zero_counters(h, chain, *table,
 					      options & OPT_VERBOSE);
 		break;
 	case CMD_LIST|CMD_ZERO:
@@ -997,21 +997,21 @@ int do_commandarp(struct nft_handle *h, int argc, char *argv[], char **table,
 				   /*options&OPT_EXPANDED*/0,
 				   options&OPT_LINENUMBERS);
 		if (ret)
-			ret = nft_chain_zero_counters(h, chain, *table,
+			ret = nft_cmd_chain_zero_counters(h, chain, *table,
 						      options & OPT_VERBOSE);
 		break;
 	case CMD_NEW_CHAIN:
-		ret = nft_chain_user_add(h, chain, *table);
+		ret = nft_cmd_chain_user_add(h, chain, *table);
 		break;
 	case CMD_DELETE_CHAIN:
-		ret = nft_chain_user_del(h, chain, *table,
+		ret = nft_cmd_chain_user_del(h, chain, *table,
 					 options & OPT_VERBOSE);
 		break;
 	case CMD_RENAME_CHAIN:
-		ret = nft_chain_user_rename(h, chain, *table, newname);
+		ret = nft_cmd_chain_user_rename(h, chain, *table, newname);
 		break;
 	case CMD_SET_POLICY:
-		ret = nft_chain_set(h, *table, chain, policy, NULL);
+		ret = nft_cmd_chain_set(h, *table, chain, policy, NULL);
 		if (ret < 0)
 			xtables_error(PARAMETER_PROBLEM, "Wrong policy `%s'\n",
 				      policy);
iptables/xtables-eb.c +13 -13
index 15b971da3d42..0b002e4d5afb 100644
@@ -150,9 +150,9 @@ append_entry(struct nft_handle *h,
 	int ret = 1;
 
 	if (append)
-		ret = nft_rule_append(h, chain, table, cs, NULL, verbose);
+		ret = nft_cmd_rule_append(h, chain, table, cs, NULL, verbose);
 	else
-		ret = nft_rule_insert(h, chain, table, cs, rule_nr, verbose);
+		ret = nft_cmd_rule_insert(h, chain, table, cs, rule_nr, verbose);
 
 	return ret;
 }
@@ -169,10 +169,10 @@ delete_entry(struct nft_handle *h,
 	int ret = 1;
 
 	if (rule_nr == -1)
-		ret = nft_rule_delete(h, chain, table, cs, verbose);
+		ret = nft_cmd_rule_delete(h, chain, table, cs, verbose);
 	else {
 		do {
-			ret = nft_rule_delete_num(h, chain, table,
+			ret = nft_cmd_rule_delete_num(h, chain, table,
 						  rule_nr, verbose);
 			rule_nr++;
 		} while (rule_nr < rule_nr_end);
@@ -427,7 +427,7 @@ static int list_rules(struct nft_handle *h, const char *chain, const char *table
 	if (!counters)
 		format |= FMT_NOCOUNTS;
 
-	return nft_rule_list(h, chain, table, rule_nr, format);
+	return nft_cmd_rule_list(h, chain, table, rule_nr, format);
 }
 
 static int parse_rule_range(const char *argv, int *rule_nr, int *rule_nr_end)
@@ -820,7 +820,7 @@ int do_commandeb(struct nft_handle *h, int argc, char *argv[], char **table,
 			flags |= OPT_COMMAND;
 
 			if (c == 'N') {
-				ret = nft_chain_user_add(h, chain, *table);
+				ret = nft_cmd_chain_user_add(h, chain, *table);
 				break;
 			} else if (c == 'X') {
 				/* X arg is optional, optarg is NULL */
@@ -828,7 +828,7 @@ int do_commandeb(struct nft_handle *h, int argc, char *argv[], char **table,
 					chain = argv[optind];
 					optind++;
 				}
-				ret = nft_chain_user_del(h, chain, *table, 0);
+				ret = nft_cmd_chain_user_del(h, chain, *table, 0);
 				break;
 			}
 
@@ -842,7 +842,7 @@ int do_commandeb(struct nft_handle *h, int argc, char *argv[], char **table,
 				else if (strchr(argv[optind], ' ') != NULL)
 					xtables_error(PARAMETER_PROBLEM, "Use of ' ' not allowed in chain names");
 
-				ret = nft_chain_user_rename(h, chain, *table,
+				ret = nft_cmd_chain_user_rename(h, chain, *table,
 							    argv[optind]);
 				if (ret != 0 && errno == ENOENT)
 					xtables_error(PARAMETER_PROBLEM, "Chain '%s' doesn't exists", chain);
@@ -1144,7 +1144,7 @@ print_zero:
 		/*case 7 :*/ /* atomic-init */
 		/*case 10:*/ /* atomic-save */
 		case 11: /* init-table */
-			nft_table_flush(h, *table);
+			nft_cmd_table_flush(h, *table);
 			return 1;
 		/*
 			replace->command = c;
@@ -1232,13 +1232,13 @@ print_zero:
 
 	if (command == 'P') {
 		if (selected_chain >= NF_BR_NUMHOOKS) {
-			ret = ebt_set_user_chain_policy(h, *table, chain, policy);
+			ret = ebt_cmd_user_chain_policy(h, *table, chain, policy);
 		} else {
 			if (strcmp(policy, "RETURN") == 0) {
 				xtables_error(PARAMETER_PROBLEM,
 					      "Policy RETURN only allowed for user defined chains");
 			}
-			ret = nft_chain_set(h, *table, chain, policy, NULL);
+			ret = nft_cmd_chain_set(h, *table, chain, policy, NULL);
 			if (ret < 0)
 				xtables_error(PARAMETER_PROBLEM, "Wrong policy");
 		}
@@ -1251,9 +1251,9 @@ print_zero:
 				 flags&LIST_C);
 	}
 	if (flags & OPT_ZERO) {
-		ret = nft_chain_zero_counters(h, chain, *table, 0);
+		ret = nft_cmd_chain_zero_counters(h, chain, *table, 0);
 	} else if (command == 'F') {
-		ret = nft_rule_flush(h, chain, *table, 0);
+		ret = nft_cmd_rule_flush(h, chain, *table, 0);
 	} else if (command == 'A') {
 		ret = append_entry(h, chain, *table, &cs, 0, 0, true);
 	} else if (command == 'I') {
iptables/xtables-restore.c +16 -16
index 63cc15cee962..cc5a9056a715 100644
@@ -61,11 +61,11 @@ static void print_usage(const char *name, const char *version)
 static const struct nft_xt_restore_cb restore_cb = {
 	.commit		= nft_commit,
 	.abort		= nft_abort,
-	.table_new	= nft_table_new,
-	.table_flush	= nft_table_flush,
+	.table_new	= nft_cmd_table_new,
+	.table_flush	= nft_cmd_table_flush,
 	.do_command	= do_commandx,
-	.chain_set	= nft_chain_set,
-	.chain_restore  = nft_chain_restore,
+	.chain_set	= nft_cmd_chain_set,
+	.chain_restore  = nft_cmd_chain_restore,
 };
 
 struct nft_xt_restore_state {
@@ -189,7 +189,7 @@ static void xtables_restore_parse_line(struct nft_handle *h,
 				      "cannot create chain '%s' (%s)\n",
 				      chain, strerror(errno));
 		} else if (h->family == NFPROTO_BRIDGE &&
-			   !ebt_set_user_chain_policy(h, state->curtable->name,
+			   !ebt_cmd_user_chain_policy(h, state->curtable->name,
 						      chain, policy)) {
 			xtables_error(OTHER_PROBLEM,
 				      "Can't set policy `%s' on `%s' line %u: %s\n",
@@ -489,20 +489,20 @@ int xtables_ip6_restore_main(int argc, char *argv[])
 				    argc, argv);
 }
 
-static int ebt_table_flush(struct nft_handle *h, const char *table)
+static int ebt_cmd_table_flush(struct nft_handle *h, const char *table)
 {
 	/* drop any pending policy rule add/removal jobs */
-	nft_abort_policy_rule(h, table);
-	return nft_table_flush(h, table);
+	nft_cmd_abort_policy_rule(h, table);
+	return nft_cmd_table_flush(h, table);
 }
 
 static const struct nft_xt_restore_cb ebt_restore_cb = {
 	.commit		= nft_bridge_commit,
-	.table_new	= nft_table_new,
-	.table_flush	= ebt_table_flush,
+	.table_new	= nft_cmd_table_new,
+	.table_flush	= ebt_cmd_table_flush,
 	.do_command	= do_commandeb,
-	.chain_set	= nft_chain_set,
-	.chain_restore  = nft_chain_restore,
+	.chain_set	= nft_cmd_chain_set,
+	.chain_restore  = nft_cmd_chain_restore,
 };
 
 static const struct option ebt_restore_options[] = {
@@ -544,11 +544,11 @@ int xtables_eb_restore_main(int argc, char *argv[])
 
 static const struct nft_xt_restore_cb arp_restore_cb = {
 	.commit		= nft_commit,
-	.table_new	= nft_table_new,
-	.table_flush	= nft_table_flush,
+	.table_new	= nft_cmd_table_new,
+	.table_flush	= nft_cmd_table_flush,
 	.do_command	= do_commandarp,
-	.chain_set	= nft_chain_set,
-	.chain_restore  = nft_chain_restore,
+	.chain_set	= nft_cmd_chain_set,
+	.chain_restore  = nft_cmd_chain_restore,
 };
 
 int xtables_arp_restore_main(int argc, char *argv[])
iptables/xtables.c +26 -26

158 lines not on this page · raw message

Signature · 1 line
2.11.0
author maintainer
#3 lore raw

PATCH 3/7 nft: calculate cache requirements from list of commands

In reply to Pablo Neira Ayuso

This patch uses the new list of commands to calculate the cache requirements, the rationale after this updates is the following:

#1 Parsing, that builds the list of commands and it also calculates
cache level requirements.
#2 Cache building.
#3 Translate commands to jobs
#4 Translate jobs to netlink

This patch disables the pre-parsing code in xtables-restore.c to calculate the cache.

After this patch, cache is calculated only once, there is no need to cancel and refetch for an in-transit transaction.

---
 iptables/nft-cache.c       | 60 +++++++++++++++++--------------------
 iptables/nft-cache.h       |  1 +
 iptables/nft-cmd.c         | 74 ++++++++++++++++++++++++++++++++++++++++++++++
 iptables/nft-cmd.h         |  1 +
 iptables/nft.c             | 17 +++++++++--
 iptables/nft.h             |  2 +-
 iptables/xtables-restore.c |  3 --
 iptables/xtables-save.c    |  3 ++
 8 files changed, 122 insertions(+), 39 deletions(-)
iptables/nft-cache.c +27 -33
index 7345a27e2894..82d6b7c2393a 100644
@@ -454,36 +454,31 @@ __nft_build_cache(struct nft_handle *h, enum nft_cache_level level,
 {
 	uint32_t genid_start, genid_stop;
 
-	if (level <= h->cache_level)
+	if (h->cache_init)
 		return;
+
+	h->cache_init = true;
 retry:
 	mnl_genid_get(h, &genid_start);
 
-	if (h->cache_level && genid_start != h->nft_genid)
-		flush_chain_cache(h, NULL);
-
 	switch (h->cache_level) {
-	case NFT_CL_NONE:
-		fetch_table_cache(h);
-		if (level == NFT_CL_TABLES)
-			break;
-		/* fall through */
 	case NFT_CL_TABLES:
-		fetch_chain_cache(h, t, chain);
-		if (level == NFT_CL_CHAINS)
-			break;
-		/* fall through */
+		fetch_table_cache(h);
+		break;
 	case NFT_CL_CHAINS:
-		fetch_set_cache(h, t, set);
-		if (level == NFT_CL_SETS)
-			break;
-		/* fall through */
+		fetch_table_cache(h);
+		fetch_chain_cache(h, t, chain);
+		break;
 	case NFT_CL_SETS:
-		fetch_rule_cache(h, t, chain);
-		if (level == NFT_CL_RULES)
-			break;
-		/* fall through */
+		fetch_table_cache(h);
+		fetch_chain_cache(h, t, chain);
+		fetch_set_cache(h, t, set);
+		break;
 	case NFT_CL_RULES:
+		fetch_table_cache(h);
+		fetch_chain_cache(h, t, chain);
+		fetch_set_cache(h, t, set);
+		fetch_rule_cache(h, t, chain);
 		break;
 	}
 
@@ -493,11 +488,6 @@ retry:
 		goto retry;
 	}
 
-	if (!t && !chain)
-		h->cache_level = level;
-	else if (h->cache_level < NFT_CL_TABLES)
-		h->cache_level = NFT_CL_TABLES;
-
 	h->nft_genid = genid_start;
 }
 
@@ -529,6 +519,7 @@ void nft_fake_cache(struct nft_handle *h)
 		h->cache->table[type].chains = nftnl_chain_list_alloc();
 	}
 	h->cache_level = NFT_CL_RULES;
+	h->cache_init = true;
 	mnl_genid_get(h, &h->nft_genid);
 }
 
@@ -627,22 +618,25 @@ static int flush_cache(struct nft_handle *h, struct nft_cache *c,
 
 void flush_chain_cache(struct nft_handle *h, const char *tablename)
 {
-	if (!h->cache_level)
+	if (!h->cache_init)
 		return;
 
 	if (flush_cache(h, h->cache, tablename))
-		h->cache_level = NFT_CL_NONE;
+		h->cache_init = false;
 }
 
 void nft_rebuild_cache(struct nft_handle *h)
 {
-	enum nft_cache_level level = h->cache_level;
-
-	if (h->cache_level)
+	if (h->cache_init) {
 		__nft_flush_cache(h);
+		h->cache_init = false;
+	}
+	__nft_build_cache(h, h->cache_level, NULL, NULL, NULL);
+}
 
-	h->cache_level = NFT_CL_NONE;
-	__nft_build_cache(h, level, NULL, NULL, NULL);
+void nft_cache_build(struct nft_handle *h)
+{
+	__nft_build_cache(h, h->cache_level, NULL, NULL, NULL);
 }
 
 void nft_release_cache(struct nft_handle *h)
iptables/nft-cache.h +1 -0
index ed498835676e..8e957c155641 100644
@@ -10,6 +10,7 @@ void nft_release_cache(struct nft_handle *h);
 void flush_chain_cache(struct nft_handle *h, const char *tablename);
 int flush_rule_cache(struct nft_handle *h, const char *table,
 		     struct nftnl_chain *c);
+void nft_cache_build(struct nft_handle *h);
 
 struct nftnl_chain_list *
 nft_chain_list_get(struct nft_handle *h, const char *table, const char *chain);
iptables/nft-cmd.c +74 -0

193 lines not on this page · raw message

iptables/nft-cmd.h +1 -0
index 3b928af3893c..6419dba7fe20 100644
@@ -18,6 +18,7 @@ struct nft_cmd {
 	unsigned int			format;
 	struct {
 		struct nftnl_rule	*rule;
+		struct nftnl_set	*set;
 	} obj;
 	const char			*policy;
 	struct xt_counters		counters;
iptables/nft.c +15 -2
index 0c450b26c06e..1ff2e93c3a3e 100644
@@ -947,6 +947,7 @@ static struct nftnl_set *add_anon_set(struct nft_handle *h, const char *table,
 {
 	static uint32_t set_id = 0;
 	struct nftnl_set *s;
+	struct nft_cmd *cmd;
 
 	s = nftnl_set_alloc();
 	if (!s)
@@ -962,7 +963,14 @@ static struct nftnl_set *add_anon_set(struct nft_handle *h, const char *table,
 	nftnl_set_set_u32(s, NFTNL_SET_KEY_LEN, key_len);
 	nftnl_set_set_u32(s, NFTNL_SET_DESC_SIZE, size);
 
-	return batch_set_add(h, NFT_COMPAT_SET_ADD, s) ? s : NULL;
+	cmd = nft_cmd_new(h, NFT_COMPAT_SET_ADD, table, NULL, NULL, -1, false);
+	if (!cmd) {
+		nftnl_set_free(s);
+		return NULL;
+	}
+	cmd->obj.set = s;
+
+	return s;
 }
 
 static struct nftnl_expr *
@@ -2964,6 +2972,8 @@ static int nft_prepare(struct nft_handle *h)
 	struct nft_cmd *cmd, *next;
 	int ret = 1;
 
+	nft_cache_build(h);
+
 	list_for_each_entry_safe(cmd, next, &h->cmd_list, head) {
 		switch (cmd->command) {
 		case NFT_COMPAT_TABLE_FLUSH:
@@ -3044,9 +3054,12 @@ static int nft_prepare(struct nft_handle *h)
 			nft_xt_builtin_init(h, cmd->table);
 			ret = 1;
 			break;
+		case NFT_COMPAT_SET_ADD:
+			batch_set_add(h, NFT_COMPAT_SET_ADD, cmd->obj.set);
+			ret = 1;
+			break;
 		case NFT_COMPAT_TABLE_ADD:
 		case NFT_COMPAT_CHAIN_ADD:
-		case NFT_COMPAT_SET_ADD:
 			assert(0);
 			break;
 		}
iptables/nft.h +1 -1
index e98fe4db07f7..a9c133934b9e 100644
@@ -30,7 +30,6 @@ struct builtin_table {
 };
 
 enum nft_cache_level {
-	NFT_CL_NONE,
 	NFT_CL_TABLES,
 	NFT_CL_CHAINS,
 	NFT_CL_SETS,
@@ -94,6 +93,7 @@ struct nft_handle {
 	bool			noflush;
 	int8_t			config_done;
 	struct list_head	cmd_list;
+	bool			cache_init;
 
 	/* meta data, for error reporting */
 	struct {
iptables/xtables-restore.c +0 -3
index cc5a9056a715..868223c8fa77 100644
@@ -319,9 +319,6 @@ void xtables_restore_parse(struct nft_handle *h,
 			ptr += blen;
 			buffer[0] = '\0';
 		}
-
-		if (do_cache)
-			nft_build_cache(h, NULL);
 	}
 
 	line = 0;
iptables/xtables-save.c +3 -0
index 3a52f8c3d820..ff7234cca41f 100644
@@ -253,6 +253,9 @@ xtables_save_main(int family, int argc, char *argv[],
 	if (!h.ops)
 		xtables_error(PARAMETER_PROBLEM, "Unknown family");
 
+	h.cache_level = NFT_CL_RULES;
+	nft_cache_build(&h);
+
 	ret = do_output(&h, tablename, &d);
 	nft_fini(&h);
 	if (dump)
Signature · 1 line
2.11.0
author maintainer
#4 lore raw

PATCH 4/7 nft: restore among support

In reply to Pablo Neira Ayuso

Update among support to work again with the new parser and cache logic.

---
 iptables/nft-bridge.c | 13 +++++++++++--
 iptables/nft.c        | 15 +++++++++++++++
 iptables/nft.h        |  6 ++++++
 3 files changed, 32 insertions(+), 2 deletions(-)
iptables/nft-bridge.c +11 -2
index a5aaa3f87187..80d7f91710c1 100644
@@ -421,11 +421,20 @@ static struct nftnl_set *set_from_lookup_expr(struct nft_xt_ctx *ctx,
 					      const struct nftnl_expr *e)
 {
 	const char *set_name = nftnl_expr_get_str(e, NFTNL_EXPR_LOOKUP_SET);
+	uint32_t set_id = nftnl_expr_get_u32(e, NFTNL_EXPR_LOOKUP_SET_ID);
 	struct nftnl_set_list *slist;
+	struct nftnl_set *set;
 
 	slist = nft_set_list_get(ctx->h, ctx->table, set_name);
-	if (slist)
-		return nftnl_set_list_lookup_byname(slist, set_name);
+	if (slist) {
+		set = nftnl_set_list_lookup_byname(slist, set_name);
+		if (set)
+			return set;
+
+		set = nft_set_batch_lookup_byid(ctx->h, set_id);
+		if (set)
+			return set;
+	}
 
 	return NULL;
 }
iptables/nft.c +15 -0
index 1ff2e93c3a3e..a1e38cbafcbe 100644
@@ -1588,6 +1588,20 @@ int nft_rule_save(struct nft_handle *h, const char *table, unsigned int format)
 	return ret == 0 ? 1 : 0;
 }
 
+struct nftnl_set *nft_set_batch_lookup_byid(struct nft_handle *h,
+					    uint32_t set_id)
+{
+	struct obj_update *n;
+
+	list_for_each_entry(n, &h->obj_list, head) {
+		if (n->type == NFT_COMPAT_SET_ADD &&
+		    nftnl_set_get_u32(n->set, NFTNL_SET_ID) == set_id)
+			return n->set;
+	}
+
+	return NULL;
+}
+
 static void
 __nft_rule_flush(struct nft_handle *h, const char *table,
 		 const char *chain, bool verbose, bool implicit)
@@ -3055,6 +3069,7 @@ static int nft_prepare(struct nft_handle *h)
 			ret = 1;
 			break;
 		case NFT_COMPAT_SET_ADD:
+			nft_xt_builtin_init(h, cmd->table);
 			batch_set_add(h, NFT_COMPAT_SET_ADD, cmd->obj.set);
 			ret = 1;
 			break;
iptables/nft.h +6 -0
index a9c133934b9e..86f19169e645 100644
@@ -144,6 +144,12 @@ void nft_bridge_chain_postprocess(struct nft_handle *h,
 
 
 /*
+ * Operations with sets.
+ */
+struct nftnl_set *nft_set_batch_lookup_byid(struct nft_handle *h,
+					    uint32_t set_id);
+
+/*
  * Operations with rule-set.
  */
 struct nftnl_rule;
Signature · 1 line
2.11.0
author maintainer
#5 lore raw

PATCH 5/7 nft: remove cache build calls

In reply to Pablo Neira Ayuso

The cache requirements are now calculated once from the parsing phase. There is no need to call __nft_build_cache() from several spots in the codepath anymore.

---
 iptables/nft-cache.c |  6 ------
 iptables/nft.c       | 21 ---------------------
 2 files changed, 27 deletions(-)
iptables/nft-cache.c +0 -6
index 82d6b7c2393a..1fb65892d898 100644
@@ -647,8 +647,6 @@ void nft_release_cache(struct nft_handle *h)
 
 struct nftnl_table_list *nftnl_table_list_get(struct nft_handle *h)
 {
-	__nft_build_cache(h, NFT_CL_TABLES, NULL, NULL, NULL);
-
 	return h->cache->tables;
 }
 
@@ -661,8 +659,6 @@ nft_set_list_get(struct nft_handle *h, const char *table, const char *set)
 	if (!t)
 		return NULL;
 
-	__nft_build_cache(h, NFT_CL_RULES, t, set, NULL);
-
 	return h->cache->table[t->type].sets;
 }
 
@@ -675,8 +671,6 @@ nft_chain_list_get(struct nft_handle *h, const char *table, const char *chain)
 	if (!t)
 		return NULL;
 
-	__nft_build_cache(h, NFT_CL_CHAINS, t, NULL, chain);
-
 	return h->cache->table[t->type].chains;
 }
 
iptables/nft.c +0 -21
index a1e38cbafcbe..e65ee028c979 100644
@@ -1342,14 +1342,6 @@ nft_rule_append(struct nft_handle *h, const char *chain, const char *table,
 
 	nft_xt_builtin_init(h, table);
 
-	/* Since ebtables user-defined chain policies are implemented as last
-	 * rule in nftables, rule cache is required here to treat them right. */
-	if (h->family == NFPROTO_BRIDGE) {
-		c = nft_chain_find(h, table, chain);
-		if (c && !nft_chain_builtin(c))
-			nft_build_cache(h, c);
-	}
-
 	nft_fn = nft_rule_append;
 
 	if (ref) {
@@ -1574,7 +1566,6 @@ int nft_rule_save(struct nft_handle *h, const char *table, unsigned int format)
 
 	c = nftnl_chain_list_iter_next(iter);
 	while (c) {
-		nft_build_cache(h, c);
 		ret = nft_chain_save_rules(h, c, format);
 		if (ret != 0)
 			break;
@@ -1782,10 +1773,6 @@ static int __nft_chain_user_del(struct nftnl_chain *c, void *data)
 		fprintf(stdout, "Deleting chain `%s'\n",
 			nftnl_chain_get_str(c, NFTNL_CHAIN_NAME));
 
-	/* This triggers required policy rule deletion. */
-	if (h->family == NFPROTO_BRIDGE)
-		nft_build_cache(h, c);
-
 	/* XXX This triggers a fast lookup from the kernel. */
 	nftnl_chain_unset(c, NFTNL_CHAIN_HANDLE);
 	ret = batch_chain_add(h, NFT_COMPAT_CHAIN_USER_DEL, c);
@@ -2068,8 +2055,6 @@ nft_rule_find(struct nft_handle *h, struct nftnl_chain *c,
 	struct nftnl_rule_iter *iter;
 	bool found = false;
 
-	nft_build_cache(h, c);
-
 	if (rulenum >= 0)
 		/* Delete by rule number case */
 		return nftnl_rule_lookup_byindex(c, rulenum);
@@ -2955,8 +2940,6 @@ int ebt_set_user_chain_policy(struct nft_handle *h, const char *table,
 	else
 		return 0;
 
-	nft_build_cache(h, c);
-
 	nftnl_chain_set_u32(c, NFTNL_CHAIN_POLICY, pval);
 	return 1;
 }
@@ -3317,8 +3300,6 @@ static int __nft_chain_zero_counters(struct nftnl_chain *c, void *data)
 			return -1;
 	}
 
-	nft_build_cache(h, c);
-
 	iter = nftnl_rule_iter_create(c);
 	if (iter == NULL)
 		return -1;
@@ -3455,8 +3436,6 @@ static int nft_is_chain_compatible(struct nftnl_chain *c, void *data)
 	enum nf_inet_hooks hook;
 	int prio;
 
-	nft_build_cache(h, c);
-
 	if (nftnl_rule_foreach(c, nft_is_rule_compatible, NULL))
 		return -1;
 
Signature · 1 line
2.11.0
author maintainer
#6 lore raw

PATCH 6/7 nft: skip table list release if uninitialized

In reply to Pablo Neira Ayuso
 # printf '%s\nCOMMIT\n' '*nat' '*raw' '*filter' | iptables-nft-restore --test && echo ok
 Segmentation fault
---
 iptables/nft-cache.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
iptables/nft-cache.c +4 -2
index 1fb65892d898..ab20eb557f4d 100644
@@ -610,8 +610,10 @@ static int flush_cache(struct nft_handle *h, struct nft_cache *c,
 			nftnl_set_list_free(c->table[i].sets);
 		c->table[i].sets = NULL;
 	}
-	nftnl_table_list_free(c->tables);
-	c->tables = NULL;
+	if (c->tables) {
+		nftnl_table_list_free(c->tables);
+		c->tables = NULL;
+	}
 
 	return 1;
 }
Signature · 1 line
2.11.0
author maintainer
#7 lore raw

PATCH 7/7 nft: missing nft_fini() call in bridge family

In reply to Pablo Neira Ayuso
---
 iptables/xtables-eb-standalone.c | 2 ++
 1 file changed, 2 insertions(+)
iptables/xtables-eb-standalone.c +2 -0
index a9081c78c3bc..ff74ddbb3733 100644
@@ -53,6 +53,8 @@ int xtables_eb_main(int argc, char *argv[])
 	if (ret)
 		ret = nft_bridge_commit(&h);
 
+	nft_fini(&h);
+
 	if (!ret)
 		fprintf(stderr, "ebtables: %s\n", nft_strerror(errno));
 
Signature · 1 line
2.11.0