Skip to contentSkip to conversation

xt: Implement dump and restore support

PATCH In review Phil Sutter 3 participants 18 messages Nov 17, 2022 – Nov 18, 2022 netfilter

lore
Share & tools
lore

Phil Sutter

author reviewer
#0 lore raw

v2 cover xt: Implement dump and restore support

If nft can't translate a compat expression, dump it in a format that can be restored later without losing data, thereby keeping the ruleset intact.

Patch 1 is preparation (more or less), patch 2 has the gory details, patch 3 is a minor code refactoring that's almost unrelated and patch 4 further sanitizes behaviour now that there's a reliable fallback in place.

Changes since v1:
- Use patch 3 to also improve the error printing if extension lookup fails.
- New patch 4.

Phil Sutter (4):
xt: Delay libxtables access until translation
xt: Implement dump and restore support
xt: Put match/target translation into own functions
xt: Detect xlate callback failure

 configure.ac              |  12 +-
 doc/libnftables-json.adoc |  15 +-
 doc/statements.txt        |  17 ++
 include/base64.h          |  17 ++
 include/json.h            |   2 +
 include/parser.h          |   1 +
 include/statement.h       |   9 +-
 include/xt.h              |   4 +
 src/Makefile.am           |   3 +-
 src/base64.c              | 170 ++++++++++++++++++++
 src/evaluate.c            |   1 +
 src/json.c                |  25 ++-
 src/netlink_linearize.c   |  32 ++++
 src/parser_bison.y        |  28 ++++
 src/parser_json.c         |  36 +++++
 src/scanner.l             |  14 ++
 src/statement.c           |   1 +
 src/xt.c                  | 317 ++++++++++++++++++++++----------------
 18 files changed, 558 insertions(+), 146 deletions(-)
 create mode 100644 include/base64.h
 create mode 100644 src/base64.c
Signature · 1 line
2.38.0
author reviewer
#1 lore raw

PATCH v2 1/4 xt: Delay libxtables access until translation

There is no point in spending efforts setting up the xt match/target when it is not printed afterwards. So just store the statement data from libnftnl in struct xt_stmt and perform the extension lookup from xt_stmt_xlate() instead.

This means some data structures are only temporarily allocated for the sake of passing to libxtables callbacks, no need to drag them around. Also no need to clone the looked up extension, it is needed only to call the functions it provides.

While being at it, select numeric output in xt_xlate_*_params - otherwise there will be reverse DNS lookups which should not happen by default.

---
 include/statement.h |   9 +--
 src/xt.c            | 192 ++++++++++++++++++--------------------------
 2 files changed, 80 insertions(+), 121 deletions(-)
include/statement.h +4 -5
index 2a2d300106181..8651fc78892c9 100644
@@ -264,12 +264,11 @@ struct xtables_target;
 struct xt_stmt {
 	const char			*name;
 	enum nft_xt_type		type;
+	uint32_t			rev;
+	uint32_t			family;
+	size_t				infolen;
+	void				*info;
 	uint32_t			proto;
-	union {
-		struct xtables_match	*match;
-		struct xtables_target	*target;
-	};
-	void				*entry;
 };
 
 extern struct stmt *xt_stmt_alloc(const struct location *loc);
src/xt.c +76 -116

278 lines not on this page · raw message

Signature · 1 line
2.38.0
author reviewer
#2 lore raw

PATCH v2 2/4 xt: Implement dump and restore support

In reply to Phil Sutter

When listing a rule with compat expressions, fall back to a textual representation which does not require knowledge of the underlying data structures (i.e. libxtables support), preserves information and gives some clue to users what it is.

The statement is printed with its type and name in clear text, the remaining private data and revision encoded as base64 string. To prevent listings from becoming a total mess, the data is zipped before encoding (if zlib is available).

Base64 en-/decoding implementation copied from FreeBSD and slightly adjusted to fit our needs.

---
 configure.ac              |  12 ++-
 doc/libnftables-json.adoc |  15 +++-
 doc/statements.txt        |  17 ++++
 include/base64.h          |  17 ++++
 include/json.h            |   2 +
 include/parser.h          |   1 +
 include/xt.h              |   4 +
 src/Makefile.am           |   3 +-
 src/base64.c              | 170 ++++++++++++++++++++++++++++++++++++++
 src/evaluate.c            |   1 +
 src/json.c                |  25 ++++--
 src/netlink_linearize.c   |  32 +++++++
 src/parser_bison.y        |  28 +++++++
 src/parser_json.c         |  36 ++++++++
 src/scanner.l             |  14 ++++
 src/statement.c           |   1 +
 src/xt.c                  |  99 ++++++++++++++++++++--
 17 files changed, 458 insertions(+), 19 deletions(-)
 create mode 100644 include/base64.h
 create mode 100644 src/base64.c
configure.ac +11 -1
index eb1882dd828e8..268255cfeb39e 100644
@@ -98,6 +98,15 @@ AC_DEFINE([HAVE_LIBXTABLES], [1], [0])
 ])
 AM_CONDITIONAL([BUILD_XTABLES], [test "x$with_xtables" = xyes])
 
+AC_ARG_WITH([zlib], [AS_HELP_STRING([--without-zlib],
+            [Disable payload compression of xt compat expressions when listing])],
+	    [], [with_zlib=yes])
+AS_IF([test "x$with_zlib" != xno], [
+AC_CHECK_LIB([z], [compress], ,
+	     AC_MSG_ERROR([No suitable version of zlib found]))
+AC_DEFINE([HAVE_ZLIB], [1], [Define if you have zlib])
+])
+
 AC_ARG_WITH([json], [AS_HELP_STRING([--with-json],
             [Enable JSON output support])],
 	    [], [with_json=no])
@@ -156,7 +165,8 @@ echo "
   use mini-gmp:			${with_mini_gmp}
   enable man page:              ${enable_man_doc}
   libxtables support:		${with_xtables}
-  json output support:          ${with_json}"
+  json output support:          ${with_json}
+  compress xt compat payloads:  ${with_zlib}"
 
 AS_IF([test "$enable_python" != "no"], [
 	echo "  enable Python:		yes (with $PYTHON_BIN)"
doc/libnftables-json.adoc +12 -3
index bb59945fc510d..6d75a9b6ed5d3 100644
@@ -1059,10 +1059,19 @@ Assign connection tracking expectation.
 
 === XT
 [verse]
-*{ "xt": null }*
+____
+*{ "xt": [* 'TYPE' *,* 'STRING' *,* 'STRING' *] }*
+
+'TYPE' := *match* | *target* | *watcher*
+____
+
+This represents an xt statement from xtables compat interface. It merely exists
+to support saving and restoring a ruleset containing these statements without
+losing data.
 
-This represents an xt statement from xtables compat interface. Sadly, at this
-point, it is not possible to provide any further information about its content.
+The textual representation was chosen to give users a rough idea of what it is
+by explicitly stating the extension's 'TYPE' and name in the first two array
+fields. The third one embeds the extension's data in compressed base64 format.
 
 == EXPRESSIONS
 Expressions are the building blocks of (most) statements. In their most basic
doc/statements.txt +17 -0
index 8076c21cded41..19ea23d1129f3 100644
@@ -783,3 +783,20 @@ ____
 # jump to different chains depending on layer 4 protocol type:
 nft add rule ip filter input ip protocol vmap { tcp : jump tcp-chain, udp : jump udp-chain , icmp : jump icmp-chain }
 ------------------------
+
+XT STATEMENT
+~~~~~~~~~~~~
+This represents an xt statement from xtables compat interface. It merely exists
+to support saving and restoring a ruleset containing these statements without
+losing data.
+
+[verse]
+____
+*xt* 'TYPE' 'NAME' 'BASE64'
+
+'TYPE' := *match* | *target* | *watcher*
+____
+
+The textual representation was chosen to give users a rough idea of what it is
+by explicitly stating the extension's 'TYPE' and 'NAME' followed by the
+extension's data in compressed 'BASE64' format.
include/base64.h new file +17 -0
new file mode 100644
index 0000000000000..aa21fd0fc1b76
@@ -0,0 +1,17 @@
+/*
+ * Base64 encoding/decoding (RFC1341)
+ * Copyright (c) 2005, Jouni Malinen <[email protected]>
+ *
+ * This software may be distributed under the terms of the BSD license.
+ * See README for more details.
+ */
+
+#ifndef BASE64_H
+#define BASE64_H
+
+unsigned char * base64_encode(const unsigned char *src, size_t len,
+			      size_t *out_len);
+unsigned char * base64_decode(const unsigned char *src, size_t len,
+			      size_t *out_len);
+
+#endif /* BASE64_H */
include/json.h +2 -0
index b0d78eb84987e..f691678d4d726 100644
@@ -92,6 +92,7 @@ json_t *connlimit_stmt_json(const struct stmt *stmt, struct output_ctx *octx);
 json_t *tproxy_stmt_json(const struct stmt *stmt, struct output_ctx *octx);
 json_t *synproxy_stmt_json(const struct stmt *stmt, struct output_ctx *octx);
 json_t *optstrip_stmt_json(const struct stmt *stmt, struct output_ctx *octx);
+json_t *xt_stmt_json(const struct stmt *stmt, struct output_ctx *octx);
 
 int do_command_list_json(struct netlink_ctx *ctx, struct cmd *cmd);
 
@@ -194,6 +195,7 @@ STMT_PRINT_STUB(connlimit)
 STMT_PRINT_STUB(tproxy)
 STMT_PRINT_STUB(synproxy)
 STMT_PRINT_STUB(optstrip)
+STMT_PRINT_STUB(xt)
 
 #undef STMT_PRINT_STUB
 #undef EXPR_PRINT_STUB
include/parser.h +1 -0
index f55da0fd47bf2..ec5b406ef89c5 100644
@@ -83,6 +83,7 @@ enum startcond_type {
 	PARSER_SC_STMT_REJECT,
 	PARSER_SC_STMT_SYNPROXY,
 	PARSER_SC_STMT_TPROXY,
+	PARSER_SC_STMT_XT,
 
 	__SC_MAX
 };
include/xt.h +4 -0
index 9fc515084d597..d365c58479e94 100644
@@ -26,4 +26,8 @@ static inline void stmt_xt_postprocess(struct rule_pp_ctx *rctx,
 
 #endif
 
+unsigned char *xt_stmt_blob_encode(const struct stmt *stmt);
+int xt_stmt_blob_decode(struct stmt *stmt, const char *b64_string,
+			const struct location *b64_loc, struct list_head *msgs);
+
 #endif /* _NFT_XT_H_ */
src/Makefile.am +2 -1
index 264d981e20c7f..cea5b8df20a51 100644
@@ -36,6 +36,7 @@ BUILT_SOURCES = parser_bison.h
 lib_LTLIBRARIES = libnftables.la
 
 libnftables_la_SOURCES =			\
+		base64.c			\
 		rule.c				\
 		statement.c			\
 		cache.c				\
@@ -89,7 +90,7 @@ libparser_la_CFLAGS = ${AM_CFLAGS} \
 		      -Wno-undef \
 		      -Wno-redundant-decls
 
-libnftables_la_LIBADD = ${LIBMNL_LIBS} ${LIBNFTNL_LIBS} libparser.la
+libnftables_la_LIBADD = ${LIBMNL_LIBS} ${LIBNFTNL_LIBS} ${Z_LIBS} libparser.la
 libnftables_la_LDFLAGS = -version-info ${libnftables_LIBVERSION} \
 			 -Wl,--version-script=$(srcdir)/libnftables.map
 
src/base64.c new file +170 -0

170 lines not on this page · raw message

src/evaluate.c +1 -0
index 0bf6a0d1b110c..13566ab3de665 100644
@@ -3989,6 +3989,7 @@ int stmt_evaluate(struct eval_ctx *ctx, struct stmt *stmt)
 	case STMT_QUOTA:
 	case STMT_NOTRACK:
 	case STMT_FLOW_OFFLOAD:
+	case STMT_XT:
 		return 0;
 	case STMT_EXPRESSION:
 		return stmt_evaluate_expr(ctx, stmt);
src/json.c +19 -6
index 6662f8087736a..1242ab04058bb 100644
@@ -7,6 +7,7 @@
 #include <netlink.h>
 #include <rule.h>
 #include <rt.h>
+#include <xt.h>
 
 #include <netdb.h>
 #include <netinet/icmp6.h>
@@ -82,12 +83,6 @@ static json_t *stmt_print_json(const struct stmt *stmt, struct output_ctx *octx)
 	char buf[1024];
 	FILE *fp;
 
-	/* XXX: Can't be supported at this point:
-	 * xt_stmt_xlate() ignores output_fp.
-	 */
-	if (stmt->ops->type == STMT_XT)
-		return json_pack("{s:n}", "xt");
-
 	if (stmt->ops->json)
 		return stmt->ops->json(stmt, octx);
 
@@ -1624,6 +1619,24 @@ json_t *optstrip_stmt_json(const struct stmt *stmt, struct output_ctx *octx)
 			 expr_print_json(stmt->optstrip.expr, octx));
 }
 
+json_t *xt_stmt_json(const struct stmt *stmt, struct output_ctx *octx)
+{
+	static const char *xt_typename[] = {
+		[NFT_XT_MATCH]		= "match",
+		[NFT_XT_TARGET]		= "target",
+		[NFT_XT_WATCHER]	= "watcher",
+		[NFT_XT_MAX]		= "unknown"
+	};
+	unsigned char *b64_buf;
+	json_t *json;
+
+	b64_buf = xt_stmt_blob_encode(stmt);
+	json = json_pack("[s, s, s]", xt_typename[stmt->xt.type],
+			 stmt->xt.name, b64_buf);
+	xfree(b64_buf);
+	return json;
+}
+
 static json_t *table_print_json_full(struct netlink_ctx *ctx,
 				     struct table *table)
 {
src/netlink_linearize.c +32 -0
index c8bbcb7452b05..d372919662578 100644
@@ -20,6 +20,7 @@
 #include <gmputil.h>
 #include <utils.h>
 #include <netinet/in.h>
+#include <zlib.h>
 
 #include <linux/netfilter.h>
 #include <libnftnl/udata.h>
@@ -1365,6 +1366,35 @@ static void netlink_gen_optstrip_stmt(struct netlink_linearize_ctx *ctx,
 	nft_rule_add_expr(ctx, nle, &expr->location);
 }
 
+static void netlink_gen_xt_stmt(struct netlink_linearize_ctx *ctx,
+				const struct stmt *stmt)
+{
+	void *data = xmalloc(stmt->xt.infolen);
+	struct nftnl_expr *nle;
+
+	memcpy(data, stmt->xt.info, stmt->xt.infolen);
+
+	switch(stmt->xt.type) {
+	case NFT_XT_MATCH:
+		nle = alloc_nft_expr("match");
+		nftnl_expr_set_str(nle, NFTNL_EXPR_MT_NAME, stmt->xt.name);
+		nftnl_expr_set_u32(nle, NFTNL_EXPR_MT_REV, stmt->xt.rev);
+		nftnl_expr_set(nle, NFTNL_EXPR_MT_INFO, data, stmt->xt.infolen);
+		break;
+	case NFT_XT_TARGET:
+	case NFT_XT_WATCHER:
+		nle = alloc_nft_expr("target");
+		nftnl_expr_set_str(nle, NFTNL_EXPR_TG_NAME, stmt->xt.name);
+		nftnl_expr_set_u32(nle, NFTNL_EXPR_TG_REV, stmt->xt.rev);
+		nftnl_expr_set(nle, NFTNL_EXPR_TG_INFO, data, stmt->xt.infolen);
+		break;
+	default:
+		free(data);
+		return;
+	}
+	nft_rule_add_expr(ctx, nle, &stmt->location);
+}
+
 static void netlink_gen_queue_stmt(struct netlink_linearize_ctx *ctx,
 				 const struct stmt *stmt)
 {
@@ -1630,6 +1660,8 @@ static void netlink_gen_stmt(struct netlink_linearize_ctx *ctx,
 		return netlink_gen_chain_stmt(ctx, stmt);
 	case STMT_OPTSTRIP:
 		return netlink_gen_optstrip_stmt(ctx, stmt);
+	case STMT_XT:
+		return netlink_gen_xt_stmt(ctx, stmt);
 	default:
 		BUG("unknown statement type %s\n", stmt->ops->name);
 	}
src/parser_bison.y +28 -0
index 760c23cf33223..c23d0178b2fab 100644
@@ -626,6 +626,13 @@ int nft_lex(void *, void *, void *);
 %token IN			"in"
 %token OUT			"out"
 
+%token XT			"xt"
+%token MATCH			"match"
+%token TARGET			"target"
+%token WATCHER			"watcher"
+%type <stmt>			xt_stmt
+%type <val>			xt_stmt_type
+
 %type <limit_rate>		limit_rate_pkts
 %type <limit_rate>		limit_rate_bytes
 
@@ -991,6 +998,7 @@ close_scope_udplite	: { scanner_pop_start_cond(nft->scanner, PARSER_SC_EXPR_UDPL
 
 close_scope_log		: { scanner_pop_start_cond(nft->scanner, PARSER_SC_STMT_LOG); }
 close_scope_synproxy	: { scanner_pop_start_cond(nft->scanner, PARSER_SC_STMT_SYNPROXY); }
+close_scope_xt		: { scanner_pop_start_cond(nft->scanner, PARSER_SC_STMT_XT); }
 
 common_block		:	INCLUDE		QUOTED_STRING	stmt_separator
 			{
@@ -2879,6 +2887,26 @@ stmt			:	verdict_stmt
 			|	synproxy_stmt	close_scope_synproxy
 			|	chain_stmt
 			|	optstrip_stmt
+			|	xt_stmt
+			;
+
+xt_stmt_type		:	MATCH	{ $$ = NFT_XT_MATCH; }
+			|	TARGET	{ $$ = NFT_XT_TARGET; }
+			|	WATCHER	{ $$ = NFT_XT_WATCHER; }
+			;
+
+xt_stmt			:	XT	xt_stmt_type	STRING	STRING	close_scope_xt
+			{
+				$$ = xt_stmt_alloc(&@$);
+				$$->xt.type = $2;
+				$$->xt.name = $3;
+				if (xt_stmt_blob_decode($$, $4, &@4, state->msgs)) {
+					xfree($$);
+					xfree($4);
+					YYERROR;
+				}
+				xfree($4);
+			}
 			;
 
 chain_stmt_type		:	JUMP	{ $$ = NFT_JUMP; }
src/parser_json.c +36 -0
index 76c268f857202..c4e4ef4f9e1f6 100644
@@ -13,6 +13,7 @@
 #include <rule.h>
 #include <sctp_chunk.h>
 #include <socket.h>
+#include <xt.h>
 
 #include <netdb.h>
 #include <netinet/icmp6.h>
@@ -2707,6 +2708,40 @@ static struct stmt *json_parse_optstrip_stmt(struct json_ctx *ctx,
 	return expr ? optstrip_stmt_alloc(int_loc, expr) : NULL;
 }
 
+static enum nft_xt_type xt_stmt_typeval(const char *type)
+{
+	if (!strcmp(type, "match"))
+		return NFT_XT_MATCH;
+	if (!strcmp(type, "target"))
+		return NFT_XT_TARGET;
+	if (!strcmp(type, "watcher"))
+		return NFT_XT_WATCHER;
+	return NFT_XT_MAX;
+}
+
+static struct stmt *json_parse_xt_stmt(struct json_ctx *ctx,
+				       const char *key, json_t *value)
+{
+	char *type, *name, *blob;
+	struct stmt *stmt;
+
+	if (json_unpack_err(ctx, value, "[s, s, s]", &type, &name, &blob))
+		return NULL;
+
+	stmt = xt_stmt_alloc(int_loc);
+	stmt->xt.type = xt_stmt_typeval(type);
+	if (stmt->xt.type == NFT_XT_MAX) {
+		json_error(ctx, "Invalid xt type name '%s'.", type);
+		return NULL;
+	}
+	stmt->xt.name = xstrdup(name);
+	if (xt_stmt_blob_decode(stmt, blob, int_loc, ctx->msgs)) {
+		json_error(ctx, "Invalid base64 data in '%s'.", blob);
+		return NULL;
+	}
+	return stmt;
+}
+
 static struct stmt *json_parse_stmt(struct json_ctx *ctx, json_t *root)
 {
 	struct {
@@ -2745,6 +2780,7 @@ static struct stmt *json_parse_stmt(struct json_ctx *ctx, json_t *root)
 		{ "synproxy", json_parse_synproxy_stmt },
 		{ "reset", json_parse_optstrip_stmt },
 		{ "secmark", json_parse_secmark_stmt },
+		{ "xt", json_parse_xt_stmt },
 	};
 	const char *type;
 	unsigned int i;
src/scanner.l +14 -0
index 1371cd044b65a..e933d32f13d1a 100644
@@ -125,6 +125,7 @@ quotedstring	\"[^"]*\"
 asteriskstring	({string}\*|{string}\\\*|\\\*|{string}\\\*{string})
 comment		#.*$
 slash		\/
+base64string	({letter}|{digit}|\+|{slash})*=*
 
 timestring	([0-9]+d)?([0-9]+h)?([0-9]+m)?([0-9]+s)?([0-9]+ms)?
 
@@ -247,6 +248,7 @@ addrstring	({macaddr}|{ip4addr}|{ip6addr})
 %s SCANSTATE_STMT_REJECT
 %s SCANSTATE_STMT_SYNPROXY
 %s SCANSTATE_STMT_TPROXY
+%s SCANSTATE_STMT_XT
 
 %%
 
@@ -411,6 +413,13 @@ addrstring	({macaddr}|{ip4addr}|{ip6addr})
 	"group"			{ return GROUP; }
 }
 
+"xt"			{ scanner_push_start_cond(yyscanner, SCANSTATE_STMT_XT); return XT; }
+<SCANSTATE_STMT_XT>{
+	"match"		{ return MATCH; }
+	"target"	{ return TARGET; }
+	"watcher"	{ return WATCHER; }
+}
+
 "queue"			{ scanner_push_start_cond(yyscanner, SCANSTATE_EXPR_QUEUE); return QUEUE;}
 <SCANSTATE_EXPR_QUEUE>{
 	"num"		{ return QUEUENUM;}
@@ -846,6 +855,11 @@ addrstring	({macaddr}|{ip4addr}|{ip6addr})
 				return STRING;
 			}
 
+{base64string}		{
+				yylval->string = xstrdup(yytext);
+				return STRING;
+			}
+
 \\{newline}		{
 				reset_pos(yyget_extra(yyscanner), yylloc);
 			}
src/statement.c +1 -0
index 327d00f99200a..eafc51c484de9 100644
@@ -997,6 +997,7 @@ static const struct stmt_ops xt_stmt_ops = {
 	.name		= "xt",
 	.print		= xt_stmt_print,
 	.destroy	= xt_stmt_destroy,
+	.json		= xt_stmt_json,
 };
 
 struct stmt *xt_stmt_alloc(const struct location *loc)
src/xt.c +91 -8

133 lines not on this page · raw message

Signature · 1 line
2.38.0
author reviewer
#3 lore raw

PATCH v2 3/4 xt: Put match/target translation into own functions

In reply to Phil Sutter

Reduce the size and indenting level of xt_stmt_xlate() a bit, also fix for error printing to stderr irrespective of octx->error_fp value.

---
Changes since v1:
- Change fprintf() calls to respect octx.

---
 src/xt.c | 144 ++++++++++++++++++++++++++++++-------------------------
 1 file changed, 78 insertions(+), 66 deletions(-)
src/xt.c +78 -66
index 9a326fd313233..e3063612c353e 100644
@@ -104,6 +104,75 @@ int xt_stmt_blob_decode(struct stmt *stmt, const char *b64_string,
 	return ret;
 }
 
+#ifdef HAVE_LIBXTABLES
+static bool xt_stmt_xlate_match(const struct stmt *stmt, void *entry,
+				struct xt_xlate *xl, struct output_ctx *octx)
+{
+	size_t size = XT_ALIGN(sizeof(struct xt_entry_match))
+			+ stmt->xt.infolen;
+	struct xt_xlate_mt_params params = {
+		.ip		= entry,
+		.numeric        = 1,
+	};
+	struct xtables_match *mt;
+	struct xt_entry_match *m;
+
+	mt = xtables_find_match(stmt->xt.name, XTF_TRY_LOAD, NULL);
+	if (!mt) {
+		fprintf(octx->error_fp,
+			"XT match %s not found\n", stmt->xt.name);
+		return false;
+	}
+	if (!mt->xlate)
+		return false;
+
+	m = xzalloc(size);
+	m->u.match_size = size;
+	m->u.user.revision = stmt->xt.rev;
+	memcpy(&m->data, stmt->xt.info, stmt->xt.infolen);
+
+	params.match = m;
+	mt->xlate(xl, &params);
+
+	xfree(m);
+	return true;
+}
+
+static bool xt_stmt_xlate_target(const struct stmt *stmt, void *entry,
+				 struct xt_xlate *xl, struct output_ctx *octx)
+{
+	size_t size = XT_ALIGN(sizeof(struct xt_entry_target))
+			+ stmt->xt.infolen;
+	struct xt_xlate_tg_params params = {
+		.ip		= entry,
+		.numeric        = 1,
+	};
+	struct xtables_target *tg;
+	struct xt_entry_target *t;
+
+	tg = xtables_find_target(stmt->xt.name, XTF_TRY_LOAD);
+	if (!tg) {
+		fprintf(octx->error_fp,
+			"XT target %s not found\n", stmt->xt.name);
+		return false;
+	}
+	if (!tg->xlate)
+		return false;
+
+	t = xzalloc(size);
+	t->u.target_size = size;
+	t->u.user.revision = stmt->xt.rev;
+	memcpy(&t->data, stmt->xt.info, stmt->xt.infolen);
+	strcpy(t->u.user.name, tg->name);
+
+	params.target = t;
+	tg->xlate(xl, &params);
+
+	xfree(t);
+	return true;
+}
+#endif
+
 void xt_stmt_xlate(const struct stmt *stmt, struct output_ctx *octx)
 {
 	static const char *xt_typename[] = {
@@ -115,11 +184,7 @@ void xt_stmt_xlate(const struct stmt *stmt, struct output_ctx *octx)
 	unsigned char *b64_buf;
 #ifdef HAVE_LIBXTABLES
 	struct xt_xlate *xl = xt_xlate_alloc(10240);
-	struct xtables_target *tg;
-	struct xt_entry_target *t;
-	struct xtables_match *mt;
-	struct xt_entry_match *m;
-	size_t size;
+	bool xlated = false;
 	void *entry;
 
 	xtables_set_nfproto(stmt->xt.family);
@@ -127,76 +192,23 @@ void xt_stmt_xlate(const struct stmt *stmt, struct output_ctx *octx)
 
 	switch (stmt->xt.type) {
 	case NFT_XT_MATCH:
-		mt = xtables_find_match(stmt->xt.name, XTF_TRY_LOAD, NULL);
-		if (!mt) {
-			fprintf(stderr, "XT match %s not found\n",
-				stmt->xt.name);
-			return;
-		}
-		size = XT_ALIGN(sizeof(*m)) + stmt->xt.infolen;
-
-		m = xzalloc(size);
-		memcpy(&m->data, stmt->xt.info, stmt->xt.infolen);
-
-		m->u.match_size = size;
-		m->u.user.revision = stmt->xt.rev;
-
-		if (mt->xlate) {
-			struct xt_xlate_mt_params params = {
-				.ip		= entry,
-				.match		= m,
-				.numeric        = 1,
-			};
-
-			mt->xlate(xl, &params);
-			nft_print(octx, "%s", xt_xlate_get(xl));
-			xfree(m);
-			xfree(entry);
-			xt_xlate_free(xl);
-			return;
-		}
-		xfree(m);
+		xlated = xt_stmt_xlate_match(stmt, entry, xl, octx);
 		break;
 	case NFT_XT_WATCHER:
 	case NFT_XT_TARGET:
-		tg = xtables_find_target(stmt->xt.name, XTF_TRY_LOAD);
-		if (!tg) {
-			fprintf(stderr, "XT target %s not found\n",
-				stmt->xt.name);
-			return;
-		}
-		size = XT_ALIGN(sizeof(*t)) + stmt->xt.infolen;
-
-		t = xzalloc(size);
-		memcpy(&t->data, stmt->xt.info, stmt->xt.infolen);
-
-		t->u.target_size = size;
-		t->u.user.revision = stmt->xt.rev;
-
-		strcpy(t->u.user.name, tg->name);
-
-		if (tg->xlate) {
-			struct xt_xlate_tg_params params = {
-				.ip		= entry,
-				.target		= t,
-				.numeric        = 1,
-			};
-
-			tg->xlate(xl, &params);
-			nft_print(octx, "%s", xt_xlate_get(xl));
-			xfree(t);
-			xfree(entry);
-			xt_xlate_free(xl);
-			return;
-		}
-		xfree(t);
+		xlated = xt_stmt_xlate_target(stmt, entry, xl, octx);
 		break;
 	default:
 		break;
 	}
 
-	xt_xlate_free(xl);
 	xfree(entry);
+	if (xlated) {
+		nft_print(octx, "%s", xt_xlate_get(xl));
+		xt_xlate_free(xl);
+		return;
+	}
+	xt_xlate_free(xl);
 #endif
 	b64_buf = xt_stmt_blob_encode(stmt);
 	nft_print(octx, "xt %s %s %s",
Signature · 1 line
2.38.0
author reviewer
#4 lore raw

PATCH v2 4/4 xt: Detect xlate callback failure

In reply to Phil Sutter

If an extension's xlate callback returns 0, translation is at least incomplete. Discard the result and resort to opaque dump format in this case.

---
 src/xt.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
src/xt.c +6 -4
index e3063612c353e..178761a42018d 100644
@@ -116,6 +116,7 @@ static bool xt_stmt_xlate_match(const struct stmt *stmt, void *entry,
 	};
 	struct xtables_match *mt;
 	struct xt_entry_match *m;
+	int rc;
 
 	mt = xtables_find_match(stmt->xt.name, XTF_TRY_LOAD, NULL);
 	if (!mt) {
@@ -132,10 +133,10 @@ static bool xt_stmt_xlate_match(const struct stmt *stmt, void *entry,
 	memcpy(&m->data, stmt->xt.info, stmt->xt.infolen);
 
 	params.match = m;
-	mt->xlate(xl, &params);
+	rc = mt->xlate(xl, &params);
 
 	xfree(m);
-	return true;
+	return rc != 0;
 }
 
 static bool xt_stmt_xlate_target(const struct stmt *stmt, void *entry,
@@ -149,6 +150,7 @@ static bool xt_stmt_xlate_target(const struct stmt *stmt, void *entry,
 	};
 	struct xtables_target *tg;
 	struct xt_entry_target *t;
+	int rc;
 
 	tg = xtables_find_target(stmt->xt.name, XTF_TRY_LOAD);
 	if (!tg) {
@@ -166,10 +168,10 @@ static bool xt_stmt_xlate_target(const struct stmt *stmt, void *entry,
 	strcpy(t->u.user.name, tg->name);
 
 	params.target = t;
-	tg->xlate(xl, &params);
+	rc = tg->xlate(xl, &params);
 
 	xfree(t);
-	return true;
+	return rc != 0;
 }
 #endif
 
Signature · 1 line
2.38.0
maintainer
#5 lore raw
In reply to Phil Sutter
Quoted 3 lines · Phil Sutter

If nft can't translate a compat expression, dump it in a format that can be restored later without losing data, thereby keeping the ruleset intact.

Why? :-( This cements nft_compat.c forever.

If we're goping to do it lets at least dump it properly, i.e. nft ... add rule compat "-m conntrack --ctstate NEW".

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

#6 lore raw
Quoted 9 lines · Florian Westphal
Quoted 3 lines · Phil Sutter

If nft can't translate a compat expression, dump it in a format that can be restored later without losing data, thereby keeping the ruleset intact.

Why? :-( This cements nft_compat.c forever.

If we're goping to do it lets at least dump it properly, i.e. nft ... add rule compat "-m conntrack --ctstate NEW".

We might have to support this mixed syntax forever. I proposed this long long time ago, when nftables was supporting ~25% of the iptables feature-set. These days, where they are almost on par (actually nftables being a lot more expressive than iptables), I am not sure it makes sense to follow this path anymore.

Unless you refer to dumping a listing which nft cannot load as a way to provide a listing that is comprehensible, but that cannot be loaded by the user.

Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

I agree we should have added native translations for iptables-nft sooner, but it was never a priority for anyone so far.

This "forward compatibility" issue (pretending old tool versions can interpret new revisions / features loaded by newer tool versions) we are trying to deal is hard, we already discussed none of the other existing tooling (ethtool, iproute2, etc.) supports for this.

If you prefer to go for the _USERDATA area as a last resort, I'm OK with it, this requires no kernel patches, and it will be used only for the "forward compatibility" scenario (last resort)

We can also resort on displaying the raw expressions, so the user gets a meaningful output that cannot be loaded again.

I think this more or less a summary of what we discussed in the NFWS.

author reviewer
#7 lore raw
Quoted 17 lines · Pablo Neira Ayuso
Quoted 9 lines · Florian Westphal
Quoted 3 lines · Phil Sutter

If nft can't translate a compat expression, dump it in a format that can be restored later without losing data, thereby keeping the ruleset intact.

Why? :-( This cements nft_compat.c forever.

If we're goping to do it lets at least dump it properly, i.e. nft ... add rule compat "-m conntrack --ctstate NEW".

We might have to support this mixed syntax forever. I proposed this long long time ago, when nftables was supporting ~25% of the iptables feature-set. These days, where they are almost on par (actually nftables being a lot more expressive than iptables), I am not sure it makes sense to follow this path anymore.

Quoted 3 lines

Unless you refer to dumping a listing which nft cannot load as a way to provide a listing that is comprehensible, but that cannot be loaded by the user.

Or a big fat warning, but I fear anything that allows | nft list ruleset | nft -f -
will simply be ignored until the problems become obvious.

Quoted 11 lines
Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

I agree we should have added native translations for iptables-nft sooner, but it was never a priority for anyone so far.

This "forward compatibility" issue (pretending old tool versions can interpret new revisions / features loaded by newer tool versions) we are trying to deal is hard, we already discussed none of the other existing tooling (ethtool, iproute2, etc.) supports for this.

Quoted 8 lines

If you prefer to go for the _USERDATA area as a last resort, I'm OK with it, this requires no kernel patches, and it will be used only for the "forward compatibility" scenario (last resort)

We can also resort on displaying the raw expressions, so the user gets a meaningful output that cannot be loaded again.

I think this more or less a summary of what we discussed in the NFWS.

Pablo, I think you're mixing up two things here:

This "support dump and load of compat expression" feature is to sanitize the current situation with up to date iptables and nftables.

The forward compat (you were right about the direction) issue is mostly about iptables-nft vs. itself in an older version.

Cheers, Phil

author reviewer
#8 lore raw
In reply to Florian Westphal
Quoted 6 lines · Florian Westphal
Quoted 3 lines · Phil Sutter

If nft can't translate a compat expression, dump it in a format that can be restored later without losing data, thereby keeping the ruleset intact.

Why? :-( This cements nft_compat.c forever.

To avoid silently breaking a ruleset. It is a last resort measure for cases where nft can't translate the ruleset into something meaningful. I already submitted a patch adding a warning when listing tables containing compat expressions (status unclear), but a real alternative to the above would be to abort the listing or to ignore the table.

Listing the ruleset in translated form when iptables-nft can't parse the translation or without translation but still maintaining syntax to be parsed without error during a later restore is almost luring users into doing stupid things.

Quoted 2 lines

If we're goping to do it lets at least dump it properly, i.e. nft ... add rule compat "-m conntrack --ctstate NEW".

This will make things worse: People will understand the format and start using it despite the warnings. This adds a new user base to compat expressions. I don't dare claiming nobody would start crafting compat expressions in my zip-dump format, but it's at least a larger obstacle. :D

Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

If you do, please convince Pablo to not push iptables commit 384958620a. I think it opened the can of worms we're trying to confine here.

Cheers, Phil

#9 lore raw

Merging threads.

On Fri, Nov 18, 2022 at 10:55:04AM +0100, Phil Sutter wrote:
[...]

Quoted 6 lines
Quoted 1 line

I think this more or less a summary of what we discussed in the NFWS.

Pablo, I think you're mixing up two things here:

This "support dump and load of compat expression" feature is to sanitize the current situation with up to date iptables and nftables.

OK, then the problem we discuss is mixing iptables-nft and nftables.

On Fri, Nov 18, 2022 at 10:47:48AM +0100, Phil Sutter wrote:
[...]

Quoted 5 lines
Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

If you do, please convince Pablo to not push iptables commit 384958620a. I think it opened the can of worms we're trying to confine here.

It could be worst, if iptables-nft would not be in place, then old iptables-legacy and new nftables rules would have no visibility each other.

With iptables-nft we have a way to move forward:

- Replace nft_compat by native expressions from iptables-nft.
- Extend iptables-nft to understand more complex expressions, worst case dump a native representation.

Why don't we just move ahead this path instead of spinning around the compat layer? This only requires userspace updates on iptables-nft.

author reviewer
#10 lore raw
Quoted 25 lines · Pablo Neira Ayuso

Merging threads.

On Fri, Nov 18, 2022 at 10:55:04AM +0100, Phil Sutter wrote:
[...]

Quoted 6 lines
Quoted 1 line

I think this more or less a summary of what we discussed in the NFWS.

Pablo, I think you're mixing up two things here:

This "support dump and load of compat expression" feature is to sanitize the current situation with up to date iptables and nftables.

OK, then the problem we discuss is mixing iptables-nft and nftables.

On Fri, Nov 18, 2022 at 10:47:48AM +0100, Phil Sutter wrote:
[...]

Quoted 5 lines
Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

If you do, please convince Pablo to not push iptables commit 384958620a. I think it opened the can of worms we're trying to confine here.

It could be worst, if iptables-nft would not be in place, then old iptables-legacy and new nftables rules would have no visibility each other.

Quoted 8 lines

With iptables-nft we have a way to move forward:

- Replace nft_compat by native expressions from iptables-nft.
- Extend iptables-nft to understand more complex expressions, worst case dump a native representation.

Why don't we just move ahead this path instead of spinning around the compat layer? This only requires userspace updates on iptables-nft.

Sure! I'm just picking low hanging fruits first. With even translation support being still incomplete, I fear it will take a while until the tools are fluent enough for this to not matter anymore. And then there's still nftables without libxtables support.

Cheers, Phil

maintainer
#11 lore raw
Quoted 32 lines · Phil Sutter
Quoted 31 lines · Pablo Neira Ayuso

Merging threads.

On Fri, Nov 18, 2022 at 10:55:04AM +0100, Phil Sutter wrote:
[...]

Quoted 6 lines
Quoted 1 line

I think this more or less a summary of what we discussed in the NFWS.

Pablo, I think you're mixing up two things here:

This "support dump and load of compat expression" feature is to sanitize the current situation with up to date iptables and nftables.

OK, then the problem we discuss is mixing iptables-nft and nftables.

On Fri, Nov 18, 2022 at 10:47:48AM +0100, Phil Sutter wrote:
[...]

Quoted 5 lines
Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

If you do, please convince Pablo to not push iptables commit 384958620a. I think it opened the can of worms we're trying to confine here.

It could be worst, if iptables-nft would not be in place, then old iptables-legacy and new nftables rules would have no visibility each other.

With iptables-nft we have a way to move forward:

- Replace nft_compat by native expressions from iptables-nft.
- Extend iptables-nft to understand more complex expressions, worst case dump a native representation.

Quoted 7 lines
Quoted 2 lines

Why don't we just move ahead this path instead of spinning around the compat layer? This only requires userspace updates on iptables-nft.

Sure! I'm just picking low hanging fruits first. With even translation support being still incomplete, I fear it will take a while until the tools are fluent enough for this to not matter anymore. And then there's still nftables without libxtables support.

Then perhaps its better to do following path:
1. Try ->xlate(), if that fails, then print a 'breaking' format?

As far as I understand the problem is the "# comment" - type syntax that makes nft just skip the incomplete rule, so perhaps just use invalid format?

Example:

counter packets 0 bytes 0 # name foo interval 250.0ms ewmalog 500.0ms
Instead make this something like
counter packets 0 bytes 0 nft_compat [ RATEEST name foo interval 250.0ms ewmalog 500.0ms ] # unsupported iptables-nft rule

?

I'd like to avoid exposure in the frontend with compatible-restore-approach if possible.

author reviewer
#12 lore raw
Quoted 50 lines · Florian Westphal
Quoted 32 lines · Phil Sutter
Quoted 31 lines · Pablo Neira Ayuso

Merging threads.

On Fri, Nov 18, 2022 at 10:55:04AM +0100, Phil Sutter wrote:
[...]

Quoted 6 lines
Quoted 1 line

I think this more or less a summary of what we discussed in the NFWS.

Pablo, I think you're mixing up two things here:

This "support dump and load of compat expression" feature is to sanitize the current situation with up to date iptables and nftables.

OK, then the problem we discuss is mixing iptables-nft and nftables.

On Fri, Nov 18, 2022 at 10:47:48AM +0100, Phil Sutter wrote:
[...]

Quoted 5 lines
Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

If you do, please convince Pablo to not push iptables commit 384958620a. I think it opened the can of worms we're trying to confine here.

It could be worst, if iptables-nft would not be in place, then old iptables-legacy and new nftables rules would have no visibility each other.

With iptables-nft we have a way to move forward:

- Replace nft_compat by native expressions from iptables-nft.
- Extend iptables-nft to understand more complex expressions, worst case dump a native representation.

Quoted 7 lines
Quoted 2 lines

Why don't we just move ahead this path instead of spinning around the compat layer? This only requires userspace updates on iptables-nft.

Sure! I'm just picking low hanging fruits first. With even translation support being still incomplete, I fear it will take a while until the tools are fluent enough for this to not matter anymore. And then there's still nftables without libxtables support.

Then perhaps its better to do following path:
1. Try ->xlate(), if that fails, then print a 'breaking' format?

As far as I understand the problem is the "# comment" - type syntax that makes nft just skip the incomplete rule, so perhaps just use invalid format?

Example:

Quoted 7 lines

counter packets 0 bytes 0 # name foo interval 250.0ms ewmalog 500.0ms
Instead make this something like
counter packets 0 bytes 0 nft_compat [ RATEEST name foo interval 250.0ms ewmalog 500.0ms ] # unsupported iptables-nft rule

?

I'd like to avoid exposure in the frontend with compatible-restore-approach if possible.

Yes, that's fine with me. Now what about translated expressions? Can we apply my warning patch until at least the majority of them is understood by iptables?

Cheers, Phil

#13 lore raw
Quoted 55 lines · Phil Sutter
Quoted 54 lines · Florian Westphal
Quoted 32 lines · Phil Sutter
Quoted 31 lines · Pablo Neira Ayuso

Merging threads.

On Fri, Nov 18, 2022 at 10:55:04AM +0100, Phil Sutter wrote:
[...]

Quoted 6 lines
Quoted 1 line

I think this more or less a summary of what we discussed in the NFWS.

Pablo, I think you're mixing up two things here:

This "support dump and load of compat expression" feature is to sanitize the current situation with up to date iptables and nftables.

OK, then the problem we discuss is mixing iptables-nft and nftables.

On Fri, Nov 18, 2022 at 10:47:48AM +0100, Phil Sutter wrote:
[...]

Quoted 5 lines
Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

If you do, please convince Pablo to not push iptables commit 384958620a. I think it opened the can of worms we're trying to confine here.

It could be worst, if iptables-nft would not be in place, then old iptables-legacy and new nftables rules would have no visibility each other.

With iptables-nft we have a way to move forward:

- Replace nft_compat by native expressions from iptables-nft.
- Extend iptables-nft to understand more complex expressions, worst case dump a native representation.

Quoted 7 lines
Quoted 2 lines

Why don't we just move ahead this path instead of spinning around the compat layer? This only requires userspace updates on iptables-nft.

Sure! I'm just picking low hanging fruits first. With even translation support being still incomplete, I fear it will take a while until the tools are fluent enough for this to not matter anymore. And then there's still nftables without libxtables support.

Then perhaps its better to do following path:
1. Try ->xlate(), if that fails, then print a 'breaking' format?

As far as I understand the problem is the "# comment" - type syntax that makes nft just skip the incomplete rule, so perhaps just use invalid format?

Example:

counter packets 0 bytes 0 # name foo interval 250.0ms ewmalog 500.0ms
Instead make this something like
counter packets 0 bytes 0 nft_compat [ RATEEST name foo interval 250.0ms ewmalog 500.0ms ] # unsupported iptables-nft rule

Quoted 7 lines
Quoted 3 lines

?

I'd like to avoid exposure in the frontend with compatible-restore-approach if possible.

Yes, that's fine with me. Now what about translated expressions? Can we apply my warning patch until at least the majority of them is understood by iptables?

Which one are you refering to?

author reviewer
#14 lore raw
Quoted 58 lines · Pablo Neira Ayuso
Quoted 57 lines · Phil Sutter
Quoted 56 lines · Florian Westphal
Quoted 32 lines · Phil Sutter
Quoted 31 lines · Pablo Neira Ayuso

Merging threads.

On Fri, Nov 18, 2022 at 10:55:04AM +0100, Phil Sutter wrote:
[...]

Quoted 6 lines
Quoted 1 line

I think this more or less a summary of what we discussed in the NFWS.

Pablo, I think you're mixing up two things here:

This "support dump and load of compat expression" feature is to sanitize the current situation with up to date iptables and nftables.

OK, then the problem we discuss is mixing iptables-nft and nftables.

On Fri, Nov 18, 2022 at 10:47:48AM +0100, Phil Sutter wrote:
[...]

Quoted 5 lines
Quoted 2 lines

At this time I'd rather like a time machine to prevent nft_compat.c from getting merged :-(

If you do, please convince Pablo to not push iptables commit 384958620a. I think it opened the can of worms we're trying to confine here.

It could be worst, if iptables-nft would not be in place, then old iptables-legacy and new nftables rules would have no visibility each other.

With iptables-nft we have a way to move forward:

- Replace nft_compat by native expressions from iptables-nft.
- Extend iptables-nft to understand more complex expressions, worst case dump a native representation.

Quoted 7 lines
Quoted 2 lines

Why don't we just move ahead this path instead of spinning around the compat layer? This only requires userspace updates on iptables-nft.

Sure! I'm just picking low hanging fruits first. With even translation support being still incomplete, I fear it will take a while until the tools are fluent enough for this to not matter anymore. And then there's still nftables without libxtables support.

Then perhaps its better to do following path:
1. Try ->xlate(), if that fails, then print a 'breaking' format?

As far as I understand the problem is the "# comment" - type syntax that makes nft just skip the incomplete rule, so perhaps just use invalid format?

Example:

counter packets 0 bytes 0 # name foo interval 250.0ms ewmalog 500.0ms
Instead make this something like
counter packets 0 bytes 0 nft_compat [ RATEEST name foo interval 250.0ms ewmalog 500.0ms ] # unsupported iptables-nft rule

?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I'd like to avoid exposure in the frontend with compatible-restore-approach if possible.

Yes, that's fine with me. Now what about translated expressions? Can we apply my warning patch until at least the majority of them is understood by iptables?

Which one are you refering to?

This one:

Subject: [nft PATCH] Warn for tables with compat expressions in rules
Date: Wed, 12 Oct 2022 17:31:07 +0200
Message-Id: <[email protected]>