Skip to contentSkip to conversation

nf_tables: follow-up on audit fix, propose kselftest

PATCH Superseded Phil Sutter 4 participants 7 messages Sep 8, 2023 – Sep 12, 2023

Subsystems (4)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Phil Sutter

author reviewer
#0 lore raw

cover nf_tables: follow-up on audit fix, propose kselftest

Patch 1 fixes/improves Pablo's fix for the bug I built into nf_tables' audit support code. I propose it for nf, but nf-next is OK from my PoV if it is deemed too risky.

Patch 2 adds a selftest for the audit notifications in nf_tables. Since audit logging can not be turned off, it may cause problems requiring a reboot to resolve. So I'd like to hear your opinions first before attempting an "official" proposal.

Phil Sutter (2):
netfilter: nf_tables: Fix entries val in rule reset audit log
selftests: netfilter: Test nf_tables audit logging

 net/netfilter/nf_tables_api.c                 | 21 ++++--
 .../testing/selftests/netfilter/nft_audit.sh  | 75 +++++++++++++++++++
 2 files changed, 88 insertions(+), 8 deletions(-)
 create mode 100755 tools/testing/selftests/netfilter/nft_audit.sh
Signature · 1 line
2.41.0
author reviewer
#1 lore raw

PATCH 1/2 netfilter: nf_tables: Fix entries val in rule reset audit log

The value in idx and the number of rules handled in that particular __nf_tables_dump_rules() call is not identical. The former is a cursor to pick up from if multiple netlink messages are needed, so its value is ever increasing. Fixing this is not just a matter of subtracting s_idx from it, though: When resetting rules in multiple chains, __nf_tables_dump_rules() is called for each and cb->args[0] is not adjusted in between.

The audit notification in __nf_tables_dump_rules() had another problem:
If nf_tables_fill_rule_info() failed (e.g. due to buffer exhaustion), no notification was sent - despite the rules having been reset already.

To catch all the above and return to a single (if possible) notification per table again, move audit logging back into the caller but into the table loop instead of past it to avoid the potential null-pointer deref.

This requires to trigger the notification in two spots. Care has to be taken in the second case as cb->args[0] is also not updated in between tables. This requires a helper variable as either it is the first table (with potential non-zero cb->args[0] cursor) or a consecutive one (with idx holding the current cursor already).

Fixes 9b5ba5c9c5109 ("netfilter: nf_tables: Unbreak audit log reset")
---
 net/netfilter/nf_tables_api.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
net/netfilter/nf_tables_api.c +13 -8
index e429ebba74b3d..d429270676131 100644
@@ -3481,9 +3481,6 @@ static int __nf_tables_dump_rules(struct sk_buff *skb,
 		(*idx)++;
 	}
 
-	if (reset && *idx)
-		audit_log_rule_reset(table, cb->seq, *idx);
-
 	return 0;
 }
 
@@ -3494,11 +3491,12 @@ static int nf_tables_dump_rules(struct sk_buff *skb,
 	const struct nft_rule_dump_ctx *ctx = cb->data;
 	struct nft_table *table;
 	const struct nft_chain *chain;
-	unsigned int idx = 0;
+	unsigned int idx = 0, s_idx;
 	struct net *net = sock_net(skb->sk);
 	int family = nfmsg->nfgen_family;
 	struct nftables_pernet *nft_net;
 	bool reset = false;
+	int ret;
 
 	if (NFNL_MSG_TYPE(cb->nlh->nlmsg_type) == NFT_MSG_GETRULE_RESET)
 		reset = true;
@@ -3529,16 +3527,23 @@ static int nf_tables_dump_rules(struct sk_buff *skb,
 						       cb, table, chain, reset);
 				break;
 			}
+			if (reset && idx > cb->args[0])
+				audit_log_rule_reset(table, cb->seq,
+						     idx - cb->args[0]);
 			goto done;
 		}
 
+		s_idx = max(idx, cb->args[0]);
 		list_for_each_entry_rcu(chain, &table->chains, list) {
-			if (__nf_tables_dump_rules(skb, &idx,
-						   cb, table, chain, reset))
-				goto done;
+			ret = __nf_tables_dump_rules(skb, &idx,
+						     cb, table, chain, reset);
+			if (ret)
+				break;
 		}
+		if (reset && idx > s_idx)
+			audit_log_rule_reset(table, cb->seq, idx - s_idx);
 
-		if (ctx && ctx->table)
+		if ((ctx && ctx->table) || ret)
 			break;
 	}
 done:
Signature · 1 line
2.41.0
#2 lore raw

Re: [nf PATCH 1/2] netfilter: nf_tables: Fix entries val in rule reset audit log

kernel test robot noticed the following build warnings:

Hi Phil,

kernel test robot noticed the following build warnings:

[auto build test WARNING on netfilter-nf/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Phil-Sutter/netfilter-nf_tables-Fix-entries-val-in-rule-reset-audit-log/20230908-082530
base:   git://git.kernel.org/pub/scm/linux/kernel/git/netfilter/nf.git main
patch link:    https://lore.kernel.org/r/20230908002229.1409-2-phil%40nwl.cc
patch subject: [nf PATCH 1/2] netfilter: nf_tables: Fix entries val in rule reset audit log
config: mips-randconfig-r002-20230908 (https://download.01.org/0day-ci/archive/20230908/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230908/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> net/netfilter/nf_tables_api.c:3536:11: warning: comparison of distinct pointer types ('typeof (idx) *' (aka 'unsigned int *') and 'typeof (cb->args[0]) *' (aka 'long *')) [-Wcompare-distinct-pointer-types]
    3536 |                 s_idx = max(idx, cb->args[0]);
         |                         ^~~~~~~~~~~~~~~~~~~~~
   include/linux/minmax.h:74:19: note: expanded from macro 'max'
      74 | #define max(x, y)       __careful_cmp(x, y, >)
         |                         ^~~~~~~~~~~~~~~~~~~~~~
   include/linux/minmax.h:36:24: note: expanded from macro '__careful_cmp'
      36 |         __builtin_choose_expr(__safe_cmp(x, y), \
         |                               ^~~~~~~~~~~~~~~~
   include/linux/minmax.h:26:4: note: expanded from macro '__safe_cmp'
      26 |                 (__typecheck(x, y) && __no_side_effects(x, y))
         |                  ^~~~~~~~~~~~~~~~~
   include/linux/minmax.h:20:28: note: expanded from macro '__typecheck'
      20 |         (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
         |                    ~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~

1 warning generated.

vim +3536 net/netfilter/nf_tables_api.c

3486
3487 static int nf_tables_dump_rules(struct sk_buff *skb,
3488 struct netlink_callback *cb)
3489 {
3490 const struct nfgenmsg *nfmsg = nlmsg_data(cb->nlh);
3491 const struct nft_rule_dump_ctx *ctx = cb->data;
3492 struct nft_table *table;
3493 const struct nft_chain *chain;
3494 unsigned int idx = 0, s_idx;
3495 struct net *net = sock_net(skb->sk);
3496 int family = nfmsg->nfgen_family;
3497 struct nftables_pernet *nft_net;
3498 bool reset = false;
3499 int ret;
3500
3501 if (NFNL_MSG_TYPE(cb->nlh->nlmsg_type) == NFT_MSG_GETRULE_RESET)
3502 reset = true;
3503
3504 rcu_read_lock();
3505 nft_net = nft_pernet(net);
3506 cb->seq = READ_ONCE(nft_net->base_seq);
3507
3508 list_for_each_entry_rcu(table, &nft_net->tables, list) {
3509 if (family != NFPROTO_UNSPEC && family != table->family)
3510 continue;
3511
3512 if (ctx && ctx->table && strcmp(ctx->table, table->name) != 0)
3513 continue;
3514
3515 if (ctx && ctx->table && ctx->chain) {
3516 struct rhlist_head *list, *tmp;
3517
3518 list = rhltable_lookup(&table->chains_ht, ctx->chain,
3519 nft_chain_ht_params);
3520 if (!list)
3521 goto done;
3522
3523 rhl_for_each_entry_rcu(chain, tmp, list, rhlhead) {
3524 if (!nft_is_active(net, chain))
3525 continue;
3526 __nf_tables_dump_rules(skb, &idx,
3527 cb, table, chain, reset);
3528 break;
3529 }
3530 if (reset && idx > cb->args[0])
3531 audit_log_rule_reset(table, cb->seq,
3532 idx - cb->args[0]);
3533 goto done;
3534 }
3535

> 3536			s_idx = max(idx, cb->args[0]);

3537 list_for_each_entry_rcu(chain, &table->chains, list) {
3538 ret = __nf_tables_dump_rules(skb, &idx,
3539 cb, table, chain, reset);
3540 if (ret)
3541 break;
3542 }
3543 if (reset && idx > s_idx)
3544 audit_log_rule_reset(table, cb->seq, idx - s_idx);
3545
3546 if ((ctx && ctx->table) || ret)
3547 break;
3548 }
3549 done:
3550 rcu_read_unlock();
3551
3552 cb->args[0] = idx;
3553 return skb->len;
3554 }
3555

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author reviewer
#3 lore raw

RFC 2/2 selftests: netfilter: Test nf_tables audit logging

In reply to Phil Sutter

Perform ruleset modifications and compare the NETFILTER_CFG type notifications emitted by auditd match expectations.

---
Calling auditd means enabling audit logging in kernel for the remaining uptime. So this test will slow down following ones or even cause spurious failures due to unexpected kernel log entries, timeouts, etc.

Is there a way to test this in a less intrusive way? Maybe fence this test so it does not run automatically (is it any good having it in kernel then)?

---
 .../testing/selftests/netfilter/nft_audit.sh  | 75 +++++++++++++++++++
 1 file changed, 75 insertions(+)
 create mode 100755 tools/testing/selftests/netfilter/nft_audit.sh
tools/testing/selftests/netfilter/nft_audit.sh new file +75 -0
new file mode 100755
index 0000000000000..55c750720137f
@@ -0,0 +1,75 @@
+#!/bin/bash
+
+SKIP_RC=4
+RC=0
+
+nft --version >/dev/null 2>&1 || {
+	echo "SKIP: missing nft tool"
+	exit $SKIP_RC
+}
+
+auditd --help >/dev/null 2>&1
+[ $? -eq 2 ] || {
+	echo "SKIP: missing auditd tool"
+	exit $SKIP_RC
+}
+
+tmpdir=$(mktemp -d)
+audit_log="$tmpdir/audit.log"
+cat >"$tmpdir/auditd.conf" <<EOF
+write_logs = no
+space_left = 75
+EOF
+auditd -f -c "$tmpdir" >"$audit_log" &
+audit_pid=$!
+trap 'kill $audit_pid; rm -rf $tmpdir' EXIT
+sleep 1
+
+logread() {
+	grep 'type=NETFILTER_CFG' "$audit_log" | \
+		sed -e 's/\(type\|msg\|pid\)=[^ ]* //g' \
+		    -e 's/\(table=[^:]*\):[0-9]*/\1/'
+}
+
+do_test() { # (cmd, log)
+	echo -n "testing for cmd: $1 ... "
+	echo >"$audit_log"
+	$1 >/dev/null || exit 1
+	diff -q <(echo "$2") <(logread) >/dev/null && { echo "OK"; return; }
+	echo "FAIL"
+	diff -u <(echo "$2") <(logread)
+	((RC++))
+}
+
+nft flush ruleset
+
+for table in t1 t2; do
+	echo "add table $table"
+	for chain in c1 c2 c3; do
+		echo "add chain $table $chain"
+		echo "add rule $table $chain counter accept"
+		echo "add rule $table $chain counter accept"
+		echo "add rule $table $chain counter accept"
+	done
+done | nft -f - || exit 1
+
+do_test 'nft reset rules t1 c2' \
+	'table=t1 family=2 entries=3 op=nft_reset_rule subj=kernel comm="nft"'
+
+do_test 'nft reset rules table t1' \
+	'table=t1 family=2 entries=9 op=nft_reset_rule subj=kernel comm="nft"'
+
+do_test 'nft reset rules' \
+	'table=t1 family=2 entries=9 op=nft_reset_rule subj=kernel comm="nft"
+table=t2 family=2 entries=9 op=nft_reset_rule subj=kernel comm="nft"'
+
+for ((i = 0; i < 500; i++)); do
+	echo "add rule t2 c3 counter accept comment \"rule $i\""
+done | nft -f - || exit 1
+
+do_test 'nft reset rules t2 c3' \
+	'table=t2 family=2 entries=189 op=nft_reset_rule subj=kernel comm="nft"
+table=t2 family=2 entries=188 op=nft_reset_rule subj=kernel comm="nft"
+table=t2 family=2 entries=126 op=nft_reset_rule subj=kernel comm="nft"'
+
+exit $RC
Signature · 1 line
2.41.0
#4 lore raw

Re: [nf-next RFC 2/2] selftests: netfilter: Test nf_tables audit logging

Quoted 4 lines · Phil Sutter

Perform ruleset modifications and compare the NETFILTER_CFG type notifications emitted by auditd match expectations.

Quoted 8 lines

---
Calling auditd means enabling audit logging in kernel for the remaining uptime. So this test will slow down following ones or even cause spurious failures due to unexpected kernel log entries, timeouts, etc.

Is there a way to test this in a less intrusive way? Maybe fence this test so it does not run automatically (is it any good having it in kernel then)?

I think you could make a small libmnl program to listen to NETLINK_AUDIT events and filter only the logs you need from there. We already have a few programs like this in the selftest folder.

Quoted 89 lines
---
 .../testing/selftests/netfilter/nft_audit.sh  | 75 +++++++++++++++++++
 1 file changed, 75 insertions(+)
 create mode 100755 tools/testing/selftests/netfilter/nft_audit.sh
tools/testing/selftests/netfilter/nft_audit.sh new file +75 -0
new file mode 100755
index 0000000000000..55c750720137f
@@ -0,0 +1,75 @@
+#!/bin/bash
+
+SKIP_RC=4
+RC=0
+
+nft --version >/dev/null 2>&1 || {
+	echo "SKIP: missing nft tool"
+	exit $SKIP_RC
+}
+
+auditd --help >/dev/null 2>&1
+[ $? -eq 2 ] || {
+	echo "SKIP: missing auditd tool"
+	exit $SKIP_RC
+}
+
+tmpdir=$(mktemp -d)
+audit_log="$tmpdir/audit.log"
+cat >"$tmpdir/auditd.conf" <<EOF
+write_logs = no
+space_left = 75
+EOF
+auditd -f -c "$tmpdir" >"$audit_log" &
+audit_pid=$!
+trap 'kill $audit_pid; rm -rf $tmpdir' EXIT
+sleep 1
+
+logread() {
+	grep 'type=NETFILTER_CFG' "$audit_log" | \
+		sed -e 's/\(type\|msg\|pid\)=[^ ]* //g' \
+		    -e 's/\(table=[^:]*\):[0-9]*/\1/'
+}
+
+do_test() { # (cmd, log)
+	echo -n "testing for cmd: $1 ... "
+	echo >"$audit_log"
+	$1 >/dev/null || exit 1
+	diff -q <(echo "$2") <(logread) >/dev/null && { echo "OK"; return; }
+	echo "FAIL"
+	diff -u <(echo "$2") <(logread)
+	((RC++))
+}
+
+nft flush ruleset
+
+for table in t1 t2; do
+	echo "add table $table"
+	for chain in c1 c2 c3; do
+		echo "add chain $table $chain"
+		echo "add rule $table $chain counter accept"
+		echo "add rule $table $chain counter accept"
+		echo "add rule $table $chain counter accept"
+	done
+done | nft -f - || exit 1
+
+do_test 'nft reset rules t1 c2' \
+	'table=t1 family=2 entries=3 op=nft_reset_rule subj=kernel comm="nft"'
+
+do_test 'nft reset rules table t1' \
+	'table=t1 family=2 entries=9 op=nft_reset_rule subj=kernel comm="nft"'
+
+do_test 'nft reset rules' \
+	'table=t1 family=2 entries=9 op=nft_reset_rule subj=kernel comm="nft"
+table=t2 family=2 entries=9 op=nft_reset_rule subj=kernel comm="nft"'
+
+for ((i = 0; i < 500; i++)); do
+	echo "add rule t2 c3 counter accept comment \"rule $i\""
+done | nft -f - || exit 1
+
+do_test 'nft reset rules t2 c3' \
+	'table=t2 family=2 entries=189 op=nft_reset_rule subj=kernel comm="nft"
+table=t2 family=2 entries=188 op=nft_reset_rule subj=kernel comm="nft"
+table=t2 family=2 entries=126 op=nft_reset_rule subj=kernel comm="nft"'
+
+exit $RC
Signature · 1 line
2.41.0
author reviewer
#5 lore raw

Re: [nf-next RFC 2/2] selftests: netfilter: Test nf_tables audit logging

Quoted 10 lines · Pablo Neira Ayuso
Quoted 9 lines · Phil Sutter

Perform ruleset modifications and compare the NETFILTER_CFG type notifications emitted by auditd match expectations.

---
Calling auditd means enabling audit logging in kernel for the remaining uptime. So this test will slow down following ones or even cause spurious failures due to unexpected kernel log entries, timeouts, etc.

Quoted 7 lines
Quoted 3 lines

Is there a way to test this in a less intrusive way? Maybe fence this test so it does not run automatically (is it any good having it in kernel then)?

I think you could make a small libmnl program to listen to NETLINK_AUDIT events and filter only the logs you need from there. We already have a few programs like this in the selftest folder.

Turns out it is indeed possible to turn audit logging off again. I was obviously misled from auditd not doing it (when killed at least). Calling 'auditctl -e 0' inside the EXIT trap does the trick. Implementing a custom audit listener tailored to our case is probably still a good idea, but at least the biggest obstacle is gone IMO.

Thanks, Phil

maintainer
#6 lore raw

Re: [nf-next RFC 2/2] selftests: netfilter: Test nf_tables audit logging

In reply to Pablo Neira Ayuso
Quoted 10 lines · Pablo Neira Ayuso
Quoted 9 lines · Phil Sutter

Perform ruleset modifications and compare the NETFILTER_CFG type notifications emitted by auditd match expectations.

---
Calling auditd means enabling audit logging in kernel for the remaining uptime. So this test will slow down following ones or even cause spurious failures due to unexpected kernel log entries, timeouts, etc.

Quoted 7 lines
Quoted 3 lines

Is there a way to test this in a less intrusive way? Maybe fence this test so it does not run automatically (is it any good having it in kernel then)?

I think you could make a small libmnl program to listen to NETLINK_AUDIT events and filter only the logs you need from there. We already have a few programs like this in the selftest folder.

Just a heads-up that the kernel sends the unicast netlink messages with a bogus nlmsghdr::nlmsg_len field, see the comments in audit_log_end() and kauditd_send_multicast_skb() for the details.

Signature · 1 line
paul-moore.com