netfilter: nf_tables: reduce set element transaction size
Subsystems (2)
Florian Westphal
maintainerv4 cover netfilter: nf_tables: reduce set element transaction size
v4: fix a typo in patch 3 commit message.
rebase on nf-next:main.
No other changes.
Changes in v3:
I failed to realize that nft_audit leaks one implementation detail to userspace: the length of the transaction log.
This gets fixed by patch 3 which adds needed helper to increment the count variable by the number of elements carried by the compacted set update.
Also fix up notifications, for update case, notifications were skipped but currently newsetelem notifications are done even if existing set element is updated.
Most patches are unchanged.
"prefer nft_trans_elem_alloc helper" is already upstreamed so its dropped from this batch.
v2: only change is in patch 3, and by extension, the last one:
During transaction abort, we need to handle an aggregate container to contain both new set elements and updates. The latter must be skipped, else we remove element that already existed at start of the transaction.
original cover letter:
When doing a flush on a set or mass adding/removing elements from a set, each element needs to allocate 96 bytes to hold the transactional state.
In such cases, virtually all the information in struct nft_trans_elem is the same.
Change nft_trans_elem to a flex-array, i.e. a single nft_trans_elem can hold multiple set element pointers.
The number of elements that can be stored in one nft_trans_elem is limited by the slab allocator, this series limits the compaction to at most 62 elements as it caps the reallocation to 2048 bytes of memory.
Florian Westphal (5):
netfilter: nf_tables: add nft_trans_commit_list_add_elem helper
netfilter: nf_tables: prepare for multiple elements in nft_trans_elem
structure
netfilter: nf_tables: preemptive fix for audit selftest failure
netfilter: nf_tables: switch trans_elem to real flex array
netfilter: nf_tables: allocate element update information dynamically
include/net/netfilter/nf_tables.h | 25 +- net/netfilter/nf_tables_api.c | 368 +++++++++++++++++++++++------- 2 files changed, 304 insertions(+), 89 deletions(-)
Signature · 1 line
2.45.2
Florian Westphal
maintainerPATCH v4 1/5 netfilter: nf_tables: add nft_trans_commit_list_add_elem helper
Add and use a wrapper to append trans_elem structures to the transaction log.
Unlike the existing helper, pass a gfp_t to indicate if sleeping is allowed.
This will be used by a followup patch to realloc nft_trans_elem structures after they gain a flexible array member to reduce number of such container structures on the transaction list.
--- net/netfilter/nf_tables_api.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
net/netfilter/nf_tables_api.c +16 -5
@@ -421,6 +421,17 @@ static void nft_trans_commit_list_add_tail(struct net *net, struct nft_trans *tr } } +static void nft_trans_commit_list_add_elem (struct net *net,struct nft_trans *trans, +gfp_t gfp) +{ +WARN_ON_ONCE (trans->msg_type !=NFT_MSG_NEWSETELEM && + trans->msg_type !=NFT_MSG_DELSETELEM ); + +might_alloc (gfp); + +nft_trans_commit_list_add_tail (net, trans); +} +static int nft_trans_table_add (struct nft_ctx *ctx,int msg_type) {struct nft_trans *trans;
@@ -7204,7 +7215,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,if (update_flags) {nft_trans_elem_priv (trans) = elem_priv;nft_trans_elem_update_flags (trans) = update_flags; -nft_trans_commit_list_add_tail (ctx->net, trans); +nft_trans_commit_list_add_elem (ctx->net, trans,GFP_KERNEL ); goto err_elem_free; } }
@@ -7228,7 +7239,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, }nft_trans_elem_priv (trans) = elem.priv; -nft_trans_commit_list_add_tail (ctx->net, trans); +nft_trans_commit_list_add_elem (ctx->net, trans,GFP_KERNEL );return 0 ; err_set_full:
@@ -7445,7 +7456,7 @@ static int nft_del_setelem(struct nft_ctx *ctx, struct nft_set *set,nft_setelem_data_deactivate (ctx->net, set, elem.priv);nft_trans_elem_priv (trans) = elem.priv; -nft_trans_commit_list_add_tail (ctx->net, trans); +nft_trans_commit_list_add_elem (ctx->net, trans,GFP_KERNEL );return 0 ; fail_ops:
@@ -7481,7 +7492,7 @@ static int nft_setelem_flush(const struct nft_ctx *ctx,nft_setelem_data_deactivate (ctx->net, set, elem_priv);nft_trans_elem_set (trans) = set;nft_trans_elem_priv (trans) = elem_priv; -nft_trans_commit_list_add_tail (ctx->net, trans); +nft_trans_commit_list_add_elem (ctx->net, trans,GFP_ATOMIC );return 0 ; }
@@ -7498,7 +7509,7 @@ static int __nft_set_catchall_flush(const struct nft_ctx *ctx,nft_setelem_data_deactivate (ctx->net, set, elem_priv);nft_trans_elem_priv (trans) = elem_priv; -nft_trans_commit_list_add_tail (ctx->net, trans); +nft_trans_commit_list_add_elem (ctx->net, trans,GFP_KERNEL );return 0 ; }
Signature · 1 line
2.45.2
Florian Westphal
maintainerPATCH v4 2/5 netfilter: nf_tables: prepare for multiple elements in nft_trans_elem structure
In reply to Florian WestphalAdd helpers to release the individual elements contained in the trans_elem container structure.
No functional change intended.
Followup patch will add 'nelems' member and will turn 'priv' into a flexible array.
These helpers can then loop over all elements.
Care needs to be taken to handle a mix of new elements and existing elements that are being updated (e.g. timeout refresh).
Before this patch, NEWSETELEM transaction with update is released early so nft_trans_set_elem_destroy() won't get called, so we need to skip elements marked as update.
--- include/net/netfilter/nf_tables.h | 21 ++- net/netfilter/nf_tables_api.c | 228 +++++++++++++++++++++--------- 2 files changed, 173 insertions(+), 76 deletions(-)
include/net/netfilter/nf_tables.h +9 -12
@@ -1755,28 +1755,25 @@ enum nft_trans_elem_flags { NFT_TRANS_UPD_EXPIRATION = (1 << 1), }; -struct nft_trans_elem { -struct nft_trans nft_trans; -struct nft_set *set; -struct nft_elem_priv *elem_priv; +struct nft_trans_one_elem { +struct nft_elem_priv *priv;u64 timeout;u64 expiration;u8 update_flags; +}; + +struct nft_trans_elem { +struct nft_trans nft_trans; +struct nft_set *set;bool bound; +unsigned int nelems; + struct nft_trans_one_elem elems[] __counted_by(nelems); };#define nft_trans_container_elem (t) \ container_of(t, struct nft_trans_elem, nft_trans)#define nft_trans_elem_set (trans) \ nft_trans_container_elem(trans)->set -#define nft_trans_elem_priv (trans) \ - nft_trans_container_elem(trans)->elem_priv -#define nft_trans_elem_update_flags (trans) \ - nft_trans_container_elem(trans)->update_flags -#define nft_trans_elem_timeout (trans) \ - nft_trans_container_elem(trans)->timeout -#define nft_trans_elem_expiration (trans) \ - nft_trans_container_elem(trans)->expiration#define nft_trans_elem_set_bound (trans) \ nft_trans_container_elem(trans)->bound
net/netfilter/nf_tables_api.c +164 -64
372 lines not on this page · raw message
Signature · 1 line
2.45.2
Florian Westphal
maintainerPATCH v4 3/5 netfilter: nf_tables: preemptive fix for audit selftest failure
In reply to Florian Westphalnftables audit log format unfortunately leaks an implementation detail, the transaction log size, to userspace:
table=t1 family=2 entries=4 op=nft_register_set
~~~~~~~~~
This 'entries' key is the number of transactions that will be applied. The upcoming set element compression (add elem x to set s, add element y to s would be placed in a single transaction request) would lower that number to 3.
~ncrement the audit counter by the number of elements to keep the reported entries value the same.
Without this, nft_audit.sh selftest fails because the recorded (expected) entries key is smaller than the expected one.
--- net/netfilter/nf_tables_api.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
net/netfilter/nf_tables_api.c +20 -3
@@ -10397,9 +10397,26 @@ static void nf_tables_commit_audit_free(struct list_head *adl) } } +/* Silly, but existing test audit test cases require a count +* value derived from the (INTERNAL!) transaction log length. +* +* Thus, compaction of NEW/DELSETELEM breaks such tests. +*/ +static unsigned int nf_tables_commit_audit_entrycount (const struct nft_trans *trans) +{ +switch (trans->msg_type) { +case NFT_MSG_NEWSETELEM : +case NFT_MSG_DELSETELEM : +return nft_trans_container_elem (trans)->nelems; + } + +return 1 ; +} +static void nf_tables_commit_audit_collect (struct list_head *adl, -struct nft_table *table,u32 op) +const struct nft_trans *trans,u32 op) { +const struct nft_table *table = trans->table;struct nft_audit_data *adp;list_for_each_entry (adp, adl, list) {
@@ -10409,7 +10426,7 @@ static void nf_tables_commit_audit_collect(struct list_head *adl,WARN_ONCE (1 ,"table=%s not expected in commit list" , table->name);return ; found: - adp->entries++; + adp->entries +=nf_tables_commit_audit_entrycount (trans);if (!adp->op || adp->op > op) adp->op = op; }
@@ -10568,7 +10585,7 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)nft_ctx_update (&ctx, trans); -nf_tables_commit_audit_collect (&adl, table, trans->msg_type); +nf_tables_commit_audit_collect (&adl, trans, trans->msg_type);switch (trans->msg_type) {case NFT_MSG_NEWTABLE :if (nft_trans_table_update (trans)) {
Signature · 1 line
2.45.2
Florian Westphal
maintainerPATCH v4 4/5 netfilter: nf_tables: switch trans_elem to real flex array
In reply to Florian WestphalWhen queueing a set element add or removal operation to the transaction log, check if the previous operation already asks for a the identical operation on the same set.
If so, store the element reference in the preceding operation. This significantlty reduces memory consumption when many set add/delete operations appear in a single transaction.
Example: 10k elements require 937kb of memory (10k allocations from kmalloc-96 slab).
Assuming we can compact 4 elements in the same set, 468 kbytes are needed (64 bytes for base struct, nft_trans_elemn, 32 bytes for nft_trans_one_elem structure, so 2500 allocations from kmalloc-192 slab).
For large batch updates we can compact up to 62 elements
into one single nft_trans_elem structure (~65% mem reduction):
(64 bytes for base struct, nft_trans_elem, 32 byte for nft_trans_one_elem
struct).
We can halve size of nft_trans_one_elem struct by moving timeout/expire/update_flags into a dynamically allocated structure, this allows to store 124 elements in a 2k slab nft_trans_elem struct. This is done in a followup patch.
--- net/netfilter/nf_tables_api.c | 71 +++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+)
net/netfilter/nf_tables_api.c +71 -0
@@ -25,6 +25,7 @@#define NFT_MODULE_AUTOLOAD_LIMIT (MODULE_NAME_LEN - sizeof("nft-expr-255-"))#define NFT_SET_MAX_ANONLEN 16 +#define NFT_MAX_SET_NELEMS ((2048 - sizeof(struct nft_trans_elem)) / sizeof(struct nft_trans_one_elem))unsigned int nf_tables_net_id__read_mostly ;
@@ -391,6 +392,69 @@ static void nf_tables_unregister_hook(struct net *net,return __nf_tables_unregister_hook (net, table, chain, false); } +static bool nft_trans_collapse_set_elem_allowed (const struct nft_trans_elem *a,const struct nft_trans_elem *b) +{ +return a->set == b->set && a->bound == b->bound && a->nelems <NFT_MAX_SET_NELEMS ; +} + +static bool nft_trans_collapse_set_elem (struct nftables_pernet *nft_net, +struct nft_trans_elem *tail, +struct nft_trans_elem *trans, +gfp_t gfp) +{ +unsigned int nelems, old_nelems = tail->nelems; +struct nft_trans_elem *new_trans; + +if (!nft_trans_collapse_set_elem_allowed (tail, trans)) +return false; + +if (WARN_ON_ONCE (trans->nelems !=1 )) +return false; + +if (check_add_overflow (old_nelems, trans->nelems, &nelems)) +return false; + +/* krealloc might free tail which invalidates list pointers */ +list_del_init (&tail->nft_trans.list); + + new_trans =krealloc (tail,struct_size (tail, elems, nelems), gfp); +if (!new_trans) { +list_add_tail (&tail->nft_trans.list, &nft_net->commit_list); +return false; + } + +INIT_LIST_HEAD (&new_trans->nft_trans.list); + new_trans->nelems = nelems; + new_trans->elems[old_nelems] = trans->elems[0 ]; +list_add_tail (&new_trans->nft_trans.list, &nft_net->commit_list); + +return true; +} + +static bool nft_trans_try_collapse (struct nftables_pernet *nft_net, +struct nft_trans *trans,gfp_t gfp) +{ +struct nft_trans *tail; + +if (list_empty (&nft_net->commit_list)) +return false; + + tail = list_last_entry(&nft_net->commit_list, struct nft_trans, list); + +if (tail->msg_type != trans->msg_type) +return false; + +switch (trans->msg_type) { +case NFT_MSG_NEWSETELEM : +case NFT_MSG_DELSETELEM : +return nft_trans_collapse_set_elem (nft_net, +nft_trans_container_elem (tail), +nft_trans_container_elem (trans), gfp); + } + +return false; +} +static void nft_trans_commit_list_add_tail (struct net *net,struct nft_trans *trans) {struct nftables_pernet *nft_net =nft_pernet (net);
@@ -424,11 +488,18 @@ static void nft_trans_commit_list_add_tail(struct net *net, struct nft_trans *trstatic void nft_trans_commit_list_add_elem (struct net *net,struct nft_trans *trans,gfp_t gfp) { +struct nftables_pernet *nft_net =nft_pernet (net); +WARN_ON_ONCE (trans->msg_type !=NFT_MSG_NEWSETELEM && trans->msg_type !=NFT_MSG_DELSETELEM );might_alloc (gfp); +if (nft_trans_try_collapse (nft_net, trans, gfp)) { +kfree (trans); +return ; + } +nft_trans_commit_list_add_tail (net, trans); }
Signature · 1 line
2.45.2
Pablo Neira Ayuso
maintainerRe: [PATCH nf-next v4 4/5] netfilter: nf_tables: switch trans_elem to real flex array
Hi Florian,
I'm making another pass on this series, a few thing I would like to ask, see below.
net/netfilter/nf_tables_api.c line 28
Quoted 9 lines · Florian Westphal
net/netfilter/nf_tables_api.c +1
-0@@ -25,6 +25,7 @@#define NFT_MODULE_AUTOLOAD_LIMIT (MODULE_NAME_LEN - sizeof("nft-expr-255-"))#define NFT_SET_MAX_ANONLEN 16 +#define NFT_MAX_SET_NELEMS ((2048 - sizeof(struct nft_trans_elem)) / sizeof(struct nft_trans_one_elem))
This NFT_MAX_SET_NELEMS is to stay in a specific kmalloc-X?
What is the logic behind this NFT_MAX_SET_NELEMS?
net/netfilter/nf_tables_api.c static void nf_tables_unregister_hook(struct net *net, · line 397
Quoted 9 lines
net/netfilter/nf_tables_api.c +3
-0unsigned int nf_tables_net_id__read_mostly ;@@ -391,6 +392,69 @@ static void nf_tables_unregister_hook(struct net *net,return __nf_tables_unregister_hook (net, table, chain, false); } +static bool nft_trans_collapse_set_elem_allowed (const struct nft_trans_elem *a,const struct nft_trans_elem *b) +{ +return a->set == b->set && a->bound == b->bound && a->nelems <NFT_MAX_SET_NELEMS ;
I think this a->bound == b->bound check defensive.
This code is collapsing only two consecutive transactions, the one at the tail (where nelems > 1) and the new transaction (where nelems == 1).
bound state should only change in case there is a NEWRULE transaction in between.
I am trying to find a error scenario where a->bound == b->bound evaluates false. I considered the following:
newelem -> newrule -> newelem
where newrule has these expressions:
lookup -> error
in this case, newrule error path is exercised:
nft_rule_expr_deactivate(&ctx, rule, NFT_TRANS_PREPARE_ERROR);
this calls nf_tables_deactivate_set() that calls nft_set_trans_unbind(), then a->bound is restored to false. Rule is released and no transaction is added.
Because if this succeeds:
newelem -> newrule -> newelem
then no element collapsing can happen, because we only collapse what is at the tail.
TLDR; Check does not harm, but it looks unlikely to happen to me.
one more comment below.
Quoted 19 lines
net/netfilter/nf_tables_api.c +19
-0+} + +static bool nft_trans_collapse_set_elem (struct nftables_pernet *nft_net, +struct nft_trans_elem *tail, +struct nft_trans_elem *trans, +gfp_t gfp) +{ +unsigned int nelems, old_nelems = tail->nelems; +struct nft_trans_elem *new_trans; + +if (!nft_trans_collapse_set_elem_allowed (tail, trans)) +return false; + +if (WARN_ON_ONCE (trans->nelems !=1 )) +return false; + +if (check_add_overflow (old_nelems, trans->nelems, &nelems)) +return false; +
net/netfilter/nf_tables_api.c static void nf_tables_unregister_hook(struct net *net, · line 426
Quoted 10 lines
net/netfilter/nf_tables_api.c +10
-0+/* krealloc might free tail which invalidates list pointers */ +list_del_init (&tail->nft_trans.list); + + new_trans =krealloc (tail,struct_size (tail, elems, nelems), gfp); +if (!new_trans) { +list_add_tail (&tail->nft_trans.list, &nft_net->commit_list); +return false; + } + +INIT_LIST_HEAD (&new_trans->nft_trans.list);
This initialization is also defensive, this element is added via list_add_tail().
Quoted 6 lines
net/netfilter/nf_tables_api.c +6
-0+ new_trans->nelems = nelems; + new_trans->elems[old_nelems] = trans->elems[0 ]; +list_add_tail (&new_trans->nft_trans.list, &nft_net->commit_list); + +return true; +}
Florian Westphal
maintainerRe: [PATCH nf-next v4 4/5] netfilter: nf_tables: switch trans_elem to real flex array
Quoted 10 lines · Pablo Neira Ayuso
I'm making another pass on this series, a few thing I would like to ask, see below.
Quoted 6 lines · Florian Westphal
net/netfilter/nf_tables_api.c +0
-0@@ -25,6 +25,7 @@
Quoted 7 lines
net/netfilter/nf_tables_api.c line 28
Quoted 3 lines
net/netfilter/nf_tables_api.c +1
-0#define NFT_MODULE_AUTOLOAD_LIMIT (MODULE_NAME_LEN - sizeof("nft-expr-255-"))#define NFT_SET_MAX_ANONLEN 16 +#define NFT_MAX_SET_NELEMS ((2048 - sizeof(struct nft_trans_elem)) / sizeof(struct nft_trans_one_elem))This
NFT_MAX_SET_NELEMSis to stay in a specific kmalloc-X?What is the logic behind this
NFT_MAX_SET_NELEMS?
I want to avoid making huge kmalloc requests, plus avoid huge krealloc overhead.
I think that kmalloc-2048 slab is a good fit.
I can add a comment, or increase to kmalloc-4096 but I'd prefer to not go over that, since kmalloc allocations > 1 page are more prone to allocation failure.
Quoted 12 lines
net/netfilter/nf_tables_api.c static void nf_tables_unregister_hook(struct net *net, · line 397
Quoted 9 lines
net/netfilter/nf_tables_api.c +3
-0unsigned int nf_tables_net_id__read_mostly ;@@ -391,6 +392,69 @@ static void nf_tables_unregister_hook(struct net *net,return __nf_tables_unregister_hook (net, table, chain, false); } +static bool nft_trans_collapse_set_elem_allowed (const struct nft_trans_elem *a,const struct nft_trans_elem *b) +{ +return a->set == b->set && a->bound == b->bound && a->nelems <NFT_MAX_SET_NELEMS ;I think this
a->bound==b->boundcheck defensive.
Quoted 3 lines
This code is collapsing only two consecutive transactions, the one at the tail (where nelems > 1) and the new transaction (where nelems == 1).
Yes.
Quoted 2 lines
bound state should only change in case there is a NEWRULE transaction in between.
Yes.
Quoted 17 lines
I am trying to find a error scenario where
a->bound==b->boundevaluates false. I considered the following:newelem -> newrule -> newelemwhere newrule has these expressions:
lookup -> errorin this case, newrule error path is exercised:
nft_rule_expr_deactivate(&ctx, rule, NFT_TRANS_PREPARE_ERROR);this calls
nf_tables_deactivate_set()that callsnft_set_trans_unbind(), thena->boundis restored to false. Rule is released and no transaction is added.
Quoted 8 lines
Because if this succeeds:
newelem -> newrule -> newelemthen no element collapsing can happen, because we only collapse what is at the tail.
TLDR; Check does not harm, but it looks unlikely to happen to me.
Yes, its defensive check. I could add a comment.
The WARN_ON_ONCE for trans->nelems != 1 exists for same reason.
Quoted 24 lines
Quoted 24 lines
net/netfilter/nf_tables_api.c +24
-0+} + +static bool nft_trans_collapse_set_elem (struct nftables_pernet *nft_net, +struct nft_trans_elem *tail, +struct nft_trans_elem *trans, +gfp_t gfp) +{ +unsigned int nelems, old_nelems = tail->nelems; +struct nft_trans_elem *new_trans; + +if (!nft_trans_collapse_set_elem_allowed (tail, trans)) +return false; + +if (WARN_ON_ONCE (trans->nelems !=1 )) +return false; + +if (check_add_overflow (old_nelems, trans->nelems, &nelems)) +return false; + +/* krealloc might free tail which invalidates list pointers */ +list_del_init (&tail->nft_trans.list); + + new_trans =krealloc (tail,struct_size (tail, elems, nelems), gfp); +if (!new_trans) {
Quoted 8 lines
net/netfilter/nf_tables_api.c static void nf_tables_unregister_hook(struct net *net, · line 426
Quoted 5 lines
net/netfilter/nf_tables_api.c +5
-0+list_add_tail (&tail->nft_trans.list, &nft_net->commit_list); +return false; + } + +INIT_LIST_HEAD (&new_trans->nft_trans.list);This initialization is also defensive, this element is added via
list_add_tail().
Yes, the first arg to list_add(_tail) can live without initialisation.
Pablo Neira Ayuso
maintainerRe: [PATCH nf-next v4 4/5] netfilter: nf_tables: switch trans_elem to real flex array
Quoted 19 lines · Florian Westphal
Quoted 10 lines · Pablo Neira Ayuso
I'm making another pass on this series, a few thing I would like to ask, see below.
Quoted 6 lines · Florian Westphal
net/netfilter/nf_tables_api.c +0
-0@@ -25,6 +25,7 @@Quoted 7 lines
net/netfilter/nf_tables_api.c line 28
Quoted 3 lines
net/netfilter/nf_tables_api.c +1
-0#define NFT_MODULE_AUTOLOAD_LIMIT (MODULE_NAME_LEN - sizeof("nft-expr-255-"))#define NFT_SET_MAX_ANONLEN 16 +#define NFT_MAX_SET_NELEMS ((2048 - sizeof(struct nft_trans_elem)) / sizeof(struct nft_trans_one_elem))This
NFT_MAX_SET_NELEMSis to stay in a specific kmalloc-X?What is the logic behind this
NFT_MAX_SET_NELEMS?
Quoted 7 lines
I want to avoid making huge kmalloc requests, plus avoid huge krealloc overhead.
I think that kmalloc-2048 slab is a good fit.
I can add a comment, or increase to kmalloc-4096 but I'd prefer to not go over that, since kmalloc allocations > 1 page are more prone to allocation failure.
Makes sense as it is now, thanks for explaining.
Quoted 42 lines
Quoted 12 lines
net/netfilter/nf_tables_api.c static void nf_tables_unregister_hook(struct net *net, · line 397
Quoted 9 lines
net/netfilter/nf_tables_api.c +3
-0unsigned int nf_tables_net_id__read_mostly ;@@ -391,6 +392,69 @@ static void nf_tables_unregister_hook(struct net *net,return __nf_tables_unregister_hook (net, table, chain, false); } +static bool nft_trans_collapse_set_elem_allowed (const struct nft_trans_elem *a,const struct nft_trans_elem *b) +{ +return a->set == b->set && a->bound == b->bound && a->nelems <NFT_MAX_SET_NELEMS ;I think this
a->bound==b->boundcheck defensive.Quoted 3 lines
This code is collapsing only two consecutive transactions, the one at the tail (where nelems > 1) and the new transaction (where nelems == 1).
Yes.
Quoted 2 lines
bound state should only change in case there is a NEWRULE transaction in between.
Yes.
Quoted 19 lines
I am trying to find a error scenario where
a->bound==b->boundevaluates false. I considered the following:newelem -> newrule -> newelemwhere newrule has these expressions:
lookup -> errorin this case, newrule error path is exercised:
nft_rule_expr_deactivate(&ctx, rule, NFT_TRANS_PREPARE_ERROR);this calls
nf_tables_deactivate_set()that callsnft_set_trans_unbind(), thena->boundis restored to false. Rule is released and no transaction is added.Because if this succeeds:
Quoted 8 lines
Quoted 6 lines
newelem -> newrule -> newelemthen no element collapsing can happen, because we only collapse what is at the tail.
TLDR; Check does not harm, but it looks unlikely to happen to me.
Yes, its defensive check. I could add a comment.
Please, do it so we don't forget about this subtle detail.
Quoted 1 line
The
WARN_ON_ONCEfortrans->nelems!= 1 exists for same reason.
Right.
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
net/netfilter/nf_tables_api.c +24
-0+} + +static bool nft_trans_collapse_set_elem (struct nftables_pernet *nft_net, +struct nft_trans_elem *tail, +struct nft_trans_elem *trans, +gfp_t gfp) +{ +unsigned int nelems, old_nelems = tail->nelems; +struct nft_trans_elem *new_trans; + +if (!nft_trans_collapse_set_elem_allowed (tail, trans)) +return false; + +if (WARN_ON_ONCE (trans->nelems !=1 )) +return false; + +if (check_add_overflow (old_nelems, trans->nelems, &nelems)) +return false; + +/* krealloc might free tail which invalidates list pointers */ +list_del_init (&tail->nft_trans.list); + + new_trans =krealloc (tail,struct_size (tail, elems, nelems), gfp); +if (!new_trans) {Quoted 8 lines
net/netfilter/nf_tables_api.c static void nf_tables_unregister_hook(struct net *net, · line 426
Quoted 5 lines
net/netfilter/nf_tables_api.c +5
-0+list_add_tail (&tail->nft_trans.list, &nft_net->commit_list); +return false; + } + +INIT_LIST_HEAD (&new_trans->nft_trans.list);This initialization is also defensive, this element is added via
list_add_tail().
Quoted 1 line
Yes, the first arg to
list_add(_tail)can live without initialisation.
Let's remove it then.
Would you submit a new revision with all these little nitpicks?
Then you also have a chance to edit your explaination on the audit aspect of this series.
If you are busy with other stuff I can take a look here, just let me know.
Thanks.
Florian Westphal
maintainerPATCH v4 5/5 netfilter: nf_tables: allocate element update information dynamically
In reply to Florian WestphalMove the timeout/expire/flag members from nft_trans_one_elem struct into a dybamically allocated structure, only needed when timeout update was requested.
This halves size of nft_trans_one_elem struct and allows to compact up to 124 elements in one transaction container rather than 62.
This halves memory requirements for a large flush or insert transaction, where ->update remains NULL.
Care has to be taken to release the extra data in all spots, including abort path.
--- include/net/netfilter/nf_tables.h | 10 ++++-- net/netfilter/nf_tables_api.c | 57 +++++++++++++++++++------------ 2 files changed, 43 insertions(+), 24 deletions(-)
include/net/netfilter/nf_tables.h +7 -3
@@ -1755,11 +1755,15 @@ enum nft_trans_elem_flags { NFT_TRANS_UPD_EXPIRATION = (1 << 1), }; -struct nft_trans_one_elem { -struct nft_elem_priv *priv; +struct nft_elem_update {u64 timeout;u64 expiration; -u8 update_flags; +u8 flags; +}; + +struct nft_trans_one_elem { +struct nft_elem_priv *priv; +struct nft_elem_update *update; };struct nft_trans_elem {
net/netfilter/nf_tables_api.c +36 -21
123 lines not on this page · raw message
Signature · 1 line
2.45.2
Pablo Neira Ayuso
maintainerHi Florian,
This series looks good to me.
Regarding 3/5, I don't see any fix or anything silly in this.
Quoted 7 lines
nftables audit log format unfortunately leaks an implementation detail, the transaction log size, to userspace:
table=t1 family=2 entries=4 op=nft_register_set
~~~~~~~~~This 'entries' key is the number of transactions that will be applied.
To my understanding, entries= is the number of entries that are either added or updated in this transaction.
Before this patch, there was a 1:1 mapping between transaction and elements, now this is not the case anymore.
If entries= exposes only the number of transactions, then this becomes useless to userspace?
In iptables, it shows the number of entries in the table after the update.
Florian Westphal
maintainerQuoted 8 lines · Pablo Neira Ayuso
Quoted 7 lines
nftables audit log format unfortunately leaks an implementation detail, the transaction log size, to userspace:
table=t1 family=2 entries=4 op=nft_register_set
~~~~~~~~~This 'entries' key is the number of transactions that will be applied.
Quoted 8 lines
To my understanding, entries= is the number of entries that are either added or updated in this transaction.
Before this patch, there was a 1:1 mapping between transaction and elements, now this is not the case anymore.
If entries= exposes only the number of transactions, then this becomes useless to userspace?
Hmm, I would need to know what this is supposed to be. Its not going to be the same in either case, iptables-legacy -A ... vs iptables-nft -A won't result in same entries due to the whole-table-replace paradigm and introduction of "update" mechanism also changes entries count.
I think its fine now, but please feel free to rewrite the commit message if you think its needed.
Pablo Neira Ayuso
maintainerQuoted 15 lines · Florian Westphal
Quoted 14 lines · Pablo Neira Ayuso
Quoted 7 lines
nftables audit log format unfortunately leaks an implementation detail, the transaction log size, to userspace:
table=t1 family=2 entries=4 op=nft_register_set
~~~~~~~~~This 'entries' key is the number of transactions that will be applied.
To my understanding, entries= is the number of entries that are either added or updated in this transaction.
Before this patch, there was a 1:1 mapping between transaction and elements, now this is not the case anymore.
Quoted 8 lines
Quoted 2 lines
If entries= exposes only the number of transactions, then this becomes useless to userspace?
Hmm, I would need to know what this is supposed to be. Its not going to be the same in either case, iptables-legacy -A ... vs iptables-nft -A won't result in same entries due to the whole-table-replace paradigm and introduction of "update" mechanism also changes entries count.
Right, there is a change between -legacy and -nft regarding audit.
Quoted 2 lines
I think its fine now, but please feel free to rewrite the commit message if you think its needed.
Thanks, I will make an edit.