Skip to contentSkip to conversation

6.18.14 netfilter/nftables consumes way more memory

REGRESSION Chris Arges 2 participants 8 messages Mar 4 – Mar 7

Subsystems (3)
Networking Stable branch netfilter
lore
Share & tools
lore

Chris Arges

author
#0 lore raw

Hello,

We've noticed significant slab unreclaimable memory increase after upgrading from 6.18.12 to 6.18.15. Other memory values look fairly close, but in my testing slab unreclaimable goes from 1.7 GB to 4.9 GB on machines.

Our use case is having nft rules like below, but adding them to 1000s of network namespaces. This is essentially running nft -f for all these namespaces every minute.

```
table inet service_1234567 {
}
delete table inet service_1234567
table inet service_1234567 {
	chain input {
		type filter hook prerouting priority filter; policy accept;
		ip saddr @account.ip_list drop
	}
	set account.ip_list {
		type ipv4_addr
		flags interval
		auto-merge
	}
}
add element inet service_1234567 account.ip_list { /* add 1000s of CIDRs here */ }
```

I suspect this is related to:
- 36ed9b6e3961 (upstream 7e43e0a1141deec651a60109dab3690854107298)
- netfilter: nft_set_rbtree: translate rbtree to array for binary search

I'm still digging into this, and plan on reverting commits and seeing if memory usage goes back to nominal in production. I don't have a trivial reproducer unfortunately.

Happy to run some additional tests, and I can easily apply patches on top of linux-6.18.y to run in a test environment.

We are using userspace nftables 1.1.3, but had to apply the patch mentioned
in this thread: https://lore.kernel.org/all/[email protected]/
In order to solve the other regression we encountered.

Thanks,
--chris

maintainer
#1 lore raw

Hi,

Quoted 5 lines · Chris Arges

Hello,

We've noticed significant slab unreclaimable memory increase after upgrading from 6.18.12 to 6.18.15. Other memory values look fairly close, but in my testing slab unreclaimable goes from 1.7 GB to 4.9 GB on machines.

From where are you collecting these memory consumption numbers?

Quoted 3 lines

Our use case is having nft rules like below, but adding them to 1000s of network namespaces. This is essentially running nft -f for all these namespaces every minute.

Those numbers for only 1000? That is too little number of entries for such increase in memory usage that you report.

Quoted 18 lines
```
table inet service_1234567 {
}
delete table inet service_1234567
table inet service_1234567 {
	chain input {
		type filter hook prerouting priority filter; policy accept;
		ip saddr @account.ip_list drop
	}
	set account.ip_list {
		type ipv4_addr
		flags interval
		auto-merge
	}
}
add element inet service_1234567 account.ip_list { /* add 1000s of CIDRs here */ }
```
Quoted 3 lines

I suspect this is related to:
- 36ed9b6e3961 (upstream 7e43e0a1141deec651a60109dab3690854107298)
- netfilter: nft_set_rbtree: translate rbtree to array for binary search

More memory consumption is expected indeed, but not so much as you are reporting.

Quoted 3 lines

I'm still digging into this, and plan on reverting commits and seeing if memory usage goes back to nominal in production. I don't have a trivial reproducer unfortunately.

The extra memory comes from the array allocation, the relevant code is here:

#define NFT_ARRAY_EXTRA_SIZE 10240

/* Similar to nft_rbtree_{u,k}size to hide details to userspace, but consider
* packed representation coming from userspace for anonymous sets too. */
static u32 nft_array_elems(const struct nft_set *set)

Quoted 2 lines

Happy to run some additional tests, and I can easily apply patches on top of linux-6.18.y to run in a test environment.

I would need need more info to propose a patch, I don't know where you are pulling such numbers. You also mention you have no reproducer.

Quoted 3 lines

We are using userspace nftables 1.1.3, but had to apply the patch mentioned
in this thread: https://lore.kernel.org/all/[email protected]/
In order to solve the other regression we encountered.

Yes, there are plans to revert a kernel patch that went in -stable to address this.

maintainer
#2 lore raw

Resending, your Reply-To: is botched.

-o-

Hi,

Quoted 5 lines · Chris Arges

Hello,

We've noticed significant slab unreclaimable memory increase after upgrading from 6.18.12 to 6.18.15. Other memory values look fairly close, but in my testing slab unreclaimable goes from 1.7 GB to 4.9 GB on machines.

From where are you collecting these memory consumption numbers?

Quoted 3 lines

Our use case is having nft rules like below, but adding them to 1000s of network namespaces. This is essentially running nft -f for all these namespaces every minute.

Those numbers for only 1000? That is too little number of entries for such increase in memory usage that you report.

Quoted 18 lines
```
table inet service_1234567 {
}
delete table inet service_1234567
table inet service_1234567 {
	chain input {
		type filter hook prerouting priority filter; policy accept;
		ip saddr @account.ip_list drop
	}
	set account.ip_list {
		type ipv4_addr
		flags interval
		auto-merge
	}
}
add element inet service_1234567 account.ip_list { /* add 1000s of CIDRs here */ }
```
Quoted 3 lines

I suspect this is related to:
- 36ed9b6e3961 (upstream 7e43e0a1141deec651a60109dab3690854107298)
- netfilter: nft_set_rbtree: translate rbtree to array for binary search

More memory consumption is expected indeed, but not so much as you are reporting.

Quoted 3 lines

I'm still digging into this, and plan on reverting commits and seeing if memory usage goes back to nominal in production. I don't have a trivial reproducer unfortunately.

The extra memory comes from the array allocation, the relevant code is here:

#define NFT_ARRAY_EXTRA_SIZE 10240

/* Similar to nft_rbtree_{u,k}size to hide details to userspace, but consider
* packed representation coming from userspace for anonymous sets too. */
static u32 nft_array_elems(const struct nft_set *set)

Quoted 2 lines

Happy to run some additional tests, and I can easily apply patches on top of linux-6.18.y to run in a test environment.

I would need need more info to propose a patch, I don't know where you are pulling such numbers. You also mention you have no reproducer.

Quoted 3 lines

We are using userspace nftables 1.1.3, but had to apply the patch mentioned
in this thread: https://lore.kernel.org/all/[email protected]/
In order to solve the other regression we encountered.

Yes, there are plans to revert a kernel patch that went in -stable to address this.

author
#3 lore raw
Quoted 4 lines · Pablo Neira Ayuso

Resending, your Reply-To: is botched.

-o-

I noticed after I sent, thanks for fixing.

Quoted 11 lines

Hi,

Quoted 5 lines · Chris Arges

Hello,

We've noticed significant slab unreclaimable memory increase after upgrading from 6.18.12 to 6.18.15. Other memory values look fairly close, but in my testing slab unreclaimable goes from 1.7 GB to 4.9 GB on machines.

From where are you collecting these memory consumption numbers?

These numbers come from the cgroup's memory.stat:

```
$ cat /sys/fs/cgroup/path/to/service/memory.stat | grep slab
slab_reclaimable 35874232
slab_unreclaimable 5343553056
slab 5379427288
```
Quoted 7 lines
Quoted 3 lines

Our use case is having nft rules like below, but adding them to 1000s of network namespaces. This is essentially running nft -f for all these namespaces every minute.

Those numbers for only 1000? That is too little number of entries for such increase in memory usage that you report.

For this workload that I suspect (since its in the cgroup) it has the following characteristics:
- 1000s of namespaces
- 1000s of CIDRs in ip list per namespace
- Updating everything frequently (<1m)

Quoted 39 lines
Quoted 18 lines
```
table inet service_1234567 {
}
delete table inet service_1234567
table inet service_1234567 {
	chain input {
		type filter hook prerouting priority filter; policy accept;
		ip saddr @account.ip_list drop
	}
	set account.ip_list {
		type ipv4_addr
		flags interval
		auto-merge
	}
}
add element inet service_1234567 account.ip_list { /* add 1000s of CIDRs here */ }
```
Quoted 3 lines

I suspect this is related to:
- 36ed9b6e3961 (upstream 7e43e0a1141deec651a60109dab3690854107298)
- netfilter: nft_set_rbtree: translate rbtree to array for binary search

More memory consumption is expected indeed, but not so much as you are reporting.

Quoted 3 lines

I'm still digging into this, and plan on reverting commits and seeing if memory usage goes back to nominal in production. I don't have a trivial reproducer unfortunately.

The extra memory comes from the array allocation, the relevant code is here:

#define NFT_ARRAY_EXTRA_SIZE 10240

/* Similar to nft_rbtree_{u,k}size to hide details to userspace, but consider
* packed representation coming from userspace for anonymous sets too. */
static u32 nft_array_elems(const struct nft_set *set)

Quoted 6 lines
Quoted 2 lines

Happy to run some additional tests, and I can easily apply patches on top of linux-6.18.y to run in a test environment.

I would need need more info to propose a patch, I don't know where you are pulling such numbers. You also mention you have no reproducer.

To clarify this issue _is_ happening in our production environments, so I can reproduce this issue there. It only happened when going from 6.18.12 to 6.18.15, and with a service inside a cgroup that is mostly applying large sets of IPs via nft. I do not have a simple reproducer script or something I can easily share yet, but am working on that.

I'm going to try and revert rbtree patch series locally and see if it still happens. I can also play with NFT_ARRAY_EXTRA_SIZE and see if that is a factor here as well.

Quoted 6 lines
Quoted 3 lines

We are using userspace nftables 1.1.3, but had to apply the patch mentioned
in this thread: https://lore.kernel.org/all/[email protected]/
In order to solve the other regression we encountered.

Yes, there are plans to revert a kernel patch that went in -stable to address this.

Thanks.
--chris

maintainer
#4 lore raw

Hi Chris,

Quoted 29 lines · Chris Arges

I noticed after I sent, thanks for fixing.

Quoted 11 lines

Hi,

Quoted 5 lines · Chris Arges

Hello,

We've noticed significant slab unreclaimable memory increase after upgrading from 6.18.12 to 6.18.15. Other memory values look fairly close, but in my testing slab unreclaimable goes from 1.7 GB to 4.9 GB on machines.

From where are you collecting these memory consumption numbers?

These numbers come from the cgroup's memory.stat:

```
$ cat /sys/fs/cgroup/path/to/service/memory.stat | grep slab
slab_reclaimable 35874232
slab_unreclaimable 5343553056
slab 5379427288
```
Quoted 7 lines
Quoted 3 lines

Our use case is having nft rules like below, but adding them to 1000s of network namespaces. This is essentially running nft -f for all these namespaces every minute.

Those numbers for only 1000? That is too little number of entries for such increase in memory usage that you report.

Quoted 5 lines

For this workload that I suspect (since its in the cgroup) it has the following characteristics:
- 1000s of namespaces
- 1000s of CIDRs in ip list per namespace
- Updating everything frequently (<1m)

I see what is going on, my resize logic is not correct. This is increasing the size for each new transaction, then the array is getting larger and larger on each transaction update.

Could you please give a try to this patch?

Thanks.

net/netfilter/nft_set_rbtree.c +1 -1
index 853ff30a208c..4462ac48fdfa 100644
@@ -646,7 +646,7 @@ static int nft_array_may_resize(const struct nft_set *set)
 	struct nft_array *array;
 
 	if (!priv->array_next) {
-		array = nft_array_alloc(nelems + NFT_ARRAY_EXTRA_SIZE);
+		array = nft_array_alloc(nelems);
 		if (!array)
 			return -ENOMEM;
maintainer
#5 lore raw
Quoted 38 lines · Pablo Neira Ayuso

Hi Chris,

Quoted 29 lines · Chris Arges

I noticed after I sent, thanks for fixing.

Quoted 11 lines

Hi,

Quoted 5 lines · Chris Arges

Hello,

We've noticed significant slab unreclaimable memory increase after upgrading from 6.18.12 to 6.18.15. Other memory values look fairly close, but in my testing slab unreclaimable goes from 1.7 GB to 4.9 GB on machines.

From where are you collecting these memory consumption numbers?

These numbers come from the cgroup's memory.stat:

```
$ cat /sys/fs/cgroup/path/to/service/memory.stat | grep slab
slab_reclaimable 35874232
slab_unreclaimable 5343553056
slab 5379427288
```
Quoted 7 lines
Quoted 3 lines

Our use case is having nft rules like below, but adding them to 1000s of network namespaces. This is essentially running nft -f for all these namespaces every minute.

Those numbers for only 1000? That is too little number of entries for such increase in memory usage that you report.

Quoted 5 lines

For this workload that I suspect (since its in the cgroup) it has the following characteristics:
- 1000s of namespaces
- 1000s of CIDRs in ip list per namespace
- Updating everything frequently (<1m)

Quoted 5 lines

I see what is going on, my resize logic is not correct. This is increasing the size for each new transaction, then the array is getting larger and larger on each transaction update.

Could you please give a try to this patch?

Scratch that.

Please, give a try to this patch.

Thanks.

net/netfilter/nft_set_rbtree.c +1 -1
index 853ff30a208c..cffeb6f5c532 100644
@@ -646,7 +646,7 @@ static int nft_array_may_resize(const struct nft_set *set)
 	struct nft_array *array;
 
 	if (!priv->array_next) {
-		array = nft_array_alloc(nelems + NFT_ARRAY_EXTRA_SIZE);
+		array = nft_array_alloc(priv->array->max_intervals);
 		if (!array)
 			return -ENOMEM;
author
#6 lore raw

On 2026-03-06 13:25:44, Pablo Neira Ayuso wrote:
<snip>

Quoted 11 lines
Quoted 5 lines

I see what is going on, my resize logic is not correct. This is increasing the size for each new transaction, then the array is getting larger and larger on each transaction update.

Could you please give a try to this patch?

Scratch that.

Please, give a try to this patch.

Thanks.

Pablo,

Thanks, I'm getting this set up on a few machines. I will have:
- 6.18.15 (original kernel version that repo'd the issue for us)
- 6.18.15 + this patch
- 6.18.15 + revert rbtree patchseries

I'll compare memory usage with those 3 variants and give a response.

--chris

Quoted 13 lines
net/netfilter/nft_set_rbtree.c +1 -1
index 853ff30a208c..cffeb6f5c532 100644
@@ -646,7 +646,7 @@ static int nft_array_may_resize(const struct nft_set *set)
 	struct nft_array *array;
 
 	if (!priv->array_next) {
-		array = nft_array_alloc(nelems + NFT_ARRAY_EXTRA_SIZE);
+		array = nft_array_alloc(priv->array->max_intervals);
 		if (!array)
 			return -ENOMEM;
maintainer
#7 lore raw
Quoted 14 lines · Chris Arges

On 2026-03-06 13:25:44, Pablo Neira Ayuso wrote:
<snip>

Quoted 11 lines
Quoted 5 lines

I see what is going on, my resize logic is not correct. This is increasing the size for each new transaction, then the array is getting larger and larger on each transaction update.

Could you please give a try to this patch?

Scratch that.

Please, give a try to this patch.

Thanks.

Quoted 8 lines

Pablo,

Thanks, I'm getting this set up on a few machines. I will have:
- 6.18.15 (original kernel version that repo'd the issue for us)
- 6.18.15 + this patch
- 6.18.15 + revert rbtree patchseries

I'll compare memory usage with those 3 variants and give a response.

I posted a new patch version, see:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/