Cannot allocate memory delete table inet filter
Hi,
we do see on occasions that we get the following error message, more so on x86 systems than on arm64:
Error: Could not process rule: Cannot allocate memory delete table inet filter
It is not a consistent error and does not happen all the time.
We are on Kernel 6.6.80, seems to me like we have something along the lines of the nf_tables: allow clone callbacks to sleep problem using GFP_ATOMIC.
Do you have any idea what I can try out/look at?
Best and thanks
Sven
**********************************************************************
DISCLAIMER:
Privileged and/or Confidential information may be contained in this message. If you are not the addressee of this message, you may not copy, use or deliver this message to anyone. In such event, you should destroy the message and kindly notify the sender by reply e-mail. It is understood that opinions or conclusions that do not relate to the official business of the company are neither given nor endorsed by the company. Thank You.
Florian Westphal
maintainerQuoted 6 lines · Sven Auhagen
we do see on occasions that we get the following error message, more so on x86 systems than on arm64:
Error: Could not process rule: Cannot allocate memory delete table inet filter
It is not a consistent error and does not happen all the time.
We are on Kernel 6.6.80, seems to me like we have something along the lines of thenf_tables: allow clone callbacks to sleep problem usingGFP_ATOMIC.
Yes, set element deletion (flushing) requires atomic (non-sleepable) allocations.
Quoted 1 line
Do you have any idea what I can try out/look at?
Do you have large sets? (I suspect yes).
As for a solution, I can see two:
1). Leverage what nft_set_pipapo.c is doing and extend
this for all sets that could use the same solution. The .walk callback for pipapo doesn't need/use rcu read locks, and could use sleepable allocations.
all set types except rhashtable could follow this.
Then, we'd just need a way for the generic flush code to see that the walk callback can sleep (e.g. by annotation in set_ops).
Upside: Clean and straightforward solution.
Downside: won't work for rhashtable which runs under rcu read lock protection.
2). Preallocate transaction elements before calling .walk
in nft_set_flush(), based on set->nelems.
2) is a bit more (w)hacky but it would work for all set types. And I could be wrong and the alloc problem isn't related to nft_set_flush.
Thanks, I checked and this setup has a lot of sets and some of them have 100-200 entries. When I clear all the sets I never have the error.
The nft_set_flush must be the reason.
I think 2. is the general solution that should work for all cases then. I will have a look into that.
Best
Sven
Quoted 6 lines · Sven Auhagen
we do see on occasions that we get the following error message, more so on x86 systems than on arm64:
Error: Could not process rule: Cannot allocate memory delete table inet filter
It is not a consistent error and does not happen all the time.
We are on Kernel 6.6.80, seems to me like we have something along the lines of thenf_tables: allow clone callbacks to sleep problem usingGFP_ATOMIC.
Yes, set element deletion (flushing) requires atomic (non-sleepable) allocations.
Quoted 1 line
Do you have any idea what I can try out/look at?
Do you have large sets? (I suspect yes).
As for a solution, I can see two:
1). Leverage what nft_set_pipapo.c is doing and extend
this for all sets that could use the same solution. The .walk callback for pipapo doesn't need/use rcu read locks, and could use sleepable allocations.
all set types except rhashtable could follow this.
Then, we'd just need a way for the generic flush code to see that the walk callback can sleep (e.g. by annotation in set_ops).
Upside: Clean and straightforward solution.
Downside: won't work for rhashtable which runs under rcu read lock protection.
2). Preallocate transaction elements before calling .walk
in nft_set_flush(), based on set->nelems.
2) is a bit more (w)hacky but it would work for all set types. And I could be wrong and the alloc problem isn't related to nft_set_flush.
**********************************************************************
DISCLAIMER:
Privileged and/or Confidential information may be contained in this message. If you are not the addressee of this message, you may not copy, use or deliver this message to anyone. In such event, you should destroy the message and kindly notify the sender by reply e-mail. It is understood that opinions or conclusions that do not relate to the official business of the company are neither given nor endorsed by the company. Thank You.
One question regarding option 2.
Don't I need to rcu read lock the nft_set_flush now because when I preallocate the transaction I need to make sure that the nelems do not change before doing the walk?
Best
Sven
________________________________________
From: Sven Auhagen <[email protected]>
Sent: Wednesday, June 25, 2025 4:51 PM
To: Florian Westphal <[email protected]>
Cc: [email protected] <[email protected]> Subject: Re: Cannot allocate memory delete table inet filter
Thanks, I checked and this setup has a lot of sets and some of them have 100-200 entries. When I clear all the sets I never have the error.
The nft_set_flush must be the reason.
I think 2. is the general solution that should work for all cases then. I will have a look into that.
Best
Sven
Quoted 6 lines · Sven Auhagen
we do see on occasions that we get the following error message, more so on x86 systems than on arm64:
Error: Could not process rule: Cannot allocate memory delete table inet filter
It is not a consistent error and does not happen all the time.
We are on Kernel 6.6.80, seems to me like we have something along the lines of thenf_tables: allow clone callbacks to sleep problem usingGFP_ATOMIC.
Yes, set element deletion (flushing) requires atomic (non-sleepable) allocations.
Quoted 1 line
Do you have any idea what I can try out/look at?
Do you have large sets? (I suspect yes).
As for a solution, I can see two:
1). Leverage what nft_set_pipapo.c is doing and extend this for all sets that could use the same solution. The .walk callback for pipapo doesn't need/use rcu read locks, and could use sleepable allocations.
all set types except rhashtable could follow this.
Then, we'd just need a way for the generic flush code to see that the walk callback can sleep (e.g. by annotation in set_ops).
Upside: Clean and straightforward solution.
Downside: won't work for rhashtable which runs under rcu read lock protection.
2). Preallocate transaction elements before calling .walk in nft_set_flush(), based on set->nelems.
2) is a bit more (w)hacky but it would work for all set types. And I could be wrong and the alloc problem isn't related to nft_set_flush.
**********************************************************************
DISCLAIMER:
Privileged and/or Confidential information may be contained in this message. If you are not the addressee of this message, you may not copy, use or deliver this message to anyone. In such event, you should destroy the message and kindly notify the sender by reply e-mail. It is understood that opinions or conclusions that do not relate to the official business of the company are neither given nor endorsed by the company. Thank You.
Florian Westphal
maintainerQuoted 2 lines · Sven Auhagen
One question regarding option 2.
Don't I need to rcu read lock thenft_set_flushnow because when I preallocate the transaction I need to make sure that the nelems do not change before doing the walk?
rcu read lock won't make nelems stable either.
The preallocation can fall short (nelems increased) or it can have leftover elements (if elements timed out or were deleted) after the walk.
So, no, you don't need to hold the rcu read lock, but you can't rely on nelems remaining stable.
Florian Westphal
maintainerQuoted 5 lines · Florian Westphal
1). Leverage what
nft_set_pipapo.cis doing and extend
this for all sets that could use the same solution. The .walk callback for pipapo doesn't need/use rcu read locks, and could use sleepable allocations.
all set types except rhashtable could follow this.
FWIW I'm exploring a change to nft_set_hash to avoid the rcu read lock when calling ->iter() for rhashtable.
If it works, this would allow to just replace the GFP_ATOMIC with GFP_KERNEL.
That would be great.
I implemented a version of 2. also and will test it today.
I preallocate the transaction elements before the walk and just allocate more atomically if the nelems changed. This should be enough to preallocate the majority of transaction elems beforehand.
Quoted 5 lines · Florian Westphal
1). Leverage what
nft_set_pipapo.cis doing and extend
this for all sets that could use the same solution. The .walk callback for pipapo doesn't need/use rcu read locks, and could use sleepable allocations.
all set types except rhashtable could follow this.
FWIW I'm exploring a change to nft_set_hash to avoid the rcu read lock when calling ->iter() for rhashtable.
If it works, this would allow to just replace the GFP_ATOMIC with GFP_KERNEL.
**********************************************************************
DISCLAIMER:
Privileged and/or Confidential information may be contained in this message. If you are not the addressee of this message, you may not copy, use or deliver this message to anyone. In such event, you should destroy the message and kindly notify the sender by reply e-mail. It is understood that opinions or conclusions that do not relate to the official business of the company are neither given nor endorsed by the company. Thank You.