kvfree_rcu() improvements
Subsystems (5)
Harry Yoo (Oracle)
maintainerv2 cover kvfree_rcu() improvements
These are a few improvements for k[v]free_rcu() API, suggested by Alexei Starovoitov. This aims to tackle two problems:
1) Allow an 8-byte field to be used as an alternative to struct rcu_head (16-byte) for 2-argument kvfree_rcu() to save memory.
2) Add kfree_rcu_nolock() API for an unknown context.
"Unknown context" means the caller does not know whether spinning on a lock is safe. For example, a BPF program attached to an arbitrary kernel function may run while the CPU already holds krcp->lock. However, in practice, it's not held most of the time.
# Discussion
Now that we have sheaves for kmalloc caches, most of frees go through the sheaves layer. However, when sheaves becomes full w/ !allow_spin, call_rcu() cannot be called because the context is unknown. (e.g., it might have preempted call_rcu()). There are two possible approaches:
a) Implement a general call_rcu_nolock() in the RCU subsystem that defers call_rcu() when it's not safe.
b) Handle this as a special case only for rcu sheaf submission in mm/slab_common.c, without touching the RCU core.
This series takes approach (b). This is because a general call_rcu_nolock() would need to flush deferred callbacks before rcu_barrier() to preserve its guarantee, increasing the cost of rcu_barrier() for all RCU users, not just kfree_rcu. By keeping the deferred call_rcu logic in the slab subsystem, only kvfree_rcu_barrier() pays the extra cost.
One downside of the current approach is that slab uses the condition !allow_spin && irqs_disabled() to determine whether it's safe to call call_rcu(), which creates a dependency on RCU's implementation details. I'd like to hear thoughts on this.
# Part 1. Allow an 8-byte field to be used as an alternative to
struct rcu_head for 2-argument kvfree_rcu() (patches 1-2)
Technically, objects that are freed with k[v]free_rcu() need only one pointer to link objects, because we already know that the callback function is always kvfree(). For this purpose, struct rcu_head is unnecessarily large (16 bytes on 64-bit).
Allow a smaller, 8-byte field (of struct rcu_ptr type) to be used with k[v]free_rcu(). Let's save one pointer per slab object.
I have to admit that my naming skill isn't great; hopefully we'll come up with a better name than struct rcu_ptr.
With this feature, either a struct rcu_ptr or rcu_head field can be used as the second argument of the k[v]free_rcu() API.
Users that only use k[v]free_rcu() are may use struct rcu_ptr to save memory (if there can be a lot of objects). However, some users, such as maple tree, may use call_rcu() or k[v]free_rcu() for objects of the same type. For such users, struct rcu_head remains the only option.
Patch 1 implements the struct rcu_ptr feature (for CONFIG_KVFREE_RCU_BATCHED), and patch 2 converts fs/dcache external_name to use struct rcu_ptr as an example user, saving a pointer per dynamically allocated external file name.
# Part 2. Add kfree_rcu_nolock() for unknown contexts
(patches 3-8)
Currently, kfree_rcu() cannot be called when the context is unknown, which might not allow spinning on a lock. In such a context, even calling call_rcu() is not legal, forcing users to implement some sort of deferred freeing. Let's make users' lives easier with a new kfree_rcu_nolock() variant.
Note that only the 2-argument variant is supported, since there is not much we can do when trylock & memory allocation fails.
When spinning on a lock is not allowed, try to acquire the spinlock using spin_trylock(). When trylock succeeds, do either:
1) Use the rcu sheaf to free the object. Note that call_rcu() cannot be called in an unknown context, because it might have preempted call_rcu(). When the rcu sheaf becomes full by freeing the object, defer the submission of the full sheaf using irq_work (defer_call_rcu).
2) Use bnode (of struct kvfree_rcu_bulk_data) to store the pointer. If trylock succeeded but no cached bnode is available, fall back and queue page cache worker just like normal 2-args kvfree_rcu() path.
In rare cases where trylock fails, a non-lazy irq_work is used to defer calling kvfree_call_rcu().
When certain debug features (kmemleak, debugobjects) are enabled, freeing is always deferred because they use spinlocks.
Patch 3 moves code for preparation.
Patch 4 introduces kfree_rcu_nolock().
Patch 5 teaches the rcu sheaf to handle the !allow_spin case. Patch 6 wraps rcu sheaf handling with CONFIG_KVFREE_RCU_BATCHED ifdef. Patch 7 introduces deferred submission of rcu sheaves for the !allow_spin case when IRQs are disabled.
Patch 8 adds a kunit test case for kfree_rcu_nolock().
Changes since RFC V1 [1]:
- Dropped the kmalloc_nolock() -> kfree[_rcu]() path support and the objexts_flags cleanup as they already have landed mainline.
- Dropped rcu_ptr conversions in mm/ (previous patch 2) and instead added struct external_name in fs/dcache.c as a user(new patch 2).
- (Fix) Handle kfence addresses correctly using is_kfence_address() and kfence_object_start().
- Reworked kfree_rcu_nolock() (patch 4):
- When trylock succeeds, now attempts to use cached bnodes (like normal kvfree_rcu 2-arg path) instead of only inserting into krcp->head.
- Added allow_spin parameter to __schedule_delayed_monitor_work() and run_page_cache_worker() to defer work submission via irq_work when spinning is not allowed (Joel).
- (Fix) Introduced defer_kvfree_rcu_barrier() to flush deferred objects before flushing rcu sheaves, preserving correctness of kvfree_rcu_barrier().
- (Fix) Moved kvfree_rcu_barrier()/kvfree_rcu_barrier_on_cache() to slab_common.c on CONFIG_KVFREE_RCU_BATCHED=n, and made them wait for deferred irq_works even without kvfree_rcu batching.
- Introduced object_start_addr() helper to deduplicate the start address calculation logic.
- Instead of falling back when the rcu sheaf becomes full, implemented deferred submission of rcu sheaves using irq_work (new patch 7) (Vlastimil, Alexei).
- Wrapped rcu sheaf handling with CONFIG_KVFREE_RCU_BATCHED ifdef
(new patch 6).
- Added a kunit test for kfree_rcu_nolock() (new patch 8).
[1] RFC V1: https://lore.kernel.org/linux-mm/[email protected]
RFC V2 branch is available at:
https://git.kernel.org/pub/scm/linux/kernel/git/harry/linux.git/log/?h=kvfree-rcu-improvements-rfc-v2r1
RFC V1 branch is available at:
https://git.kernel.org/pub/scm/linux/kernel/git/harry/linux.git/log/?h=kvfree-rcu-improvements-rfc-v1r1
What haven't changed since RFC v1:
- PREEMPT_RT support for kfree_rcu_sheaf() (Vlastimil): that is worth addressing and I think it's doable, but it'll be a too big change to be part of this series.
- Reducing struct rcu_ptr on !KVFREE_RCU_BATCHED (Vlastimil): I tried, but I'm not still sure it's worth the complexity for CONFIG_KVFREE_RCU_BATCHED=n users. Also, this inevitably introduces some delay in freeing objects which is against the purpose of RCU_STRICT_GRACE_PERIOD.
- While writing this cover letter, just realized that I should probably try to reduce the number of irq work structures (pointed out by Joel) (at least to 2 for lazy and non-lazy instead of 4). Will explore this in the next version.
Harry Yoo (Oracle) (8):
mm/slab: introduce k[v]free_rcu() with struct rcu_ptr fs/dcache: use rcu_ptr instead of rcu_head for external names mm/slab: move kfree_rcu_cpu[_work] definitions mm/slab: introduce kfree_rcu_nolock()
mm/slab: make kfree_rcu_nolock() work with sheaves mm/slab: wrap rcu sheaf handling with ifdef mm/slab: introduce deferred submission of rcu sheaves lib/tests/slub_kunit: add a test case for kfree_rcu_nolock()
fs/dcache.c | 8 +- include/linux/rcupdate.h | 64 ++++-- include/linux/slab.h | 16 +- include/linux/types.h | 9 + lib/tests/slub_kunit.c | 73 +++++++ mm/slab.h | 8 +- mm/slab_common.c | 452 +++++++++++++++++++++++++++++---------- mm/slub.c | 47 +++- 8 files changed, 514 insertions(+), 163 deletions(-)
base-commit: 7e0445f673205fd045f3358cacb52b3557627317
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 1/8 mm/slab: introduce k[v]free_rcu() with struct rcu_ptr
k[v]free_rcu() repurposes two fields of struct rcu_head: 'func' to store the start address of the object, and 'next' to link objects.
However, using 'func' to store the start address is unnecessary:
1. slab can get the start address from the address of struct rcu_head field via nearest_obj(), and
2. vmalloc and large kmalloc can get the start address by aligning down the address of the struct rcu_head field to the page boundary.
Therefore, allow an 8-byte (on 64-bit) field (of a new type called struct rcu_ptr) to be used with k[v]free_rcu() with two arguments.
Some users use both call_rcu() and k[v]free_rcu() to process callbacks (e.g., maple tree), so it makes sense to have struct rcu_head field to handle both cases. However, many users that simply free objects via kvfree_rcu() can save one pointer by using struct rcu_ptr instead of struct rcu_head.
Note that struct rcu_ptr is a single pointer only when CONFIG_KVFREE_RCU_BATCHED=y. To keep kvfree_rcu() implementation minimal when CONFIG_KVFREE_RCU_BATCHED is disabled, struct rcu_ptr is the size as struct rcu_head, and the implementation of kvfree_rcu() remains unchanged in that configuration.
Note that implementing a kvfree_rcu batching on !KVFREE_RCU_BATCHED is against the purpose of RCU_STRICT_GRACE_PERIOD which is often used to catch use-after-free bugs.
--- include/linux/rcupdate.h | 61 +++++++++++++++++++++++++++------------- include/linux/types.h | 9 ++++++ mm/slab_common.c | 46 +++++++++++++++++++----------- 3 files changed, 81 insertions(+), 35 deletions(-)
include/linux/rcupdate.h +42 -19
@@ -1057,22 +1057,30 @@ static inline void rcu_read_unlock_migrate(void)/** * kfree_rcu() - kfree an object after a grace period. * @ptr: pointer to kfree for double-argument invocations. -* @rhf: the name of the struct rcu_head within the type of @ptr. +* @rf: the name of the struct rcu_head or struct rcu_ptr within the type of @ptr. * * Many rcu callbacks functions just call kfree() on the base structure. * These functions are trivial, but their size adds up, and furthermore * when they are used in a kernel module, that module must invoke the * high-latency rcu_barrier() function at module-unload time. +* The kfree_rcu() function handles this issue by batching. * -* The kfree_rcu() function handles this issue. In order to have a universal -* callback function handling different offsets of rcu_head, the callback needs -* to determine the starting address of the freed object, which can be a large -* kmalloc or vmalloc allocation. To allow simply aligning the pointer down to -* page boundary for those, only offsets up to 4095 bytes can be accommodated. -* If the offset is larger than 4095 bytes, a compile-time error will -* be generated in kvfree_rcu_arg_2(). If this error is triggered, you can -* either fall back to use of call_rcu() or rearrange the structure to -* position the rcu_head structure into the first 4096 bytes. +* Typically, struct rcu_head is used to process RCU callbacks, but it requires +* two pointers. However, since kfree_rcu() uses kfree() as the callback +* function, it can process callbacks with struct rcu_ptr, which is only +* one pointer in size (unless !CONFIG_KVFREE_RCU_BATCHED). +* +* The type of @rf can be either struct rcu_head or struct rcu_ptr, and when +* possible, it is recommended to use struct rcu_ptr due to its smaller size. +* +* In order to have a universal callback function handling different offsets +* of @rf, the callback needs to determine the starting address of the freed +* object, which can be a large kmalloc or vmalloc allocation. To allow simply +* aligning the pointer down to page boundary for those, only offsets up to +* 4095 bytes can be accommodated. If the offset is larger than 4095 bytes, +* a compile-time error will be generated in kvfree_rcu_arg_2(). +* If this error is triggered, you can either fall back to use of call_rcu() +* or rearrange the structure to position @rf into the first 4096 bytes. * * The object to be freed can be allocated either by kmalloc(), * kmalloc_nolock(), or kmem_cache_alloc().
@@ -1082,8 +1090,8 @@ static inline void rcu_read_unlock_migrate(void)* The BUILD_BUG_ON check must not involve any function calls, hence the * checks are done in macros here. */ -#define kfree_rcu (ptr, rhf) kvfree_rcu_arg_2(ptr, rhf) -#define kvfree_rcu (ptr, rhf) kvfree_rcu_arg_2(ptr, rhf) +#define kfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf) +#define kvfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf)/** * kfree_rcu_mightsleep() - kfree an object after a grace period.
@@ -1105,22 +1113,37 @@ static inline void rcu_read_unlock_migrate(void)#define kfree_rcu_mightsleep (ptr) kvfree_rcu_arg_1(ptr)#define kvfree_rcu_mightsleep (ptr) kvfree_rcu_arg_1(ptr) -/* -* In mm/slab_common.c, no suitable header to include here. -*/ -void kvfree_call_rcu (struct rcu_head *head,void *ptr); + +#ifdef CONFIG_KVFREE_RCU_BATCHED +void kvfree_call_rcu_ptr (struct rcu_ptr *head,void *ptr); +#define kvfree_call_rcu (head, ptr) \ + _Generic((head), \ + struct rcu_head *: kvfree_call_rcu_ptr, \ + struct rcu_ptr *: kvfree_call_rcu_ptr, \ + void *: kvfree_call_rcu_ptr \ + )((struct rcu_ptr *)(head), (ptr)) +#else +void kvfree_call_rcu_head (struct rcu_head *head,void *ptr); +static_assert (sizeof (struct rcu_head ) ==sizeof (struct rcu_ptr )); +#define kvfree_call_rcu (head, ptr) \ + _Generic((head), \ + struct rcu_head *: kvfree_call_rcu_head, \ + struct rcu_ptr *: kvfree_call_rcu_head, \ + void *: kvfree_call_rcu_head \ + )((struct rcu_head *)(head), (ptr)) +#endif /* * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the * comment of kfree_rcu() for details. */ -#define kvfree_rcu_arg_2 (ptr, rhf) \ +#define kvfree_rcu_arg_2 (ptr, rf) \ do { \ typeof (ptr) ___p = (ptr); \ \ if (___p) { \ - BUILD_BUG_ON(offsetof(typeof(*(ptr)), rhf) >= 4096); \ - kvfree_call_rcu(&((___p)->rhf), (void *) (___p)); \ + BUILD_BUG_ON(offsetof(typeof(*(ptr)), rf) >= 4096); \ + kvfree_call_rcu(&((___p)->rf), (void *) (___p)); \ } \ } while (0)
include/linux/types.h +9 -0
@@ -249,6 +249,15 @@ struct callback_head { } __attribute__((aligned(sizeof(void *))));#define rcu_head callback_head + +struct rcu_ptr { +#ifdef CONFIG_KVFREE_RCU_BATCHED +struct rcu_ptr *next; +#else +struct callback_head ; +#endif +} __attribute__((aligned (sizeof (void *)))); +typedef void (*rcu_callback_t )(struct rcu_head *head);typedef void (*call_rcu_func_t )(struct rcu_head *head,rcu_callback_t func);
mm/slab_common.c +30 -16
119 lines not on this page · raw message
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 2/8 fs/dcache: use rcu_ptr instead of rcu_head for external names
In reply to Harry Yoo (Oracle)When a file name length exceeds 31 (DCACHE_INLINE_LEN-1), struct external_name is dynamically allocated. Because only kfree_rcu() is used to free the objects, struct rcu_ptr is enough and saves a pointer per object.
Under the author's home directory, there are 230k unique file names that are longer than 31. Some memory saving benefit is expected.
--- fs/dcache.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
fs/dcache.c +4 -4
@@ -325,12 +325,12 @@ static inline int dentry_cmp(const struct dentry *dentry, const unsigned char *c/* * long names are allocated separately from dentry and never modified. * Refcounted, freeing is RCU-delayed. See take_dentry_name_snapshot() -* for the reason why ->count and ->head can't be combined into a union. +* for the reason why ->count and ->rcu can't be combined into a union. * dentry_string_cmp() relies upon ->name[] being word-aligned. */ struct external_name {atomic_t count; -struct rcu_head head; +struct rcu_ptr rcu;unsigned char name[]__aligned(sizeof(unsigned long)) ; };
@@ -393,7 +393,7 @@ void release_dentry_name_snapshot(struct name_snapshot *name)struct external_name *p; p = container_of(name->name.name, struct external_name, name[0]);if (unlikely (atomic_dec_and_test (&p->count))) -kfree_rcu (p, head); +kfree_rcu (p, rcu); } }EXPORT_SYMBOL (release_dentry_name_snapshot);
@@ -2863,7 +2863,7 @@ static void copy_name(struct dentry *dentry, struct dentry *target) dentry->__d_name.hash_len = target->__d_name.hash_len; }if (old_name &&likely (atomic_dec_and_test (&old_name->count))) -kfree_rcu (old_name, head); +kfree_rcu (old_name, rcu); }/*
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 3/8 mm/slab: move kfree_rcu_cpu[_work] definitions
In reply to Harry Yoo (Oracle)In preparation for defining kfree_rcu_cpu under CONFIG_KVFREE_RCU_BATCHED=n and adding a new function common to both configurations, move the existing kfree_rcu_cpu[_work] definitions to just before the beginning of the kfree_rcu batching infrastructure.
--- mm/slab_common.c | 142 ++++++++++++++++++++++++----------------------- 1 file changed, 72 insertions(+), 70 deletions(-)
mm/slab_common.c +72 -70
@@ -1263,78 +1263,9 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);EXPORT_TRACEPOINT_SYMBOL (kfree);EXPORT_TRACEPOINT_SYMBOL (kmem_cache_free); -#ifndef CONFIG_KVFREE_RCU_BATCHED - -void kvfree_call_rcu_head (struct rcu_head *head,void *ptr) -{ -if (head) { -kasan_record_aux_stack (ptr); -call_rcu (head, kvfree_rcu_cb); -return ; - } - -// kvfree_rcu(one_arg) call. -might_sleep (); -synchronize_rcu (); -kvfree (ptr); -} -EXPORT_SYMBOL_GPL (kvfree_call_rcu_head); - -void __init kvfree_rcu_init (void ) -{ -} - -#else /* CONFIG_KVFREE_RCU_BATCHED */ - -/* -* This rcu parameter is runtime-read-only. It reflects -* a minimum allowed number of objects which can be cached -* per-CPU. Object size is equal to one page. This value -* can be changed at boot time. -*/ -static int rcu_min_cached_objs =5 ; -module_param (rcu_min_cached_objs, int,0444 ); - -// A page shrinker can ask for pages to be freed to make them -// available for other parts of the system. This usually happens -// under low memory conditions, and in that case we should also -// defer page-cache filling for a short time period. -// -// The default value is 5 seconds, which is long enough to reduce -// interference with the shrinker while it asks other systems to -// drain their caches. -static int rcu_delay_page_cache_fill_msec =5000 ; -module_param (rcu_delay_page_cache_fill_msec, int,0444 ); - -static struct workqueue_struct *rcu_reclaim_wq; - -/* Maximum number of jiffies to wait before draining a batch. */ -#define KFREE_DRAIN_JIFFIES (5 * HZ) +#ifdef CONFIG_KVFREE_RCU_BATCHED #define KFREE_N_BATCHES 2#define FREE_N_CHANNELS 2 - -/** -* struct kvfree_rcu_bulk_data - single block to store kvfree_rcu() pointers -* @list: List node. All blocks are linked between each other -* @gp_snap: Snapshot of RCU state for objects placed to this bulk -* @nr_records: Number of active pointers in the array -* @records: Array of the kvfree_rcu() pointers -*/ -struct kvfree_rcu_bulk_data { -struct list_head list; -struct rcu_gp_oldstate gp_snap; -unsigned long nr_records; - void *records[] __counted_by(nr_records); -}; - -/* -* This macro defines how many entries the "records" array -* will contain. It is based on the fact that the size of -* kvfree_rcu_bulk_data structure becomes exactly one page. -*/ -#define KVFREE_BULK_MAX_ENTR \ - ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) -/** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
@@ -1402,6 +1333,77 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; }; +#endif + +#ifndef CONFIG_KVFREE_RCU_BATCHED + +void kvfree_call_rcu_head (struct rcu_head *head,void *ptr) +{ +if (head) { +kasan_record_aux_stack (ptr); +call_rcu (head, kvfree_rcu_cb); +return ; + } + +// kvfree_rcu(one_arg) call. +might_sleep (); +synchronize_rcu (); +kvfree (ptr); +} +EXPORT_SYMBOL_GPL (kvfree_call_rcu_head); + +void __init kvfree_rcu_init (void ) +{ +} + +#else /* CONFIG_KVFREE_RCU_BATCHED */ + +/* +* This rcu parameter is runtime-read-only. It reflects +* a minimum allowed number of objects which can be cached +* per-CPU. Object size is equal to one page. This value +* can be changed at boot time. +*/ +static int rcu_min_cached_objs =5 ; +module_param (rcu_min_cached_objs, int,0444 ); + +// A page shrinker can ask for pages to be freed to make them +// available for other parts of the system. This usually happens +// under low memory conditions, and in that case we should also +// defer page-cache filling for a short time period. +// +// The default value is 5 seconds, which is long enough to reduce +// interference with the shrinker while it asks other systems to +// drain their caches. +static int rcu_delay_page_cache_fill_msec =5000 ; +module_param (rcu_delay_page_cache_fill_msec, int,0444 ); + +static struct workqueue_struct *rcu_reclaim_wq; + +/* Maximum number of jiffies to wait before draining a batch. */ +#define KFREE_DRAIN_JIFFIES (5 * HZ) + +/** +* struct kvfree_rcu_bulk_data - single block to store kvfree_rcu() pointers +* @list: List node. All blocks are linked between each other +* @gp_snap: Snapshot of RCU state for objects placed to this bulk +* @nr_records: Number of active pointers in the array +* @records: Array of the kvfree_rcu() pointers +*/ +struct kvfree_rcu_bulk_data { +struct list_head list; +struct rcu_gp_oldstate gp_snap; +unsigned long nr_records; + void *records[] __counted_by(nr_records); +}; + +/* +* This macro defines how many entries the "records" array +* will contain. It is based on the fact that the size of +* kvfree_rcu_bulk_data structure becomes exactly one page. +*/ +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { .lock = __RAW_SPIN_LOCK_UNLOCKED(krc.lock),
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 4/8 mm/slab: introduce kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Currently, kfree_rcu() cannot be called when the context is unknown, which might not allow spinning on a lock. In such an unknown context, even calling call_rcu() is not legal, forcing users to implement some sort of deferred freeing.
Make users' lives easier by introducing kfree_rcu_nolock() variant. It passes allow_spin = false to kvfree_call_rcu(), which means spinning on a lock is not allowed because the context is unknown.
Unlike kfree_rcu(), kfree_rcu_nolock() only supports a 2-argument variant because, in the worst case where memory allocation fails, the caller cannot synchronously wait for the grace period to finish.
kfree_rcu_nolock() tries to acquire kfree_rcu_cpu spinlock. When trylock succeeds, get a cached bnode and use it to store the pointer. Just like existing kvfree_rcu() with 2-arg variant, fall back if there's no cached bnode available.
If trylock fails, insert the object to the per-cpu lockless list and defer freeing using irq_work that calls kvfree_call_rcu() later. Note that in the most of the cases the context allows spinning, and thus it is worth trying to acquire the lock.
To ensure rcu sheaves are flushed in flush_rcu_all_sheaves() and flush_rcu_sheaves_on_cache(), deferred objects must be processed before calling them. Otherwise, irq work might insert objects to a sheaf and end up not flushing it. Implement a defer_kvfree_rcu_barrier() and call it before flushing rcu sheaves.
In case kmemleak or debug objects is enabled, always defer freeing as those debug features use spinlocks.
Determine whether work items (page cache worker or delayed monitor) need to be queued under krcp->lock. If so, use irq_work to defer the actual work submission. The existing logic prevents excessive irq_work queueing.
For now, the sheaves layer is bypassed if spinning is not allowed.
Without CONFIG_KVFREE_RCU_BATCHED, all frees in the !allow_spin case are deferred using irq_work. Move kvfree_rcu_barrier[_on_cache]() to mm/slab_common.c and let them wait for irq_works.
--- include/linux/rcupdate.h | 23 ++-- include/linux/slab.h | 16 +-- mm/slab.h | 1 + mm/slab_common.c | 260 +++++++++++++++++++++++++++++++-------- mm/slub.c | 6 +- 5 files changed, 231 insertions(+), 75 deletions(-)
include/linux/rcupdate.h +12 -11
@@ -1090,8 +1090,9 @@ static inline void rcu_read_unlock_migrate(void)* The BUILD_BUG_ON check must not involve any function calls, hence the * checks are done in macros here. */ -#define kfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf) -#define kvfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf) +#define kfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf, true) +#define kfree_rcu_nolock (ptr, rf) kvfree_rcu_arg_2(ptr, rf, false) +#define kvfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf, true)/** * kfree_rcu_mightsleep() - kfree an object after a grace period.
@@ -1115,35 +1116,35 @@ static inline void rcu_read_unlock_migrate(void)#ifdef CONFIG_KVFREE_RCU_BATCHED -void kvfree_call_rcu_ptr (struct rcu_ptr *head,void *ptr); -#define kvfree_call_rcu (head, ptr) \ +void kvfree_call_rcu_ptr (struct rcu_ptr *head,void *ptr,bool allow_spin); +#define kvfree_call_rcu (head, ptr, spin) \ _Generic((head), \ struct rcu_head *: kvfree_call_rcu_ptr, \ struct rcu_ptr *: kvfree_call_rcu_ptr, \ void *: kvfree_call_rcu_ptr \ - )((struct rcu_ptr *)(head), (ptr)) + )((struct rcu_ptr *)(head), (ptr), spin)#else -void kvfree_call_rcu_head (struct rcu_head *head,void *ptr); +void kvfree_call_rcu_head (struct rcu_head *head,void *ptr,bool allow_spin);static_assert (sizeof (struct rcu_head ) ==sizeof (struct rcu_ptr )); -#define kvfree_call_rcu (head, ptr) \ +#define kvfree_call_rcu (head, ptr, spin) \ _Generic((head), \ struct rcu_head *: kvfree_call_rcu_head, \ struct rcu_ptr *: kvfree_call_rcu_head, \ void *: kvfree_call_rcu_head \ - )((struct rcu_head *)(head), (ptr)) + )((struct rcu_head *)(head), (ptr), spin)#endif /* * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the * comment of kfree_rcu() for details. */ -#define kvfree_rcu_arg_2 (ptr, rf) \ +#define kvfree_rcu_arg_2 (ptr, rf, spin) \ do { \ typeof (ptr) ___p = (ptr); \ \ if (___p) { \ BUILD_BUG_ON(offsetof(typeof(*(ptr)), rf) >= 4096); \ - kvfree_call_rcu(&((___p)->rf), (void *) (___p)); \ + kvfree_call_rcu(&((___p)->rf), (void *) (___p), spin); \ } \ } while (0)
@@ -1152,7 +1153,7 @@ do { \typeof (ptr) ___p = (ptr); \ \if (___p) \ -kvfree_call_rcu (NULL , (void *) (___p)); \ +kvfree_call_rcu (NULL , (void *) (___p), true); \ }while (0 )/*
include/linux/slab.h +3 -13
@@ -1238,23 +1238,13 @@ extern void kvfree_sensitive(const void *addr, size_t len);unsigned int kmem_cache_size (struct kmem_cache *s); -#ifndef CONFIG_KVFREE_RCU_BATCHED -static inline void kvfree_rcu_barrier (void ) -{ -rcu_barrier (); -} - -static inline void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) -{ -rcu_barrier (); -} - -static inline void kfree_rcu_scheduler_running (void ) { } -#else void kvfree_rcu_barrier (void );void kvfree_rcu_barrier_on_cache (struct kmem_cache *s); +#ifndef CONFIG_KVFREE_RCU_BATCHED +static inline void kfree_rcu_scheduler_running (void ) { } +#else void kfree_rcu_scheduler_running (void );#endif
mm/slab.h +1 -0
@@ -412,6 +412,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj);void flush_all_rcu_sheaves (void );void flush_rcu_sheaves_on_cache (struct kmem_cache *s); +void defer_kvfree_rcu_barrier (void );#define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ SLAB_CACHE_DMA32 | SLAB_PANIC | \
mm/slab_common.c +210 -50
415 lines not on this page · raw message
mm/slub.c +5 -1
@@ -4018,7 +4018,10 @@ static void flush_rcu_sheaf(struct work_struct *w) } -/* needed for kvfree_rcu_barrier() */ +/* +* Needed for kvfree_rcu_barrier(). The caller should invoke +* defer_kvfree_rcu_barrier() before calling this function. +*/ void flush_rcu_sheaves_on_cache (struct kmem_cache *s) {struct slub_flush_work *sfw;
@@ -4053,6 +4056,7 @@ void flush_all_rcu_sheaves(void) {struct kmem_cache *s; +defer_kvfree_rcu_barrier ();cpus_read_lock ();mutex_lock (&slab_mutex);
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 5/8 mm/slab: make kfree_rcu_nolock() work with sheaves
In reply to Harry Yoo (Oracle)Teach kfree_rcu_sheaf() how to handle the !allow_spin case. Similar to __pcs_replace_full_main(), try to get an empty sheaf from pcs->spare or the barn, but don't add !allow_spin support for alloc_empty_sheaf() and fail early instead.
Since call_rcu() does not support NMI contexts, kfree_rcu_sheaf() fails when the rcu sheaf becomes full.
--- mm/slab.h | 2 +- mm/slab_common.c | 7 +++---- mm/slub.c | 14 ++++++++++++-- 3 files changed, 16 insertions(+), 7 deletions(-)
mm/slab.h +1 -1
@@ -409,7 +409,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)return !(s->flags & (SLAB_CACHE_DMA |SLAB_ACCOUNT |SLAB_RECLAIM_ACCOUNT )); } -bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj); +bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin);void flush_all_rcu_sheaves (void );void flush_rcu_sheaves_on_cache (struct kmem_cache *s);void defer_kvfree_rcu_barrier (void );
mm/slab_common.c +3 -4
@@ -1716,7 +1716,7 @@ static void kfree_rcu_work(struct work_struct *work)kvfree_rcu_list (head); } -static bool kfree_rcu_sheaf (void *obj) +static bool kfree_rcu_sheaf (void *obj,bool allow_spin) {struct kmem_cache *s;struct slab *slab;
@@ -1730,7 +1730,7 @@ static bool kfree_rcu_sheaf(void *obj) s = slab->slab_cache;if (likely (!IS_ENABLED (CONFIG_NUMA ) ||slab_nid (slab) ==numa_mem_id ())) -return __kfree_rcu_sheaf (s, obj); +return __kfree_rcu_sheaf (s, obj, allow_spin);return false; }
@@ -2111,8 +2111,7 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr, bool allow_spin) IS_ENABLED(CONFIG_DEBUG_KMEMLEAK))) goto defer_free; -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) && - (allow_spin &&kfree_rcu_sheaf (ptr))) +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) &&kfree_rcu_sheaf (ptr, allow_spin))return ;// Queue the object but don't yet schedule the batch.
mm/slub.c +12 -2
@@ -5895,7 +5895,7 @@ static void rcu_free_sheaf(struct rcu_head *head)*/ static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG); -bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj) +bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin) {struct slub_percpu_sheaves *pcs;struct slab_sheaf *rcu_sheaf;
@@ -5933,7 +5933,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) goto fail; } - empty =barn_get_empty_sheaf (barn, true); + empty =barn_get_empty_sheaf (barn, allow_spin);if (empty) { pcs->rcu_free = empty;
@@ -5942,6 +5942,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)local_unlock (&s->cpu_sheaves->lock); +/* It's easier to fall back than trying harder with !allow_spin */ +if (!allow_spin) + goto fail; + empty =alloc_empty_sheaf (s,GFP_NOWAIT );if (!empty)
@@ -5973,6 +5977,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)if (likely (rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =NULL ; }else { +if (unlikely (!allow_spin)) { +/* call_rcu() cannot be called in an unknown context */ + rcu_sheaf->size--; +local_unlock (&s->cpu_sheaves->lock); + goto fail; + } pcs->rcu_free =NULL ; rcu_sheaf->node =numa_node_id (); }
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 6/8 mm/slab: wrap rcu sheaf handling with ifdef
In reply to Harry Yoo (Oracle)Freeing objects via rcu sheaves is only done with CONFIG_KVFREE_RCU_BATCHED. Wrap the related functions and struct fields with ifdef to make this dependency explicit.
Also remove a TODO about implementing __kvfree_rcu_barrier_on_cache() for a specific slab cache, as there doesn't seem to be a simple and effective way to do so.
--- mm/slab.h | 3 +++ mm/slab_common.c | 4 ---- mm/slub.c | 27 +++++++++++++++++++++++++-- 3 files changed, 28 insertions(+), 6 deletions(-)
mm/slab.h +3 -0
@@ -409,9 +409,12 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)return !(s->flags & (SLAB_CACHE_DMA |SLAB_ACCOUNT |SLAB_RECLAIM_ACCOUNT )); } +#ifdef CONFIG_KVFREE_RCU_BATCHED bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin);void flush_all_rcu_sheaves (void );void flush_rcu_sheaves_on_cache (struct kmem_cache *s); +#endif +void defer_kvfree_rcu_barrier (void );#define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
mm/slab_common.c +0 -4
@@ -2289,10 +2289,6 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)rcu_barrier (); } -/* -* TODO: Introduce a version of __kvfree_rcu_barrier() that works -* on a specific slab cache. -*/ __kvfree_rcu_barrier (); }EXPORT_SYMBOL_GPL (kvfree_rcu_barrier_on_cache);
mm/slub.c +25 -2
@@ -421,7 +421,9 @@ struct slub_percpu_sheaves {local_trylock_t lock;struct slab_sheaf *main;/* never NULL when unlocked */ struct slab_sheaf *spare;/* empty or full, may be NULL */ +#ifdef CONFIG_KVFREE_RCU_BATCHED struct slab_sheaf *rcu_free;/* for batching kfree_rcu() */ +#endif };/*
@@ -2923,6 +2925,7 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf) sheaf->size =0 ; } +#ifdef CONFIG_KVFREE_RCU_BATCHED static bool __rcu_free_sheaf_prepare (struct kmem_cache *s,struct slab_sheaf *sheaf) {
@@ -2965,6 +2968,7 @@ static void rcu_free_sheaf_nobarn(struct rcu_head *head) free_empty_sheaf(s, sheaf); } +#endif /* * Caller needs to make sure migration is disabled in order to fully flush
@@ -2978,7 +2982,10 @@ static void rcu_free_sheaf_nobarn(struct rcu_head *head)static void pcs_flush_all (struct kmem_cache *s) {struct slub_percpu_sheaves *pcs; -struct slab_sheaf *spare, *rcu_free; +struct slab_sheaf *spare; +#ifdef CONFIG_KVFREE_RCU_BATCHED +struct slab_sheaf *rcu_free; +#endif local_lock (&s->cpu_sheaves->lock); pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -2986,8 +2993,10 @@ static void pcs_flush_all(struct kmem_cache *s) spare = pcs->spare; pcs->spare =NULL ; +#ifdef CONFIG_KVFREE_RCU_BATCHED rcu_free = pcs->rcu_free; pcs->rcu_free =NULL ; +#endif local_unlock (&s->cpu_sheaves->lock);
@@ -2996,8 +3005,10 @@ static void pcs_flush_all(struct kmem_cache *s)free_empty_sheaf (s, spare); } +#ifdef CONFIG_KVFREE_RCU_BATCHED if (rcu_free)call_rcu (&rcu_free->rcu_head, rcu_free_sheaf_nobarn); +#endif sheaf_flush_main (s); }
@@ -3016,10 +3027,12 @@ static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu) pcs->spare =NULL ; } +#ifdef CONFIG_KVFREE_RCU_BATCHED if (pcs->rcu_free) {call_rcu (&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn); pcs->rcu_free =NULL ; } +#endif }static void pcs_destroy (struct kmem_cache *s)
@@ -3056,7 +3069,9 @@ static void pcs_destroy(struct kmem_cache *s)*/ WARN_ON (pcs->spare); +#ifdef CONFIG_KVFREE_RCU_BATCHED WARN_ON (pcs->rcu_free); +#endif if (!WARN_ON (pcs->main->size)) {free_empty_sheaf (s, pcs->main);
@@ -3937,7 +3952,11 @@ static bool has_pcs_used(int cpu, struct kmem_cache *s) pcs =per_cpu_ptr (s->cpu_sheaves, cpu); -return (pcs->spare || pcs->rcu_free || pcs->main->size); +#ifdef CONFIG_KVFREE_RCU_BATCHED +if (pcs->rcu_free) +return true; +#endif +return (pcs->spare || pcs->main->size); }/*
@@ -3995,6 +4014,7 @@ static void flush_all(struct kmem_cache *s)cpus_read_unlock (); } +#ifdef CONFIG_KVFREE_RCU_BATCHED static void flush_rcu_sheaf (struct work_struct *w) {struct slub_percpu_sheaves *pcs;
@@ -4071,6 +4091,7 @@ void flush_all_rcu_sheaves(void) rcu_barrier(); } +#endif /* CONFIG_KVFREE_RCU_BATCHED */ static int slub_cpu_setup (unsigned int cpu) {
@@ -5825,6 +5846,7 @@ bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin)return true; } +#ifdef CONFIG_KVFREE_RCU_BATCHED static void rcu_free_sheaf (struct rcu_head *head) {struct slab_sheaf *sheaf;
@@ -6005,6 +6027,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)lock_map_release (&kfree_rcu_sheaf_map);return false; } +#endif /* CONFIG_KVFREE_RCU_BATCHED */ static __always_inline bool can_free_to_pcs(struct slab *slab) {
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 7/8 mm/slab: introduce deferred submission of rcu sheaves
In reply to Harry Yoo (Oracle)Instead of falling back when the rcu sheaf becomes full, implement deferred submission of rcu sheaves. If kfree_rcu_sheaf() is invoked by kfree_rcu_nolock() (!allow_spin) and IRQs are disabled, the CPU might be in the middle of call_rcu() and thus defer call_rcu() with irq_work.
Submit all deferred RCU sheaves to call_rcu() before calling rcu_barrier() to ensure the promise of kvfree_rcu_barrier().
An alternative approach could be to implement this in the RCU subsystem, tracking if it's safe to call call_rcu() and allowing falling back to deferred call_rcu() at the cost of more expensive rcu_barrier() calls.
--- mm/slab.h | 2 ++ mm/slab_common.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++-- mm/slub.c | 12 ++++-------- 3 files changed, 53 insertions(+), 10 deletions(-)
mm/slab.h +2 -0
@@ -411,6 +411,8 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)#ifdef CONFIG_KVFREE_RCU_BATCHED bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin); +void rcu_free_sheaf (struct rcu_head *head); +void submit_rcu_sheaf (struct rcu_head *head,bool allow_spin);void flush_all_rcu_sheaves (void );void flush_rcu_sheaves_on_cache (struct kmem_cache *s);#endif
mm/slab_common.c +47 -2
86 lines not on this page · raw message
mm/slub.c +4 -8
@@ -4152,6 +4152,8 @@ static int slub_cpu_dead(unsigned int cpu)__pcs_flush_all_cpu (s, cpu); }mutex_unlock (&slab_mutex); + +/* pending IRQ work should have been flushed before going offline */ return 0 ; }
@@ -5847,7 +5849,7 @@ bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin) }#ifdef CONFIG_KVFREE_RCU_BATCHED -static void rcu_free_sheaf (struct rcu_head *head) +void rcu_free_sheaf (struct rcu_head *head) {struct slab_sheaf *sheaf;struct node_barn *barn =NULL ;
@@ -5999,12 +6001,6 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)if (likely (rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =NULL ; }else { -if (unlikely (!allow_spin)) { -/* call_rcu() cannot be called in an unknown context */ - rcu_sheaf->size--; -local_unlock (&s->cpu_sheaves->lock); - goto fail; - } pcs->rcu_free =NULL ; rcu_sheaf->node =numa_node_id (); }
@@ -6014,7 +6010,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)* flush_all_rcu_sheaves() doesn't miss this sheaf */ if (rcu_sheaf) -call_rcu (&rcu_sheaf->rcu_head, rcu_free_sheaf); +submit_rcu_sheaf (&rcu_sheaf->rcu_head, allow_spin);local_unlock (&s->cpu_sheaves->lock);
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 8/8 lib/tests/slub_kunit: add a test case for kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Similar to test_kmalloc_kfree_nolock, add a test that allocates objects via kmalloc_nolock() and frees them via kfree_rcu_nolock() in a perf overflow handler (NMI or hardirq depending on the arch), while the main loop allocates and frees objects via kmalloc() and kfree_rcu().
--- lib/tests/slub_kunit.c | 73 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+)
lib/tests/slub_kunit.c +73 -0
@@ -367,6 +367,78 @@ static void test_kmalloc_kfree_nolock(struct kunit *test)kfree (objects[j]); } +cleanup: +perf_event_disable (ctx.event); +perf_event_release_kernel (ctx.event); + +kunit_info (test,"callback_count: %d, alloc_ok: %d, alloc_fail: %d\n" , + ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail); + +if (alloc_fail) +kunit_skip (test,"Allocation failed" ); +KUNIT_EXPECT_EQ (test,0 , slab_errors); +} + +struct dummy_struct { +struct rcu_ptr rcu; +}; + +static void overflow_handler_test_kfree_rcu_nolock (struct perf_event *event, +struct perf_sample_data *data, +struct pt_regs *regs) +{ +struct dummy_struct *dummy; +gfp_t gfp; +struct test_nolock_context *ctx = event->overflow_handler_context; + +/* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */ + gfp = (ctx->callback_count %2 ) ?0 : __GFP_ACCOUNT; + dummy =kmalloc_nolock (sizeof (*dummy), gfp,NUMA_NO_NODE ); + +if (dummy) { + ctx->alloc_ok++; +kfree_rcu_nolock (dummy, rcu); + }else { + ctx->alloc_fail++; + } + ctx->callback_count++; +} + +static void test_kfree_rcu_nolock (struct kunit *test) +{ +int i, j; +struct test_nolock_context ctx = { .test = test }; +struct perf_event *event; +bool alloc_fail = false; +struct dummy_struct *dummy; + +if (IS_BUILTIN (CONFIG_SLUB_KUNIT_TEST )) +kunit_skip (test,"can't do kfree_rcu_nolock() when test is built-in" ); + + event =perf_event_create_kernel_counter (&hw_attr,-1 , current, + overflow_handler_test_kfree_rcu_nolock, + &ctx); +if (IS_ERR (event)) +kunit_skip (test,"Failed to create perf event" ); + ctx.event = event; +perf_event_enable (ctx.event); +for (i =0 ; i <NR_ITERATIONS ; i++) { +for (j =0 ; j <NR_OBJECTS ; j++) { +gfp_t gfp = (i %2 ) ?GFP_KERNEL :GFP_KERNEL_ACCOUNT ; + + objects[j] =kmalloc (sizeof (*dummy), gfp); +if (!objects[j]) { + j--; +while (j >=0 ) +kfree (objects[j--]); + alloc_fail = true; + goto cleanup; + } + } +for (j =0 ; j <NR_OBJECTS ; j++) +kfree_rcu ((struct dummy_struct *)objects[j], rcu); + } + cleanup:perf_event_disable (ctx.event);perf_event_release_kernel (ctx.event);
@@ -406,6 +478,7 @@ static struct kunit_case test_cases[] = { KUNIT_CASE(test_krealloc_redzone_zeroing),#ifdef CONFIG_PERF_EVENTS KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock), + KUNIT_CASE_SLOW(test_kfree_rcu_nolock),#endif {} };
Signature · 1 line
2.43.0
Al Viro
maintainerRe: [PATCH 2/8] fs/dcache: use rcu_ptr instead of rcu_head for external names
In reply to Harry Yoo (Oracle)Quoted 1 line · Harry Yoo (Oracle)
When a file name length exceeds 31 (DCACHE_INLINE_LEN-1),
#ifdef CONFIG_64BIT
# define DNAME_INLINE_WORDS 5 /* 192 bytes */ #else # ifdef CONFIG_SMP # define DNAME_INLINE_WORDS 9 /* 128 bytes */ # else # define DNAME_INLINE_WORDS 11 /* 128 bytes */ # endif #endif
Where do you get that 31 from?
Re: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Quoted 20 lines · Harry Yoo (Oracle)
struct
kfree_rcu_cpu{mm/slab_common.c +12
-0+// Objects queued on a lockless linked list, used to free objects +// in unknown contexts when trylock fails. +struct llist_head defer_head; + +struct irq_work defer_free; +struct irq_work sched_delayed_monitor; +struct irq_work run_page_cache_worker; +// Objects queued on a linked list struct rcu_ptr *head;unsigned long head_gp_snap;@@ -1333,12 +1341,99 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; }; + +static void defer_kfree_rcu_irq_work_fn (struct irq_work *work); +static void sched_delayed_monitor_irq_work_fn (struct irq_work *work); +static void run_page_cache_worker_irq_work_fn (struct irq_work *work);
mm/slab_common.c struct kfree_rcu_cpu { · line 1357
Quoted 10 lines
mm/slab_common.c +10
-0+ +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .lock =__RAW_SPIN_LOCK_UNLOCKED (krc.lock), + .defer_head =LLIST_HEAD_INIT (defer_head), + .defer_free =IRQ_WORK_INIT (defer_kfree_rcu_irq_work_fn), + .sched_delayed_monitor = +IRQ_WORK_INIT_LAZY (sched_delayed_monitor_irq_work_fn), + .run_page_cache_worker = +IRQ_WORK_INIT_LAZY (run_page_cache_worker_irq_work_fn), +};
I think kfree_rcu_cpu doesn't need to be per-cpu. It can be global llist with single irq_work for them all. Not sure about sched_delayed_monitor/run_page_cache_worker. Do they have to be per-cpu ? Can all 3 share single irq_work?
Re: [PATCH 7/8] mm/slab: introduce deferred submission of rcu sheaves
In reply to Harry Yoo (Oracle)Quoted 11 lines · Harry Yoo (Oracle)
Instead of falling back when the rcu sheaf becomes full, implement deferred submission of rcu sheaves. If
kfree_rcu_sheaf()is invoked bykfree_rcu_nolock()(!allow_spin) and IRQs are disabled, the CPU might be in the middle ofcall_rcu()and thus defercall_rcu()withirq_work.Submit all deferred RCU sheaves to
call_rcu()before callingrcu_barrier()to ensure the promise ofkvfree_rcu_barrier().An alternative approach could be to implement this in the RCU subsystem, tracking if it's safe to call
call_rcu()and allowing falling back to deferredcall_rcu()at the cost of more expensivercu_barrier()calls.
Yeah. call_rcu_nolock() will be really handy here and in other places.
When you respin pls pick some tree that sashiko knows about.
So it can apply it all and review it all.
Currently it reviewed only patch 1 and failed to apply the most interesting 4+
https://sashiko.dev/#/patchset/20260416091022.36823-1-harry%40kernel.org
Paul E. McKenney
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Alexei StarovoitovQuoted 26 lines · Alexei Starovoitov
Quoted 25 lines · Harry Yoo (Oracle)
struct
kfree_rcu_cpu{mm/slab_common.c +17
-0+// Objects queued on a lockless linked list, used to free objects +// in unknown contexts when trylock fails. +struct llist_head defer_head; + +struct irq_work defer_free; +struct irq_work sched_delayed_monitor; +struct irq_work run_page_cache_worker; +// Objects queued on a linked list struct rcu_ptr *head;unsigned long head_gp_snap;@@ -1333,12 +1341,99 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; }; + +static void defer_kfree_rcu_irq_work_fn (struct irq_work *work); +static void sched_delayed_monitor_irq_work_fn (struct irq_work *work); +static void run_page_cache_worker_irq_work_fn (struct irq_work *work); + +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .lock =__RAW_SPIN_LOCK_UNLOCKED (krc.lock), + .defer_head =LLIST_HEAD_INIT (defer_head), + .defer_free =IRQ_WORK_INIT (defer_kfree_rcu_irq_work_fn),
Quoted 8 lines
mm/slab_common.c struct kfree_rcu_cpu { · line 1357
Quoted 5 lines
mm/slab_common.c +5
-0+ .sched_delayed_monitor = + IRQ_WORK_INIT_LAZY(sched_delayed_monitor_irq_work_fn), + .run_page_cache_worker = + IRQ_WORK_INIT_LAZY(run_page_cache_worker_irq_work_fn), +};I think
kfree_rcu_cpudoesn't need to be per-cpu. It can be global llist with singleirq_workfor them all.
I would be quite nervous about that, but you might well be right, given that this is a trylock-acquisition failure path. Give or take people and/or machines analyzing the code for potential denial-of-service attacks. :-/
Quoted 2 lines
Not sure about sched_delayed_monitor/run_page_cache_worker. Do they have to be per-cpu ? Can all 3 share single
irq_work?
On the other hand, if all CPUs are doing kfree_rcu() in even a semi-tight loop, having them all unconditionally use global state is not going to make for a fun time on large systems. And there already are situations where user code can make all CPUs to call_rcu() in a semi-tight loop, so even if that is not yet the case for kfree_rcu(), past experience indicates that it soon will be.
And noted on the desirability of call_rcu_nolock(), apologies for being slow.
Thanx, Paul
Re: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
Quoted 45 lines · Paul E. McKenney
Quoted 26 lines · Alexei Starovoitov
Quoted 25 lines · Harry Yoo (Oracle)
struct
kfree_rcu_cpu{mm/slab_common.c +17
-0+// Objects queued on a lockless linked list, used to free objects +// in unknown contexts when trylock fails. +struct llist_head defer_head; + +struct irq_work defer_free; +struct irq_work sched_delayed_monitor; +struct irq_work run_page_cache_worker; +// Objects queued on a linked list struct rcu_ptr *head;unsigned long head_gp_snap;@@ -1333,12 +1341,99 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; }; + +static void defer_kfree_rcu_irq_work_fn (struct irq_work *work); +static void sched_delayed_monitor_irq_work_fn (struct irq_work *work); +static void run_page_cache_worker_irq_work_fn (struct irq_work *work); + +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .lock =__RAW_SPIN_LOCK_UNLOCKED (krc.lock), + .defer_head =LLIST_HEAD_INIT (defer_head), + .defer_free =IRQ_WORK_INIT (defer_kfree_rcu_irq_work_fn),Quoted 8 lines
mm/slab_common.c struct kfree_rcu_cpu { · line 1357
Quoted 5 lines
mm/slab_common.c +5
-0+ .sched_delayed_monitor = + IRQ_WORK_INIT_LAZY(sched_delayed_monitor_irq_work_fn), + .run_page_cache_worker = + IRQ_WORK_INIT_LAZY(run_page_cache_worker_irq_work_fn), +};I think
kfree_rcu_cpudoesn't need to be per-cpu. It can be global llist with singleirq_workfor them all.I would be quite nervous about that, but you might well be right, given that this is a trylock-acquisition failure path. Give or take people and/or machines analyzing the code for potential denial-of-service attacks. :-/
Quoted 2 lines
Not sure about sched_delayed_monitor/run_page_cache_worker. Do they have to be per-cpu ? Can all 3 share single
irq_work?
Quoted 6 lines
On the other hand, if all CPUs are doing
kfree_rcu()in even a semi-tight loop, having them all unconditionally use global state is not going to make for a fun time on large systems. And there already are situations where user code can make all CPUs tocall_rcu()in a semi-tight loop, so even if that is not yet the case forkfree_rcu(), past experience indicates that it soon will be.
My understanding is that defer_head/defer_free will be used when per-cpu trylock fails, so it should be rare. Hence the suggestion to optimize for memory.
Harry Yoo (Oracle)
maintainerRe: [PATCH 2/8] fs/dcache: use rcu_ptr instead of rcu_head for external names
In reply to Al ViroQuoted 2 lines · Al Viro
Quoted 1 line · Harry Yoo (Oracle)
When a file name length exceeds 31 (DCACHE_INLINE_LEN-1),
^ Now I see a typo here
Quoted 11 lines
#ifdef
CONFIG_64BIT# define DNAME_INLINE_WORDS 5 /* 192 bytes */ #else # ifdef CONFIG_SMP # define DNAME_INLINE_WORDS 9 /* 128 bytes */ # else # define DNAME_INLINE_WORDS 11 /* 128 bytes */ # endif #endifWhere do you get that 31 from?
Oh god, I blindly got it from the wrong file:
$ csdef DNAME_INLINE_LEN # my custom alias for cscope tools/testing/selftests/bpf/progs/find_vma.c DNAME_INLINE_LEN 14 #define DNAME_INLINE_LEN 32 include/linux/dcache.h DNAME_INLINE_LEN 82 #define DNAME_INLINE_LEN (DNAME_INLINE_WORDS*sizeof(unsigned long ))
Thanks for pointing it out.
Should have been 39.
Will fix.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Paul E. McKenneyQuoted 26 lines · Paul E. McKenney
Quoted 25 lines · Alexei Starovoitov
Quoted 24 lines · Harry Yoo (Oracle)
struct
kfree_rcu_cpu{mm/slab_common.c +16
-0+// Objects queued on a lockless linked list, used to free objects +// in unknown contexts when trylock fails. +struct llist_head defer_head; + +struct irq_work defer_free; +struct irq_work sched_delayed_monitor; +struct irq_work run_page_cache_worker; +// Objects queued on a linked list struct rcu_ptr *head;unsigned long head_gp_snap;@@ -1333,12 +1341,99 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; }; + +static void defer_kfree_rcu_irq_work_fn (struct irq_work *work); +static void sched_delayed_monitor_irq_work_fn (struct irq_work *work); +static void run_page_cache_worker_irq_work_fn (struct irq_work *work); + +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .lock =__RAW_SPIN_LOCK_UNLOCKED (krc.lock), + .defer_head =LLIST_HEAD_INIT (defer_head),
Quoted 8 lines
Quoted 8 lines
mm/slab_common.c struct kfree_rcu_cpu { · line 1357
Quoted 6 lines
mm/slab_common.c +6
-0+ .defer_free = IRQ_WORK_INIT(defer_kfree_rcu_irq_work_fn), + .sched_delayed_monitor = + IRQ_WORK_INIT_LAZY(sched_delayed_monitor_irq_work_fn), + .run_page_cache_worker = + IRQ_WORK_INIT_LAZY(run_page_cache_worker_irq_work_fn), +};I think
kfree_rcu_cpudoesn't need to be per-cpu.
After reading this, I was like "Oh, that's quite a drastic change?", but looks like I misread it. I didn't create a new percpu structure, but extended the existing one.
I guess you meant the new fields added (defer_head, and irq works) to struct kfree_rcu_cpu, not the whole structure.
Quoted 1 line
Quoted 1 line
It can be global llist with single
irq_workfor them all.
It could be, but what is the benefit of separating them from existing kfree_rcu_cpu and making them global?
Quoted 4 lines
I would be quite nervous about that, but you might well be right, given that this is a trylock-acquisition failure path. Give or take people and/or machines analyzing the code for potential denial-of-service attacks. :-/
It'll probably not that bad because it's trylock-acquisiion failure path of per-cpu lock; IIRC during my test, falling back to defer_free happened only a few times (< 10) when the kunit test is calling kfree_rcu() in a tight loop (100k calls) while concurrently invoking kfree_rcu_nolock() ~10k times on the same CPU.
Quoted 2 lines
Quoted 2 lines
Not sure about sched_delayed_monitor/run_page_cache_worker. Do they have to be per-cpu ?
Since existing sched_delayed_monitor/run_page_cache_worker works are per-cpu, I think it's better to keep those irq_works per-cpu as well.
Quoted 1 line
Quoted 1 line
Can all 3 share single
irq_work?
I thought defer_free and defer_call_rcu should be non-lazy irq work and others should be lazy irq work. And I was thinking of having one lazy and one non-lazy IRQ work (two instead of four).
But given that sched_delayed_monitor and run_page_cache_worker should not triggered that frequently anyway, it'll probably be okay for all of them to share a single non-lazy IRQ work.
Quoted 6 lines
On the other hand, if all CPUs are doing
kfree_rcu()in even a semi-tight loop, having them all unconditionally use global state is not going to make for a fun time on large systems. And there already are situations where user code can make all CPUs tocall_rcu()in a semi-tight loop, so even if that is not yet the case forkfree_rcu(), past experience indicates that it soon will be.
A tight loop for kfree_rcu() should be fine.
I think the question is "Can a malicious user can make all CPUs to kfree_rcu() in a tight loop AND concurrently trigger kfree_rcu_nolock() on those CPUs, so that trylock will mostly fail"
Quoted 2 lines
And noted on the desirability of
call_rcu_nolock(), apologies for being slow.
No problem. Really appreciate looking into it, Alexei and Paul!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerRe: [PATCH 7/8] mm/slab: introduce deferred submission of rcu sheaves
In reply to Alexei StarovoitovQuoted 6 lines · Alexei Starovoitov
Quoted 5 lines · Harry Yoo (Oracle)
Instead of falling back when the rcu sheaf becomes full, implement deferred submission of rcu sheaves. If
kfree_rcu_sheaf()is invoked bykfree_rcu_nolock()(!allow_spin) and IRQs are disabled, the CPU might be in the middle ofcall_rcu()and thus defercall_rcu()withirq_work.
Quoted 8 lines
Quoted 6 lines
Submit all deferred RCU sheaves to
call_rcu()before callingrcu_barrier()to ensure the promise ofkvfree_rcu_barrier().An alternative approach could be to implement this in the RCU subsystem, tracking if it's safe to call
call_rcu()and allowing falling back to deferredcall_rcu()at the cost of more expensivercu_barrier()calls.Yeah.
call_rcu_nolock()will be really handy here and in other places.
*curiously watching whether RCU folks show interest, so that he could
drop ad-hoc code in slab*
Quoted 1 line
When you respin pls pick some tree that sashiko knows about.
Yeah, adding base-commit id appareantly didn't help. Sashiko doesn't know that we have the slab tree :'(
Quoted 3 lines
So it can apply it all and review it all.
Currently it reviewed only patch 1 and failed to apply the most interesting 4+
https://sashiko.dev/#/patchset/20260416091022.36823-1-harry%40kernel.org
I heard that at least Sashiko knows about mm, linux-next, and mainline trees. Will try linux-next next time.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerQuoted 9 lines · Harry Yoo (Oracle)
# Part 1. Allow an 8-byte field to be used as an alternative to
structrcu_headfor 2-argumentkvfree_rcu()(patches 1-2)Technically, objects that are freed with k[v]free_rcu() need only one pointer to link objects, because we already know that the callback function is always
kvfree(). For this purpose, structrcu_headis unnecessarily large (16 bytes on 64-bit).
Quoted 5 lines
Allow a smaller, 8-byte field (of struct
rcu_ptrtype) to be used with k[v]free_rcu(). Let's save one pointer per slab object.I have to admit that my naming skill isn't great; hopefully we'll come up with a better name than
struct rcu_ptr.
"rcu_ptr" sounds too generic to me. Things like rcu_dereference() talk about "RCU-protected pointer" and it could be misleading that "rcu_ptr" has something to do with this. Only if Paul think it's ok :)
But maybe just "kvfree_rcu_head"? As IMHO It's an rcu_head specialized for kvfree.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 1/8] mm/slab: introduce k[v]free_rcu() with struct rcu_ptr
In reply to Harry Yoo (Oracle)Quoted 18 lines · Harry Yoo (Oracle)
mm/slab_common.c +6
-4@@ -1523,18 +1522,34 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, }static void -kvfree_rcu_list (struct rcu_head *head) +kvfree_rcu_list (struct rcu_ptr *head) { -struct rcu_head *next; +struct rcu_ptr *next;for (; head; head = next) { -void *ptr = (void *) head->func; -unsigned long offset = (void *) head - ptr; +void *ptr; +unsigned long offset; +struct slab *slab; +if (is_vmalloc_addr (head)) {
mm/slab_common.c kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, · line 1544
Quoted 10 lines
mm/slab_common.c +10
-0+ ptr = (void *)PAGE_ALIGN_DOWN ((unsigned long )head); + }else { + slab =virt_to_slab (head); +if (!slab) + ptr = (void *)PAGE_ALIGN_DOWN ((unsigned long )head); +else if (is_kfence_address (head)) + ptr =kfence_object_start (head); +else + ptr =nearest_obj (slab->slab_cache, slab, head); + }
It seems a bit wasteful to do this... (especially the virt_to_slab()) part.
mm/slab_common.c kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, · line 1554
Quoted 11 lines
mm/slab_common.c +5
-1+ + offset = (void *)head - ptr; next = head->next;debug_rcu_head_unqueue ((struct rcu_head *)ptr);rcu_lock_acquire (&rcu_callback_map); -trace_rcu_invoke_kvfree_callback ("slab" , head, offset); +trace_rcu_invoke_kvfree_callback ("slab" , + (struct rcu_head *)head, + offset);kvfree (ptr);
... and then throw it all away and let kvfree(ptr) do it again. So maybe we could then call something more internal and specific based on the result of the above.
Uladzislau Rezki
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Quoted 630 lines · Harry Yoo (Oracle)
Currently,
kfree_rcu()cannot be called when the context is unknown, which might not allow spinning on a lock. In such an unknown context, even callingcall_rcu()is not legal, forcing users to implement some sort of deferred freeing.Make users' lives easier by introducing
kfree_rcu_nolock()variant. It passesallow_spin= false tokvfree_call_rcu(), which means spinning on a lock is not allowed because the context is unknown.Unlike
kfree_rcu(),kfree_rcu_nolock()only supports a 2-argument variant because, in the worst case where memory allocation fails, the caller cannot synchronously wait for the grace period to finish.
kfree_rcu_nolock()tries to acquirekfree_rcu_cpuspinlock. When trylock succeeds, get a cached bnode and use it to store the pointer. Just like existingkvfree_rcu()with 2-arg variant, fall back if there's no cached bnode available.If trylock fails, insert the object to the per-cpu lockless list and defer freeing using
irq_workthat callskvfree_call_rcu()later. Note that in the most of the cases the context allows spinning, and thus it is worth trying to acquire the lock.To ensure rcu sheaves are flushed in
flush_rcu_all_sheaves()andflush_rcu_sheaves_on_cache(), deferred objects must be processed before calling them. Otherwise, irq work might insert objects to a sheaf and end up not flushing it. Implement adefer_kvfree_rcu_barrier()and call it before flushing rcu sheaves.In case kmemleak or debug objects is enabled, always defer freeing as those debug features use spinlocks.
Determine whether work items (page cache worker or delayed monitor) need to be queued under
krcp->lock. If so, useirq_workto defer the actual work submission. The existing logic prevents excessiveirq_workqueueing.For now, the sheaves layer is bypassed if spinning is not allowed.
Without
CONFIG_KVFREE_RCU_BATCHED, all frees in the !allow_spin case are deferred usingirq_work. Move kvfree_rcu_barrier[_on_cache]() tomm/slab_common.cand let them wait forirq_works.Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Harry Yoo (Oracle) <[email protected]>--- include/linux/rcupdate.h | 23 ++-- include/linux/slab.h | 16 +-- mm/slab.h | 1 + mm/slab_common.c | 260 +++++++++++++++++++++++++++++++-------- mm/slub.c | 6 +- 5 files changed, 231 insertions(+), 75 deletions(-)include/linux/rcupdate.h +12
-11@@ -1090,8 +1090,9 @@ static inline void rcu_read_unlock_migrate(void)* The BUILD_BUG_ON check must not involve any function calls, hence the * checks are done in macros here. */ -#define kfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf) -#define kvfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf) +#define kfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf, true) +#define kfree_rcu_nolock (ptr, rf) kvfree_rcu_arg_2(ptr, rf, false) +#define kvfree_rcu (ptr, rf) kvfree_rcu_arg_2(ptr, rf, true)/** * kfree_rcu_mightsleep() - kfree an object after a grace period. @@ -1115,35 +1116,35 @@ static inline void rcu_read_unlock_migrate(void)#ifdef CONFIG_KVFREE_RCU_BATCHED -void kvfree_call_rcu_ptr (struct rcu_ptr *head,void *ptr); -#define kvfree_call_rcu (head, ptr) \ +void kvfree_call_rcu_ptr (struct rcu_ptr *head,void *ptr,bool allow_spin); +#define kvfree_call_rcu (head, ptr, spin) \ _Generic((head), \ struct rcu_head *: kvfree_call_rcu_ptr, \ struct rcu_ptr *: kvfree_call_rcu_ptr, \ void *: kvfree_call_rcu_ptr \ - )((struct rcu_ptr *)(head), (ptr)) + )((struct rcu_ptr *)(head), (ptr), spin)#else -void kvfree_call_rcu_head (struct rcu_head *head,void *ptr); +void kvfree_call_rcu_head (struct rcu_head *head,void *ptr,bool allow_spin);static_assert (sizeof (struct rcu_head ) ==sizeof (struct rcu_ptr )); -#define kvfree_call_rcu (head, ptr) \ +#define kvfree_call_rcu (head, ptr, spin) \ _Generic((head), \ struct rcu_head *: kvfree_call_rcu_head, \ struct rcu_ptr *: kvfree_call_rcu_head, \ void *: kvfree_call_rcu_head \ - )((struct rcu_head *)(head), (ptr)) + )((struct rcu_head *)(head), (ptr), spin)#endif /* * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the * comment of kfree_rcu() for details. */ -#define kvfree_rcu_arg_2 (ptr, rf) \ +#define kvfree_rcu_arg_2 (ptr, rf, spin) \ do { \ typeof (ptr) ___p = (ptr); \ \ if (___p) { \ BUILD_BUG_ON(offsetof(typeof(*(ptr)), rf) >= 4096); \ - kvfree_call_rcu(&((___p)->rf), (void *) (___p)); \ + kvfree_call_rcu(&((___p)->rf), (void *) (___p), spin); \ } \ } while (0)@@ -1152,7 +1153,7 @@ do { \typeof (ptr) ___p = (ptr); \ \if (___p) \ -kvfree_call_rcu (NULL , (void *) (___p)); \ +kvfree_call_rcu (NULL , (void *) (___p), true); \ }while (0 )/* include/linux/slab.h +3
-13@@ -1238,23 +1238,13 @@ extern void kvfree_sensitive(const void *addr, size_t len);unsigned int kmem_cache_size (struct kmem_cache *s); -#ifndef CONFIG_KVFREE_RCU_BATCHED -static inline void kvfree_rcu_barrier (void ) -{ -rcu_barrier (); -} - -static inline void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) -{ -rcu_barrier (); -} - -static inline void kfree_rcu_scheduler_running (void ) { } -#else void kvfree_rcu_barrier (void );void kvfree_rcu_barrier_on_cache (struct kmem_cache *s); +#ifndef CONFIG_KVFREE_RCU_BATCHED +static inline void kfree_rcu_scheduler_running (void ) { } +#else void kfree_rcu_scheduler_running (void );#endif mm/slab.h +1
-0@@ -412,6 +412,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj);void flush_all_rcu_sheaves (void );void flush_rcu_sheaves_on_cache (struct kmem_cache *s); +void defer_kvfree_rcu_barrier (void );#define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ SLAB_CACHE_DMA32 | SLAB_PANIC | \mm/slab_common.c +210
-50@@ -1311,6 +1311,14 @@ struct kfree_rcu_cpu_work {* the interactions with the slab allocators. */ struct kfree_rcu_cpu { +// Objects queued on a lockless linked list, used to free objects +// in unknown contexts when trylock fails. +struct llist_head defer_head; + +struct irq_work defer_free; +struct irq_work sched_delayed_monitor; +struct irq_work run_page_cache_worker; +// Objects queued on a linked list struct rcu_ptr *head;unsigned long head_gp_snap;@@ -1333,12 +1341,99 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; }; + +static void defer_kfree_rcu_irq_work_fn (struct irq_work *work); +static void sched_delayed_monitor_irq_work_fn (struct irq_work *work); +static void run_page_cache_worker_irq_work_fn (struct irq_work *work); + +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .lock =__RAW_SPIN_LOCK_UNLOCKED (krc.lock), + .defer_head =LLIST_HEAD_INIT (defer_head), + .defer_free =IRQ_WORK_INIT (defer_kfree_rcu_irq_work_fn), + .sched_delayed_monitor = +IRQ_WORK_INIT_LAZY (sched_delayed_monitor_irq_work_fn), + .run_page_cache_worker = +IRQ_WORK_INIT_LAZY (run_page_cache_worker_irq_work_fn), +}; +#else +struct kfree_rcu_cpu { +struct llist_head defer_head; +struct irq_work defer_free; +}; + +static void defer_kfree_rcu_irq_work_fn (struct irq_work *work); + +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .defer_head =LLIST_HEAD_INIT (defer_head), + .defer_free =IRQ_WORK_INIT (defer_kfree_rcu_irq_work_fn), +};#endif -#ifndef CONFIG_KVFREE_RCU_BATCHED +/* Wait for deferred work from kfree_rcu_nolock() */ +void defer_kvfree_rcu_barrier (void ) +{ +int cpu; + +for_each_possible_cpu (cpu) +irq_work_sync (&per_cpu_ptr (&krc, cpu)->defer_free); +} + +static void *object_start_addr (void *ptr) +{ +struct slab *slab; +void *start; + +if (is_vmalloc_addr (ptr)) { + start = (void *)PAGE_ALIGN_DOWN ((unsigned long )ptr); + }else { + slab =virt_to_slab (ptr); +if (!slab) + start = (void *)PAGE_ALIGN_DOWN ((unsigned long )ptr); +else if (is_kfence_address (ptr)) + start =kfence_object_start (ptr); +else + start =nearest_obj (slab->slab_cache, slab, ptr); + } -void kvfree_call_rcu_head (struct rcu_head *head,void *ptr) +return start; +} + +static void defer_kfree_rcu_irq_work_fn (struct irq_work *work) { +struct kfree_rcu_cpu *krcp; +struct llist_head *head; +struct llist_node *llnode, *pos, *t; + + krcp = container_of(work, struct kfree_rcu_cpu, defer_free); + head = &krcp->defer_head; + +if (llist_empty (head)) +return ; + + llnode =llist_del_all (head); +llist_for_each_safe (pos, t, llnode) { +void *objp; +struct rcu_ptr *rcup = (struct rcu_ptr *)pos; + + objp =object_start_addr (rcup); +kvfree_call_rcu (rcup, objp, true); + } +} + +#ifndef CONFIG_KVFREE_RCU_BATCHED +void kvfree_call_rcu_head (struct rcu_head *head,void *ptr,bool allow_spin) +{ +if (!allow_spin) { +struct kfree_rcu_cpu *krcp; + +guard (preempt)(); + + krcp =this_cpu_ptr (&krc); +if (llist_add ((struct llist_node *)head, &krcp->defer_head)) +irq_work_queue (&krcp->defer_free); +return ; + } +if (head) {kasan_record_aux_stack (ptr);call_rcu (head, kvfree_rcu_cb);@@ -1356,6 +1451,19 @@ void __init kvfree_rcu_init(void) { } +void kvfree_rcu_barrier (void ) +{ +defer_kvfree_rcu_barrier (); +rcu_barrier (); +} +EXPORT_SYMBOL_GPL (kvfree_rcu_barrier); + +void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) +{ +kvfree_rcu_barrier (); +} +EXPORT_SYMBOL_GPL (kvfree_rcu_barrier_on_cache); +#else /* CONFIG_KVFREE_RCU_BATCHED */ /* @@ -1405,9 +1513,16 @@ struct kvfree_rcu_bulk_data {#define KVFREE_BULK_MAX_ENTR \ ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) -static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { - .lock =__RAW_SPIN_LOCK_UNLOCKED (krc.lock), -}; + +static void schedule_delayed_monitor_work (struct kfree_rcu_cpu *krcp); + +static void sched_delayed_monitor_irq_work_fn (struct irq_work *work) +{ +struct kfree_rcu_cpu *krcp; + + krcp = container_of(work, struct kfree_rcu_cpu, sched_delayed_monitor); +schedule_delayed_monitor_work (krcp); +} static __always_inline voiddebug_rcu_bhead_unqueue (struct kvfree_rcu_bulk_data *bhead)@@ -1421,13 +1536,18 @@ debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead) }static inline struct kfree_rcu_cpu * -krc_this_cpu_lock (unsigned long *flags) +krc_this_cpu_lock (unsigned long *flags,bool allow_spin) {struct kfree_rcu_cpu *krcp;local_irq_save (*flags);// For safely calling this_cpu_ptr(). krcp =this_cpu_ptr (&krc); -raw_spin_lock (&krcp->lock); +if (allow_spin) { +raw_spin_lock (&krcp->lock); + }else if (!raw_spin_trylock (&krcp->lock)) { +local_irq_restore (*flags); +return NULL ; + }return krcp; }@@ -1531,20 +1651,8 @@ kvfree_rcu_list(struct rcu_ptr *head)for (; head; head = next) {void *ptr;unsigned long offset; -struct slab *slab; - -if (is_vmalloc_addr (head)) { - ptr = (void *)PAGE_ALIGN_DOWN ((unsigned long )head); - }else { - slab =virt_to_slab (head); -if (!slab) - ptr = (void *)PAGE_ALIGN_DOWN ((unsigned long )head); -else if (is_kfence_address (head)) - ptr =kfence_object_start (head); -else - ptr =nearest_obj (slab->slab_cache, slab, head); - } + ptr =object_start_addr (head); offset = (void *)head - ptr; next = head->next;debug_rcu_head_unqueue ((struct rcu_head *)ptr);@@ -1663,18 +1771,26 @@ static int krc_count(struct kfree_rcu_cpu *krcp) }static void -__schedule_delayed_monitor_work (struct kfree_rcu_cpu *krcp) +__schedule_delayed_monitor_work (struct kfree_rcu_cpu *krcp,bool allow_spin) {long delay, delay_left; delay =krc_count (krcp) >=KVFREE_BULK_MAX_ENTR ?1 :KFREE_DRAIN_JIFFIES ;if (delayed_work_pending (&krcp->monitor_work)) { delay_left = krcp->monitor_work.timer.expires - jiffies; -if (delay < delay_left) -mod_delayed_work (rcu_reclaim_wq, &krcp->monitor_work, delay); +if (delay < delay_left) { +if (allow_spin) +mod_delayed_work (rcu_reclaim_wq, &krcp->monitor_work, delay); +else +irq_work_queue (&krcp->sched_delayed_monitor); + }return ; } -queue_delayed_work (rcu_reclaim_wq, &krcp->monitor_work, delay); + +if (allow_spin) +queue_delayed_work (rcu_reclaim_wq, &krcp->monitor_work, delay); +else +irq_work_queue (&krcp->sched_delayed_monitor); } static void@@ -1683,7 +1799,7 @@ schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)unsigned long flags;raw_spin_lock_irqsave (&krcp->lock, flags); -__schedule_delayed_monitor_work (krcp); +__schedule_delayed_monitor_work (krcp, true);raw_spin_unlock_irqrestore (&krcp->lock, flags); }@@ -1847,25 +1963,25 @@ static void fill_page_cache_func(struct work_struct *work)// Returns true if ptr was successfully recorded, else the caller must // use a fallback. static inline bool -add_ptr_to_bulk_krc_lock (struct kfree_rcu_cpu **krcp, -unsigned long *flags,void *ptr,bool can_alloc) +add_ptr_to_bulk_krc_lock (struct kfree_rcu_cpu *krcp, +unsigned long *flags,void *ptr,bool can_alloc,bool allow_spin) {struct kvfree_rcu_bulk_data *bnode;int idx; - *krcp =krc_this_cpu_lock (flags); -if (unlikely (!(*krcp)->initialized)) +if (unlikely (!krcp->initialized))return false; idx = !!is_vmalloc_addr (ptr); - bnode =list_first_entry_or_null (&(*krcp)->bulk_head[idx], + bnode =list_first_entry_or_null (&krcp->bulk_head[idx], struct kvfree_rcu_bulk_data, list);/* Check if a new block is required. */ if (!bnode || bnode->nr_records ==KVFREE_BULK_MAX_ENTR ) { - bnode =get_cached_bnode (*krcp); + bnode =get_cached_bnode (krcp);if (!bnode && can_alloc) { -krc_this_cpu_unlock (*krcp, *flags); +krc_this_cpu_unlock (krcp, *flags); +VM_WARN_ON_ONCE (!allow_spin);// __GFP_NORETRY - allows a light-weight direct reclaim // what is OK from minimizing of fallback hitting point of @@ -1880,7 +1996,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,// scenarios. bnode = (struct kvfree_rcu_bulk_data *)__get_free_page (GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN); -raw_spin_lock_irqsave (&(*krcp)->lock, *flags); +raw_spin_lock_irqsave (&krcp->lock, *flags); }if (!bnode)@@ -1888,14 +2004,14 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,// Initialize the new block and attach it. bnode->nr_records =0 ; -list_add (&bnode->list, &(*krcp)->bulk_head[idx]); +list_add (&bnode->list, &krcp->bulk_head[idx]); }// Finally insert and update the GP for this page. bnode->nr_records++; bnode->records[bnode->nr_records -1 ] = ptr;get_state_synchronize_rcu_full (&bnode->gp_snap); -atomic_inc (&(*krcp)->bulk_count[idx]); +atomic_inc (&krcp->bulk_count[idx]);return true; }@@ -1911,7 +2027,32 @@ schedule_page_work_fn(struct hrtimer *t) }static void -run_page_cache_worker (struct kfree_rcu_cpu *krcp) +__run_page_cache_worker (struct kfree_rcu_cpu *krcp) +{ +if (atomic_read (&krcp->backoff_page_cache_fill)) { +queue_delayed_work (rcu_reclaim_wq, + &krcp->page_cache_work, +msecs_to_jiffies (rcu_delay_page_cache_fill_msec)); + }else { +hrtimer_setup (&krcp->hrtimer, schedule_page_work_fn,CLOCK_MONOTONIC , +HRTIMER_MODE_REL ); +hrtimer_start (&krcp->hrtimer,0 ,HRTIMER_MODE_REL ); + } +} + +static void run_page_cache_worker_irq_work_fn (struct irq_work *work) +{ +unsigned long flags; +struct kfree_rcu_cpu *krcp = + container_of(work, struct kfree_rcu_cpu, run_page_cache_worker); + +raw_spin_lock_irqsave (&krcp->lock, flags); +__run_page_cache_worker (krcp); +raw_spin_unlock_irqrestore (&krcp->lock, flags); +} + +static void +run_page_cache_worker (struct kfree_rcu_cpu *krcp,bool allow_spin) {// If cache disabled, bail out. if (!rcu_min_cached_objs)@@ -1919,15 +2060,10 @@ run_page_cache_worker(struct kfree_rcu_cpu *krcp)if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING && !atomic_xchg (&krcp->work_in_progress,1 )) { -if (atomic_read (&krcp->backoff_page_cache_fill)) { -queue_delayed_work (rcu_reclaim_wq, - &krcp->page_cache_work, -msecs_to_jiffies (rcu_delay_page_cache_fill_msec)); - }else { -hrtimer_setup (&krcp->hrtimer, schedule_page_work_fn,CLOCK_MONOTONIC , -HRTIMER_MODE_REL ); -hrtimer_start (&krcp->hrtimer,0 ,HRTIMER_MODE_REL ); - } +if (allow_spin) +__run_page_cache_worker (krcp); +else +irq_work_queue (&krcp->run_page_cache_worker); } }@@ -1955,7 +2091,7 @@ void __init kfree_rcu_scheduler_running(void)* be free'd in workqueue context. This allows us to: batch requests together to * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ -void kvfree_call_rcu_ptr (struct rcu_ptr *head,void *ptr) +void kvfree_call_rcu_ptr (struct rcu_ptr *head,void *ptr,bool allow_spin) {unsigned long flags;struct kfree_rcu_cpu *krcp;@@ -1971,7 +2107,12 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr)if (!head)might_sleep (); -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) &&kfree_rcu_sheaf (ptr)) +if (!allow_spin && (IS_ENABLED (CONFIG_DEBUG_OBJECTS_RCU_HEAD ) || +IS_ENABLED (CONFIG_DEBUG_KMEMLEAK ))) + goto defer_free; + +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) && + (allow_spin &&kfree_rcu_sheaf (ptr)))return ;// Queue the object but don't yet schedule the batch. @@ -1985,9 +2126,14 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr) }kasan_record_aux_stack (ptr); - success =add_ptr_to_bulk_krc_lock (&krcp, &flags, ptr, !head); + + krcp =krc_this_cpu_lock (&flags, allow_spin); +if (!krcp) + goto defer_free; + + success =add_ptr_to_bulk_krc_lock (krcp, &flags, ptr, !head, allow_spin);if (!success) { -run_page_cache_worker (krcp); +run_page_cache_worker (krcp, allow_spin);if (head ==NULL )// Inline if kvfree_rcu(one_arg) call. @@ -2012,7 +2158,7 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr)// Set timer to drain after KFREE_DRAIN_JIFFIES. if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING ) -__schedule_delayed_monitor_work (krcp); +__schedule_delayed_monitor_work (krcp, allow_spin); unlock_return:krc_this_cpu_unlock (krcp, flags);@@ -2023,10 +2169,22 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr)* CPU can pass the QS state. */ if (!success) { +VM_WARN_ON_ONCE (!allow_spin);debug_rcu_head_unqueue ((struct rcu_head *) ptr);synchronize_rcu ();kvfree (ptr); } +return ; + +defer_free: +VM_WARN_ON_ONCE (allow_spin); +guard (preempt)(); + + krcp =this_cpu_ptr (&krc); +if (llist_add ((struct llist_node *)head, &krcp->defer_head)) +irq_work_queue (&krcp->defer_free); +return ; + }EXPORT_SYMBOL_GPL (kvfree_call_rcu_ptr);@@ -2125,6 +2283,8 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);*/ void kvfree_rcu_barrier_on_cache (struct kmem_cache *s) { +defer_kvfree_rcu_barrier (); +if (cache_has_sheaves (s)) {flush_rcu_sheaves_on_cache (s);rcu_barrier ();mm/slub.c +5
-1@@ -4018,7 +4018,10 @@ static void flush_rcu_sheaf(struct work_struct *w) } -/* needed for kvfree_rcu_barrier() */ +/* +* Needed for kvfree_rcu_barrier(). The caller should invoke +* defer_kvfree_rcu_barrier() before calling this function. +*/ void flush_rcu_sheaves_on_cache (struct kmem_cache *s) {struct slub_flush_work *sfw;@@ -4053,6 +4056,7 @@ void flush_all_rcu_sheaves(void) {struct kmem_cache *s; +defer_kvfree_rcu_barrier ();cpus_read_lock ();mutex_lock (&slab_mutex);Signature · 1 line
2.43.0
As discussed or noted earlier, having third argument and check the entire path with "if (allow_spin)" is not optimal and is not good approach. I do not think it this would be a good fit for mainline.
Also, re-entering the same path with allow_spin = false feels awkward.
I think a better option is to add a separate kvfree_rcu_nmi() helper, or similar, and avoid complicating the generic implementation. Otherwise, the common path risks becoming harder to maintain.
Below is a simple implementation.
<patch>
include/linux/rcupdate.h +11 -0
@@ -1109,6 +1109,7 @@ static inline void rcu_read_unlock_migrate(void)* In mm/slab_common.c, no suitable header to include here. */ void kvfree_call_rcu (struct rcu_head *head,void *ptr); +void kvfree_call_rcu_nolock (struct rcu_head *head,void *ptr);/* * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the
@@ -1132,6 +1133,16 @@ do { \kvfree_call_rcu (NULL , (void *) (___p)); \ } while (0) +#define kvfree_rcu_nmi (ptr, rhf) \ +do { \ + typeof (ptr) ___p = (ptr); \ + \ + if (___p) { \ + BUILD_BUG_ON(offsetof(typeof(*(ptr)), rhf) >= 4096); \ + kvfree_call_rcu_nolock(&((___p)->rhf), (void *) (___p));\ + } \ +} while (0) +/* * Place this after a lock-acquisition primitive to guarantee that * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
mm/slab_common.c +75 -0
92 lines not on this page · raw message
<patch>
I can prepare a patch to handle NMI safe or unknown contexts.
--
Uladzislau Rezki
Paul E. McKenney
maintainerQuoted 13 lines · Vlastimil Babka (SUSE)
Quoted 12 lines · Harry Yoo (Oracle)
# Part 1. Allow an 8-byte field to be used as an alternative to
structrcu_headfor 2-argumentkvfree_rcu()(patches 1-2)Technically, objects that are freed with k[v]free_rcu() need only one pointer to link objects, because we already know that the callback function is always
kvfree(). For this purpose, structrcu_headis unnecessarily large (16 bytes on 64-bit).Allow a smaller, 8-byte field (of struct
rcu_ptrtype) to be used with k[v]free_rcu(). Let's save one pointer per slab object.
Quoted 8 lines
Quoted 2 lines
I have to admit that my naming skill isn't great; hopefully we'll come up with a better name than
struct rcu_ptr."
rcu_ptr" sounds too generic to me. Things likercu_dereference()talk about "RCU-protected pointer" and it could be misleading that "rcu_ptr" has something to do with this. Only if Paul think it's ok :)
But maybe just "kvfree_rcu_head"? As IMHO It's anrcu_headspecialized for kvfree.
What is in a name? ;-)
I do have a slight preference for kvfree_rcu_head.
Thanx, Paul
Harry Yoo (Oracle)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Uladzislau RezkiQuoted 4 lines · Uladzislau Rezki
On Thu, Apr 16, 2026 at 06:10:18PM +0900, Harry Yoo (Oracle) wrote:
As discussed or noted earlier, having third argument and check the entire path with "if (allow_spin)" is not optimal and is not good approach. I do not think it this would be a good fit for mainline.
I understand that this pattern is pretty uncommon and adds complexity. But there is a reason why we're doing this. As Alexei pointed out earlier, even rare spikes of irq_work hurt performance [1]; that can be avoided in most cases where the lock is not in use. That's why buddy [2] and slab [3] accepted this pattern despite the added complexity?
[1] https://lore.kernel.org/linux-mm/CAADnVQKk_Bgi0bc-td_3pVpHYXR3CpC3R8rg-NHwdLEDiQSeNg@mail.gmail.com
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] https://lore.kernel.org/linux-mm/[email protected]
Quoted 3 lines
I think a better option is to add a separate
kvfree_rcu_nmi()helper, or similar, and avoid complicating the generic implementation. Otherwise, the common path risks becoming harder to maintain.
It's unfortunate that I didn't provide performance data to justify adding complexity. I can try, but apparently you're not talking about that.
Rather, you seem to be arguing that you're against this pattern no matter what the benefits are?
I'm confused.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerQuoted 10 lines · Vlastimil Babka (SUSE)
Quoted 9 lines · Harry Yoo (Oracle)
# Part 1. Allow an 8-byte field to be used as an alternative to
structrcu_headfor 2-argumentkvfree_rcu()(patches 1-2)Technically, objects that are freed with k[v]free_rcu() need only one pointer to link objects, because we already know that the callback function is always
kvfree(). For this purpose, structrcu_headis unnecessarily large (16 bytes on 64-bit).
Quoted 7 lines
Quoted 5 lines
Allow a smaller, 8-byte field (of struct
rcu_ptrtype) to be used with k[v]free_rcu(). Let's save one pointer per slab object.I have to admit that my naming skill isn't great; hopefully we'll come up with a better name than
struct rcu_ptr."
rcu_ptr" sounds too generic to me.
to me too ;)
Quoted 3 lines
Things like
rcu_dereference()talk about
"RCU-protected pointer" and it could be misleading that "rcu_ptr" has something to do with this.
Agreed.
Quoted 2 lines
But maybe just "
kvfree_rcu_head"? As IMHO It's anrcu_headspecialized for kvfree.
I like the name better and will rename it, thanks!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerRe: [PATCH 1/8] mm/slab: introduce k[v]free_rcu() with struct rcu_ptr
In reply to Vlastimil Babka (SUSE)Quoted 42 lines · Vlastimil Babka (SUSE)
Quoted 18 lines · Harry Yoo (Oracle)
mm/slab_common.c +6
-4@@ -1523,18 +1522,34 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, }static void -kvfree_rcu_list (struct rcu_head *head) +kvfree_rcu_list (struct rcu_ptr *head) { -struct rcu_head *next; +struct rcu_ptr *next;for (; head; head = next) { -void *ptr = (void *) head->func; -unsigned long offset = (void *) head - ptr; +void *ptr; +unsigned long offset; +struct slab *slab; +if (is_vmalloc_addr (head)) {mm/slab_common.c kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, · line 1544
Quoted 10 lines
mm/slab_common.c +10
-0+ ptr = (void *)PAGE_ALIGN_DOWN ((unsigned long )head); + }else { + slab =virt_to_slab (head); +if (!slab) + ptr = (void *)PAGE_ALIGN_DOWN ((unsigned long )head); +else if (is_kfence_address (head)) + ptr =kfence_object_start (head); +else + ptr =nearest_obj (slab->slab_cache, slab, head); + }It seems a bit wasteful to do this... (especially the
virt_to_slab()) part.Quoted 10 lines
mm/slab_common.c +5
-1+ + offset = (void *)head - ptr; next = head->next;debug_rcu_head_unqueue ((struct rcu_head *)ptr);rcu_lock_acquire (&rcu_callback_map); -trace_rcu_invoke_kvfree_callback ("slab" , head, offset); +trace_rcu_invoke_kvfree_callback ("slab" , + (struct rcu_head *)head, + offset);
Quoted 5 lines
Review kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, · line 1554
Quoted 1 line
(quoted diff) +0
-0kvfree(ptr);... and then throw it all away and let kvfree(ptr) do it again. So maybe we could then call something more internal and specific based on the result of the above.
Good point, will do.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Quoted 11 lines · Harry Yoo (Oracle)
Quoted 4 lines · Uladzislau Rezki
On Thu, Apr 16, 2026 at 06:10:18PM +0900, Harry Yoo (Oracle) wrote:
As discussed or noted earlier, having third argument and check the entire path with "if (allow_spin)" is not optimal and is not good approach. I do not think it this would be a good fit for mainline.I understand that this pattern is pretty uncommon and adds complexity. But there is a reason why we're doing this. As Alexei pointed out earlier, even rare spikes of
irq_workhurt performance [1]; that can be avoided in most cases where the lock is not in use. That's why buddy [2] and slab [3] accepted this pattern despite the added complexity?
Perhaps we can avoid complicating existing kvfree_rcu batching (and still avoid hurting performance).
Given that kfree_rcu_nolock() (at least BPF) doesn't need to free vmalloc objects, we can make kfree_rcu_sheaf() handle most of frees (slab already knows how to handle allow_spin = false) and fall back to a simple irq_work as Ulad suggested.
But then we need to make kfree_rcu_sheaf() try harder (currently it gives up when it fails to get an empty sheaf from the barn).
And I guess the tricky part is making kfree_rcu_sheaf() work on RT ;)
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Quoted 6 lines · Harry Yoo (Oracle)
Quoted 4 lines · Uladzislau Rezki
On Thu, Apr 16, 2026 at 06:10:18PM +0900, Harry Yoo (Oracle) wrote:
I think a better option is to add a separatekvfree_rcu_nmi()helper, or similar, and avoid complicating the generic implementation. Otherwise, the common path risks becoming harder to maintain.
Quoted 8 lines
It's unfortunate that I didn't provide performance data to justify adding complexity. I can try, but apparently you're not talking about that.
Rather, you seem to be arguing that you're against this pattern no matter what the benefits are?
I'm confused.
I was indeed confused, but before writing this, I should have thought that you might be considering a simple but still efficient solution.
Apologies, my reply may have sounded a bit offensive.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Uladzislau RezkiQuoted 5 lines · Uladzislau Rezki
I think a better option is to add a separate
kvfree_rcu_nmi()helper, or similar, and avoid complicating the generic implementation. Otherwise, the common path risks becoming harder to maintain.Below is a simple implementation.
I'm happy to keep things simple as long as that doesn't mean compromising performance.
We can discuss that.
mm/slab_common.c struct kfree_rcu_cpu { · line 1406
Quoted 11 lines
<patch>
mm/slab_common.c +2
-0@@ -1402,6 +1402,14 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; + +/* For NMI context. */
I think "unknown context" is a better term since it includes NMI context as well as other contexts.
(I'm also slightly moving towards the term, :D)
mm/slab_common.c struct kfree_rcu_cpu { · line 1413
Quoted 7 lines
mm/slab_common.c +6
-0+struct llist_head drain_list; +struct llist_node *pending_list; + +struct rcu_work drain_rcu_work; +struct irq_work drain_irqwork; +atomic_t drain_in_progress; };
[... changing the order of functions a little bit to help review ...]
Quoted 7 lines
mm/slab_common.c +3
-0static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = {@@ -1926,6 +1934,69 @@ void __init kfree_rcu_scheduler_running(void) } } + +/* +* Queue a request for lazy invocation.
mm/slab_common.c void __init kfree_rcu_scheduler_running(void) · line 1948
Quoted 10 lines
mm/slab_common.c +10
-0+* Context: For NMI contexts or unknown contexts only. +*/ +void +kvfree_call_rcu_nolock (struct rcu_head *head,void *ptr) +{ +struct kfree_rcu_cpu *krcp =this_cpu_ptr (&krc); + + head->func = ptr; +llist_add ((struct llist_node *) head, &krcp->drain_list); +
So it inserts objects to the list,
mm/slab_common.c void __init kfree_rcu_scheduler_running(void) · line 1995
Quoted 4 lines
mm/slab_common.c +4
-0+if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING ) { +/* Only first(and only one) user rings the bell. */ +if (!atomic_cmpxchg (&krcp->drain_in_progress,0 ,1 )) +irq_work_queue (&krcp->drain_irqwork);
and only the task that succeeds cmpxchg queues the IRQ work. The IRQ work queues an RCU work which iterates over the list of objects and frees them.
Draining will be performed a little bit more frequently (every call_rcu_hurry() + work queue delay) compared to the ordinary kvfree_rcu path (every 1-5 seconds)
The question is how frequent is too frequent, when it comes to additional IRQ/RCU work invocations affecting performance.
Quoted 19 lines
mm/slab_common.c +18
-0+static void +kvfree_rcu_nolock_irqwork (struct irq_work *irqwork) +{ +struct kfree_rcu_cpu *krcp = + container_of(irqwork, struct kfree_rcu_cpu, drain_irqwork); > + bool queued; + + krcp->pending_list =llist_del_all (&krcp->drain_list); +ASSERT_EXCLUSIVE_WRITER (krcp->pending_list); + queued =queue_rcu_work (rcu_reclaim_wq, &krcp->drain_rcu_work); +WARN_ON_ONCE (!queued); +} +static void +kvfree_rcu_nolock_work (struct work_struct *work) +{ +struct kfree_rcu_cpu *krcp =container_of (to_rcu_work (work), + struct kfree_rcu_cpu, drain_rcu_work); +struct llist_node *pos, *n, *pending; +bool queued;
mm/slab_common.c void __init kfree_rcu_scheduler_running(void) · line 1953
Quoted 10 lines
mm/slab_common.c +10
-0+ + pending = krcp->pending_list; + krcp->pending_list =NULL ; +ASSERT_EXCLUSIVE_WRITER (krcp->pending_list); + +llist_for_each_safe (pos, n, pending) { +struct rcu_head *rcu = (struct rcu_head *) pos; +void *ptr = (void *) rcu->func; +kvfree (ptr); + }
This is pretty similar to what a kvfree_rcu(two_arg) call does in the slowpath (kvfree_rcu_list), except that we don't maintain RCU state explicitly.
How much performance do we sacrifice compared to letting them go through the kvfree_rcu() fastpath?
Quoted 17 lines
mm/slab_common.c +14
-0+atomic_set (&krcp->drain_in_progress,0 ); +if (!llist_empty (&krcp->drain_list)) { +if (!atomic_cmpxchg (&krcp->drain_in_progress,0 ,1 )) { + krcp->pending_list =llist_del_all (&krcp->drain_list); +ASSERT_EXCLUSIVE_WRITER (krcp->pending_list); + queued =queue_rcu_work (rcu_reclaim_wq, &krcp->drain_rcu_work); +WARN_ON_ONCE (!queued); + } + } +} + } +} +EXPORT_SYMBOL_GPL (kvfree_call_rcu_nolock); +/* * Queue a request for lazy invocation of the appropriate free routine * after a grace period. Please note that three paths are maintained,
Signature · 2 lines
Cheers, Harry / Hyeonggon
Uladzislau Rezki
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
Quoted 108 lines · Harry Yoo (Oracle)
Quoted 5 lines · Uladzislau Rezki
I think a better option is to add a separate
kvfree_rcu_nmi()helper, or similar, and avoid complicating the generic implementation. Otherwise, the common path risks becoming harder to maintain.Below is a simple implementation.
I'm happy to keep things simple as long as that doesn't mean compromising performance.
We can discuss that.
mm/slab_common.c struct kfree_rcu_cpu { · line 1406
Quoted 11 lines
<patch>
mm/slab_common.c +2
-0@@ -1402,6 +1402,14 @@ struct kfree_rcu_cpu {struct llist_head bkvcache;int nr_bkv_objs; + +/* For NMI context. */ I think "unknown context" is a better term since it includes NMI context as well as other contexts.
(I'm also slightly moving towards the term, :D)mm/slab_common.c struct kfree_rcu_cpu { · line 1413
Quoted 7 lines
mm/slab_common.c +6
-0+struct llist_head drain_list; +struct llist_node *pending_list; + +struct rcu_work drain_rcu_work; +struct irq_work drain_irqwork; +atomic_t drain_in_progress; };[... changing the order of functions a little bit to help review ...]
Quoted 7 lines
mm/slab_common.c +3
-0static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = {@@ -1926,6 +1934,69 @@ void __init kfree_rcu_scheduler_running(void) } } + +/* +* Queue a request for lazy invocation. mm/slab_common.c void __init kfree_rcu_scheduler_running(void) · line 1948
Quoted 10 lines
mm/slab_common.c +10
-0+* Context: For NMI contexts or unknown contexts only. +*/ +void +kvfree_call_rcu_nolock (struct rcu_head *head,void *ptr) +{ +struct kfree_rcu_cpu *krcp =this_cpu_ptr (&krc); + + head->func = ptr; +llist_add ((struct llist_node *) head, &krcp->drain_list); +So it inserts objects to the list,
mm/slab_common.c void __init kfree_rcu_scheduler_running(void) · line 1995
Quoted 4 lines
mm/slab_common.c +4
-0+if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING ) { +/* Only first(and only one) user rings the bell. */ +if (!atomic_cmpxchg (&krcp->drain_in_progress,0 ,1 )) +irq_work_queue (&krcp->drain_irqwork);and only the task that succeeds cmpxchg queues the IRQ work. The IRQ work queues an RCU work which iterates over the list of objects and frees them.
Draining will be performed a little bit more frequently (every
call_rcu_hurry()+ work queue delay) compared to the ordinarykvfree_rcupath (every 1-5 seconds)The question is how frequent is too frequent, when it comes to additional IRQ/RCU work invocations affecting performance.
Quoted 19 lines
mm/slab_common.c +18
-0+static void +kvfree_rcu_nolock_irqwork (struct irq_work *irqwork) +{ +struct kfree_rcu_cpu *krcp = + container_of(irqwork, struct kfree_rcu_cpu, drain_irqwork); > + bool queued; + + krcp->pending_list =llist_del_all (&krcp->drain_list); +ASSERT_EXCLUSIVE_WRITER (krcp->pending_list); + queued =queue_rcu_work (rcu_reclaim_wq, &krcp->drain_rcu_work); +WARN_ON_ONCE (!queued); +} +static void +kvfree_rcu_nolock_work (struct work_struct *work) +{ +struct kfree_rcu_cpu *krcp =container_of (to_rcu_work (work), + struct kfree_rcu_cpu, drain_rcu_work); +struct llist_node *pos, *n, *pending; +bool queued;mm/slab_common.c void __init kfree_rcu_scheduler_running(void) · line 1953
Quoted 10 lines
mm/slab_common.c +10
-0+ + pending = krcp->pending_list; + krcp->pending_list =NULL ; +ASSERT_EXCLUSIVE_WRITER (krcp->pending_list); + +llist_for_each_safe (pos, n, pending) { +struct rcu_head *rcu = (struct rcu_head *) pos; +void *ptr = (void *) rcu->func; +kvfree (ptr); + }This is pretty similar to what a
kvfree_rcu(two_arg)call does in the slowpath (kvfree_rcu_list), except that we don't maintain RCU state explicitly.
Quoted 3 lines
How much performance do we sacrifice compared to letting them go through the
kvfree_rcu()fastpath?
I think we may be focusing too much on performance here. In fact it depends on use cases which we try to improve or fix. Freeing an object over RCU from NMI context is a corner case. It is __not_ generic. We even do not have(now in mainline) users because we never support it from NMI, just like call_rcu().
From the other hand, if you heavily reclaim from NMI, this is likely not a common or intended usage pattern this is how i understand it. If BPF needs it, then the first question which comes to mind is not about performance. It is how to support this case in kfree_rcu() without adding noticeable complexity or overhead or hacks to the generic path without making it harder to maintain.
Performance wise you noted, you mean:
a) call latency(this is probably the most important for NMI)?
b) memory footprint?
c) pointer-chasing overhead?
Please note, when we are talking about NMI support, we are in a special conditions thus we should no overthinking here. This is how i look at it.
--
Uladzislau Rezki
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Quoted 4 lines · Harry Yoo (Oracle)
mm/slab_common.c +4
-0+#ifndef CONFIG_KVFREE_RCU_BATCHED +void kvfree_call_rcu_head (struct rcu_head *head,void *ptr,bool allow_spin) +{ +if (!allow_spin) {
mm/slab_common.c struct kfree_rcu_cpu { · line 1436
Quoted 10 lines
mm/slab_common.c +10
-0+struct kfree_rcu_cpu *krcp; + +guard (preempt)(); + + krcp =this_cpu_ptr (&krc); +if (llist_add ((struct llist_node *)head, &krcp->defer_head)) +irq_work_queue (&krcp->defer_free); +return ; + } +
Given "allow_spin" is compile-time constant, it would make more sense to have different functions for each variant rather than calling a common one with the extra param?
Quoted 3 lines
mm/slab_common.c +0
-0if (head) {kasan_record_aux_stack (ptr);call_rcu (head, kvfree_rcu_cb);
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 5/8] mm/slab: make kfree_rcu_nolock() work with sheaves
In reply to Harry Yoo (Oracle)Quoted 51 lines · Harry Yoo (Oracle)
Teach
kfree_rcu_sheaf()how to handle the !allow_spin case. Similar to__pcs_replace_full_main(), try to get an empty sheaf frompcs->spareor the barn, but don't add !allow_spin support foralloc_empty_sheaf()and fail early instead.Since
call_rcu()does not support NMI contexts,kfree_rcu_sheaf()fails when the rcu sheaf becomes full.Signed-off-by Harry Yoo (Oracle) <[email protected]>--- mm/slab.h | 2 +- mm/slab_common.c | 7 +++---- mm/slub.c | 14 ++++++++++++-- 3 files changed, 16 insertions(+), 7 deletions(-)mm/slab.h +1
-1@@ -409,7 +409,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)return !(s->flags & (SLAB_CACHE_DMA |SLAB_ACCOUNT |SLAB_RECLAIM_ACCOUNT )); } -bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj); +bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin);void flush_all_rcu_sheaves (void );void flush_rcu_sheaves_on_cache (struct kmem_cache *s);void defer_kvfree_rcu_barrier (void );mm/slab_common.c +2
-2@@ -1716,7 +1716,7 @@ static void kfree_rcu_work(struct work_struct *work)kvfree_rcu_list (head); } -static bool kfree_rcu_sheaf (void *obj) +static bool kfree_rcu_sheaf (void *obj,bool allow_spin) {struct kmem_cache *s;struct slab *slab;@@ -1730,7 +1730,7 @@ static bool kfree_rcu_sheaf(void *obj) s = slab->slab_cache;if (likely (!IS_ENABLED (CONFIG_NUMA ) ||slab_nid (slab) ==numa_mem_id ())) -return __kfree_rcu_sheaf (s, obj); +return __kfree_rcu_sheaf (s, obj, allow_spin);return false; }
mm/slab_common.c void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr, bool allow_spin) · line 2115
Quoted 7 lines
mm/slab_common.c +1
-2@@ -2111,8 +2111,7 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr, bool allow_spin) IS_ENABLED(CONFIG_DEBUG_KMEMLEAK))) goto defer_free; -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) && - (allow_spin &&kfree_rcu_sheaf (ptr))) +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) &&kfree_rcu_sheaf (ptr, allow_spin))return ;
I wonder if this patch is enough to drop (or reduce a lot) the PREEMPT_RT incompatibility? It IIRC came from Ulad's review point [1] that since RT spinlock is in fact a mutex, it's not unsafe to take it with e.g. disabled preemption, and it's also why kfree_rcu_cpu uses a raw_spin_lock_t, etc.
But with kfree_rcu_sheaf now having allow_spin parameter and only doing a trylock at most with that false, it should now be safe to allow it for CONFIG_PREEMPT_RT, only pass allow_spin = false unconditionally there?
Quoted 48 lines
mm/slab_common.c +0
-0// Queue the object but don't yet schedule the batch. mm/slub.c +12
-2@@ -5895,7 +5895,7 @@ static void rcu_free_sheaf(struct rcu_head *head)*/ static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG); -bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj) +bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin) {struct slub_percpu_sheaves *pcs;struct slab_sheaf *rcu_sheaf;@@ -5933,7 +5933,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) goto fail; } - empty =barn_get_empty_sheaf (barn, true); + empty =barn_get_empty_sheaf (barn, allow_spin);if (empty) { pcs->rcu_free = empty;@@ -5942,6 +5942,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)local_unlock (&s->cpu_sheaves->lock); +/* It's easier to fall back than trying harder with !allow_spin */ +if (!allow_spin) + goto fail; + empty =alloc_empty_sheaf (s,GFP_NOWAIT );if (!empty)@@ -5973,6 +5977,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)if (likely (rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =NULL ; }else { +if (unlikely (!allow_spin)) { +/* call_rcu() cannot be called in an unknown context */ + rcu_sheaf->size--; +local_unlock (&s->cpu_sheaves->lock); + goto fail; + } pcs->rcu_free =NULL ; rcu_sheaf->node =numa_node_id (); }
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 5/8] mm/slab: make kfree_rcu_nolock() work with sheaves
Quoted 15 lines · Vlastimil Babka (SUSE)
mm/slab_common.c void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr, bool allow_spin) · line 2115
Quoted 8 lines · Harry Yoo (Oracle)
mm/slab_common.c +1
-2@@ -2111,8 +2111,7 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr, bool allow_spin) IS_ENABLED(CONFIG_DEBUG_KMEMLEAK))) goto defer_free; -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) && - (allow_spin &&kfree_rcu_sheaf (ptr))) +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) &&kfree_rcu_sheaf (ptr, allow_spin))return ;I wonder if this patch is enough to drop (or reduce a lot) the
PREEMPT_RTincompatibility? It IIRC came from Ulad's review point [1] that since RT spinlock is in fact a mutex, it's not unsafe to take it with e.g. disabled preemption, and it's also whykfree_rcu_cpuuses araw_spin_lock_t, etc.
Quoted 3 lines
But with
kfree_rcu_sheafnow havingallow_spinparameter and only doing a trylock at most with that false, it should now be safe to allow it forCONFIG_PREEMPT_RT, only passallow_spin= false unconditionally there?
Oh I forgot about...
Quoted 36 lines
[1] https://lore.kernel.org/all/aL_uhPtztx7Ef0T2@pc636/
Quoted 34 lines
mm/slab_common.c +0
-0// Queue the object but don't yet schedule the batch. mm/slub.c +6
-2@@ -5895,7 +5895,7 @@ static void rcu_free_sheaf(struct rcu_head *head)*/ static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG); -bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj) +bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin) {struct slub_percpu_sheaves *pcs;struct slab_sheaf *rcu_sheaf;@@ -5933,7 +5933,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) goto fail; } - empty =barn_get_empty_sheaf (barn, true); + empty =barn_get_empty_sheaf (barn, allow_spin);if (empty) { pcs->rcu_free = empty;@@ -5942,6 +5942,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)local_unlock (&s->cpu_sheaves->lock); +/* It's easier to fall back than trying harder with !allow_spin */ +if (!allow_spin) + goto fail; + empty =alloc_empty_sheaf (s,GFP_NOWAIT );
Quoted 11 lines
mm/slub.c bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) · line 5985
Quoted 11 lines
mm/slub.c +6
-0if (!empty)@@ -5973,6 +5977,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)if (likely (rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =NULL ; }else { +if (unlikely (!allow_spin)) { +/* call_rcu() cannot be called in an unknown context */ + rcu_sheaf->size--; +local_unlock (&s->cpu_sheaves->lock); + goto fail; + }
... this part, so an unconditional allow_spin=false on RT would only fill the rcu_sheaf once and then stop working.
However, can we condition this on in_nmi() or something, and pretend the other cases of "unknown context" exist? Installing BPF hooks allocating/freeing memory into call_rcu() internals sounds crazy, no?
Quoted 4 lines
Quoted 3 lines
mm/slub.c +0
-0pcs->rcu_free =NULL ; rcu_sheaf->node =numa_node_id (); }
Re: [PATCH 5/8] mm/slab: make kfree_rcu_nolock() work with sheaves
Quoted 74 lines · Vlastimil Babka (SUSE)
Quoted 15 lines · Vlastimil Babka (SUSE)
mm/slab_common.c void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr, bool allow_spin) · line 2115
Quoted 8 lines · Harry Yoo (Oracle)
mm/slab_common.c +1
-2@@ -2111,8 +2111,7 @@ void kvfree_call_rcu_ptr(struct rcu_ptr *head, void *ptr, bool allow_spin) IS_ENABLED(CONFIG_DEBUG_KMEMLEAK))) goto defer_free; -if (!IS_ENABLED (CONFIG_PREEMPT_RT ) && - (allow_spin &&kfree_rcu_sheaf (ptr))) +if (!IS_ENABLED (CONFIG_PREEMPT_RT ) &&kfree_rcu_sheaf (ptr, allow_spin))return ;I wonder if this patch is enough to drop (or reduce a lot) the
PREEMPT_RTincompatibility? It IIRC came from Ulad's review point [1] that since RT spinlock is in fact a mutex, it's not unsafe to take it with e.g. disabled preemption, and it's also whykfree_rcu_cpuuses araw_spin_lock_t, etc.Quoted 3 lines
But with
kfree_rcu_sheafnow havingallow_spinparameter and only doing a trylock at most with that false, it should now be safe to allow it forCONFIG_PREEMPT_RT, only passallow_spin= false unconditionally there?Oh I forgot about...
Quoted 36 lines
[1] https://lore.kernel.org/all/aL_uhPtztx7Ef0T2@pc636/
Quoted 34 lines
mm/slab_common.c +0
-0// Queue the object but don't yet schedule the batch. mm/slub.c +6
-2@@ -5895,7 +5895,7 @@ static void rcu_free_sheaf(struct rcu_head *head)*/ static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG); -bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj) +bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin) {struct slub_percpu_sheaves *pcs;struct slab_sheaf *rcu_sheaf;@@ -5933,7 +5933,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) goto fail; } - empty =barn_get_empty_sheaf (barn, true); + empty =barn_get_empty_sheaf (barn, allow_spin);if (empty) { pcs->rcu_free = empty;@@ -5942,6 +5942,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)local_unlock (&s->cpu_sheaves->lock); +/* It's easier to fall back than trying harder with !allow_spin */ +if (!allow_spin) + goto fail; + empty =alloc_empty_sheaf (s,GFP_NOWAIT );Quoted 11 lines
mm/slub.c bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) · line 5985
Quoted 11 lines
mm/slub.c +6
-0if (!empty)@@ -5973,6 +5977,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)if (likely (rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =NULL ; }else { +if (unlikely (!allow_spin)) { +/* call_rcu() cannot be called in an unknown context */ + rcu_sheaf->size--; +local_unlock (&s->cpu_sheaves->lock); + goto fail; + }... this part, so an unconditional allow_spin=false on RT would only fill the
rcu_sheafonce and then stop working.
Quoted 3 lines
However, can we condition this on
in_nmi()or something, and pretend the other cases of "unknown context" exist? Installing BPF hooks allocating/freeing memory intocall_rcu()internals sounds crazy, no?
Unfortunately no. People attach to tracepoint lock contention which can be inside call_rcu.
As discussed call_rcu_nolock() is only way forward.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 5/8] mm/slab: make kfree_rcu_nolock() work with sheaves
Quoted 16 lines · Alexei Starovoitov
Quoted 14 lines · Vlastimil Babka (SUSE)
Quoted 10 lines
mm/slub.c bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) · line 5985
Quoted 10 lines
mm/slub.c +6
-0@@ -5973,6 +5977,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)if (likely (rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =NULL ; }else { +if (unlikely (!allow_spin)) { +/* call_rcu() cannot be called in an unknown context */ + rcu_sheaf->size--; +local_unlock (&s->cpu_sheaves->lock); + goto fail; + }... this part, so an unconditional allow_spin=false on RT would only fill the
rcu_sheafonce and then stop working.
Quoted 7 lines
Quoted 3 lines
However, can we condition this on
in_nmi()or something, and pretend the other cases of "unknown context" exist? Installing BPF hooks allocating/freeing memory intocall_rcu()internals sounds crazy, no?Unfortunately no. People attach to tracepoint lock contention which can be inside
call_rcu.
As discussedcall_rcu_nolock()is only way forward.
Alright. But we assume those tracepoint-attached callers know to use kfree_rcu_nolock(), while kfree_rcu() callers under PREEMPT_RT can still assume it's not called in NMI or tracepoint-attached context, right?
So I think what __kfree_rcu_sheaf() could do is
- with CONFIG_PREEMPT_RT, treat allow_spin == true as allow_spin == false for purposes of barn_get_empty_sheaf() and to bail out before trying alloc_empty_sheaf() - this is to avoid any lock activity that's not a trylock.
- allow_spin controls ability to do call_rcu() as above, regardless of PREEMPT_RT, to avoid it from kfree_rcu_nolock()
Re: [PATCH 5/8] mm/slab: make kfree_rcu_nolock() work with sheaves
Quoted 22 lines · Vlastimil Babka (SUSE)
Quoted 20 lines · Alexei Starovoitov
Quoted 17 lines · Vlastimil Babka (SUSE)
Quoted 10 lines
mm/slub.c bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) · line 5985
Quoted 10 lines
mm/slub.c +6
-0@@ -5973,6 +5977,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)if (likely (rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =NULL ; }else { +if (unlikely (!allow_spin)) { +/* call_rcu() cannot be called in an unknown context */ + rcu_sheaf->size--; +local_unlock (&s->cpu_sheaves->lock); + goto fail; + }... this part, so an unconditional allow_spin=false on RT would only fill the
rcu_sheafonce and then stop working.However, can we condition this on
in_nmi()or something, and pretend the other cases of "unknown context" exist? Installing BPF hooks allocating/freeing memory intocall_rcu()internals sounds crazy, no?
Quoted 7 lines
Quoted 3 lines
Unfortunately no. People attach to tracepoint lock contention which can be inside
call_rcu.
As discussedcall_rcu_nolock()is only way forward.Alright. But we assume those tracepoint-attached callers know to use
kfree_rcu_nolock(), whilekfree_rcu()callers underPREEMPT_RTcan still assume it's not called in NMI or tracepoint-attached context, right?
right. I see what you mean.
Quoted 6 lines
So I think what
__kfree_rcu_sheaf()could do is
- withCONFIG_PREEMPT_RT, treatallow_spin== true asallow_spin== false for purposes ofbarn_get_empty_sheaf()and to bail out before tryingalloc_empty_sheaf()- this is to avoid any lock activity that's not a trylock.
-allow_spincontrols ability to docall_rcu()as above, regardless ofPREEMPT_RT, to avoid it fromkfree_rcu_nolock()
makes sense.
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 6/8] mm/slab: wrap rcu sheaf handling with ifdef
In reply to Harry Yoo (Oracle)Quoted 3 lines · Harry Yoo (Oracle)
Freeing objects via rcu sheaves is only done with
CONFIG_KVFREE_RCU_BATCHED. Wrap the related functions and struct fields with ifdef to make this dependency explicit.
With SLUB_TINY we have recently took the approach of limiting it to only necessary parts, and not care about unused code etc where it doesn't cause issues. I think it's good to minimize the ifdef complexity, so maybe it's possible to limit it only to allow compiling with no warnings (including IIRC W=1 clang that IIRC is stricter about unused functions)?
Quoted 3 lines
Also remove a TODO about implementing
__kvfree_rcu_barrier_on_cache()for a specific slab cache, as there doesn't seem to be a simple and effective way to do so.
Ack.
Quoted 172 lines
Signed-off-by Harry Yoo (Oracle) <[email protected]>--- mm/slab.h | 3 +++ mm/slab_common.c | 4 ---- mm/slub.c | 27 +++++++++++++++++++++++++-- 3 files changed, 28 insertions(+), 6 deletions(-)mm/slab.h +3
-0@@ -409,9 +409,12 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)return !(s->flags & (SLAB_CACHE_DMA |SLAB_ACCOUNT |SLAB_RECLAIM_ACCOUNT )); } +#ifdef CONFIG_KVFREE_RCU_BATCHED bool __kfree_rcu_sheaf (struct kmem_cache *s,void *obj,bool allow_spin);void flush_all_rcu_sheaves (void );void flush_rcu_sheaves_on_cache (struct kmem_cache *s); +#endif +void defer_kvfree_rcu_barrier (void );#define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \mm/slab_common.c +0
-4@@ -2289,10 +2289,6 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)rcu_barrier (); } -/* -* TODO: Introduce a version of __kvfree_rcu_barrier() that works -* on a specific slab cache. -*/ __kvfree_rcu_barrier (); }EXPORT_SYMBOL_GPL (kvfree_rcu_barrier_on_cache);mm/slub.c +25
-2@@ -421,7 +421,9 @@ struct slub_percpu_sheaves {local_trylock_t lock;struct slab_sheaf *main;/* never NULL when unlocked */ struct slab_sheaf *spare;/* empty or full, may be NULL */ +#ifdef CONFIG_KVFREE_RCU_BATCHED struct slab_sheaf *rcu_free;/* for batching kfree_rcu() */ +#endif };/* @@ -2923,6 +2925,7 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf) sheaf->size =0 ; } +#ifdef CONFIG_KVFREE_RCU_BATCHED static bool __rcu_free_sheaf_prepare (struct kmem_cache *s,struct slab_sheaf *sheaf) {@@ -2965,6 +2968,7 @@ static void rcu_free_sheaf_nobarn(struct rcu_head *head) free_empty_sheaf(s, sheaf); } +#endif /* * Caller needs to make sure migration is disabled in order to fully flush @@ -2978,7 +2982,10 @@ static void rcu_free_sheaf_nobarn(struct rcu_head *head)static void pcs_flush_all (struct kmem_cache *s) {struct slub_percpu_sheaves *pcs; -struct slab_sheaf *spare, *rcu_free; +struct slab_sheaf *spare; +#ifdef CONFIG_KVFREE_RCU_BATCHED +struct slab_sheaf *rcu_free; +#endif local_lock (&s->cpu_sheaves->lock); pcs =this_cpu_ptr (s->cpu_sheaves);@@ -2986,8 +2993,10 @@ static void pcs_flush_all(struct kmem_cache *s) spare = pcs->spare; pcs->spare =NULL ; +#ifdef CONFIG_KVFREE_RCU_BATCHED rcu_free = pcs->rcu_free; pcs->rcu_free =NULL ; +#endif local_unlock (&s->cpu_sheaves->lock);@@ -2996,8 +3005,10 @@ static void pcs_flush_all(struct kmem_cache *s)free_empty_sheaf (s, spare); } +#ifdef CONFIG_KVFREE_RCU_BATCHED if (rcu_free)call_rcu (&rcu_free->rcu_head, rcu_free_sheaf_nobarn); +#endif sheaf_flush_main (s); }@@ -3016,10 +3027,12 @@ static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu) pcs->spare =NULL ; } +#ifdef CONFIG_KVFREE_RCU_BATCHED if (pcs->rcu_free) {call_rcu (&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn); pcs->rcu_free =NULL ; } +#endif }static void pcs_destroy (struct kmem_cache *s)@@ -3056,7 +3069,9 @@ static void pcs_destroy(struct kmem_cache *s)*/ WARN_ON (pcs->spare); +#ifdef CONFIG_KVFREE_RCU_BATCHED WARN_ON (pcs->rcu_free); +#endif if (!WARN_ON (pcs->main->size)) {free_empty_sheaf (s, pcs->main);@@ -3937,7 +3952,11 @@ static bool has_pcs_used(int cpu, struct kmem_cache *s) pcs =per_cpu_ptr (s->cpu_sheaves, cpu); -return (pcs->spare || pcs->rcu_free || pcs->main->size); +#ifdef CONFIG_KVFREE_RCU_BATCHED +if (pcs->rcu_free) +return true; +#endif +return (pcs->spare || pcs->main->size); }/* @@ -3995,6 +4014,7 @@ static void flush_all(struct kmem_cache *s)cpus_read_unlock (); } +#ifdef CONFIG_KVFREE_RCU_BATCHED static void flush_rcu_sheaf (struct work_struct *w) {struct slub_percpu_sheaves *pcs;@@ -4071,6 +4091,7 @@ void flush_all_rcu_sheaves(void) rcu_barrier(); } +#endif /* CONFIG_KVFREE_RCU_BATCHED */ static int slub_cpu_setup (unsigned int cpu) {@@ -5825,6 +5846,7 @@ bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin)return true; } +#ifdef CONFIG_KVFREE_RCU_BATCHED static void rcu_free_sheaf (struct rcu_head *head) {struct slab_sheaf *sheaf;@@ -6005,6 +6027,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)lock_map_release (&kfree_rcu_sheaf_map);return false; } +#endif /* CONFIG_KVFREE_RCU_BATCHED */ static __always_inline bool can_free_to_pcs(struct slab *slab) {
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 7/8] mm/slab: introduce deferred submission of rcu sheaves
In reply to Harry Yoo (Oracle)Quoted 4 lines · Harry Yoo (Oracle)
Instead of falling back when the rcu sheaf becomes full, implement deferred submission of rcu sheaves. If
kfree_rcu_sheaf()is invoked bykfree_rcu_nolock()(!allow_spin) and IRQs are disabled, the CPU might be in the middle ofcall_rcu()and thus defercall_rcu()withirq_work.
Seems like we're doing two things here. Implementing a fallback is the advertised one. But also narrowing the definition of "unknown context where call_rcu() is unsafe" to mean irqs_disabled().
Where does that condition (!irqs_disabled() means we're safe) come from exactly, does RCU guarantee it anywhere? And if that condition is indeed reliable, shouldn't it have been used by patch 5 already?
Quoted 9 lines
Submit all deferred RCU sheaves to
call_rcu()before callingrcu_barrier()to ensure the promise ofkvfree_rcu_barrier().An alternative approach could be to implement this in the RCU subsystem, tracking if it's safe to call
call_rcu()and allowing falling back to deferredcall_rcu()at the cost of more expensivercu_barrier()calls.Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Harry Yoo (Oracle) <[email protected]>
Vlastimil Babka (SUSE)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Harry Yoo (Oracle)Quoted 16 lines · Harry Yoo (Oracle)
Quoted 11 lines · Harry Yoo (Oracle)
Quoted 4 lines · Uladzislau Rezki
On Thu, Apr 16, 2026 at 06:10:18PM +0900, Harry Yoo (Oracle) wrote:
As discussed or noted earlier, having third argument and check the entire path with "if (allow_spin)" is not optimal and is not good approach. I do not think it this would be a good fit for mainline.I understand that this pattern is pretty uncommon and adds complexity. But there is a reason why we're doing this. As Alexei pointed out earlier, even rare spikes of
irq_workhurt performance [1]; that can be avoided in most cases where the lock is not in use. That's why buddy [2] and slab [3] accepted this pattern despite the added complexity?Perhaps we can avoid complicating existing
kvfree_rcubatching (and still avoid hurting performance).
Quoted 7 lines
Given that
kfree_rcu_nolock()(at least BPF) doesn't need to free vmalloc objects, we can makekfree_rcu_sheaf()handle most of frees (slab already knows how to handleallow_spin= false) and fall back to a simpleirq_workas Ulad suggested.But then we need to make
kfree_rcu_sheaf()try harder (currently it gives up when it fails to get an empty sheaf from the barn).
Yeah, if there's a burst of kfree_rcu_nolock() calls, it might exhaust that.
Quoted 1 line
And I guess the tricky part is making
kfree_rcu_sheaf()work on RT ;)
I hope we're close to that, as discussed on patch 5/8 :)
Paul E. McKenney
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
Quoted 22 lines · Vlastimil Babka (SUSE)
Quoted 21 lines · Harry Yoo (Oracle)
Quoted 11 lines · Harry Yoo (Oracle)
Quoted 4 lines · Uladzislau Rezki
On Thu, Apr 16, 2026 at 06:10:18PM +0900, Harry Yoo (Oracle) wrote:
As discussed or noted earlier, having third argument and check the entire path with "if (allow_spin)" is not optimal and is not good approach. I do not think it this would be a good fit for mainline.I understand that this pattern is pretty uncommon and adds complexity. But there is a reason why we're doing this. As Alexei pointed out earlier, even rare spikes of
irq_workhurt performance [1]; that can be avoided in most cases where the lock is not in use. That's why buddy [2] and slab [3] accepted this pattern despite the added complexity?Perhaps we can avoid complicating existing
kvfree_rcubatching (and still avoid hurting performance).Given that
kfree_rcu_nolock()(at least BPF) doesn't need to free vmalloc objects, we can makekfree_rcu_sheaf()handle most of frees (slab already knows how to handleallow_spin= false) and fall back to a simpleirq_workas Ulad suggested.
Quoted 4 lines
Quoted 2 lines
But then we need to make
kfree_rcu_sheaf()try harder (currently it gives up when it fails to get an empty sheaf from the barn).Yeah, if there's a burst of
kfree_rcu_nolock()calls, it might exhaust that.
The usual trick is to do an irq-work or similar and have the handler do the allocation.
Thanx, Paul
Quoted 4 lines
Quoted 1 line
And I guess the tricky part is making
kfree_rcu_sheaf()work on RT ;)I hope we're close to that, as discussed on patch 5/8 :)
Harry Yoo (Oracle)
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
In reply to Uladzislau RezkiHi Ulad. Apologies for the delayed response.
I meant to reply sooner but sidetracked by other issues.
Your questions are fair, but let me try to clarify the current situation.
And before diving into details, I would like to reiterate that there are potentially two points to discuss here:
Point 1. Can we justify complicating subsystems by passing
allow_spin parameter all over the place?
Point 2. Can we avoid adding this complexity to kvfree_rcu() and
let slab handle it instead? (as mentioned in [4])
On Point 1: IMHO it could be justified, but at the same time I hope we end up avoiding more complexity in the long term by working on Point 2.
This reply focuses only on Point 1 and explains why it could be justified.
Quoted 7 lines · Uladzislau Rezki
Quoted 3 lines · Harry Yoo (Oracle)
On Wed, Apr 22, 2026 at 04:42:28PM +0200, Uladzislau Rezki wrote:
How much performance do we sacrifice compared to letting them go through thekvfree_rcu()fastpath?Freeing an object over RCU from
NMI context is a corner case. It is__not_generic.
First, I want to clarify that kfree_rcu_nolock() is not just for NMI context. It is intended to be used when the context is unknown (because it can be called in an arbitrary code locations).
There are two kinds of problematic situations where BPF programs are attached to:
- 1) a tracepoint or a function that can be invoked in a critical section (w/ a lock held), or
- 2) a function that can be called in an NMI context, which might preempt an arbitrary context holding a lock.
While 1) and 2) are not (I think) dominant use cases, and although most of users can legally call kvfree_rcu(), BPF can't use kvfree_rcu() and must consider the most restrictive contexts.
Quoted 3 lines
We even do not have(now
in mainline) users because we never support it from NMI, just likecall_rcu().
Unfortunately, we've had this use case (of allocating memory for BPF programs) for a long time in the mainline. There are two current approaches to mitigate the limitation:
- 1) Pre-allocate all memory. e.g.) allocate all hash table elements when creating a BPF map, rather than allocating them on demand. This ensures correctness but sacrifices memory.
- 2) Use the BPF-specific memory allocator [1] [2] to allocate memory on demand and avoid preallocation. While this wastes less memory than 1) and also maintains performance, it is re-inventing yet another memory allocator.
Also, the allocator reinvented kfree_rcu batching as well.
Now, we're trying to avoid 1) and 2) as much as possible and use kmalloc_nolock() instead [3].
Quoted 5 lines
If BPF needs
it, then the first question which comes to mind is not about performance. It is how to support this case inkfree_rcu()without adding noticeable complexity or overhead or hacks to the generic path without making it harder to maintain.
Since there will be only few subsystems that needs it, and because they already use it on production systems, I don't see much value in maintaining a simple implementation if that compromises performance (and thus make the transition harder).
Quoted 5 lines
Performance wise you noted, you mean:
a) call latency(this is probably the most important for NMI)?
b) memory footprint?
c) pointer-chasing overhead?
I think it's either
- The performance of kfree_rcu_nolock() itself (a), or
- Not distrubing workloads running on the machine (b and c)
depending on what people use BPF for.
[1] "A BPF-specific memory allocator", Jonathan Corbet, LWN.net, 2022.
Link: https://lwn.net/Articles/899274
[2] [LSF/MM/BPF TOPIC] Reentrant kmalloc usable from any context,
Alexei Starovoitov, the linux-mm mailing list, 2025.
Topic proposal: https://lore.kernel.org/linux-mm/CAADnVQ+=x31Rcjh6-qX1C+d2q0LBRCH=1gPqfKOaMXYx_fkQjw@mail.gmail.com
Slides: https://github.com/4ast/docs/blob/main/Reentrant%20kmalloc.pdf
[3] [GIT PULL] Introduce try_alloc_pages for 6.15, Alexei Starovoitov, the linux-mm mailing list, 2025.
[4] https://lore.kernel.org/linux-mm/ael8R0vOeZ0Xbcoe@hyeyoo
After finishing this email, now I realize that I probably should have elaborated the cover letter ;)
Hope this helps.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Uladzislau Rezki
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
Hello, Harry!
Quoted 4 lines
Hi Ulad. Apologies for the delayed response.
I meant to reply sooner but sidetracked by other issues.
No problem, sometimes i also can lag because of other tasks :)
Quoted 9 lines
Your questions are fair, but let me try to clarify the current situation.
And before diving into details, I would like to reiterate that there are potentially two points to discuss here:
Point 1. Can we justify complicating subsystems by passing
allow_spinparameter all over the place?
Yes, we can. But as i noted i see some drawbacks :)
- all new incoming patches have to respect that new third argument;
- the fallback mechanism which uses irq-work is not optimal in my opinion:
a) We introduce an extra window between queuing a pointer, mark irq-work to be executed and then reenter the kfree_rcu() with no-sync flag and now we need to wait a GP for them. But the GP might be already passed for such pointers. So we potentially need more time to offload. This is rather minus.
b) Since it is for BPF, allow_spin is always false, thus only fallback path is used. Decoupling comes to mind.
c)
Why should we mix those? What it is worth to do, is to prevent mixing "unknown path which is for BPF/others" with generic kfree_rcu().
It is easier to go that way and more cleaner, IMO. We need less code and we fix a specific requirements.
Quoted 4 lines
Point 2. Can we avoid adding this complexity to
kvfree_rcu()and
let slab handle it instead? (as mentioned in [4])
it depends if BPF people want to free a pointer using RCU machinery? Do you know if that an intention?
Quoted 12 lines
On Point 1: IMHO it could be justified, but at the same time I hope we end up avoiding more complexity in the long term by working on Point 2.
This reply focuses only on Point 1 and explains why it could be justified.
Quoted 5 lines · Uladzislau Rezki
Quoted 3 lines · Harry Yoo (Oracle)
On Wed, Apr 22, 2026 at 04:42:28PM +0200, Uladzislau Rezki wrote:
How much performance do we sacrifice compared to letting them go through thekvfree_rcu()fastpath?
Quoted 7 lines
Quoted 2 lines
Freeing an object over RCU from
NMI context is a corner case. It is__not_generic.First, I want to clarify that
kfree_rcu_nolock()is not just for NMI context. It is intended to be used when the context is unknown (because it can be called in an arbitrary code locations).
When we say "unknown" to me it sounds like a worst case, which is NMI :)
Quoted 52 lines
There are two kinds of problematic situations where BPF programs are attached to:
- 1) a tracepoint or a function that can be invoked in a critical section (w/ a lock held), or
- 2) a function that can be called in an NMI context, which might preempt an arbitrary context holding a lock.
While 1) and 2) are not (I think) dominant use cases, and although most of users can legally call
kvfree_rcu(), BPF can't usekvfree_rcu()and must consider the most restrictive contexts.Quoted 3 lines
We even do not have(now
in mainline) users because we never support it from NMI, just likecall_rcu().Unfortunately, we've had this use case (of allocating memory for BPF programs) for a long time in the mainline. There are two current approaches to mitigate the limitation:
- 1) Pre-allocate all memory. e.g.) allocate all hash table elements when creating a BPF map, rather than allocating them on demand. This ensures correctness but sacrifices memory.
- 2) Use the BPF-specific memory allocator [1] [2] to allocate memory on demand and avoid preallocation. While this wastes less memory than 1) and also maintains performance, it is re-inventing yet another memory allocator.
Also, the allocator reinvented
kfree_rcubatching as well.Now, we're trying to avoid 1) and 2) as much as possible and use
kmalloc_nolock()instead [3].Quoted 5 lines
If BPF needs
it, then the first question which comes to mind is not about performance. It is how to support this case inkfree_rcu()without adding noticeable complexity or overhead or hacks to the generic path without making it harder to maintain.Since there will be only few subsystems that needs it, and because they already use it on production systems, I don't see much value in maintaining a simple implementation if that compromises performance (and thus make the transition harder).
Quoted 5 lines
Performance wise you noted, you mean:
a) call latency(this is probably the most important for NMI)?
b) memory footprint?
c) pointer-chasing overhead?
Quoted 7 lines
I think it's either
- The performance of
kfree_rcu_nolock()itself (a), or
- Not distrubing workloads running on the machine (b and c)depending on what people use BPF for.
Are you aware of any specific workloads which we can run? To test and see what we have when it comes to performance metrics? I mean exact uses cases with exact steps who to trigger them?
That would be useful to see on behaviour.
Thank you!
--
Uladzislau Rezki
Harry Yoo
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
[Resending as it's rejected by mailing lists due to my broken email
setup. Apologies for the noise.]
*shows up late again after LSFMM and processing some backlog*
Quoted 14 lines · Uladzislau Rezki
Hello, Harry!
Quoted 4 lines
Hi Ulad. Apologies for the delayed response.
I meant to reply sooner but sidetracked by other issues.No problem, sometimes i also can lag because of other tasks :)
Quoted 6 lines
Your questions are fair, but let me try to clarify the current situation.
And before diving into details, I would like to reiterate that there are potentially two points to discuss here:
Quoted 6 lines
Quoted 3 lines
Point 1. Can we justify complicating subsystems by passing
allow_spinparameter all over the place?Yes, we can. But as i noted i see some drawbacks :)
- all new incoming patches have to respect that new third argument;
That is true :)
Quoted 2 lines
- the fallback mechanism which uses irq-work is not optimal in my opinion:
In most cases it would not fall back because most likely trylock would succeed. If most of the calls do not fall back, a bit of suboptimality on the fallback path is acceptable.
Quoted 8 lines
a) We introduce an extra window between queuing a pointer, mark irq-work to be executed and then reenter the
kfree_rcu()with no-sync flag and now we need to wait a GP for them. But the GP might be already passed for such pointers. So we potentially need more time to offload. This is rather minus.b) Since it is for BPF,
allow_spinis always false, thus only fallback path is used. Decoupling comes to mind.
No, allow_spin == true means spinning on a lock is safe. If allow_spin is false, it would do a trylock instead of spinning, and it is expected to succeed most of the time. As long as trylock succeeds, it uses the same data structures as the existing kvfree_rcu batching without fallback.
Quoted 5 lines
c)Why should we mix those? What it is worth to do, is to prevent mixing "unknown path which is for BPF/others" with generic
kfree_rcu().
Because we want to reuse the existing kvfree_rcu batching infrastructure without reinventing a new feature to do the same thing.
The intent is to avoid the fallback in most cases when allow_spin is false, with fallback being there for correctness.
Quoted 5 lines
Quoted 3 lines
Point 2. Can we avoid adding this complexity to
kvfree_rcu()and
let slab handle it instead? (as mentioned in [4])it depends if BPF people want to free a pointer using RCU machinery? Do you know if that an intention?
They want to free slab objects after RCU grace period. Freeing slab objects without involving RCU is already supported by kfree_nolock(). (There are other use cases as well, as recently posted in [1])
I meant RCU sheaves can handle freeing slab objects after RCU grace period, and kfree_rcu_nolock() users don't need to handle vmalloc pages. So technically we don't have to add this complexity to kvfree_rcu batching and handle it in slab.
But to do that, we shouldn't disable kfree_rcu_sheaf() completely on RT. Apparently Vlastimil has a suggestion to address this, and I'm going to digest his suggestion and explore that aspect.
[1] https://lore.kernel.org/linux-mm/esepccfhqg7m6jo76ns2znj2cnuaepx2xvw5zaygtwohq4psma@563ypprp6rr3
[2] https://lore.kernel.org/linux-mm/[email protected]
Quoted 12 lines
Quoted 12 lines
On Point 1: IMHO it could be justified, but at the same time I hope we end up avoiding more complexity in the long term by working on Point 2.
This reply focuses only on Point 1 and explains why it could be justified.
Quoted 5 lines · Uladzislau Rezki
Quoted 3 lines · Harry Yoo (Oracle)
On Wed, Apr 22, 2026 at 04:42:28PM +0200, Uladzislau Rezki wrote:
How much performance do we sacrifice compared to letting them go through thekvfree_rcu()fastpath?
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Freeing an object over RCU from
NMI context is a corner case. It is__not_generic.First, I want to clarify that
kfree_rcu_nolock()is not just for NMI context. It is intended to be used when the context is unknown (because it can be called in an arbitrary code locations).When we say "unknown" to me it sounds like a worst case, which is NMI :)
If we say "allow_spin = false assumes the most restrictive context, such as NMI context", that is misleading. It sounds like we always fall back, but we don't. Even when the context is unknown, fallback isn't required most of the time.
So I would like to say "the context is unknown", meaning that technically kvfree_rcu could be re-entered in the middle of kvfree_rcu and we need to be able to handle that for correctness (although in most cases there's no re-entry and no fallback).
Quoted 57 lines
Quoted 57 lines
There are two kinds of problematic situations where BPF programs are attached to:
- 1) a tracepoint or a function that can be invoked in a critical section (w/ a lock held), or
- 2) a function that can be called in an NMI context, which might preempt an arbitrary context holding a lock.
While 1) and 2) are not (I think) dominant use cases, and although most of users can legally call
kvfree_rcu(), BPF can't usekvfree_rcu()and must consider the most restrictive contexts.Quoted 3 lines
We even do not have(now
in mainline) users because we never support it from NMI, just likecall_rcu().Unfortunately, we've had this use case (of allocating memory for BPF programs) for a long time in the mainline. There are two current approaches to mitigate the limitation:
- 1) Pre-allocate all memory. e.g.) allocate all hash table elements when creating a BPF map, rather than allocating them on demand. This ensures correctness but sacrifices memory.
- 2) Use the BPF-specific memory allocator [1] [2] to allocate memory on demand and avoid preallocation. While this wastes less memory than 1) and also maintains performance, it is re-inventing yet another memory allocator.
Also, the allocator reinvented
kfree_rcubatching as well.Now, we're trying to avoid 1) and 2) as much as possible and use
kmalloc_nolock()instead [3].Quoted 5 lines
If BPF needs
it, then the first question which comes to mind is not about performance. It is how to support this case inkfree_rcu()without adding noticeable complexity or overhead or hacks to the generic path without making it harder to maintain.Since there will be only few subsystems that needs it, and because they already use it on production systems, I don't see much value in maintaining a simple implementation if that compromises performance (and thus make the transition harder).
Quoted 5 lines
Performance wise you noted, you mean:
a) call latency(this is probably the most important for NMI)?
b) memory footprint?
c) pointer-chasing overhead?I think it's either
- The performance of
kfree_rcu_nolock()itself (a), or
- Not distrubing workloads running on the machine (b and c)
Quoted 7 lines
Quoted 2 lines
depending on what people use BPF for.
Are you aware of any specific workloads which we can run? To test and see what we have when it comes to performance metrics? I mean exact uses cases with exact steps who to trigger them?
That would be useful to see on behaviour.
I'll share once I find what BPF folks are using for performance benchmarks. (Which means I'm not aware at the moment :D)
[+Cc [email protected]]
Thanks!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Uladzislau Rezki
maintainerRe: [PATCH 4/8] mm/slab: introduce kfree_rcu_nolock()
Quoted 20 lines · Harry Yoo
[Resending as it's rejected by mailing lists due to my broken email
setup. Apologies for the noise.]*shows up late again after LSFMM and processing some backlog*
Quoted 14 lines · Uladzislau Rezki
Hello, Harry!
Quoted 4 lines
Hi Ulad. Apologies for the delayed response.
I meant to reply sooner but sidetracked by other issues.No problem, sometimes i also can lag because of other tasks :)
Quoted 6 lines
Your questions are fair, but let me try to clarify the current situation.
And before diving into details, I would like to reiterate that there are potentially two points to discuss here:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Point 1. Can we justify complicating subsystems by passing
allow_spinparameter all over the place?Yes, we can. But as i noted i see some drawbacks :)
- all new incoming patches have to respect that new third argument;
That is true :)
This is i would like to avoid :)
Quoted 24 lines
Quoted 2 lines
- the fallback mechanism which uses irq-work is not optimal in my opinion:
In most cases it would not fall back because most likely trylock would succeed. If most of the calls do not fall back, a bit of suboptimality on the fallback path is acceptable.
Quoted 8 lines
a) We introduce an extra window between queuing a pointer, mark irq-work to be executed and then reenter the
kfree_rcu()with no-sync flag and now we need to wait a GP for them. But the GP might be already passed for such pointers. So we potentially need more time to offload. This is rather minus.b) Since it is for BPF,
allow_spinis always false, thus only fallback path is used. Decoupling comes to mind.No,
allow_spin== true means spinning on a lock is safe. Ifallow_spinis false, it would do a trylock instead of spinning, and it is expected to succeed most of the time. As long as trylock succeeds, it uses the same data structures as the existingkvfree_rcubatching without fallback.Quoted 3 lines
c)
Quoted 9 lines
Quoted 2 lines
Why should we mix those? What it is worth to do, is to prevent mixing "unknown path which is for BPF/others" with generic
kfree_rcu().Because we want to reuse the existing
kvfree_rcubatching infrastructure without reinventing a new feature to do the same thing.The intent is to avoid the fallback in most cases when
allow_spinis false, with fallback being there for correctness.
The problem is that, it is a random behaviour with trylocking, i.e. it is not deterministic. If you apply some noise you end up in kicking two paths anyway.
If the idea is to reuse "existing kvfree_rcu batching" you need to access the array in lock-free way. If you can do that, i would agree with it.
Quoted 114 lines
Quoted 6 lines
Quoted 4 lines
Point 2. Can we avoid adding this complexity to
kvfree_rcu()and
let slab handle it instead? (as mentioned in [4])it depends if BPF people want to free a pointer using RCU machinery? Do you know if that an intention?
They want to free slab objects after RCU grace period. Freeing slab objects without involving RCU is already supported by
kfree_nolock(). (There are other use cases as well, as recently posted in [1])I meant RCU sheaves can handle freeing slab objects after RCU grace period, and
kfree_rcu_nolock()users don't need to handle vmalloc pages. So technically we don't have to add this complexity tokvfree_rcubatching and handle it in slab.But to do that, we shouldn't disable
kfree_rcu_sheaf()completely on RT. Apparently Vlastimil has a suggestion to address this, and I'm going to digest his suggestion and explore that aspect.[1] https://lore.kernel.org/linux-mm/esepccfhqg7m6jo76ns2znj2cnuaepx2xvw5zaygtwohq4psma@563ypprp6rr3
[2] https://lore.kernel.org/linux-mm/[email protected]
Quoted 12 lines
Quoted 12 lines
On Point 1: IMHO it could be justified, but at the same time I hope we end up avoiding more complexity in the long term by working on Point 2.
This reply focuses only on Point 1 and explains why it could be justified.
Quoted 5 lines · Uladzislau Rezki
Quoted 3 lines · Harry Yoo (Oracle)
On Wed, Apr 22, 2026 at 04:42:28PM +0200, Uladzislau Rezki wrote:
How much performance do we sacrifice compared to letting them go through thekvfree_rcu()fastpath?Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Freeing an object over RCU from
NMI context is a corner case. It is__not_generic.First, I want to clarify that
kfree_rcu_nolock()is not just for NMI context. It is intended to be used when the context is unknown (because it can be called in an arbitrary code locations).When we say "unknown" to me it sounds like a worst case, which is NMI :)
If we say "
allow_spin= false assumes the most restrictive context, such as NMI context", that is misleading. It sounds like we always fall back, but we don't. Even when the context is unknown, fallback isn't required most of the time.So I would like to say "the context is unknown", meaning that technically
kvfree_rcucould be re-entered in the middle ofkvfree_rcuand we need to be able to handle that for correctness (although in most cases there's no re-entry and no fallback).Quoted 59 lines
Quoted 59 lines
There are two kinds of problematic situations where BPF programs are attached to:
- 1) a tracepoint or a function that can be invoked in a critical section (w/ a lock held), or
- 2) a function that can be called in an NMI context, which might preempt an arbitrary context holding a lock.
While 1) and 2) are not (I think) dominant use cases, and although most of users can legally call
kvfree_rcu(), BPF can't usekvfree_rcu()and must consider the most restrictive contexts.Quoted 3 lines
We even do not have(now
in mainline) users because we never support it from NMI, just likecall_rcu().Unfortunately, we've had this use case (of allocating memory for BPF programs) for a long time in the mainline. There are two current approaches to mitigate the limitation:
- 1) Pre-allocate all memory. e.g.) allocate all hash table elements when creating a BPF map, rather than allocating them on demand. This ensures correctness but sacrifices memory.
- 2) Use the BPF-specific memory allocator [1] [2] to allocate memory on demand and avoid preallocation. While this wastes less memory than 1) and also maintains performance, it is re-inventing yet another memory allocator.
Also, the allocator reinvented
kfree_rcubatching as well.Now, we're trying to avoid 1) and 2) as much as possible and use
kmalloc_nolock()instead [3].Quoted 5 lines
If BPF needs
it, then the first question which comes to mind is not about performance. It is how to support this case inkfree_rcu()without adding noticeable complexity or overhead or hacks to the generic path without making it harder to maintain.Since there will be only few subsystems that needs it, and because they already use it on production systems, I don't see much value in maintaining a simple implementation if that compromises performance (and thus make the transition harder).
Quoted 5 lines
Performance wise you noted, you mean:
a) call latency(this is probably the most important for NMI)?
b) memory footprint?
c) pointer-chasing overhead?I think it's either
- The performance of
kfree_rcu_nolock()itself (a), or
- Not distrubing workloads running on the machine (b and c)depending on what people use BPF for.
Quoted 9 lines
Quoted 5 lines
Are you aware of any specific workloads which we can run? To test and see what we have when it comes to performance metrics? I mean exact uses cases with exact steps who to trigger them?
That would be useful to see on behaviour.
I'll share once I find what BPF folks are using for performance benchmarks. (Which means I'm not aware at the moment :D)
Please share.
--
Uladzislau Rezki