slab: add struct kmem_cache_args
Subsystems (4)
Christian Brauner
maintainerv4 cover slab: add struct kmem_cache_args
Hey,
This is v4 which allows NULL to be passed in the struct kmem_cache_args argument of kmem_cache_create() and substitutes default parameters in this case.
As discussed last week the various kmem_cache_*() functions should be replaced by a unified function that is based around a struct, with only the basic parameters passed separately.
Vlastimil already said that he would like to keep core parameters out of the struct: name, object size, and flags. I personally don't care much and would not object to moving everything into the struct but that's a matter of taste and I yield that decision power to the maintainer.
In the first version I pointed out that the choice of name is somewhat forced as kmem_cache_create() is taken and the only way to reuse it would be to replace all users in one go. Or to do a global sed/kmem_cache_create()/kmem_cache_create2()/g. And then introduce kmem_cache_setup(). That doesn't strike me as a viable option.
If we really cared about the *_create() suffix then an alternative might be to do a sed/kmem_cache_setup()/kmem_cache_create()/g after every user in the kernel is ported. I honestly don't think that's worth it but I wanted to at least mention it to highlight the fact that this might lead to a naming compromise.
However, I came up with an alternative using _Generic() to create a compatibility layer that will call the correct variant of kmem_cache_create() depending on whether struct kmem_cache_args is passed or not. That compatibility layer can stay in place until we updated all calls to be based on struct kmem_cache_args.
From a cursory grep (and not excluding Documentation mentions) we will have to replace 44 kmem_cache_create_usercopy() calls and about 463 kmem_cache_create() calls which makes for a bit above 500 calls to port to kmem_cache_setup(). That'll probably be good work for people getting into kernel development.
---
Changes in v4:
- Allow NULL to be passed in the struct kmem_cache_args argument of kmem_cache_create() and use default parameters in this case.
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Reworded some commit messages.
- Picked up various RvBs.
- Added two patches to make two functions static inline.
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Remove kmem_cache_setup() and add a compatibility layer built around
_Generic() so that we can keep the kmem_cache_create() name and type
switch on the third argument to either call __kmem_cache_create() or __kmem_cache_create_args().
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Christian Brauner (17):
slab: s/__kmem_cache_create/do_kmem_cache_create/g
slab: add struct kmem_cache_args
slab: port kmem_cache_create() to struct kmem_cache_args
slab: port kmem_cache_create_rcu() to struct kmem_cache_args
slab: port kmem_cache_create_usercopy() to struct kmem_cache_args
slab: pass struct kmem_cache_args to create_cache()
slab: pull kmem_cache_open() into do_kmem_cache_create()
slab: pass struct kmem_cache_args to do_kmem_cache_create()
slab: remove rcu_freeptr_offset from struct kmem_cache
slab: port KMEM_CACHE() to struct kmem_cache_args
slab: port KMEM_CACHE_USERCOPY() to struct kmem_cache_args
slab: create kmem_cache_create() compatibility layer
file: port to struct kmem_cache_args
slab: remove kmem_cache_create_rcu()
slab: make kmem_cache_create_usercopy() static inline
slab: make __kmem_cache_create() static inline
io_uring: port to struct kmem_cache_args
fs/file_table.c | 11 ++- include/linux/slab.h | 130 +++++++++++++++++++++++++++------ io_uring/io_uring.c | 14 ++-- mm/slab.h | 6 +- mm/slab_common.c | 197 +++++++++++---------------------------------------- mm/slub.c | 162 +++++++++++++++++++++--------------------- 6 files changed, 250 insertions(+), 270 deletions(-)
---
base-commit: 6e016babce7c845ed015da25c7a097fa3482d95a
change-id: 20240902-work-kmem_cache_args-e9760972c7d4
Christian Brauner
maintainerPATCH v4 1/17 slab: s/__kmem_cache_create/do_kmem_cache_create/g
Free up reusing the double-underscore variant for follow-up patches.
--- mm/slab.h | 2 +- mm/slab_common.c | 4 ++-- mm/slub.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
mm/slab.h +1 -1
@@ -424,7 +424,7 @@ kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller)gfp_t kmalloc_fix_flags (gfp_t flags);/* Functions provided by the slab allocators */ -int __kmem_cache_create (struct kmem_cache *,slab_flags_t flags); +int do_kmem_cache_create (struct kmem_cache *,slab_flags_t flags);void __init kmem_cache_init (void ); extern void create_boot_cache(struct kmem_cache *, const char *name,
mm/slab_common.c +2 -2
@@ -234,7 +234,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize;#endif - err =__kmem_cache_create (s, flags); + err =do_kmem_cache_create (s, flags);if (err) goto out_free_cache;
@@ -778,7 +778,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->usersize = usersize;#endif - err =__kmem_cache_create (s, flags); + err =do_kmem_cache_create (s, flags);if (err) panic("Creation of kmalloc slab %s size=%u failed. Reason %d\n",
mm/slub.c +1 -1
@@ -5902,7 +5902,7 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align,return s; } -int __kmem_cache_create (struct kmem_cache *s,slab_flags_t flags) +int do_kmem_cache_create (struct kmem_cache *s,slab_flags_t flags) {int err;
Signature · 1 line
2.45.2
Re: [PATCH v4 01/17] slab: s/__kmem_cache_create/do_kmem_cache_create/g
Quoted 7 lines · Christian Brauner
Free up reusing the double-underscore variant for follow-up patches.
Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Thanks
Christian Brauner
maintainerPATCH v4 2/17 slab: add struct kmem_cache_args
In reply to Christian BraunerCurrently we have multiple kmem_cache_create*() variants that take up to seven separate parameters with one of the functions having to grow an eigth parameter in the future to handle both usercopy and a custom freelist pointer.
Add a struct kmem_cache_args structure and move less common parameters into it. Core parameters such as name, object size, and flags continue to be passed separately.
Add a new function __kmem_cache_create_args() that takes a struct kmem_cache_args pointer and port do_kmem_cache_create_usercopy() over to it.
In follow-up patches we will port the other kmem_cache_create*() variants over to it as well.
--- include/linux/slab.h | 21 ++++++++++++++++ mm/slab_common.c | 67 +++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 72 insertions(+), 16 deletions(-)
include/linux/slab.h +21 -0
@@ -240,6 +240,27 @@ struct mem_cgroup;*/ bool slab_is_available (void ); +/** +* @align: The required alignment for the objects. +* @useroffset: Usercopy region offset +* @usersize: Usercopy region size +* @freeptr_offset: Custom offset for the free pointer in RCU caches +* @use_freeptr_offset: Whether a @freeptr_offset is used +* @ctor: A constructor for the objects. +*/ +struct kmem_cache_args { +unsigned int align; +unsigned int useroffset; +unsigned int usersize; +unsigned int freeptr_offset; +bool use_freeptr_offset; +void (*ctor)(void *); +}; + +struct kmem_cache *__kmem_cache_create_args (const char *name, +unsigned int object_size, +struct kmem_cache_args *args, +slab_flags_t flags);struct kmem_cache *kmem_cache_create (const char *name,unsigned int size,unsigned int align,slab_flags_t flags,void (*ctor)(void *));
mm/slab_common.c +51 -16
99 lines not on this page · raw message
Signature · 1 line
2.45.2
Quoted 13 lines · Christian Brauner
Currently we have multiple kmem_cache_create*() variants that take up to seven separate parameters with one of the functions having to grow an eigth parameter in the future to handle both usercopy and a custom freelist pointer.
Add a struct
kmem_cache_argsstructure and move less common parameters into it. Core parameters such as name, object size, and flags continue to be passed separately.Add a new function
__kmem_cache_create_args()that takes a structkmem_cache_argspointer and portdo_kmem_cache_create_usercopy()over to it.
Quoted 8 lines
In follow-up patches we will port the other kmem_cache_create*() variants over to it as well.
Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 3/17 slab: port kmem_cache_create() to struct kmem_cache_args
In reply to Christian BraunerPort kmem_cache_create() to struct kmem_cache_args.
--- mm/slab_common.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/slab_common.c +6 -2
@@ -439,8 +439,12 @@ struct kmem_cache * kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)) { - return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, - 0, 0, ctor); + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + }; + +return __kmem_cache_create_args (name, size, &kmem_args, flags); }EXPORT_SYMBOL (kmem_cache_create);
Signature · 1 line
2.45.2
Re: [PATCH v4 03/17] slab: port kmem_cache_create() to struct kmem_cache_args
Quoted 7 lines · Christian Brauner
Port
kmem_cache_create()to structkmem_cache_args.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 4/17 slab: port kmem_cache_create_rcu() to struct kmem_cache_args
In reply to Christian BraunerPort kmem_cache_create_rcu() to struct kmem_cache_args.
--- mm/slab_common.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
mm/slab_common.c +7 -3
@@ -481,9 +481,13 @@ struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, unsigned int freeptr_offset, slab_flags_t flags) { -return do_kmem_cache_create_usercopy (name, size, freeptr_offset,0 , - flags |SLAB_TYPESAFE_BY_RCU ,0 ,0 , -NULL ); +struct kmem_cache_args kmem_args = { + .freeptr_offset = freeptr_offset, + .use_freeptr_offset = true, + }; + +return __kmem_cache_create_args (name, size, &kmem_args, + flags |SLAB_TYPESAFE_BY_RCU ); }EXPORT_SYMBOL (kmem_cache_create_rcu);
Signature · 1 line
2.45.2
Re: [PATCH v4 04/17] slab: port kmem_cache_create_rcu() to struct kmem_cache_args
Quoted 7 lines · Christian Brauner
Port
kmem_cache_create_rcu()to structkmem_cache_args.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 5/17 slab: port kmem_cache_create_usercopy() to struct kmem_cache_args
In reply to Christian BraunerPort kmem_cache_create_usercopy() to struct kmem_cache_args and remove the now unused do_kmem_cache_create_usercopy() helper.
--- mm/slab_common.c | 30 ++++++++---------------------- 1 file changed, 8 insertions(+), 22 deletions(-)
mm/slab_common.c +8 -22
@@ -351,26 +351,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, }EXPORT_SYMBOL (__kmem_cache_create_args); -static struct kmem_cache * -do_kmem_cache_create_usercopy (const char *name, -unsigned int size,unsigned int freeptr_offset, -unsigned int align,slab_flags_t flags, -unsigned int useroffset,unsigned int usersize, -void (*ctor)(void *)) -{ -struct kmem_cache_args kmem_args = { - .align = align, - .use_freeptr_offset = freeptr_offset !=UINT_MAX , - .freeptr_offset = freeptr_offset, - .useroffset = useroffset, - .usersize = usersize, - .ctor = ctor, - }; - -return __kmem_cache_create_args (name, size, &kmem_args, flags); -} - -/** * kmem_cache_create_usercopy - Create a cache with a region suitable * for copying to userspace
@@ -405,8 +385,14 @@ kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)) { - return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, - useroffset, usersize, ctor); + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + .useroffset = useroffset, + .usersize = usersize, + }; + +return __kmem_cache_create_args (name, size, &kmem_args, flags); }EXPORT_SYMBOL (kmem_cache_create_usercopy);
Signature · 1 line
2.45.2
Re: [PATCH v4 05/17] slab: port kmem_cache_create_usercopy() to struct kmem_cache_args
Quoted 8 lines · Christian Brauner
Port
kmem_cache_create_usercopy()to structkmem_cache_argsand remove the now unuseddo_kmem_cache_create_usercopy()helper.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 6/17 slab: pass struct kmem_cache_args to create_cache()
In reply to Christian BraunerPass struct kmem_cache_args to create_cache() so that we can later simplify further helpers.
--- mm/slab_common.c | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-)
mm/slab_common.c +19 -20
@@ -202,22 +202,22 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, }static struct kmem_cache *create_cache (const char *name, -unsigned int object_size,unsigned int freeptr_offset, -unsigned int align,slab_flags_t flags, -unsigned int useroffset,unsigned int usersize, -void (*ctor)(void *)) +unsigned int object_size, +struct kmem_cache_args *args, +slab_flags_t flags) {struct kmem_cache *s;int err; -if (WARN_ON (useroffset + usersize > object_size)) - useroffset = usersize =0 ; +if (WARN_ON (args->useroffset + args->usersize > object_size)) + args->useroffset = args->usersize =0 ;/* If a custom freelist pointer is requested make sure it's sane. */ err = -EINVAL ; -if (freeptr_offset !=UINT_MAX && - (freeptr_offset >= object_size || !(flags &SLAB_TYPESAFE_BY_RCU ) || - !IS_ALIGNED (freeptr_offset,sizeof (freeptr_t)))) +if (args->use_freeptr_offset && + (args->freeptr_offset >= object_size || + !(flags &SLAB_TYPESAFE_BY_RCU ) || + !IS_ALIGNED (args->freeptr_offset,sizeof (freeptr_t)))) goto out; err = -ENOMEM ;
@@ -227,12 +227,15 @@ static struct kmem_cache *create_cache(const char *name, s->name = name; s->size = s->object_size = object_size; - s->rcu_freeptr_offset = freeptr_offset; - s->align = align; - s->ctor = ctor; +if (args->use_freeptr_offset) + s->rcu_freeptr_offset = args->freeptr_offset; +else + s->rcu_freeptr_offset =UINT_MAX ; + s->align = args->align; + s->ctor = args->ctor;#ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = useroffset; - s->usersize = usersize; + s->useroffset = args->useroffset; + s->usersize = args->usersize;#endif err =do_kmem_cache_create (s, flags);if (err)
@@ -265,7 +268,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, slab_flags_t flags) {struct kmem_cache *s =NULL ; -unsigned int freeptr_offset =UINT_MAX ;const char *cache_name;int err;
@@ -323,11 +325,8 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, goto out_unlock; } -if (args->use_freeptr_offset) - freeptr_offset = args->freeptr_offset; - s =create_cache (cache_name, object_size, freeptr_offset, -calculate_alignment (flags, args->align, object_size), - flags, args->useroffset, args->usersize, args->ctor); + args->align =calculate_alignment (flags, args->align, object_size); + s =create_cache (cache_name, object_size, args, flags);if (IS_ERR (s)) { err =PTR_ERR (s);kfree_const (cache_name);
Signature · 1 line
2.45.2
Re: [PATCH v4 06/17] slab: pass struct kmem_cache_args to create_cache()
Quoted 8 lines · Christian Brauner
Pass struct
kmem_cache_argstocreate_cache()so that we can later simplify further helpers.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 7/17 slab: pull kmem_cache_open() into do_kmem_cache_create()
In reply to Christian Braunerdo_kmem_cache_create() is the only caller and we're going to pass down struct kmem_cache_args in a follow-up patch.
--- mm/slub.c | 132 +++++++++++++++++++++++++++++--------------------------------- 1 file changed, 62 insertions(+), 70 deletions(-)
mm/slub.c +62 -70
@@ -5290,65 +5290,6 @@ static int calculate_sizes(struct kmem_cache *s)return !!oo_objects (s->oo); } -static int kmem_cache_open (struct kmem_cache *s,slab_flags_t flags) -{ - s->flags =kmem_cache_flags (flags, s->name); -#ifdef CONFIG_SLAB_FREELIST_HARDENED - s->random =get_random_long (); -#endif - -if (!calculate_sizes (s)) - goto error; -if (disable_higher_order_debug) { -/* -* Disable debugging flags that store metadata if the min slab -* order increased. -*/ -if (get_order (s->size) >get_order (s->object_size)) { - s->flags &= ~DEBUG_METADATA_FLAGS ; - s->offset =0 ; -if (!calculate_sizes (s)) - goto error; - } - } - -#ifdef system_has_freelist_aba -if (system_has_freelist_aba () && !(s->flags &SLAB_NO_CMPXCHG )) { -/* Enable fast mode */ - s->flags |= __CMPXCHG_DOUBLE; - } -#endif - -/* -* The larger the object size is, the more slabs we want on the partial -* list to avoid pounding the page allocator excessively. -*/ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); - -set_cpu_partial (s); - -#ifdef CONFIG_NUMA - s->remote_node_defrag_ratio =1000 ; -#endif - -/* Initialize the pre-computed randomized freelist if slab is up */ -if (slab_state >=UP ) { -if (init_cache_random_seq (s)) - goto error; - } - -if (!init_kmem_cache_nodes (s)) - goto error; - -if (alloc_kmem_cache_cpus (s)) -return 0 ; - -error: -__kmem_cache_release (s); -return -EINVAL ; -} -static void list_slab_objects (struct kmem_cache *s,struct slab *slab,const char *text) {
@@ -5904,26 +5845,77 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align,int do_kmem_cache_create (struct kmem_cache *s,slab_flags_t flags) { -int err; +int err = -EINVAL ; - err =kmem_cache_open (s, flags); -if (err) -return err; + s->flags =kmem_cache_flags (flags, s->name); +#ifdef CONFIG_SLAB_FREELIST_HARDENED + s->random =get_random_long (); +#endif + +if (!calculate_sizes (s)) + goto out; +if (disable_higher_order_debug) { +/* +* Disable debugging flags that store metadata if the min slab +* order increased. +*/ +if (get_order (s->size) >get_order (s->object_size)) { + s->flags &= ~DEBUG_METADATA_FLAGS ; + s->offset =0 ; +if (!calculate_sizes (s)) + goto out; + } + } + +#ifdef system_has_freelist_aba +if (system_has_freelist_aba () && !(s->flags &SLAB_NO_CMPXCHG )) { +/* Enable fast mode */ + s->flags |= __CMPXCHG_DOUBLE; + } +#endif + +/* +* The larger the object size is, the more slabs we want on the partial +* list to avoid pounding the page allocator excessively. +*/ + s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); + s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); + +set_cpu_partial (s); + +#ifdef CONFIG_NUMA + s->remote_node_defrag_ratio =1000 ; +#endif + +/* Initialize the pre-computed randomized freelist if slab is up */ +if (slab_state >=UP ) { +if (init_cache_random_seq (s)) + goto out; + } + +if (!init_kmem_cache_nodes (s)) + goto out; + +if (!alloc_kmem_cache_cpus (s)) + goto out;/* Mutex is not taken during early boot */ -if (slab_state <=UP ) -return 0 ; +if (slab_state <=UP ) { + err =0 ; + goto out; + } err =sysfs_slab_add (s); -if (err) { -__kmem_cache_release (s); -return err; - } +if (err) + goto out;if (s->flags &SLAB_STORE_USER )debugfs_slab_add (s); -return 0 ; +out: +if (err) +__kmem_cache_release (s); +return err; }#ifdef SLAB_SUPPORTS_SYSFS
Signature · 1 line
2.45.2
Re: [PATCH v4 07/17] slab: pull kmem_cache_open() into do_kmem_cache_create()
Quoted 8 lines · Christian Brauner
do_kmem_cache_create()is the only caller and we're going to pass down structkmem_cache_argsin a follow-up patch.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 8/17 slab: pass struct kmem_cache_args to do_kmem_cache_create()
In reply to Christian Braunerand initialize most things in do_kmem_cache_create(). In a follow-up patch we'll remove rcu_freeptr_offset from struct kmem_cache.
--- mm/slab.h | 4 +++- mm/slab_common.c | 27 ++++++--------------------- mm/slub.c | 17 ++++++++++++++++- 3 files changed, 25 insertions(+), 23 deletions(-)
mm/slab.h +3 -1
@@ -424,7 +424,9 @@ kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller)gfp_t kmalloc_fix_flags (gfp_t flags);/* Functions provided by the slab allocators */ -int do_kmem_cache_create (struct kmem_cache *,slab_flags_t flags); +int do_kmem_cache_create (struct kmem_cache *s,const char *name, +unsigned int size,struct kmem_cache_args *args, +slab_flags_t flags);void __init kmem_cache_init (void ); extern void create_boot_cache(struct kmem_cache *, const char *name,
mm/slab_common.c +6 -21
@@ -224,20 +224,7 @@ static struct kmem_cache *create_cache(const char *name, s =kmem_cache_zalloc (kmem_cache,GFP_KERNEL );if (!s) goto out; - - s->name = name; - s->size = s->object_size = object_size; -if (args->use_freeptr_offset) - s->rcu_freeptr_offset = args->freeptr_offset; -else - s->rcu_freeptr_offset =UINT_MAX ; - s->align = args->align; - s->ctor = args->ctor; -#ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = args->useroffset; - s->usersize = args->usersize; -#endif - err =do_kmem_cache_create (s, flags); + err =do_kmem_cache_create (s, name, object_size, args, flags);if (err) goto out_free_cache;
@@ -788,9 +775,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, {int err;unsigned int align =ARCH_KMALLOC_MINALIGN ; - - s->name = name; - s->size = s->object_size = size; +struct kmem_cache_args kmem_args = {};/* * kmalloc caches guarantee alignment of at least the largest
@@ -799,14 +784,14 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,*/ if (flags &SLAB_KMALLOC ) align =max (align,1U << (ffs (size) -1 )); - s->align =calculate_alignment (flags, align, size); + kmem_args.align =calculate_alignment (flags, align, size);#ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = useroffset; - s->usersize = usersize; + kmem_args.useroffset = useroffset; + kmem_args.usersize = usersize;#endif - err =do_kmem_cache_create (s, flags); + err =do_kmem_cache_create (s, name, size, &kmem_args, flags);if (err) panic("Creation of kmalloc slab %s size=%u failed. Reason %d\n",
mm/slub.c +16 -1
@@ -5843,14 +5843,29 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align,return s; } -int do_kmem_cache_create (struct kmem_cache *s,slab_flags_t flags) +int do_kmem_cache_create (struct kmem_cache *s,const char *name, +unsigned int size,struct kmem_cache_args *args, +slab_flags_t flags) {int err = -EINVAL ; + s->name = name; + s->size = s->object_size = size; + s->flags =kmem_cache_flags (flags, s->name);#ifdef CONFIG_SLAB_FREELIST_HARDENED s->random =get_random_long ();#endif +if (args->use_freeptr_offset) + s->rcu_freeptr_offset = args->freeptr_offset; +else + s->rcu_freeptr_offset =UINT_MAX ; + s->align = args->align; + s->ctor = args->ctor; +#ifdef CONFIG_HARDENED_USERCOPY + s->useroffset = args->useroffset; + s->usersize = args->usersize; +#endif if (!calculate_sizes (s)) goto out;
Signature · 1 line
2.45.2
Re: [PATCH v4 08/17] slab: pass struct kmem_cache_args to do_kmem_cache_create()
Quoted 8 lines · Christian Brauner
and initialize most things in
do_kmem_cache_create(). In a follow-up patch we'll removercu_freeptr_offsetfrom structkmem_cache.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 9/17 slab: remove rcu_freeptr_offset from struct kmem_cache
In reply to Christian BraunerPass down struct kmem_cache_args to calculate_sizes() so we can use args->{use}_freeptr_offset directly. This allows us to remove ->rcu_freeptr_offset from struct kmem_cache.
--- mm/slab.h | 2 -- mm/slub.c | 25 +++++++------------------ 2 files changed, 7 insertions(+), 20 deletions(-)
mm/slab.h +0 -2
@@ -261,8 +261,6 @@ struct kmem_cache {unsigned int object_size;/* Object size without metadata */ struct reciprocal_value reciprocal_size;unsigned int offset;/* Free pointer offset */ -/* Specific free pointer requested (if not UINT_MAX) */ -unsigned int rcu_freeptr_offset;#ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial;
mm/slub.c +7 -18
@@ -3916,8 +3916,7 @@ static void *__slab_alloc_node(struct kmem_cache *s,* If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer. * -* Note that we also wipe custom freelist pointers specified via -* s->rcu_freeptr_offset. +* Note that we also wipe custom freelist pointers. */ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s,void *obj)
@@ -5141,17 +5140,11 @@ static void set_cpu_partial(struct kmem_cache *s)#endif } -/* Was a valid freeptr offset requested? */ -static inline bool has_freeptr_offset (const struct kmem_cache *s) -{ -return s->rcu_freeptr_offset !=UINT_MAX ; -} -/* * calculate_sizes() determines the order and the distribution of data within * a slab object. */ -static int calculate_sizes (struct kmem_cache *s) +static int calculate_sizes (struct kmem_cache_args *args,struct kmem_cache *s) {slab_flags_t flags = s->flags;unsigned int size = s->object_size;
@@ -5192,7 +5185,7 @@ static int calculate_sizes(struct kmem_cache *s)*/ s->inuse = size; -if (((flags &SLAB_TYPESAFE_BY_RCU ) && !has_freeptr_offset (s)) || +if (((flags &SLAB_TYPESAFE_BY_RCU ) && !args->use_freeptr_offset) || (flags &SLAB_POISON ) || s->ctor || ((flags &SLAB_RED_ZONE ) && (s->object_size <sizeof (void *) ||slub_debug_orig_size (s)))) {
@@ -5214,8 +5207,8 @@ static int calculate_sizes(struct kmem_cache *s)*/ s->offset = size; size +=sizeof (void *); - }else if ((flags &SLAB_TYPESAFE_BY_RCU ) &&has_freeptr_offset (s)) { - s->offset = s->rcu_freeptr_offset; + }else if ((flags &SLAB_TYPESAFE_BY_RCU ) && args->use_freeptr_offset) { + s->offset = args->freeptr_offset; }else {/* * Store freelist pointer near middle of object to keep
@@ -5856,10 +5849,6 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name,#ifdef CONFIG_SLAB_FREELIST_HARDENED s->random =get_random_long ();#endif -if (args->use_freeptr_offset) - s->rcu_freeptr_offset = args->freeptr_offset; -else - s->rcu_freeptr_offset =UINT_MAX ; s->align = args->align; s->ctor = args->ctor;#ifdef CONFIG_HARDENED_USERCOPY
@@ -5867,7 +5856,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, s->usersize = args->usersize;#endif -if (!calculate_sizes (s)) +if (!calculate_sizes (args, s)) goto out;if (disable_higher_order_debug) {/*
@@ -5877,7 +5866,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name,if (get_order (s->size) >get_order (s->object_size)) { s->flags &= ~DEBUG_METADATA_FLAGS ; s->offset =0 ; -if (!calculate_sizes (s)) +if (!calculate_sizes (args, s)) goto out; } }
Signature · 1 line
2.45.2
Re: [PATCH v4 09/17] slab: remove rcu_freeptr_offset from struct kmem_cache
Quoted 9 lines · Christian Brauner
Pass down struct
kmem_cache_argstocalculate_sizes()so we can use args->{use}_freeptr_offset directly. This allows us to remove ->rcu_freeptr_offset from structkmem_cache.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 10/17 slab: port KMEM_CACHE() to struct kmem_cache_args
In reply to Christian BraunerMake KMEM_CACHE() use struct kmem_cache_args.
--- include/linux/slab.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
include/linux/slab.h +5 -3
@@ -283,9 +283,11 @@ int kmem_cache_shrink(struct kmem_cache *s);* f.e. add ____cacheline_aligned_in_smp to the struct declaration * then the objects will be properly aligned in SMP configurations. */ -#define KMEM_CACHE (__struct, __flags) \ - kmem_cache_create(#__struct, sizeof(struct __struct), \ - __alignof__(struct __struct), (__flags), NULL) +#define KMEM_CACHE (__struct, __flags) \ + __kmem_cache_create_args(#__struct, sizeof(struct __struct), \ + &(struct kmem_cache_args) { \ + .align = __alignof__(struct __struct), \ + }, (__flags))/* * To whitelist a single field for copying to/from usercopy, use this
Signature · 1 line
2.45.2
Re: [PATCH v4 10/17] slab: port KMEM_CACHE() to struct kmem_cache_args
Quoted 7 lines · Christian Brauner
Make
KMEM_CACHE()use structkmem_cache_args.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 11/17 slab: port KMEM_CACHE_USERCOPY() to struct kmem_cache_args
In reply to Christian BraunerMake KMEM_CACHE_USERCOPY() use struct kmem_cache_args.
--- include/linux/slab.h | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
include/linux/slab.h +7 -6
@@ -293,12 +293,13 @@ int kmem_cache_shrink(struct kmem_cache *s);* To whitelist a single field for copying to/from usercopy, use this * macro instead for KMEM_CACHE() above. */ -#define KMEM_CACHE_USERCOPY (__struct, __flags, __field) \ - kmem_cache_create_usercopy(#__struct, \ - sizeof(struct __struct), \ - __alignof__(struct __struct), (__flags), \ - offsetof(struct __struct, __field), \ - sizeof_field(struct __struct, __field), NULL) +#define KMEM_CACHE_USERCOPY (__struct, __flags, __field) \ + __kmem_cache_create_args(#__struct, sizeof(struct __struct), \ + &(struct kmem_cache_args) { \ + .align = __alignof__(struct __struct), \ + .useroffset = offsetof(struct __struct, __field), \ + .usersize = sizeof_field(struct __struct, __field), \ + }, (__flags))/* * Common kmalloc functions provided by all allocators
Signature · 1 line
2.45.2
Re: [PATCH v4 11/17] slab: port KMEM_CACHE_USERCOPY() to struct kmem_cache_args
Quoted 7 lines · Christian Brauner
Make
KMEM_CACHE_USERCOPY()use structkmem_cache_args.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 12/17 slab: create kmem_cache_create() compatibility layer
In reply to Christian BraunerUse _Generic() to create a compatibility layer that type switches on the third argument to either call __kmem_cache_create() or __kmem_cache_create_args(). If NULL is passed for the struct kmem_cache_args argument use default args making porting for callers that don't care about additional arguments easy.
--- include/linux/slab.h | 29 ++++++++++++++++++++++++++--- mm/slab_common.c | 10 +++++----- 2 files changed, 31 insertions(+), 8 deletions(-)
include/linux/slab.h +26 -3
@@ -261,9 +261,10 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); -struct kmem_cache *kmem_cache_create (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -void (*ctor)(void *)); + +struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +void (*ctor)(void *)); struct kmem_cache *kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int align, slab_flags_t flags,
@@ -272,6 +273,28 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name,struct kmem_cache *kmem_cache_create_rcu (const char *name,unsigned int size,unsigned int freeptr_offset,slab_flags_t flags); + +/* If NULL is passed for @args, use this variant with default arguments. */ +static inline struct kmem_cache * +__kmem_cache_default_args (const char *name,unsigned int size, +struct kmem_cache_args *args, +slab_flags_t flags) +{ +struct kmem_cache_args kmem_default_args = {}; + +/* Make sure we don't get passed garbage. */ +if (WARN_ON_ONCE (args)) +return NULL ; + +return __kmem_cache_create_args (name, size, &kmem_default_args, flags); +} + +#define kmem_cache_create (__name, __object_size, __args, ...) \ + _Generic((__args), \ + struct kmem_cache_args *: __kmem_cache_create_args, \ + void *: __kmem_cache_default_args, \ + default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__) +void kmem_cache_destroy (struct kmem_cache *s);int kmem_cache_shrink (struct kmem_cache *s);
mm/slab_common.c +5 -5
@@ -383,7 +383,7 @@ kmem_cache_create_usercopy(const char *name, unsigned int size,EXPORT_SYMBOL (kmem_cache_create_usercopy);/** -* kmem_cache_create - Create a cache. +* __kmem_cache_create - Create a cache. * @name: A string which is used in /proc/slabinfo to identify this cache. * @size: The size of objects to be created in this cache. * @align: The required alignment for the objects.
@@ -407,9 +407,9 @@ EXPORT_SYMBOL(kmem_cache_create_usercopy);* * Return: a pointer to the cache on success, NULL on failure. */ -struct kmem_cache * -kmem_cache_create (const char *name,unsigned int size,unsigned int align, -slab_flags_t flags,void (*ctor)(void *)) +struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +void (*ctor)(void *)) {struct kmem_cache_args kmem_args = { .align = align,
@@ -418,7 +418,7 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align,return __kmem_cache_create_args (name, size, &kmem_args, flags); } -EXPORT_SYMBOL (kmem_cache_create); +EXPORT_SYMBOL (__kmem_cache_create);/** * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache.
Signature · 1 line
2.45.2
Re: [PATCH v4 12/17] slab: create kmem_cache_create() compatibility layer
Quoted 9 lines · Christian Brauner
Use
_Generic()to create a compatibility layer that type switches on the third argument to either call__kmem_cache_create()or__kmem_cache_create_args(). If NULL is passed for the structkmem_cache_argsargument use default args making porting for callers that don't care about additional arguments easy.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Quoted 91 lines
--- include/linux/slab.h | 29 ++++++++++++++++++++++++++--- mm/slab_common.c | 10 +++++----- 2 files changed, 31 insertions(+), 8 deletions(-)include/linux/slab.h +26
-3@@ -261,9 +261,10 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); -struct kmem_cache *kmem_cache_create (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -void (*ctor)(void *)); + +struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +void (*ctor)(void *)); struct kmem_cache *kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int align, slab_flags_t flags,@@ -272,6 +273,28 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name,struct kmem_cache *kmem_cache_create_rcu (const char *name,unsigned int size,unsigned int freeptr_offset,slab_flags_t flags); + +/* If NULL is passed for @args, use this variant with default arguments. */ +static inline struct kmem_cache * +__kmem_cache_default_args (const char *name,unsigned int size, +struct kmem_cache_args *args, +slab_flags_t flags) +{ +struct kmem_cache_args kmem_default_args = {}; + +/* Make sure we don't get passed garbage. */ +if (WARN_ON_ONCE (args)) +return NULL ; + +return __kmem_cache_create_args (name, size, &kmem_default_args, flags); +} + +#define kmem_cache_create (__name, __object_size, __args, ...) \ + _Generic((__args), \ + struct kmem_cache_args *: __kmem_cache_create_args, \ + void *: __kmem_cache_default_args, \ + default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__) +void kmem_cache_destroy (struct kmem_cache *s);int kmem_cache_shrink (struct kmem_cache *s);mm/slab_common.c +5
-5@@ -383,7 +383,7 @@ kmem_cache_create_usercopy(const char *name, unsigned int size,EXPORT_SYMBOL (kmem_cache_create_usercopy);/** -* kmem_cache_create - Create a cache. +* __kmem_cache_create - Create a cache. * @name: A string which is used in /proc/slabinfo to identify this cache. * @size: The size of objects to be created in this cache. * @align: The required alignment for the objects. @@ -407,9 +407,9 @@ EXPORT_SYMBOL(kmem_cache_create_usercopy);* * Return: a pointer to the cache on success, NULL on failure. */ -struct kmem_cache * -kmem_cache_create (const char *name,unsigned int size,unsigned int align, -slab_flags_t flags,void (*ctor)(void *)) +struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +void (*ctor)(void *)) {struct kmem_cache_args kmem_args = { .align = align,@@ -418,7 +418,7 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align,return __kmem_cache_create_args (name, size, &kmem_args, flags); } -EXPORT_SYMBOL (kmem_cache_create); +EXPORT_SYMBOL (__kmem_cache_create);/** * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache. Signature · 2 lines
-- 2.45.2
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v4 12/17] slab: create kmem_cache_create() compatibility layer
In reply to Christian BraunerQuoted 38 lines · Christian Brauner
Use
_Generic()to create a compatibility layer that type switches on the third argument to either call__kmem_cache_create()or__kmem_cache_create_args(). If NULL is passed for the structkmem_cache_argsargument use default args making porting for callers that don't care about additional arguments easy.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Signed-off-by Christian Brauner <[email protected]>--- include/linux/slab.h | 29 ++++++++++++++++++++++++++--- mm/slab_common.c | 10 +++++----- 2 files changed, 31 insertions(+), 8 deletions(-)include/linux/slab.h +6
-3@@ -261,9 +261,10 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); -struct kmem_cache *kmem_cache_create (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -void (*ctor)(void *)); + +struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +void (*ctor)(void *)); struct kmem_cache *kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int align, slab_flags_t flags,@@ -272,6 +273,28 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name,struct kmem_cache *kmem_cache_create_rcu (const char *name,unsigned int size,unsigned int freeptr_offset,slab_flags_t flags); + +/* If NULL is passed for @args, use this variant with default arguments. */
include/linux/slab.h struct kmem_cache *kmem_cache_create_usercopy(const char *name, · line 287
Quoted 10 lines
include/linux/slab.h +10
-0+static inline struct kmem_cache * +__kmem_cache_default_args (const char *name,unsigned int size, +struct kmem_cache_args *args, +slab_flags_t flags) +{ +struct kmem_cache_args kmem_default_args = {}; + +/* Make sure we don't get passed garbage. */ +if (WARN_ON_ONCE (args)) +return NULL ;
I'm changing to return ERR_PTR(-EINVAL); locally
although it's not too important
Quoted 49 lines
include/linux/slab.h +10
-0+ +return __kmem_cache_create_args (name, size, &kmem_default_args, flags); +} + +#define kmem_cache_create (__name, __object_size, __args, ...) \ + _Generic((__args), \ + struct kmem_cache_args *: __kmem_cache_create_args, \ + void *: __kmem_cache_default_args, \ + default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__) +void kmem_cache_destroy (struct kmem_cache *s);int kmem_cache_shrink (struct kmem_cache *s);mm/slab_common.c +5
-5@@ -383,7 +383,7 @@ kmem_cache_create_usercopy(const char *name, unsigned int size,EXPORT_SYMBOL (kmem_cache_create_usercopy);/** -* kmem_cache_create - Create a cache. +* __kmem_cache_create - Create a cache. * @name: A string which is used in /proc/slabinfo to identify this cache. * @size: The size of objects to be created in this cache. * @align: The required alignment for the objects. @@ -407,9 +407,9 @@ EXPORT_SYMBOL(kmem_cache_create_usercopy);* * Return: a pointer to the cache on success, NULL on failure. */ -struct kmem_cache * -kmem_cache_create (const char *name,unsigned int size,unsigned int align, -slab_flags_t flags,void (*ctor)(void *)) +struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +void (*ctor)(void *)) {struct kmem_cache_args kmem_args = { .align = align,@@ -418,7 +418,7 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align,return __kmem_cache_create_args (name, size, &kmem_args, flags); } -EXPORT_SYMBOL (kmem_cache_create); +EXPORT_SYMBOL (__kmem_cache_create);/** * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache.
Re: [PATCH v4 12/17] slab: create kmem_cache_create() compatibility layer
In reply to Christian BraunerQuoted 9 lines · Christian Brauner
Use
_Generic()to create a compatibility layer that type switches on the third argument to either call__kmem_cache_create()or__kmem_cache_create_args(). If NULL is passed for the structkmem_cache_argsargument use default args making porting for callers that don't care about additional arguments easy.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 13/17 file: port to struct kmem_cache_args
In reply to Christian BraunerPort filp_cache to struct kmem_cache_args.
--- fs/file_table.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
fs/file_table.c +8 -3
@@ -511,9 +511,14 @@ EXPORT_SYMBOL(__fput_sync);void __init files_init (void ) { - filp_cachep =kmem_cache_create_rcu ("filp" ,sizeof (struct file ), - offsetof(struct file , f_freeptr), -SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT ); +struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct file , f_freeptr), + }; + + filp_cachep =kmem_cache_create ("filp" ,sizeof (struct file ), &args, +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_ACCOUNT |SLAB_TYPESAFE_BY_RCU );percpu_counter_init (&nr_files,0 ,GFP_KERNEL ); }
Signature · 1 line
2.45.2
Re: [PATCH v4 13/17] file: port to struct kmem_cache_args
Quoted 7 lines · Christian Brauner
Port
filp_cacheto structkmem_cache_args.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 14/17 slab: remove kmem_cache_create_rcu()
In reply to Christian BraunerNow that we have ported all users of kmem_cache_create_rcu() to struct kmem_cache_args the function is unused and can be removed.
--- include/linux/slab.h | 3 --- mm/slab_common.c | 43 ------------------------------------------- 2 files changed, 46 deletions(-)
include/linux/slab.h +0 -3
@@ -270,9 +270,6 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name,slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); -struct kmem_cache *kmem_cache_create_rcu (const char *name,unsigned int size, -unsigned int freeptr_offset, -slab_flags_t flags);/* If NULL is passed for @args, use this variant with default arguments. */ static inline struct kmem_cache *
mm/slab_common.c +0 -43
49 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [PATCH v4 14/17] slab: remove kmem_cache_create_rcu()
Quoted 8 lines · Christian Brauner
Now that we have ported all users of
kmem_cache_create_rcu()to structkmem_cache_argsthe function is unused and can be removed.Reviewed-by Kees Cook <[email protected]>Reviewed-by Jens Axboe <[email protected]>Reviewed-by Mike Rapoport (Microsoft) <[email protected]>Reviewed-by Vlastimil Babka <[email protected]>Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 15/17 slab: make kmem_cache_create_usercopy() static inline
In reply to Christian BraunerMake kmem_cache_create_usercopy() a static inline function.
--- include/linux/slab.h | 49 ++++++++++++++++++++++++++++++++++++++++++++----- mm/slab_common.c | 45 --------------------------------------------- 2 files changed, 44 insertions(+), 50 deletions(-)
include/linux/slab.h +44 -5
@@ -265,11 +265,50 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size,unsigned int align,slab_flags_t flags,void (*ctor)(void *)); -struct kmem_cache *kmem_cache_create_usercopy (const char *name, -unsigned int size,unsigned int align, -slab_flags_t flags, -unsigned int useroffset,unsigned int usersize, -void (*ctor)(void *)); + +/** +* kmem_cache_create_usercopy - Create a cache with a region suitable +* for copying to userspace +* @name: A string which is used in /proc/slabinfo to identify this cache. +* @size: The size of objects to be created in this cache. +* @align: The required alignment for the objects. +* @flags: SLAB flags +* @useroffset: Usercopy region offset +* @usersize: Usercopy region size +* @ctor: A constructor for the objects. +* +* Cannot be called within a interrupt, but can be interrupted. +* The @ctor is run when new pages are allocated by the cache. +* +* The flags are +* +* %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) +* to catch references to uninitialised memory. +* +* %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check +* for buffer overruns. +* +* %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware +* cacheline. This can be beneficial if you're counting cycles as closely +* as davem. +* +* Return: a pointer to the cache on success, NULL on failure. +*/ +static inline struct kmem_cache * +kmem_cache_create_usercopy (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +unsigned int useroffset,unsigned int usersize, +void (*ctor)(void *)) +{ +struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + .useroffset = useroffset, + .usersize = usersize, + }; + +return __kmem_cache_create_args (name, size, &kmem_args, flags); +}/* If NULL is passed for @args, use this variant with default arguments. */ static inline struct kmem_cache *
mm/slab_common.c +0 -45
51 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [PATCH v4 15/17] slab: make kmem_cache_create_usercopy() static inline
Quoted 3 lines · Christian Brauner
Make
kmem_cache_create_usercopy()a static inline function.Signed-off-by Christian Brauner <[email protected]>
Quoted 125 lines
--- include/linux/slab.h | 49 ++++++++++++++++++++++++++++++++++++++++++++----- mm/slab_common.c | 45 --------------------------------------------- 2 files changed, 44 insertions(+), 50 deletions(-)include/linux/slab.h +44
-5@@ -265,11 +265,50 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size,unsigned int align,slab_flags_t flags,void (*ctor)(void *)); -struct kmem_cache *kmem_cache_create_usercopy (const char *name, -unsigned int size,unsigned int align, -slab_flags_t flags, -unsigned int useroffset,unsigned int usersize, -void (*ctor)(void *)); + +/** +* kmem_cache_create_usercopy - Create a cache with a region suitable +* for copying to userspace +* @name: A string which is used in /proc/slabinfo to identify this cache. +* @size: The size of objects to be created in this cache. +* @align: The required alignment for the objects. +* @flags: SLAB flags +* @useroffset: Usercopy region offset +* @usersize: Usercopy region size +* @ctor: A constructor for the objects. +* +* Cannot be called within a interrupt, but can be interrupted. +* The @ctor is run when new pages are allocated by the cache. +* +* The flags are +* +* %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) +* to catch references to uninitialised memory. +* +* %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check +* for buffer overruns. +* +* %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware +* cacheline. This can be beneficial if you're counting cycles as closely +* as davem. +* +* Return: a pointer to the cache on success, NULL on failure. +*/ +static inline struct kmem_cache * +kmem_cache_create_usercopy (const char *name,unsigned int size, +unsigned int align,slab_flags_t flags, +unsigned int useroffset,unsigned int usersize, +void (*ctor)(void *)) +{ +struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + .useroffset = useroffset, + .usersize = usersize, + }; + +return __kmem_cache_create_args (name, size, &kmem_args, flags); +}/* If NULL is passed for @args, use this variant with default arguments. */ static inline struct kmem_cache *mm/slab_common.c +0
-45@@ -337,51 +337,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, }EXPORT_SYMBOL (__kmem_cache_create_args); -/** -* kmem_cache_create_usercopy - Create a cache with a region suitable -* for copying to userspace -* @name: A string which is used in /proc/slabinfo to identify this cache. -* @size: The size of objects to be created in this cache. -* @align: The required alignment for the objects. -* @flags: SLAB flags -* @useroffset: Usercopy region offset -* @usersize: Usercopy region size -* @ctor: A constructor for the objects. -* -* Cannot be called within a interrupt, but can be interrupted. -* The @ctor is run when new pages are allocated by the cache. -* -* The flags are -* -* %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) -* to catch references to uninitialised memory. -* -* %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check -* for buffer overruns. -* -* %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware -* cacheline. This can be beneficial if you're counting cycles as closely -* as davem. -* -* Return: a pointer to the cache on success, NULL on failure. -*/ -struct kmem_cache * -kmem_cache_create_usercopy (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -unsigned int useroffset,unsigned int usersize, -void (*ctor)(void *)) -{ -struct kmem_cache_args kmem_args = { - .align = align, - .ctor = ctor, - .useroffset = useroffset, - .usersize = usersize, - }; - -return __kmem_cache_create_args (name, size, &kmem_args, flags); -} -EXPORT_SYMBOL (kmem_cache_create_usercopy); -/** * __kmem_cache_create - Create a cache. * @name: A string which is used in /proc/slabinfo to identify this cache. Signature · 2 lines
-- 2.45.2
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v4 15/17] slab: make kmem_cache_create_usercopy() static inline
In reply to Christian BraunerQuoted 3 lines · Christian Brauner
Make
kmem_cache_create_usercopy()a static inline function.Signed-off-by Christian Brauner <[email protected]>
Christian Brauner
maintainerPATCH v4 16/17 slab: make __kmem_cache_create() static inline
In reply to Christian BraunerMake __kmem_cache_create() a static inline function.
--- include/linux/slab.h | 13 ++++++++++--- mm/slab_common.c | 38 -------------------------------------- 2 files changed, 10 insertions(+), 41 deletions(-)
include/linux/slab.h +10 -3
@@ -261,10 +261,17 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); +static inline struct kmem_cache * +__kmem_cache_create (const char *name,unsigned int size,unsigned int align, +slab_flags_t flags,void (*ctor)(void *)) +{ +struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + }; -struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -void (*ctor)(void *)); +return __kmem_cache_create_args (name, size, &kmem_args, flags); +}/** * kmem_cache_create_usercopy - Create a cache with a region suitable
mm/slab_common.c +0 -38
@@ -337,44 +337,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, }EXPORT_SYMBOL (__kmem_cache_create_args); -/** -* __kmem_cache_create - Create a cache. -* @name: A string which is used in /proc/slabinfo to identify this cache. -* @size: The size of objects to be created in this cache. -* @align: The required alignment for the objects. -* @flags: SLAB flags -* @ctor: A constructor for the objects. -* -* Cannot be called within a interrupt, but can be interrupted. -* The @ctor is run when new pages are allocated by the cache. -* -* The flags are -* -* %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) -* to catch references to uninitialised memory. -* -* %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check -* for buffer overruns. -* -* %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware -* cacheline. This can be beneficial if you're counting cycles as closely -* as davem. -* -* Return: a pointer to the cache on success, NULL on failure. -*/ -struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -void (*ctor)(void *)) -{ -struct kmem_cache_args kmem_args = { - .align = align, - .ctor = ctor, - }; - -return __kmem_cache_create_args (name, size, &kmem_args, flags); -} -EXPORT_SYMBOL (__kmem_cache_create); -static struct kmem_cache *kmem_buckets_cache__ro_after_init ;/**
Signature · 1 line
2.45.2
Re: [PATCH v4 16/17] slab: make __kmem_cache_create() static inline
Quoted 3 lines · Christian Brauner
Make
__kmem_cache_create()a static inline function.Signed-off-by Christian Brauner <[email protected]>
Quoted 83 lines
--- include/linux/slab.h | 13 ++++++++++--- mm/slab_common.c | 38 -------------------------------------- 2 files changed, 10 insertions(+), 41 deletions(-)include/linux/slab.h +10
-3@@ -261,10 +261,17 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); +static inline struct kmem_cache * +__kmem_cache_create (const char *name,unsigned int size,unsigned int align, +slab_flags_t flags,void (*ctor)(void *)) +{ +struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + }; -struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -void (*ctor)(void *)); +return __kmem_cache_create_args (name, size, &kmem_args, flags); +}/** * kmem_cache_create_usercopy - Create a cache with a region suitable mm/slab_common.c +0
-38@@ -337,44 +337,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, }EXPORT_SYMBOL (__kmem_cache_create_args); -/** -* __kmem_cache_create - Create a cache. -* @name: A string which is used in /proc/slabinfo to identify this cache. -* @size: The size of objects to be created in this cache. -* @align: The required alignment for the objects. -* @flags: SLAB flags -* @ctor: A constructor for the objects. -* -* Cannot be called within a interrupt, but can be interrupted. -* The @ctor is run when new pages are allocated by the cache. -* -* The flags are -* -* %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) -* to catch references to uninitialised memory. -* -* %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check -* for buffer overruns. -* -* %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware -* cacheline. This can be beneficial if you're counting cycles as closely -* as davem. -* -* Return: a pointer to the cache on success, NULL on failure. -*/ -struct kmem_cache *__kmem_cache_create (const char *name,unsigned int size, -unsigned int align,slab_flags_t flags, -void (*ctor)(void *)) -{ -struct kmem_cache_args kmem_args = { - .align = align, - .ctor = ctor, - }; - -return __kmem_cache_create_args (name, size, &kmem_args, flags); -} -EXPORT_SYMBOL (__kmem_cache_create); -static struct kmem_cache *kmem_buckets_cache__ro_after_init ;/** Signature · 2 lines
-- 2.45.2
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v4 16/17] slab: make __kmem_cache_create() static inline
In reply to Christian BraunerQuoted 3 lines · Christian Brauner
Make
__kmem_cache_create()a static inline function.Signed-off-by Christian Brauner <[email protected]>
Nice series, thank you!
Christian Brauner
maintainerPATCH v4 17/17 io_uring: port to struct kmem_cache_args
In reply to Christian BraunerPort req_cachep to struct kmem_cache_args.
--- io_uring/io_uring.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
io_uring/io_uring.c +8 -6
@@ -3638,6 +3638,11 @@ SYSCALL_DEFINE2(io_uring_setup, u32, entries,static int __init io_uring_init (void ) { +struct kmem_cache_args kmem_args = { + .useroffset = offsetof(struct io_kiocb, cmd.data), + .usersize = sizeof_field(struct io_kiocb, cmd.data), + }; +#define __BUILD_BUG_VERIFY_OFFSET_SIZE (stype, eoffset, esize, ename) do { \ BUILD_BUG_ON(offsetof(stype, ename) != eoffset); \ BUILD_BUG_ON(sizeof_field(stype, ename) != esize); \
@@ -3722,12 +3727,9 @@ static int __init io_uring_init(void)* range, and HARDENED_USERCOPY will complain if we haven't * correctly annotated this range. */ - req_cachep =kmem_cache_create_usercopy ("io_kiocb" , -sizeof (struct io_kiocb ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC | -SLAB_ACCOUNT |SLAB_TYPESAFE_BY_RCU , - offsetof(struct io_kiocb, cmd.data), - sizeof_field(struct io_kiocb, cmd.data), NULL); + req_cachep =kmem_cache_create ("io_kiocb" ,sizeof (struct io_kiocb ), &kmem_args, +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_TYPESAFE_BY_RCU ); io_buf_cachep =KMEM_CACHE (io_buffer,SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT );
Signature · 1 line
2.45.2
Quoted 5 lines · Christian Brauner
Hey,
This is v4 which allows NULL to be passed in the struct
kmem_cache_argsargument ofkmem_cache_create()and substitutes default parameters in this case.
Thanks, I've followed your earlier suggestion and put this series to a branch that merges the vfs.file first:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-6.12/kmem_cache_args
and I'm including it to slab/for-next
Meanwhile I think we should look at the kerneldocs for the new kmem_cache_create() (hopefully kerneldoc can be persuaded to document it without throwing errors) and the kmem_cache_args, because patches 16 and 14 just removed what we had before, including the details about freeptr_offset in patch 14.
Quoted 85 lines
As discussed last week the various kmem_cache_*() functions should be replaced by a unified function that is based around a struct, with only the basic parameters passed separately.
Vlastimil already said that he would like to keep core parameters out of the struct: name, object size, and flags. I personally don't care much and would not object to moving everything into the struct but that's a matter of taste and I yield that decision power to the maintainer.
In the first version I pointed out that the choice of name is somewhat forced as
kmem_cache_create()is taken and the only way to reuse it would be to replace all users in one go. Or to do a global sed/kmem_cache_create()/kmem_cache_create2()/g. And then introducekmem_cache_setup(). That doesn't strike me as a viable option.If we really cared about the
*_create()suffix then an alternative might be to do a sed/kmem_cache_setup()/kmem_cache_create()/g after every user in the kernel is ported. I honestly don't think that's worth it but I wanted to at least mention it to highlight the fact that this might lead to a naming compromise.However, I came up with an alternative using
_Generic()to create a compatibility layer that will call the correct variant ofkmem_cache_create()depending on whether structkmem_cache_argsis passed or not. That compatibility layer can stay in place until we updated all calls to be based on structkmem_cache_args.From a cursory grep (and not excluding Documentation mentions) we will have to replace 44
kmem_cache_create_usercopy()calls and about 463kmem_cache_create()calls which makes for a bit above 500 calls to port tokmem_cache_setup(). That'll probably be good work for people getting into kernel development.Signed-off-by Christian Brauner <[email protected]>---
Changes in v4:
- Allow NULL to be passed in the structkmem_cache_argsargument ofkmem_cache_create()and use default parameters in this case.
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- Reworded some commit messages.
- Picked up various RvBs.
- Added two patches to make two functions static inline.
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Removekmem_cache_setup()and add a compatibility layer built around
_Generic()so that we can keep thekmem_cache_create()name and type
switch on the third argument to either call__kmem_cache_create()or__kmem_cache_create_args().
- Link to v1: https://lore.kernel.org/r/[email protected]---
Christian Brauner (17):
slab: s/__kmem_cache_create/do_kmem_cache_create/g
slab: add structkmem_cache_args
slab: portkmem_cache_create()to structkmem_cache_args
slab: portkmem_cache_create_rcu()to structkmem_cache_args
slab: portkmem_cache_create_usercopy()to structkmem_cache_args
slab: pass structkmem_cache_argstocreate_cache()
slab: pullkmem_cache_open()intodo_kmem_cache_create()
slab: pass structkmem_cache_argstodo_kmem_cache_create()
slab: removercu_freeptr_offsetfrom structkmem_cache
slab: portKMEM_CACHE()to structkmem_cache_args
slab: portKMEM_CACHE_USERCOPY()to structkmem_cache_args
slab: createkmem_cache_create()compatibility layer
file: port to structkmem_cache_args
slab: removekmem_cache_create_rcu()
slab: makekmem_cache_create_usercopy()static inline
slab: make__kmem_cache_create()static inline
io_uring: port to structkmem_cache_argsfs/file_table.c | 11 ++- include/linux/slab.h | 130 +++++++++++++++++++++++++++------ io_uring/io_uring.c | 14 ++-- mm/slab.h | 6 +- mm/slab_common.c | 197 +++++++++++---------------------------------------- mm/slub.c | 162 +++++++++++++++++++++--------------------- 6 files changed, 250 insertions(+), 270 deletions(-)---
base-commit:6e016babce7c845ed015da25c7a097fa3482d95a
change-id: 20240902-work-kmem_cache_args-e9760972c7d4