slab: replace cache_from_obj() with inline checks
Subsystems (2)
PATCH v2 1/1 slab: replace cache_from_obj() with inline checks
Eric Dumazet has noticed cache_from_obj() is not inlined with clang and suggested splitting it into two functions, where the smaller inlined one assumes the fastpath is !CONFIG_SLAB_FREELIST_HARDENED. However most distros enable it these days and so this would likely add a function call to the object free fastpaths.
Instead take a step back and consider that cache_from_obj() is a relict from when memcgs created their separate kmem_cache copies, as the outdated comment in build_detached_freelist() reminds us.
Meanwhile hardening/debugging had reused cache_from_obj() to validate that the freed object really belongs to a slab from the cache we think we are freeing from.
In build_detached_freelist() simply remove this, because it did not handle the NULL result from cache_from_obj() failure properly, nor validate objects (for the NULL slab->slab_cache pointer) when called via kfree_bulk(). If anyone is motivated to implement it properly, it should be possible in a similar way to kmem_cache_free().
In kmem_cache_free(), do the hardening/debugging checks directly so they are inlined by definition and virt_to_slab(obj) is performed just once. In case they failed, call a newly introduced warn_free_bad_obj() that performs the warnings outside of the fastpath, and leak the object.
As an intentional change, leak the object when slab->slab_cache differs from the cache given to kmem_cache_free(). Previously we would only leak when the object is not in a valid slab page or the slab->slab_cache pointer is NULL, and otherwise trust the slab->slab_cache over the kmem_cache_free() argument. But if those differ, it means something went wrong enough that it's best not to continue freeing.
As a result the fastpath should be inlined in all configs and the warnings are moved away.
---
I'll put this to slab/for-next. Thanks for the reviews!
Changes in v2:
- Add a comment and mention in the commit log that we are leaking the
freed object also in cases where s != slab->slab_cache (Hao Li)
- Link to v1: https://patch.msgid.link/[email protected]
--- mm/slub.c | 56 +++++++++++++++++++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 23 deletions(-)
mm/slub.c +33 -23
@@ -6738,30 +6738,26 @@ void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr) }#endif -static inline struct kmem_cache *virt_to_cache (const void *obj) +static noinline void warn_free_bad_obj(struct kmem_cache *s, void *obj) { +struct kmem_cache *cachep;struct slab *slab; slab =virt_to_slab (obj); -if (WARN_ONCE (!slab,"%s: Object is not a Slab page!\n" , __func__)) -return NULL ; -return slab->slab_cache; -} - -static inline struct kmem_cache *cache_from_obj (struct kmem_cache *s,void *x) -{ -struct kmem_cache *cachep; +if (WARN_ONCE (!slab, +"kmem_cache_free(%s, %p): object is not in a slab page\n" , + s->name, obj)) +return ; -if (!IS_ENABLED (CONFIG_SLAB_FREELIST_HARDENED ) && - !kmem_cache_debug_flags (s,SLAB_CONSISTENCY_CHECKS )) -return s; + cachep = slab->slab_cache; - cachep =virt_to_cache (x); -if (WARN (cachep && cachep != s, -"%s: Wrong slab cache. %s but object is from %s\n" , - __func__, s->name, cachep->name)) -print_tracking (cachep, x); -return cachep; +if (WARN_ONCE (cachep != s, +"kmem_cache_free(%s, %p): object belongs to different cache %s\n" , + s->name, obj, cachep ? cachep->name :"(NULL)" )) { +if (cachep) +print_tracking (cachep, obj); +return ; + } }/**
@@ -6774,11 +6770,25 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)*/ void kmem_cache_free (struct kmem_cache *s,void *x) { - s =cache_from_obj (s, x); -if (!s) -return ; +struct slab *slab; + + slab =virt_to_slab (x); + +if (IS_ENABLED (CONFIG_SLAB_FREELIST_HARDENED ) || +kmem_cache_debug_flags (s,SLAB_CONSISTENCY_CHECKS )) { + +/* +* Intentionally leak the object in these cases, because it +* would be too dangerous to continue. +*/ +if (unlikely (!slab || (slab->slab_cache != s))) { +warn_free_bad_obj (s, x); +return ; + } + } +trace_kmem_cache_free (_RET_IP_, x, s); -slab_free (s,virt_to_slab (x), x, _RET_IP_); +slab_free (s, slab, x, _RET_IP_); }EXPORT_SYMBOL (kmem_cache_free);
@@ -7305,7 +7315,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, df->s = slab->slab_cache; }else { df->slab = slab; - df->s =cache_from_obj (s, object);/* Support for memcg */ + df->s = s; }/* Start new detached freelist */
---
base-commit: 0f61b1860cc3f52aef9036d7235ed1f017632193
change-id: 20260120-b4-remove_cache_from_obj-190fcaf16789
Best regards,
Signature · 1 line
Vlastimil Babka <[email protected]>
Quoted 33 lines · Vlastimil Babka
Eric Dumazet has noticed
cache_from_obj()is not inlined with clang and suggested splitting it into two functions, where the smaller inlined one assumes the fastpath is !CONFIG_SLAB_FREELIST_HARDENED. However most distros enable it these days and so this would likely add a function call to the object free fastpaths.Instead take a step back and consider that
cache_from_obj()is a relict from when memcgs created their separatekmem_cachecopies, as the outdated comment inbuild_detached_freelist()reminds us.Meanwhile hardening/debugging had reused
cache_from_obj()to validate that the freed object really belongs to a slab from the cache we think we are freeing from.In
build_detached_freelist()simply remove this, because it did not handle the NULL result fromcache_from_obj()failure properly, nor validate objects (for the NULLslab->slab_cachepointer) when called viakfree_bulk(). If anyone is motivated to implement it properly, it should be possible in a similar way tokmem_cache_free().In
kmem_cache_free(), do the hardening/debugging checks directly so they are inlined by definition andvirt_to_slab(obj)is performed just once. In case they failed, call a newly introducedwarn_free_bad_obj()that performs the warnings outside of the fastpath, and leak the object.As an intentional change, leak the object when
slab->slab_cachediffers from the cache given tokmem_cache_free(). Previously we would only leak when the object is not in a valid slab page or theslab->slab_cachepointer is NULL, and otherwise trust theslab->slab_cacheover thekmem_cache_free()argument. But if those differ, it means something went wrong enough that it's best not to continue freeing.
Quoted 8 lines
As a result the fastpath should be inlined in all configs and the warnings are moved away.
Reported-by Eric Dumazet <[email protected]>Reviewed-by Harry Yoo <[email protected]>Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>
Thanks !
Quoted 35 lines · Eric Dumazet
Quoted 35 lines
Eric Dumazet has noticed
cache_from_obj()is not inlined with clang and suggested splitting it into two functions, where the smaller inlined one assumes the fastpath is !CONFIG_SLAB_FREELIST_HARDENED. However most distros enable it these days and so this would likely add a function call to the object free fastpaths.Instead take a step back and consider that
cache_from_obj()is a relict from when memcgs created their separatekmem_cachecopies, as the outdated comment inbuild_detached_freelist()reminds us.Meanwhile hardening/debugging had reused
cache_from_obj()to validate that the freed object really belongs to a slab from the cache we think we are freeing from.In
build_detached_freelist()simply remove this, because it did not handle the NULL result fromcache_from_obj()failure properly, nor validate objects (for the NULLslab->slab_cachepointer) when called viakfree_bulk(). If anyone is motivated to implement it properly, it should be possible in a similar way tokmem_cache_free().In
kmem_cache_free(), do the hardening/debugging checks directly so they are inlined by definition andvirt_to_slab(obj)is performed just once. In case they failed, call a newly introducedwarn_free_bad_obj()that performs the warnings outside of the fastpath, and leak the object.As an intentional change, leak the object when
slab->slab_cachediffers from the cache given tokmem_cache_free(). Previously we would only leak when the object is not in a valid slab page or theslab->slab_cachepointer is NULL, and otherwise trust theslab->slab_cacheover thekmem_cache_free()argument. But if those differ, it means something went wrong enough that it's best not to continue freeing.As a result the fastpath should be inlined in all configs and the warnings are moved away.
Quoted 8 lines
Quoted 5 lines
Reported-by Eric Dumazet <[email protected]>Reviewed-by Harry Yoo <[email protected]>Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka <[email protected]>Acked-by Eric Dumazet <[email protected]>