mm/slab: Add kvfree_atomic() helper
Subsystems (5)
PATCH 1/2 mm/slab: Add kvfree_atomic() helper
kvmalloc() now supports non-sleeping GFP flags, including the vmalloc fallback path. This means it may return vmalloc memory even for GFP_ATOMIC and GFP_NOWAIT allocations.
Freeing such memory with kvfree() may then end up calling vfree(), which is not safe for non-sleeping contexts.
Introduce kvfree_atomic() helper for such cases. It mirrors kvfree(), but uses vfree_atomic() for vmalloced memory.
--- include/linux/slab.h | 3 +++ mm/slub.c | 16 ++++++++++++++++ 2 files changed, 19 insertions(+)
include/linux/slab.h +3 -0
@@ -1234,6 +1234,9 @@ void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long aligextern void kvfree (const void *addr); DEFINE_FREE(kvfree, void *, if (!IS_ERR_OR_NULL(_T)) kvfree(_T)) +extern void kvfree_atomic (const void *addr); +DEFINE_FREE(kvfree_atomic, void *, if (!IS_ERR_OR_NULL(_T)) kvfree_atomic(_T)) +extern void kvfree_sensitive (const void *addr,size_t len);unsigned int kmem_cache_size (struct kmem_cache *s);
mm/slub.c +16 -0
@@ -6802,6 +6802,22 @@ void kvfree(const void *addr) }EXPORT_SYMBOL (kvfree); +/** +* kvfree_atomic() - Free memory. +* @addr: Pointer to allocated memory. +* +* Same as kvfree(), but uses vfree_atomic() for vmalloc +* backed memory. Must not be called from NMI context. +*/ +void kvfree_atomic (const void *addr) +{ +if (is_vmalloc_addr (addr)) +vfree_atomic (addr); +else +kfree (addr); +} +EXPORT_SYMBOL (kvfree_atomic); +/** * kvfree_sensitive - Free a data object containing sensitive information. * @addr: address of the data object to be freed.
Signature · 1 line
2.47.3
PATCH 2/2 rhashtable: Add bucket_table_free_atomic() helper
rhashtable_insert_rehash() allocates a new bucket table with GFP_ATOMIC, as it is called from an RCU read-side critical section.
If rhashtable_rehash_attach() then fails, the new table is freed via kvfree(). This is unsafe, since kvfree() may fall back to vfree() for vmalloc-backed allocations, which can sleep and trigger:
BUG: sleeping function called from invalid context
Add bucket_table_free_atomic(), which uses kvfree_atomic() so the table can be freed safely from non-sleeping context.
--- lib/rhashtable.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
lib/rhashtable.c +9 -1
@@ -114,6 +114,14 @@ static void bucket_table_free(const struct bucket_table *tbl)kvfree (tbl); } +static void bucket_table_free_atomic (const struct bucket_table *tbl) +{ +if (tbl->nest) +nested_bucket_table_free (tbl); + +kvfree_atomic (tbl); +} +static void bucket_table_free_rcu (struct rcu_head *head) {bucket_table_free (container_of(head ,struct bucket_table ,rcu ));
@@ -473,7 +481,7 @@ static int rhashtable_insert_rehash(struct rhashtable *ht, err =rhashtable_rehash_attach (ht, tbl, new_tbl);if (err) { -bucket_table_free (new_tbl); +bucket_table_free_atomic (new_tbl);if (err == -EEXIST ) err =0 ; }else
Signature · 1 line
2.47.3
+Cc SLAB maintainers. Please use get_maintainers.pl next time.
Quoted 11 lines · Uladzislau Rezki (Sony)
kvmalloc()now supports non-sleeping GFP flags, including the vmalloc fallback path. This means it may return vmalloc memory even forGFP_ATOMICandGFP_NOWAITallocations.Freeing such memory with
kvfree()may then end up callingvfree(), which is not safe for non-sleeping contexts.Introduce
kvfree_atomic()helper for such cases. It mirrorskvfree(), but usesvfree_atomic()for vmalloced memory.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>
Quoted 46 lines
--- include/linux/slab.h | 3 +++ mm/slub.c | 16 ++++++++++++++++ 2 files changed, 19 insertions(+)include/linux/slab.h +3
-0@@ -1234,6 +1234,9 @@ void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long aligextern void kvfree (const void *addr); DEFINE_FREE(kvfree, void *, if (!IS_ERR_OR_NULL(_T)) kvfree(_T)) +extern void kvfree_atomic (const void *addr); +DEFINE_FREE(kvfree_atomic, void *, if (!IS_ERR_OR_NULL(_T)) kvfree_atomic(_T)) +extern void kvfree_sensitive (const void *addr,size_t len);unsigned int kmem_cache_size (struct kmem_cache *s);mm/slub.c +16
-0@@ -6802,6 +6802,22 @@ void kvfree(const void *addr) }EXPORT_SYMBOL (kvfree); +/** +* kvfree_atomic() - Free memory. +* @addr: Pointer to allocated memory. +* +* Same as kvfree(), but uses vfree_atomic() for vmalloc +* backed memory. Must not be called from NMI context. +*/ +void kvfree_atomic (const void *addr) +{ +if (is_vmalloc_addr (addr)) +vfree_atomic (addr); +else +kfree (addr); +} +EXPORT_SYMBOL (kvfree_atomic); +/** * kvfree_sensitive - Free a data object containing sensitive information. * @addr: address of the data object to be freed.
Quoted 10 lines · Vlastimil Babka
+Cc SLAB maintainers. Please use get_maintainers.pl next time.
Quoted 7 lines · Uladzislau Rezki (Sony)
kvmalloc()now supports non-sleeping GFP flags, including the vmalloc fallback path. This means it may return vmalloc memory even forGFP_ATOMICandGFP_NOWAITallocations.Freeing such memory with
kvfree()may then end up callingvfree(), which is not safe for non-sleeping contexts.
Quoted 7 lines
Quoted 4 lines
Introduce
kvfree_atomic()helper for such cases. It mirrorskvfree(), but usesvfree_atomic()for vmalloced memory.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>
Thank you and thank you for adding SLAB maintainers!
--
Uladzislau Rezki
Quoted 4 lines · Uladzislau Rezki (Sony)
kvmalloc()now supports non-sleeping GFP flags, including the vmalloc fallback path. This means it may return vmalloc memory even forGFP_ATOMICandGFP_NOWAITallocations.
Quoted 8 lines
Freeing such memory with
kvfree()may then end up callingvfree(), which is not safe for non-sleeping contexts.Introduce
kvfree_atomic()helper for such cases. It mirrorskvfree(), but usesvfree_atomic()for vmalloced memory.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>---
Oh, allowing k[v]free() to be called in interrupt context but not in non-sleepable context is confusing... but that's not new.
Looks good to me,
Thanks for fixing it, Ulad!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Herbert Xu
maintainerQuoted 7 lines · Uladzislau Rezki (Sony)
kvmalloc()now supports non-sleeping GFP flags, including the vmalloc fallback path. This means it may return vmalloc memory even forGFP_ATOMICandGFP_NOWAITallocations.Freeing such memory with
kvfree()may then end up callingvfree(), which is not safe for non-sleeping contexts.
Quoted 8 lines
Introduce
kvfree_atomic()helper for such cases. It mirrorskvfree(), but usesvfree_atomic()for vmalloced memory.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- include/linux/slab.h | 3 +++ mm/slub.c | 16 ++++++++++++++++ 2 files changed, 19 insertions(+)
All applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt