Skip to contentSkip to conversation

mm/slab: introduce alloc_flags and slab_alloc_context

PATCH In review Vlastimil Babka (SUSE) 5 participants 39 messages Jun 15 – Jun 18

Subsystems (5)
lore
Share & tools
lore

Vlastimil Babka (SUSE)

author maintainer
#0 lore raw

v3 cover mm/slab: introduce alloc_flags and slab_alloc_context

This series is now in slab/for-next, based on the slab-for-7.2 tag that was sent as first PR to Linus. Posting new version due to many accumulated changes, for final rounds of review. The plan is to send a second slab PR with this early next week, if nothing explodes.

Git: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/slab_alloc_flags

The slab implementation currently relies on gfp flags to convey some context information internally:

- The absence of both __GFP_RECLAIM flags is interpreted as "cannot spin on locks", and intended to be used by kmalloc_nolock(). But false positives are possible e.g. during early boot where gfp_allowed_mask clears __GFP_RECLAIM from all allocations. This leads to unnecessary allocation failures and workarounds such as fd3634312a04 ("debugobject:
Make it work with deferred page initialization - again").

- __GFP_NO_OBJ_EXT exists and takes up valuable bit in the gfp flags space, only to prevent recursive kmalloc() allocations for obj_ext arrays and sheaves.

The page allocator uses its internal alloc_flags to convey various context information, including ALLOC_TRYLOCK (meaning "cannot spin"). This series copies that concept for the slab allocator, with its own slab-specific internal flags:

- SLAB_ALLOC_DEFAULT - no extra flags (the value is 0), but explicit
- SLAB_ALLOC_NOLOCK - do not spin on locks (used by kmalloc_nolock())
- SLAB_ALLOC_NEW_SLAB - replacing existing 'bool new_slab' parameter for allocating obj_ext arrays
- SLAB_ALLOC_NO_RECURSE - replacing usage of __GFP_NO_OBJ_EXT

To reduce the amount of parameters in various internal functions, we additionally introduce slab_alloc_context (also inspired by page allocator's alloc_context) for passing a number of existing arguments and the new alloc_flags:

/* Structure holding extra parameters for slab allocations */

struct slab_alloc_context {
	unsigned long caller_addr;
	size_t orig_size;
	unsigned int alloc_flags;
	struct list_lru *lru;
};

This also replaces the existing struct partial_context.

The last necessary piece is kmalloc_flags() which can take the alloc_flags in addition to gfp flags and is intended for the recursive allocations of sheaves and obj_ext arrays, so that both SLAB_ALLOC_NOLOCK and SLAB_ALLOC_NO_RECURSE can be communicated. Internally it decides between kmalloc_nolock() and normal kmalloc() depending on SLAB_ALLOC_NOLOCK.

The rest of the series is gradually expanding the usage of both alloc_flags and slab_alloc_context as necessary, with bits of refactoring. Then, __GFP_NO_OBJ_EXT is removed completely.

Note that some usage of gfpflags_allow_spinning() relying on absence of __GFP_RECLAIM remains outside of slab (and page allocator) in memcg, page_owner and stackdepot code. These can thus yield false-positive decisions that spinning is not allowed, but should not result in important allocations failing anymore.

---
Changes in v3:
- Applied R-b tags from Harry, Hao, Suren (thanks!)
- Former Patch 1 "mm/slab: do not limit zeroing to orig_size when only
red zoning is enabled" fast tracked as a fix to slab-for-7.2 PR.
- Patch 1: refactor kasan_init handling (Harry).
- Constify struct slab_alloc_context usage eveywhere (Suren)
- Rename SLAB_ALLOC_TRYLOCK to SLAB_ALLOC_NOLOCK (Suren, Alexei)
- Reorder patches 5 and 6 (formerly 6 7) (Suren)
- Move trynode_flags refactoring from 7 to 6 to avoid bisection
hazard.
- In Patch 14, support temporarily both __GFP_NO_OBJ_EXT and SLAB_ALLOC_NO_RECURSE to prevent obj_ext -> sheaves -> obj_ext
recursion (Sashiko)
- Expand OBJCGS_CLEAR_MASK to allow kmalloc_nolock() warnings
(Hao Li, Shengming Hu).
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- Due to Sashiko review, drop the idea of zeroing orig_size
unconditionally, as it can break krealloc(). Thanks to that found a pre-existing bug fixed by the new Patch 1. The kfence zeroing related cleanup is implemented differently in Patch 2.
- Prevent nested kmalloc_nolock warnings due to added gfp flags
(Sashiko)
- Fix a pre-existing issue with opportunistic slab allocation from the
target node only effectively dropping __GFP_NOMEMALLOC and __GFP_RECLAIM. (Sashiko)
- Move kmalloc_flags() definitions to mm/slab.h (per Harry).
- Link to v1: https://patch.msgid.link/[email protected]

---
Vlastimil Babka (SUSE) (15):
mm/slab: do not init any kfence objects on allocation mm/slab: stop inlining __slab_alloc_node()
mm/slab: introduce slab_alloc_context
mm/slab: introduce alloc_flags and SLAB_ALLOC_NOLOCK mm/slab: replace struct partial_context with slab_alloc_context mm/slab: add alloc_flags to slab_alloc_context
mm/slab: pass alloc_flags to new slab allocation
mm/slab: pass alloc_flags through slab_post_alloc_hook() chain mm/slab: replace slab_alloc_node() parameters with slab_alloc_context mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags mm/slab: pass slab_alloc_context to __do_kmalloc_node() mm/slab: allow __GFP_NOMEMALLOC and __GFP_NOWARN for kmalloc_nolock() mm/slab: introduce kmalloc_flags()
mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts() mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves

 include/linux/slab.h |   5 +-
 mm/kfence/core.c     |   2 +-
 mm/memcontrol.c      |   5 +-
 mm/slab.h            |  29 ++-
 mm/slub.c            | 488 +++++++++++++++++++++++++++++++--------------------
 5 files changed, 329 insertions(+), 200 deletions(-)

---
base-commit: dfdfd58cce1c3f5df8733b64595448996c08e424
change-id: 20260601-slab_alloc_flags-25c782b0c57c

Best regards,

Signature · 1 line
Vlastimil Babka (SUSE) <[email protected]>
author maintainer
#1 lore raw

PATCH v3 1/15 mm/slab: do not init any kfence objects on allocation

When init (zeroing) on allocation is requested, for kmalloc() we generally have to zero the full object size even if a smaller size is requested, in order to provide krealloc()'s __GFP_ZERO guarantees.

When we end up allocating a kfence object, kfence performs the zeroing on its own because it has its own redzone beyond the requested size. Thus slab_post_alloc_hook() has an 'init' parameter which has to be evaluated in all callers (via slab_want_init_on_alloc()) and should be false for kfence allocations.

For kfence allocations in slab_alloc_node() this is achieved by subtly skipping over the slab_want_init_on_alloc() call. Other callers (i.e. kmem_cache_alloc_bulk_noprof()) however evaluate it unconditionally even if they do end up with a kfence allocation. This is only subtly not a problem, as those are not kmalloc allocations and thus the "requested size" equals s->object_size and thus it cannot interfere with kfence's redzone. There's just a unnecessary double zeroing (in both kfence and slab_post_alloc_hook()), but it's all very fragile and contradicts the comment in kfence_guarded_alloc().

Remove this subtlety and simplify the code by eliminating the init parameter from slab_post_alloc_hook() and make it call slab_want_init_on_alloc() itself. Instead add a is_kfence_address() check before performing the memset, which will start doing the right thing for all callers of slab_post_alloc_hook().

This potentially adds overhead of the is_kfence_address() check to allocation hotpath, but that one is designed to be as small as possible, and it's only evaluated if zeroing is about to happen. This means (aside from init_on_alloc hardening) only for __GFP_ZERO allocations, and the zeroing itself comes with an overhead likely larger than the added check.

While at it, refactor the handling of evaluating when KASAN does the init instead of SLUB, with no intended functional changes. A non-functional change is that we don't pass kasan_init as true to kasan_slab_alloc() if kasan has no integrated init, but then the value is ignored anyway, so it's theoretically more correct.

Thanks to Harry Yoo for the initial refactoring attempt, and for updated comments that are used here.

---
 mm/kfence/core.c |  2 +-
 mm/slub.c        | 60 ++++++++++++++++++++++++++------------------------------
 2 files changed, 29 insertions(+), 33 deletions(-)
mm/kfence/core.c +1 -1
index 655dc5ce3240..5e0b406924e9 100644
@@ -500,7 +500,7 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t g
 
 	/*
 	 * We check slab_want_init_on_alloc() ourselves, rather than letting
-	 * SL*B do the initialization, as otherwise we might overwrite KFENCE's
+	 * slab do the initialization, as otherwise it might overwrite KFENCE's
 	 * redzone.
 	 */
 	if (unlikely(slab_want_init_on_alloc(gfp, cache)))
mm/slub.c +28 -32

125 lines not on this page · raw message

Signature · 1 line
2.54.0
reviewer Reviewed-by on v3 1/15
#2 lore raw

Re: [PATCH v3 01/15] mm/slab: do not init any kfence objects on allocation

Quoted 39 lines · Vlastimil Babka (SUSE)

When init (zeroing) on allocation is requested, for kmalloc() we generally have to zero the full object size even if a smaller size is requested, in order to provide krealloc()'s __GFP_ZERO guarantees.

When we end up allocating a kfence object, kfence performs the zeroing on its own because it has its own redzone beyond the requested size. Thus slab_post_alloc_hook() has an 'init' parameter which has to be evaluated in all callers (via slab_want_init_on_alloc()) and should be false for kfence allocations.

For kfence allocations in slab_alloc_node() this is achieved by subtly skipping over the slab_want_init_on_alloc() call. Other callers (i.e. kmem_cache_alloc_bulk_noprof()) however evaluate it unconditionally even if they do end up with a kfence allocation. This is only subtly not a problem, as those are not kmalloc allocations and thus the "requested size" equals s->object_size and thus it cannot interfere with kfence's redzone. There's just a unnecessary double zeroing (in both kfence and slab_post_alloc_hook()), but it's all very fragile and contradicts the comment in kfence_guarded_alloc().

Remove this subtlety and simplify the code by eliminating the init parameter from slab_post_alloc_hook() and make it call slab_want_init_on_alloc() itself. Instead add a is_kfence_address() check before performing the memset, which will start doing the right thing for all callers of slab_post_alloc_hook().

This potentially adds overhead of the is_kfence_address() check to allocation hotpath, but that one is designed to be as small as possible, and it's only evaluated if zeroing is about to happen. This means (aside from init_on_alloc hardening) only for __GFP_ZERO allocations, and the zeroing itself comes with an overhead likely larger than the added check.

While at it, refactor the handling of evaluating when KASAN does the init instead of SLUB, with no intended functional changes. A non-functional change is that we don't pass kasan_init as true to kasan_slab_alloc() if kasan has no integrated init, but then the value is ignored anyway, so it's theoretically more correct.

Quoted 7 lines

Thanks to Harry Yoo for the initial refactoring attempt, and for updated comments that are used here.

Signature · 2 lines
Thanks,
Hao
maintainer Reviewed-by on v3 1/15
#3 lore raw

Re: [PATCH v3 01/15] mm/slab: do not init any kfence objects on allocation

In reply to Vlastimil Babka (SUSE)
Quoted 40 lines · Vlastimil Babka (SUSE)

When init (zeroing) on allocation is requested, for kmalloc() we generally have to zero the full object size even if a smaller size is requested, in order to provide krealloc()'s __GFP_ZERO guarantees.

When we end up allocating a kfence object, kfence performs the zeroing on its own because it has its own redzone beyond the requested size. Thus slab_post_alloc_hook() has an 'init' parameter which has to be evaluated in all callers (via slab_want_init_on_alloc()) and should be false for kfence allocations.

For kfence allocations in slab_alloc_node() this is achieved by subtly skipping over the slab_want_init_on_alloc() call. Other callers (i.e. kmem_cache_alloc_bulk_noprof()) however evaluate it unconditionally even if they do end up with a kfence allocation. This is only subtly not a problem, as those are not kmalloc allocations and thus the "requested size" equals s->object_size and thus it cannot interfere with kfence's redzone. There's just a unnecessary double zeroing (in both kfence and slab_post_alloc_hook()), but it's all very fragile and contradicts the comment in kfence_guarded_alloc().

Remove this subtlety and simplify the code by eliminating the init parameter from slab_post_alloc_hook() and make it call slab_want_init_on_alloc() itself. Instead add a is_kfence_address() check before performing the memset, which will start doing the right thing for all callers of slab_post_alloc_hook().

This potentially adds overhead of the is_kfence_address() check to allocation hotpath, but that one is designed to be as small as possible, and it's only evaluated if zeroing is about to happen. This means (aside from init_on_alloc hardening) only for __GFP_ZERO allocations, and the zeroing itself comes with an overhead likely larger than the added check.

While at it, refactor the handling of evaluating when KASAN does the init instead of SLUB, with no intended functional changes. A non-functional change is that we don't pass kasan_init as true to kasan_slab_alloc() if kasan has no integrated init, but then the value is ignored anyway, so it's theoretically more correct.

Quoted 7 lines

Thanks to Harry Yoo for the initial refactoring attempt, and for updated comments that are used here.

Quoted 159 lines
---
 mm/kfence/core.c |  2 +-
 mm/slub.c        | 60 ++++++++++++++++++++++++++------------------------------
 2 files changed, 29 insertions(+), 33 deletions(-)
mm/kfence/core.c +1 -1
index 655dc5ce3240..5e0b406924e9 100644
@@ -500,7 +500,7 @@ static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t g
 
        /*
         * We check slab_want_init_on_alloc() ourselves, rather than letting
-        * SL*B do the initialization, as otherwise we might overwrite KFENCE's
+        * slab do the initialization, as otherwise it might overwrite KFENCE's
         * redzone.
         */
        if (unlikely(slab_want_init_on_alloc(gfp, cache)))
mm/slub.c +28 -33
index e2ee8f1aaccf..d762cbe5d040 100644
@@ -4565,13 +4565,13 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags)
 
 static __fastpath_inline
 bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
-                         gfp_t flags, size_t size, void **p, bool init,
+                         gfp_t flags, size_t size, void **p,
                          unsigned int orig_size)
 {
+       bool init = slab_want_init_on_alloc(flags, s);
        unsigned int zero_size = s->object_size;
-       bool kasan_init = init;
-       size_t i;
        gfp_t init_flags = flags & gfp_allowed_mask;
+       bool kasan_init = false;
 
        /*
         * For kmalloc object, the allocated size (object_size) can be larger
@@ -4588,28 +4588,33 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
                zero_size = orig_size;
 
        /*
-        * When slab_debug is enabled, avoid memory initialization integrated
-        * into KASAN and instead zero out the memory via the memset below with
-        * the proper size. Otherwise, KASAN might overwrite SLUB redzones and
-        * cause false-positive reports. This does not lead to a performance
+        * ARM64 can set memory tags and zero the memory using a single
+        * instruction. Since HW_TAGS KASAN uses that while tagging the object,
+        * separate zeroing is unnecessary.
+        *
+        * However, KASAN never zeroes memory when slab_debug is enabled to
+        * avoid overwriting SLUB redzones. This does not lead to a performance
         * penalty on production builds, as slab_debug is not intended to be
         * enabled there.
         */
-       if (__slub_debug_enabled())
-               kasan_init = false;
+       if (kasan_has_integrated_init() && !__slub_debug_enabled()) {
+               kasan_init = init;
+               init = false;
+       }
 
-       /*
-        * As memory initialization might be integrated into KASAN,
-        * kasan_slab_alloc and initialization memset must be
-        * kept together to avoid discrepancies in behavior.
-        *
-        * As p[i] might get tagged, memset and kmemleak hook come after KASAN.
-        */
-       for (i = 0; i < size; i++) {
+       for (size_t i = 0; i < size; i++) {
                p[i] = kasan_slab_alloc(s, p[i], init_flags, kasan_init);
-               if (p[i] && init && (!kasan_init ||
-                                    !kasan_has_integrated_init()))
+
+               /*
+                * memset and hooks come after KASAN as p[i] might get tagged
+                *
+                * kfence zeroes the object instead of SLUB to avoid overwriting
+                * its own redzone starting at orig_size, which could happen
+                * with SLUB zeroing full s->object_size
+                */
+               if (init && p[i] && !is_kfence_address(p[i]))
                        memset(p[i], 0, zero_size);
+
                if (gfpflags_allow_spinning(flags))
                        kmemleak_alloc_recursive(p[i], s->object_size, 1,
                                                 s->flags, init_flags);
@@ -4910,7 +4915,6 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
                gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
 {
        void *object;
-       bool init = false;
 
        s = slab_pre_alloc_hook(s, gfpflags);
        if (unlikely(!s))
@@ -4926,16 +4930,13 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
                object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
 
        maybe_wipe_obj_freeptr(s, object);
-       init = slab_want_init_on_alloc(gfpflags, s);
 
 out:
        /*
-        * When init equals 'true', like for kzalloc() family, only
-        * @orig_size bytes might be zeroed instead of s->object_size
         * In case this fails due to memcg_slab_post_alloc_hook(),
         * object is set to NULL
         */
-       slab_post_alloc_hook(s, lru, gfpflags, 1, &object, init, orig_size);
+       slab_post_alloc_hook(s, lru, gfpflags, 1, &object, orig_size);
 
        return object;
 }
@@ -5230,7 +5231,6 @@ kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
                                   struct slab_sheaf *sheaf)
 {
        void *ret = NULL;
-       bool init;
 
        if (sheaf->size == 0)
                goto out;
@@ -5240,10 +5240,8 @@ kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
        if (likely(!ret))
                ret = sheaf->objects[--sheaf->size];
 
-       init = slab_want_init_on_alloc(gfp, s);
-
        /* add __GFP_NOFAIL to force successful memcg charging */
-       slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, init, s->object_size);
+       slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, s->object_size);
 out:
        trace_kmem_cache_alloc(_RET_IP_, ret, s, gfp, NUMA_NO_NODE);
 
@@ -5423,8 +5421,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 
 success:
        maybe_wipe_obj_freeptr(s, ret);
-       slab_post_alloc_hook(s, NULL, alloc_gfp, 1, &ret,
-                            slab_want_init_on_alloc(alloc_gfp, s), orig_size);
+       slab_post_alloc_hook(s, NULL, alloc_gfp, 1, &ret, orig_size);
 
        ret = kasan_kmalloc(s, ret, orig_size, alloc_gfp);
        return ret;
@@ -7339,8 +7336,7 @@ bool kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags,
 
 out:
        /* memcg and kmem_cache debug support and memory initialization */
-       return likely(slab_post_alloc_hook(s, NULL, flags, size, p,
-                       slab_want_init_on_alloc(flags, s), s->object_size));
+       return likely(slab_post_alloc_hook(s, NULL, flags, size, p, s->object_size));
 }
 EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof);
 
 
--

2.54.0

author maintainer
#4 lore raw

PATCH v3 2/15 mm/slab: stop inlining __slab_alloc_node()

In reply to Vlastimil Babka (SUSE)

With sheaves, this is no longer part of the allocation fastpath. For the same reason, also mark the call to it from slab_alloc_node() as unlikely().

---
 mm/slub.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/slub.c +3 -3
index d762cbe5d040..8845e15cb152 100644
@@ -4519,8 +4519,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	return object;
 }
 
-static __always_inline void *__slab_alloc_node(struct kmem_cache *s,
-		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+static void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node,
+			       unsigned long addr, size_t orig_size)
 {
 	void *object;
 
@@ -4926,7 +4926,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
 
 	object = alloc_from_pcs(s, gfpflags, node);
 
-	if (!object)
+	if (unlikely(!object))
 		object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
 
 	maybe_wipe_obj_freeptr(s, object);
Signature · 1 line
2.54.0
author maintainer
#5 lore raw

PATCH v3 3/15 mm/slab: introduce slab_alloc_context

In reply to Vlastimil Babka (SUSE)

Similarly to page allocator's struct alloc_context, introduce a helper struct to hold a part of the allocation arguments. This will allow reducing the number of parameters in many functions of the implementation, and extend them easily if needed.

For now, make it hold the caller address and the originally requested allocation size.

Convert alloc_single_from_new_slab(), __slab_alloc_node() and ___slab_alloc(). No functional change intended.

---
 mm/slub.c | 45 ++++++++++++++++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 13 deletions(-)
mm/slub.c +32 -13
index 8845e15cb152..8a0c5553876e 100644
@@ -213,6 +213,12 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled);
 static DEFINE_STATIC_KEY_FALSE(strict_numa);
 #endif
 
+/* Structure holding extra parameters for slab allocations */
+struct slab_alloc_context {
+	unsigned long caller_addr;
+	size_t orig_size;
+};
+
 /* Structure holding parameters for get_from_partial() call chain */
 struct partial_context {
 	gfp_t flags;
@@ -3687,7 +3693,8 @@ static inline void init_slab_obj_iter(struct kmem_cache *s, struct slab *slab,
  * and put the slab to the partial (or full) list.
  */
 static void *alloc_single_from_new_slab(struct kmem_cache *s, struct slab *slab,
-					int orig_size, bool allow_spin)
+					const struct slab_alloc_context *ac,
+					bool allow_spin)
 {
 	struct kmem_cache_node *n;
 	struct slab_obj_iter iter;
@@ -3705,7 +3712,7 @@ static void *alloc_single_from_new_slab(struct kmem_cache *s, struct slab *slab,
 	/* alloc_debug_processing() always expects a valid freepointer */
 	set_freepointer(s, object, slab->freelist);
 
-	if (!alloc_debug_processing(s, slab, object, orig_size)) {
+	if (!alloc_debug_processing(s, slab, object, ac->orig_size)) {
 		/*
 		 * It's not really expected that this would fail on a
 		 * freshly allocated slab, but a concurrent memory
@@ -4443,7 +4450,7 @@ static unsigned int alloc_from_new_slab(struct kmem_cache *s, struct slab *slab,
  * slab.
  */
 static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
-			   unsigned long addr, unsigned int orig_size)
+			   const struct slab_alloc_context *ac)
 {
 	bool allow_spin = gfpflags_allow_spinning(gfpflags);
 	void *object;
@@ -4476,7 +4483,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 			pc.flags = GFP_NOWAIT | __GFP_THISNODE;
 	}
 
-	pc.orig_size = orig_size;
+	pc.orig_size = ac->orig_size;
 	object = get_from_partial(s, node, &pc);
 	if (object)
 		goto success;
@@ -4496,7 +4503,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	stat(s, ALLOC_SLAB);
 
 	if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
-		object = alloc_single_from_new_slab(s, slab, orig_size, allow_spin);
+		object = alloc_single_from_new_slab(s, slab, ac, allow_spin);
 
 		if (likely(object))
 			goto success;
@@ -4514,13 +4521,13 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 
 success:
 	if (kmem_cache_debug_flags(s, SLAB_STORE_USER))
-		set_track(s, object, TRACK_ALLOC, addr, gfpflags);
+		set_track(s, object, TRACK_ALLOC, ac->caller_addr, gfpflags);
 
 	return object;
 }
 
 static void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node,
-			       unsigned long addr, size_t orig_size)
+			       const struct slab_alloc_context *ac)
 {
 	void *object;
 
@@ -4545,7 +4552,7 @@ static void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node,
 	}
 #endif
 
-	object = ___slab_alloc(s, gfpflags, node, addr, orig_size);
+	object = ___slab_alloc(s, gfpflags, node, ac);
 
 	return object;
 }
@@ -4926,8 +4933,13 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
 
 	object = alloc_from_pcs(s, gfpflags, node);
 
-	if (unlikely(!object))
-		object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
+	if (unlikely(!object)) {
+		const struct slab_alloc_context ac = {
+			.caller_addr = addr,
+			.orig_size = orig_size,
+		};
+		object = __slab_alloc_node(s, gfpflags, node, &ac);
+	}
 
 	maybe_wipe_obj_freeptr(s, object);
 
@@ -5353,6 +5365,10 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 	struct kmem_cache *s;
 	bool can_retry = true;
 	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = orig_size,
+	};
 
 	VM_WARN_ON_ONCE(gfp_flags & ~(__GFP_ACCOUNT | __GFP_ZERO |
 				      __GFP_NO_OBJ_EXT));
@@ -5398,7 +5414,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 	 * kfence_alloc. Hence call __slab_alloc_node() (at most twice)
 	 * and slab_post_alloc_hook() directly.
 	 */
-	ret = __slab_alloc_node(s, alloc_gfp, node, _RET_IP_, orig_size);
+	ret = __slab_alloc_node(s, alloc_gfp, node, &ac);
 
 	/*
 	 * It's possible we failed due to trylock as we preempted someone with
@@ -7240,10 +7256,13 @@ static bool __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
 	int i;
 
 	if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
+		const struct slab_alloc_context ac = {
+			.caller_addr = _RET_IP_,
+			.orig_size = s->object_size,
+		};
 		for (i = 0; i < size; i++) {
 
-			p[i] = ___slab_alloc(s, flags, NUMA_NO_NODE, _RET_IP_,
-					     s->object_size);
+			p[i] = ___slab_alloc(s, flags, NUMA_NO_NODE, &ac);
 			if (unlikely(!p[i]))
 				goto error;
 
Signature · 1 line
2.54.0
reviewer Reviewed-by on v3 3/15
#6 lore raw

Re: [PATCH v3 03/15] mm/slab: introduce slab_alloc_context

Quoted 8 lines · Vlastimil Babka (SUSE)

Similarly to page allocator's struct alloc_context, introduce a helper struct to hold a part of the allocation arguments. This will allow reducing the number of parameters in many functions of the implementation, and extend them easily if needed.

For now, make it hold the caller address and the originally requested allocation size.

Quoted 8 lines

Convert alloc_single_from_new_slab(), __slab_alloc_node() and ___slab_alloc(). No functional change intended.

---

Signature · 2 lines
Thanks,
Hao
author maintainer
#7 lore raw

PATCH v3 4/15 mm/slab: introduce alloc_flags and SLAB_ALLOC_NOLOCK

In reply to Vlastimil Babka (SUSE)

Similarly to the page allocators, introduce slab-allocator specific alloc flags that internally control allocation behavior in addition to gfp_flags, without occupying the limited gfp flags space.

Introduce the first flag SLAB_ALLOC_NOLOCK that behaves similarly to page allocator's ALLOC_TRYLOCK and will be used to reimplement kmalloc_nolock()'s "!allow_spin" behavior. That currently relies on gfpflags_allow_spinning() and thus the lack of both __GFP_RECLAIM flags, importantly __GFP_KSWAPD_RECLAIM. This can give false-positive results e.g. in early boot with a restricted gfp_allowed_mask.

Also introduce alloc_flags_allow_spinning() to replace the usage of gfpflags_allow_spinning().

Start using alloc_flags and the new check first in alloc_from_pcs() and __pcs_replace_empty_main(). This means some slab allocations that were falsely treated as kmalloc_nolock() due to their gfp flags will now have higher chances of success, and this will further increase with followup changes.

Remove a WARN_ON_ONCE() from refill_objects() as it's now legitimate to reach it from a slab allocation that's not _nolock() and yet lacks __GFP_KSWAPD_RECLAIM for other reasons.

---
 mm/slab.h |  9 +++++++++
 mm/slub.c | 17 ++++++++---------
 2 files changed, 17 insertions(+), 9 deletions(-)
mm/slab.h +9 -0
index 1bf9c3021ae3..f1246f0c9f74 100644
@@ -16,6 +16,15 @@
  * Internal slab definitions
  */
 
+/* slab's alloc_flags definitions */
+#define SLAB_ALLOC_DEFAULT	0x00 /* no flags */
+#define SLAB_ALLOC_NOLOCK	0x01 /* a kmalloc_nolock() allocation */
+
+static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
+{
+	return !(alloc_flags & SLAB_ALLOC_NOLOCK);
+}
+
 #ifdef CONFIG_64BIT
 # ifdef system_has_cmpxchg128
 # define system_has_freelist_aba()	system_has_cmpxchg128()
mm/slub.c +8 -9
index 8a0c5553876e..c53592baa027 100644
@@ -4641,7 +4641,8 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
  * unlocked.
  */
 static struct slub_percpu_sheaves *
-__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp)
+__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
+			 gfp_t gfp, unsigned int alloc_flags)
 {
 	struct slab_sheaf *empty = NULL;
 	struct slab_sheaf *full;
@@ -4667,7 +4668,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return NULL;
 	}
 
-	allow_spin = gfpflags_allow_spinning(gfp);
+	allow_spin = alloc_flags_allow_spinning(alloc_flags);
 
 	full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
 
@@ -4753,7 +4754,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 }
 
 static __fastpath_inline
-void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, int node)
+void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags, int node)
 {
 	struct slub_percpu_sheaves *pcs;
 	bool node_requested;
@@ -4798,7 +4799,7 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, int node)
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	if (unlikely(pcs->main->size == 0)) {
-		pcs = __pcs_replace_empty_main(s, pcs, gfp);
+		pcs = __pcs_replace_empty_main(s, pcs, gfp, alloc_flags);
 		if (unlikely(!pcs))
 			return NULL;
 	}
@@ -4931,7 +4932,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
 	if (unlikely(object))
 		goto out;
 
-	object = alloc_from_pcs(s, gfpflags, node);
+	object = alloc_from_pcs(s, gfpflags, SLAB_ALLOC_DEFAULT, node);
 
 	if (unlikely(!object)) {
 		const struct slab_alloc_context ac = {
@@ -5362,6 +5363,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 {
 	gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
 	size_t orig_size = size;
+	unsigned int alloc_flags = SLAB_ALLOC_NOLOCK;
 	struct kmem_cache *s;
 	bool can_retry = true;
 	void *ret;
@@ -5404,7 +5406,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 		 */
 		return NULL;
 
-	ret = alloc_from_pcs(s, alloc_gfp, node);
+	ret = alloc_from_pcs(s, alloc_gfp, alloc_flags, node);
 	if (ret)
 		goto success;
 
@@ -7218,9 +7220,6 @@ refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
 	unsigned int refilled;
 	struct slab *slab;
 
-	if (WARN_ON_ONCE(!gfpflags_allow_spinning(gfp)))
-		return 0;
-
 	refilled = __refill_objects_node(s, p, gfp, min, max,
 					 get_node(s, local_node),
 					 /* allow_spin = */ true);
Signature · 1 line
2.54.0
author maintainer
#8 lore raw

PATCH v3 5/15 mm/slab: replace struct partial_context with slab_alloc_context

In reply to Vlastimil Babka (SUSE)

Refactor get_from_partial_node(), get_from_any_partial(), get_from_partial() and ___slab_alloc().

Remove struct partial_context, which used to be more substantial but shrank as part of the sheaves conversion. Instead pass gfp_flags and pointer to the new slab_alloc_context, which together is a superset of partial_context, and alloc_flags are about to be added to slab_alloc_context as well.

No functional change intended.

---
 mm/slub.c | 54 +++++++++++++++++++++++++-----------------------------
 1 file changed, 25 insertions(+), 29 deletions(-)
mm/slub.c +25 -29
index c53592baa027..6f6c15d796e1 100644
@@ -219,12 +219,6 @@ struct slab_alloc_context {
 	size_t orig_size;
 };
 
-/* Structure holding parameters for get_from_partial() call chain */
-struct partial_context {
-	gfp_t flags;
-	unsigned int orig_size;
-};
-
 /* Structure holding parameters for get_partial_node_bulk() */
 struct partial_bulk_context {
 	gfp_t flags;
@@ -3825,7 +3819,8 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
  */
 static void *get_from_partial_node(struct kmem_cache *s,
 				   struct kmem_cache_node *n,
-				   struct partial_context *pc)
+				   gfp_t gfp_flags,
+				   const struct slab_alloc_context *ac)
 {
 	struct slab *slab, *slab2;
 	unsigned long flags;
@@ -3840,7 +3835,7 @@ static void *get_from_partial_node(struct kmem_cache *s,
 	if (!n || !n->nr_partial)
 		return NULL;
 
-	if (gfpflags_allow_spinning(pc->flags))
+	if (gfpflags_allow_spinning(gfp_flags))
 		spin_lock_irqsave(&n->list_lock, flags);
 	else if (!spin_trylock_irqsave(&n->list_lock, flags))
 		return NULL;
@@ -3848,12 +3843,12 @@ static void *get_from_partial_node(struct kmem_cache *s,
 
 		struct freelist_counters old, new;
 
-		if (!pfmemalloc_match(slab, pc->flags))
+		if (!pfmemalloc_match(slab, gfp_flags))
 			continue;
 
 		if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
 			object = alloc_single_from_partial(s, n, slab,
-							pc->orig_size);
+							ac->orig_size);
 			if (object)
 				break;
 			continue;
@@ -3887,15 +3882,16 @@ static void *get_from_partial_node(struct kmem_cache *s,
 /*
  * Get an object from somewhere. Search in increasing NUMA distances.
  */
-static void *get_from_any_partial(struct kmem_cache *s, struct partial_context *pc)
+static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
+				  const struct slab_alloc_context *ac)
 {
 #ifdef CONFIG_NUMA
 	struct zonelist *zonelist;
 	struct zoneref *z;
 	struct zone *zone;
-	enum zone_type highest_zoneidx = gfp_zone(pc->flags);
+	enum zone_type highest_zoneidx = gfp_zone(gfp_flags);
 	unsigned int cpuset_mems_cookie;
-	bool allow_spin = gfpflags_allow_spinning(pc->flags);
+	bool allow_spin = gfpflags_allow_spinning(gfp_flags);
 
 	/*
 	 * The defrag ratio allows a configuration of the tradeoffs between
@@ -3929,16 +3925,17 @@ static void *get_from_any_partial(struct kmem_cache *s, struct partial_context *
 		if (allow_spin)
 			cpuset_mems_cookie = read_mems_allowed_begin();
 
-		zonelist = node_zonelist(mempolicy_slab_node(), pc->flags);
+		zonelist = node_zonelist(mempolicy_slab_node(), gfp_flags);
 		for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) {
 			struct kmem_cache_node *n;
 
 			n = get_node(s, zone_to_nid(zone));
 
-			if (n && cpuset_zone_allowed(zone, pc->flags) &&
+			if (n && cpuset_zone_allowed(zone, gfp_flags) &&
 					n->nr_partial > s->min_partial) {
 
-				void *object = get_from_partial_node(s, n, pc);
+				void *object = get_from_partial_node(s, n,
+								gfp_flags, ac);
 
 				if (object) {
 					/*
@@ -3960,8 +3957,8 @@ static void *get_from_any_partial(struct kmem_cache *s, struct partial_context *
 /*
  * Get an object from a partial slab
  */
-static void *get_from_partial(struct kmem_cache *s, int node,
-			      struct partial_context *pc)
+static void *get_from_partial(struct kmem_cache *s, int node, gfp_t flags,
+			      const struct slab_alloc_context *ac)
 {
 	int searchnode = node;
 	void *object;
@@ -3969,11 +3966,11 @@ static void *get_from_partial(struct kmem_cache *s, int node,
 	if (node == NUMA_NO_NODE)
 		searchnode = numa_mem_id();
 
-	object = get_from_partial_node(s, get_node(s, searchnode), pc);
-	if (object || (node != NUMA_NO_NODE && (pc->flags & __GFP_THISNODE)))
+	object = get_from_partial_node(s, get_node(s, searchnode), flags, ac);
+	if (object || (node != NUMA_NO_NODE && (flags & __GFP_THISNODE)))
 		return object;
 
-	return get_from_any_partial(s, pc);
+	return get_from_any_partial(s, flags, ac);
 }
 
 static bool has_pcs_used(int cpu, struct kmem_cache *s)
@@ -4453,21 +4450,21 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 			   const struct slab_alloc_context *ac)
 {
 	bool allow_spin = gfpflags_allow_spinning(gfpflags);
+	gfp_t trynode_flags;
 	void *object;
 	struct slab *slab;
-	struct partial_context pc;
 	bool try_thisnode = true;
 
 	stat(s, ALLOC_SLOWPATH);
 
 new_objects:
 
-	pc.flags = gfpflags;
+	trynode_flags = gfpflags;
 	/*
 	 * When a preferred node is indicated but no __GFP_THISNODE
 	 *
 	 * 1) try to get a partial slab from target node only by having
-	 *    __GFP_THISNODE in pc.flags for get_from_partial()
+	 *    __GFP_THISNODE in trynode_flags for get_from_partial()
 	 * 2) if 1) failed, try to allocate a new slab from target node with
 	 *    GPF_NOWAIT | __GFP_THISNODE opportunistically
 	 * 3) if 2) failed, retry with original gfpflags which will allow
@@ -4478,17 +4475,16 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 		     && try_thisnode)) {
 		if (unlikely(!allow_spin))
 			/* Do not upgrade gfp to NOWAIT from more restrictive mode */
-			pc.flags = gfpflags | __GFP_THISNODE;
+			trynode_flags = gfpflags | __GFP_THISNODE;
 		else
-			pc.flags = GFP_NOWAIT | __GFP_THISNODE;
+			trynode_flags = GFP_NOWAIT | __GFP_THISNODE;
 	}
 
-	pc.orig_size = ac->orig_size;
-	object = get_from_partial(s, node, &pc);
+	object = get_from_partial(s, node, trynode_flags, ac);
 	if (object)
 		goto success;
 
-	slab = new_slab(s, pc.flags, node);
+	slab = new_slab(s, trynode_flags, node);
 
 	if (unlikely(!slab)) {
 		if (node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE)
Signature · 1 line
2.54.0
author maintainer
#9 lore raw

PATCH v3 6/15 mm/slab: add alloc_flags to slab_alloc_context

In reply to Vlastimil Babka (SUSE)

Add alloc_flags as a new field to the slab_alloc_context helper struct, so we can pass it to more functions in the slab implementation without adding another function parameter.

Start checking them via alloc_flags_allow_spinning() in alloc_single_from_new_slab() (where we can drop the allow_spin parameter), ___slab_alloc(), get_from_partial_node() and get_from_any_partial(). This further reduces false-positive spinning-not-allowed from allocations that are not kmalloc_nolock() but lack __GFP_RECLAIM flags.

_kmalloc_nolock_noprof() initializes ac.alloc_flags using its flags that are SLAB_ALLOC_NOLOCK. slab_alloc_node() and __kmem_cache_alloc_bulk() are not reachable from kmalloc_nolock() and all their callers expect spinning to be allowed, so they can use SLAB_ALLOC_DEFAULT. This is temporary as the scope of slab_alloc_context will further move to the callers, making the alloc_flags usage more obvious.

Also change how trynode_flags are constructed in ___slab_alloc() to achieve the same "do not upgrade to GFP_NOWAIT" by using masking instead of checking allow_spin. We need to do that because we now determine allow_spin from alloc_flags, and would otherwise start to upgrade e.g. kmalloc() allocations without __GFP_KSWAPD_RECLAIM (that however do allow spinning) to GFP_NOWAIT, thus including __GFP_KSWAPD_RECLAIM.

During the masking keep also existing __GFP_NOMEMALLOC (pointed out by Sashiko) and __GFP_ACCOUNT. Previously the hardcoded GFP_NOWAIT would eliminate them, but it's not a big problem that would need a separate fix.

---
 mm/slub.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)
mm/slub.c +15 -13
index 6f6c15d796e1..3a34907b881b 100644
@@ -217,6 +217,7 @@ static DEFINE_STATIC_KEY_FALSE(strict_numa);
 struct slab_alloc_context {
 	unsigned long caller_addr;
 	size_t orig_size;
+	unsigned int alloc_flags;
 };
 
 /* Structure holding parameters for get_partial_node_bulk() */
@@ -3687,9 +3688,9 @@ static inline void init_slab_obj_iter(struct kmem_cache *s, struct slab *slab,
  * and put the slab to the partial (or full) list.
  */
 static void *alloc_single_from_new_slab(struct kmem_cache *s, struct slab *slab,
-					const struct slab_alloc_context *ac,
-					bool allow_spin)
+					const struct slab_alloc_context *ac)
 {
+	bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
 	struct kmem_cache_node *n;
 	struct slab_obj_iter iter;
 	bool needs_add_partial;
@@ -3835,7 +3836,7 @@ static void *get_from_partial_node(struct kmem_cache *s,
 	if (!n || !n->nr_partial)
 		return NULL;
 
-	if (gfpflags_allow_spinning(gfp_flags))
+	if (alloc_flags_allow_spinning(ac->alloc_flags))
 		spin_lock_irqsave(&n->list_lock, flags);
 	else if (!spin_trylock_irqsave(&n->list_lock, flags))
 		return NULL;
@@ -3891,7 +3892,7 @@ static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
 	struct zone *zone;
 	enum zone_type highest_zoneidx = gfp_zone(gfp_flags);
 	unsigned int cpuset_mems_cookie;
-	bool allow_spin = gfpflags_allow_spinning(gfp_flags);
+	bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
 
 	/*
 	 * The defrag ratio allows a configuration of the tradeoffs between
@@ -4449,7 +4450,7 @@ static unsigned int alloc_from_new_slab(struct kmem_cache *s, struct slab *slab,
 static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 			   const struct slab_alloc_context *ac)
 {
-	bool allow_spin = gfpflags_allow_spinning(gfpflags);
+	bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
 	gfp_t trynode_flags;
 	void *object;
 	struct slab *slab;
@@ -4466,18 +4467,15 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	 * 1) try to get a partial slab from target node only by having
 	 *    __GFP_THISNODE in trynode_flags for get_from_partial()
 	 * 2) if 1) failed, try to allocate a new slab from target node with
-	 *    GPF_NOWAIT | __GFP_THISNODE opportunistically
+	 *    (at most) GFP_NOWAIT | __GFP_THISNODE opportunistically
 	 * 3) if 2) failed, retry with original gfpflags which will allow
 	 *    get_from_partial() try partial lists of other nodes before
 	 *    potentially allocating new page from other nodes
 	 */
 	if (unlikely(node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE)
 		     && try_thisnode)) {
-		if (unlikely(!allow_spin))
-			/* Do not upgrade gfp to NOWAIT from more restrictive mode */
-			trynode_flags = gfpflags | __GFP_THISNODE;
-		else
-			trynode_flags = GFP_NOWAIT | __GFP_THISNODE;
+		trynode_flags &= GFP_NOWAIT | __GFP_NOMEMALLOC | __GFP_ACCOUNT;
+		trynode_flags |= __GFP_NOWARN | __GFP_THISNODE;
 	}
 
 	object = get_from_partial(s, node, trynode_flags, ac);
@@ -4499,7 +4497,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	stat(s, ALLOC_SLAB);
 
 	if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
-		object = alloc_single_from_new_slab(s, slab, ac, allow_spin);
+		object = alloc_single_from_new_slab(s, slab, ac);
 
 		if (likely(object))
 			goto success;
@@ -4918,6 +4916,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
 static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
 		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
 {
+	const unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
 	void *object;
 
 	s = slab_pre_alloc_hook(s, gfpflags);
@@ -4928,12 +4927,13 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
 	if (unlikely(object))
 		goto out;
 
-	object = alloc_from_pcs(s, gfpflags, SLAB_ALLOC_DEFAULT, node);
+	object = alloc_from_pcs(s, gfpflags, alloc_flags, node);
 
 	if (unlikely(!object)) {
 		const struct slab_alloc_context ac = {
 			.caller_addr = addr,
 			.orig_size = orig_size,
+			.alloc_flags = alloc_flags,
 		};
 		object = __slab_alloc_node(s, gfpflags, node, &ac);
 	}
@@ -5366,6 +5366,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 	const struct slab_alloc_context ac = {
 		.caller_addr = _RET_IP_,
 		.orig_size = orig_size,
+		.alloc_flags = alloc_flags,
 	};
 
 	VM_WARN_ON_ONCE(gfp_flags & ~(__GFP_ACCOUNT | __GFP_ZERO |
@@ -7254,6 +7255,7 @@ static bool __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
 		const struct slab_alloc_context ac = {
 			.caller_addr = _RET_IP_,
 			.orig_size = s->object_size,
+			.alloc_flags = SLAB_ALLOC_DEFAULT,
 		};
 		for (i = 0; i < size; i++) {
 
Signature · 1 line
2.54.0
Reviewed-by on v3 6/15
#10 lore raw

Re: [PATCH v3 06/15] mm/slab: add alloc_flags to slab_alloc_context

Quoted 31 lines · Vlastimil Babka (SUSE)

Add alloc_flags as a new field to the slab_alloc_context helper struct, so we can pass it to more functions in the slab implementation without adding another function parameter.

Start checking them via alloc_flags_allow_spinning() in alloc_single_from_new_slab() (where we can drop the allow_spin parameter), ___slab_alloc(), get_from_partial_node() and get_from_any_partial(). This further reduces false-positive spinning-not-allowed from allocations that are not kmalloc_nolock() but lack __GFP_RECLAIM flags.

_kmalloc_nolock_noprof() initializes ac.alloc_flags using its flags that are SLAB_ALLOC_NOLOCK. slab_alloc_node() and __kmem_cache_alloc_bulk() are not reachable from kmalloc_nolock() and all their callers expect spinning to be allowed, so they can use SLAB_ALLOC_DEFAULT. This is temporary as the scope of slab_alloc_context will further move to the callers, making the alloc_flags usage more obvious.

Also change how trynode_flags are constructed in ___slab_alloc() to achieve the same "do not upgrade to GFP_NOWAIT" by using masking instead of checking allow_spin. We need to do that because we now determine allow_spin from alloc_flags, and would otherwise start to upgrade e.g. kmalloc() allocations without __GFP_KSWAPD_RECLAIM (that however do allow spinning) to GFP_NOWAIT, thus including __GFP_KSWAPD_RECLAIM.

During the masking keep also existing __GFP_NOMEMALLOC (pointed out by Sashiko) and __GFP_ACCOUNT. Previously the hardcoded GFP_NOWAIT would eliminate them, but it's not a big problem that would need a separate fix.

Quoted 4 lines
Quoted 129 lines
---
 mm/slub.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)
mm/slub.c +15 -14
index 6f6c15d796e1..3a34907b881b 100644
@@ -217,6 +217,7 @@ static DEFINE_STATIC_KEY_FALSE(strict_numa);
 struct slab_alloc_context {
        unsigned long caller_addr;
        size_t orig_size;
+       unsigned int alloc_flags;
 };
 
 /* Structure holding parameters for get_partial_node_bulk() */
@@ -3687,9 +3688,9 @@ static inline void init_slab_obj_iter(struct kmem_cache *s, struct slab *slab,
  * and put the slab to the partial (or full) list.
  */
 static void *alloc_single_from_new_slab(struct kmem_cache *s, struct slab *slab,
-                                       const struct slab_alloc_context *ac,
-                                       bool allow_spin)
+                                       const struct slab_alloc_context *ac)
 {
+       bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
        struct kmem_cache_node *n;
        struct slab_obj_iter iter;
        bool needs_add_partial;
@@ -3835,7 +3836,7 @@ static void *get_from_partial_node(struct kmem_cache *s,
        if (!n || !n->nr_partial)
                return NULL;
 
-       if (gfpflags_allow_spinning(gfp_flags))
+       if (alloc_flags_allow_spinning(ac->alloc_flags))
                spin_lock_irqsave(&n->list_lock, flags);
        else if (!spin_trylock_irqsave(&n->list_lock, flags))
                return NULL;
@@ -3891,7 +3892,7 @@ static void *get_from_any_partial(struct kmem_cache *s, gfp_t gfp_flags,
        struct zone *zone;
        enum zone_type highest_zoneidx = gfp_zone(gfp_flags);
        unsigned int cpuset_mems_cookie;
-       bool allow_spin = gfpflags_allow_spinning(gfp_flags);
+       bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
 
        /*
         * The defrag ratio allows a configuration of the tradeoffs between
@@ -4449,7 +4450,7 @@ static unsigned int alloc_from_new_slab(struct kmem_cache *s, struct slab *slab,
 static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
                           const struct slab_alloc_context *ac)
 {
-       bool allow_spin = gfpflags_allow_spinning(gfpflags);
+       bool allow_spin = alloc_flags_allow_spinning(ac->alloc_flags);
        gfp_t trynode_flags;
        void *object;
        struct slab *slab;
@@ -4466,18 +4467,15 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
         * 1) try to get a partial slab from target node only by having
         *    __GFP_THISNODE in trynode_flags for get_from_partial()
         * 2) if 1) failed, try to allocate a new slab from target node with
-        *    GPF_NOWAIT | __GFP_THISNODE opportunistically
+        *    (at most) GFP_NOWAIT | __GFP_THISNODE opportunistically
         * 3) if 2) failed, retry with original gfpflags which will allow
         *    get_from_partial() try partial lists of other nodes before
         *    potentially allocating new page from other nodes
         */
        if (unlikely(node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE)
                     && try_thisnode)) {
-               if (unlikely(!allow_spin))
-                       /* Do not upgrade gfp to NOWAIT from more restrictive mode */
-                       trynode_flags = gfpflags | __GFP_THISNODE;
-               else
-                       trynode_flags = GFP_NOWAIT | __GFP_THISNODE;
+               trynode_flags &= GFP_NOWAIT | __GFP_NOMEMALLOC | __GFP_ACCOUNT;
+               trynode_flags |= __GFP_NOWARN | __GFP_THISNODE;
        }
 
        object = get_from_partial(s, node, trynode_flags, ac);
@@ -4499,7 +4497,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
        stat(s, ALLOC_SLAB);
 
        if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) {
-               object = alloc_single_from_new_slab(s, slab, ac, allow_spin);
+               object = alloc_single_from_new_slab(s, slab, ac);
 
                if (likely(object))
                        goto success;
@@ -4918,6 +4916,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
 static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
                gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
 {
+       const unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
        void *object;
 
        s = slab_pre_alloc_hook(s, gfpflags);
@@ -4928,12 +4927,13 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
        if (unlikely(object))
                goto out;
 
-       object = alloc_from_pcs(s, gfpflags, SLAB_ALLOC_DEFAULT, node);
+       object = alloc_from_pcs(s, gfpflags, alloc_flags, node);
 
        if (unlikely(!object)) {
                const struct slab_alloc_context ac = {
                        .caller_addr = addr,
                        .orig_size = orig_size,
+                       .alloc_flags = alloc_flags,
                };
                object = __slab_alloc_node(s, gfpflags, node, &ac);
        }
@@ -5366,6 +5366,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
        const struct slab_alloc_context ac = {
                .caller_addr = _RET_IP_,
                .orig_size = orig_size,
+               .alloc_flags = alloc_flags,
        };
 
        VM_WARN_ON_ONCE(gfp_flags & ~(__GFP_ACCOUNT | __GFP_ZERO |
@@ -7254,6 +7255,7 @@ static bool __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
                const struct slab_alloc_context ac = {
                        .caller_addr = _RET_IP_,
                        .orig_size = s->object_size,
+                       .alloc_flags = SLAB_ALLOC_DEFAULT,
                };
                for (i = 0; i < size; i++) {
 
 
--

2.54.0

author maintainer
#11 lore raw

PATCH v3 7/15 mm/slab: pass alloc_flags to new slab allocation

In reply to Vlastimil Babka (SUSE)

Add the alloc_flags parameter to allocate_slab() and new_slab() so it can be used to determine if spinning is allowed, independently from gfp flags.

refill_objects() passes SLAB_ALLOC_DEFAULT because it can only be reached from contexts that allow spinning.

---
 mm/slub.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/slub.c +11 -12
index 3a34907b881b..a975a2e727c8 100644
@@ -3378,9 +3378,10 @@ static __always_inline void unaccount_slab(struct slab *slab, int order,
 }
 
 /* Allocate and initialize a slab without building its freelist. */
-static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
+static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
+				  unsigned int alloc_flags, int node)
 {
-	bool allow_spin = gfpflags_allow_spinning(flags);
+	bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
 	struct slab *slab;
 	struct kmem_cache_order_objects oo = s->oo;
 	gfp_t alloc_gfp;
@@ -3398,10 +3399,6 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	if ((alloc_gfp & __GFP_DIRECT_RECLAIM) && oo_order(oo) > oo_order(s->min))
 		alloc_gfp = (alloc_gfp | __GFP_NOMEMALLOC) & ~__GFP_RECLAIM;
 
-	/*
-	 * __GFP_RECLAIM could be cleared on the first allocation attempt,
-	 * so pass allow_spin flag directly.
-	 */
 	slab = alloc_slab_page(alloc_gfp, node, oo, allow_spin);
 	if (unlikely(!slab)) {
 		oo = s->min;
@@ -3438,15 +3435,17 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	return slab;
 }
 
-static struct slab *new_slab(struct kmem_cache *s, gfp_t flags, int node)
+static struct slab *new_slab(struct kmem_cache *s, gfp_t flags,
+			     unsigned int alloc_flags, int node)
 {
 	if (unlikely(flags & GFP_SLAB_BUG_MASK))
 		flags = kmalloc_fix_flags(flags);
 
 	WARN_ON_ONCE(s->ctor && (flags & __GFP_ZERO));
 
-	return allocate_slab(s,
-		flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node);
+	flags &= GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK;
+
+	return allocate_slab(s, flags, alloc_flags, node);
 }
 
 static void __free_slab(struct kmem_cache *s, struct slab *slab, bool allow_spin)
@@ -4482,7 +4481,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	if (object)
 		goto success;
 
-	slab = new_slab(s, trynode_flags, node);
+	slab = new_slab(s, trynode_flags, ac->alloc_flags, node);
 
 	if (unlikely(!slab)) {
 		if (node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE)
@@ -7230,7 +7229,7 @@ refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
 
 new_slab:
 
-	slab = new_slab(s, gfp, local_node);
+	slab = new_slab(s, gfp, SLAB_ALLOC_DEFAULT, local_node);
 	if (!slab)
 		goto out;
 
@@ -7578,7 +7577,7 @@ static void early_kmem_cache_node_alloc(int node)
 
 	BUG_ON(kmem_cache_node->size < sizeof(struct kmem_cache_node));
 
-	slab = new_slab(kmem_cache_node, GFP_NOWAIT, node);
+	slab = new_slab(kmem_cache_node, GFP_NOWAIT, SLAB_ALLOC_DEFAULT, node);
 
 	BUG_ON(!slab);
 	if (slab_nid(slab) != node) {
Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 7/15
#12 lore raw

Re: [PATCH v3 07/15] mm/slab: pass alloc_flags to new slab allocation

Quoted 4 lines · Vlastimil Babka (SUSE)

Add the alloc_flags parameter to allocate_slab() and new_slab() so it can be used to determine if spinning is allowed, independently from gfp flags.

Quoted 8 lines

refill_objects() passes SLAB_ALLOC_DEFAULT because it can only be reached from contexts that allow spinning.

---

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#13 lore raw

PATCH v3 8/15 mm/slab: pass alloc_flags through slab_post_alloc_hook() chain

In reply to Vlastimil Babka (SUSE)

Convert the whole following call stack to pass either slab_alloc_context (thus including alloc_flags) or just alloc_flags as necessary:

slab_post_alloc_hook()
alloc_tagging_slab_alloc_hook()
__alloc_tagging_slab_alloc_hook()
prepare_slab_obj_exts_hook()
alloc_slab_obj_exts()
memcg_slab_post_alloc_hook()

    __memcg_slab_post_alloc_hook()
      alloc_slab_obj_exts()

Converting all these at once avoids unnecessary churn and is mostly mechanical.

This ultimately allows to decide if spinning is allowed using alloc_flags in alloc_slab_obj_exts(), as well as slab_post_alloc_hook(). Aside from alloc_from_pcs_bulk() (to be handled next) there is nothing else in slab itself relying on gfpflags_allow_spinning() which can be false even if not called from kmalloc_nolock().

A followup change will also use the alloc_flags availability in the call stack above to remove the __GFP_NO_OBJ_EXT flag.

For alloc_slab_obj_exts(), also replace the suboptimal "bool new_slab" parameter with a SLAB_ALLOC_NEW_SLAB flag with identical functionality.

To further reduce the number of parameters of slab_post_alloc_hook(), also make 'struct list_lru *lru' (which is NULL for most callers) a new field of slab_alloc_context.

---
 mm/memcontrol.c |  5 +--
 mm/slab.h       |  6 ++--
 mm/slub.c       | 94 +++++++++++++++++++++++++++++++++------------------------
 3 files changed, 62 insertions(+), 43 deletions(-)
mm/memcontrol.c +3 -2
index c03d4787d466..29390ba13baa 100644
@@ -3424,7 +3424,8 @@ static inline size_t obj_full_size(struct kmem_cache *s)
 }
 
 bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
-				  gfp_t flags, size_t size, void **p)
+				  gfp_t flags, unsigned int slab_alloc_flags,
+				  size_t size, void **p)
 {
 	size_t obj_size = obj_full_size(s);
 	struct obj_cgroup *objcg;
@@ -3472,7 +3473,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
 		slab = virt_to_slab(p[i]);
 
 		if (!slab_obj_exts(slab) &&
-		    alloc_slab_obj_exts(slab, s, flags, false)) {
+		    alloc_slab_obj_exts(slab, s, flags, slab_alloc_flags)) {
 			continue;
 		}
 
mm/slab.h +4 -2
index f1246f0c9f74..d86203131f58 100644
@@ -19,6 +19,7 @@
 /* slab's alloc_flags definitions */
 #define SLAB_ALLOC_DEFAULT	0x00 /* no flags */
 #define SLAB_ALLOC_NOLOCK	0x01 /* a kmalloc_nolock() allocation */
+#define SLAB_ALLOC_NEW_SLAB	0x02 /* a flag for alloc_slab_obj_exts() */
 
 static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
 {
@@ -612,7 +613,7 @@ static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
 }
 
 int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
-                        gfp_t gfp, bool new_slab);
+			gfp_t gfp, unsigned int alloc_flags);
 
 #else /* CONFIG_SLAB_OBJ_EXT */
 
@@ -642,7 +643,8 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s)
 
 #ifdef CONFIG_MEMCG
 bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
-				  gfp_t flags, size_t size, void **p);
+				  gfp_t flags, unsigned int slab_alloc_flags,
+				  size_t size, void **p);
 void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
 			    void **p, int objects, unsigned long obj_exts);
 #endif
mm/slub.c +55 -39

263 lines not on this page · raw message

Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 8/15
#14 lore raw

Re: [PATCH v3 08/15] mm/slab: pass alloc_flags through slab_post_alloc_hook() chain

Quoted 27 lines · Vlastimil Babka (SUSE)

Convert the whole following call stack to pass either slab_alloc_context (thus including alloc_flags) or just alloc_flags as necessary:

slab_post_alloc_hook()
alloc_tagging_slab_alloc_hook()
__alloc_tagging_slab_alloc_hook()
prepare_slab_obj_exts_hook()
alloc_slab_obj_exts()
memcg_slab_post_alloc_hook()

    __memcg_slab_post_alloc_hook()
      alloc_slab_obj_exts()

Converting all these at once avoids unnecessary churn and is mostly mechanical.

This ultimately allows to decide if spinning is allowed using alloc_flags in alloc_slab_obj_exts(), as well as slab_post_alloc_hook(). Aside from alloc_from_pcs_bulk() (to be handled next) there is nothing else in slab itself relying on gfpflags_allow_spinning() which can be false even if not called from kmalloc_nolock().

A followup change will also use the alloc_flags availability in the call stack above to remove the __GFP_NO_OBJ_EXT flag.

For alloc_slab_obj_exts(), also replace the suboptimal "bool new_slab" parameter with a SLAB_ALLOC_NEW_SLAB flag with identical functionality.

Quoted 7 lines

To further reduce the number of parameters of slab_post_alloc_hook(), also make 'struct list_lru *lru' (which is NULL for most callers) a new field of slab_alloc_context.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
Reviewed-by
#15 lore raw

Re: [PATCH v3 08/15] mm/slab: pass alloc_flags through slab_post_alloc_hook() chain

Quoted 35 lines · Harry Yoo
Quoted 31 lines · Vlastimil Babka (SUSE)

Convert the whole following call stack to pass either slab_alloc_context (thus including alloc_flags) or just alloc_flags as necessary:

slab_post_alloc_hook()
alloc_tagging_slab_alloc_hook()
__alloc_tagging_slab_alloc_hook()
prepare_slab_obj_exts_hook()
alloc_slab_obj_exts()
memcg_slab_post_alloc_hook()

    __memcg_slab_post_alloc_hook()
      alloc_slab_obj_exts()

Converting all these at once avoids unnecessary churn and is mostly mechanical.

This ultimately allows to decide if spinning is allowed using alloc_flags in alloc_slab_obj_exts(), as well as slab_post_alloc_hook(). Aside from alloc_from_pcs_bulk() (to be handled next) there is nothing else in slab itself relying on gfpflags_allow_spinning() which can be false even if not called from kmalloc_nolock().

A followup change will also use the alloc_flags availability in the call stack above to remove the __GFP_NO_OBJ_EXT flag.

For alloc_slab_obj_exts(), also replace the suboptimal "bool new_slab" parameter with a SLAB_ALLOC_NEW_SLAB flag with identical functionality.

To further reduce the number of parameters of slab_post_alloc_hook(), also make 'struct list_lru *lru' (which is NULL for most callers) a new field of slab_alloc_context.

Quoted 6 lines
Quoted 3 lines

---

Looks good to me,

Quoted 4 lines

--
Cheers,
Harry / Hyeonggon

reviewer Reviewed-by on v3 8/15
#16 lore raw

Re: [PATCH v3 08/15] mm/slab: pass alloc_flags through slab_post_alloc_hook() chain

In reply to Vlastimil Babka (SUSE)
Quoted 27 lines · Vlastimil Babka (SUSE)

Convert the whole following call stack to pass either slab_alloc_context (thus including alloc_flags) or just alloc_flags as necessary:

slab_post_alloc_hook()
alloc_tagging_slab_alloc_hook()
__alloc_tagging_slab_alloc_hook()
prepare_slab_obj_exts_hook()
alloc_slab_obj_exts()
memcg_slab_post_alloc_hook()

    __memcg_slab_post_alloc_hook()
      alloc_slab_obj_exts()

Converting all these at once avoids unnecessary churn and is mostly mechanical.

This ultimately allows to decide if spinning is allowed using alloc_flags in alloc_slab_obj_exts(), as well as slab_post_alloc_hook(). Aside from alloc_from_pcs_bulk() (to be handled next) there is nothing else in slab itself relying on gfpflags_allow_spinning() which can be false even if not called from kmalloc_nolock().

A followup change will also use the alloc_flags availability in the call stack above to remove the __GFP_NO_OBJ_EXT flag.

For alloc_slab_obj_exts(), also replace the suboptimal "bool new_slab" parameter with a SLAB_ALLOC_NEW_SLAB flag with identical functionality.

Quoted 7 lines

To further reduce the number of parameters of slab_post_alloc_hook(), also make 'struct list_lru *lru' (which is NULL for most callers) a new field of slab_alloc_context.

---

Looks good to me.

Signature · 2 lines
Thanks,
Hao
author maintainer
#17 lore raw

PATCH v3 9/15 mm/slab: replace slab_alloc_node() parameters with slab_alloc_context

In reply to Vlastimil Babka (SUSE)

The function takes all the parameters that exist as fields in slab_alloc_context, except alloc_flags. Replace them with a single pointer.

This moves slab_alloc_context initialization to a number of callers, which is more verbose, but arguably also more clear than a long list of parameters, and most do not use the 'lru' field.

This will also allow kmalloc_nolock() to call slab_alloc_node() and reduce the special open-coding it currently has.

---
 mm/slub.c | 75 ++++++++++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 53 insertions(+), 22 deletions(-)
mm/slub.c +53 -22
index 465eb4db5770..562495b80d74 100644
@@ -4920,30 +4920,23 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
  *
  * Otherwise we can simply pick the next object from the lockless free list.
  */
-static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
-		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s,
+		gfp_t gfpflags, int node, const struct slab_alloc_context *ac)
 {
-	const unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
 	void *object;
-	const struct slab_alloc_context ac = {
-		.caller_addr = addr,
-		.orig_size = orig_size,
-		.alloc_flags = alloc_flags,
-		.lru = lru,
-	};
 
 	s = slab_pre_alloc_hook(s, gfpflags);
 	if (unlikely(!s))
 		return NULL;
 
-	object = kfence_alloc(s, orig_size, gfpflags);
+	object = kfence_alloc(s, ac->orig_size, gfpflags);
 	if (unlikely(object))
 		goto out;
 
-	object = alloc_from_pcs(s, gfpflags, alloc_flags, node);
+	object = alloc_from_pcs(s, gfpflags, ac->alloc_flags, node);
 
 	if (unlikely(!object))
-		object = __slab_alloc_node(s, gfpflags, node, &ac);
+		object = __slab_alloc_node(s, gfpflags, node, ac);
 
 	maybe_wipe_obj_freeptr(s, object);
 
@@ -4952,15 +4945,21 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
 	 * In case this fails due to memcg_slab_post_alloc_hook(),
 	 * object is set to NULL
 	 */
-	slab_post_alloc_hook(s, gfpflags, 1, &object, &ac);
+	slab_post_alloc_hook(s, gfpflags, 1, &object, ac);
 
 	return object;
 }
 
 void *kmem_cache_alloc_noprof(struct kmem_cache *s, gfp_t gfpflags)
 {
-	void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_,
-				    s->object_size);
+	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = s->object_size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
+
+	ret = slab_alloc_node(s, gfpflags, NUMA_NO_NODE, &ac);
 
 	trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
 
@@ -4971,8 +4970,15 @@ EXPORT_SYMBOL(kmem_cache_alloc_noprof);
 void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru,
 			   gfp_t gfpflags)
 {
-	void *ret = slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _RET_IP_,
-				    s->object_size);
+	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = s->object_size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+		.lru = lru,
+	};
+
+	ret = slab_alloc_node(s, gfpflags, NUMA_NO_NODE, &ac);
 
 	trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
 
@@ -5004,7 +5010,14 @@ EXPORT_SYMBOL(kmem_cache_charge);
  */
 void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int node)
 {
-	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size);
+	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = s->object_size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
+
+	ret = slab_alloc_node(s, gfpflags, node, &ac);
 
 	trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, node);
 
@@ -5334,6 +5347,11 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
 {
 	struct kmem_cache *s;
 	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = caller,
+		.orig_size = size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
 
 	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
 		ret = __kmalloc_large_node_noprof(size, flags, node);
@@ -5347,7 +5365,7 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
 
 	s = kmalloc_slab(size, b, flags, token);
 
-	ret = slab_alloc_node(s, NULL, flags, node, caller, size);
+	ret = slab_alloc_node(s, flags, node, &ac);
 	ret = kasan_kmalloc(s, ret, size, flags);
 	trace_kmalloc(caller, ret, size, s->size, flags, node);
 	return ret;
@@ -5465,8 +5483,14 @@ EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);
 
 void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size)
 {
-	void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE,
-					    _RET_IP_, size);
+	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
+
+	ret = slab_alloc_node(s, gfpflags, NUMA_NO_NODE, &ac);
 
 	trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags, NUMA_NO_NODE);
 
@@ -5478,7 +5502,14 @@ EXPORT_SYMBOL(__kmalloc_cache_noprof);
 void *__kmalloc_cache_node_noprof(struct kmem_cache *s, gfp_t gfpflags,
 				  int node, size_t size)
 {
-	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size);
+	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
+
+	ret = slab_alloc_node(s, gfpflags, node, &ac);
 
 	trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags, node);
 
Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 9/15
#18 lore raw

Re: [PATCH v3 09/15] mm/slab: replace slab_alloc_node() parameters with slab_alloc_context

Quoted 8 lines · Vlastimil Babka (SUSE)

The function takes all the parameters that exist as fields in slab_alloc_context, except alloc_flags. Replace them with a single pointer.

This moves slab_alloc_context initialization to a number of callers, which is more verbose, but arguably also more clear than a long list of parameters, and most do not use the 'lru' field.

Quoted 8 lines

This will also allow kmalloc_nolock() to call slab_alloc_node() and reduce the special open-coding it currently has.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#19 lore raw

PATCH v3 10/15 mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags

In reply to Vlastimil Babka (SUSE)

The last user of gfpflags_allow_spinning() in slab is alloc_from_pcs_bulk(), which is only called from kmem_cache_alloc_bulk().

It turns out that gfpflags_allow_spinning() is not necessary, because kmem_cache_alloc_bulk() is only expected to be called from context that does allow spinning, so simply replace it with 'true'. This means we can also drop the gfp parameter from alloc_from_pcs_bulk().

With that, we can remove the "@flags must allow spinning" part of the kernel doc, as there is no more connection to the gfp flags in the slab implementation.

Also remove a comment in alloc_slab_obj_exts() because there should be no more false positives possible due to gfp_allowed_mask during early boot.

---
 mm/slub.c | 16 ++++------------
 1 file changed, 4 insertions(+), 12 deletions(-)
mm/slub.c +4 -12
index 562495b80d74..81938774098b 100644
@@ -2171,12 +2171,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 
 	sz = obj_exts_alloc_size(s, slab, gfp);
 
-	/*
-	 * Note that allow_spin may be false during early boot and its
-	 * restricted GFP_BOOT_MASK. Due to kmalloc_nolock() only supporting
-	 * architectures with cmpxchg16b, early obj_exts will be missing for
-	 * very early allocations on those.
-	 */
 	if (unlikely(!allow_spin))
 		vec = kmalloc_nolock(sz, __GFP_ZERO | __GFP_NO_OBJ_EXT,
 				     slab_nid(slab));
@@ -4830,8 +4824,7 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags,
 }
 
 static __fastpath_inline
-unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
-				 void **p)
+unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 {
 	struct slub_percpu_sheaves *pcs;
 	struct slab_sheaf *main;
@@ -4866,7 +4859,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, gfp_t gfp, size_t size,
 		}
 
 		full = barn_replace_empty_sheaf(barn, pcs->main,
-						gfpflags_allow_spinning(gfp));
+						/* allow_spin = */ true);
 
 		if (full) {
 			stat(s, BARN_GET);
@@ -7331,8 +7324,7 @@ static bool __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
  * Allocate @size objects from @s and places them into @p.  @size must be larger
  * than 0.
  *
- * Interrupts must be enabled when calling this function and @flags must allow
- * spinning.
+ * Interrupts must be enabled when calling this function.
  *
  * Unlike alloc_pages_bulk(), this function does not check for already allocated
  * objects in @p, and thus the caller does not need to zero it.
@@ -7370,7 +7362,7 @@ bool kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags,
 		size--;
 	}
 
-	i = alloc_from_pcs_bulk(s, flags, size, p);
+	i = alloc_from_pcs_bulk(s, size, p);
 	if (i < size) {
 		/*
 		 * If we ran out of memory, don't bother with freeing back to
Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 10/15
#20 lore raw

Re: [PATCH v3 10/15] mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags

Quoted 13 lines · Vlastimil Babka (SUSE)

The last user of gfpflags_allow_spinning() in slab is alloc_from_pcs_bulk(), which is only called from kmem_cache_alloc_bulk().

It turns out that gfpflags_allow_spinning() is not necessary, because kmem_cache_alloc_bulk() is only expected to be called from context that does allow spinning, so simply replace it with 'true'. This means we can also drop the gfp parameter from alloc_from_pcs_bulk().

With that, we can remove the "@flags must allow spinning" part of the kernel doc, as there is no more connection to the gfp flags in the slab implementation.

Quoted 8 lines

Also remove a comment in alloc_slab_obj_exts() because there should be no more false positives possible due to gfp_allowed_mask during early boot.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
reviewer Reviewed-by on v3 10/15
#21 lore raw

Re: [PATCH v3 10/15] mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags

In reply to Vlastimil Babka (SUSE)
Quoted 13 lines · Vlastimil Babka (SUSE)

The last user of gfpflags_allow_spinning() in slab is alloc_from_pcs_bulk(), which is only called from kmem_cache_alloc_bulk().

It turns out that gfpflags_allow_spinning() is not necessary, because kmem_cache_alloc_bulk() is only expected to be called from context that does allow spinning, so simply replace it with 'true'. This means we can also drop the gfp parameter from alloc_from_pcs_bulk().

With that, we can remove the "@flags must allow spinning" part of the kernel doc, as there is no more connection to the gfp flags in the slab implementation.

Quoted 8 lines

Also remove a comment in alloc_slab_obj_exts() because there should be no more false positives possible due to gfp_allowed_mask during early boot.

---

Looks good to me.

Signature · 2 lines
Thanks,
Hao
author maintainer
#22 lore raw

PATCH v3 11/15 mm/slab: pass slab_alloc_context to __do_kmalloc_node()

In reply to Vlastimil Babka (SUSE)

With alloc_flags usage in slab, we can replace __GFP_NO_OBJ_EXT with an alloc flag that prevents kmalloc recursion. For that we need a version of kmalloc() that takes alloc_flags and use it in places that perform these potentially recursive kmalloc allocations (of sheaves or obj_ext arrays).

As a preparatory step, make __do_kmalloc_node() take a pointer to slab_alloc_context. This replaces the 'size' and 'caller' parameters and includes alloc_flags which we'll make use of.

---
 mm/slub.c | 54 ++++++++++++++++++++++++++++++++++++------------------
 1 file changed, 36 insertions(+), 18 deletions(-)
mm/slub.c +36 -18
index 81938774098b..537ea68f417b 100644
@@ -5335,20 +5335,16 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)
 EXPORT_SYMBOL(__kmalloc_large_node_noprof);
 
 static __always_inline
-void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
-			unsigned long caller, kmalloc_token_t token)
+void *__do_kmalloc_node(kmem_buckets *b, gfp_t flags, int node,
+			kmalloc_token_t token, const struct slab_alloc_context *ac)
 {
+	const size_t size = ac->orig_size;
 	struct kmem_cache *s;
 	void *ret;
-	const struct slab_alloc_context ac = {
-		.caller_addr = caller,
-		.orig_size = size,
-		.alloc_flags = SLAB_ALLOC_DEFAULT,
-	};
 
 	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
 		ret = __kmalloc_large_node_noprof(size, flags, node);
-		trace_kmalloc(caller, ret, size,
+		trace_kmalloc(ac->caller_addr, ret, size,
 			      PAGE_SIZE << get_order(size), flags, node);
 		return ret;
 	}
@@ -5358,22 +5354,34 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
 
 	s = kmalloc_slab(size, b, flags, token);
 
-	ret = slab_alloc_node(s, flags, node, &ac);
+	ret = slab_alloc_node(s, flags, node, ac);
 	ret = kasan_kmalloc(s, ret, size, flags);
-	trace_kmalloc(caller, ret, size, s->size, flags, node);
+	trace_kmalloc(ac->caller_addr, ret, size, s->size, flags, node);
 	return ret;
 }
 void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
 {
-	return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
-				 _RET_IP_, PASS_TOKEN_PARAM(token));
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
+
+	return __do_kmalloc_node(PASS_BUCKET_PARAM(b), flags, node,
+				 PASS_TOKEN_PARAM(token), &ac);
 }
 EXPORT_SYMBOL(__kmalloc_node_noprof);
 
 void *__kmalloc_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags)
 {
-	return __do_kmalloc_node(size, NULL, flags,  NUMA_NO_NODE, _RET_IP_,
-				 PASS_TOKEN_PARAM(token));
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
+
+	return __do_kmalloc_node(NULL, flags,  NUMA_NO_NODE,
+				 PASS_TOKEN_PARAM(token), &ac);
 }
 EXPORT_SYMBOL(__kmalloc_noprof);
 
@@ -5468,9 +5476,14 @@ EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);
 void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,
 					 int node, unsigned long caller)
 {
-	return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
-				 caller, PASS_TOKEN_PARAM(token));
+	const struct slab_alloc_context ac = {
+		.caller_addr = caller,
+		.orig_size = size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
 
+	return __do_kmalloc_node(PASS_BUCKET_PARAM(b), flags, node,
+				 PASS_TOKEN_PARAM(token), &ac);
 }
 EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);
 
@@ -6871,14 +6884,19 @@ void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long
 {
 	bool allow_block;
 	void *ret;
+	const struct slab_alloc_context ac = {
+		.caller_addr = _RET_IP_,
+		.orig_size = size,
+		.alloc_flags = SLAB_ALLOC_DEFAULT,
+	};
 
 	/*
 	 * It doesn't really make sense to fallback to vmalloc for sub page
 	 * requests
 	 */
-	ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
+	ret = __do_kmalloc_node(PASS_BUCKET_PARAM(b),
 				kmalloc_gfp_adjust(flags, size),
-				node, _RET_IP_, PASS_TOKEN_PARAM(token));
+				node, PASS_TOKEN_PARAM(token), &ac);
 	if (ret || size <= PAGE_SIZE)
 		return ret;
 
Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 11/15
#23 lore raw

Re: [PATCH v3 11/15] mm/slab: pass slab_alloc_context to __do_kmalloc_node()

Quoted 6 lines · Vlastimil Babka (SUSE)

With alloc_flags usage in slab, we can replace __GFP_NO_OBJ_EXT with an alloc flag that prevents kmalloc recursion. For that we need a version of kmalloc() that takes alloc_flags and use it in places that perform these potentially recursive kmalloc allocations (of sheaves or obj_ext arrays).

Quoted 8 lines

As a preparatory step, make __do_kmalloc_node() take a pointer to slab_alloc_context. This replaces the 'size' and 'caller' parameters and includes alloc_flags which we'll make use of.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
Reviewed-by
#24 lore raw

Re: [PATCH v3 11/15] mm/slab: pass slab_alloc_context to __do_kmalloc_node()

Quoted 14 lines · Harry Yoo
Quoted 10 lines · Vlastimil Babka (SUSE)

With alloc_flags usage in slab, we can replace __GFP_NO_OBJ_EXT with an alloc flag that prevents kmalloc recursion. For that we need a version of kmalloc() that takes alloc_flags and use it in places that perform these potentially recursive kmalloc allocations (of sheaves or obj_ext arrays).

As a preparatory step, make __do_kmalloc_node() take a pointer to slab_alloc_context. This replaces the 'size' and 'caller' parameters and includes alloc_flags which we'll make use of.

Quoted 7 lines
Quoted 4 lines

---

Looks good to me,

Quoted 4 lines

--
Cheers,
Harry / Hyeonggon

author maintainer
#25 lore raw

PATCH v3 12/15 mm/slab: allow __GFP_NOMEMALLOC and __GFP_NOWARN for kmalloc_nolock()

In reply to Vlastimil Babka (SUSE)

The two flags are added internally so there's no point for warning if they are passed by the caller as well, so allow them. This will allow simplifying obj_ext allocation under kmalloc_nolock().

Also it's not necessary to have the extra alloc_gfp variable for adding the two flags. The original gfp_flags parameter is not used anywhere except for the warning. So remove alloc_gfp and directly modify and use gfp_flags everywhere.

---
 include/linux/slab.h |  3 ++-
 mm/slub.c            | 19 ++++++++++---------
 2 files changed, 12 insertions(+), 10 deletions(-)
include/linux/slab.h +2 -1
index ce1c867dc0ba..b955f3cbb732 100644
@@ -1040,7 +1040,8 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
  * kmalloc_nolock - Allocate an object of given size from any context.
  * @size: size to allocate
  * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
- * allowed.
+ * allowed. Also __GFP_NOWARN and __GFP_NOMEMALLOC are allowed but added
+ * internally thus not necessary.
  * @node: node number of the target node.
  *
  * Return: pointer to the new object or NULL in case of error.
mm/slub.c +10 -9
index 537ea68f417b..8769083bec81 100644
@@ -5387,7 +5387,6 @@ EXPORT_SYMBOL(__kmalloc_noprof);
 
 void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node)
 {
-	gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
 	size_t orig_size = size;
 	unsigned int alloc_flags = SLAB_ALLOC_NOLOCK;
 	struct kmem_cache *s;
@@ -5400,7 +5399,9 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 	};
 
 	VM_WARN_ON_ONCE(gfp_flags & ~(__GFP_ACCOUNT | __GFP_ZERO |
-				      __GFP_NO_OBJ_EXT));
+			__GFP_NO_OBJ_EXT | __GFP_NOWARN | __GFP_NOMEMALLOC));
+
+	gfp_flags |= __GFP_NOWARN | __GFP_NOMEMALLOC;
 
 	if (unlikely(!size))
 		return ZERO_SIZE_PTR;
@@ -5419,7 +5420,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 retry:
 	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
 		return NULL;
-	s = kmalloc_slab(size, NULL, alloc_gfp, PASS_TOKEN_PARAM(token));
+	s = kmalloc_slab(size, NULL, gfp_flags, PASS_TOKEN_PARAM(token));
 
 	if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
 		/*
@@ -5433,7 +5434,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 		 */
 		return NULL;
 
-	ret = alloc_from_pcs(s, alloc_gfp, alloc_flags, node);
+	ret = alloc_from_pcs(s, gfp_flags, alloc_flags, node);
 	if (ret)
 		goto success;
 
@@ -5443,7 +5444,7 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 	 * kfence_alloc. Hence call __slab_alloc_node() (at most twice)
 	 * and slab_post_alloc_hook() directly.
 	 */
-	ret = __slab_alloc_node(s, alloc_gfp, node, &ac);
+	ret = __slab_alloc_node(s, gfp_flags, node, &ac);
 
 	/*
 	 * It's possible we failed due to trylock as we preempted someone with
@@ -5456,8 +5457,8 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 		size = s->object_size + 1;
 		/*
 		 * Another alternative is to
-		 * if (memcg) alloc_gfp &= ~__GFP_ACCOUNT;
-		 * else if (!memcg) alloc_gfp |= __GFP_ACCOUNT;
+		 * if (memcg) gfp_flags &= ~__GFP_ACCOUNT;
+		 * else if (!memcg) gfp_flags |= __GFP_ACCOUNT;
 		 * to retry from bucket of the same size.
 		 */
 		can_retry = false;
@@ -5466,9 +5467,9 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 
 success:
 	maybe_wipe_obj_freeptr(s, ret);
-	slab_post_alloc_hook(s, alloc_gfp, 1, &ret, &ac);
+	slab_post_alloc_hook(s, gfp_flags, 1, &ret, &ac);
 
-	ret = kasan_kmalloc(s, ret, orig_size, alloc_gfp);
+	ret = kasan_kmalloc(s, ret, orig_size, gfp_flags);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);
Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 12/15
#26 lore raw

Re: [PATCH v3 12/15] mm/slab: allow __GFP_NOMEMALLOC and __GFP_NOWARN for kmalloc_nolock()

Quoted 9 lines · Vlastimil Babka (SUSE)

The two flags are added internally so there's no point for warning if they are passed by the caller as well, so allow them. This will allow simplifying obj_ext allocation under kmalloc_nolock().

Also it's not necessary to have the extra alloc_gfp variable for adding the two flags. The original gfp_flags parameter is not used anywhere except for the warning. So remove alloc_gfp and directly modify and use gfp_flags everywhere.

Quoted 5 lines

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#27 lore raw

PATCH v3 13/15 mm/slab: introduce kmalloc_flags()

In reply to Vlastimil Babka (SUSE)

With alloc_flags usage in slab, we can replace __GFP_NO_OBJ_EXT with an alloc flag that prevents kmalloc recursion. For that we need a version of kmalloc() that takes alloc_flags and use it in places that perform these potentially recursive kmalloc allocations (of sheaves or obj_ext arrays).

Add this function, named kmalloc_flags(). Right now it's only useful for these nested allocations, so it doesn't need to optimize build-time constant sizes like kmalloc() or kmalloc_buckets.

Since we need it to support both normal and non-spinning kmalloc_nolock() context through the SLAB_ALLOC_NOLOCK flag, split out most of the special _kmalloc_nolock_noprof() implementation to __kmalloc_nolock_noprof() that takes a slab_alloc_context, and make _kmalloc_nolock_noprof() a simple tail calling wrapper with the proper context.

kmalloc_flags() can thus determine whether to call __kmalloc_nolock_noprof() or __do_kmalloc_node(), based on the given alloc_flags.

---
 mm/slab.h | 13 +++++++++++++
 mm/slub.c | 55 +++++++++++++++++++++++++++++++++++++++++++------------
 2 files changed, 56 insertions(+), 12 deletions(-)
mm/slab.h +13 -0
index d86203131f58..482b8e0fe797 100644
@@ -11,6 +11,7 @@
 #include <linux/memcontrol.h>
 #include <linux/kfence.h>
 #include <linux/kasan.h>
+#include <linux/slab.h>
 
 /*
  * Internal slab definitions
@@ -26,6 +27,18 @@ static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
 	return !(alloc_flags & SLAB_ALLOC_NOLOCK);
 }
 
+void *__kmalloc_flags_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags,
+				  unsigned int alloc_flags, int node)
+				  __assume_kmalloc_alignment __alloc_size(1);
+
+static __always_inline __alloc_size(1) void *_kmalloc_flags_noprof(size_t size,
+		gfp_t flags, unsigned int alloc_flags, int node, kmalloc_token_t token)
+{
+	return __kmalloc_flags_noprof(PASS_TOKEN_PARAMS(size, token), flags, alloc_flags, node);
+}
+#define kmalloc_flags_noprof(...)	_kmalloc_flags_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
+#define kmalloc_flags(...)		alloc_hooks(kmalloc_flags_noprof(__VA_ARGS__))
+
 #ifdef CONFIG_64BIT
 # ifdef system_has_cmpxchg128
 # define system_has_freelist_aba()	system_has_cmpxchg128()
mm/slub.c +43 -12

93 lines not on this page · raw message

Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 13/15
#28 lore raw

Re: [PATCH v3 13/15] mm/slab: introduce kmalloc_flags()

Quoted 21 lines · Vlastimil Babka (SUSE)

With alloc_flags usage in slab, we can replace __GFP_NO_OBJ_EXT with an alloc flag that prevents kmalloc recursion. For that we need a version of kmalloc() that takes alloc_flags and use it in places that perform these potentially recursive kmalloc allocations (of sheaves or obj_ext arrays).

Add this function, named kmalloc_flags(). Right now it's only useful for these nested allocations, so it doesn't need to optimize build-time constant sizes like kmalloc() or kmalloc_buckets.

Since we need it to support both normal and non-spinning kmalloc_nolock() context through the SLAB_ALLOC_NOLOCK flag, split out most of the special _kmalloc_nolock_noprof() implementation to __kmalloc_nolock_noprof() that takes a slab_alloc_context, and make _kmalloc_nolock_noprof() a simple tail calling wrapper with the proper context.

kmalloc_flags() can thus determine whether to call __kmalloc_nolock_noprof() or __do_kmalloc_node(), based on the given alloc_flags.

Quoted 5 lines

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#29 lore raw

PATCH v3 14/15 mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

In reply to Vlastimil Babka (SUSE)

__GFP_NO_OBJ_EXT has limited scope within the slab allocator itself and gfp flags are a scarce resource, unlike slab's alloc_flags.

Introduce SLAB_ALLOC_NO_RECURSE alloc flag that has the same intent as __GFP_NO_OBJ_EXT but a more generic name, meaning that a kmalloc() family function should not recurse into another kmalloc*() for the purposes of allocating auxiliary structures (obj_ext arrays or sheaves).

First, replace the __GFP_NO_OBJ_EXT for allocating obj_ext arrays in alloc_slab_obj_exts(). Make use of the newly added kmalloc_flags() function, where we can pass alloc_flags with SLAB_ALLOC_NO_RECURSE added. This will also pass through SLAB_ALLOC_NOLOCK so we don't need to special case kmalloc_nolock() anymore.

Note that until now the kmalloc_nolock() ignored the incoming gfp flags and hardcoded __GFP_ZERO | __GFP_NO_OBJ_EXT. But it's correct to pass on the incoming gfp flags (only augmented with __GFP_ZERO), because if alloc_flags contain SLAB_ALLOC_NOLOCK, the incoming gfp flags have to be also compatible with it. However, we might have added __GFP_THISNODE for opportunistic slab allocation, as pointed out by Hao Li, and __GFP_COMP by allocate_slab() as pointed out by Shengming Hu. Solve this by adding both flags to OBJCGS_CLEAR_MASK as it makes sense to strip them anyway for non-kmalloc_nolock() allocations of sheaves or obj_ext arrays as well.

To avoid recursion of sheaf -> obj_ext -> sheaf -> ... allocations at this patch, until the next patch converts sheaves to SLAB_ALLOC_NO_RECURSE, use both gfp and alloc_flags for obj_ext. The next patch will remove the gfp part.

---
 mm/slab.h |  1 +
 mm/slub.c | 22 ++++++++++++----------
 2 files changed, 13 insertions(+), 10 deletions(-)
mm/slab.h +1 -0
index 482b8e0fe797..281a65233795 100644
@@ -21,6 +21,7 @@
 #define SLAB_ALLOC_DEFAULT	0x00 /* no flags */
 #define SLAB_ALLOC_NOLOCK	0x01 /* a kmalloc_nolock() allocation */
 #define SLAB_ALLOC_NEW_SLAB	0x02 /* a flag for alloc_slab_obj_exts() */
+#define SLAB_ALLOC_NO_RECURSE	0x04 /* prevent kmalloc() recursion */
 
 static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
 {
mm/slub.c +12 -10
index 383d39a22561..fc5b8c85b690 100644
@@ -2047,12 +2047,16 @@ static inline void dec_slabs_node(struct kmem_cache *s, int node,
 #endif /* CONFIG_SLUB_DEBUG */
 
 /*
- * The allocated objcg pointers array is not accounted directly.
+ * The allocated objcg pointers array or sheaf is not accounted directly.
  * Moreover, it should not come from DMA buffer and is not readily
- * reclaimable. So those GFP bits should be masked off.
+ * reclaimable. Node restriction for the parent allocation also should
+ * not apply to the slab's internal objects, as well as __GFP_COMP used
+ * for new slab allocations.
+ * So those GFP bits should be masked off.
  */
 #define OBJCGS_CLEAR_MASK	(__GFP_DMA | __GFP_RECLAIMABLE | \
-				__GFP_ACCOUNT | __GFP_NOFAIL)
+				__GFP_ACCOUNT | __GFP_NOFAIL | \
+				__GFP_THISNODE | __GFP_COMP)
 
 #ifdef CONFIG_SLAB_OBJ_EXT
 
@@ -2168,14 +2172,12 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 	gfp &= ~OBJCGS_CLEAR_MASK;
 	/* Prevent recursive extension vector allocation */
 	gfp |= __GFP_NO_OBJ_EXT;
+	alloc_flags |= SLAB_ALLOC_NO_RECURSE;
 
 	sz = obj_exts_alloc_size(s, slab, gfp);
 
-	if (unlikely(!allow_spin))
-		vec = kmalloc_nolock(sz, __GFP_ZERO | __GFP_NO_OBJ_EXT,
-				     slab_nid(slab));
-	else
-		vec = kmalloc_node(sz, gfp | __GFP_ZERO, slab_nid(slab));
+	/* This will use kmalloc_nolock() if alloc_flags say so */
+	vec = kmalloc_flags(sz, gfp | __GFP_ZERO, alloc_flags, slab_nid(slab));
 
 	if (!vec) {
 		/*
@@ -2251,7 +2253,7 @@ static inline void free_slab_obj_exts(struct slab *slab, bool allow_spin)
 	}
 
 	/*
-	 * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its
+	 * obj_exts was created with SLAB_ALLOC_NO_RECURSE flag, therefore its
 	 * corresponding extension will be NULL. alloc_tag_sub() will throw a
 	 * warning if slab has extensions but the extension of an object is
 	 * NULL, therefore replace NULL with CODETAG_EMPTY to indicate that
@@ -2374,7 +2376,7 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
 	if (s->flags & (SLAB_NO_OBJ_EXT | SLAB_NOLEAKTRACE))
 		return;
 
-	if (flags & __GFP_NO_OBJ_EXT)
+	if (alloc_flags & SLAB_ALLOC_NO_RECURSE || flags & __GFP_NO_OBJ_EXT)
 		return;
 
 	slab = virt_to_slab(object);
Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 14/15
#30 lore raw

Re: [PATCH v3 14/15] mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

Quoted 25 lines · Vlastimil Babka (SUSE)

__GFP_NO_OBJ_EXT has limited scope within the slab allocator itself and gfp flags are a scarce resource, unlike slab's alloc_flags.

Introduce SLAB_ALLOC_NO_RECURSE alloc flag that has the same intent as __GFP_NO_OBJ_EXT but a more generic name, meaning that a kmalloc() family function should not recurse into another kmalloc*() for the purposes of allocating auxiliary structures (obj_ext arrays or sheaves).

First, replace the __GFP_NO_OBJ_EXT for allocating obj_ext arrays in alloc_slab_obj_exts(). Make use of the newly added kmalloc_flags() function, where we can pass alloc_flags with SLAB_ALLOC_NO_RECURSE added. This will also pass through SLAB_ALLOC_NOLOCK so we don't need to special case kmalloc_nolock() anymore.

Note that until now the kmalloc_nolock() ignored the incoming gfp flags and hardcoded __GFP_ZERO | __GFP_NO_OBJ_EXT. But it's correct to pass on the incoming gfp flags (only augmented with __GFP_ZERO), because if alloc_flags contain SLAB_ALLOC_NOLOCK, the incoming gfp flags have to be also compatible with it. However, we might have added __GFP_THISNODE for opportunistic slab allocation, as pointed out by Hao Li, and __GFP_COMP by allocate_slab() as pointed out by Shengming Hu. Solve this by adding both flags to OBJCGS_CLEAR_MASK as it makes sense to strip them anyway for non-kmalloc_nolock() allocations of sheaves or obj_ext arrays as well.

Quoted 8 lines

To avoid recursion of sheaf -> obj_ext -> sheaf -> ... allocations at this patch, until the next patch converts sheaves to SLAB_ALLOC_NO_RECURSE, use both gfp and alloc_flags for obj_ext. The next patch will remove the gfp part.

---

Looks good to me,

With some comments below.

I was worried that perhaps replacing SLAB_ALLOC_NO_RECURSE with __GFP_NO_OBJ_EXT will create a cycle of

alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)
-> kmalloc_flags(SLAB_ALLOC_NO_RECURSE)
-> alloc_from_pcs(SLAB_ALLOC_NO_RECURSE)
-> refill_objects(SLAB_ALLOC_DEFAULT)
-> new_slab(SLAB_ALLOC_DEFAULT)
-> account_slab(SLAB_ALLOC_DEFAULT)
-> alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)

with __GFP_NO_OBJ_EXT, it would have been passed to refill_objects(), but SLAB_ALLOC_NO_RECURSE is not. However this cycle does not exist because alloc_slab_obj_exts() clears __GFP_ACCOUNT (as part of OBJCG_CLEAR_MASK) and memory profiling itself does not invoke alloc_slab_obj_exts() when allocating new slabs if SLAB_ACCOUNT is not set (which is interesting, by the way).

Also alloc_slab_obj_exts() propagating SLAB_ALLOC_NEW_SLAB to kmalloc_flags() is little bit confusing because it does not have any effect due to SLAB_ALLOC_NO_RECURSE.

Those are quite subtle and perhaps worth some attention. But technically, should not be a blocker for the patch.

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#31 lore raw

Re: [PATCH v3 14/15] mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

Quoted 33 lines · Harry Yoo
Quoted 30 lines · Vlastimil Babka (SUSE)

__GFP_NO_OBJ_EXT has limited scope within the slab allocator itself and gfp flags are a scarce resource, unlike slab's alloc_flags.

Introduce SLAB_ALLOC_NO_RECURSE alloc flag that has the same intent as __GFP_NO_OBJ_EXT but a more generic name, meaning that a kmalloc() family function should not recurse into another kmalloc*() for the purposes of allocating auxiliary structures (obj_ext arrays or sheaves).

First, replace the __GFP_NO_OBJ_EXT for allocating obj_ext arrays in alloc_slab_obj_exts(). Make use of the newly added kmalloc_flags() function, where we can pass alloc_flags with SLAB_ALLOC_NO_RECURSE added. This will also pass through SLAB_ALLOC_NOLOCK so we don't need to special case kmalloc_nolock() anymore.

Note that until now the kmalloc_nolock() ignored the incoming gfp flags and hardcoded __GFP_ZERO | __GFP_NO_OBJ_EXT. But it's correct to pass on the incoming gfp flags (only augmented with __GFP_ZERO), because if alloc_flags contain SLAB_ALLOC_NOLOCK, the incoming gfp flags have to be also compatible with it. However, we might have added __GFP_THISNODE for opportunistic slab allocation, as pointed out by Hao Li, and __GFP_COMP by allocate_slab() as pointed out by Shengming Hu. Solve this by adding both flags to OBJCGS_CLEAR_MASK as it makes sense to strip them anyway for non-kmalloc_nolock() allocations of sheaves or obj_ext arrays as well.

To avoid recursion of sheaf -> obj_ext -> sheaf -> ... allocations at this patch, until the next patch converts sheaves to SLAB_ALLOC_NO_RECURSE, use both gfp and alloc_flags for obj_ext. The next patch will remove the gfp part.

Quoted 6 lines
Quoted 3 lines

---

Looks good to me,

Thanks!

Quoted 13 lines

With some comments below.

I was worried that perhaps replacing SLAB_ALLOC_NO_RECURSE with __GFP_NO_OBJ_EXT will create a cycle of

alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)
-> kmalloc_flags(SLAB_ALLOC_NO_RECURSE)
-> alloc_from_pcs(SLAB_ALLOC_NO_RECURSE)
-> refill_objects(SLAB_ALLOC_DEFAULT)
-> new_slab(SLAB_ALLOC_DEFAULT)
-> account_slab(SLAB_ALLOC_DEFAULT)
-> alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)

Quoted 6 lines

with __GFP_NO_OBJ_EXT, it would have been passed to refill_objects(), but SLAB_ALLOC_NO_RECURSE is not. However this cycle does not exist because alloc_slab_obj_exts() clears __GFP_ACCOUNT (as part of OBJCG_CLEAR_MASK) and memory profiling itself does not invoke alloc_slab_obj_exts() when allocating new slabs if SLAB_ACCOUNT is not set (which is interesting, by the way).

Hm yeah I think we should propagate alloc_flags to refill_objects() etc, to avoid later surprise. But can be done as a later cleanup.

Quoted 3 lines

Also alloc_slab_obj_exts() propagating SLAB_ALLOC_NEW_SLAB to kmalloc_flags() is little bit confusing because it does not have any effect due to SLAB_ALLOC_NO_RECURSE.

OK let's address this one by this fixup:

mm/slub.c +3 -1
index fc5b8c85b690..dc4b4ae874ce 100644
@@ -2164,6 +2164,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 {
 	const bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
 	unsigned int objects = objs_per_slab(s, slab);
+	bool new_slab = alloc_flags & SLAB_ALLOC_NEW_SLAB;
 	unsigned long new_exts;
 	unsigned long old_exts;
 	struct slabobj_ext *vec;
@@ -2173,6 +2174,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 	/* Prevent recursive extension vector allocation */
 	gfp |= __GFP_NO_OBJ_EXT;
 	alloc_flags |= SLAB_ALLOC_NO_RECURSE;
+	alloc_flags &= ~SLAB_ALLOC_NEW_SLAB;
 
 	sz = obj_exts_alloc_size(s, slab, gfp);
 
@@ -2203,7 +2205,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 	old_exts = READ_ONCE(slab->obj_exts);
 	handle_failed_objexts_alloc(old_exts, vec, objects);
 
-	if (alloc_flags & SLAB_ALLOC_NEW_SLAB) {
+	if (new_slab) {
 		/*
 		 * If the slab is brand new and nobody can yet access its
 		 * obj_exts, no synchronization is required and obj_exts can
maintainer
#32 lore raw

Re: [PATCH v3 14/15] mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

Quoted 61 lines · Vlastimil Babka (SUSE)
Quoted 31 lines · Harry Yoo
Quoted 30 lines · Vlastimil Babka (SUSE)

__GFP_NO_OBJ_EXT has limited scope within the slab allocator itself and gfp flags are a scarce resource, unlike slab's alloc_flags.

Introduce SLAB_ALLOC_NO_RECURSE alloc flag that has the same intent as __GFP_NO_OBJ_EXT but a more generic name, meaning that a kmalloc() family function should not recurse into another kmalloc*() for the purposes of allocating auxiliary structures (obj_ext arrays or sheaves).

First, replace the __GFP_NO_OBJ_EXT for allocating obj_ext arrays in alloc_slab_obj_exts(). Make use of the newly added kmalloc_flags() function, where we can pass alloc_flags with SLAB_ALLOC_NO_RECURSE added. This will also pass through SLAB_ALLOC_NOLOCK so we don't need to special case kmalloc_nolock() anymore.

Note that until now the kmalloc_nolock() ignored the incoming gfp flags and hardcoded __GFP_ZERO | __GFP_NO_OBJ_EXT. But it's correct to pass on the incoming gfp flags (only augmented with __GFP_ZERO), because if alloc_flags contain SLAB_ALLOC_NOLOCK, the incoming gfp flags have to be also compatible with it. However, we might have added __GFP_THISNODE for opportunistic slab allocation, as pointed out by Hao Li, and __GFP_COMP by allocate_slab() as pointed out by Shengming Hu. Solve this by adding both flags to OBJCGS_CLEAR_MASK as it makes sense to strip them anyway for non-kmalloc_nolock() allocations of sheaves or obj_ext arrays as well.

To avoid recursion of sheaf -> obj_ext -> sheaf -> ... allocations at this patch, until the next patch converts sheaves to SLAB_ALLOC_NO_RECURSE, use both gfp and alloc_flags for obj_ext. The next patch will remove the gfp part.

Quoted 6 lines
Quoted 3 lines

---

Looks good to me,

Thanks!

Quoted 13 lines

With some comments below.

I was worried that perhaps replacing SLAB_ALLOC_NO_RECURSE with __GFP_NO_OBJ_EXT will create a cycle of

alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)
-> kmalloc_flags(SLAB_ALLOC_NO_RECURSE)
-> alloc_from_pcs(SLAB_ALLOC_NO_RECURSE)
-> refill_objects(SLAB_ALLOC_DEFAULT)
-> new_slab(SLAB_ALLOC_DEFAULT)
-> account_slab(SLAB_ALLOC_DEFAULT)
-> alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)

Quoted 6 lines

with __GFP_NO_OBJ_EXT, it would have been passed to refill_objects(), but SLAB_ALLOC_NO_RECURSE is not. However this cycle does not exist because alloc_slab_obj_exts() clears __GFP_ACCOUNT (as part of OBJCG_CLEAR_MASK) and memory profiling itself does not invoke alloc_slab_obj_exts() when allocating new slabs if SLAB_ACCOUNT is not set (which is interesting, by the way).

Quoted 2 lines

Hm yeah I think we should propagate alloc_flags to refill_objects() etc, to avoid later surprise. But can be done as a later cleanup.

Ack.

Quoted 5 lines
Quoted 3 lines

Also alloc_slab_obj_exts() propagating SLAB_ALLOC_NEW_SLAB to kmalloc_flags() is little bit confusing because it does not have any effect due to SLAB_ALLOC_NO_RECURSE.

OK let's address this one by this fixup:

The fixup looks good to me, thanks!

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author maintainer
#33 lore raw

Re: [PATCH v3 14/15] mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

In reply to Vlastimil Babka (SUSE)
Quoted 21 lines · Vlastimil Babka (SUSE)
Quoted 14 lines

With some comments below.

I was worried that perhaps replacing SLAB_ALLOC_NO_RECURSE with __GFP_NO_OBJ_EXT will create a cycle of

alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)
-> kmalloc_flags(SLAB_ALLOC_NO_RECURSE)
-> alloc_from_pcs(SLAB_ALLOC_NO_RECURSE)
-> refill_objects(SLAB_ALLOC_DEFAULT)
-> new_slab(SLAB_ALLOC_DEFAULT)
-> account_slab(SLAB_ALLOC_DEFAULT)
-> alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)

Quoted 6 lines

with __GFP_NO_OBJ_EXT, it would have been passed to refill_objects(), but SLAB_ALLOC_NO_RECURSE is not. However this cycle does not exist because alloc_slab_obj_exts() clears __GFP_ACCOUNT (as part of OBJCG_CLEAR_MASK) and memory profiling itself does not invoke alloc_slab_obj_exts() when allocating new slabs if SLAB_ACCOUNT is not set (which is interesting, by the way).

Quoted 2 lines

Hm yeah I think we should propagate alloc_flags to refill_objects() etc, to avoid later surprise. But can be done as a later cleanup.

It's also not a new hazard I think because while previously gfp flags with __GFP_NO_OBJ_EXT would could be propagated more thoroughly than alloc_flags for obj_exts only __alloc_tagging_slab_alloc_hook() looks at them, and alloc_slab_obj_exts() (from account_slab()) didn't either, so the amount of (finite) recursion is the same I think.

Quoted 36 lines
Quoted 3 lines

Also alloc_slab_obj_exts() propagating SLAB_ALLOC_NEW_SLAB to kmalloc_flags() is little bit confusing because it does not have any effect due to SLAB_ALLOC_NO_RECURSE.

OK let's address this one by this fixup:

mm/slub.c +3 -1
index fc5b8c85b690..dc4b4ae874ce 100644
@@ -2164,6 +2164,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 {
 	const bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
 	unsigned int objects = objs_per_slab(s, slab);
+	bool new_slab = alloc_flags & SLAB_ALLOC_NEW_SLAB;
 	unsigned long new_exts;
 	unsigned long old_exts;
 	struct slabobj_ext *vec;
@@ -2173,6 +2174,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 	/* Prevent recursive extension vector allocation */
 	gfp |= __GFP_NO_OBJ_EXT;
 	alloc_flags |= SLAB_ALLOC_NO_RECURSE;
+	alloc_flags &= ~SLAB_ALLOC_NEW_SLAB;
 
 	sz = obj_exts_alloc_size(s, slab, gfp);
 
@@ -2203,7 +2205,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 	old_exts = READ_ONCE(slab->obj_exts);
 	handle_failed_objexts_alloc(old_exts, vec, objects);
 
-	if (alloc_flags & SLAB_ALLOC_NEW_SLAB) {
+	if (new_slab) {
 		/*
 		 * If the slab is brand new and nobody can yet access its
 		 * obj_exts, no synchronization is required and obj_exts can
Reviewed-by
#34 lore raw

Re: [PATCH v3 14/15] mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

Quoted 23 lines · Vlastimil Babka (SUSE)
Quoted 21 lines · Vlastimil Babka (SUSE)
Quoted 20 lines

With some comments below.

I was worried that perhaps replacing SLAB_ALLOC_NO_RECURSE with __GFP_NO_OBJ_EXT will create a cycle of

alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)
-> kmalloc_flags(SLAB_ALLOC_NO_RECURSE)
-> alloc_from_pcs(SLAB_ALLOC_NO_RECURSE)
-> refill_objects(SLAB_ALLOC_DEFAULT)
-> new_slab(SLAB_ALLOC_DEFAULT)
-> account_slab(SLAB_ALLOC_DEFAULT)
-> alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)

with __GFP_NO_OBJ_EXT, it would have been passed to refill_objects(), but SLAB_ALLOC_NO_RECURSE is not. However this cycle does not exist because alloc_slab_obj_exts() clears __GFP_ACCOUNT (as part of OBJCG_CLEAR_MASK) and memory profiling itself does not invoke alloc_slab_obj_exts() when allocating new slabs if SLAB_ACCOUNT is not set (which is interesting, by the way).

Quoted 8 lines
Quoted 2 lines

Hm yeah I think we should propagate alloc_flags to refill_objects() etc, to avoid later surprise. But can be done as a later cleanup.

It's also not a new hazard I think because while previously gfp flags with __GFP_NO_OBJ_EXT would could be propagated more thoroughly than alloc_flags for obj_exts only __alloc_tagging_slab_alloc_hook() looks at them, and alloc_slab_obj_exts() (from account_slab()) didn't either, so the amount of (finite) recursion is the same I think.

True but I think we should clean that up anyway after this change.

With the fixup you showed, LGTM.

Quoted 38 lines
Quoted 37 lines
Quoted 4 lines

Also alloc_slab_obj_exts() propagating SLAB_ALLOC_NEW_SLAB to kmalloc_flags() is little bit confusing because it does not have any effect due to SLAB_ALLOC_NO_RECURSE.

OK let's address this one by this fixup:

mm/slub.c +3 -1
index fc5b8c85b690..dc4b4ae874ce 100644
@@ -2164,6 +2164,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 {
      const bool allow_spin = alloc_flags_allow_spinning(alloc_flags);
      unsigned int objects = objs_per_slab(s, slab);
+     bool new_slab = alloc_flags & SLAB_ALLOC_NEW_SLAB;
      unsigned long new_exts;
      unsigned long old_exts;
      struct slabobj_ext *vec;
@@ -2173,6 +2174,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
      /* Prevent recursive extension vector allocation */
      gfp |= __GFP_NO_OBJ_EXT;
      alloc_flags |= SLAB_ALLOC_NO_RECURSE;
+     alloc_flags &= ~SLAB_ALLOC_NEW_SLAB;
 
      sz = obj_exts_alloc_size(s, slab, gfp);
 
@@ -2203,7 +2205,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
      old_exts = READ_ONCE(slab->obj_exts);
      handle_failed_objexts_alloc(old_exts, vec, objects);
 
-     if (alloc_flags & SLAB_ALLOC_NEW_SLAB) {
+     if (new_slab) {
              /*
               * If the slab is brand new and nobody can yet access its
               * obj_exts, no synchronization is required and obj_exts can
reviewer Reviewed-by
#35 lore raw

Re: [PATCH v3 14/15] mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

In reply to Vlastimil Babka (SUSE)
Quoted 63 lines · Vlastimil Babka (SUSE)
Quoted 33 lines · Harry Yoo
Quoted 30 lines · Vlastimil Babka (SUSE)

__GFP_NO_OBJ_EXT has limited scope within the slab allocator itself and gfp flags are a scarce resource, unlike slab's alloc_flags.

Introduce SLAB_ALLOC_NO_RECURSE alloc flag that has the same intent as __GFP_NO_OBJ_EXT but a more generic name, meaning that a kmalloc() family function should not recurse into another kmalloc*() for the purposes of allocating auxiliary structures (obj_ext arrays or sheaves).

First, replace the __GFP_NO_OBJ_EXT for allocating obj_ext arrays in alloc_slab_obj_exts(). Make use of the newly added kmalloc_flags() function, where we can pass alloc_flags with SLAB_ALLOC_NO_RECURSE added. This will also pass through SLAB_ALLOC_NOLOCK so we don't need to special case kmalloc_nolock() anymore.

Note that until now the kmalloc_nolock() ignored the incoming gfp flags and hardcoded __GFP_ZERO | __GFP_NO_OBJ_EXT. But it's correct to pass on the incoming gfp flags (only augmented with __GFP_ZERO), because if alloc_flags contain SLAB_ALLOC_NOLOCK, the incoming gfp flags have to be also compatible with it. However, we might have added __GFP_THISNODE for opportunistic slab allocation, as pointed out by Hao Li, and __GFP_COMP by allocate_slab() as pointed out by Shengming Hu. Solve this by adding both flags to OBJCGS_CLEAR_MASK as it makes sense to strip them anyway for non-kmalloc_nolock() allocations of sheaves or obj_ext arrays as well.

To avoid recursion of sheaf -> obj_ext -> sheaf -> ... allocations at this patch, until the next patch converts sheaves to SLAB_ALLOC_NO_RECURSE, use both gfp and alloc_flags for obj_ext. The next patch will remove the gfp part.

Quoted 6 lines
Quoted 3 lines

---

Looks good to me,

Thanks!

Quoted 13 lines

With some comments below.

I was worried that perhaps replacing SLAB_ALLOC_NO_RECURSE with __GFP_NO_OBJ_EXT will create a cycle of

alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)
-> kmalloc_flags(SLAB_ALLOC_NO_RECURSE)
-> alloc_from_pcs(SLAB_ALLOC_NO_RECURSE)
-> refill_objects(SLAB_ALLOC_DEFAULT)
-> new_slab(SLAB_ALLOC_DEFAULT)
-> account_slab(SLAB_ALLOC_DEFAULT)
-> alloc_slab_obj_exts(SLAB_ALLOC_DEFAULT)

Quoted 6 lines

with __GFP_NO_OBJ_EXT, it would have been passed to refill_objects(), but SLAB_ALLOC_NO_RECURSE is not. However this cycle does not exist because alloc_slab_obj_exts() clears __GFP_ACCOUNT (as part of OBJCG_CLEAR_MASK) and memory profiling itself does not invoke alloc_slab_obj_exts() when allocating new slabs if SLAB_ACCOUNT is not set (which is interesting, by the way).

Quoted 8 lines

Hm yeah I think we should propagate alloc_flags to refill_objects() etc, to avoid later surprise. But can be done as a later cleanup.

Quoted 3 lines

Also alloc_slab_obj_exts() propagating SLAB_ALLOC_NEW_SLAB to kmalloc_flags() is little bit confusing because it does not have any effect due to SLAB_ALLOC_NO_RECURSE.

OK let's address this one by this fixup:

Both the patch and the fix looks good to me.

Signature · 2 lines
Thanks,
Hao
author maintainer
#36 lore raw

PATCH v3 15/15 mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves

In reply to Vlastimil Babka (SUSE)

Finish the switch away from __GFP_NO_OBJ_EXT by replacing it with SLAB_ALLOC_NO_RECURSE when allocating empty sheaves. Pass alloc_flags to [__]alloc_empty_sheaf(). Callers that can't be part of a recursive kmalloc() chain simply pass SLAB_ALLOC_DEFAULT. Use kmalloc_flags() instead of kzalloc() for allocating the sheaf.

With that we can finalize the removal the __GFP_NO_OBJ_EXT handling from obj_ext allocations as well, leaving only SLAB_ALLOC_NO_RECURSE in place.

This leaves __GFP_NO_OBJ_EXT with no users in slab, so stop allowing the flag in kmalloc_nolock().

---
 include/linux/slab.h |  6 +++---
 mm/slub.c            | 34 +++++++++++++++++-----------------
 2 files changed, 20 insertions(+), 20 deletions(-)
include/linux/slab.h +3 -3
index b955f3cbb732..43c3d9b51107 100644
@@ -1039,9 +1039,9 @@ void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, in
 /**
  * kmalloc_nolock - Allocate an object of given size from any context.
  * @size: size to allocate
- * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
- * allowed. Also __GFP_NOWARN and __GFP_NOMEMALLOC are allowed but added
- * internally thus not necessary.
+ * @gfp_flags: GFP flags. Only __GFP_ACCOUNT and __GFP_ZERO allowed.  Also
+ * __GFP_NOWARN and __GFP_NOMEMALLOC are allowed but added internally thus not
+ * necessary.
  * @node: node number of the target node.
  *
  * Return: pointer to the new object or NULL in case of error.
mm/slub.c +17 -17
index fc5b8c85b690..62e9cd46916f 100644
@@ -2171,7 +2171,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 
 	gfp &= ~OBJCGS_CLEAR_MASK;
 	/* Prevent recursive extension vector allocation */
-	gfp |= __GFP_NO_OBJ_EXT;
 	alloc_flags |= SLAB_ALLOC_NO_RECURSE;
 
 	sz = obj_exts_alloc_size(s, slab, gfp);
@@ -2376,7 +2375,7 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,
 	if (s->flags & (SLAB_NO_OBJ_EXT | SLAB_NOLEAKTRACE))
 		return;
 
-	if (alloc_flags & SLAB_ALLOC_NO_RECURSE || flags & __GFP_NO_OBJ_EXT)
+	if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
 		return;
 
 	slab = virt_to_slab(object);
@@ -2761,7 +2760,7 @@ static inline void *setup_object(struct kmem_cache *s, void *object)
 }
 
 static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp,
-					      unsigned int capacity)
+				unsigned int alloc_flags, unsigned int capacity)
 {
 	struct slab_sheaf *sheaf;
 	size_t sheaf_size;
@@ -2772,10 +2771,10 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp,
 	 * bucket)
 	 */
 	if (s->flags & SLAB_KMALLOC)
-		gfp |= __GFP_NO_OBJ_EXT;
+		alloc_flags |= SLAB_ALLOC_NO_RECURSE;
 
 	sheaf_size = struct_size(sheaf, objects, capacity);
-	sheaf = kzalloc(sheaf_size, gfp);
+	sheaf = kmalloc_flags(sheaf_size, gfp | __GFP_ZERO, alloc_flags, NUMA_NO_NODE);
 
 	if (unlikely(!sheaf))
 		return NULL;
@@ -2788,20 +2787,20 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp,
 }
 
 static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s,
-						   gfp_t gfp)
+				gfp_t gfp, unsigned int alloc_flags)
 {
-	if (gfp & __GFP_NO_OBJ_EXT)
+	if (alloc_flags & SLAB_ALLOC_NO_RECURSE)
 		return NULL;
 
 	gfp &= ~OBJCGS_CLEAR_MASK;
 
-	return __alloc_empty_sheaf(s, gfp, s->sheaf_capacity);
+	return __alloc_empty_sheaf(s, gfp, alloc_flags, s->sheaf_capacity);
 }
 
 static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
 {
 	/*
-	 * If the sheaf was created with __GFP_NO_OBJ_EXT flag then its
+	 * If the sheaf was created with SLAB_ALLOC_NO_RECURSE flag then its
 	 * corresponding extension is NULL and alloc_tag_sub() will throw a
 	 * warning, therefore replace NULL with CODETAG_EMPTY to indicate
 	 * that the extension for this sheaf is expected to be NULL.
@@ -4693,7 +4692,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return NULL;
 
 	if (!empty) {
-		empty = alloc_empty_sheaf(s, gfp);
+		empty = alloc_empty_sheaf(s, gfp, alloc_flags);
 		if (!empty)
 			return NULL;
 	}
@@ -5066,7 +5065,7 @@ kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
 
 	if (unlikely(size > s->sheaf_capacity)) {
 
-		sheaf = __alloc_empty_sheaf(s, gfp, size);
+		sheaf = __alloc_empty_sheaf(s, gfp, SLAB_ALLOC_DEFAULT, size);
 		if (!sheaf)
 			return NULL;
 
@@ -5111,7 +5110,7 @@ kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
 
 
 	if (!sheaf)
-		sheaf = alloc_empty_sheaf(s, gfp);
+		sheaf = alloc_empty_sheaf(s, gfp, SLAB_ALLOC_DEFAULT);
 
 	if (sheaf) {
 		sheaf->capacity = s->sheaf_capacity;
@@ -5396,7 +5395,7 @@ static void *__kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_f
 
 	VM_WARN_ON_ONCE(alloc_flags_allow_spinning(ac->alloc_flags));
 	VM_WARN_ON_ONCE(gfp_flags & ~(__GFP_ACCOUNT | __GFP_ZERO |
-			__GFP_NO_OBJ_EXT | __GFP_NOWARN | __GFP_NOMEMALLOC));
+				      __GFP_NOWARN | __GFP_NOMEMALLOC));
 
 	gfp_flags |= __GFP_NOWARN | __GFP_NOMEMALLOC;
 
@@ -5911,7 +5910,7 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!allow_spin)
 		return NULL;
 
-	empty = alloc_empty_sheaf(s, GFP_NOWAIT);
+	empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
 	if (empty)
 		goto got_empty;
 
@@ -6095,7 +6094,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 
 		local_unlock(&s->cpu_sheaves->lock);
 
-		empty = alloc_empty_sheaf(s, GFP_NOWAIT);
+		empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
 
 		if (!empty)
 			goto fail;
@@ -7640,7 +7639,7 @@ static int init_percpu_sheaves(struct kmem_cache *s)
 		if (!s->sheaf_capacity)
 			pcs->main = &bootstrap_sheaf;
 		else
-			pcs->main = alloc_empty_sheaf(s, GFP_KERNEL);
+			pcs->main = alloc_empty_sheaf(s, GFP_KERNEL, SLAB_ALLOC_DEFAULT);
 
 		if (!pcs->main)
 			return -ENOMEM;
@@ -8506,7 +8505,8 @@ static void __init bootstrap_cache_sheaves(struct kmem_cache *s)
 
 		pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
 
-		pcs->main = __alloc_empty_sheaf(s, GFP_KERNEL, capacity);
+		pcs->main = __alloc_empty_sheaf(s, GFP_KERNEL,
+				SLAB_ALLOC_DEFAULT, capacity);
 
 		if (!pcs->main) {
 			failed = true;
Signature · 1 line
2.54.0
maintainer Reviewed-by on v3 15/15
#37 lore raw

Re: [PATCH v3 15/15] mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves

Quoted 10 lines · Vlastimil Babka (SUSE)

Finish the switch away from __GFP_NO_OBJ_EXT by replacing it with SLAB_ALLOC_NO_RECURSE when allocating empty sheaves. Pass alloc_flags to [__]alloc_empty_sheaf(). Callers that can't be part of a recursive kmalloc() chain simply pass SLAB_ALLOC_DEFAULT. Use kmalloc_flags() instead of kzalloc() for allocating the sheaf.

With that we can finalize the removal the __GFP_NO_OBJ_EXT handling from obj_ext allocations as well, leaving only SLAB_ALLOC_NO_RECURSE in place.

Quoted 7 lines

This leaves __GFP_NO_OBJ_EXT with no users in slab, so stop allowing the flag in kmalloc_nolock().

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
Reviewed-by
#38 lore raw

Re: [PATCH v3 15/15] mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves

Quoted 17 lines · Harry Yoo
Quoted 13 lines · Vlastimil Babka (SUSE)

Finish the switch away from __GFP_NO_OBJ_EXT by replacing it with SLAB_ALLOC_NO_RECURSE when allocating empty sheaves. Pass alloc_flags to [__]alloc_empty_sheaf(). Callers that can't be part of a recursive kmalloc() chain simply pass SLAB_ALLOC_DEFAULT. Use kmalloc_flags() instead of kzalloc() for allocating the sheaf.

With that we can finalize the removal the __GFP_NO_OBJ_EXT handling from obj_ext allocations as well, leaving only SLAB_ALLOC_NO_RECURSE in place.

This leaves __GFP_NO_OBJ_EXT with no users in slab, so stop allowing the flag in kmalloc_nolock().

Quoted 7 lines
Quoted 4 lines

---

Looks good to me,

Quoted 4 lines

--
Cheers,
Harry / Hyeonggon