Skip to contentSkip to conversation

mm/slub: check pfmemalloc_match in slab_alloc_node fastpath

PATCH In review Ohhoon Kwon 2 participants 3 messages Apr 9, 2022 – Apr 10, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/slub: check pfmemalloc_match in slab_alloc_node fastpath

If current alloc context does not have __GFP_MEMALLOC in its gfpflags, then slab objects that were previously created with __GFP_MEMALLOC should not be given.

This criteria is well kept in slab alloc slowpath:
When gfpflags does not contain __GFP_MEMALLOC but if per-cpu slab page was allocated with __GFP_MEMALLOC, then allocator first deactivates per-cpu slab page and then again allocates new slab page with the current context's gfpflags.

However, this criteria is not checked in fastpath. It should also be checked in the fastpath, too.

---
 mm/slub.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/slub.c +2 -1
index 74d92aa4a3a2..c77cd548e106 100644
@@ -3179,7 +3179,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l
 	 * there is a suitable cpu freelist.
 	 */
 	if (IS_ENABLED(CONFIG_PREEMPT_RT) ||
-	    unlikely(!object || !slab || !node_match(slab, node))) {
+	    unlikely(!object || !slab || !node_match(slab, node) ||
+			!pfmemalloc_match(slab, gfpflags))) {
 		object = __slab_alloc(s, gfpflags, node, addr, c);
 	} else {
 		void *next_object = get_freepointer_safe(s, object);
Signature · 1 line
2.25.1
#1 lore raw
Quoted 23 lines · Ohhoon Kwon

If current alloc context does not have __GFP_MEMALLOC in its gfpflags, then slab objects that were previously created with __GFP_MEMALLOC should not be given.

This criteria is well kept in slab alloc slowpath:
When gfpflags does not contain __GFP_MEMALLOC but if per-cpu slab page was allocated with __GFP_MEMALLOC, then allocator first deactivates per-cpu slab page and then again allocates new slab page with the current context's gfpflags.

However, this criteria is not checked in fastpath. It should also be checked in the fastpath, too.

---
 mm/slub.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/slub.c +0 -0
index 74d92aa4a3a2..c77cd548e106 100644

mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l · line 3186

Quoted 9 lines
mm/slub.c +2 -1
@@ -3179,7 +3179,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l
 	 * there is a suitable cpu freelist.
 	 */
 	if (IS_ENABLED(CONFIG_PREEMPT_RT) ||
-	    unlikely(!object || !slab || !node_match(slab, node))) {
+	    unlikely(!object || !slab || !node_match(slab, node) ||
+			!pfmemalloc_match(slab, gfpflags))) {
 		object = __slab_alloc(s, gfpflags, node, addr, c);
 	} else {
 		void *next_object = get_freepointer_safe(s, object);

The missing pfmemalloc check in fastpath was intended.

pfmemalloc check in fast did exist in Mel's commit 072bb0aa5e0629 ("mm:
sl[au]b: add knowledge of PFMEMALLOC reserve pages").

But later removed by Christoph's commit 5091b74a95d4 ("mm: slub: optimise the SLUB fast path to avoid pfmemalloc checks").

Any thoughts?

Thanks!

Signature · 2 lines
Thanks,
Hyeonggon
author
#2 lore raw

Oh I was not aware of those histories.
I checked the commits, and it seems it is better to leave it optimized(as it is now).

Thanks for your help.
Ohhoon Kwon.

Quoted 50 lines · Hyeonggon Yoo
Quoted 23 lines · Ohhoon Kwon

If current alloc context does not have __GFP_MEMALLOC in its gfpflags, then slab objects that were previously created with __GFP_MEMALLOC should not be given.

This criteria is well kept in slab alloc slowpath:
When gfpflags does not contain __GFP_MEMALLOC but if per-cpu slab page was allocated with __GFP_MEMALLOC, then allocator first deactivates per-cpu slab page and then again allocates new slab page with the current context's gfpflags.

However, this criteria is not checked in fastpath. It should also be checked in the fastpath, too.

---
 mm/slub.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/slub.c +0 -0
index 74d92aa4a3a2..c77cd548e106 100644

mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l · line 3186

Quoted 9 lines
mm/slub.c +2 -1
@@ -3179,7 +3179,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l
       * there is a suitable cpu freelist.
       */
      if (IS_ENABLED(CONFIG_PREEMPT_RT) ||
-         unlikely(!object || !slab || !node_match(slab, node))) {
+         unlikely(!object || !slab || !node_match(slab, node) ||
+                     !pfmemalloc_match(slab, gfpflags))) {
              object = __slab_alloc(s, gfpflags, node, addr, c);
      } else {
              void *next_object = get_freepointer_safe(s, object);

The missing pfmemalloc check in fastpath was intended.

pfmemalloc check in fast did exist in Mel's commit 072bb0aa5e0629 ("mm:
sl[au]b: add knowledge of PFMEMALLOC reserve pages").

But later removed by Christoph's commit 5091b74a95d4 ("mm: slub: optimise the SLUB fast path to avoid pfmemalloc checks").

Any thoughts?

Thanks!

--
Thanks,
Hyeonggon