mm/slub: check pfmemalloc_match in slab_alloc_node fastpath
Subsystems (2)
PATCH 1/1 mm/slub: check pfmemalloc_match in slab_alloc_node fastpath
If current alloc context does not have __GFP_MEMALLOC in its gfpflags, then slab objects that were previously created with __GFP_MEMALLOC should not be given.
This criteria is well kept in slab alloc slowpath:
When gfpflags does not contain __GFP_MEMALLOC but if per-cpu slab page was allocated with __GFP_MEMALLOC, then allocator first deactivates per-cpu slab page and then again allocates new slab page with the current context's gfpflags.
However, this criteria is not checked in fastpath. It should also be checked in the fastpath, too.
--- mm/slub.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/slub.c +2 -1
@@ -3179,7 +3179,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l* there is a suitable cpu freelist. */ if (IS_ENABLED (CONFIG_PREEMPT_RT ) || -unlikely (!object || !slab || !node_match (slab, node))) { +unlikely (!object || !slab || !node_match (slab, node) || + !pfmemalloc_match (slab, gfpflags))) { object =__slab_alloc (s, gfpflags, node, addr, c); }else {void *next_object =get_freepointer_safe (s, object);
Signature · 1 line
2.25.1
Quoted 23 lines · Ohhoon Kwon
If current alloc context does not have
__GFP_MEMALLOCin its gfpflags, then slab objects that were previously created with__GFP_MEMALLOCshould not be given.This criteria is well kept in slab alloc slowpath:
When gfpflags does not contain__GFP_MEMALLOCbut if per-cpu slab page was allocated with__GFP_MEMALLOC, then allocator first deactivates per-cpu slab page and then again allocates new slab page with the current context's gfpflags.However, this criteria is not checked in fastpath. It should also be checked in the fastpath, too.
Signed-off-by Ohhoon Kwon <[email protected]>--- mm/slub.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/slub.c +0
-0
mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l · line 3186
Quoted 9 lines
mm/slub.c +2
-1@@ -3179,7 +3179,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l* there is a suitable cpu freelist. */ if (IS_ENABLED (CONFIG_PREEMPT_RT ) || -unlikely (!object || !slab || !node_match (slab, node))) { +unlikely (!object || !slab || !node_match (slab, node) || + !pfmemalloc_match (slab, gfpflags))) { object =__slab_alloc (s, gfpflags, node, addr, c); }else {void *next_object =get_freepointer_safe (s, object);
The missing pfmemalloc check in fastpath was intended.
pfmemalloc check in fast did exist in Mel's commit 072bb0aa5e0629 ("mm:
sl[au]b: add knowledge of PFMEMALLOC reserve pages").
But later removed by Christoph's commit 5091b74a95d4 ("mm: slub: optimise the SLUB fast path to avoid pfmemalloc checks").
Any thoughts?
Thanks!
Signature · 2 lines
Thanks, Hyeonggon
Oh I was not aware of those histories.
I checked the commits, and it seems it is better to leave it optimized(as it is now).
Thanks for your help.
Ohhoon Kwon.
Quoted 50 lines · Hyeonggon Yoo
Quoted 23 lines · Ohhoon Kwon
If current alloc context does not have
__GFP_MEMALLOCin its gfpflags, then slab objects that were previously created with__GFP_MEMALLOCshould not be given.This criteria is well kept in slab alloc slowpath:
When gfpflags does not contain__GFP_MEMALLOCbut if per-cpu slab page was allocated with__GFP_MEMALLOC, then allocator first deactivates per-cpu slab page and then again allocates new slab page with the current context's gfpflags.However, this criteria is not checked in fastpath. It should also be checked in the fastpath, too.
Signed-off-by Ohhoon Kwon <[email protected]>--- mm/slub.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/slub.c +0
-0mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l · line 3186
Quoted 9 lines
mm/slub.c +2
-1@@ -3179,7 +3179,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_l* there is a suitable cpu freelist. */ if (IS_ENABLED (CONFIG_PREEMPT_RT ) || -unlikely (!object || !slab || !node_match (slab, node))) { +unlikely (!object || !slab || !node_match (slab, node) || + !pfmemalloc_match (slab, gfpflags))) { object =__slab_alloc (s, gfpflags, node, addr, c); }else {void *next_object =get_freepointer_safe (s, object);The missing pfmemalloc check in fastpath was intended.
pfmemalloc check in fast did exist in Mel's commit
072bb0aa5e0629("mm:
sl[au]b: add knowledge of PFMEMALLOC reserve pages").But later removed by Christoph's commit
5091b74a95d4("mm: slub: optimise the SLUB fast path to avoid pfmemalloc checks").Any thoughts?
Thanks!
--
Thanks,
Hyeonggon