mm: slub: fix flush_cpu_slab()/__free_slab() invocations in task context.
Subsystems (2)
PATCH v3 1/1 mm: slub: fix flush_cpu_slab()/__free_slab() invocations in task context.
Commit 5a836bf6b09f ("mm: slub: move flush_cpu_slab() invocations __free_slab() invocations out of IRQ context") moved all flush_cpu_slab() invocations to the global workqueue to avoid a problem related with deactivate_slab()/__free_slab() being called from an IRQ context on PREEMPT_RT kernels.
When the flush_all_cpu_locked() function is called from a task context it may happen that a workqueue with WQ_MEM_RECLAIM bit set ends up flushing the global workqueue, this will cause a dependency issue.
workqueue: WQ_MEM_RECLAIM nvme-delete-wq:nvme_delete_ctrl_work [nvme_core]
is flushing !WQ_MEM_RECLAIM events:flush_cpu_slab
WARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637 check_flush_dependency+0x10a/0x120 Workqueue: nvme-delete-wq nvme_delete_ctrl_work [nvme_core] RIP: 0010:check_flush_dependency+0x10a/0x120[ 453.262125] Call Trace: __flush_work.isra.0+0xbf/0x220 ? __queue_work+0x1dc/0x420 flush_all_cpus_locked+0xfb/0x120 __kmem_cache_shutdown+0x2b/0x320 kmem_cache_destroy+0x49/0x100 bioset_exit+0x143/0x190 blk_release_queue+0xb9/0x100 kobject_cleanup+0x37/0x130 nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc] nvme_free_ctrl+0x1ac/0x2b0 [nvme_core]
Fix this bug by creating a workqueue for the flush operation with the WQ_MEM_RECLAIM bit set.
v2: Create a workqueue with WQ_MEM_RECLAIM
instead of trying to revert the changes.
v3: replace create_workqueue() with alloc_workqueue() and BUG_ON() with
WARN_ON()
--- mm/slub.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/slub.c +8 -1
@@ -310,6 +310,11 @@ static inline void stat(const struct kmem_cache *s, enum stat_item si)*/ static nodemask_t slab_nodes; +/* +* Workqueue used for flush_cpu_slab(). +*/ +static struct workqueue_struct *flushwq; +/******************************************************************** * Core slab cache functions *******************************************************************/
@@ -2730,7 +2735,7 @@ static void flush_all_cpus_locked(struct kmem_cache *s)INIT_WORK (&sfw->work, flush_cpu_slab); sfw->skip = false; sfw->s = s; -schedule_work_on (cpu, &sfw->work); +queue_work_on (cpu, flushwq, &sfw->work); }for_each_online_cpu (cpu) {
@@ -4858,6 +4863,8 @@ void __init kmem_cache_init(void)void __init kmem_cache_init_late (void ) { + flushwq =alloc_workqueue ("slub_flushwq" ,WQ_MEM_RECLAIM ,0 ); +WARN_ON (!flushwq); } struct kmem_cache *
Signature · 1 line
2.31.1
Quoted 30 lines · Maurizio Lombardi
Commit
5a836bf6b09f("mm: slub: moveflush_cpu_slab()invocations__free_slab()invocations out of IRQ context") moved allflush_cpu_slab()invocations to the global workqueue to avoid a problem related with deactivate_slab()/__free_slab() being called from an IRQ context onPREEMPT_RTkernels.When the
flush_all_cpu_locked()function is called from a task context it may happen that a workqueue withWQ_MEM_RECLAIMbit set ends up flushing the global workqueue, this will cause a dependency issue.workqueue:
WQ_MEM_RECLAIMnvme-delete-wq:nvme_delete_ctrl_work [nvme_core]
is flushing !WQ_MEM_RECLAIM events:flush_cpu_slabWARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637 check_flush_dependency+0x10a/0x120 Workqueue: nvme-delete-wq nvme_delete_ctrl_work [nvme_core] RIP: 0010:check_flush_dependency+0x10a/0x120[ 453.262125] Call Trace: __flush_work.isra.0+0xbf/0x220 ? __queue_work+0x1dc/0x420 flush_all_cpus_locked+0xfb/0x120 __kmem_cache_shutdown+0x2b/0x320 kmem_cache_destroy+0x49/0x100 bioset_exit+0x143/0x190 blk_release_queue+0xb9/0x100 kobject_cleanup+0x37/0x130 nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc] nvme_free_ctrl+0x1ac/0x2b0 [nvme_core]Fix this bug by creating a workqueue for the flush operation with the
WQ_MEM_RECLAIMbit set.
Quoted 7 lines
v2: Create a workqueue with
WQ_MEM_RECLAIM
instead of trying to revert the changes.v3: replace
create_workqueue()withalloc_workqueue()andBUG_ON()withWARN_ON()Signed-off-by Maurizio Lombardi <[email protected]>
Thanks, I added a Fixes: and Cc stable as AFAICS the warnings are not under a debugging config option, and it could bite somebody near OOM. Added to slab.git for-6.0/fixes and will include in pullrq this week.
Quoted 58 lines · Maurizio Lombardi
Commit
5a836bf6b09f("mm: slub: moveflush_cpu_slab()invocations__free_slab()invocations out of IRQ context") moved allflush_cpu_slab()invocations to the global workqueue to avoid a problem related with deactivate_slab()/__free_slab() being called from an IRQ context onPREEMPT_RTkernels.When the
flush_all_cpu_locked()function is called from a task context it may happen that a workqueue withWQ_MEM_RECLAIMbit set ends up flushing the global workqueue, this will cause a dependency issue.workqueue:
WQ_MEM_RECLAIMnvme-delete-wq:nvme_delete_ctrl_work [nvme_core]
is flushing !WQ_MEM_RECLAIM events:flush_cpu_slabWARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637 check_flush_dependency+0x10a/0x120 Workqueue: nvme-delete-wq nvme_delete_ctrl_work [nvme_core] RIP: 0010:check_flush_dependency+0x10a/0x120[ 453.262125] Call Trace: __flush_work.isra.0+0xbf/0x220 ? __queue_work+0x1dc/0x420 flush_all_cpus_locked+0xfb/0x120 __kmem_cache_shutdown+0x2b/0x320 kmem_cache_destroy+0x49/0x100 bioset_exit+0x143/0x190 blk_release_queue+0xb9/0x100 kobject_cleanup+0x37/0x130 nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc] nvme_free_ctrl+0x1ac/0x2b0 [nvme_core]Fix this bug by creating a workqueue for the flush operation with the
WQ_MEM_RECLAIMbit set.v2: Create a workqueue with
WQ_MEM_RECLAIM
instead of trying to revert the changes.v3: replace
create_workqueue()withalloc_workqueue()andBUG_ON()withWARN_ON()Signed-off-by Maurizio Lombardi <[email protected]>--- mm/slub.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)mm/slub.c +5
-0@@ -310,6 +310,11 @@ static inline void stat(const struct kmem_cache *s, enum stat_item si)*/ static nodemask_t slab_nodes; +/* +* Workqueue used for flush_cpu_slab(). +*/ +static struct workqueue_struct *flushwq; +/******************************************************************** * Core slab cache functions *******************************************************************/
mm/slub.c static void flush_all_cpus_locked(struct kmem_cache *s) · line 2738
Quoted 5 lines
mm/slub.c +1
-1@@ -2730,7 +2735,7 @@ static void flush_all_cpus_locked(struct kmem_cache *s)INIT_WORK (&sfw->work, flush_cpu_slab); sfw->skip = false; sfw->s = s; -schedule_work_on (cpu, &sfw->work); +queue_work_on (cpu, flushwq, &sfw->work);
Hi. what happens here if flushwq failed?
I think avoiding BUG_ON() makes sense,
but shouldn't we have fallback method?
Quoted 17 lines
mm/slub.c +2
-0}for_each_online_cpu (cpu) {@@ -4858,6 +4863,8 @@ void __init kmem_cache_init(void)void __init kmem_cache_init_late (void ) { + flushwq =alloc_workqueue ("slub_flushwq" ,WQ_MEM_RECLAIM ,0 ); +WARN_ON (!flushwq); }struct
kmem_cache*Signature · 1 line
2.31.1
Signature · 2 lines
Thanks, Hyeonggon
Quoted 11 lines · Hyeonggon Yoo
mm/slub.c static void flush_all_cpus_locked(struct kmem_cache *s) · line 2738
Quoted 6 lines
mm/slub.c +1
-1@@ -2730,7 +2735,7 @@ static void flush_all_cpus_locked(struct kmem_cache *s)INIT_WORK (&sfw->work, flush_cpu_slab); sfw->skip = false; sfw->s = s; -schedule_work_on (cpu, &sfw->work); +queue_work_on (cpu, flushwq, &sfw->work);Hi. what happens here if flushwq failed?
I think avoiding
BUG_ON()makes sense,
but shouldn't we have fallback method?
You get an output to act on and fix. The point is that it shouldn't have happen in the first place. With the bug_on() that early, chances are that you never see anything but a blank screen. So with the warn_on you get probably to see the warn_on before you get here.
Sebastian
Quoted 13 lines · Sebastian Andrzej Siewior
Quoted 11 lines · Hyeonggon Yoo
mm/slub.c static void flush_all_cpus_locked(struct kmem_cache *s) · line 2738
Quoted 6 lines
mm/slub.c +1
-1@@ -2730,7 +2735,7 @@ static void flush_all_cpus_locked(struct kmem_cache *s)INIT_WORK (&sfw->work, flush_cpu_slab); sfw->skip = false; sfw->s = s; -schedule_work_on (cpu, &sfw->work); +queue_work_on (cpu, flushwq, &sfw->work);Hi. what happens here if flushwq failed?
I think avoiding
BUG_ON()makes sense,
but shouldn't we have fallback method?
Quoted 6 lines
You get an output to act on and fix. The point is that it shouldn't have happen in the first place. With the
bug_on()that early, chances are that you never see anything but a blank screen. So with thewarn_onyou get probably to see thewarn_onbefore you get here.Sebastian
Thank you for kind explanation.
Makes sense!
Signature · 2 lines
Thanks, Hyeonggon
Quoted 78 lines · Maurizio Lombardi
Commit
5a836bf6b09f("mm: slub: moveflush_cpu_slab()invocations__free_slab()invocations out of IRQ context") moved allflush_cpu_slab()invocations to the global workqueue to avoid a problem related with deactivate_slab()/__free_slab() being called from an IRQ context onPREEMPT_RTkernels.When the
flush_all_cpu_locked()function is called from a task context it may happen that a workqueue withWQ_MEM_RECLAIMbit set ends up flushing the global workqueue, this will cause a dependency issue.workqueue:
WQ_MEM_RECLAIMnvme-delete-wq:nvme_delete_ctrl_work [nvme_core]
is flushing !WQ_MEM_RECLAIM events:flush_cpu_slabWARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637 check_flush_dependency+0x10a/0x120 Workqueue: nvme-delete-wq nvme_delete_ctrl_work [nvme_core] RIP: 0010:check_flush_dependency+0x10a/0x120[ 453.262125] Call Trace: __flush_work.isra.0+0xbf/0x220 ? __queue_work+0x1dc/0x420 flush_all_cpus_locked+0xfb/0x120 __kmem_cache_shutdown+0x2b/0x320 kmem_cache_destroy+0x49/0x100 bioset_exit+0x143/0x190 blk_release_queue+0xb9/0x100 kobject_cleanup+0x37/0x130 nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc] nvme_free_ctrl+0x1ac/0x2b0 [nvme_core]Fix this bug by creating a workqueue for the flush operation with the
WQ_MEM_RECLAIMbit set.v2: Create a workqueue with
WQ_MEM_RECLAIM
instead of trying to revert the changes.v3: replace
create_workqueue()withalloc_workqueue()andBUG_ON()withWARN_ON()Signed-off-by Maurizio Lombardi <[email protected]>--- mm/slub.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)mm/slub.c +8
-1@@ -310,6 +310,11 @@ static inline void stat(const struct kmem_cache *s, enum stat_item si)*/ static nodemask_t slab_nodes; +/* +* Workqueue used for flush_cpu_slab(). +*/ +static struct workqueue_struct *flushwq; +/******************************************************************** * Core slab cache functions *******************************************************************/ @@ -2730,7 +2735,7 @@ static void flush_all_cpus_locked(struct kmem_cache *s)INIT_WORK (&sfw->work, flush_cpu_slab); sfw->skip = false; sfw->s = s; -schedule_work_on (cpu, &sfw->work); +queue_work_on (cpu, flushwq, &sfw->work); }for_each_online_cpu (cpu) {@@ -4858,6 +4863,8 @@ void __init kmem_cache_init(void)void __init kmem_cache_init_late (void ) { + flushwq =alloc_workqueue ("slub_flushwq" ,WQ_MEM_RECLAIM ,0 ); +WARN_ON (!flushwq); } struct kmem_cache *Signature · 1 line
2.31.1
Looks good to me.
Signature · 2 lines
Thanks, Hyeonggon