stackdepot, kasan, workqueue: Avoid expanding stackdepot slabs when holding raw_spin_lock
Subsystems (5)
cover stackdepot, kasan, workqueue: Avoid expanding stackdepot slabs when holding raw_spin_lock
Shuah Khan reported [1]:
| When CONFIG_PROVE_RAW_LOCK_NESTING=y and CONFIG_KASAN are enabled, | kasan_record_aux_stack() runs into "BUG: Invalid wait context" when | it tries to allocate memory attempting to acquire spinlock in page | allocation code while holding workqueue pool raw_spinlock. | | There are several instances of this problem when block layer tries | to __queue_work(). Call trace from one of these instances is below: | | kblockd_mod_delayed_work_on() | mod_delayed_work_on() | __queue_delayed_work() | __queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) | insert_work() | kasan_record_aux_stack() | kasan_save_stack() | stack_depot_save() | alloc_pages() | __alloc_pages() | get_page_from_freelist() | rm_queue() | rm_queue_pcplist() | local_lock_irqsave(&pagesets.lock, flags); | [ BUG: Invalid wait context triggered ]
[1] https://lkml.kernel.org/r/[email protected]
PROVE_RAW_LOCK_NESTING is pointing out that (on RT kernels) the locking rules are being violated. More generally, memory is being allocated from a non-preemptive context (raw_spin_lock'd c-s) where it is not allowed.
To properly fix this, we must prevent stackdepot from replenishing its "stack slab" pool if memory allocations cannot be done in the current
context: it's a bug to use either GFP_ATOMIC nor GFP_NOWAIT in certain non-preemptive contexts, including raw_spin_locks (see gfp.h and ab00db216c9c7).
The only downside is that saving a stack trace may fail if: stackdepot runs out of space AND the same stack trace has not been recorded before. I expect this to be unlikely, and a simple experiment (boot the kernel) didn't result in any failure to record stack trace from insert_work().
The series includes a few minor fixes to stackdepot that I noticed in preparing the series. It then introduces __stack_depot_save(), which exposes the option to force stackdepot to not allocate any memory. Finally, KASAN is changed to use the new stackdepot interface and provide kasan_record_aux_stack_noalloc(), which is then used by workqueue code.
Marco Elver (6):
lib/stackdepot: include gfp.h
lib/stackdepot: remove unused function argument lib/stackdepot: introduce __stack_depot_save()
kasan: common: provide can_alloc in kasan_save_stack()
kasan: generic: introduce kasan_record_aux_stack_noalloc() workqueue, kasan: avoid alloc_pages() when recording stack
include/linux/kasan.h | 2 ++ include/linux/stackdepot.h | 6 +++++ kernel/workqueue.c | 2 +- lib/stackdepot.c | 51 ++++++++++++++++++++++++++++++-------- mm/kasan/common.c | 6 ++--- mm/kasan/generic.c | 14 +++++++++-- mm/kasan/kasan.h | 2 +- 7 files changed, 65 insertions(+), 18 deletions(-)
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
PATCH 1/6 lib/stackdepot: include gfp.h
<linux/stackdepot.h> refers to gfp_t, but doesn't include gfp.h.
Fix it by including <linux/gfp.h>.
--- include/linux/stackdepot.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/stackdepot.h +2 -0
@@ -11,6 +11,8 @@#ifndef _LINUX_STACKDEPOT_H#define _LINUX_STACKDEPOT_H +#include <linux/gfp.h> +typedef u32 depot_stack_handle_t ; depot_stack_handle_t stack_depot_save(unsigned long *entries,
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
Re: [PATCH 1/6] lib/stackdepot: include gfp.h
Quoted 6 lines · Marco Elver
<
linux/stackdepot.h> refers togfp_t, but doesn't includegfp.h.Fix it by including <
linux/gfp.h>.Signed-off-by Marco Elver <[email protected]>
Quoted 20 lines
--- include/linux/stackdepot.h | 2 ++ 1 file changed, 2 insertions(+)include/linux/stackdepot.h +2
-1@@ -11,6 +11,8 @@#ifndef _LINUX_STACKDEPOT_H#define _LINUX_STACKDEPOT_H +#include <linux/gfp.h> +typedef u32 depot_stack_handle_t ; depot_stack_handle_t stack_depot_save(unsigned long *entries, --2.33.0.153.gba50c8fa24-goog
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
PATCH 2/6 lib/stackdepot: remove unused function argument
In reply to Marco Elveralloc_flags in depot_alloc_stack() is no longer used; remove it.
--- lib/stackdepot.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
lib/stackdepot.c +4 -5
@@ -102,8 +102,8 @@ static bool init_stack_slab(void **prealloc) }/* Allocation of a new stack in raw storage */ -static struct stack_record *depot_alloc_stack (unsigned long *entries,int size, -u32 hash,void **prealloc,gfp_t alloc_flags) +static struct stack_record * +depot_alloc_stack (unsigned long *entries,int size,u32 hash,void **prealloc) {struct stack_record *stack;size_t required_size =struct_size (stack, entries, size);
@@ -309,9 +309,8 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries, found =find_stack (*bucket, entries, nr_entries, hash);if (!found) { -struct stack_record *new = -depot_alloc_stack (entries, nr_entries, - hash, &prealloc, alloc_flags); +struct stack_record *new =depot_alloc_stack (entries, nr_entries, hash, &prealloc); +if (new) { new->next = *bucket;/*
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
Re: [PATCH 2/6] lib/stackdepot: remove unused function argument
Quoted 4 lines · Marco Elver
alloc_flagsindepot_alloc_stack()is no longer used; remove it.Signed-off-by Marco Elver <[email protected]>
Quoted 34 lines
--- lib/stackdepot.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)lib/stackdepot.c +4
-6@@ -102,8 +102,8 @@ static bool init_stack_slab(void **prealloc) }/* Allocation of a new stack in raw storage */ -static struct stack_record *depot_alloc_stack (unsigned long *entries,int size, -u32 hash,void **prealloc,gfp_t alloc_flags) +static struct stack_record * +depot_alloc_stack (unsigned long *entries,int size,u32 hash,void **prealloc) {struct stack_record *stack;size_t required_size =struct_size (stack, entries, size);@@ -309,9 +309,8 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries, found =find_stack (*bucket, entries, nr_entries, hash);if (!found) { -struct stack_record *new = -depot_alloc_stack (entries, nr_entries, - hash, &prealloc, alloc_flags); +struct stack_record *new =depot_alloc_stack (entries, nr_entries, hash, &prealloc); +if (new) { new->next = *bucket;/* -- 2.33.0.153.gba50c8fa24-goog
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
PATCH 3/6 lib/stackdepot: introduce __stack_depot_save()
In reply to Marco ElverAdd __stack_depot_save(), which provides more fine-grained control over stackdepot's memory allocation behaviour, in case stackdepot runs out of "stack slabs".
Normally stackdepot uses alloc_pages() in case it runs out of space; passing can_alloc==false to __stack_depot_save() prohibits this, at the cost of more likely failure to record a stack trace.
--- include/linux/stackdepot.h | 4 ++++ lib/stackdepot.c | 42 ++++++++++++++++++++++++++++++++------ 2 files changed, 40 insertions(+), 6 deletions(-)
include/linux/stackdepot.h +4 -0
@@ -15,6 +15,10 @@typedef u32 depot_stack_handle_t ; +depot_stack_handle_t __stack_depot_save (unsigned long *entries, +unsigned int nr_entries, +gfp_t gfp_flags,bool can_alloc); +depot_stack_handle_t stack_depot_save (unsigned long *entries,unsigned int nr_entries,gfp_t gfp_flags);
lib/stackdepot.c +36 -6
66 lines not on this page · raw message
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
Re: [PATCH 3/6] lib/stackdepot: introduce __stack_depot_save()
Quoted 8 lines · Marco Elver
lib/stackdepot.c +0
-0@@ -339,6 +350,25 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries, fast_exit:return retval; }
lib/stackdepot.c depot_stack_handle_t stack_depot_save(unsigned long *entries, · line 362
Quoted 10 lines
lib/stackdepot.c +10
-0+EXPORT_SYMBOL_GPL (__stack_depot_save); + +/** +* stack_depot_save - Save a stack trace from an array +* +* @entries: Pointer to storage array +* @nr_entries: Size of the storage array +* @alloc_flags: Allocation gfp flags +* +* Context: Contexts where allocations via alloc_pages() are allowed.
Could we add here something like (see __stack_depot_save() for details) since it has more verbose.
Sebastian
PATCH 4/6 kasan: common: provide can_alloc in kasan_save_stack()
In reply to Marco ElverAdd another argument, can_alloc, to kasan_save_stack() which is passed as-is to __stack_depot_save().
No functional change intended.
--- mm/kasan/common.c | 6 +++--- mm/kasan/generic.c | 2 +- mm/kasan/kasan.h | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)
mm/kasan/common.c +3 -3
@@ -30,20 +30,20 @@#include "kasan.h" #include "../slab.h" -depot_stack_handle_t kasan_save_stack (gfp_t flags) +depot_stack_handle_t kasan_save_stack (gfp_t flags,bool can_alloc) {unsigned long entries[KASAN_STACK_DEPTH ];unsigned int nr_entries; nr_entries =stack_trace_save (entries,ARRAY_SIZE (entries),0 ); nr_entries =filter_irq_stacks (entries, nr_entries); -return stack_depot_save (entries, nr_entries, flags); +return __stack_depot_save (entries, nr_entries, flags, can_alloc); }void kasan_set_track (struct kasan_track *track,gfp_t flags) { track->pid = current->pid; - track->stack =kasan_save_stack (flags); + track->stack =kasan_save_stack (flags, true); }#if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS )
mm/kasan/generic.c +1 -1
@@ -345,7 +345,7 @@ void kasan_record_aux_stack(void *addr)return ; alloc_meta->aux_stack[1 ] = alloc_meta->aux_stack[0 ]; - alloc_meta->aux_stack[0 ] =kasan_save_stack (GFP_NOWAIT ); + alloc_meta->aux_stack[0 ] =kasan_save_stack (GFP_NOWAIT , true); } void kasan_set_free_info(struct kmem_cache *cache,
mm/kasan/kasan.h +1 -1
@@ -260,7 +260,7 @@ void kasan_report_invalid_free(void *object, unsigned long ip);struct page *kasan_addr_to_page (const void *addr); -depot_stack_handle_t kasan_save_stack (gfp_t flags); +depot_stack_handle_t kasan_save_stack (gfp_t flags,bool can_alloc);void kasan_set_track (struct kasan_track *track,gfp_t flags);void kasan_set_free_info (struct kmem_cache *cache,void *object,u8 tag); struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
PATCH 5/6 kasan: generic: introduce kasan_record_aux_stack_noalloc()
In reply to Marco ElverIntroduce a variant of kasan_record_aux_stack() that does not do any memory allocation through stackdepot. This will permit using it in contexts that cannot allocate any memory.
--- include/linux/kasan.h | 2 ++ mm/kasan/generic.c | 14 ++++++++++++-- 2 files changed, 14 insertions(+), 2 deletions(-)
include/linux/kasan.h +2 -0
@@ -370,12 +370,14 @@ static inline void kasan_unpoison_task_stack(struct task_struct *task) {}void kasan_cache_shrink (struct kmem_cache *cache);void kasan_cache_shutdown (struct kmem_cache *cache);void kasan_record_aux_stack (void *ptr); +void kasan_record_aux_stack_noalloc (void *ptr);#else /* CONFIG_KASAN_GENERIC */ static inline void kasan_cache_shrink (struct kmem_cache *cache) {}static inline void kasan_cache_shutdown (struct kmem_cache *cache) {}static inline void kasan_record_aux_stack (void *ptr) {} +static inline void kasan_record_aux_stack_noalloc (void *ptr) {}#endif /* CONFIG_KASAN_GENERIC */
mm/kasan/generic.c +12 -2
@@ -328,7 +328,7 @@ DEFINE_ASAN_SET_SHADOW(f3);DEFINE_ASAN_SET_SHADOW (f5);DEFINE_ASAN_SET_SHADOW (f8); -void kasan_record_aux_stack (void *addr) +static void __kasan_record_aux_stack (void *addr,bool can_alloc) {struct page *page =kasan_addr_to_page (addr);struct kmem_cache *cache;
@@ -345,7 +345,17 @@ void kasan_record_aux_stack(void *addr)return ; alloc_meta->aux_stack[1 ] = alloc_meta->aux_stack[0 ]; - alloc_meta->aux_stack[0 ] =kasan_save_stack (GFP_NOWAIT , true); + alloc_meta->aux_stack[0 ] =kasan_save_stack (GFP_NOWAIT , can_alloc); +} + +void kasan_record_aux_stack (void *addr) +{ +return __kasan_record_aux_stack (addr, true); +} + +void kasan_record_aux_stack_noalloc (void *addr) +{ +return __kasan_record_aux_stack (addr, false); } void kasan_set_free_info(struct kmem_cache *cache,
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
PATCH 6/6 workqueue, kasan: avoid alloc_pages() when recording stack
In reply to Marco ElverShuah Khan reported:
| When CONFIG_PROVE_RAW_LOCK_NESTING=y and CONFIG_KASAN are enabled, | kasan_record_aux_stack() runs into "BUG: Invalid wait context" when | it tries to allocate memory attempting to acquire spinlock in page | allocation code while holding workqueue pool raw_spinlock. | | There are several instances of this problem when block layer tries | to __queue_work(). Call trace from one of these instances is below: | | kblockd_mod_delayed_work_on() | mod_delayed_work_on() | __queue_delayed_work() | __queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) | insert_work() | kasan_record_aux_stack() | kasan_save_stack() | stack_depot_save() | alloc_pages() | __alloc_pages() | get_page_from_freelist() | rm_queue() | rm_queue_pcplist() | local_lock_irqsave(&pagesets.lock, flags); | [ BUG: Invalid wait context triggered ]
The default kasan_record_aux_stack() calls stack_depot_save() with GFP_NOWAIT, which in turn can then call alloc_pages(GFP_NOWAIT, ...). In general, however, it is not even possible to use either GFP_ATOMIC nor GFP_NOWAIT in certain non-preemptive contexts, including raw_spin_locks (see gfp.h and ab00db216c9c7).
Fix it by instructing stackdepot to not expand stack storage via alloc_pages() in case it runs out by using kasan_record_aux_stack_noalloc().
While there is an increased risk of failing to insert the stack trace, this is typically unlikely, especially if the same insertion had already succeeded previously (stack depot hit). For frequent calls from the same location, it therefore becomes extremely unlikely that kasan_record_aux_stack_noalloc() fails.
--- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/workqueue.c +1 -1
@@ -1329,7 +1329,7 @@ static void insert_work(struct pool_workqueue *pwq, struct work_struct *work,struct worker_pool *pool = pwq->pool;/* record the work call stack in order to print it in KASAN reports */ -kasan_record_aux_stack (work); +kasan_record_aux_stack_noalloc (work);/* we own @work, set data and link */ set_work_pwq (work, pwq, extra_flags);
Signature · 1 line
2.33.0.153.gba50c8fa24-goog
Re: [PATCH 6/6] workqueue, kasan: avoid alloc_pages() when recording stack
Quoted 41 lines · Marco Elver
Shuah Khan reported:
| When CONFIG_PROVE_RAW_LOCK_NESTING=y and CONFIG_KASAN are enabled, | kasan_record_aux_stack() runs into "BUG: Invalid wait context" when | it tries to allocate memory attempting to acquire spinlock in page | allocation code while holding workqueue pool raw_spinlock. | | There are several instances of this problem when block layer tries | to __queue_work(). Call trace from one of these instances is below: | | kblockd_mod_delayed_work_on() | mod_delayed_work_on() | __queue_delayed_work() | __queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) | insert_work() | kasan_record_aux_stack() | kasan_save_stack() | stack_depot_save() | alloc_pages() | __alloc_pages() | get_page_from_freelist() | rm_queue() | rm_queue_pcplist() | local_lock_irqsave(&pagesets.lock, flags); | [ BUG: Invalid wait context triggered ]The default
kasan_record_aux_stack()callsstack_depot_save()withGFP_NOWAIT, which in turn can then callalloc_pages(GFP_NOWAIT, ...). In general, however, it is not even possible to use eitherGFP_ATOMICnorGFP_NOWAITin certain non-preemptive contexts, includingraw_spin_locks(seegfp.handab00db216c9c7).Fix it by instructing stackdepot to not expand stack storage via
alloc_pages()in case it runs out by usingkasan_record_aux_stack_noalloc().While there is an increased risk of failing to insert the stack trace, this is typically unlikely, especially if the same insertion had already succeeded previously (stack depot hit). For frequent calls from the same location, it therefore becomes extremely unlikely that
kasan_record_aux_stack_noalloc()fails.
Quoted 3 lines
Reported-by Shuah Khan <[email protected]>Signed-off-by Marco Elver <[email protected]>
Thanks.
Signature · 1 line
tejun
[+Cc: Thomas, Sebastian]
Sorry, forgot to Cc you... :-/
Quoted 71 lines · Marco Elver
Shuah Khan reported [1]:
| When CONFIG_PROVE_RAW_LOCK_NESTING=y and CONFIG_KASAN are enabled, | kasan_record_aux_stack() runs into "BUG: Invalid wait context" when | it tries to allocate memory attempting to acquire spinlock in page | allocation code while holding workqueue pool raw_spinlock. | | There are several instances of this problem when block layer tries | to __queue_work(). Call trace from one of these instances is below: | | kblockd_mod_delayed_work_on() | mod_delayed_work_on() | __queue_delayed_work() | __queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) | insert_work() | kasan_record_aux_stack() | kasan_save_stack() | stack_depot_save() | alloc_pages() | __alloc_pages() | get_page_from_freelist() | rm_queue() | rm_queue_pcplist() | local_lock_irqsave(&pagesets.lock, flags); | [ BUG: Invalid wait context triggered ][1] https://lkml.kernel.org/r/[email protected]
PROVE_RAW_LOCK_NESTINGis pointing out that (on RT kernels) the locking rules are being violated. More generally, memory is being allocated from a non-preemptive context (raw_spin_lock'd c-s) where it is not allowed.To properly fix this, we must prevent stackdepot from replenishing its "stack slab" pool if memory allocations cannot be done in the current
context: it's a bug to use eitherGFP_ATOMICnorGFP_NOWAITin certain non-preemptive contexts, includingraw_spin_locks(seegfp.handab00db216c9c7).The only downside is that saving a stack trace may fail if: stackdepot runs out of space AND the same stack trace has not been recorded before. I expect this to be unlikely, and a simple experiment (boot the kernel) didn't result in any failure to record stack trace from
insert_work().The series includes a few minor fixes to stackdepot that I noticed in preparing the series. It then introduces
__stack_depot_save(), which exposes the option to force stackdepot to not allocate any memory. Finally, KASAN is changed to use the new stackdepot interface and providekasan_record_aux_stack_noalloc(), which is then used by workqueue code.Marco Elver (6):
lib/stackdepot: includegfp.h
lib/stackdepot: remove unused function argument lib/stackdepot: introduce__stack_depot_save()
kasan: common: providecan_allocinkasan_save_stack()
kasan: generic: introducekasan_record_aux_stack_noalloc()workqueue, kasan: avoidalloc_pages()when recording stackinclude/linux/kasan.h | 2 ++ include/linux/stackdepot.h | 6 +++++ kernel/workqueue.c | 2 +- lib/stackdepot.c | 51 ++++++++++++++++++++++++++++++-------- mm/kasan/common.c | 6 ++--- mm/kasan/generic.c | 14 +++++++++-- mm/kasan/kasan.h | 2 +- 7 files changed, 65 insertions(+), 18 deletions(-)--
2.33.0.153.gba50c8fa24-goog
Quoted 58 lines · Marco Elver
Shuah Khan reported [1]:
| When CONFIG_PROVE_RAW_LOCK_NESTING=y and CONFIG_KASAN are enabled, | kasan_record_aux_stack() runs into "BUG: Invalid wait context" when | it tries to allocate memory attempting to acquire spinlock in page | allocation code while holding workqueue pool raw_spinlock. | | There are several instances of this problem when block layer tries | to __queue_work(). Call trace from one of these instances is below: | | kblockd_mod_delayed_work_on() | mod_delayed_work_on() | __queue_delayed_work() | __queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) | insert_work() | kasan_record_aux_stack() | kasan_save_stack() | stack_depot_save() | alloc_pages() | __alloc_pages() | get_page_from_freelist() | rm_queue() | rm_queue_pcplist() | local_lock_irqsave(&pagesets.lock, flags); | [ BUG: Invalid wait context triggered ][1] https://lkml.kernel.org/r/[email protected]
PROVE_RAW_LOCK_NESTINGis pointing out that (on RT kernels) the locking rules are being violated. More generally, memory is being allocated from a non-preemptive context (raw_spin_lock'd c-s) where it is not allowed.To properly fix this, we must prevent stackdepot from replenishing its "stack slab" pool if memory allocations cannot be done in the current
context: it's a bug to use eitherGFP_ATOMICnorGFP_NOWAITin certain non-preemptive contexts, includingraw_spin_locks(seegfp.handab00db216c9c7).The only downside is that saving a stack trace may fail if: stackdepot runs out of space AND the same stack trace has not been recorded before. I expect this to be unlikely, and a simple experiment (boot the kernel) didn't result in any failure to record stack trace from
insert_work().The series includes a few minor fixes to stackdepot that I noticed in preparing the series. It then introduces
__stack_depot_save(), which exposes the option to force stackdepot to not allocate any memory. Finally, KASAN is changed to use the new stackdepot interface and providekasan_record_aux_stack_noalloc(), which is then used by workqueue code.Marco Elver (6):
lib/stackdepot: includegfp.h
lib/stackdepot: remove unused function argument lib/stackdepot: introduce__stack_depot_save()
kasan: common: providecan_allocinkasan_save_stack()
kasan: generic: introducekasan_record_aux_stack_noalloc()workqueue, kasan: avoidalloc_pages()when recording stack
Quoted 9 lines
include/linux/kasan.h | 2 ++ include/linux/stackdepot.h | 6 +++++ kernel/workqueue.c | 2 +- lib/stackdepot.c | 51 ++++++++++++++++++++++++++++++-------- mm/kasan/common.c | 6 ++--- mm/kasan/generic.c | 14 +++++++++-- mm/kasan/kasan.h | 2 +- 7 files changed, 65 insertions(+), 18 deletions(-)
Thank you. Tested all the 6 patches in this series on Linux 5.14. This problem exists in 5.13 and needs to be marked for both 5.14 and 5.13 stable releases.
Here is my
thanks,
-- Shuah
Quoted 70 lines · Shuah Khan
Quoted 59 lines · Marco Elver
Shuah Khan reported [1]:
| When CONFIG_PROVE_RAW_LOCK_NESTING=y and CONFIG_KASAN are enabled, | kasan_record_aux_stack() runs into "BUG: Invalid wait context" when | it tries to allocate memory attempting to acquire spinlock in page | allocation code while holding workqueue pool raw_spinlock. | | There are several instances of this problem when block layer tries | to __queue_work(). Call trace from one of these instances is below: | | kblockd_mod_delayed_work_on() | mod_delayed_work_on() | __queue_delayed_work() | __queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) | insert_work() | kasan_record_aux_stack() | kasan_save_stack() | stack_depot_save() | alloc_pages() | __alloc_pages() | get_page_from_freelist() | rm_queue() | rm_queue_pcplist() | local_lock_irqsave(&pagesets.lock, flags); | [ BUG: Invalid wait context triggered ][1]
https://lkml.kernel.org/r/[email protected]
PROVE_RAW_LOCK_NESTINGis pointing out that (on RT kernels) the locking rules are being violated. More generally, memory is being allocated from a non-preemptive context (raw_spin_lock'd c-s) where it is not allowed.To properly fix this, we must prevent stackdepot from replenishing its "stack slab" pool if memory allocations cannot be done in the current
context: it's a bug to use eitherGFP_ATOMICnorGFP_NOWAITin certain non-preemptive contexts, includingraw_spin_locks(seegfp.handab00db216c9c7).The only downside is that saving a stack trace may fail if: stackdepot runs out of space AND the same stack trace has not been recorded before. I expect this to be unlikely, and a simple experiment (boot the kernel) didn't result in any failure to record stack trace from
insert_work().The series includes a few minor fixes to stackdepot that I noticed in preparing the series. It then introduces
__stack_depot_save(), which exposes the option to force stackdepot to not allocate any memory. Finally, KASAN is changed to use the new stackdepot interface and providekasan_record_aux_stack_noalloc(), which is then used by workqueue code.Marco Elver (6):
lib/stackdepot: includegfp.h
lib/stackdepot: remove unused function argument lib/stackdepot: introduce__stack_depot_save()
kasan: common: providecan_allocinkasan_save_stack()
kasan: generic: introducekasan_record_aux_stack_noalloc()workqueue, kasan: avoidalloc_pages()when recording stackQuoted 9 lines
include/linux/kasan.h | 2 ++
include/linux/stackdepot.h| 6 +++++
kernel/workqueue.c | 2 +-
lib/stackdepot.c | 51 ++++++++++++++++++++++++++++++-------- mm/kasan/common.c | 6 ++---
mm/kasan/generic.c | 14 +++++++++-- mm/kasan/kasan.h | 2 +-7 files changed, 65 insertions(+), 18 deletions(-)
Quoted 2 lines
Thank you. Tested all the 6 patches in this series on Linux 5.14. This problem exists in 5.13 and needs to be marked for both 5.14 and 5.13 stable releases.
I think if this problem manifests only with CONFIG_PROVE_RAW_LOCK_NESTING then it shouldn't be backported to stable. CONFIG_PROVE_RAW_LOCK_NESTING is an experimental/development option to earlier discover what will collide with RT lock semantics, without needing the full RT tree.
Thus, good to fix going forward, but not necessary to stable backport.
Quoted 7 lines
Here is my
Tested-by Shuah Khan <[email protected]>thanks,
-- Shuah
Sebastian Andrzej Siewior
Acked-byQuoted 8 lines · Vlastimil Babka
Quoted 2 lines
Thank you. Tested all the 6 patches in this series on Linux 5.14. This problem exists in 5.13 and needs to be marked for both 5.14 and 5.13 stable releases.
I think if this problem manifests only with
CONFIG_PROVE_RAW_LOCK_NESTINGthen it shouldn't be backported to stable.CONFIG_PROVE_RAW_LOCK_NESTINGis an experimental/development option to earlier discover what will collide with RT lock semantics, without needing the full RT tree.
Thus, good to fix going forward, but not necessary to stable backport.
for the series. Thank you.
As for the backport I agree here with Vlastimil.
I pulled it into my RT tree for some testing and it looked good. I had to
kernel/rcu/tree.c +1 -1
@@ -3030,7 +3030,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) head->func = func; head->next =NULL ;local_irq_save (flags); -kasan_record_aux_stack (head); +kasan_record_aux_stack_noalloc (head); rdp =this_cpu_ptr (&rcu_data);/* Add the callback to our list. */
We could move kasan_record_aux_stack() before that local_irq_save() but then call_rcu() can be called preempt-disabled section so we would have the same problem.
The second warning came from kasan_quarantine_remove_cache(). At the end per_cpu_remove_cache() -> qlist_free_all() will free memory with disabled interrupts (due to that smp-function call).
Moving it to kworker would solve the problem. I don't mind keeping that smp_function call assuming that it is all debug-code and it increases overall latency anyway. But then could we maybe move all those objects to a single list which freed after on_each_cpu()?
Otherwise I haven't seen any new warnings showing up with KASAN enabled.
Sebastian
Quoted 4 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Vlastimil Babka
Quoted 2 lines
Thank you. Tested all the 6 patches in this series on Linux 5.14. This problem exists in 5.13 and needs to be marked for both 5.14 and 5.13 stable releases.
Quoted 8 lines
Quoted 5 lines
I think if this problem manifests only with
CONFIG_PROVE_RAW_LOCK_NESTINGthen it shouldn't be backported to stable.CONFIG_PROVE_RAW_LOCK_NESTINGis an experimental/development option to earlier discover what will collide with RT lock semantics, without needing the full RT tree.
Thus, good to fix going forward, but not necessary to stable backport.for the series. Thank you.
Thank you. I'll send v2 with Acks/Tested-by added and the comment addition you suggested.
Quoted 20 lines
As for the backport I agree here with Vlastimil.
I pulled it into my RT tree for some testing and it looked good. I had to
kernel/rcu/tree.c +1
-1@@ -3030,7 +3030,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) head->func = func; head->next =NULL ;local_irq_save (flags); -kasan_record_aux_stack (head); +kasan_record_aux_stack_noalloc (head); rdp =this_cpu_ptr (&rcu_data);/* Add the callback to our list. */ We could move
kasan_record_aux_stack()before thatlocal_irq_save()but thencall_rcu()can be called preempt-disabled section so we would have the same problem.
Quoted 7 lines
The second warning came from
kasan_quarantine_remove_cache(). At the endper_cpu_remove_cache()->qlist_free_all()will free memory with disabled interrupts (due to that smp-function call).
Moving it to kworker would solve the problem. I don't mind keeping thatsmp_functioncall assuming that it is all debug-code and it increases overall latency anyway. But then could we maybe move all those objects to a single list which freed afteron_each_cpu()?
The quarantine is per-CPU, and I think what you suggest would fundamentally change its design. If you have something that works on RT without a fundamental change would be ideal (it is all debug code and not used on non-KASAN kernels).
Quoted 3 lines
Otherwise I haven't seen any new warnings showing up with KASAN enabled.
Sebastian
Quoted 51 lines · Marco Elver
Shuah Khan reported [1]:
| When CONFIG_PROVE_RAW_LOCK_NESTING=y and CONFIG_KASAN are enabled, | kasan_record_aux_stack() runs into "BUG: Invalid wait context" when | it tries to allocate memory attempting to acquire spinlock in page | allocation code while holding workqueue pool raw_spinlock. | | There are several instances of this problem when block layer tries | to __queue_work(). Call trace from one of these instances is below: | | kblockd_mod_delayed_work_on() | mod_delayed_work_on() | __queue_delayed_work() | __queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) | insert_work() | kasan_record_aux_stack() | kasan_save_stack() | stack_depot_save() | alloc_pages() | __alloc_pages() | get_page_from_freelist() | rm_queue() | rm_queue_pcplist() | local_lock_irqsave(&pagesets.lock, flags); | [ BUG: Invalid wait context triggered ][1] https://lkml.kernel.org/r/[email protected]
PROVE_RAW_LOCK_NESTINGis pointing out that (on RT kernels) the locking rules are being violated. More generally, memory is being allocated from a non-preemptive context (raw_spin_lock'd c-s) where it is not allowed.To properly fix this, we must prevent stackdepot from replenishing its "stack slab" pool if memory allocations cannot be done in the current
context: it's a bug to use eitherGFP_ATOMICnorGFP_NOWAITin certain non-preemptive contexts, includingraw_spin_locks(seegfp.handab00db216c9c7).The only downside is that saving a stack trace may fail if: stackdepot runs out of space AND the same stack trace has not been recorded before. I expect this to be unlikely, and a simple experiment (boot the kernel) didn't result in any failure to record stack trace from
insert_work().The series includes a few minor fixes to stackdepot that I noticed in preparing the series. It then introduces
__stack_depot_save(), which exposes the option to force stackdepot to not allocate any memory. Finally, KASAN is changed to use the new stackdepot interface and providekasan_record_aux_stack_noalloc(), which is then used by workqueue code.
Quoted 7 lines
Marco Elver (6):
lib/stackdepot: includegfp.h
lib/stackdepot: remove unused function argument lib/stackdepot: introduce__stack_depot_save()
kasan: common: providecan_allocinkasan_save_stack()
kasan: generic: introducekasan_record_aux_stack_noalloc()workqueue, kasan: avoidalloc_pages()when recording stack
for the whole series.
Quoted 13 lines
include/linux/kasan.h | 2 ++ include/linux/stackdepot.h | 6 +++++ kernel/workqueue.c | 2 +- lib/stackdepot.c | 51 ++++++++++++++++++++++++++++++-------- mm/kasan/common.c | 6 ++--- mm/kasan/generic.c | 14 +++++++++-- mm/kasan/kasan.h | 2 +- 7 files changed, 65 insertions(+), 18 deletions(-)--
2.33.0.153.gba50c8fa24-goog
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg