mm: avoid slub allocation while holding list_lock
Subsystems (2)
PATCH 1/1 mm: avoid slub allocation while holding list_lock
If we are already under list_lock, don't call kmalloc(). Otherwise we will run into deadlock because kmalloc() also tries to grab the same lock.
Instead, allocate pages directly. Given currently page->objects has 15 bits, we only need 1 page. We may waste some memory but we only do so when slub debug is on.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at: ___slab_alloc+0x104/0x437
but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at: __kmem_cache_shutdown+0x81/0x3cb
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&n->list_lock)->rlock);
lock(&(&n->list_lock)->rlock);*** DEADLOCK ***
--- mm/slub.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/slub.c +6 -2
@@ -3683,7 +3683,11 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page,#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); +int order; +unsigned long *map; + + order =get_order (DIV_ROUND_UP (page->objects,BITS_PER_BYTE )); + map = (void *)__get_free_pages (GFP_ATOMIC | __GFP_ZERO, order);if (!map)return ;slab_err (s, page, text, s->name);
@@ -3698,7 +3702,7 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page, } }slab_unlock (page); -bitmap_free (map); +free_pages ((unsigned long )map, order);#endif }
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Quoted 19 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.Instead, allocate pages directly. Given currently
page->objectshas 15 bits, we only need 1 page. We may waste some memory but we only do so when slub debug is on.WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Possible unsafe locking scenario:
Quoted 8 lines
CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);*** DEADLOCK ***Signed-off-by Yu Zhao <[email protected]>
Looks sane to me:
Signature · 1 line
Kirill A. Shutemov
Quoted 25 lines · Kirill A. Shutemov
Quoted 24 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.Instead, allocate pages directly. Given currently
page->objectshas 15 bits, we only need 1 page. We may waste some memory but we only do so when slub debug is on.WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Possible unsafe locking scenario:CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);
Quoted 8 lines
Quoted 3 lines
*** DEADLOCK ***Signed-off-by Yu Zhao <[email protected]>Looks sane to me:
Really?
Since page->objects is handled as bitmap, alignment should be BITS_PER_LONG than BITS_PER_BYTE (though in this particular case, get_order() would implicitly align BITS_PER_BYTE * PAGE_SIZE). But get_order(0) is an undefined behavior.
Quoted 35 lines · Tetsuo Handa
Quoted 25 lines · Kirill A. Shutemov
Quoted 24 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.Instead, allocate pages directly. Given currently
page->objectshas 15 bits, we only need 1 page. We may waste some memory but we only do so when slub debug is on.WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Possible unsafe locking scenario:CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);Quoted 8 lines
Quoted 3 lines
*** DEADLOCK ***Signed-off-by Yu Zhao <[email protected]>Looks sane to me:
Quoted 6 lines
Really?
Since
page->objectsis handled as bitmap, alignment should beBITS_PER_LONGthanBITS_PER_BYTE(though in this particular case,get_order()would implicitly alignBITS_PER_BYTE*PAGE_SIZE). Butget_order(0)is an undefined behavior.
I think we can safely assume PAGE_SIZE is unsigned long aligned and page->objects is non-zero. But if you don't feel comfortable with these assumptions, I'd be happy to ensure them explicitly.
Quoted 3 lines · Yu Zhao
I think we can safely assume
PAGE_SIZEis unsigned long aligned andpage->objectsis non-zero. But if you don't feel comfortable with these assumptions, I'd be happy to ensure them explicitly.
I know PAGE_SIZE is unsigned long aligned. If someone by chance happens to change from "dynamic allocation" to "on stack", get_order() will no longer be called and the bug will show up.
I don't know whether __get_free_page(GFP_ATOMIC) can temporarily consume more than 4096 bytes, but if it can, we might want to avoid "dynamic allocation".
By the way, if "struct kmem_cache_node" is object which won't have many thousands of instances, can't we embed that buffer into "struct kmem_cache_node" because max size of that buffer is only 4096 bytes?
Quoted 11 lines · Tetsuo Handa
Quoted 3 lines · Yu Zhao
I think we can safely assume
PAGE_SIZEis unsigned long aligned andpage->objectsis non-zero. But if you don't feel comfortable with these assumptions, I'd be happy to ensure them explicitly.I know
PAGE_SIZEis unsigned long aligned. If someone by chance happens to change from "dynamic allocation" to "on stack",get_order()will no longer be called and the bug will show up.I don't know whether
__get_free_page(GFP_ATOMIC)can temporarily consume more than 4096 bytes, but if it can, we might want to avoid "dynamic allocation".
With GFP_ATOMIC and ~~__GFP_HIGHMEM, it shouldn't.
Quoted 3 lines
By the way, if "struct
kmem_cache_node" is object which won't have many thousands of instances, can't we embed that buffer into "structkmem_cache_node" because max size of that buffer is only 4096 bytes?
It seems to me allocation in error path is better than always keeping a page around. But the latter may still be acceptable given it's done only when debug is on and, of course, on a per-node scale.
Quoted 38 lines · Yu Zhao
Quoted 37 lines · Tetsuo Handa
Quoted 25 lines · Kirill A. Shutemov
Quoted 24 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.Instead, allocate pages directly. Given currently
page->objectshas 15 bits, we only need 1 page. We may waste some memory but we only do so when slub debug is on.WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Possible unsafe locking scenario:CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);Quoted 8 lines
Quoted 3 lines
*** DEADLOCK ***Signed-off-by Yu Zhao <[email protected]>Looks sane to me:
Really?
Quoted 7 lines
Quoted 4 lines
Since
page->objectsis handled as bitmap, alignment should beBITS_PER_LONGthanBITS_PER_BYTE(though in this particular case,get_order()would implicitly alignBITS_PER_BYTE*PAGE_SIZE). Butget_order(0)is an undefined behavior.I think we can safely assume
PAGE_SIZEis unsigned long aligned andpage->objectsis non-zero.
I think it's better to handle page->objects == 0 gracefully. It should not happen, but this code handles situation that should not happen.
Signature · 1 line
Kirill A. Shutemov
Andrew Morton
maintainerQuoted 17 lines · On Mon, 9 Sep 2019 00:10:16 -0600 Yu Zhao <[email protected]> wrote:
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.Instead, allocate pages directly. Given currently
page->objectshas 15 bits, we only need 1 page. We may waste some memory but we only do so when slub debug is on.WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Quoted 9 lines
Possible unsafe locking scenario:CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);*** DEADLOCK ***
It would be better if a silly low-level debug function like this weren't to try to allocate memory at all. Did you consider simply using a statically allocated buffer?
{
static char buffer[something large enough];
static spinlock_t lock_to_protect_it;Alternatively, do we need to call get_map() at all in there? We could simply open-code the get_map() functionality inside list_slab_objects(). It would be slower, but printk is already slow. Potentially extremely slow.
PATCH 1/3 mm: correct mask size for slub page->objects
Mask of slub objects per page shouldn't be larger than what page->objects can hold.
It requires more than 2^15 objects to hit the problem, and I don't think anybody would. It'd be nice to have the mask fixed, but not really worth cc'ing the stable.
--- mm/slub.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/slub.c +3 -1
@@ -187,7 +187,7 @@ static inline bool kmem_cache_has_cpu_partial(struct kmem_cache *s)*/ #define DEBUG_METADATA_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER) -#define OO_SHIFT 16 +#define OO_SHIFT 15#define OO_MASK ((1 << OO_SHIFT) - 1)#define MAX_OBJS_PER_PAGE 32767/* since page.objects is u15 */
@@ -343,6 +343,8 @@ static inline unsigned int oo_order(struct kmem_cache_order_objects x)static inline unsigned int oo_objects (struct kmem_cache_order_objects x) { +BUILD_BUG_ON (OO_MASK >MAX_OBJS_PER_PAGE ); +return x.x &OO_MASK ; }
Signature · 1 line
2.23.0.162.g0b9fbb3734-goog
PATCH 2/3 mm: avoid slub allocation while holding list_lock
If we are already under list_lock, don't call kmalloc(). Otherwise we will run into deadlock because kmalloc() also tries to grab the same lock.
Instead, statically allocate bitmap in struct kmem_cache_node. Given currently page->objects has 15 bits, we bloat the per-node struct by 4K. So we waste some memory but only do so when slub debug is on.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at: ___slab_alloc+0x104/0x437
but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at: __kmem_cache_shutdown+0x81/0x3cb
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&n->list_lock)->rlock);
lock(&(&n->list_lock)->rlock);*** DEADLOCK ***
--- include/linux/slub_def.h | 4 ++++ mm/slab.h | 1 + mm/slub.c | 44 ++++++++++++++-------------------------- 3 files changed, 20 insertions(+), 29 deletions(-)
include/linux/slub_def.h +4 -0
@@ -9,6 +9,10 @@*/ #include <linux/kobject.h> +#define OO_SHIFT 15 +#define OO_MASK ((1 << OO_SHIFT) - 1) +#define MAX_OBJS_PER_PAGE 32767/* since page.objects is u15 */ +enum stat_item {ALLOC_FASTPATH ,/* Allocation from cpu slab */ ALLOC_SLOWPATH ,/* Allocation by getting a new cpu slab */
mm/slab.h +1 -0
@@ -556,6 +556,7 @@ struct kmem_cache_node {atomic_long_t nr_slabs;atomic_long_t total_objects;struct list_head full; +unsigned long object_map[BITS_TO_LONGS (MAX_OBJS_PER_PAGE )];#endif #endif
mm/slub.c +15 -29
139 lines not on this page · raw message
Signature · 1 line
2.23.0.162.g0b9fbb3734-goog
Quoted 7 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.Instead, statically allocate bitmap in struct
kmem_cache_node. Given currentlypage->objectshas 15 bits, we bloat the per-node struct by 4K. So we waste some memory but only do so when slub debug is on.
Why not have single page total protected by a lock?
Listing object from two pages at the same time doesn't make sense anyway. Cuncurent validating is not something sane to do.
Signature · 1 line
Kirill A. Shutemov
Quoted 5 lines · Kirill A. Shutemov
Quoted 4 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.
Quoted 8 lines
Quoted 3 lines
Instead, statically allocate bitmap in struct
kmem_cache_node. Given currentlypage->objectshas 15 bits, we bloat the per-node struct by 4K. So we waste some memory but only do so when slub debug is on.Why not have single page total protected by a lock?
Listing object from two pages at the same time doesn't make sense anyway. Cuncurent validating is not something sane to do.
Okay, cutting down to static global bitmap.
PATCH v2 1/4 mm: correct mask size for slub page->objects
In reply to Kirill A. ShutemovMask of slub objects per page shouldn't be larger than what page->objects can hold.
It requires more than 2^15 objects to hit the problem, and I don't think anybody would. It'd be nice to have the mask fixed, but not really worth cc'ing the stable.
--- mm/slub.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/slub.c +3 -1
@@ -187,7 +187,7 @@ static inline bool kmem_cache_has_cpu_partial(struct kmem_cache *s)*/ #define DEBUG_METADATA_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER) -#define OO_SHIFT 16 +#define OO_SHIFT 15#define OO_MASK ((1 << OO_SHIFT) - 1)#define MAX_OBJS_PER_PAGE 32767/* since page.objects is u15 */
@@ -343,6 +343,8 @@ static inline unsigned int oo_order(struct kmem_cache_order_objects x)static inline unsigned int oo_objects (struct kmem_cache_order_objects x) { +BUILD_BUG_ON (OO_MASK >MAX_OBJS_PER_PAGE ); +return x.x &OO_MASK ; }
Signature · 1 line
2.23.0.162.g0b9fbb3734-goog
PATCH v2 2/4 mm: clean up validate_slab()
The function doesn't need to return any value, and the check can be done in one pass.
There is a behavior change: before the patch, we stop at the first invalid free object; after the patch, we stop at the first invalid object, free or in use. This shouldn't matter because the original behavior isn't intended anyway.
--- mm/slub.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-)
mm/slub.c +8 -13
@@ -4386,31 +4386,26 @@ static int count_total(struct page *page)#endif #ifdef CONFIG_SLUB_DEBUG -static int validate_slab (struct kmem_cache *s,struct page *page, +static void validate_slab (struct kmem_cache *s,struct page *page,unsigned long *map) {void *p;void *addr =page_address (page); -if (!check_slab (s, page) || - !on_freelist (s, page,NULL )) -return 0 ; +if (!check_slab (s, page) || !on_freelist (s, page,NULL )) +return ;/* Now we know that a valid freelist exists */ bitmap_zero (map, page->objects);get_map (s, page, map);for_each_object (p, s, addr, page->objects) { -if (test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_INACTIVE )) -return 0 ; - } +u8 val =test_bit (slab_index (p, s, addr), map) ? +SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ; -for_each_object (p, s, addr, page->objects) -if (!test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_ACTIVE )) -return 0 ; -return 1 ; +if (!check_object (s, page, p, val)) +break ; + } } static void validate_slab_slab(struct kmem_cache *s, struct page *page,
Signature · 1 line
2.23.0.162.g0b9fbb3734-goog
Re: [PATCH v2 2/4] mm: clean up validate_slab()
Quoted 35 lines · Yu Zhao
The function doesn't need to return any value, and the check can be done in one pass.
There is a behavior change: before the patch, we stop at the first invalid free object; after the patch, we stop at the first invalid object, free or in use. This shouldn't matter because the original behavior isn't intended anyway.
Signed-off-by Yu Zhao <[email protected]>--- mm/slub.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-)mm/slub.c +3
-4@@ -4386,31 +4386,26 @@ static int count_total(struct page *page)#endif #ifdef CONFIG_SLUB_DEBUG -static int validate_slab (struct kmem_cache *s,struct page *page, +static void validate_slab (struct kmem_cache *s,struct page *page,unsigned long *map) {void *p;void *addr =page_address (page); -if (!check_slab (s, page) || - !on_freelist (s, page,NULL )) -return 0 ; +if (!check_slab (s, page) || !on_freelist (s, page,NULL )) +return ;/* Now we know that a valid freelist exists */
mm/slub.c static int count_total(struct page *page) · line 4404
Quoted 10 lines
mm/slub.c +2
-4bitmap_zero (map, page->objects);get_map (s, page, map);for_each_object (p, s, addr, page->objects) { -if (test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_INACTIVE )) -return 0 ; - } +u8 val =test_bit (slab_index (p, s, addr), map) ? +SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ;
Proper 'if' would be more readable.
Other than that look fine to me.
Quoted 15 lines
mm/slub.c +3
-5-for_each_object (p, s, addr, page->objects) -if (!test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_ACTIVE )) -return 0 ; -return 1 ; +if (!check_object (s, page, p, val)) +break ; + } } static void validate_slab_slab(struct kmem_cache *s, struct page *page,Signature · 1 line
2.23.0.162.g0b9fbb3734-goog
Signature · 1 line
Kirill A. Shutemov
PATCH v2 3/4 mm: avoid slub allocation while holding list_lock
In reply to Yu ZhaoIf we are already under list_lock, don't call kmalloc(). Otherwise we will run into deadlock because kmalloc() also tries to grab the same lock.
Fixing the problem by using a static bitmap instead.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at: ___slab_alloc+0x104/0x437
but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at: __kmem_cache_shutdown+0x81/0x3cb
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&n->list_lock)->rlock);
lock(&(&n->list_lock)->rlock);*** DEADLOCK ***
--- mm/slub.c | 88 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 47 insertions(+), 41 deletions(-)
mm/slub.c +47 -41
@@ -443,19 +443,38 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, }#ifdef CONFIG_SLUB_DEBUG +static unsigned long object_map[BITS_TO_LONGS (MAX_OBJS_PER_PAGE )]; +static DEFINE_SPINLOCK (object_map_lock); +/* * Determine a map of object in use on a page. * * Node listlock must be held to guarantee that the page does * not vanish from under us. */ -static void get_map (struct kmem_cache *s,struct page *page,unsigned long *map) +static unsigned long *get_map (struct kmem_cache *s,struct page *page) {void *p;void *addr =page_address (page); +VM_BUG_ON (!irqs_disabled ()); + +spin_lock (&object_map_lock); + +bitmap_zero (object_map, page->objects); +for (p = page->freelist; p; p =get_freepointer (s, p)) -set_bit (slab_index (p, s, addr), map); +set_bit (slab_index (p, s, addr), object_map); + +return object_map; +} + +static void put_map (unsigned long *map) +{ +VM_BUG_ON (map != object_map); +lockdep_assert_held (&object_map_lock); + +spin_unlock (&object_map_lock); }static inline unsigned int size_from_object (struct kmem_cache *s)
@@ -3685,13 +3704,12 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page,#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); -if (!map) -return ; +unsigned long *map; +slab_err (s, page, text, s->name);slab_lock (page); -get_map (s, page, map); + map =get_map (s, page);for_each_object (p, s, addr, page->objects) {if (!test_bit (slab_index (p, s, addr), map)) {
@@ -3699,8 +3717,9 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page, print_tracking(s, p); } } +put_map (map); +slab_unlock (page); -bitmap_free (map);#endif }
@@ -4386,19 +4405,19 @@ static int count_total(struct page *page)#endif #ifdef CONFIG_SLUB_DEBUG -static void validate_slab (struct kmem_cache *s,struct page *page, -unsigned long *map) +static void validate_slab (struct kmem_cache *s,struct page *page) {void *p;void *addr =page_address (page); +unsigned long *map; + +slab_lock (page);if (!check_slab (s, page) || !on_freelist (s, page,NULL )) -return ; + goto unlock;/* Now we know that a valid freelist exists */ -bitmap_zero (map, page->objects); - -get_map (s, page, map); + map =get_map (s, page);for_each_object (p, s, addr, page->objects) {u8 val =test_bit (slab_index (p, s, addr), map) ?SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ;
@@ -4406,18 +4425,13 @@ static void validate_slab(struct kmem_cache *s, struct page *page,if (!check_object (s, page, p, val))break ; } -} - -static void validate_slab_slab (struct kmem_cache *s,struct page *page, -unsigned long *map) -{ -slab_lock (page); -validate_slab (s, page, map); +put_map (map); +unlock:slab_unlock (page); }static int validate_slab_node (struct kmem_cache *s, -struct kmem_cache_node *n,unsigned long *map) +struct kmem_cache_node *n) {unsigned long count =0 ;struct page *page;
@@ -4426,7 +4440,7 @@ static int validate_slab_node(struct kmem_cache *s,spin_lock_irqsave (&n->list_lock, flags);list_for_each_entry (page, &n->partial, slab_list) { -validate_slab_slab (s, page, map); +validate_slab (s, page); count++; }if (count != n->nr_partial)
@@ -4437,7 +4451,7 @@ static int validate_slab_node(struct kmem_cache *s, goto out;list_for_each_entry (page, &n->full, slab_list) { -validate_slab_slab (s, page, map); +validate_slab (s, page); count++; }if (count !=atomic_long_read (&n->nr_slabs))
@@ -4454,15 +4468,11 @@ static long validate_slab_cache(struct kmem_cache *s)int node;unsigned long count =0 ;struct kmem_cache_node *n; -unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); - -if (!map) -return -ENOMEM ;flush_all (s);for_each_kmem_cache_node (s, node, n) - count +=validate_slab_node (s, n, map); -bitmap_free (map); + count +=validate_slab_node (s, n); +return count; }/*
@@ -4592,18 +4602,17 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, }static void process_slab (struct loc_track *t,struct kmem_cache *s, -struct page *page,enum track_item alloc, -unsigned long *map) +struct page *page,enum track_item alloc) {void *addr =page_address (page);void *p; +unsigned long *map; -bitmap_zero (map, page->objects); -get_map (s, page, map); - + map =get_map (s, page);for_each_object (p, s, addr, page->objects)if (!test_bit (slab_index (p, s, addr), map))add_location (t, s,get_track (s, p, alloc)); +put_map (map); } static int list_locations(struct kmem_cache *s, char *buf,
@@ -4614,11 +4623,9 @@ static int list_locations(struct kmem_cache *s, char *buf,struct loc_track t = {0 ,0 ,NULL };int node;struct kmem_cache_node *n; -unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); -if (!map || !alloc_loc_track (&t,PAGE_SIZE /sizeof (struct location ), -GFP_KERNEL )) { -bitmap_free (map); +if (!alloc_loc_track (&t,PAGE_SIZE /sizeof (struct location ), +GFP_KERNEL )) {return sprintf (buf,"Out of memory\n" ); }/* Push back cpu slabs */
@@ -4633,9 +4640,9 @@ static int list_locations(struct kmem_cache *s, char *buf,spin_lock_irqsave (&n->list_lock, flags);list_for_each_entry (page, &n->partial, slab_list) -process_slab (&t, s, page, alloc, map); +process_slab (&t, s, page, alloc);list_for_each_entry (page, &n->full, slab_list) -process_slab (&t, s, page, alloc, map); +process_slab (&t, s, page, alloc);spin_unlock_irqrestore (&n->list_lock, flags); }
@@ -4684,7 +4691,6 @@ static int list_locations(struct kmem_cache *s, char *buf, }free_loc_track (&t); -bitmap_free (map);if (!t.count) len +=sprintf (buf,"No data\n" );return len;
Signature · 1 line
2.23.0.162.g0b9fbb3734-goog
Quoted 17 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.Fixing the problem by using a static bitmap instead.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Possible unsafe locking scenario:
Quoted 8 lines
CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);*** DEADLOCK ***Signed-off-by Yu Zhao <[email protected]>
Signature · 1 line
Kirill A. Shutemov
PATCH v2 4/4 mm: lock slub page when listing objects
In reply to Yu ZhaoThough I have no idea what the side effect of such race would be, apparently we want to prevent the free list from being changed while debugging the objects.
--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/slub.c +4 -0
@@ -4608,11 +4608,15 @@ static void process_slab(struct loc_track *t, struct kmem_cache *s,void *p;unsigned long *map; +slab_lock (page); + map =get_map (s, page);for_each_object (p, s, addr, page->objects)if (!test_bit (slab_index (p, s, addr), map))add_location (t, s,get_track (s, p, alloc));put_map (map); + +slab_unlock (page); } static int list_locations(struct kmem_cache *s, char *buf,
Signature · 1 line
2.23.0.162.g0b9fbb3734-goog
Re: [PATCH v2 4/4] mm: lock slub page when listing objects
Quoted 3 lines · Yu Zhao
Though I have no idea what the side effect of such race would be, apparently we want to prevent the free list from being changed while debugging the objects.
Codewise looks good to me. But commit message can be better.
Can we repharase it to indicate that slab_lock is required to protect page->objects?
Signature · 1 line
Kirill A. Shutemov
Re: [PATCH v2 4/4] mm: lock slub page when listing objects
Quoted 9 lines · Kirill A. Shutemov
Quoted 3 lines · Yu Zhao
Though I have no idea what the side effect of such race would be, apparently we want to prevent the free list from being changed while debugging the objects.
Codewise looks good to me. But commit message can be better.
Can we repharase it to indicate that
slab_lockis required to protectpage->objects?
Will do.
Re: [PATCH v2 4/4] mm: lock slub page when listing objects
In reply to Yu ZhaoQuoted 3 lines · Yu Zhao
Though I have no idea what the side effect of such race would be, apparently we want to prevent the free list from being changed while debugging the objects.
process_slab() is called under the list_lock which prevents any allocation from the free list in the slab page. This means that new objects can be added to the freelist which occurs by updating the freelist pointer in the slab page with a pointer to the newly free object which in turn contains the old freelist pointr.
It is therefore possible to safely traverse the objects on the freelist after the pointer has been retrieved
NAK.
Re: [PATCH v2 1/4] mm: correct mask size for slub page->objects
In reply to Yu ZhaoQuoted 9 lines · Yu Zhao
Mask of slub objects per page shouldn't be larger than what
page->objectscan hold.It requires more than 2^15 objects to hit the problem, and I don't think anybody would. It'd be nice to have the mask fixed, but not really worth cc'ing the stable.
Fixes 50d5c41cd151 ("slub: Do not use frozen page flag but a bit in the page counters")Signed-off-by Yu Zhao <[email protected]>
I don't think the patch fixes anything.
Yes, we have one spare bit between order and number of object that is not used and always zero. So what?
I can imagine for some microarchitecures accessing higher 16 bits of int is cheaper than shifting by 15.
Signature · 1 line
Kirill A. Shutemov
Re: [PATCH v2 1/4] mm: correct mask size for slub page->objects
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Yu Zhao
Mask of slub objects per page shouldn't be larger than what
page->objectscan hold.
Quoted 8 lines
Quoted 6 lines
It requires more than 2^15 objects to hit the problem, and I don't think anybody would. It'd be nice to have the mask fixed, but not really worth cc'ing the stable.
Fixes 50d5c41cd151 ("slub: Do not use frozen page flag but a bit in the page counters")Signed-off-by Yu Zhao <[email protected]>I don't think the patch fixes anything.
Technically it does. It makes no sense for a mask to have more bits than the variable that holds the masked value. I had to look up the commit history to find out why and go through the code to make sure it doesn't actually cause any problem.
My hope is that nobody else would have to go through the same trouble.
Quoted 5 lines
Yes, we have one spare bit between order and number of object that is not used and always zero. So what?
I can imagine for some microarchitecures accessing higher 16 bits of int is cheaper than shifting by 15.
Well, I highly doubt the inconsistency is intended for such optimization, even it existed.
Re: [PATCH v2 1/4] mm: correct mask size for slub page->objects
Quoted 12 lines · Yu Zhao
Quoted 11 lines · Kirill A. Shutemov
Quoted 9 lines · Yu Zhao
Mask of slub objects per page shouldn't be larger than what
page->objectscan hold.It requires more than 2^15 objects to hit the problem, and I don't think anybody would. It'd be nice to have the mask fixed, but not really worth cc'ing the stable.
Fixes 50d5c41cd151 ("slub: Do not use frozen page flag but a bit in the page counters")Signed-off-by Yu Zhao <[email protected]>
Quoted 8 lines
Quoted 1 line
I don't think the patch fixes anything.
Technically it does. It makes no sense for a mask to have more bits than the variable that holds the masked value. I had to look up the commit history to find out why and go through the code to make sure it doesn't actually cause any problem.
My hope is that nobody else would have to go through the same trouble.
Just put some comments then.
Signature · 1 line
Kirill A. Shutemov
PATCH v3 1/2 mm: clean up validate_slab()
In reply to Yu ZhaoThe function doesn't need to return any value, and the check can be done in one pass.
There is a behavior change: before the patch, we stop at the first invalid free object; after the patch, we stop at the first invalid object, free or in use. This shouldn't matter because the original behavior isn't intended anyway.
--- mm/slub.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-)
mm/slub.c +8 -13
@@ -4384,31 +4384,26 @@ static int count_total(struct page *page)#endif #ifdef CONFIG_SLUB_DEBUG -static int validate_slab (struct kmem_cache *s,struct page *page, +static void validate_slab (struct kmem_cache *s,struct page *page,unsigned long *map) {void *p;void *addr =page_address (page); -if (!check_slab (s, page) || - !on_freelist (s, page,NULL )) -return 0 ; +if (!check_slab (s, page) || !on_freelist (s, page,NULL )) +return ;/* Now we know that a valid freelist exists */ bitmap_zero (map, page->objects);get_map (s, page, map);for_each_object (p, s, addr, page->objects) { -if (test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_INACTIVE )) -return 0 ; - } +u8 val =test_bit (slab_index (p, s, addr), map) ? +SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ; -for_each_object (p, s, addr, page->objects) -if (!test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_ACTIVE )) -return 0 ; -return 1 ; +if (!check_object (s, page, p, val)) +break ; + } } static void validate_slab_slab(struct kmem_cache *s, struct page *page,
Signature · 1 line
2.23.0.237.gc6a4ce50a0-goog
PATCH v3 2/2 mm: avoid slub allocation while holding list_lock
If we are already under list_lock, don't call kmalloc(). Otherwise we will run into deadlock because kmalloc() also tries to grab the same lock.
Fixing the problem by using a static bitmap instead.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at: ___slab_alloc+0x104/0x437
but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at: __kmem_cache_shutdown+0x81/0x3cb
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&n->list_lock)->rlock);
lock(&(&n->list_lock)->rlock);*** DEADLOCK ***
--- mm/slub.c | 88 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 47 insertions(+), 41 deletions(-)
mm/slub.c +47 -41
@@ -441,19 +441,38 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, }#ifdef CONFIG_SLUB_DEBUG +static unsigned long object_map[BITS_TO_LONGS (MAX_OBJS_PER_PAGE )]; +static DEFINE_SPINLOCK (object_map_lock); +/* * Determine a map of object in use on a page. * * Node listlock must be held to guarantee that the page does * not vanish from under us. */ -static void get_map (struct kmem_cache *s,struct page *page,unsigned long *map) +static unsigned long *get_map (struct kmem_cache *s,struct page *page) {void *p;void *addr =page_address (page); +VM_BUG_ON (!irqs_disabled ()); + +spin_lock (&object_map_lock); + +bitmap_zero (object_map, page->objects); +for (p = page->freelist; p; p =get_freepointer (s, p)) -set_bit (slab_index (p, s, addr), map); +set_bit (slab_index (p, s, addr), object_map); + +return object_map; +} + +static void put_map (unsigned long *map) +{ +VM_BUG_ON (map != object_map); +lockdep_assert_held (&object_map_lock); + +spin_unlock (&object_map_lock); }static inline unsigned int size_from_object (struct kmem_cache *s)
@@ -3683,13 +3702,12 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page,#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); -if (!map) -return ; +unsigned long *map; +slab_err (s, page, text, s->name);slab_lock (page); -get_map (s, page, map); + map =get_map (s, page);for_each_object (p, s, addr, page->objects) {if (!test_bit (slab_index (p, s, addr), map)) {
@@ -3697,8 +3715,9 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page, print_tracking(s, p); } } +put_map (map); +slab_unlock (page); -bitmap_free (map);#endif }
@@ -4384,19 +4403,19 @@ static int count_total(struct page *page)#endif #ifdef CONFIG_SLUB_DEBUG -static void validate_slab (struct kmem_cache *s,struct page *page, -unsigned long *map) +static void validate_slab (struct kmem_cache *s,struct page *page) {void *p;void *addr =page_address (page); +unsigned long *map; + +slab_lock (page);if (!check_slab (s, page) || !on_freelist (s, page,NULL )) -return ; + goto unlock;/* Now we know that a valid freelist exists */ -bitmap_zero (map, page->objects); - -get_map (s, page, map); + map =get_map (s, page);for_each_object (p, s, addr, page->objects) {u8 val =test_bit (slab_index (p, s, addr), map) ?SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ;
@@ -4404,18 +4423,13 @@ static void validate_slab(struct kmem_cache *s, struct page *page,if (!check_object (s, page, p, val))break ; } -} - -static void validate_slab_slab (struct kmem_cache *s,struct page *page, -unsigned long *map) -{ -slab_lock (page); -validate_slab (s, page, map); +put_map (map); +unlock:slab_unlock (page); }static int validate_slab_node (struct kmem_cache *s, -struct kmem_cache_node *n,unsigned long *map) +struct kmem_cache_node *n) {unsigned long count =0 ;struct page *page;
@@ -4424,7 +4438,7 @@ static int validate_slab_node(struct kmem_cache *s,spin_lock_irqsave (&n->list_lock, flags);list_for_each_entry (page, &n->partial, slab_list) { -validate_slab_slab (s, page, map); +validate_slab (s, page); count++; }if (count != n->nr_partial)
@@ -4435,7 +4449,7 @@ static int validate_slab_node(struct kmem_cache *s, goto out;list_for_each_entry (page, &n->full, slab_list) { -validate_slab_slab (s, page, map); +validate_slab (s, page); count++; }if (count !=atomic_long_read (&n->nr_slabs))
@@ -4452,15 +4466,11 @@ static long validate_slab_cache(struct kmem_cache *s)int node;unsigned long count =0 ;struct kmem_cache_node *n; -unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); - -if (!map) -return -ENOMEM ;flush_all (s);for_each_kmem_cache_node (s, node, n) - count +=validate_slab_node (s, n, map); -bitmap_free (map); + count +=validate_slab_node (s, n); +return count; }/*
@@ -4590,18 +4600,17 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, }static void process_slab (struct loc_track *t,struct kmem_cache *s, -struct page *page,enum track_item alloc, -unsigned long *map) +struct page *page,enum track_item alloc) {void *addr =page_address (page);void *p; +unsigned long *map; -bitmap_zero (map, page->objects); -get_map (s, page, map); - + map =get_map (s, page);for_each_object (p, s, addr, page->objects)if (!test_bit (slab_index (p, s, addr), map))add_location (t, s,get_track (s, p, alloc)); +put_map (map); } static int list_locations(struct kmem_cache *s, char *buf,
@@ -4612,11 +4621,9 @@ static int list_locations(struct kmem_cache *s, char *buf,struct loc_track t = {0 ,0 ,NULL };int node;struct kmem_cache_node *n; -unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); -if (!map || !alloc_loc_track (&t,PAGE_SIZE /sizeof (struct location ), -GFP_KERNEL )) { -bitmap_free (map); +if (!alloc_loc_track (&t,PAGE_SIZE /sizeof (struct location ), +GFP_KERNEL )) {return sprintf (buf,"Out of memory\n" ); }/* Push back cpu slabs */
@@ -4631,9 +4638,9 @@ static int list_locations(struct kmem_cache *s, char *buf,spin_lock_irqsave (&n->list_lock, flags);list_for_each_entry (page, &n->partial, slab_list) -process_slab (&t, s, page, alloc, map); +process_slab (&t, s, page, alloc);list_for_each_entry (page, &n->full, slab_list) -process_slab (&t, s, page, alloc, map); +process_slab (&t, s, page, alloc);spin_unlock_irqrestore (&n->list_lock, flags); }
@@ -4682,7 +4689,6 @@ static int list_locations(struct kmem_cache *s, char *buf, }free_loc_track (&t); -bitmap_free (map);if (!t.count) len +=sprintf (buf,"No data\n" );return len;
Signature · 1 line
2.23.0.237.gc6a4ce50a0-goog
Re: [PATCH v3 1/2] mm: clean up validate_slab()
In reply to Yu ZhaoQuoted 9 lines · Yu Zhao
The function doesn't need to return any value, and the check can be done in one pass.
There is a behavior change: before the patch, we stop at the first invalid free object; after the patch, we stop at the first invalid object, free or in use. This shouldn't matter because the original behavior isn't intended anyway.
Signed-off-by Yu Zhao <[email protected]>
Signature · 1 line
Kirill A. Shutemov
PATCH v4 1/2 mm: clean up validate_slab()
In reply to Yu ZhaoThe function doesn't need to return any value, and the check can be done in one pass.
There is a behavior change: before the patch, we stop at the first invalid free object; after the patch, we stop at the first invalid object, free or in use. This shouldn't matter because the original behavior isn't intended anyway.
--- mm/slub.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-)
mm/slub.c +8 -13
@@ -4404,31 +4404,26 @@ static int count_total(struct page *page)#endif #ifdef CONFIG_SLUB_DEBUG -static int validate_slab (struct kmem_cache *s,struct page *page, +static void validate_slab (struct kmem_cache *s,struct page *page,unsigned long *map) {void *p;void *addr =page_address (page); -if (!check_slab (s, page) || - !on_freelist (s, page,NULL )) -return 0 ; +if (!check_slab (s, page) || !on_freelist (s, page,NULL )) +return ;/* Now we know that a valid freelist exists */ bitmap_zero (map, page->objects);get_map (s, page, map);for_each_object (p, s, addr, page->objects) { -if (test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_INACTIVE )) -return 0 ; - } +u8 val =test_bit (slab_index (p, s, addr), map) ? +SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ; -for_each_object (p, s, addr, page->objects) -if (!test_bit (slab_index (p, s, addr), map)) -if (!check_object (s, page, p,SLUB_RED_ACTIVE )) -return 0 ; -return 1 ; +if (!check_object (s, page, p, val)) +break ; + } } static void validate_slab_slab(struct kmem_cache *s, struct page *page,
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
PATCH v4 2/2 mm: avoid slub allocation while holding list_lock
If we are already under list_lock, don't call kmalloc(). Otherwise we will run into deadlock because kmalloc() also tries to grab the same lock.
Fixing the problem by using a static bitmap instead.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at: ___slab_alloc+0x104/0x437
but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at: __kmem_cache_shutdown+0x81/0x3cb
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&n->list_lock)->rlock);
lock(&(&n->list_lock)->rlock);*** DEADLOCK ***
--- mm/slub.c | 88 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 47 insertions(+), 41 deletions(-)
mm/slub.c +47 -41
@@ -441,19 +441,38 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, }#ifdef CONFIG_SLUB_DEBUG +static unsigned long object_map[BITS_TO_LONGS (MAX_OBJS_PER_PAGE )]; +static DEFINE_SPINLOCK (object_map_lock); +/* * Determine a map of object in use on a page. * * Node listlock must be held to guarantee that the page does * not vanish from under us. */ -static void get_map (struct kmem_cache *s,struct page *page,unsigned long *map) +static unsigned long *get_map (struct kmem_cache *s,struct page *page) {void *p;void *addr =page_address (page); +VM_BUG_ON (!irqs_disabled ()); + +spin_lock (&object_map_lock); + +bitmap_zero (object_map, page->objects); +for (p = page->freelist; p; p =get_freepointer (s, p)) -set_bit (slab_index (p, s, addr), map); +set_bit (slab_index (p, s, addr), object_map); + +return object_map; +} + +static void put_map (unsigned long *map) +{ +VM_BUG_ON (map != object_map); +lockdep_assert_held (&object_map_lock); + +spin_unlock (&object_map_lock); }static inline unsigned int size_from_object (struct kmem_cache *s)
@@ -3695,13 +3714,12 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page,#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); -if (!map) -return ; +unsigned long *map; +slab_err (s, page, text, s->name);slab_lock (page); -get_map (s, page, map); + map =get_map (s, page);for_each_object (p, s, addr, page->objects) {if (!test_bit (slab_index (p, s, addr), map)) {
@@ -3709,8 +3727,9 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page, print_tracking(s, p); } } +put_map (map); +slab_unlock (page); -bitmap_free (map);#endif }
@@ -4404,19 +4423,19 @@ static int count_total(struct page *page)#endif #ifdef CONFIG_SLUB_DEBUG -static void validate_slab (struct kmem_cache *s,struct page *page, -unsigned long *map) +static void validate_slab (struct kmem_cache *s,struct page *page) {void *p;void *addr =page_address (page); +unsigned long *map; + +slab_lock (page);if (!check_slab (s, page) || !on_freelist (s, page,NULL )) -return ; + goto unlock;/* Now we know that a valid freelist exists */ -bitmap_zero (map, page->objects); - -get_map (s, page, map); + map =get_map (s, page);for_each_object (p, s, addr, page->objects) {u8 val =test_bit (slab_index (p, s, addr), map) ?SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ;
@@ -4424,18 +4443,13 @@ static void validate_slab(struct kmem_cache *s, struct page *page,if (!check_object (s, page, p, val))break ; } -} - -static void validate_slab_slab (struct kmem_cache *s,struct page *page, -unsigned long *map) -{ -slab_lock (page); -validate_slab (s, page, map); +put_map (map); +unlock:slab_unlock (page); }static int validate_slab_node (struct kmem_cache *s, -struct kmem_cache_node *n,unsigned long *map) +struct kmem_cache_node *n) {unsigned long count =0 ;struct page *page;
@@ -4444,7 +4458,7 @@ static int validate_slab_node(struct kmem_cache *s,spin_lock_irqsave (&n->list_lock, flags);list_for_each_entry (page, &n->partial, slab_list) { -validate_slab_slab (s, page, map); +validate_slab (s, page); count++; }if (count != n->nr_partial)
@@ -4455,7 +4469,7 @@ static int validate_slab_node(struct kmem_cache *s, goto out;list_for_each_entry (page, &n->full, slab_list) { -validate_slab_slab (s, page, map); +validate_slab (s, page); count++; }if (count !=atomic_long_read (&n->nr_slabs))
@@ -4472,15 +4486,11 @@ static long validate_slab_cache(struct kmem_cache *s)int node;unsigned long count =0 ;struct kmem_cache_node *n; -unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); - -if (!map) -return -ENOMEM ;flush_all (s);for_each_kmem_cache_node (s, node, n) - count +=validate_slab_node (s, n, map); -bitmap_free (map); + count +=validate_slab_node (s, n); +return count; }/*
@@ -4610,18 +4620,17 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, }static void process_slab (struct loc_track *t,struct kmem_cache *s, -struct page *page,enum track_item alloc, -unsigned long *map) +struct page *page,enum track_item alloc) {void *addr =page_address (page);void *p; +unsigned long *map; -bitmap_zero (map, page->objects); -get_map (s, page, map); - + map =get_map (s, page);for_each_object (p, s, addr, page->objects)if (!test_bit (slab_index (p, s, addr), map))add_location (t, s,get_track (s, p, alloc)); +put_map (map); } static int list_locations(struct kmem_cache *s, char *buf,
@@ -4632,11 +4641,9 @@ static int list_locations(struct kmem_cache *s, char *buf,struct loc_track t = {0 ,0 ,NULL };int node;struct kmem_cache_node *n; -unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); -if (!map || !alloc_loc_track (&t,PAGE_SIZE /sizeof (struct location ), -GFP_KERNEL )) { -bitmap_free (map); +if (!alloc_loc_track (&t,PAGE_SIZE /sizeof (struct location ), +GFP_KERNEL )) {return sprintf (buf,"Out of memory\n" ); }/* Push back cpu slabs */
@@ -4651,9 +4658,9 @@ static int list_locations(struct kmem_cache *s, char *buf,spin_lock_irqsave (&n->list_lock, flags);list_for_each_entry (page, &n->partial, slab_list) -process_slab (&t, s, page, alloc, map); +process_slab (&t, s, page, alloc);list_for_each_entry (page, &n->full, slab_list) -process_slab (&t, s, page, alloc, map); +process_slab (&t, s, page, alloc);spin_unlock_irqrestore (&n->list_lock, flags); }
@@ -4702,7 +4709,6 @@ static int list_locations(struct kmem_cache *s, char *buf, }free_loc_track (&t); -bitmap_free (map);if (!t.count) len +=sprintf (buf,"No data\n" );return len;
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
Quoted 3 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.
How did this happen? The kmalloc needs to be always done before the list_lock is taken.
Quoted 11 lines
Fixing the problem by using a static bitmap instead.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Quoted 8 lines
Possible unsafe locking scenario:CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);*** DEADLOCK ***
Ahh. list_slab_objects() in shutdown?
There is a much easier fix for this:
[FIX] slub: Remove kmalloc under list_lock from list_slab_objects()
list_slab_objects() is called when a slab is destroyed and there are objects still left to list the objects in the syslog. This is a pretty rare event.
And there it seems we take the list_lock and call kmalloc while holding that lock.
Perform the allocation in free_partial() before the list_lock is taken.
mm/slub.c +7 -5
@@ -3690,14 +3690,11 @@ error: }static void list_slab_objects (struct kmem_cache *s,struct page *page, -const char *text) +const char *text,unsigned long *map) {#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); -if (!map) -return ;slab_err (s, page, text, s->name);slab_lock (page);
@@ -3723,6 +3720,10 @@ static void free_partial(struct kmem_cac {LIST_HEAD (discard);struct page *page, *h; +unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); + +if (!map) +return ;BUG_ON (irqs_disabled ());spin_lock_irq (&n->list_lock);
@@ -3732,7 +3733,8 @@ static void free_partial(struct kmem_caclist_add (&page->slab_list, &discard); }else {list_slab_objects (s, page, -"Objects remaining in %s on __kmem_cache_shutdown()" ); +"Objects remaining in %s on __kmem_cache_shutdown()" , + map); } }spin_unlock_irq (&n->list_lock);
Quoted 69 lines · Christopher Lameter
Quoted 3 lines · Yu Zhao
If we are already under
list_lock, don't callkmalloc(). Otherwise we will run into deadlock becausekmalloc()also tries to grab the same lock.How did this happen? The kmalloc needs to be always done before the
list_lockis taken.Quoted 11 lines
Fixing the problem by using a static bitmap instead.
WARNING: possible recursive locking detected
--------------------------------------------
mount-encrypted/4921 is trying to acquire lock:
(&(&n->list_lock)->rlock){-.-.}, at:___slab_alloc+0x104/0x437but task is already holding lock:
(&(&n->list_lock)->rlock){-.-.}, at:__kmem_cache_shutdown+0x81/0x3cbother info that might help us debug this:
Quoted 8 lines
Possible unsafe locking scenario:CPU0 ---- lock(&(&n->list_lock)->rlock); lock(&(&n->list_lock)->rlock);*** DEADLOCK ***Ahh.
list_slab_objects()in shutdown?There is a much easier fix for this:
[FIX] slub: Remove kmalloc under
list_lockfromlist_slab_objects()
list_slab_objects()is called when a slab is destroyed and there are objects still left to list the objects in the syslog. This is a pretty rare event.And there it seems we take the
list_lockand call kmalloc while holding that lock.Perform the allocation in
free_partial()before thelist_lockis taken.Fixes bbd7d57bfe852d9788bae5fb171c7edb4021d8ac ("slub: Potential stack overflow")Signed-off-by Christoph Lameter <[email protected]>Index:
linux/mm/slub.c
===================================================================mm/slub.c +1
-4@@ -3690,14 +3690,11 @@ error: }static void list_slab_objects (struct kmem_cache *s,struct page *page, -const char *text) +const char *text,unsigned long *map) {#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); -if (!map) -return ;slab_err (s, page, text, s->name);slab_lock (page);
mm/slub.c static void free_partial(struct kmem_cac · line 3726
Quoted 7 lines
mm/slub.c +4
-0@@ -3723,6 +3720,10 @@ static void free_partial(struct kmem_cac {LIST_HEAD (discard);struct page *page, *h; +unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); + +if (!map) +return ;
What would happen if we are trying to allocate from the slab that is being shut down? And shouldn't the allocation be conditional (i.e., only when CONFIG_SLUB_DEBUG=y)?
Quoted 9 lines · Yu Zhao
mm/slub.c static void free_partial(struct kmem_cac · line 3726
Quoted 5 lines
mm/slub.c +4
-0struct page *page, *h; +unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); + +if (!map) +return ;What would happen if we are trying to allocate from the slab that is being shut down? And shouldn't the allocation be conditional (i.e., only when
CONFIG_SLUB_DEBUG=y)?
Kmalloc slabs are never shut down.
The allocation does not hurt and CONFIG_SLUB_DEBUG is on in most configurations.
Quoted 12 lines · Christopher Lameter
Quoted 9 lines · Yu Zhao
mm/slub.c static void free_partial(struct kmem_cac · line 3726
Quoted 5 lines
mm/slub.c +4
-0struct page *page, *h; +unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); + +if (!map) +return ;What would happen if we are trying to allocate from the slab that is being shut down? And shouldn't the allocation be conditional (i.e., only when
CONFIG_SLUB_DEBUG=y)?
Quoted 1 line
Kmalloc slabs are never shut down.
Maybe I'm not thinking straight -- isn't it what caused the deadlock in the first place?
Kmalloc slabs can be shut down when memcg is on.
Quoted 13 lines · Yu Zhao
Quoted 12 lines · Christopher Lameter
Quoted 9 lines · Yu Zhao
mm/slub.c static void free_partial(struct kmem_cac · line 3726
Quoted 5 lines
mm/slub.c +4
-0struct page *page, *h; +unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); + +if (!map) +return ;What would happen if we are trying to allocate from the slab that is being shut down? And shouldn't the allocation be conditional (i.e., only when
CONFIG_SLUB_DEBUG=y)?
Quoted 4 lines
Quoted 1 line
Kmalloc slabs are never shut down.
Maybe I'm not thinking straight -- isn't it what caused the deadlock in the first place?
Well if kmalloc allocations become a problem then we have numerous issues all over the kernel to fix.
Quoted 1 line
Kmalloc slabs can be shut down when memcg is on.
Kmalloc needs to work even during shutdown of a memcg.
Maybe we need to fix memcg to not allocate from the current memcg during shutdown?
[FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Regardless of the issue with memcgs allowing allocations from its kmalloc array during shutdown: This patch cleans things up and properly allocates the bitmap outside of the list_lock.
[FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
V1->V2 : Properly handle CONFIG_SLUB_DEBUG. Handle bitmap free correctly.
list_slab_objects() is called when a slab is destroyed and there are objects still left to list the objects in the syslog. This is a pretty rare event.
And there it seems we take the list_lock and call kmalloc while holding that lock.
Perform the allocation in free_partial() before the list_lock is taken.
mm/slub.c +14 -4
@@ -3690,14 +3690,15 @@ error: }static void list_slab_objects (struct kmem_cache *s,struct page *page, -const char *text) +const char *text,unsigned long *map) {#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); +if (!map)return ; +slab_err (s, page, text, s->name);slab_lock (page);
@@ -3710,7 +3711,6 @@ static void list_slab_objects(struct kme } }slab_unlock (page); -bitmap_free (map);#endif }
@@ -3723,6 +3723,11 @@ static void free_partial(struct kmem_cac {LIST_HEAD (discard);struct page *page, *h; +unsigned long *map =NULL ; + +#ifdef CONFIG_SLUB_DEBUG + map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); +#endif BUG_ON (irqs_disabled ());spin_lock_irq (&n->list_lock);
@@ -3732,11 +3737,16 @@ static void free_partial(struct kmem_caclist_add (&page->slab_list, &discard); }else {list_slab_objects (s, page, -"Objects remaining in %s on __kmem_cache_shutdown()" ); +"Objects remaining in %s on __kmem_cache_shutdown()" , + map); } }spin_unlock_irq (&n->list_lock); +#ifdef CONFIG_SLUB_DEBUG +bitmap_free (map); +#endif +list_for_each_entry_safe (page, h, &discard, slab_list)discard_slab (s, page); }
Andrew Morton
maintainerRe: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Quoted 7 lines · On Mon, 11 Nov 2019 15:55:05 +0000 (UTC) Christopher Lameter <[email protected]>…
Regardless of the issue with memcgs allowing allocations from its kmalloc array during shutdown: This patch cleans things up and properly allocates the bitmap outside of the
list_lock.[FIX] slub: Remove kmalloc under
list_lockfromlist_slab_objects()V2
Quoted 8 lines
V1->V2: Properly handleCONFIG_SLUB_DEBUG. Handle bitmap free correctly.
list_slab_objects()is called when a slab is destroyed and there are objects still left to list the objects in the syslog. This is a pretty rare event.And there it seems we take the
list_lockand call kmalloc while holding that lock.Perform the allocation in
free_partial()before thelist_lockis taken.
No response here? It looks a lot simpler than the originally proposed patch?
Quoted 58 lines
mm/slub.c +14
-4@@ -3690,14 +3690,15 @@ error: }static void list_slab_objects (struct kmem_cache *s,struct page *page, -const char *text) +const char *text,unsigned long *map) {#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC ); +if (!map)return ; +slab_err (s, page, text, s->name);slab_lock (page);@@ -3710,7 +3711,6 @@ static void list_slab_objects(struct kme } }slab_unlock (page); -bitmap_free (map);#endif }@@ -3723,6 +3723,11 @@ static void free_partial(struct kmem_cac {LIST_HEAD (discard);struct page *page, *h; +unsigned long *map =NULL ; + +#ifdef CONFIG_SLUB_DEBUG + map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); +#endif BUG_ON (irqs_disabled ());spin_lock_irq (&n->list_lock);@@ -3732,11 +3737,16 @@ static void free_partial(struct kmem_caclist_add (&page->slab_list, &discard); }else {list_slab_objects (s, page, -"Objects remaining in %s on __kmem_cache_shutdown()" ); +"Objects remaining in %s on __kmem_cache_shutdown()" , + map); } }spin_unlock_irq (&n->list_lock); +#ifdef CONFIG_SLUB_DEBUG +bitmap_free (map); +#endif +list_for_each_entry_safe (page, h, &discard, slab_list)discard_slab (s, page); }
Re: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Quoted 10 lines · Andrew Morton
Quoted 1 line
Perform the allocation in
free_partial()before thelist_lockis taken.No response here? It looks a lot simpler than the originally proposed patch?
Quoted 5 lines
mm/slub.c +0
-0@@ -3690,14 +3690,15 @@ error: }
Quoted 8 lines
mm/slub.c error: · line 3698
Quoted 8 lines
mm/slub.c +1
-2static void list_slab_objects (struct kmem_cache *s,struct page *page, -const char *text) +const char *text,unsigned long *map) {#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC );
Changing from !(__GFP_IO | __GFP_FS) allocation to
Quoted 7 lines
Quoted 7 lines
mm/slub.c +2
-0+if (!map)return ; +slab_err (s, page, text, s->name);slab_lock (page);
Quoted 8 lines
mm/slub.c static void free_partial(struct kmem_cac · line 3729
Quoted 8 lines
mm/slub.c +4
-0@@ -3723,6 +3723,11 @@ static void free_partial(struct kmem_cac {LIST_HEAD (discard);struct page *page, *h; +unsigned long *map =NULL ; + +#ifdef CONFIG_SLUB_DEBUG + map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL );
__GFP_IO | __GFP_FS allocation.
How is this path guaranteed to be safe to perform __GFP_IO | __GFP_FS reclaim?
Quoted 4 lines
Quoted 4 lines
mm/slub.c +1
-0+#endif BUG_ON (irqs_disabled ());spin_lock_irq (&n->list_lock);
Re: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
In reply to Andrew MortonQuoted 4 lines · Andrew Morton
Quoted 1 line
Perform the allocation in
free_partial()before thelist_lockis taken.No response here? It looks a lot simpler than the originally proposed patch?
Yup. I prefer this one but its my own patch so I cannot Ack this.
Re: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Quoted 8 lines · Christopher Lameter
Quoted 4 lines · Andrew Morton
Quoted 1 line
Perform the allocation in
free_partial()before thelist_lockis taken.No response here? It looks a lot simpler than the originally proposed patch?
Yup. I prefer this one but its my own patch so I cannot Ack this.
Hi, there is a pending question from Tetsuo-san. I'd be happy to ack once it's address.
Re: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Quoted 5 lines · Yu Zhao
Quoted 4 lines · Christopher Lameter
Quoted 2 lines · Andrew Morton
Quoted 1 line
Perform the allocation in
free_partial()before thelist_lockis taken.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
No response here? It looks a lot simpler than the originally proposed patch?
Yup. I prefer this one but its my own patch so I cannot Ack this.
Hi, there is a pending question from Tetsuo-san. I'd be happy to ack once it's address.
Tetsuo's mails don't reach linux-mm for a while and he has given up trying to do something about it. It's hard to discuss anything outside the direct CC group then. I don't know what's the pending question, for example.
Re: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Quoted 11 lines · Vlastimil Babka
Quoted 10 lines · Yu Zhao
Quoted 8 lines · Christopher Lameter
Quoted 4 lines · Andrew Morton
Quoted 1 line
Perform the allocation in
free_partial()before thelist_lockis taken.No response here? It looks a lot simpler than the originally proposed patch?
Yup. I prefer this one but its my own patch so I cannot Ack this.
Quoted 8 lines
Quoted 2 lines
Hi, there is a pending question from Tetsuo-san. I'd be happy to ack once it's address.
Tetsuo's mails don't reach linux-mm for a while and he has given up trying to do something about it. It's hard to discuss anything outside the direct CC group then. I don't know what's the pending question, for example.
Hmm, this one? Even non-ML destinations are sometimes rejected (e.g.
554 5.7.1 Service unavailable; Client host [202.181.97.72] blocked using b.barracudacentral.org; http://www.barracudanetworks.com/reputation/?pr=1&ip=202.181.97.72
). Anyway, I just worried whether it is really safe to do memory allocation
which might involve memory reclaim. You MM guys know better...
-------- Forwarded Message --------
Subject: Re: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Message-ID: <[email protected]>
Date: Sun, 1 Dec 2019 10:17:38 +0900
Quoted 10 lines · Andrew Morton
Quoted 1 line
Perform the allocation in
free_partial()before thelist_lockis taken.No response here? It looks a lot simpler than the originally proposed patch?
Quoted 5 lines
mm/slub.c +0
-0@@ -3690,14 +3690,15 @@ error: }
Quoted 8 lines
mm/slub.c error: · line 3698
Quoted 8 lines
mm/slub.c +1
-2static void list_slab_objects (struct kmem_cache *s,struct page *page, -const char *text) +const char *text,unsigned long *map) {#ifdef CONFIG_SLUB_DEBUG void *addr =page_address (page);void *p; -unsigned long *map =bitmap_zalloc (page->objects,GFP_ATOMIC );
Changing from !(__GFP_IO | __GFP_FS) allocation to
Quoted 7 lines
Quoted 7 lines
mm/slub.c +2
-0+if (!map)return ; +slab_err (s, page, text, s->name);slab_lock (page);
Quoted 8 lines
mm/slub.c static void free_partial(struct kmem_cac · line 3729
Quoted 8 lines
mm/slub.c +4
-0@@ -3723,6 +3723,11 @@ static void free_partial(struct kmem_cac {LIST_HEAD (discard);struct page *page, *h; +unsigned long *map =NULL ; + +#ifdef CONFIG_SLUB_DEBUG + map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL );
__GFP_IO | __GFP_FS allocation.
How is this path guaranteed to be safe to perform __GFP_IO | __GFP_FS reclaim?
Quoted 4 lines
Quoted 4 lines
mm/slub.c +1
-0+#endif BUG_ON (irqs_disabled ());spin_lock_irq (&n->list_lock);
Re: [FIX] slub: Remove kmalloc under list_lock from list_slab_objects() V2
Quoted 5 lines · Tetsuo Handa
On 2020/01/10 23:11, Vlastimil Babka wrote:
Hmm, this one? Even non-ML destinations are sometimes rejected (e.g.
554 5.7.1 Service unavailable; Client host [202.181.97.72] blocked using b.barracudacentral.org; http://www.barracudanetworks.com/reputation/?pr=1&ip=202.181.97.72
). Anyway, I just worried whether it is really safe to do memory allocation
which might involve memory reclaim. You MM guys know better...
We are talking about a call to destroy the kmem_cache. This is not done under any lock. The lock was taken inside that function before the call to list_slab_objects. That can be avoided.
+Roman Gushchin
Quoted 24 lines · Christopher Lameter
Quoted 9 lines · Yu Zhao
Quoted 8 lines · Christopher Lameter
Quoted 6 lines · Yu Zhao
Quoted 5 lines
struct page *page, *h;mm/slub.c +4
-0+unsigned long *map =bitmap_alloc (oo_objects (s->max),GFP_KERNEL ); + +if (!map) +return ;Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What would happen if we are trying to allocate from the slab that is being shut down? And shouldn't the allocation be conditional (i.e., only when
CONFIG_SLUB_DEBUG=y)?Kmalloc slabs are never shut down.
Maybe I'm not thinking straight -- isn't it what caused the deadlock in the first place?
Well if kmalloc allocations become a problem then we have numerous issues all over the kernel to fix.
Quoted 8 lines
Quoted 1 line
Kmalloc slabs can be shut down when memcg is on.
Kmalloc needs to work even during shutdown of a memcg.
Maybe we need to fix memcg to not allocate from the current memcg during shutdown?
Roman recently added reparenting of memcg kmem caches on memcg offline and can comment in more detail but we don't shutdown a kmem cache until all the in-fly memcg allocations are resolved. Also the allocation here does not look like a __GFP_ACCOUNT allocation.
PATCH 3/3 mm: lock slub page when listing objects
In reply to Yu ZhaoThough I have no idea what the side effect of a race would be, apparently we want to prevent the free list from being changed while debugging objects in general.
--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/slub.c +4 -0
@@ -4594,10 +4594,14 @@ static void process_slab(struct loc_track *t, struct kmem_cache *s,void *addr =page_address (page);void *p; +slab_lock (page); +get_map (s, page, map);for_each_object (p, s, addr, page->objects)if (!test_bit (slab_index (p, s, addr), map))add_location (t, s,get_track (s, p, alloc)); + +slab_unlock (page); } static int list_locations(struct kmem_cache *s, char *buf,
Signature · 1 line
2.23.0.162.g0b9fbb3734-goog