slub: Do not assert not having lock in removing freed partial
Subsystems (2)
PATCH 1/1 slub: Do not assert not having lock in removing freed partial
Vladimir reported the following issue:
Commit c65c1877bd68 ("slub: use lockdep_assert_held") requires remove_partial() to be called with n->list_lock held, but free_partial() called from kmem_cache_close() on cache destruction does not follow this rule, leading to a warning:
WARNING: CPU: 0 PID: 2787 at mm/slub.c:1536 __kmem_cache_shutdown+0x1b2/0x1f0() Modules linked in: CPU: 0 PID: 2787 Comm: modprobe Tainted: G W 3.14.0-rc1-mm1+ #1 Hardware name: 0000000000000600 ffff88003ae1dde8 ffffffff816d9583 0000000000000600 0000000000000000 ffff88003ae1de28 ffffffff8107c107 0000000000000000 ffff880037ab2b00 ffff88007c240d30 ffffea0001ee5280 ffffea0001ee52a0 Call Trace: [<ffffffff816d9583>] dump_stack+0x51/0x6e [<ffffffff8107c107>] warn_slowpath_common+0x87/0xb0 [<ffffffff8107c145>] warn_slowpath_null+0x15/0x20 [<ffffffff811c7fe2>] __kmem_cache_shutdown+0x1b2/0x1f0 [<ffffffff811908d3>] kmem_cache_destroy+0x43/0xf0 [<ffffffffa013a123>] xfs_destroy_zones+0x103/0x110 [xfs] [<ffffffffa0192b54>] exit_xfs_fs+0x38/0x4e4 [xfs] [<ffffffff811036fa>] SyS_delete_module+0x19a/0x1f0 [<ffffffff816dfcd8>] ? retint_swapgs+0x13/0x1b [<ffffffff810d2125>] ? trace_hardirqs_on_caller+0x105/0x1d0 [<ffffffff81359efe>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff816e8539>] system_call_fastpath+0x16/0x1b
His solution was to add a spinlock in order to quiet lockdep. Although there would be no contention to adding the lock, that lock also requires disabling of interrupts which will have a larger impact on the system.
Instead of adding a spinlock to a location where it is not needed for lockdep, make a remove_freed_partial() function that does not test if the list_lock is held, as no one should have it due to it being freed.
mm/slub.c +20 -3
@@ -1530,13 +1530,30 @@ static inline void add_partial(struct kmlist_add (&page->lru, &n->partial); } +static __always_inline void +__remove_partial (struct kmem_cache_node *n,struct page *page) +{ +list_del (&page->lru); + n->nr_partial--; +} +static inline void remove_partial (struct kmem_cache_node *n,struct page *page) {lockdep_assert_held (&n->list_lock); +__remove_partial (n, page); +} -list_del (&page->lru); - n->nr_partial--; +/* +* The difference between remove_partial and remove_freed_partial +* is that remove_freed_partial happens only on a a freed slab +* that should not have anyone accessing it, and thus does not +* require the n->list_lock. +*/ +static inline void remove_freed_partial (struct kmem_cache_node *n, +struct page *page) +{ +__remove_partial (n, page); }/*
@@ -3195,7 +3212,7 @@ static void free_partial(struct kmem_caclist_for_each_entry_safe (page, h, &n->partial, lru) {if (!page->inuse) { -remove_partial (n, page); +remove_freed_partial (n, page);discard_slab (s, page); }else { list_slab_objects(s, page,
David Rientjes
reviewerQuoted 58 lines · Steven Rostedt
Vladimir reported the following issue:
Commit
c65c1877bd68("slub: uselockdep_assert_held") requiresremove_partial()to be called withn->list_lockheld, butfree_partial()called fromkmem_cache_close()on cache destruction does not follow this rule, leading to a warning:WARNING: CPU: 0 PID: 2787 at mm/slub.c:1536 __kmem_cache_shutdown+0x1b2/0x1f0() Modules linked in: CPU: 0 PID: 2787 Comm: modprobe Tainted: G W 3.14.0-rc1-mm1+ #1 Hardware name: 0000000000000600 ffff88003ae1dde8 ffffffff816d9583 0000000000000600 0000000000000000 ffff88003ae1de28 ffffffff8107c107 0000000000000000 ffff880037ab2b00 ffff88007c240d30 ffffea0001ee5280 ffffea0001ee52a0 Call Trace: [<ffffffff816d9583>] dump_stack+0x51/0x6e [<ffffffff8107c107>] warn_slowpath_common+0x87/0xb0 [<ffffffff8107c145>] warn_slowpath_null+0x15/0x20 [<ffffffff811c7fe2>] __kmem_cache_shutdown+0x1b2/0x1f0 [<ffffffff811908d3>] kmem_cache_destroy+0x43/0xf0 [<ffffffffa013a123>] xfs_destroy_zones+0x103/0x110 [xfs] [<ffffffffa0192b54>] exit_xfs_fs+0x38/0x4e4 [xfs] [<ffffffff811036fa>] SyS_delete_module+0x19a/0x1f0 [<ffffffff816dfcd8>] ? retint_swapgs+0x13/0x1b [<ffffffff810d2125>] ? trace_hardirqs_on_caller+0x105/0x1d0 [<ffffffff81359efe>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff816e8539>] system_call_fastpath+0x16/0x1bHis solution was to add a spinlock in order to quiet lockdep. Although there would be no contention to adding the lock, that lock also requires disabling of interrupts which will have a larger impact on the system.
Instead of adding a spinlock to a location where it is not needed for lockdep, make a
remove_freed_partial()function that does not test if thelist_lockis held, as no one should have it due to it being freed.Reported-by Vladimir Davydov <[email protected]>Signed-off-by Steven Rostedt <[email protected]>Index:
linux-trace.git/mm/slub.c
===================================================================mm/slub.c +7
-0@@ -1530,13 +1530,30 @@ static inline void add_partial(struct kmlist_add (&page->lru, &n->partial); } +static __always_inline void +__remove_partial (struct kmem_cache_node *n,struct page *page) +{ +list_del (&page->lru); + n->nr_partial--; +} +static inline void remove_partial (struct kmem_cache_node *n,struct page *page)
mm/slub.c static inline void add_partial(struct km · line 1549
Quoted 10 lines
mm/slub.c +5
-2{lockdep_assert_held (&n->list_lock); +__remove_partial (n, page); +} -list_del (&page->lru); - n->nr_partial--; +/* +* The difference between remove_partial and remove_freed_partial +* is that remove_freed_partial happens only on a a freed slab
Duplicate "a" there.
Quoted 11 lines
mm/slub.c +7
-0+* that should not have anyone accessing it, and thus does not +* require the n->list_lock. +*/ +static inline void remove_freed_partial (struct kmem_cache_node *n, +struct page *page) +{ +__remove_partial (n, page); }/*
mm/slub.c static void free_partial(struct kmem_cac · line 3218
Quoted 8 lines
mm/slub.c +1
-1@@ -3195,7 +3212,7 @@ static void free_partial(struct kmem_caclist_for_each_entry_safe (page, h, &n->partial, lru) {if (!page->inuse) { -remove_partial (n, page); +remove_freed_partial (n, page);discard_slab (s, page); }else { list_slab_objects(s, page,
We'll want to do something similiar for the add_partial() called from early_kmem_cache_node_alloc(), right? It had the added n->list_lock for the same reason and is done during early init where nobody else can be referencing a kmem_cache_node.
It would probably be better to define these in terms of "partial slabs that cannot have anyone else accessing it" rather than "freed slabs".
Quoted 5 lines · On Wed, 5 Feb 2014 16:46:43 -0800 (PST) David Rientjes <[email protected]>…
Quoted 3 lines
+/*
+ * The difference betweenremove_partialandremove_freed_partial
+ * is thatremove_freed_partialhappens only on a a freed slabDuplicate "a" there.
oops.
Quoted 26 lines
Quoted 12 lines
+ * that should not have anyone accessing it, and thus does not
+ * require then->list_lock.
+ */mm/slub.c +4
-0+static inline void remove_freed_partial (struct kmem_cache_node *n, +struct page *page) +{ +__remove_partial (n, page); }/* mm/slub.c static void free_partial(struct kmem_cac · line 3218
Quoted 8 lines
mm/slub.c +1
-1@@ -3195,7 +3212,7 @@ static void free_partial(struct kmem_caclist_for_each_entry_safe (page, h, &n->partial, lru) {if (!page->inuse) { -remove_partial (n, page); +remove_freed_partial (n, page);discard_slab (s, page); }else { list_slab_objects(s, page,We'll want to do something similiar for the
add_partial()called fromearly_kmem_cache_node_alloc(), right? It had the addedn->list_lockfor the same reason and is done during early init where nobody else can be referencing akmem_cache_node.
Quoted 2 lines
It would probably be better to define these in terms of "partial slabs that cannot have anyone else accessing it" rather than "freed slabs".
Perhaps then we just use the __remove_partial() and __add_partial() that does not do the checks. That's common practice to use a "__" to denote that it's special and usually doesn't require locking.
-- Steve
David Rientjes
reviewerQuoted 5 lines · Steven Rostedt
Quoted 5 lines
We'll want to do something similiar for the
add_partial()called fromearly_kmem_cache_node_alloc(), right? It had the addedn->list_lockfor the same reason and is done during early init where nobody else can be referencing akmem_cache_node.
Quoted 7 lines
Quoted 2 lines
It would probably be better to define these in terms of "partial slabs that cannot have anyone else accessing it" rather than "freed slabs".
Perhaps then we just use the
__remove_partial()and__add_partial()that does not do the checks. That's common practice to use a "__" to denote that it's special and usually doesn't require locking.
Sounds appropriate. Andrew released a mmotm today so I'm assuming it will be in linux-next tomorrow with my {add,remove}_full() patch but it shouldn't matter if you're going to be playing with {add,remove}_partial().