SLUB BUG: check_slab called with interrupts enabled

Hi Christoph,

last night I got an interesting backtrace running 3.0-rc3 (Fedora Rawhide kernel package). Unfortunately netconsole seems to be incompatible with KVM at the moment, so I had to capture the oops on my digital camera and will be transcribing just the backtrace.

Essentially, kernel 3.0-rc3 hit this bug:

static int check_slab(struct kmem_cache *s, struct page *page)

{
        int maxobj;

        VM_BUG_ON(!irqs_disabled());

The call trace:

check_slab
alloc_debug_processing
__slab_alloc
kmem_cache_alloc
bvec_alloc_bs
bio_alloc_bioset
bio_alloc
mpage_alloc
do_mpage_readpage
... followed by ext4 and VFS code, obviously innocent

Is this a known issue, Christoph?

If not, anything I can do to help debug/fix this?

All rights reversed

__slab_alloc() disables interrupts so alloc_debug_processing() should not run into this issue.

There are no additional special slub patches applied right? Because some of the patches under discussion change the interrupt disable handling a bit.

Just the two attached ones, which don't seem to touch the code path in question...

All rights reversed

From [email protected] Fri May 13 10:04:18 2011
From: Mel Gorman <[email protected]>
To: Andrew Morton <[email protected]>
Cc: James Bottomley <[email protected]>,
Colin King <[email protected]>,
Raghavendra D Prabhu <[email protected]>,
Jan Kara <[email protected]>, Chris Mason <[email protected]>,
Christoph Lameter <[email protected]>,
Pekka Enberg <[email protected]>,
Rik van Riel <[email protected]>,
Johannes Weiner <[email protected]>,
linux-fsdevel <[email protected]>,
linux-mm <[email protected]>,
linux-kernel <[email protected]>,
linux-ext4 <[email protected]>,
Mel Gorman <[email protected]>
Subject: [PATCH 3/4] mm: slub: Do not take expensive steps for SLUBs speculative high-order allocations
Date: Fri, 13 May 2011 15:03:23 +0100
Message-Id: <[email protected]>
X-Mailing-List: [email protected]

To avoid locking and per-cpu overhead, SLUB optimisically uses high-order allocations and falls back to lower allocations if they fail. However, by simply trying to allocate, the caller can enter compaction or reclaim - both of which are likely to cost more than the benefit of using high-order pages in SLUB. On a desktop system, two users report that the system is getting stalled with kswapd using large amounts of CPU.

This patch prevents SLUB taking any expensive steps when trying to use high-order allocations. Instead, it is expected to fall back to smaller orders more aggressively. Testing was somewhat inconclusive on how much this helped but it makes sense that falling back to order-0 allocations is faster than entering compaction or direct reclaim.

---
 mm/page_alloc.c |    3 ++-
 mm/slub.c       |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

index 9f8a97b..057f1e2 100644
@@ -1972,6 +1972,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
 {
 	int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET;
 	const gfp_t wait = gfp_mask & __GFP_WAIT;
+	const gfp_t can_wake_kswapd = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */
 	BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH);

@@ -1984,7 +1985,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
 	 */
 	alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH);
 
-	if (!wait) {
+	if (!wait && can_wake_kswapd) {
 		/*
 		 * Not worth trying to allocate harder for
 		 * __GFP_NOMEMALLOC even if it can't schedule.

index 98c358d..c5797ab 100644
@@ -1170,7 +1170,8 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	 * Let the initial higher-order allocation fail under memory pressure
 	 * so we fall-back to the minimum order allocation.
 	 */
-	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
+	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NO_KSWAPD) &
+			~(__GFP_NOFAIL | __GFP_WAIT | __GFP_REPEAT);
 
 	page = alloc_slab_page(alloc_gfp, node, oo);
 	if (unlikely(!page)) {

1.7.3.4

From [email protected] Fri May 13 10:04:00 2011
From: Mel Gorman <[email protected]>
To: Andrew Morton <[email protected]>
Cc: James Bottomley <[email protected]>,
Colin King <[email protected]>,
Raghavendra D Prabhu <[email protected]>,
Jan Kara <[email protected]>, Chris Mason <[email protected]>, Christoph Lameter <[email protected]>,
Pekka Enberg <[email protected]>,
Rik van Riel <[email protected]>,
Johannes Weiner <[email protected]>,
linux-fsdevel <[email protected]>,
linux-mm <[email protected]>,
linux-kernel <[email protected]>,
linux-ext4 <[email protected]>,
Mel Gorman <[email protected]>
Subject: [PATCH 2/4] mm: slub: Do not wake kswapd for SLUBs speculative high-order allocations
Date: Fri, 13 May 2011 15:03:22 +0100
Message-Id: <[email protected]>
X-Mailing-List: [email protected]

To avoid locking and per-cpu overhead, SLUB optimisically uses high-order allocations and falls back to lower allocations if they fail. However, by simply trying to allocate, kswapd is woken up to start reclaiming at that order. On a desktop system, two users report that the system is getting locked up with kswapd using large amounts of CPU. Using SLAB instead of SLUB made this problem go away.

This patch prevents kswapd being woken up for high-order allocations. Testing indicated that with this patch applied, the system was much harder to hang and even when it did, it eventually recovered.

---
 mm/slub.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

index 9d2e5e4..98c358d 100644
@@ -1170,7 +1170,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	 * Let the initial higher-order allocation fail under memory pressure
 	 * so we fall-back to the minimum order allocation.
 	 */
-	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
+	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
 
 	page = alloc_slab_page(alloc_gfp, node, oo);
 	if (unlikely(!page)) {

1.7.3.4

I also do not see how these could break something. But they are mucking around with the __GFP_WAIT flag. __GFP_WAIT determines the reenabling and redisabling of interrupts in __slab_alloc(). If some variables gets corrupted then this could be the result.

Print out the value of gfpflags before and after the call to new_slab() from __slab_alloc()?

From [email protected] Fri May 13 10:04:18 2011
From: Mel Gorman <[email protected]>
To: Andrew Morton <[email protected]>
Cc: James Bottomley <[email protected]>,
Colin King <[email protected]>,
Raghavendra D Prabhu <[email protected]>,
Jan Kara <[email protected]>, Chris Mason <[email protected]>,
Christoph Lameter <[email protected]>,
Pekka Enberg <[email protected]>,
Rik van Riel <[email protected]>,
Johannes Weiner <[email protected]>,
linux-fsdevel <[email protected]>,
linux-mm <[email protected]>,
linux-kernel <[email protected]>,
linux-ext4 <[email protected]>,
Mel Gorman <[email protected]>
Subject: [PATCH 3/4] mm: slub: Do not take expensive steps for SLUBs speculative high-order allocations
Date: Fri, 13 May 2011 15:03:23 +0100
Message-Id: <[email protected]>
X-Mailing-List: [email protected]

To avoid locking and per-cpu overhead, SLUB optimisically uses high-order allocations and falls back to lower allocations if they fail. However, by simply trying to allocate, the caller can enter compaction or reclaim - both of which are likely to cost more than the benefit of using high-order pages in SLUB. On a desktop system, two users report that the system is getting stalled with kswapd using large amounts of CPU.

This patch prevents SLUB taking any expensive steps when trying to use high-order allocations. Instead, it is expected to fall back to smaller orders more aggressively. Testing was somewhat inconclusive on how much this helped but it makes sense that falling back to order-0 allocations is faster than entering compaction or direct reclaim.

---
 mm/page_alloc.c |    3 ++-
 mm/slub.c       |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

index 9f8a97b..057f1e2 100644
@@ -1972,6 +1972,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
 {
 	int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET;
 	const gfp_t wait = gfp_mask & __GFP_WAIT;
+	const gfp_t can_wake_kswapd = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */
 	BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH);

@@ -1984,7 +1985,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
 	 */
 	alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH);
 
-	if (!wait) {
+	if (!wait && can_wake_kswapd) {
 		/*
 		 * Not worth trying to allocate harder for
 		 * __GFP_NOMEMALLOC even if it can't schedule.

index 98c358d..c5797ab 100644
@@ -1170,7 +1170,8 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	 * Let the initial higher-order allocation fail under memory pressure
 	 * so we fall-back to the minimum order allocation.
 	 */
-	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
+	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NO_KSWAPD) &
+			~(__GFP_NOFAIL | __GFP_WAIT | __GFP_REPEAT);
 
 	page = alloc_slab_page(alloc_gfp, node, oo);
 	if (unlikely(!page)) {

1.7.3.4

From [email protected] Fri May 13 10:04:00 2011
From: Mel Gorman <[email protected]>
To: Andrew Morton <[email protected]>
Cc: James Bottomley <[email protected]>,
Colin King <[email protected]>,
Raghavendra D Prabhu <[email protected]>,
Jan Kara <[email protected]>, Chris Mason <[email protected]>, Christoph Lameter <[email protected]>,
Pekka Enberg <[email protected]>,
Rik van Riel <[email protected]>,
Johannes Weiner <[email protected]>,
linux-fsdevel <[email protected]>,
linux-mm <[email protected]>,
linux-kernel <[email protected]>,
linux-ext4 <[email protected]>,
Mel Gorman <[email protected]>
Subject: [PATCH 2/4] mm: slub: Do not wake kswapd for SLUBs speculative high-order allocations
Date: Fri, 13 May 2011 15:03:22 +0100
Message-Id: <[email protected]>
X-Mailing-List: [email protected]

To avoid locking and per-cpu overhead, SLUB optimisically uses high-order allocations and falls back to lower allocations if they fail. However, by simply trying to allocate, kswapd is woken up to start reclaiming at that order. On a desktop system, two users report that the system is getting locked up with kswapd using large amounts of CPU. Using SLAB instead of SLUB made this problem go away.

This patch prevents kswapd being woken up for high-order allocations. Testing indicated that with this patch applied, the system was much harder to hang and even when it did, it eventually recovered.

---
 mm/slub.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

index 9d2e5e4..98c358d 100644
@@ -1170,7 +1170,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	 * Let the initial higher-order allocation fail under memory pressure
 	 * so we fall-back to the minimum order allocation.
 	 */
-	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
+	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
 
 	page = alloc_slab_page(alloc_gfp, node, oo);
 	if (unlikely(!page)) {

1.7.3.4

Since I'm building a new kernel anyway (with Neil Horman's patch to make netconsole work with tun/tap), I have commented out these two slub patches.

Lets see what happens...

All rights reversed

After backing them out, I got 18 hours of uptime so far.

This could just be dumb luck, or it could be some slub vs. kswapd interaction. Or maybe I simply am not triggering the original bug any more because kswapd is now doing all the work, but may still be able to trigger it under more memory pressure...

Either way, since this could still be dumb luck, I'll let you guys know if/when I see a next crash :)

All rights reversed

Could be some memory issue or stack corruption. This is a machine with ECC ram right?

OK.

Yes it is, 12GB of ECC memory.

Running that much without ECC is probably a bad idea :)

30 hours uptime already.

Just a little beyond the "dumb luck" threshold.

However, I suspect the bug may still be there and the fact that kswapd is doing more work may simply be hiding it.

All rights reversed