Skip to contentSkip to conversation

2.6.6 is crashing repeatedly

DISCUSSION [email protected] 3 participants 4 messages May 15, 2004

lore
Share & tools
lore
#0 lore raw

[All this time and I still havitually send e-mail to vger.rutgers.edu...]

Sorry for the delay in getting this.

I have now captured a kernel crash. Everything after iput in the second crash was hand-coped, and may suffer from transcription errors, but it was done quite carefully.

System has ECC memory and has been very stable, with uptimes in excess of 1 year when kernel upgrades were infrequent (2.5 development).

Stock 2.6.6 kernel, config as posted before.

Unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c012a392
*pde = 00000000
Oops: 0002 [#1]
CPU:    0
EIP:    0060:[<c012a392>]    Not tainted
EFLAGS: 00010012   (2.6.6) 

EIP is at free_block+0x52/0xd0
eax: 00000000 ebx: e9a3f000 ecx: e9a3f200 edx: df654000
esi: f7f8a560 edi: 00000016 ebp: f7f8a56c esp: f7d89dec
ds: 007b es: 007b ss: 0068
Process kswapd0 (pid: 8, threadinfo=f7d88000 task=f7d8eb50)

Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 dc3cdac0 0000001b c012a449 
       f7fe73dc c17fd774 c17fd774 00000296 dc3cdac0 c037a304 c012a61a dc3cdb40 
       f7d89e5c 0000003d c014f385 dc3cdb40 c014f5c3 dc19c0c8 dc19c0c0 00000080 
Call Trace:
 [<c012a449>] cache_flusharray+0x39/0xc0
 [<c012a61a>] kmem_cache_free+0x3a/0x50
 [<c014f385>] destroy_inode+0x35/0x40
 [<c014f5c3>] dispose_list+0x43/0x70
 [<c014f87e>] prune_icache+0xae/0x1b0
 [<c014f995>] shrink_icache_memory+0x15/0x20
 [<c012bfa9>] shrink_slab+0x129/0x180
 [<c012d0b3>] balance_pgdat+0x1f3/0x260
 [<c012d20e>] kswapd+0xee/0xf0
 [<c0110e30>] autoremove_wake_function+0x0/0x40
 [<c0110e30>] autoremove_wake_function+0x0/0x40
 [<c012d120>] kswapd+0x0/0xf0
 [<c0101fbd>] kernel_thread_helper+0x5/0x18

Code: 89 50 04 89 02 8b 43 0c 31 d2 c7 03 00 01 10 00 c7 43 04 00 
 <1>Unable to handle kernel paging request at virtual address 00200204
 printing eip:
c012a38d
*pde = 00000000
Oops: 0000 [#2]
CPU:    0
EIP:    0060:[<c012a38d>]    Not tainted
EFLAGS: 00010012   (2.6.6) 

EIP is at free_block+0x4d/0xd0
eax: 00383380 ebx: 00200200 ecx: dc19cac0 edx: c1000000
esi: f7f8a560 edi: 00000000 ebp: f7f8a56c esp: f42f5f20
ds: 007b es: 007b ss: 0068
Process qmail-clean (pid: 1027, threadinfo=f42f4000 task=f433c6f0)

Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 da337200 0000001b c012a449 
       f7fe73dc c17fd774 c17fd774 00000296 da337200 f42f5f80 c012a61a da337280 
       00000000 da337280 c014f385 da337280 c01501ac f6e4f000 c0147f44 ec9fc8b8 
Call Trace:
 [<c012a449>] cache_flusharray+0x39/0xc0
 [<c012a61a>] kmem_cache_free+0x3a/0x50
 [<c014f385>] destroy_inode+0x35/0x40
 [<c01501ac>] iput+0x4c/0x60
 [<c0147f44>] sys_unlink+0xf4/0x120
 [<c013b538>] sys_read+0x38/0x60
 [<c01039ff>] syscall_call+0x7/0xb

Code: 8b 53 04 8b 03 89 50 04 89 02 86 43 0c 31 d2 c7 03 00 01 10
 <1>Unable to handle kernel paging request at virtual address 00200204

<cascading errors not copied, ending with>

Kernel panic: Fatal exception in interrupt
#1 lore raw
Quoted 25 lines · [email protected] wrote:

I have now captured a kernel crash. Everything after iput in the second crash was hand-coped, and may suffer from transcription errors, but it was done quite carefully.

System has ECC memory and has been very stable, with uptimes in excess of 1 year when kernel upgrades were infrequent (2.5 development).

Stock 2.6.6 kernel, config as posted before.

 Unable to handle kernel NULL pointer dereference at virtual address 00000004
  printing eip:
 c012a392
 *pde = 00000000
 Oops: 0002 [#1]
 CPU:    0
 EIP:    0060:[<c012a392>]    Not tainted
 EFLAGS: 00010012   (2.6.6) 
 EIP is at free_block+0x52/0xd0
 eax: 00000000   ebx: e9a3f000   ecx: e9a3f200   edx: df654000
 esi: f7f8a560   edi: 00000016   ebp: f7f8a56c   esp: f7d89dec
 ds: 007b   es: 007b   ss: 0068
 Process kswapd0 (pid: 8, threadinfo=f7d88000 task=f7d8eb50)
 Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 dc3cdac0 0000001b c012a449 
        f7fe73dc c17fd774 c17fd774 00000296 dc3cdac0 c037a304 c012a61a dc3cdb40 
Quoted 8 lines
        f7d89e5c 0000003d c014f385 dc3cdb40 c014f5c3 dc19c0c8 dc19c0c0 00000080 
 Call Trace:
  [<c012a449>] cache_flusharray+0x39/0xc0
  [<c012a61a>] kmem_cache_free+0x3a/0x50
  [<c014f385>] destroy_inode+0x35/0x40
  [<c014f5c3>] dispose_list+0x43/0x70
  [<c014f87e>] prune_icache+0xae/0x1b0
  [<c014f995>] shrink_icache_memory+0x15/0x20

Drat, random memory corruption.

Can you enable CONFIG_SLAB_DEBUG? And CONFIG_DEBUG_PAGEALLOC too, although beware that the latter is a bit costly in terms of CPU cycles and memory usage.

#2 lore raw
Quoted 3 lines

Can you enable CONFIG_SLAB_DEBUG? And CONFIG_DEBUG_PAGEALLOC too, although beware that the latter is a bit costly in terms of CPU cycles and memory usage.

Just rebooted with
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_PAGEALLOC=y

More news when something happens.

Thanks for your instant diagnosis!

#3 lore raw
In reply to Andrew Morton
Quoted 41 lines · Andrew Morton

Date: Fri, 14 May 2004 23:28:42 -0700
From: Andrew Morton <[email protected]>
To: [email protected]
Cc: [email protected]
Subject: Re: 2.6.6 is crashing repeatedly

Quoted 25 lines · [email protected] wrote:

I have now captured a kernel crash. Everything after iput in the second crash was hand-coped, and may suffer from transcription errors, but it was done quite carefully.

System has ECC memory and has been very stable, with uptimes in excess of 1 year when kernel upgrades were infrequent (2.5 development).

Stock 2.6.6 kernel, config as posted before.

 Unable to handle kernel NULL pointer dereference at virtual address 00000004
  printing eip:
 c012a392
 *pde = 00000000
 Oops: 0002 [#1]
 CPU:    0
 EIP:    0060:[<c012a392>]    Not tainted
 EFLAGS: 00010012   (2.6.6)
 EIP is at free_block+0x52/0xd0
 eax: 00000000   ebx: e9a3f000   ecx: e9a3f200   edx: df654000
 esi: f7f8a560   edi: 00000016   ebp: f7f8a56c   esp: f7d89dec
 ds: 007b   es: 007b   ss: 0068
 Process kswapd0 (pid: 8, threadinfo=f7d88000 task=f7d8eb50)
 Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 dc3cdac0 0000001b c012a449
        f7fe73dc c17fd774 c17fd774 00000296 dc3cdac0 c037a304 c012a61a dc3cdb40
Quoted 8 lines
        f7d89e5c 0000003d c014f385 dc3cdb40 c014f5c3 dc19c0c8 dc19c0c0 00000080
 Call Trace:
  [<c012a449>] cache_flusharray+0x39/0xc0
  [<c012a61a>] kmem_cache_free+0x3a/0x50
  [<c014f385>] destroy_inode+0x35/0x40
  [<c014f5c3>] dispose_list+0x43/0x70
  [<c014f87e>] prune_icache+0xae/0x1b0
  [<c014f995>] shrink_icache_memory+0x15/0x20
Quoted 2 lines

Drat, random memory corruption.

Interesting...

FWIW, a data point: I've been chasing memory corruption in 2.6.5-mm2. (Although I suspect it pre-dates that level.)

The first 2K of a DMA page is overlayed w what looks like code (even disassembles to something almost meaningful, looking like interrupt handling code ie w several iret's. Althought I've yet to find where exactly it's from.) I obtain dma mem from the page via dma_pool_alloc, having created the pool w dma_pool_create. The hi 2k of the page w the overlay, remains in tact.

I've been running w CONFIG_SLAB_DEBUG And CONFIG_DEBUG_PAGEALLOC usually to no avail. However I do sometimes find freed memory 0xA7 poison words occasionally involved. I temporarily commented out the dma_pool_free that would release the memory and I still get the 2K overlay.

johnh

Quoted 10 lines

Can you enable CONFIG_SLAB_DEBUG? And CONFIG_DEBUG_PAGEALLOC too, although beware that the latter is a bit costly in terms of CPU cycles and memory usage.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
-----------------------------------------------------------------
John Heil
South Coast Software
Custom systems software for UNIX and IBM MVS mainframes 1-714-774-6952
[email protected]
http://www.sc-software.com
-----------------------------------------------------------------