2.6.0-test7 DEBUG_PAGEALLOC oops

Fault trying to read from address 0xc034a000: the page is not mapped.

In store_stackinfo: the function stores a backtrace of the last kmem_cache_free caller in the object - might be useful, and the memory is not used.

The esp value is sane, the stack is at 0xc0348000, and the fault is at 'a000: just behind the end of the stack. I assume the fauling line is

                       svalue = *sptr++;

It looks like store stackinfo accesses memory behind the end of the stack. Which gcc version do you use? Could you send me mm/slab.o?

--
Manfred

Exactly.

Yeah, I'm trying to figure out why. The below (if dang mailer actually inlines it) kludge allows me to boot, so I suppose I need to ponder addr wrt _stext and _etext.

gcc-2.95.3. slab.o coming via private mail.

-Mike

@@ -864,12 +864,11 @@
 
 		while (((long) sptr & (THREAD_SIZE-1)) != 0) {
 			svalue = *sptr++;
-			if (kernel_text_address(svalue)) {
+			if (kernel_text_address(svalue))
 				*addr++=svalue;
-				size -= sizeof(unsigned long);
-				if (size <= sizeof(unsigned long))
-					break;
-			}
+			size -= sizeof(unsigned long);
+			if (size <= sizeof(unsigned long))
+				break;
 		}
 
 	}

I'm blind. The esp value is the culprit:
It's not 32-bit aligned. Someone misaligned the stack, and thus
if(stack_ptr & (THREAD_SIZE-1))
didn't notice the end of the stack.
The generated assembly of store_slabinfo is correct:

    1d2:       f7 c6 ff 1f 00 00       test   $0x1fff,%esi
Check sptr against THREAD_SIZE -1
    1d8:       74 21                   je     1fb <store_stackinfo+0x6f>
    1da:       8b 3e                   mov    (%esi),%edi

And load *sptr.

Wrong direction: Right now it crashes because it runs over the end of the stack.
With your patch applied, the allocated object is too small to hold all entries on the stack, and thus store_stackinfo aborts before it runs into the next page.

I'd increase kstack_depth_to_print to 140. Do not increase it too much, otherwise it will oops due to the misaligned stack. Then check the EBP values: They are pushed after the return address. The return addresses are listed in the Call Trace section. Example:
0xc01316aa8 pushes 0xc0349dd6 -> odd.
0xc0131b6c pushes 0xc0349de6 -> odd.

0xc0131b3e pushes c0349e02 -> odd.

Proper values for EBP are multiples of 4. One you find where the stack got misaligned, disassemble the offending function (or send me the .o file)

--
Manfred

Ok, you want do_IRQ assembler, correct?

fwiw, building with gcc-3.3 didn't help, nor did disabling frame pointers.

-Mike

Unable to handle kernel paging request at virtual address c034a000
 printing eip:
c0134d5a
*pde = 00102027
*pte = 0034a000
Oops: 0000 [#1]
CPU:    0
EIP:    0060:[<c0134d5a>]    Not tainted
EFLAGS: 00010002

EIP is at store_stackinfo+0x4e/0x80
eax: 00000000 ebx: c7436f88 ecx: c0301390 edx: c030138c
esi: c0349ffe edi: 017e0008 ebp: c0349d46 esp: c0349d36
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, threadinfo=c0348000 task=c02fcbe0)

Stack: c74b59c4 c1173488 c7436000 00000070 c0349d76 c0136aa8 c1173488 c7436000 
       0000006b c72b1bb4 c72b1bb4 c7fecafc 00001000 c0131b6c c7f75f78 00000086 
       c0349d86 c0131b6c c1173488 c7436f38 c0349da2 c0131b3e c7436f38 c1173488 
       c72b1bb4 c72b1bb4 00000001 c0349db6 c014cf76 c7436f38 c7fecafc c7436f2c 
       c0349dc2 c014d158 c72b1bb4 c0349dda c016548a c72b1bb4 c72b1bb4 00000000 
       c04a070c c0349df6 c014d830 c72b1bb4 00010000 00000000 00010000 c72b1bb4 
       c0349e26 c01e45af c72b1bb4 00010000 00000000 c7467f58 00000080 c04a070c 
       00000000 00000000 00000000 00000000 c0349e3a c01e4697 c7467f58 00000001 
       00010000 c0349e62 c01f0b7d c7467f58 00000001 00000080 c04a070c 00000000 
       c04a070c 00000001 00000296 c0349e76 c01f9f68 c04a070c 00000001 00000080 
       c0349e96 c0201df8 c04a070c 00000001 00000080 c0348000 c7af5ef8 50005ef8 
       c0349eba c01f2024 c04a070c c7b2ca48 04000001 00000000 c0201d98 c04a0660 
       00000292 c0349eda c010a423 0000000e c7af5ef8 c0349f0a c0348000 c0348000 
       0000000e c0349f02 c010a70e 0000000e c0349f0a c7b2ca48 00000000 00000010 
       c0348000 c7b2ca48 c0346bc0 000000c0 c0109014 00000000 00000000 00000000 
       00000010 c0348000 000000c0 00005305 9f280000 9f280000 ffffff0e 00008026 
       000000b8 00000216 803600c0 9f8800b8 2a13c034 0060c011 9f880000 8000c034 
       007bc034 007b0000 9c2c0000 0010fffb 
Call Trace:
 [<c0136aa8>] kmem_cache_free+0x218/0x294
 [<c0131b6c>] mempool_free_slab+0x10/0x14
 [<c0131b6c>] mempool_free_slab+0x10/0x14
 [<c0131b3e>] mempool_free+0x7a/0x84
 [<c014cf76>] bio_destructor+0x36/0x4c
 [<c014d158>] bio_put+0x2c/0x30
 [<c016548a>] mpage_end_io_read+0x6a/0x78
 [<c014d830>] bio_endio+0x50/0x5c
 [<c01e45af>] __end_that_request_first+0xef/0x1c0
 [<c01e4697>] end_that_request_first+0x17/0x1c
 [<c01f0b7d>] ide_end_request+0x8d/0x124
 [<c01f9f68>] default_end_request+0x14/0x18
 [<c0201df8>] ide_dma_intr+0x60/0x98
 [<c01f2024>] ide_intr+0x108/0x17c
 [<c0201d98>] ide_dma_intr+0x0/0x98
 [<c010a423>] handle_IRQ_event+0x2b/0x58
 [<c010a70e>] do_IRQ+0x92/0x130
 [<c0109014>] common_interrupt+0x18/0x20

000004ec <do_IRQ>:

 4ec:	55                   	push   %ebp
 4ed:	89 e5                	mov    %esp,%ebp
 4ef:	83 ec 08             	sub    $0x8,%esp
 4f2:	57                   	push   %edi
 4f3:	56                   	push   %esi
 4f4:	53                   	push   %ebx
 4f5:	be 00 e0 ff ff       	mov    $0xffffe000,%esi
 4fa:	0f b6 7d 2c          	movzbl 0x2c(%ebp),%edi
 4fe:	21 e6                	and    %esp,%esi
 500:	89 fb                	mov    %edi,%ebx
 502:	c1 e3 05             	shl    $0x5,%ebx
 505:	8d 83 00 00 00 00    	lea    0x0(%ebx),%eax
 50b:	89 45 fc             	mov    %eax,0xfffffffc(%ebp)
 50e:	81 46 14 00 00 01 00 	addl   $0x10000,0x14(%esi)
 515:	ff 04 bd 1c 00 00 00 	incl   0x1c(,%edi,4)
 51c:	ff 46 14             	incl   0x14(%esi)
 51f:	8b 83 04 00 00 00    	mov    0x4(%ebx),%eax
 525:	57                   	push   %edi
 526:	8b 40 14             	mov    0x14(%eax),%eax
 529:	ff d0                	call   *%eax
 52b:	8b 83 00 00 00 00    	mov    0x0(%ebx),%eax
 531:	83 c4 04             	add    $0x4,%esp
 534:	24 d7                	and    $0xd7,%al
 536:	c7 45 f8 00 00 00 00 	movl   $0x0,0xfffffff8(%ebp)
 53d:	0c 04                	or     $0x4,%al
 53f:	a8 03                	test   $0x3,%al
 541:	75 0d                	jne    550 <do_IRQ+0x64>
 543:	8b 93 08 00 00 00    	mov    0x8(%ebx),%edx
 549:	24 fb                	and    $0xfb,%al
 54b:	89 55 f8             	mov    %edx,0xfffffff8(%ebp)
 54e:	0c 01                	or     $0x1,%al
 550:	89 83 00 00 00 00    	mov    %eax,0x0(%ebx)
 556:	83 7d f8 00          	cmpl   $0x0,0xfffffff8(%ebp)
 55a:	74 60                	je     5bc <do_IRQ+0xd0>
 55c:	89 f3                	mov    %esi,%ebx
 55e:	89 f6                	mov    %esi,%esi
 560:	ff 4b 14             	decl   0x14(%ebx)
 563:	8b 43 08             	mov    0x8(%ebx),%eax
 566:	a8 08                	test   $0x8,%al
 568:	74 06                	je     570 <do_IRQ+0x84>
 56a:	e8 fc ff ff ff       	call   56b <do_IRQ+0x7f>
 56f:	90                   	nop    
 570:	8b 45 f8             	mov    0xfffffff8(%ebp),%eax
 573:	8d 55 08             	lea    0x8(%ebp),%edx
 576:	50                   	push   %eax
 577:	52                   	push   %edx
 578:	57                   	push   %edi
 579:	e8 fc ff ff ff       	call   57a <do_IRQ+0x8e>
 57e:	83 c4 0c             	add    $0xc,%esp
 581:	ff 43 14             	incl   0x14(%ebx)
 584:	83 3d 04 00 00 00 00 	cmpl   $0x0,0x4
 58b:	75 13                	jne    5a0 <do_IRQ+0xb4>
 58d:	50                   	push   %eax
 58e:	8b 45 fc             	mov    0xfffffffc(%ebp),%eax
 591:	50                   	push   %eax
 592:	57                   	push   %edi
 593:	e8 e0 fd ff ff       	call   378 <note_interrupt>
 598:	83 c4 0c             	add    $0xc,%esp
 59b:	90                   	nop    
 59c:	8d 74 26 00          	lea    0x0(%esi,1),%esi
 5a0:	8b 55 fc             	mov    0xfffffffc(%ebp),%edx
 5a3:	8b 02                	mov    (%edx),%eax
 5a5:	89 c2                	mov    %eax,%edx
 5a7:	a8 04                	test   $0x4,%al
 5a9:	74 0a                	je     5b5 <do_IRQ+0xc9>
 5ab:	83 e2 fb             	and    $0xfffffffb,%edx
 5ae:	8b 45 fc             	mov    0xfffffffc(%ebp),%eax
 5b1:	89 10                	mov    %edx,(%eax)
 5b3:	eb ab                	jmp    560 <do_IRQ+0x74>
 5b5:	24 fe                	and    $0xfe,%al
 5b7:	8b 55 fc             	mov    0xfffffffc(%ebp),%edx
 5ba:	89 02                	mov    %eax,(%edx)
 5bc:	8b 55 fc             	mov    0xfffffffc(%ebp),%edx
 5bf:	8b 42 04             	mov    0x4(%edx),%eax
 5c2:	57                   	push   %edi
 5c3:	8b 40 18             	mov    0x18(%eax),%eax
 5c6:	ff d0                	call   *%eax
 5c8:	83 c4 04             	add    $0x4,%esp
 5cb:	bb 00 e0 ff ff       	mov    $0xffffe000,%ebx
 5d0:	21 e3                	and    %esp,%ebx
 5d2:	ff 4b 14             	decl   0x14(%ebx)
 5d5:	8b 43 08             	mov    0x8(%ebx),%eax
 5d8:	a8 08                	test   $0x8,%al
 5da:	74 05                	je     5e1 <do_IRQ+0xf5>
 5dc:	e8 fc ff ff ff       	call   5dd <do_IRQ+0xf1>
 5e1:	8b 43 14             	mov    0x14(%ebx),%eax
 5e4:	05 01 00 ff ff       	add    $0xffff0001,%eax
 5e9:	89 43 14             	mov    %eax,0x14(%ebx)
 5ec:	a9 00 ff ff 00       	test   $0xffff00,%eax
 5f1:	75 0e                	jne    601 <do_IRQ+0x115>
 5f3:	83 3d 00 00 00 00 00 	cmpl   $0x0,0x0
 5fa:	74 05                	je     601 <do_IRQ+0x115>
 5fc:	e8 fc ff ff ff       	call   5fd <do_IRQ+0x111>
 601:	b8 00 e0 ff ff       	mov    $0xffffe000,%eax
 606:	21 e0                	and    %esp,%eax
 608:	ff 48 14             	decl   0x14(%eax)
 60b:	b8 01 00 00 00       	mov    $0x1,%eax
 610:	8d 65 ec             	lea    0xffffffec(%ebp),%esp
 613:	5b                   	pop    %ebx
 614:	5e                   	pop    %esi
 615:	5f                   	pop    %edi
 616:	89 ec                	mov    %ebp,%esp
 618:	5d                   	pop    %ebp

619: c3 ret

No - I need the function that was interrupted by common_interrupt.

I found only one valid function pointer in the stack dump above common_interrupt:

0xc0112a13, EBP=0xc0349f88

Could you look it up in your System.map?
Which power management do you use? apm or acpi?

--
Manfred

Aha. (/me sees light [a wee bit late])

It's in apm_bios_call_simple()...

...and booting with apm=off cures the explosions. (uhoh... crud bios?)

-Mike

00000134 <apm_bios_call_simple>:

     134:	55                   	push   %ebp
     135:	b8 00 e0 ff ff       	mov    $0xffffe000,%eax
     13a:	89 e5                	mov    %esp,%ebp
     13c:	21 e0                	and    %esp,%eax
     13e:	83 ec 1c             	sub    $0x1c,%esp
     141:	57                   	push   %edi
     142:	56                   	push   %esi
     143:	53                   	push   %ebx

}

/**
* apm_bios_call_simple - make a simple APM BIOS 32bit call
* @func: APM function to invoke
* @ebx_in: EBX register value for BIOS call
* @ecx_in: ECX register value for BIOS call
* @eax: EAX register on return from the BIOS call
*
* Make a BIOS call that does only returns one value, or just status.
* If there is an error, then the error code is returned in AH
* (bits 8-15 of eax) and this function returns non-zero. This is
* used for simpler BIOS operations. This call may hold interrupts
* off for a long time on some laptops.
*/

static u8 apm_bios_call_simple(u32 func, u32 ebx_in, u32 ecx_in, u32 *eax)

{
     144:	8b 5d 0c             	mov    0xc(%ebp),%ebx
     147:	8b 4d 10             	mov    0x10(%ebp),%ecx
	u8			error;
	APM_DECL_SEGS
	unsigned long		flags;
	cpumask_t		cpus;
	int			cpu;
	struct desc_struct	save_desc_40;


	cpus = apm_save_cpus();
	
	cpu = get_cpu();
     14a:	ff 40 14             	incl   0x14(%eax)
	save_desc_40 = cpu_gdt_table[cpu][0x40 / 8];
     14d:	a1 40 00 00 00       	mov    0x40,%eax
     152:	8b 15 44 00 00 00    	mov    0x44,%edx
     158:	89 45 f0             	mov    %eax,0xfffffff0(%ebp)
     15b:	89 55 f4             	mov    %edx,0xfffffff4(%ebp)
	cpu_gdt_table[cpu][0x40 / 8] = bad_bios_desc;
     15e:	a1 34 00 00 00       	mov    0x34,%eax
     163:	8b 15 38 00 00 00    	mov    0x38,%edx
     169:	a3 40 00 00 00       	mov    %eax,0x40
     16e:	89 15 44 00 00 00    	mov    %edx,0x44

	local_save_flags(flags);
     174:	9c                   	pushf  
     175:	8f 45 ec             	popl   0xffffffec(%ebp)
	APM_DO_CLI;
     178:	83 3d 20 00 00 00 00 	cmpl   $0x0,0x20
     17f:	74 03                	je     184 <apm_bios_call_simple+0x50>
     181:	fb                   	sti    
     182:	eb 01                	jmp    185 <apm_bios_call_simple+0x51>
     184:	fa                   	cli    
	APM_DO_SAVE_SEGS;
     185:	8c 65 fc             	movl   %fs,0xfffffffc(%ebp)
     188:	8c 6d f8             	movl   %gs,0xfffffff8(%ebp)
	/*
	 * N.B. We do NOT need a cld after the BIOS call
	 * because we always save and restore the flags.
	 */
	__asm__ __volatile__(APM_DO_ZERO_SEGS
     18b:	8b 45 08             	mov    0x8(%ebp),%eax
     18e:	1e                   	push   %ds
     18f:	06                   	push   %es
     190:	31 d2                	xor    %edx,%edx
     192:	8e da                	mov    %edx,%ds
     194:	8e c2                	mov    %edx,%es
     196:	8e e2                	mov    %edx,%fs
     198:	8e ea                	mov    %edx,%gs
     19a:	57                   	push   %edi
     19b:	55                   	push   %ebp
     19c:	2e ff 1d 20 00 00 00 	lcall  *%cs:0x20
     1a3:	0f 92 c3             	setb   %bl <== 0xc0112a13 is HERE
     1a6:	5d                   	pop    %ebp
     1a7:	5f                   	pop    %edi
     1a8:	07                   	pop    %es
     1a9:	1f                   	pop    %ds
     1aa:	89 45 e8             	mov    %eax,0xffffffe8(%ebp)
     1ad:	8b 45 14             	mov    0x14(%ebp),%eax
     1b0:	8b 55 e8             	mov    0xffffffe8(%ebp),%edx
     1b3:	89 10                	mov    %edx,(%eax)
	error = apm_bios_call_simple_asm(func, ebx_in, ecx_in, eax);
	APM_DO_RESTORE_SEGS;
     1b5:	8e 65 fc             	movl   0xfffffffc(%ebp),%fs
     1b8:	8e 6d f8             	movl   0xfffffff8(%ebp),%gs
	local_irq_restore(flags);
     1bb:	ff 75 ec             	pushl  0xffffffec(%ebp)
     1be:	9d                   	popf   
	cpu_gdt_table[smp_processor_id()][0x40 / 8] = save_desc_40;
     1bf:	8b 45 f0             	mov    0xfffffff0(%ebp),%eax
     1c2:	8b 55 f4             	mov    0xfffffff4(%ebp),%edx
     1c5:	a3 40 00 00 00       	mov    %eax,0x40
     1ca:	89 15 44 00 00 00    	mov    %edx,0x44
/* how to get the thread information struct from C */
static inline struct thread_info *current_thread_info(void)
{
	struct thread_info *ti;
	__asm__("andl %%esp,%0; ":"=r" (ti) : "0" (~8191UL));
     1d0:	b8 00 e0 ff ff       	mov    $0xffffe000,%eax
     1d5:	21 e0                	and    %esp,%eax
	put_cpu();
     1d7:	ff 48 14             	decl   0x14(%eax)

#endif

static __inline__ int constant_test_bit(int nr, const volatile unsigned long * addr)

{
	return ((1UL << (nr & 31)) & (((const volatile unsigned int *) addr)[nr >> 5])) != 0;
     1da:	8b 40 08             	mov    0x8(%eax),%eax
     1dd:	a8 08                	test   $0x8,%al
     1df:	74 05                	je     1e6 <apm_bios_call_simple+0xb2>
     1e1:	e8 fc ff ff ff       	call   1e2 <apm_bios_call_simple+0xae>
	apm_restore_cpus(cpus);
	return error;
     1e6:	0f b6 c3             	movzbl %bl,%eax
     1e9:	5b                   	pop    %ebx
     1ea:	5e                   	pop    %esi
     1eb:	5f                   	pop    %edi
     1ec:	89 ec                	mov    %ebp,%esp
     1ee:	5d                   	pop    %ebp
     1ef:	c3                   	ret    

Could you try the attached patch?
It updates the end of stack detection to handle unaligned stacks.

--
Manfred

// $Header$
// Kernel Version:

//  VERSION = 2
//  PATCHLEVEL = 6
//  SUBLEVEL = 0
//  EXTRAVERSION = -test7

@@ -101,6 +101,16 @@
 
 #endif
 
+static inline int kstack_end(void *addr)
+{
+	unsigned long offset = (unsigned long)addr & (THREAD_SIZE-1);
+
+	/* Some APM bios versions misalign the stack */
+	if (offset == 0 || offset > (THREAD_SIZE-sizeof(void*)))
+			return 1;
+	return 0;
+}
+
 /*
  * thread information flags
  * - these are process state flags that various assembly files may need to access

@@ -862,7 +862,7 @@
 		unsigned long *sptr = &caller;
 		unsigned long svalue;
 
-		while (((long) sptr & (THREAD_SIZE-1)) != 0) {
+		while (!kstack_end(sptr)) {
 			svalue = *sptr++;
 			if (kernel_text_address(svalue)) {
 				*addr++=svalue;

@@ -104,7 +104,7 @@
 #ifdef CONFIG_KALLSYMS
 	printk("\n");
 #endif
-	while (((long) stack & (THREAD_SIZE-1)) != 0) {
+	while (!kstack_end(stack)) {
 		addr = *stack++;
 		if (kernel_text_address(addr)) {
 			printk(" [<%08lx>] ", addr);

@@ -138,7 +138,7 @@
 
 	stack = esp;
 	for(i = 0; i < kstack_depth_to_print; i++) {
-		if (((long) stack & (THREAD_SIZE-1)) == 0)
+		if (kstack_end(stack))
 			break;
 		if (i && ((i % 8) == 0))
 			printk("\n       ");

Works fine. (modulo moving kstack_end above ASSEMBLY)

Thanks,

-Mike

I'm the one with bugzilla 973. I'm trying the patch with a source tree synced up from bk this morning and having a few problems. My in-laws are visiting today, so my work on this will be intermittent. I am interested, however.

Note to self: next time read the whole message, including the part in parenthesis. The patch, modulo Mike's modulo (move the function where I was told to move the function), does indeed work fine.

Testing continues, but thanks!

I've attached a rediff of your patch against test7 bitkeeper. It has the kstack_end function moved up above ASSEMBLY as suggested by Mike. I've tested this version and it works for me (tm). Thanks a bunch.

diff -ur linux-2.5-tma/arch/i386/kernel/traps.c linux-2.5-tm/arch/i386/kernel/traps.c
@@ -104,7 +104,7 @@
 #ifdef CONFIG_KALLSYMS
 	printk("\n");
 #endif
-	while (((long) stack & (THREAD_SIZE-1)) != 0) {
+	while (!kstack_end(stack)) {
 		addr = *stack++;
 		if (kernel_text_address(addr)) {
 			printk(" [<%08lx>] ", addr);

@@ -138,7 +138,7 @@
 
 	stack = esp;
 	for(i = 0; i < kstack_depth_to_print; i++) {
-		if (((long) stack & (THREAD_SIZE-1)) == 0)
+		if (kstack_end(stack))
 			break;
 		if (i && ((i % 8) == 0))
 			printk("\n       ");

diff -ur linux-2.5-tma/include/asm-i386/thread_info.h linux-2.5-tm/include/asm-i386/thread_info.h
@@ -92,6 +92,16 @@
 #define get_thread_info(ti) get_task_struct((ti)->task)
 #define put_thread_info(ti) put_task_struct((ti)->task)
 
+static inline int kstack_end(void *addr)
+{
+        unsigned long offset = (unsigned long)addr & (THREAD_SIZE-1);
+
+        /* Some APM bios versions misalign the stack */
+        if (offset == 0 || offset > (THREAD_SIZE-sizeof(void*)))
+                        return 1;
+        return 0;
+}
+
 #else /* !__ASSEMBLY__ */
 
 /* how to get the thread information struct from ASM */

diff -ur linux-2.5-tma/mm/slab.c linux-2.5-tm/mm/slab.c
@@ -862,7 +862,7 @@
 		unsigned long *sptr = &caller;
 		unsigned long svalue;
 
-		while (((long) sptr & (THREAD_SIZE-1)) != 0) {
+		while (!kstack_end(sptr)) {
 			svalue = *sptr++;
 			if (kernel_text_address(svalue)) {
 				*addr++=svalue;

Only in linux-2.5-tm: stack.patch