BUG 2.6.7 hangs on boot (rx2600)

Hi,
just a bug report.

I tried building a 2.6.7 kernel using my "standard" .config that worked for 2.6.6. The only source code changes were to include openib.org infiniband driver in the tree as modules.

Last output was:

	Console: colour VGA+ 80x25

THe next line for 2.6.6 kernel would be:

	Memory: 6225280k/6263872k available (6560k code, 38224k reserved, 3069k data, 208k init)

After 2 minutes I still couldn't ping the box and reset it. I should have "INIT" the box to see where it hung but wasn't thinking.

I wonder if this might be related to KALLSYMS bugs recently discussed. Trying with CONFIG_KALLSYMS and CONFIG_KALLSYMS_ALL off next.

hth,
grant

EFI Boot Manager ver 1.10 [14.61] Firmware ver 2.21 [4334]

Please select a boot option

Debian GNU/Linux
EFI Shell [Built-in]
Boot Option Maintenance Menu
System Configuration Menu

Use ^ and v to change option(s). Use Enter to select an option
Loading.: Debian GNU/Linux
Starting: Debian GNU/Linux
ELILO

ELILO boot:
Linux266 Linux267 Linux265 Linux2424 (or a kernel file name: [[dev_name:/]path/]kernel_image cmdline options)

ELILO boot: Linux267
Uncompressing Linux... done
Linux version 2.6.7 ([email protected]) (gcc version 3.3.4 (Debian)) #1 SMP Mon Jun 21 22:36:17 PDT 2004 EFI v1.10 by HP: SALsystab=0x3fb38000 ACPI 2.0=0x3fb2e000 SMBIOS=0x3fb3a000 HCDP=0x3fb2c000
ACPI: RSDP (v002 HP ) @ 0x000000003fb2e000
ACPI: XSDT (v001 HP zx6000 0x00000000 HP 0x00000000) @ 0x000000003fb2e02c
ACPI: FADT (v003 HP zx6000 0x00000000 HP 0x00000000) @ 0x000000003fb369e0
ACPI: SPCR (v001 HP zx6000 0x00000000 HP 0x00000000) @ 0x000000003fb36b18
ACPI: DBGP (v001 HP zx6000 0x00000000 HP 0x00000000) @ 0x000000003fb36b68
ACPI: MADT (v001 HP zx6000 0x00000000 HP 0x00000000) @ 0x000000003fb36c28
ACPI: SPMI (v004 HP zx6000 0x00000000 HP 0x00000000) @ 0x0003fb36ba0
ACPI: CPEP (v001 HP zx6000 0x00000000 HP 0x00000000) @ 0x000000003fb36bf0
ACPI: SSDT (v001 HP zx6000 0x006 INTL 0x02012044) @ 0x000000003fb33870
ACPI: SSDT (v001 HP zx6000 0x00000006 INTL 0x02012044) @ 0x000000003fb33a50

ACPIT (v001     HP   zx6000 0x00000006 INTL 0x02012044) @ 0x000000003fb33da0
ACPI: SSDT (v001     HP   zx6000 0x00000006 INTL 0x020) @ 0x000000003fb347c0
ACPI: SSDT (v001     HP   zx6000 0x00000006 INTL 0x02012044) @ 0x000000003fb351e0
ACPI: SSDT (v001     zx6000 0x00000006 INTL 0x02012044) @ 0x000000003fb35c00
ACPI: SSDT (v001     HP   zx6000 0x00000006 INTL 0x02012044) @ 0x00000036620
ACPI: SSDT (v001     HP   zx6000 0x00000006 INTL 0x02012044) @ 0x000000003fb36800
ACPI: SSDT (v001     HP   zx6000 0x000 INTL 0x02012044) @ 0x000000003fb368f0
ACPI: DSDT (v001     HP   zx6000 0x00000007 INTL 0x02012044) @ 0x0000000000000000

efi.top: ignoring 4KB of memory at 0x0 due to granule hole at 0x0
efi.trim_top: ignoring 636KB of memory at 0x1000 due to granule ho 0x0
efi.trim_bottom: ignoring 15360KB of memory at 0x100000 due to granule hole at 0x0
SAL 3.1: HP version 2.21
SAL Platformures: None
SAL: AP wakeup using external interrupt vector 0xff
ACPI: Local APIC address 0xc0000000fee00000
ACPI: LAPIC_ADDR_Oddress[00000000fee00000])
ACPI: LSAPIC (acpi_id[0x00] lsapic_id[0x00] lsapic_eid[0x00] enabled)
CPU 0 (0x0000) enabled (BSP)
: LSAPIC (acpi_id[0x01] lsapic_id[0x01] lsapic_eid[0x00] enabled)
CPU 1 (0x0100) enabled
ACPI: IOSAPIC (id[0x0] global_irq_bas0] address[00000000fed20800])
ACPI: IOSAPIC (id[0x1] global_irq_base[0x1b] address[00000000fed22800])
ACPI: IOSAPIC (id[0x2] g_irq_base[0x26] address[00000000fed24800])
ACPI: IOSAPIC (id[0x3] global_irq_base[0x31] address[00000000fed26800])
ACPI: IOSAPd[0x4] global_irq_base[0x3c] address[00000000fed28800])
ACPI: IOSAPIC (id[0x6] global_irq_base[0x47] address[00000000fed2c800])PI: IOSAPIC (id[0x7] global_irq_base[0x52] address[00000000fed2e800]) GSI 0x24(low,level) -> CPU 0x0000 vector 48
2 CPUs avail 2 CPUs total
GSI 0x52(low,level) -> CPU 0x0000 vector 49
MCA related initialization done
On node 0 totalpages: 391492
DMA: 391492 pages, LIFO batch:4
Normal zone: 0 pages, LIFO batch:1
HighMem zone: 0 pages, LIFO batch:1
Virtual mem_map start0xa0007fffc7200000
Built 1 zonelists
Kernel command line: BOOT_IMAGE=scsi0:/EFI/debian/boot/vmlinuz-2.6.7 root=/dev/sdb3 consoyS0 ro
PID hash table entries: 4096 (order 12: 65536 bytes)
CPU 0: base freq 0.000MHz, ITC ratio/2, ITC freq00.000MHz+ppm
Console: colour VGA+ 80x25
<End Of Output>

Did you have any MCA records after you rebooted? This is about the time it should have chosen a default console device and started printing to it, so if it tried using the VGA console and you didn't have one, it may have just fallen over. The MCA record would be pretty clear about that though...

Jesse

Sorry - I forgot to mention that. I did check and there was none.

I was thinking it just picked a different serial port for ttyS0 output. That's why I waited a couple of minutes to see if I could ping/ssh to the box. But after two minutes it still didn't respond and should have by then.

The systems does have a VGA output but nothing is connected. USB keyboard/mouse is not connected either.

However, you just gave me the idea to enable CONFIG_IA64_EARLY_PRINTK. Trying that now.

thanks,
grant

David saw this late last week on a 2.6.7 kernel compiled for UP, and I reproduced it yesterday using the current linux-ia64-2.5 BK bits. I'm using a zx6000 with serial console (VGA also present, but unused). No MCA records. I collected info from an INIT, but haven't had a chance to look very far. Something in the error records pointed to efi_memmap_walk(), so I added a printk there, and the problem disappeared.

David mentioned the linker in Debian/testing as a possibility. I'm using ld 2.14.90.0.7 from Debian binutils 2.14.90.0.7-3.

This is for SMP

*ugh*.

Debian/testing is currently:
ii binutils 2.14.90.0.7-8 The GNU assembler, linker and binary utiliti

I suppose that's a possibility. I have a TOC dump on the machine and just need to "harvest" it, then see where it's spinning. Probably get to that tomorrow.

thanks,
grant

Yeah, I saw it with slightly different "console=" parameters than David did. So I think it's just a heisenbug and he happened to trip over it with a UP kernel.

Have you used salinfo_decode? I think there's a debian package for it now ("salinfo").

ok

no...too cool.

gsyprf3:~# apt-get install salinfo
Reading Package Lists... Done
Building Dependency Tree... Done
The following NEW packages will be installed:
  salinfo
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 29.5kB of archives.
After unpacking 197kB of additional disk space will be used.
Get:1 http://mirrors.kernel.org testing/main salinfo 0.5-1 [29.5kB]
Fetched 29.5kB in 2s (14.3kB/s)
Selecting previously deselected package salinfo.
(Reading database ... 63823 files and directories currently installed.)
Unpacking salinfo (from .../salinfo_0.5-1_ia64.deb) ...
Setting up salinfo (0.5-1) ...
Starting salinfo decode daemons: salinfo.

gsyprf3:~# man salinfo 
No manual entry for salinfo
gsyprf3:~# 

ah it's "man salinfo_decode".

gsyprf3:/var/log/salinfo/decoded# ls
2004-06-22-06:44:21-cpu0-init.0  2004-06-22-06:44:21-cpu1-init.0  old

bits of cpu0-init.0 data:
...
Processor static data:

    xip  : 0xa00000010005b330  xfs  : 0x8000000000000000
    xpsr : 0x00001010084a6010
...
    iip  : 0xa000000100005400  iipa : 0xa00000010005b320
    ipsr : 0x0000001008000010
...
        b0 : 0xa000000100013cb0 0x000000003f7e4620 0x0000000000000000 0x0000000000000000
        b4 : 0x0000000000000000 0x0000000000000000 0xe00000003fa43280 0xa00000010005b2a0

Reading the corresponding System.map-2.6.7:
...
a000000100005400 t general_exception
...
a00000010005b2a0 t count_reserved_pages
a00000010005b380 T mem_init
a00000010005b8c0 T ia64_set_rbs_bot
a00000010005b920 t mapped_kernel_page_is_present
a00000010005ba40 T ia64_do_page_fault

I'm wasn't 100% certain I'm looking at the right System.map file. But both see to be identical for the above symbols.

I've made both System.map files (with and w/o EARLY_PRINTK) and salinfo records available on

	ftp://gsyprf3.external.hp.com/pub/2.6.7/

hth,
grant

That didn't matter. Still "hangs".

thanks,
grant

I saw it on a 4 way Tiger as well. The kernel was compiled using the RHEL3 toolchain.
Saw the machine sitting in dispatch_to_fault_handler: with the IIP pointing to count_reserved_pages(). I think IFA was 0.

Didn't get a chance to debug this further.

-Arun

RHEL3

pointing to

I also saw such hang on Tiger, when rpmbuild on SLES9-RC2. I'm suspecting that recent patch which moves init_task to region 5 may break sth there.

Env:
Tiger 4 with 4 Madison
SLES9-RC2 (linux-2.6.5-7.79)

Description:
Before rpmbuild, I applied partial-page patch derived from David's tree, which happened to contain another patch to move init_task to region 5. Then when I booted the new kernel created by rpmbuild (so by default config file), Tiger hangs immediately.

Excluding patch for moving init_task made problem away. ======Kernel just stopped at general_exception(), with IIP pointing to count_reserved_pages():
IIP: 0xa0000001000594a0 (In count_reserved_pages)
ISR: 0x00000028400000030
IFA: 0xe00000004fefe6018

Then by single step:
General_exception() -> dispatch_to_default_handler() -> SAVE_MIN_WITH_COVER_R19:

	MINSTATE_GET_CURRENT(r16);	//r16 = 0xa00000010070c000
(&init_task)
	...
	adds r16=IA64_TASK_THREAD_ON_USTACK_OFFSET,r16;
\
	;;
\
	ld1 r17=[r16];	<------Just in this point, ITP also hangs and

seems another fault happening.

I'm suspecting line in ia64_switch_to:

	/*
	* If we've already mapped this task's page, we can skip doing it
again.
	*/

(p6) cmp.eq p7,p6=r27,r27 <----- Should here cmp.eq.unc be used
instead? No time to test it now...

Thanks,
Kevin

Thanks a lot Kevin, that worked great! Here's the patch for people who want something a little easier to apply. Too bad the init_task move patch was mixed up with some ia32 stuff or it would have been easier to revert this change for testing.

Clear both p7 and p6 predicates in the check for task struct mapping in ia64_switch_to.

Jesse

===== arch/ia64/kernel/entry.S 1.61 vs edited =====

@@ -191,7 +191,7 @@
 	/*
 	 * If we've already mapped this task's page, we can skip doing it again.
 	 */
-(p6)	cmp.eq p7,p6=r26,r27
+(p6)	cmp.eq.unc p7,p6=r26,r27
 (p6)	br.cond.dpnt .map
 	;;
 .done:

Your change evidently solves the problem, but I don't understand how. Can you enlighten me? Here's the essence of the code:

	        cmp.eq p7,p6=r25,in0
	(p6)    cmp.eq p7,p6=r26,r27
	(p6)    br.cond.dpnt .map

As I understand it, adding ".unc" to the second cmp instruction should only make a difference when p6=0. In that case, after the old cmp (no ".unc") we'd have p6=0 and p7=1, while after the new cmp (with ".unc") we'd have p6=0 and p7=0. But we never test p7, so I don't see what difference it makes.

Ah, agree with you on this point. So maybe this touch just affects sth behind indirectly. I'm looking into it again... Anyone has a quick answer? :)

Thanks,
Kevin

This is called black magic and pure coincidence. Welcome to the world of randomness. If I boot that "unc" Kernel frequent enough, it will hang eventually. Without "unc" it also has 30/70 fail/pass rate.

The regression is coming from moving init_task from region 7 to region
5. The hang was a nested fault with no valid dtlb mapping for the init task's stack. The problem was from physical mode efi call. efi_call_phys
does: ia64_switch_mode_phys, call the function, then ia64_switch_mode_virt. The ia64_switch_mode_virt now need to special case the init task to put sp and ar.bspstore into region5 instead of region7. I have a quick patch that fix the hang. Let me polish it a bit more and then post.

Oh yeah, baby, the first two hunk in head.S is just plain wrong in this
patch: http://www.gelato.unsw.edu.au/linux-ia64/0406/10047.html. Let me work on that too .....

- Ken

Regarding to rev 1.24 in head.S:
http://lia64.bkbits.net:8080/linux-ia64-2.5/diffs/arch/ia64/kernel/[email protected]?nav=index.html|src/.|src/arch|src/arch/ia64|src/arch/
ia64/kernel|hist/arch/ia64/kernel/head.S

The relocation of r16 is incorrect. For BP, we are not installing any region 7 TLB mapping. But this patch will put a valid kernel granule index into kr(stack). Equivalently, it lies to the rest of the kernel that it installed an entry at the index kernel image locates. If the first task coming out of this init_task happens to have its task struct in that very same granule, the stack will not be mapped by any DTLB. Then bad things happen like random hang because of nested fault.

The first two hunks in the following patch reverse that relocation. The next two hunks fix the random hang observed when moving init_task from region 7 to region 5. As explained earlier, BP did a call to efi_call_phys which switches to physical mode and then back to virtual. When going back to virtual, it converts ar.bspstore and sp to region 7 address. After that, any heavy weight fault will lead into nested fault because there are no region 7 dtlb mapping for its stack. The fix is to special case ia64_switch_mode_virt to compute ar.bspstore and sp's virtual addresses into region 5.

Tested on Intel tiger machine with several hundred consecutive boots.

@@ -154,10 +154,9 @@ start_ap:
 #endif
 	;;
 	tpa r3=r2		// r3 = phys addr of task struct
+	mov r16=-1
+(isBP)	br.cond.dpnt .load_current // no need to map region 5 init_task
 	;;
-	shr.u r16=r3,IA64_GRANULE_SHIFT
-(isBP)	br.cond.dpnt .load_current // BP stack is on region 5 --- no need to map it
-
 	// load mapping for stack (virtaddr in r2, physaddr in r3)
 	rsm psr.ic
 	movl r17=PAGE_KERNEL

@@ -169,6 +168,7 @@ start_ap:
 	dep r2=-1,r3,61,3	// IMVA of task
 	;;
 	mov r17=rr[r2]
+	shr.u r16=r3,IA64_GRANULE_SHIFT
 	;;
 	dep r17=0,r17,8,24
 	;;

@@ -766,7 +766,9 @@ GLOBAL_ENTRY(ia64_switch_mode_virt)
 	flushrs				// must be first insn in group
 	srlz.i
  }
+	movl r19=init_task
 	;;
+	cmp.eq p7,p6=r19,r13		// special case init_task
 	mov cr.ipsr=r16			// set new PSR
 	add r3-ia64_switch_mode_virt,r15
 

@@ -781,11 +783,15 @@ GLOBAL_ENTRY(ia64_switch_mode_virt)
 	movl r18=KERNEL_START
 	dep r3=0,r3,KERNEL_TR_PAGE_SHIFT,64-KERNEL_TR_PAGE_SHIFT
 	dep r14=0,r14,KERNEL_TR_PAGE_SHIFT,64-KERNEL_TR_PAGE_SHIFT
-	dep r17=-1,r17,61,3
-	dep sp=-1,sp,61,3
+(p6)	dep r17=-1,r17,61,3
+(p6)	dep sp=-1,sp,61,3
+(p7)	dep r17=0,r17,KERNEL_TR_PAGE_SHIFT,64-KERNEL_TR_PAGE_SHIFT
+(p7)	dep sp =0, sp,KERNEL_TR_PAGE_SHIFT,64-KERNEL_TR_PAGE_SHIFT
 	;;
 	or r3=r3,r18
 	or r14=r14,r18
+(p7)	or r17=r17,r18
+(p7)	or sp=sp,r18
 	;;
 
 	mov r18=ar.rnat			// save ar.rnat

Ken> Regarding to rev 1.24 in head.S:
Ken> http://lia64.bkbits.net:8080/linux-ia64-2.5/diffs/arch/ia64/kernel/[email protected]?nav=index.html|src/.|src/arch|src/arch/ia64|src/arch/
Ken> ia64/kernel|hist/arch/ia64/kernel/head.S

Ken> The relocation of r16 is incorrect. For BP, we are not installing any Ken> region 7 TLB mapping.

Indeed! I must be missing something though: with your patch, r13 will be initialized to the region 7 address again, which defeats the purpose of the original patch. I think the initialization of r13 needs to be conditional on whether we're dealing with init_task or anything else.

Ken> As explained earlier, BP did a call to efi_call_phys which Ken> switches to physical mode and then back to virtual. When going Ken> back to virtual, it converts ar.bspstore and sp to region 7 Ken> address. After that, any heavy weight fault will lead into Ken> nested fault because there are no region 7 dtlb mapping for its Ken> stack. The fix is to special case ia64_switch_mode_virt to Ken> compute ar.bspstore and sp's virtual addresses into region 5.

True, but it's really ugly to add more special cases. Wouldn't it be better to explicitly pass the sp/bsp that need to be restored? (Caveat: can't use the normal calling conventions there; perhaps r17 and r18 could be used?)

	--david

David Mosberger wrote on Friday, June 25, 2004 10:29 PM

Not sure what you mean. The first two hunks are trying to revert the change in rev 1.24 and r16 initialization, which gets put into kr(stack) later. I'm not touching r13.

Yeah, but we have to update all the call sites, current efi_call_phys and two other PAL static/stacked calls.

Ken> David Mosberger wrote on Friday, June 25, 2004 10:29 PM

Ken> The relocation of r16 is incorrect. For BP, we are not installing any Ken> region 7 TLB mapping.

Ken> Not sure what you mean. The first two hunks are trying to revert the Ken> change in rev 1.24 and r16 initialization, which gets put into kr(stack) Ken> later. I'm not touching r13.

Ah, yes, my bad (I guess I really should catch up sleep first...).

Ken> Yeah, but we have to update all the call sites, current efi_call_phys Ken> and two other PAL static/stacked calls.

True, but I think there are only 3 call-sites. If it turns out to be _really_ ugly we can reonsider, but I think it might be a better choice in the long run.

	--david

David Mosberger wrote on Friday, June 25, 2004 10:55 PM

How does this patch look? It is a bit big. But what it does is really
simple: change 3 call sites to save/restore virtual address of sp and ar.bsp/ar.bspstore.

=== arch/ia64/kernel/efi_stub.S 1.5 vs edited ==--- 1.5/arch/ia64/kernel/efi_stub.S Thu May 15 04:45:02 2003
+++ edited/arch/ia64/kernel/efi_stub.S Mon Jun 28 21:55:00 2004
@@ -44,7 +44,7 @@

GLOBAL_ENTRY(efi_call_phys)
.prologue ASM_UNW_PRLG_RP|ASM_UNW_PRLG_PFS, ASM_UNW_PRLG_GRSAVE(8)
- alloc loc1=ar.pfs,8,5,7,0
+ alloc loc1=ar.pfs,8,7,7,0
ld8 r2=[in0],8 // load EFI function's entry point
mov loc0=rp
.body
@@ -70,9 +70,13 @@
mov out3=in4
mov out5=in6
mov out6=in7
+ mov loc5=r19
+ mov loc6=r20
br.call.sptk.many rp¶ // call the EFI function
.ret1: mov ar.rsc=0 // put RSE in enforced lazy, LE mode
mov r16=loc3
+ mov r19=loc5
+ mov r20=loc6
br.call.sptk.many rp=ia64_switch_mode_virt // return to virtual mode
.ret2: mov ar.rsc=loc4 // restore RSE configuration
mov ar.pfs=loc1
=== arch/ia64/kernel/head.S 1.24 vs edited ==--- 1.24/arch/ia64/kernel/head.S Wed Jun 16 18:09:33 2004
+++ edited/arch/ia64/kernel/head.S Mon Jun 28 21:55:01 2004
@@ -706,6 +706,9 @@
*
* Inputs:
* r16 = new psr to establish
+ * Output:
+ * r19 = old virtual address of ar.bsp
+ * r20 = old virtual address of sp
*
* Note: RSE must already be in enforced lazy mode
*/
@@ -724,12 +727,13 @@
mov cr.ipsr=r16 // set new PSR
add r3-ia64_switch_mode_phys,r15

- mov r17=ar.bsp
+ mov r19=ar.bsp
+ mov r20=sp
mov r14=rp // get return address into a general register
;;

// going to physical mode, use tpa to translate virt->phys
- tpa r17=r17
+ tpa r17=r19
tpa r3=r3
tpa sp=sp
tpa r14=r14
@@ -752,6 +756,8 @@
*
* Inputs:
* r16 = new psr to establish
+ * r19 = new bspstore to establish
+ * r20 = new sp to establish
*
* Note: RSE must already be in enforced lazy mode
*/
@@ -770,7 +776,6 @@
mov cr.ipsr=r16 // set new PSR
add r3-ia64_switch_mode_virt,r15

- mov r17=ar.bsp
mov r14=rp // get return address into a general register
;;

@@ -781,15 +786,14 @@
movl r18=KERNEL_START
dep r3=0,r3,KERNEL_TR_PAGE_SHIFT,64-KERNEL_TR_PAGE_SHIFT dep r14=0,r14,KERNEL_TR_PAGE_SHIFT,64-KERNEL_TR_PAGE_SHIFT
- dep r17=-1,r17,61,3
- dep sp=-1,sp,61,3
+ mov sp=r20
;;
or r3=r3,r18
or r14=r14,r18
;;

mov r18=ar.rnat // save ar.rnat
- mov ar.bspstore=r17 // this steps on ar.rnat
+ mov ar.bspstore=r19 // this steps on ar.rnat
mov cr.iip=r3
mov cr.ifs=r0
;;
=== arch/ia64/kernel/pal.S 1.7 vs edited ==--- 1.7/arch/ia64/kernel/pal.S Thu May 15 04:45:02 2003
+++ edited/arch/ia64/kernel/pal.S Mon Jun 28 21:55:03 2004
@@ -176,10 +176,14 @@
andcm r16=loc3,r16 // removes bits to clear from psr
br.call.sptk.many rp=ia64_switch_mode_phys
.ret1: mov rp = r8 // install return address (physical)
+ mov loc5 = r19
+ mov loc6 = r20
br.cond.sptk.many b7
1:
mov ar.rsc=0 // put RSE in enforced lazy, LE mode
mov r16=loc3 // r16= original psr
+ mov r19=loc5
+ mov r20=loc6
br.call.sptk.many rp=ia64_switch_mode_virt // return to virtual mode
.ret2:
mov psr.l = loc3 // restore init PSR
@@ -201,7 +205,7 @@
*/
GLOBAL_ENTRY(ia64_pal_call_phys_stacked)
.prologue ASM_UNW_PRLG_RP|ASM_UNW_PRLG_PFS, ASM_UNW_PRLG_GRSAVE(5)
- alloc loc1 = ar.pfs,5,5,86,0
+ alloc loc1 = ar.pfs,5,7,4,0
movl loc2 = pal_entry_point
1: {
mov r28 = in0 // copy procedure index
@@ -230,10 +234,14 @@
andcm r16=loc3,r16 // removes bits to clear from psr
br.call.sptk.many rp=ia64_switch_mode_phys
.ret6:
+ mov loc5 = r19
+ mov loc6 = r20
br.call.sptk.many rp· // now make the call
.ret7:
mov ar.rsc=0 // put RSE in enforced lazy, LE mode
mov r16=loc3 // r16= original psr
+ mov r19=loc5
+ mov r20=loc6
br.call.sptk.many rp=ia64_switch_mode_virt // return to virtual mode

.ret8: mov psr.l = loc3 // restore init PSR

Chen, Kenneth W wrote on Tuesday, June 29, 2004 8:09 AM

To follow up the other bug in head.S, here is the fix.

---------
For BP, we are not installing any region 7 DTLB mapping for init_task. However, kr(stack) is being initialized to a legal kernel granule that the kernel resides. If the first task context switch out of this init_task happens to have its task struct in that very same granule, the stack will not be mapped by any DTLB. Patch to properly initialize kr(stack) for BP.

=== arch/ia64/kernel/head.S 1.25 vs edited ==--- 1.25/arch/ia64/kernel/head.S Mon Jun 28 22:07:49 2004
+++ edited/arch/ia64/kernel/head.S Mon Jun 28 22:11:32 2004
@@ -154,8 +154,7 @@
#endif
;;
tpa r3=r2 // r3 = phys addr of task struct
- ;;
- shr.u r16=r3,IA64_GRANULE_SHIFT
+ mov r16=-1
(isBP) br.cond.dpnt .load_current // BP stack is on region 5 --- no need to map it

// load mapping for stack (virtaddr in r2, physaddr in r3)
@@ -169,6 +168,7 @@
dep r2=-1,r3,61,3 // IMVA of task
;;
mov r17=rr[r2]
+ shr.u r16=r3,IA64_GRANULE_SHIFT
;;
dep r17=0,r17,8,24
;;

I tried both of these on a machine that doesn't have memory at the stock kernel load address, and it failed very early on. However, it works with the attached patch.

Linux version 2.6.7 ([email protected]) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-24)) #3 SMP Tue Jun 29 10:01:08 PDT 2004 EFI v1.02 by SGI: SALsystab=0x230047e5150 ACPI 2.0=0x230047e5920
ACPI: RSDP (v002 SGI ) @
0x00000230047e5920
ACPI: XSDT (v001 SGI XSDTSN2 0x00010001 0x00000001) @ 0x00000230047e5960
ACPI: MADT (v001 SGI APICSN2 0x00010001 0x00000001) @ 0x00000230047e59c0
ACPI: SRAT (v001 SGI SRATSN2 0x00010001 0x00000001) @ 0x00000230047e5a30
ACPI: SLIT (v001 SGI SLITSN2 0x00010001 0x00000001) @ 0x00000230047e5b00
ACPI: FADT (v003 SGI FACPSN2 0x00030001 0x00000001) @ 0x00000230047e5c00
ACPI: DSDT (v001 SGI DSDTSN2 0x00010001 0x00000001) @ 0x00000230047e5bc0
ACPI: DSDT (v001 SGI DSDTSN2 0x00010001 0x00000001) @ 0x0000000000000000
ACPI: SRAT revision 0
ACPI: SLIT localities 6x6
Number of logical nodes in system = 2
Number of memory chunks in system = 2
SAL 2.9: SGI SN2 version 3.32
SAL Platform features: ITC_Drift
SAL: AP wakeup using external interrupt vector 0x12

Jesse

===== arch/ia64/kernel/entry.S 1.61 vs edited =====

@@ -179,19 +179,17 @@
 	.body
 
 	adds r22=IA64_TASK_THREAD_KSP_OFFSET,r13
-	movl r25=init_task
 	mov r27=IA64_KR(CURRENT_STACK)
-	adds r21=IA64_TASK_THREAD_KSP_OFFSET,in0
 	dep r20=0,in0,61,3		// physical address of "current"
 	;;
 	st8 [r22]=sp			// save kernel stack pointer of old task
 	shr.u r26=r20,IA64_GRANULE_SHIFT
-	cmp.eq p7,p6=r25,in0
+	adds r21=IA64_TASK_THREAD_KSP_OFFSET,in0
 	;;
 	/*
 	 * If we've already mapped this task's page, we can skip doing it again.
 	 */
-(p6)	cmp.eq p7,p6=r26,r27
+	cmp.eq p7,p6=r26,r27
 (p6)	br.cond.dpnt .map
 	;;
 .done:

===== arch/ia64/kernel/head.S 1.24 vs edited =====

@@ -154,10 +154,6 @@
 #endif
 	;;
 	tpa r3=r2		// r3 == phys addr of task struct
-	;;
-	shr.u r16=r3,IA64_GRANULE_SHIFT
-(isBP)	br.cond.dpnt .load_current // BP stack is on region 5 --- no need to map it
-
 	// load mapping for stack (virtaddr in r2, physaddr in r3)
 	rsm psr.ic
 	movl r17=PAGE_KERNEL

@@ -169,6 +165,7 @@
 	dep r2=-1,r3,61,3	// IMVA of task
 	;;
 	mov r17=rr[r2]
+	shr.u r16=r3,IA64_GRANULE_SHIFT
 	;;
 	dep r17=0,r17,8,24
 	;;

@@ -183,7 +180,6 @@
 	srlz.d
   	;;
 
-.load_current:
 	// load the "current" pointer (r13) and ar.k6 with the current task
 	mov IA64_KR(CURRENT)=r2		// virtual address
 	mov IA64_KR(CURRENT_STACK)=r16

Jesse Barnes wrote on Tuesday, June 29, 2004 10:32 AM

Let me confirm what I understand:
David's bk tree doesn't boot.
David's bk tree + 2 patches I posted this morning doesn't boot.

Is that correct?

Correct. David's tree hangs at a later point though, after printing "Console: ... 80x25".

Jesse

Jesse Barnes wrote on Tuesday, June 29, 2004 10:46 AM

I presume David's bk tree plus this patch also hung on your machine?
http://www.gelato.unsw.edu.au/linux-ia64/0406/10162.html

- Ken

Yep. It gets as far as David's tree plus your patches from this morning.

Jesse

Linux version 2.6.7 ([email protected]) (gcc version 3.3.2) #2 SMP Tue Jun 29 11:10:12 PDT 2004
EFI v1.02 by SGI: SALsystab=0x230047e5150 ACPI 2.0=0x230047e5920
ACPI: RSDP (v002 SGI ) @
0x00000230047e5920
ACPI: XSDT (v001 SGI XSDTSN2 0x00010001 0x00000001) @ 0x00000230047e5960
ACPI: MADT (v001 SGI APICSN2 0x00010001 0x00000001) @ 0x00000230047e59c0
ACPI: SRAT (v001 SGI SRATSN2 0x00010001 0x00000001) @ 0x00000230047e5a30
ACPI: SLIT (v001 SGI SLITSN2 0x00010001 0x00000001) @ 0x00000230047e5b00
ACPI: FADT (v003 SGI FACPSN2 0x00030001 0x00000001) @ 0x00000230047e5c00
ACPI: DSDT (v001 SGI DSDTSN2 0x00010001 0x00000001) @ 0x00000230047e5bc0
ACPI: DSDT (v001 SGI DSDTSN2 0x00010001 0x00000001) @ 0x0000000000000000
ACPI: SRAT revision 0
ACPI: SLIT localities 6x6
Number of logical nodes in system = 2
Number of memory chunks in system = 2
SAL 2.9: SGI SN2 version 3.32
SAL Platform features: ITC_Drift
SAL: AP wakeup using external interrupt vector 0x12

Jesse Barnes wrote on Tuesday, June 29, 2004 11:14 AM

Looks like there are more places where kernel does "virt -> phys -> virt". And conversion from phys to virt is setting 3 msb to 1.

- Ken

Ken> How does this patch look? It is a bit big. But what it does is really Ken> simple: change 3 call sites to save/restore virtual address of sp and Ken> ar.bsp/ar.bspstore.

Looks fine to me.

Thanks,

	--david

Ken> Looks like there are more places where kernel does "virt -> Ken> phys -> virt". And conversion from phys to virt is setting 3 Ken> msb to 1.

Your fixes work fine on the machines I have tried so far (variety of Itanium 2 boxen and a Big Sur Merced box).

Jesse, do you know if unapplying the "move current to region 5" patch fixes your boot-problem? For convenience, I attached the (original) patch below.

	--david

# arch/ia64/kernel/head.S

#   2004/06/16 18:09:33-07:00 [email protected] +5 -1
#   (_start): Initialize "current" pointer for init-task to be in
#   	region 5, not 7.
# 
# arch/ia64/kernel/entry.S
#   2004/06/16 18:09:33-07:00 [email protected] +4 -2
#   (ia64_switch_to): Don't try to map "current"-pointers which are
#   	inside region 5.

#

diff -Nru a/arch/ia64/kernel/entry.S b/arch/ia64/kernel/entry.S
@@ -179,17 +179,19 @@
 	.body
 
 	adds r22=IA64_TASK_THREAD_KSP_OFFSET,r13
+	movl r25=init_task
 	mov r27=IA64_KR(CURRENT_STACK)
+	adds r21=IA64_TASK_THREAD_KSP_OFFSET,in0
 	dep r20=0,in0,61,3		// physical address of "current"
 	;;
 	st8 [r22]=sp			// save kernel stack pointer of old task
 	shr.u r26=r20,IA64_GRANULE_SHIFT
-	adds r21=IA64_TASK_THREAD_KSP_OFFSET,in0
+	cmp.eq p7,p6=r25,in0
 	;;
 	/*
 	 * If we've already mapped this task's page, we can skip doing it again.
 	 */
-	cmp.eq p7,p6=r26,r27
+(p6)	cmp.eq p7,p6=r26,r27
 (p6)	br.cond.dpnt .map
 	;;
 .done:

diff -Nru a/arch/ia64/kernel/head.S b/arch/ia64/kernel/head.S
@@ -154,6 +154,10 @@
 #endif
 	;;
 	tpa r3=r2		// r3 = phys addr of task struct
+	;;
+	shr.u r16=r3,IA64_GRANULE_SHIFT
+(isBP)	br.cond.dpnt .load_current // BP stack is on region 5 --- no need to map it
+
 	// load mapping for stack (virtaddr in r2, physaddr in r3)
 	rsm psr.ic
 	movl r17=PAGE_KERNEL

@@ -165,7 +169,6 @@
 	dep r2=-1,r3,61,3	// IMVA of task
 	;;
 	mov r17=rr[r2]
-	shr.u r16=r3,IA64_GRANULE_SHIFT
 	;;
 	dep r17=0,r17,8,24
 	;;

@@ -180,6 +183,7 @@
 	srlz.d
   	;;
 
+.load_current:
 	// load the "current" pointer (r13) and ar.k6 with the current task
 	mov IA64_KR(CURRENT)=r2		// virtual address
 	mov IA64_KR(CURRENT_STACK)=r16

Yes, reverting that patch does fix the hang I'm seeing.

Jesse

Yep, that looks like a problem. The kernel hangs right after the local_flush_tlb_all in ia64_tlb_init, and if I comment it out I get a "Unable to handle kernel paging request at virtual address a000003004289f70" which, on sn2, looks like a virt -> phys -> virt conversion (i.e. 0x3004289f70 is a valid physical address).

Jesse

Ken> Looks like there are more places where kernel does "virt -> Ken> phys -> virt". And conversion from phys to virt is setting 3 Ken> msb to 1.

Jesse> Yes, reverting that patch does fix the hang I'm seeing.

A quick grep shows that MINSTATE_END_SAVE_MIN_PHYS would also convert the region 5 "current" into a region 7 address. An MCA at the right time might cause problems, I think.

	--david

Jesse> Yep, that looks like a problem. The kernel hangs right after Jesse> the local_flush_tlb_all in ia64_tlb_init, and if I comment it Jesse> out I get a "Unable to handle kernel paging request at Jesse> virtual address a000003004289f70" which, on sn2, looks like a Jesse> virt -> phys -> virt conversion (i.e. 0x3004289f70 is a valid Jesse> physical address).

I'm not sure this is related. local_flush_tlb_all() simply flushes the entire tlb it doesn't touch the "current" pointer in any shape or fashion. Not doing that flush will leave the kernel with stale TLB entries, so all bets are off.

	--david

Yeah, this is happening later, so it's probably a separate issue (and maybe not an issue at all). Nonetheless, we still see a hang at boot with the latest BK tree.

Jesse

If you hack your elilo to load the kernel higher in memory do you see the same thing?

Jesse

Applying this small patch to head.S gets me as far as Grant's original report.

-(isBP) br.cond.dpnt .load_current
+//(isBP) br.cond.dpnt .load_current

This lets me get to:

PID hash table entries: 4096 (order 12: 65536 bytes)
CPU 0: base freq 0.000MHz, ITC ratio/2, ITC freq00.000MHz+/--1ppm
Console: colour dummy device 80x25

And like I've already mentioned, if I revert the whole thing the kernel boots fine. Ugg, I'm really starting to dislike the move of the init_task...

Jesse

It is something related to stack pointer (r12). If sp stays in region 7 (providing there is a corresponding dtr mapping), kernel boots fine. David mentioned that MCA code also has tpa thingy in it that needs to be converted. One other thing I'm getting frustrated at is I'm keep on receiving mca while doing a ptc.e (only on sgi altix). Have no idea why.

- Ken

I get different behavior depending on whether I'm on a partitioned system or
not: either a machine check caused by a null pointer dereference in the unwind code on a non-partitioned machine, or a hang on a partitioned machine.

The difference could be in leftover tc entries from the PROM. They'll get purged when the first ptc.e happens, causing a fault on the very next instruction or data reference following the purge. Just a guess.

Jesse

Jesse> And like I've already mentioned, if I revert the whole thing Jesse> the kernel boots fine. Ugg, I'm really starting to dislike Jesse> the move of the init_task...

I suppose you Would rather have me undo the virtual mapping of the kernel? That's the only "easy" solution I can think of, but that would be a definite step back in other ways.

	--david

Yeah, that would be much worse. This is just an ugly problem, that's all.

Jesse

That's what I'm seeing too, any ideas where to look for the problem? I'm happy to test any patches you might have :) I'm not really sure what's going on, but it looks like it might be:

_start -> start_kernel -> setup_arch -> ia64_mmu_init -> ptc.e -> page fault
-> die

which would indicate (given that we know that the ptc.e arguments are valid) that we're purging a TC entry that keeps us going up to that point, and that we're not using the region 5 mapping like we should. Does that sound right at all?

But that's not the cause of the problem we're seeing, right?

You don't think this is a symptom of the region 7 vs. region 5 problem? It looks to me like we're doing the ptc.e with the same arguments in either case...

Thanks,
Jesse