Skip to contentSkip to conversation

kernel 2.6.16-1.2097_FC6 unbootable on Itanium

Fedora-ia64-list Prarit Bhargava 14 participants 33 messages Mar 29, 2006 – Mar 30, 2006

lore
Share & tools
lore
#0 lore raw
Quoted 11 lines · Prarit Bhargava
Quoted 6 lines · Émeric Maschino

Hi,

There's something wrong with kernel 2.6.16-1.2097_FC6. The last displayed message is "Freeing unused kernel memory: 400kB freed" and then nothing. kernel-2.6.16-1.2088_FC6 runs fine.

I saw an error spit out while the RPM install was assembling the initrd.
I'll look into it.

Looks like upstream 2.6.16-1 doesn't boot either.

Adding linux-ia64 to the list ...

P.

Quoted 3 lines

P.

#1 lore raw
Quoted 4 lines · Prarit Bhargava
Quoted 2 lines · Émeric Maschino

Hi,

Quoted 8 lines
Quoted 4 lines

There's something wrong with kernel 2.6.16-1.2097_FC6. The last displayed message is "Freeing unused kernel memory: 400kB freed" and then nothing. kernel-2.6.16-1.2088_FC6 runs fine.

I saw an error spit out while the RPM install was assembling the initrd.
I'll look into it.

Looks like upstream 2.6.16-1 doesn't boot either.

Adding linux-ia64 to the list ...

P.

Quoted 2 lines

P.

#2 lore raw
In reply to Prarit Bhargava

Prarit Bhargava wrote on Wednesday, March 29, 2006 8:34 AM

Quoted 3 lines

Looks like upstream 2.6.16-1 doesn't boot either.

Adding linux-ia64 to the list ...

Just tried upstream 2.6.16.1 on my Intel tiger4 machine. It boots OK. What kind of problem do you have? Where does it hang?

- Ken

#3 lore raw
In reply to Prarit Bhargava
Quoted 10 lines · Kenneth W

Prarit Bhargava wrote on Wednesday, March 29, 2006 8:34 AM

Quoted 3 lines

Looks like upstream 2.6.16-1 doesn't boot either.

Adding linux-ia64 to the list ...

Just tried upstream 2.6.16.1 on my Intel tiger4 machine. It boots OK. What kind of problem do you have? Where does it hang?

I'm trying to get a hold of Emeric to find out what system he's using :)

But it looks like it is hanging on Altix's at

Calling initcall 0xa0000001006ad230: bictcp_register+0x0/0x40() TCP bic registered
Calling initcall 0xa0000001006b1160: xfrm_user_init+0x0/0xe0() Initializing IPsec netlink socket
Calling initcall 0xa0000001006ade70: af_unix_init+0x0/0x120()
NET: Registered protocol family 1
Calling initcall 0xa0000001006adfc0: packet_init+0x0/0x100()
NET: Registered protocol family 17
Calling initcall 0xa0000001006a9670: acpi_poweroff_init+0x0/0x100() Calling initcall 0xa0000001006af2d0: seqgen_init+0x0/0x40() Calling initcall 0xa0000001006b18d0: early_uart_console_switch+0x0/0x120() Calling initcall 0xa0000001006a8e10: net_random_reseed+0x0/0x100() Freeing unused kernel memory: 400kB freed

P.

Quoted 6 lines

- Ken
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
In reply to Prarit Bhargava

On Wed, Mar 29, 2006 at 01:55:26PM -0500, Prarit Bhargava wrote:

| But it looks like it is hanging on Altix's at
| 
| Calling initcall 0xa0000001006ad230: bictcp_register+0x0/0x40()
| TCP bic registered
| Calling initcall 0xa0000001006b1160: xfrm_user_init+0x0/0xe0()
| Initializing IPsec netlink socket
| Calling initcall 0xa0000001006ade70: af_unix_init+0x0/0x120()
| NET: Registered protocol family 1
| Calling initcall 0xa0000001006adfc0: packet_init+0x0/0x100()
| NET: Registered protocol family 17
| Calling initcall 0xa0000001006a9670: acpi_poweroff_init+0x0/0x100()
| Calling initcall 0xa0000001006af2d0: seqgen_init+0x0/0x40()
| Calling initcall 0xa0000001006b18d0: early_uart_console_switch+0x0/0x120()
| Calling initcall 0xa0000001006a8e10: net_random_reseed+0x0/0x100()
| Freeing unused kernel memory: 400kB freed

Is it hung, or did it MCA? Check the leds (ctrl-t to L2 from console, then 'leds'). If you repeat this procedure, you should see the LED values changing.

Greg

#5 lore raw
In reply to Prarit Bhargava

Prarit Bhargava wrote on Wednesday, March 29, 2006 10:55 AM

Quoted 8 lines
Quoted 3 lines

Just tried upstream 2.6.16.1 on my Intel tiger4 machine. It boots OK. What kind of problem do you have? Where does it hang?

I'm trying to get a hold of Emeric to find out what system he's using :)

But it looks like it is hanging on Altix's at

Quoted 13 lines

Calling initcall 0xa0000001006ad230: bictcp_register+0x0/0x40() TCP bic registered
Calling initcall 0xa0000001006b1160: xfrm_user_init+0x0/0xe0() Initializing IPsec netlink socket
Calling initcall 0xa0000001006ade70: af_unix_init+0x0/0x120()
NET: Registered protocol family 1
Calling initcall 0xa0000001006adfc0: packet_init+0x0/0x100()
NET: Registered protocol family 17
Calling initcall 0xa0000001006a9670: acpi_poweroff_init+0x0/0x100() Calling initcall 0xa0000001006af2d0: seqgen_init+0x0/0x40() Calling initcall 0xa0000001006b18d0: early_uart_console_switch+0x0/0x120() Calling initcall 0xa0000001006a8e10: net_random_reseed+0x0/0x100() Freeing unused kernel memory: 400kB freed

I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't boot :-(

So far we have reports on:

(1) 2.6.16.1 doesn't boot on altix
(2) 2.6.16-git17 doesn't boot on tiger4
(3) Jes's strange .section causing some hangs.

Hmm, I don't know what's going on ....

- Ken

#6 lore raw
In reply to Prarit Bhargava
Quoted 8 lines

I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't boot :-(

So far we have reports on:

(1) 2.6.16.1 doesn't boot on altix
(2) 2.6.16-git17 doesn't boot on tiger4
(3) Jes's strange .section causing some hangs.

Tiger still boots with my release tree (just has seven patches that haven't been sent to Linus yet, including Russ' fix for my merge goof that created item "3" above). I haven't pulled in from Linus in about three days, so I'm about 458 commits behind the bleeding edge. So a "git bisect" between Linus head and the point where I last merged (commit 64bc0430) might yield the problem (for Ken's tiger boot issue).

-Tony

Here's what's in release/test that hasn't gone to Linus yet:

 arch/ia64/kernel/iosapic.c          |  279 +++++++++++++++++++++---------------
 arch/ia64/kernel/vmlinux.lds.S      |   18 +-
 arch/ia64/mm/init.c                 |    8 -
 arch/ia64/mm/tlb.c                  |   12 -
 arch/ia64/sn/kernel/sn2/sn_hwperf.c |    8 -
 5 files changed, 198 insertions(+), 127 deletions(-)

Chen, Kenneth W:
[IA64] optimize flush_tlb_range on large numa box

Dean Roe:
[IA64-SGI] fix for-loop in sn_hwperf_geoid_to_cnode()

[email protected]:
[IA64-SGI] sn_hwperf use of num_online_cpus()

Russ Anderson:
[IA64] Move __mca_table out of the __init section

Satoru Takeuchi:
[IA64] correct some messages and fixes some minor things [IA64] simplify some condition checks in iosapic_check_gsi_range

Zhang, Yanmin:
[IA64] lazy_mmu_prot_update needs to be aware of huge pages

#7 lore raw
In reply to Prarit Bhargava
Quoted 8 lines · Kenneth W

I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't boot :-(

So far we have reports on:

(1) 2.6.16.1 doesn't boot on altix
(2) 2.6.16-git17 doesn't boot on tiger4
(3) Jes's strange .section causing some hangs.

(4) James Bottomley couldn't boot his ZX2000 (HP ZX1 chipset) from git yesterday either.
He had to update firmware to 2.31 version because of some changes in 2.6.16 SAL usage exposed an older SAL bug. Alex Williamson had dug up the details and I'm blanking on them right now.
But his zx2000 is still not able to boot.

Could be multiple, unrelated problems being reported now.

grant

#8 lore raw

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on

In reply to Prarit Bhargava
Quoted 8 lines · Grant Grundler

(4) James Bottomley couldn't boot his ZX2000 (HP ZX1 chipset) from git yesterday either.
He had to update firmware to 2.31 version because of some changes in 2.6.16 SAL usage exposed an older SAL bug. Alex Williamson had dug up the details and I'm blanking on them right now.
But his zx2000 is still not able to boot.

Could be multiple, unrelated problems being reported now.

No, I can confirm that reverting

[IA64] MCA Recovery: kernel context recovery table

fixes my boot problems.

I actually got slightly further than Jes with the analysis. The problem occurs because something in this changeset makes code that dynamically loads the TLS libraries segfault. If you look, Jes, you'll find init is continually segfaulting and respawning.

James

#9 lore raw
In reply to Prarit Bhargava
Quoted 7 lines

Tiger still boots with my release tree (just has seven patches that haven't been sent to Linus yet, including Russ' fix for my merge goof that created item "3" above). I haven't pulled in from Linus in about three days, so I'm about 458 commits behind the bleeding edge. So a "git bisect" between Linus head and the point where I last merged (commit 64bc0430) might yield the problem (for Ken's tiger boot issue).

Just started the bisection ... and the current tip of Linus' git tree (f3cab8a0) booted just fine on tiger! There are 85 commits in there since git-17 was cut, but a quick glance at gitk doesn't show any obvious fixes or reverts that might affect ia64.

-Tony

#10 lore raw
In reply to Prarit Bhargava

This is my turn to fill in the report: yesterday's Rawhide kernel 2.6.16-1.2097_FC6 (based on 2.6.16-git9, git10 and git 13) and today's Rawhide kernel 2.6.16-1.2102_FC6 (based on 2.6.16-git14, 2.6.16-git15) don't boot on hp workstation zx6000.

Émeric

Quoted 9 lines

So far we have reports on:

(1) 2.6.16.1 doesn't boot on altix
(2) 2.6.16-git17 doesn't boot on tiger4
(3) Jes's strange .section causing some hangs.

Hmm, I don't know what's going on ....

- Ken

#11 lore raw
In reply to Prarit Bhargava
Quoted 8 lines · Tony
Quoted 7 lines

Tiger still boots with my release tree (just has seven patches that haven't been sent to Linus yet, including Russ' fix for my merge goof that created item "3" above). I haven't pulled in from Linus in about three days, so I'm about 458 commits behind the bleeding edge. So a "git bisect" between Linus head and the point where I last merged (commit 64bc0430) might yield the problem (for Ken's tiger boot issue).

Quoted 5 lines

Just started the bisection ... and the current tip of Linus' git tree (f3cab8a0) booted just fine on tiger! There are 85 commits in there since git-17 was cut, but a quick glance at gitk doesn't show any obvious fixes or reverts that might affect ia64.

I don't know if this helps or not.

I ran Jes's kernel on the simulator. Unfortunately, he sent me a stripped kernel so I have no symbol table.

The kernel blows up right after printing:

	...
	Virtual mem_map starts at 0xa0007fffd5f2c000
	Built 1 zonelists
	Kernel command line: root=/dev/hda2 init=/bin/bash console=ttyS0
	PID hash table entries: 1024 (order: 10, 32768 bytes)
	Console: colour dummy device 80x25

The failure is an MCA caused by a cache hit on a memory reference to an uncached address. The simulator detects this error & stops.

The code that took the failure was memcopy (or equiv). I recognized the code from the prefetchs and ld/st sequence. The data appears to be an ACPI table that is being copied into kernel memory. The current reference is using uncached addresses but 15M instructions in the past, the table was referenced cached.

Does this ring any bells with anyone? This failure may occur only in our simulator environment, so don't spent any time on this unless it sounds familar.

As soon as I get a symbol table, I'll know a lot more about the failure.

---
Jack

#12 lore raw
In reply to Prarit Bhargava
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Tony" = Luck, Tony <[email protected]> writes:

I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't boot :-(

So far we have reports on:

(1) 2.6.16.1 doesn't boot on altix (2) 2.6.16-git17 doesn't boot on tiger4 (3) Jes's strange .section causing some hangs.

Tony> Tiger still boots with my release tree (just has seven patches Tony> that haven't been sent to Linus yet, including Russ' fix for my Tony> merge goof that created item "3" above). I haven't pulled in Tony> from Linus in about three days, so I'm about 458 commits behind Tony> the bleeding edge. So a "git bisect" between Linus head and the Tony> point where I last merged (commit 64bc0430) might yield the Tony> problem (for Ken's tiger boot issue).

Tony,

Please note that if I compile the kernel on a box with an old installation and old toolchain the kernel boots, same source tree built on a system with a new installation and it doesn't boot.

Cheers,
Jes

#13 lore raw
In reply to Prarit Bhargava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jack" = Jack Steiner <[email protected]> writes:

Jack> ... Virtual mem_map starts at 0xa0007fffd5f2c000 Built 1
Jack> zonelists Kernel command line: root=/dev/hda2 init=/bin/bash Jack> console=ttyS0 PID hash table entries: 1024 (order: 10, 32768 Jack> bytes) Console: colour dummy device 80x25

Jack> The failure is an MCA caused by a cache hit on a memory Jack> reference to an uncached address. The simulator detects this Jack> error & stops.

Hi Jack,

Sorry about the missing symbol table, was a bit late.

I am not sure it's the same thing, as my kernel gets a lot further and normally spawns init before exploding.

Jack> The code that took the failure was memcopy (or equiv). I Jack> recognized the code from the prefetchs and ld/st sequence. The Jack> data appears to be an ACPI table that is being copied into Jack> kernel memory. The current reference is using uncached addresses Jack> but 15M instructions in the past, the table was referenced Jack> cached.

Jack> Does this ring any bells with anyone? This failure may occur Jack> only in our simulator environment, so don't spent any time on Jack> this unless it sounds familar.

Jack> As soon as I get a symbol table, I'll know a lot more about the Jack> failure.

I put one up on attica for you ~jes/System.map-jack

It seems the kernel explodes in __copy_user in your case.

Cheers,
Jes

#14 lore raw
In reply to Prarit Bhargava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Prarit" = Prarit Bhargava <[email protected]> writes:

Prarit> Looks like upstream 2.6.16-1 doesn't boot either.

Prarit> Adding linux-ia64 to the list ...

Please don't cross post to linux-ia64 when you have a fscked up list on the CC, the fedora list is one of those hopeless subscribers only things ;-(

Jes

#15 lore raw
In reply to Prarit Bhargava
Quoted 18 lines · Jes Sorensen (on 30 Mar 2006 03:57:04 -0500) wrote:
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Tony" = Luck, Tony <[email protected]> writes:

I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't boot :-(

So far we have reports on:

(1) 2.6.16.1 doesn't boot on altix (2) 2.6.16-git17 doesn't boot on tiger4 (3) Jes's strange .section causing some hangs.

Tony> Tiger still boots with my release tree (just has seven patches Tony> that haven't been sent to Linus yet, including Russ' fix for my Tony> merge goof that created item "3" above). I haven't pulled in Tony> from Linus in about three days, so I'm about 458 commits behind Tony> the bleeding edge. So a "git bisect" between Linus head and the Tony> point where I last merged (commit 64bc0430) might yield the Tony> problem (for Ken's tiger boot issue).

Quoted 5 lines

Tony,

Please note that if I compile the kernel on a box with an old installation and old toolchain the kernel boots, same source tree built on a system with a new installation and it doesn't boot.

This sounds like a random data overwrite which depends on the exact kernel layout. Somebody else in SGI (Prarit ?) said that they compiled your failing tree on the same platform and their kernel worked. Since the build userid and the pathname to the kernel source tree are embedded in the binary, changing either of those will disturb the kernel layout, possibly making the problem move.

Jes, try renaming your source tree to something with a different pathname length and rebuild the kernel. Does the problem change or go away?

#16 lore raw
In reply to Prarit Bhargava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Keith" = Keith Owens <[email protected]> writes:

Quoted 3 lines · Keith> Jes Sorensen (on 30 Mar 2006 03:57:04 -0500) wrote:
Quoted 3 lines

Please note that if I compile the kernel on a box with an old installation and old toolchain the kernel boots, same source tree built on a system with a new installation and it doesn't boot.

Keith> This sounds like a random data overwrite which depends on the Keith> exact kernel layout. Somebody else in SGI (Prarit ?) said that Keith> they compiled your failing tree on the same platform and their Keith> kernel worked. Since the build userid and the pathname to the Keith> kernel source tree are embedded in the binary, changing either Keith> of those will disturb the kernel layout, possibly making the Keith> problem move.

Keith> Jes, try renaming your source tree to something with a Keith> different pathname length and rebuild the kernel. Does the Keith> problem change or go away?

Moved it to /tmp/a and built there - no go.

Cheers,
Jes

#17 lore raw
In reply to Prarit Bhargava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"James" = James Bottomley <[email protected]> writes:

James> No, I can confirm that reverting

James> [IA64] MCA Recovery: kernel context recovery table

James> fixes my boot problems.

James> I actually got slightly further than Jes with the analysis. James> The problem occurs because something in this changeset makes James> code that dynamically loads the TLS libraries segfault. If you James> look, Jes, you'll find init is continually segfaulting and James> respawning.

Yay!

I am not the only one seeing this! I am not insane!

I bet that if you revert the patch above but then add these two lines to arch/ia64/kernel/gate.S you'll see the same failure:

	.section "__mca_table", "a"
	.previous

Tracked it down further. If I take the gate.o file from a working kernel and copy it into the broken tree and run make again, the resulting kernel boots.

I compared the code between a working and a broken gate.o with objdump and it's identical. However the good file is 3552 bytes whereas the broken one is 3648.

objdump -h reveals one extra empty section which seems to be what makes all the difference. The below output is from a broken gate.o, the only difference I can find comparing it to the good one is that idx 4 is missing in the good one and the following ones are renamed. Do we have some strange restrictions in the linker on the number of sections it can handle?

Cheers,
Jes

gate.o: file format elf64-ia64-little

Sections:

Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         000003b0  0000000000000000  0000000000000000  00000040  2**5

CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .data 00000000 0000000000000000 0000000000000000 000003f0 2**0
CONTENTS, ALLOC, LOAD, DATA
2 .bss 00000000 0000000000000000 0000000000000000 000003f0 2**0
ALLOC
3 __ex_table 00000000 0000000000000000 0000000000000000 000003f0 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
4 __mca_table 00000000 0000000000000000 0000000000000000 000003f0 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
5 .data.patch.vtop 00000000 0000000000000000 0000000000000000 000003f0 2**0
CONTENTS, ALLOC, LOAD, DATA
6 .data.patch.mckinley_e9 00000004 0000000000000000 0000000000000000 000003f0 2**2
CONTENTS, ALLOC, LOAD, RELOC, DATA
7 .data.patch.fsyscall_table 00000004 0000000000000000 0000000000000000 000003f4 2**2
CONTENTS, ALLOC, LOAD, RELOC, DATA
8 .data.patch.brl_fsys_bubble_down 00000004 0000000000000000 0000000000000000 000003f8 2**2
CONTENTS, ALLOC, LOAD, RELOC, DATA
9 .IA_64.unwind_info 000000b8 0000000000000000 0000000000000000 00000400 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
10 .IA_64.unwind 00000048 0000000000000000 0000000000000000 000004b8 2**3
CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA

#18 lore raw
In reply to Prarit Bhargava
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jes" = Jes Sorensen <[email protected]> writes:

Jes> Tracked it down further. If I take the gate.o file from a working Jes> kernel and copy it into the broken tree and run make again, the Jes> resulting kernel boots.

Jes> I compared the code between a working and a broken gate.o with Jes> objdump and it's identical. However the good file is 3552 bytes Jes> whereas the broken one is 3648.

Looks like something goes wrong with some linking references due to this. I ran objdump on the two kernels I get when replacing gate.o and the diff is as attached below. Not sure if this is an ia64 specific bug or we're just 'lucky'.

Cheers,
Jes

broken/vmlinux.s renamed from good/vmlinux.s +6 -6
@@ -4981,8 +4981,8 @@
 a00000010000bd96:	30 01 44 44 08 40 	            mov.m r19=ar.bsp
 a00000010000bd9c:	01 00 40 00       	            mov f10ð
 a00000010000bda0:	05 00 00 00 01 00 	[MLX]       nop.m 0x0
-a00000010000bda6:	00 00 00 00 20 c0 	            movl r14=0xa000000000010640;;
-a00000010000bdac:	01 24 30 68 
+a00000010000bda6:	00 00 00 00 00 c0 	            movl r14=0x10080;;
+a00000010000bdac:	01 20 04 60 
 a00000010000bdb0:	08 00 00 32 2a 04 	[MMI]       mov.m ar25=r0
 a00000010000bdb6:	00 00 80 54 08 e0 	            mov.m ar.ccv=r0
 a00000010000bdbc:	e0 08 00 07       	            mov b7=r14
@@ -8192,8 +8192,8 @@
 a00000010000fe06:	00 01 00 00 20 c0 	            movl r14=0xa00000010000bc60
 a00000010000fe0c:	01 06 e0 6d 
 a00000010000fe10:	05 00 00 00 01 00 	[MLX]       nop.m 0x0
-a00000010000fe16:	00 00 00 00 20 80 	            movl r28=0xa000000000010620;;
-a00000010000fe1c:	03 22 30 68 
+a00000010000fe16:	00 00 00 00 00 80 	            movl r28=0x10060;;
+a00000010000fe1c:	03 26 00 60 
 a00000010000fe20:	01 10 00 20 00 21 	[MII]       mov r2=r16
 a00000010000fe26:	00 21 42 0c 42 e0 	            adds r16ƒ6,r16
 a00000010000fe2c:	03 00 cc 00       	            mov r31=pr;;
@@ -1563052,7 +1563052,7 @@
 a00000010091856c:	03 73 ff 00       	            data8 0x1fee607fe
 a000000100918570:	73 ec 02 f0 05 44 	[MBB] (p35) mov pmd[r120]=r0
 a000000100918576:	ee 02 f0 06 46 e8 	            data8 0x1181bc00bb9
-a00000010091857c:	02 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff957570 <__kernel_sigtramp+0xff946e50>;;
+a00000010091857c:	02 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff957570 <__end_gate_brl_fsys_bubble_down_patchlist+0xff957064>;;
 a000000100918580:	e4 02 f0 01 36 e6 	[MLX] (p23) mov r0=-2090116
 a000000100918586:	02 f0 02 38 e1 02 	            data8 0x1be0edc605
 a00000010091858c:	e3 76 f0 0d 
@@ -1563061,7 +1563061,7 @@
 a00000010091859c:	03 73 ff 00       	            data8 0x1fee607fe
 a0000001009185a0:	73 ec 00 f0 05 44 	[MBB] (p35) mov pmd[r120]=r0
 a0000001009185a6:	ee 00 f0 06 46 e8 	            data8 0x1181bc003b9
-a0000001009185ac:	00 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff9575a0 <__kernel_sigtramp+0xff946e80>;;
+a0000001009185ac:	00 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff9575a0 <__end_gate_brl_fsys_bubble_down_patchlist+0xff957094>;;
 a0000001009185b0:	e4 00 f0 01 36 e6 	[MLX] (p07) mov r0=-2090116
 a0000001009185b6:	00 f0 02 38 e1 00 	            data8 0x1cdf2edc601
 a0000001009185bc:	e3 76 f9 e6 
#19 lore raw
In reply to Prarit Bhargava
Quoted 14 lines

I don't know if this helps or not.

I ran Jes's kernel on the simulator. Unfortunately, he sent me a stripped kernel so I have no symbol table.

The kernel blows up right after printing:

	...
	Virtual mem_map starts at 0xa0007fffd5f2c000
	Built 1 zonelists
	Kernel command line: root=/dev/hda2 init=/bin/bash console=ttyS0
	PID hash table entries: 1024 (order: 10, 32768 bytes)
	Console: colour dummy device 80x25
Quoted 8 lines

The failure is an MCA caused by a cache hit on a memory reference to an uncached address. The simulator detects this error & stops.

The code that took the failure was memcopy (or equiv). I recognized the code from the prefetchs and ld/st sequence. The data appears to be an ACPI table that is being copied into kernel memory. The current reference is using uncached addresses but 15M instructions in the past, the table was referenced cached.

I found the problem that is causing the cache/uncached collision. This is most likely NOT what is causing the problem you are currently chasing, but it is a problem never-the-less.

(Background: I noticed that recent IA64 kernels are making references to the same address using both cached & uncached addresses. The kernel fails to boot on the simulator because it detects a cache-hit on an uncached reference. Mixing cached & uncached references is not supported).

acpi_os_map_memory() has changed the order of checking the EFI attributes on pages being mapped. In 2.6.15, if the EFI map indicates that an address supports both cached & uncached references, acpi_os_map_memory() would map it as cached.

The most recent kernel has changed this behavior so that the address will be referenced as uncached.

OLD:

        acpi_os_map_memory
                if (EFI_MEMORY_WB & efi_mem_attributes(phys)) {
                        *virt = (void __iomem *)phys_to_virt(phys);
                } else {
                        *virt = ioremap(phys, size);
                }

NEW
acpi_os_map_memory() unconditionally calls ioremap() to do the mapping

ioremap
if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
return __ioremap(offset, size);

if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
return phys_to_virt(offset);

Earlier in the boot, the ACPI tables are unconditionally references as cached:

        char *__acpi_map_table(unsigned long phys_addr, unsigned long size)
        {
                return __va(phys_addr);
        }

I suspect there was a reason for this change. I'll add Bjorn to the cc list.

Is this problem unique to SN systems. The BIOS reports that most memory ranges support both CACHED & UNCACHED references. I _think_ this is correct.

#20 lore raw
In reply to Prarit Bhargava

Hello all,

Quoted 5 lines
---
 arch/ia64/kernel/gate.lds.S |    1 +
 include/asm-ia64/asmmacro.h |    4 ++++
 2 files changed, 5 insertions(+)

I applied this patch and compiled using gcc-4.1.0-3.

I now see hundreds of warnings like:

WARNING: drivers/atm/he.o - Section mismatch: reference to .init.text:he_start from .text after 'he_init_one' (at offset 0x81a2)
WARNING: drivers/block/cpqarray.o - Section mismatch: reference to .init.text:cpqarray_init_one from .data.rel.local between 'cpqarray_pci_driver' (at offset 0x20) and 'smart1_access'

Just an FYI,

P.

#21 lore raw
In reply to Prarit Bhargava
Quoted 1 line · Jack Steiner

Is this problem unique to SN systems

No, the same will happen on all other systems (that I know of).

Quoted 3 lines

The BIOS reports that most
memory ranges support both CACHED & UNCACHED references. I _think_ this is correct.

That's correct. The map shows the ways the page *can* be mapped, not the way it *should* be mapped.

It's strange that ACPI would prefer to use WC when WB mapping is possible. They definitely need to pick one way and stick with it though. As you say, mapping the same page with different cacheability is a no-no (and at least in theory, it should cause an MCA even on real hardware).

--david

--
Mosberger Consulting LLC, http://www.mosberger-consulting.com/

#22 lore raw
In reply to Prarit Bhargava
Quoted 10 lines · David Mosberger-Tang
Quoted 1 line · Jack Steiner

Is this problem unique to SN systems

No, the same will happen on all other systems (that I know of).

Quoted 3 lines

The BIOS reports that most
memory ranges support both CACHED & UNCACHED references. I _think_ this is correct.

Quoted 8 lines

That's correct. The map shows the ways the page *can* be mapped, not the way it *should* be mapped.

It's strange that ACPI would prefer to use WC when WB mapping is possible. They definitely need to pick one way and stick with it though. As you say, mapping the same page with different cacheability is a no-no (and at least in theory, it should cause an MCA even on real hardware).

It does, at least on our chipset. If the chipset detects a simultaneous UC & C reference, it generates a BUS error. It is surprising how quickly this MCA occurs when we break the rules.

Quoted 5 lines

--david

--
Mosberger Consulting LLC, http://www.mosberger-consulting.com/

Signature · 1 line
Jack
#23 lore raw
In reply to Prarit Bhargava
Quoted 7 lines · Jack Steiner
Quoted 2 lines · David Mosberger-Tang

(and at least in theory, it should cause an MCA even on real hardware).

It does, at least on our chipset. If the chipset detects a simultaneous UC & C reference, it generates a BUS error. It is surprising how quickly this MCA occurs when we break the rules.

That's good. While MCAs are a pain, it's a lot better than having to track down data corruption due to (in-)coherency.

--david
--
Mosberger Consulting LLC, http://www.mosberger-consulting.com/

#24 lore raw
In reply to Prarit Bhargava
Quoted 10 lines · Prarit Bhargava

Hello all,

Quoted 5 lines

---

arch/ia64/kernel/gate.lds.S |    1 +
include/asm-ia64/asmmacro.h |    4 ++++
2 files changed, 5 insertions(+)

I applied this patch and compiled using gcc-4.1.0-3.

Quoted 8 lines

I now see hundreds of warnings like:

WARNING: drivers/atm/he.o - Section mismatch: reference to .init.text:he_start from .text after 'he_init_one' (at offset 0x81a2)
WARNING: drivers/block/cpqarray.o - Section mismatch: reference to .init.text:cpqarray_init_one from .data.rel.local between 'cpqarray_pci_driver' (at offset 0x20) and 'smart1_access'

Where did they come from?

H.J.

#28 lore raw
In reply to Prarit Bhargava
Quoted 17 lines · Jack Steiner

acpi_os_map_memory() has changed the order of checking the EFI attributes on pages being mapped. In 2.6.15, if the EFI map indicates that an address supports both cached & uncached references, acpi_os_map_memory() would map it as cached.

OLD:

        acpi_os_map_memory
                if (EFI_MEMORY_WB & efi_mem_attributes(phys)) {
                        *virt = (void __iomem *)phys_to_virt(phys);
                } else {
                        *virt = ioremap(phys, size);
                }

NEW
acpi_os_map_memory() unconditionally calls ioremap() to do the mapping

Quoted 6 lines

ioremap
if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
return __ioremap(offset, size);

if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
return phys_to_virt(offset);

I debated whether to check for UC or WB first, and I think I got it wrong. My reasoning for UC first was that it was a smaller change for the typical case CSR mapping case, because ioremap() used to *always* return UC mappings.

But efi_memmap_init() collects full granules of WB memory, without regard for whether they also support UC. So if ioremap() needs to work for main memory (which is slightly strange in itself, but seems to be the precedent), it needs to prefer WB mappings when possible.

Quoted 6 lines

Earlier in the boot, the ACPI tables are unconditionally references as cached:

        char *__acpi_map_table(unsigned long phys_addr, unsigned long size)
        {
                return __va(phys_addr);
        }

This seems suspicious to me. But we look at some of these ACPI tables prior to calling efi_memmap_init(), so it might not be safe to use ioremap() yet. It's conceivable that a platform could place ACPI tables in UC-only ROM, and this would blow up. But I guess we haven't tripped over that yet.

Does the patch below solve your immediate problem?

[IA64] ioremap() should prefer WB over UC

efi_memmap_init() collects full granules of WB memory, without regard for whether they also support UC. So in order for ioremap() to work for main memory, it must prefer WB mappings when possible.

Index: work-mm5/arch/ia64/mm/ioremap.c
=================================--- work-mm5.orig/arch/ia64/mm/ioremap.c 2006-03-23 10:22:40.000000000 -0700
+++ work-mm5/arch/ia64/mm/ioremap.c 2006-03-30 09:36:57.000000000 -0700
@@ -21,12 +21,12 @@
void __iomem *
ioremap (unsigned long offset, unsigned long size)
{
- if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
- return __ioremap(offset, size);
-
if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
return phys_to_virt(offset);

+ if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
+ return __ioremap(offset, size);
+
/*
* Someday this should check ACPI resources so we
* can do the right thing for hot-plugged regions.

#29 lore raw
In reply to Prarit Bhargava
Quoted 42 lines · Bjorn Helgaas
Quoted 17 lines · Jack Steiner

acpi_os_map_memory() has changed the order of checking the EFI attributes on pages being mapped. In 2.6.15, if the EFI map indicates that an address supports both cached & uncached references, acpi_os_map_memory() would map it as cached.

OLD:

        acpi_os_map_memory
                if (EFI_MEMORY_WB & efi_mem_attributes(phys)) {
                        *virt = (void __iomem *)phys_to_virt(phys);
                } else {
                        *virt = ioremap(phys, size);
                }

NEW
acpi_os_map_memory() unconditionally calls ioremap() to do the mapping

Quoted 6 lines

ioremap
if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
return __ioremap(offset, size);

if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
return phys_to_virt(offset);

I debated whether to check for UC or WB first, and I think I got it wrong. My reasoning for UC first was that it was a smaller change for the typical case CSR mapping case, because ioremap() used to *always* return UC mappings.

But efi_memmap_init() collects full granules of WB memory, without regard for whether they also support UC. So if ioremap() needs to work for main memory (which is slightly strange in itself, but seems to be the precedent), it needs to prefer WB mappings when possible.

Quoted 6 lines

Earlier in the boot, the ACPI tables are unconditionally references as cached:

        char *__acpi_map_table(unsigned long phys_addr, unsigned long size)
        {
                return __va(phys_addr);
        }
Quoted 7 lines

This seems suspicious to me. But we look at some of these ACPI tables prior to calling efi_memmap_init(), so it might not be safe to use ioremap() yet. It's conceivable that a platform could place ACPI tables in UC-only ROM, and this would blow up. But I guess we haven't tripped over that yet.

Does the patch below solve your immediate problem?

Looks good.

Quoted 29 lines

[IA64] ioremap() should prefer WB over UC

efi_memmap_init() collects full granules of WB memory, without regard for whether they also support UC. So in order for ioremap() to work for main memory, it must prefer WB mappings when possible.

Index: work-mm5/arch/ia64/mm/ioremap.c
=================================> --- work-mm5.orig/arch/ia64/mm/ioremap.c 2006-03-23 10:22:40.000000000 -0700
+++ work-mm5/arch/ia64/mm/ioremap.c 2006-03-30 09:36:57.000000000 -0700

(quoted diff) +3 -3
@@ -21,12 +21,12 @@
 void __iomem *
 ioremap (unsigned long offset, unsigned long size)
 {
-	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
-		return __ioremap(offset, size);
-
 	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
 		return phys_to_virt(offset);
 
+	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
+		return __ioremap(offset, size);
+
 	/*
 	 * Someday this should check ACPI resources so we
 	 * can do the right thing for hot-plugged regions.
Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.
#30 lore raw
In reply to Prarit Bhargava
Quoted 9 lines

I applied this patch and compiled using gcc-4.1.0-3.

I now see hundreds of warnings like:

WARNING: drivers/atm/he.o - Section mismatch: reference to .init.text:he_start from .text after 'he_init_one' (at offset 0x81a2)
WARNING: drivers/block/cpqarray.o - Section mismatch: reference to .init.text:cpqarray_init_one from .data.rel.local between 'cpqarray_pci_driver' (at offset 0x20) and 'smart1_access'

Just tried with vanilla gcc 4.1.0, and I don't see these warnings.

BTW, these messages presumably come from scripts/mod/modpost.c

-Tony

#31 lore raw
In reply to Prarit Bhargava
Quoted 7 lines · Prarit Bhargava
Quoted 3 lines

Which program generated them?

[root@altix2 ~]# gcc --version
gcc (GCC) 4.1.0 20060304 (Red Hat 4.1.0-3)

Humph. I can't get this to happen again :/. Oh well ... if anyone else sees this issue please let me know.

P.

Quoted 6 lines

P.
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#32 lore raw
In reply to Prarit Bhargava

On Thu, Mar 30, 2006 at 09:16:11AM -0600, Jack Steiner wrote:
...

Quoted 2 lines

The most recent kernel has changed this behavior so that the address will be referenced as uncached.

...

Quoted 7 lines

NEW
acpi_os_map_memory() unconditionally calls ioremap() to do the mapping

ioremap
if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
return __ioremap(offset, size);

Could this be related to the issue of ioremap() vs ioremap_nocache() that Mathew Wilcox proposed a patch for?

	http://lkml.org/lkml/2006/3/30/251

grant