IBM x360 2.2.x boot failure, 2.4.9 works fine

IBM has a brand new xSeries PC server called the x360.
It has 1-4 Xeon MP's, DDR SDRAM, PCI-X backplane, IBM Summit chipset. Full specs are here:

http://www.pc.ibm.com/us/eserver/xseries/x360.html

Has anyone tried running Linux on one of these ?
I have tried a few versions and found:

2.2.18 fails in boot
2.2.20 fails in boot
2.2.21rc2 fails in boot
2.4.9 works fine!

I know the hardware is in good shape because a 2.4.9 kernel works fine on this machine. I have scoured the IBM site, linux-kernel, and Google for clues, but to no avail.

The boot sequence failure under the 2.2.x versions that I tried is always the same, it fails to recognize the IDE and SCSI devices. From the messages, the system appears to be deaf to interrupts and so it cannot recognize its devices. Notable messages from the boot sequence that support this idea are:

hda: IRQ probe failed (0)
hda: lost interrupt
floppy0: no floppy controllers found
keyboard: Timeout - AT keyboard not present?
scsi : aborting command due to timeout : pid 0, scsi0, channel 0, id 0, lun 0 Test Unit Ready 00 00 00 00 00

Below is the full boot sequence output as best I could reconstruct it. It is a bit fuzzy because the x360 lacks a serial port (maybe it has one accessible through the USB, but I don't have a cable). I pulled this output together through their ASM remote console redirect feature, which is less than convenient, leaving me with an error-probe cut and paste exercise.

Unfortunately I have this machine for a limited eval and must return it on Monday. I do hope to get it back ASAP to continue debugging, but that is at IBM's discretion.

Any suggestions welcome, I will post more as I learn it.

-=-=-=-=-=-=-

LILO boot:
Loading linux-2220...............
Uncompressing Linux... Ok, booting the kernel.

per-CPU timeslice cutoff: 0.00 usecs.
CPU0: Intel Intel(R) Xeon(TM) CPU 1.60GHz stepping 01
calibrating APIC timer ...
..... CPU clock speed is 1600.2578 MHz.
..... system bus clock speed is 100.0160 MHz. Booting processor 4 eip 2000
Calibrating delay loop... 3198.15 BogoMIPS
Intel machine check reporting enabled on CPU#4.
8K L1 data cache
1024K L3 cache
12K L1 instruction cache
8192K L1 instruction cache
CPU: L1 I Cache: 8204K L1 D Cache: 8K
CPU: L3 Cache: 1024K
CPU: Intel(R) Xeon(TM) CPU 1.60GHz
OK.
CPU4: Intel Intel(R) Xeon(TM) CPU 1.60GHz stepping 01
Total of 2 processors activated (6389.76 BogoMIPS). enabling symmetric IO mode... ...done.
ENABLING IO-APIC IRQs
init IO_APIC IRQs
IO-APIC (apicid-pin) 14-0, 14-3, 14-5, 14-11, 14-13, 14-15,
14-18WARNING: ASSIGN_IRQ_VECTOR wrapped back to 52

Linux version 2.2.20 ([email protected]) (gcc version egcs-2.91.66 1
9990314/Linux (egcs-1.1.2 release)) #1 SMP Fri Mar 15 14:12:38 EST 2002 BIOS-provided physical RAM map:
BIOS-e820: 0009d000 @ 00000000 (usable)
BIOS-e820: 7efea400 @ 00100000 (usable)
Warning only 960MB will be used.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: IBM ENSW Product ID: NF 6000R SMP APIC at: 0xFEE00000 Processor #0 Unknown CPU [15:1] APIC version 20
Processor #4 Unknown CPU [15:1] APIC version 20
I/O APIC #14 Version 17 at 0xFEC00000.
Processors: 2
WARNING: MP table in the EBDA can be UNSAFE, contact [email protected] if you experience SMP problems! mapped APIC to ffffe000 (fee00000)
mapped IOAPIC to ffffd000 (fec00000)
Detected 1600371 kHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop...

PCI: Probing PCI hardware
PCI: Discovered primary peer bus 01
PCI->APIC IRQ transform: (B0,I2,P0) -> 17
PCI->APIC IRQ transform: (B0,I3,P0) -> 42
PCI->APIC IRQ transform: (B0,I4,P0) -> 41
PCI->APIC IRQ transform: (B0,I15,P0) -> 16
PCI->APIC IRQ transform: (B1,I1,P0) -> 43
PCI->APIC IRQ transform: (B1,I2,P0) -> 24
PCI: BIOS reporting unknown device 0a:00
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Unix domain sockets 1.0 for Linux NET4.0.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
TCP: Hash tables configured (ehash 524288 bhash 65536)
Linux IP multicast router 0.06 plus PIM-SM
Initializing RT netlink socket
Starting kswapd v 1.5
Detected PS/2 Mouse Port.
Serial driver version 4.27 with no serial options enabled ttyS00 at 0x03f8 (irq = 4) is a 16550A
pty: 256 Unix98 ptys configured
PCI_IDE: unknown IDE controller on PCI bus 00 device 79, VID=1166, DID=0211
PCI_IDE: not 100% native mode: will probe irqs later
keyboard: Timeout - AT keyboard not present?
ide0: BM-DMA at 0x0700-0x0707, BIOS settings: hda:DMA, hdb:DMA
ide1: BM-DMA at 0x0708-0x070f, BIOS settings: hdc:pio, hdd:pio
hda: IRQ probe failed (0)
hda: IRQ probe failed (0)
hda: LG CD-ROM CRN-8245B, ATAPI CDROM drive
hda: IRQ probe failed (0)
hdb: IRQ probe failed (0)
hdb: IRQ probe failed (0)
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: lost interrupt
hda: lost interrupt
hda: ATAPI 24X CD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.11
Floppy drive(s): fd0 is 1.44M
floppy0: no floppy controllers found
sim710: No NCR53C710 adapter found.
(scsi0) <Adaptec AIC-7892 Ultra 160/m SCSI host adapter> found at PCI 0/4/0
(scsi0) Wide Channel, SCSI ID=7, 32/255 SCBs
(scsi0) Downloading sequencer code... 396 instructions downloaded
NCR53c406a: no available ports found
sym53c416.c: Version 1.0.0
DC390: 0 adapters found
scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.33/3.2.4
<Adaptec AIC-7892 Ultra 160/m SCSI host adapter>
scsi : 1 host.
scsi : aborting command due to timeout : pid 0, scsi0, channel 0, id 0, lun 0 Test Unit Ready 00 00 00 00 00
scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.33/3.2.4
<Adaptec AIC-7892 Ultra 160/m SCSI host adapter>
scsi : 1 host.
scsi : aborting command due to timeout : pid 0, scsi0, channel 0, id 0, lun 0 Test Unit Ready 00 00 00 00 00

begin:vcard
n:Lee;Gordon
tel;fax:(617) 354-9272
tel;home:(617) 576-1779
tel;work:(617) 354-9292 x108
x-mozilla-html:TRUE
url:http://www.mazunetworks.com
org:Mazu Networks
adr:;;50 Cushing Street;Cambridge;MA;02138;USA
version:2.1
email;internet:[email protected]
title:Principal Software Engineer
note;quoted-printable:125 Cambridge Park Drive=0D=0ACambridge, MA 02140=0D=0A=0D=0A x-mozilla-cpt:;0
fn:Gordon Lee
end:vcard

Eeek, these machines are now in the wild? Didn't realize this :)

Yes.

Ouch :(

Glad to see this.

I don't know if anyone ever tried a 2.2.x kernel on these boxes :) Is there a reason you _really_ need a 2.2.x kernel for this machine?

You also might try a UP 2.2.x kernel on this box to see if the problem is in the parsing of the APIC tables (as I think it is.)

thanks,

greg k-h

Forgot to mention, how many processors does this kernel show you having? I think you need to run the latest 2.4.19-ac kernel to get the second processors to show up properly.

thanks,

greg k-h

They've officially been released for a while now. (December? January? Something like that.)

Not surprising. x360s have XAPICs in them and need reasonably up-to-date APIC handling code. Try 2.4.14 or higher. That also includes Intel's "hyperthreading" code, should you want to try that. (Well, minus my forthcoming patch to that. It's in Alan's 2.4 tree.)

Recent 2.4.x also properly detects and uses APICs without going through a PIC phase. This is important as many x360 IRQ sources do not run through a PIC, and will fail any attempt to use PIC mode interrupts.

Also, I've got some patches that avoid a problem on SMP XAPIC boxes: all the external interrupts hit on CPU 0.

I use a simple static round robin routine to bind the irqs roughly evenly across all CPUs. Ingo has a different patch that rotates the irqs randomly to idle CPUs on every received interrupt. This spreads the load much more evenly, but has a bit more overhead and limits cache warmth for the handler code/data. If you're going to be beating on the x360 much, you'll probably want to install one patch or the other.

See above; 2.2.x is not expected to work.

James Cleverdon, IBM xSeries Platform (NUMA), Beaverton
[email protected]

Thanks for the rapid replies,

Yes. They are still ramping up production, and evals are scarce. I am pretty excited about it, because on paper, even without the hyperthreading, they should run pretty fast for I/O intensive workloads. My current eval project is to get some empirical performance numbers on a particular application.

I'm first! Lucky me! :-)

Longterm no, shortterm yes,
We have some modifications to the 2.2.x kernel/drivers that would cost us some time to migrate to 2.4.x. We expect to do this, but not within the short eval period during which I have the box. My immediate goal is to get it running enough to take performance measurements so we can clearly quantify the cost/benefit of migrating to this box.

As a matter of fact, we did try a UP 2.2.x kernel, and it worked. But then

we only have one CPU, and where is the fun in that ? :-) So I suppose this gives further support to the mishandled APIC table theory.

I am interested and motivated to understand the details of APIC's further. If I were to attempt to patch up a 2.2.x kernel to workaround this problem,

what documentation should I have on hand ? I have an Intel SMP 1.4 doc, although I haven't studied it in detail yet. Is this sufficient or are
there other Must Have documents that I will need ?

- GL

begin:vcard
n:Lee;Gordon
tel;fax:(617) 354-9272
tel;home:(617) 576-1779
tel;work:(617) 354-9292 x108
x-mozilla-html:TRUE
url:http://www.mazunetworks.com
org:Mazu Networks
adr:;;50 Cushing Street;Cambridge;MA;02138;USA
version:2.1
email;internet:[email protected]
title:Principal Software Engineer
note;quoted-printable:125 Cambridge Park Drive=0D=0ACambridge, MA 02140=0D=0A=0D=0A x-mozilla-cpt:;0
fn:Gordon Lee
end:vcard

It has two physical packages, and shows two processors. See below.

I suppose that you mean the second logical processors, right ? It appears to show only one logical processor for each physical package. Is there anything else you would like to see while I have an xterm open on this ?

Straight from the horses mouth, an x360 running 2.4.9:

[root@x360-gw /root]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.337
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm
bogomips : 3191.60

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.337
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm
bogomips : 3198.15

Ah, can you try the latest 2.4.19-ac tree and make sure that the rest of your processors (the "evil" twins) show up?

thanks,

greg k-h

Great, it will be nice to see some real world use of these machines to see how well the HyperThreading works out.

Yes it does, thanks for testing this.

James would be the best person for this, as he got this machine up and working on Linux properly.

thanks,

greg k-h

Yes, they show up. I tried 'cat /proc/cpuinfo' on the following:

2.4.18 shows two processors
2.4.19-pre3 shows two processors
2.4.19-pre3-ac1 shows four processors

Here is the output from 2.4.19-pre3-ac1:

[root@x360 /root]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3191.60

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3198.15

processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3198.15

processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 1
model name : Intel(R) Xeon(TM) CPU 1.60GHz
stepping : 1
cpu MHz : 1600.400
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 3198.15

Great, thanks for testing. I'd recommend using this hardware :)

greg k-h

layer bug in /proc/cpuinfo, and that kernel versions previous to 2.4.19-pre3-ac1 can actually use all of the logical processors. Is this correct ?

If so, at which 2.4.x kernel did support for hyperthreading show up?

- GL

Not exactly no.

2.4.19pre, and you want -ac patches for autodetect right now - that should all be in the main tree for 2.4.19 proper

Sorry, I meant to say, "I'd recommend using this kernel version".

No, the other processors are not recognized by Linux at all. You need that kernel version to properly use all of the logical processors.

In one of the 2.4.19-ac kernels from what I remember, sorry I don't know the exact version.

Hope this helps,

greg k-h

Interesting changelog entries..

2.4.14: hyperthreaded P4's
2.4.17:- Pentium IV Hyperthreading support (Alan Cox)
2.4.18ac:o Hyperthreading awareness for MTRR driver

Shame that .14 and .17 aren't more descriptive. I'm guessing that they provided different bits. Not sure from memory what order things happened though.

Maybe .14 was "Boot, and don't do anything silly" patches, whilst .17 was the actual "take advantage of this feature" patch.
*shrug*

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

The x360 doesnt require James' current patches (that arent in the mainstream quite yet), but what it DOES require that you wont have in the 2.2 stream is ACPI parsing.

The Linus kernel has support to boot fully HT on these boxes as of 2.4.17. There are a number of changes related to the processor in it. In order to detect HT you will need the acpitable code. In order to have the machine function well beyond that you will want changes that Ingo put into the scheduler :).

Personally I cant tell how much work you will have porting drivers to 2.4 but it might be easier than getting all the support you need back into the 2.2 kernel, and even if you do backport it I dont know that you'd get it in officially :)

Cheers,

Jack F. Vogel
IBM  Linux Solutions
[email protected]  (work)
[email protected] (home)