Skip to contentSkip to conversation

[2.4][PATCH] Xeon HT - SMT+SMP interrupt balancing

DISCUSSION Ken 3 participants 7 messages Dec 11, 2003 – Dec 12, 2003

lore
Share & tools
lore

Ken

author
#0 lore raw

I am a long time Linux user who has observed a behavior in recent 2.4.x kernels that I consider undesirable. I have read the FAQ, 'googled'/searched the LKML archives, "lurked" on the list via archives, and contacted the Maintainer directly prior to posting. Since Marcelo has announced the freezing of 2.4, I believe this is the appropriate time/forum to raise this issue. This issue may be related to other recent threads with similar subjects, but has not been addressed IMHO.

I have found that each kernel in 2.4.[19|20|21|22|23] detects my Dual (2) Xeon HT CPUs and maps hardware IRQs, but does not distribute interrupts from the IRQs across the 4 logical CPUs. All interrupts seem to be handled by CPU0 except for LOC, which is per CPU. In each case, I have corrected the behavior by applying Ingo's "linux-2.4.18-irqbalance.patch" which I found in an archive. Thank you, Ingo, for posting it -- it has been a lifesaver. ;-) I've found that it now exists by other names in various places.

For example, I have an Intel SR2300 based on an Intel SE7501-WV2 MB with
two Xeon 2.4G (HT) CPUs and E7500 (Plumas) chip set. Booting the
vanilla kernel results in this:

# cat /proc/interrupts from 2.4.23 (essentially vanilla)

CPU0 CPU1 CPU2 CPU3

  0:      23147          0          0          0    IO-APIC-edge  timer
  1:         49          0          0          0    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  8:          1          0          0          0    IO-APIC-edge  rtc
 15:          4          1          0          0    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level  usb-uhci
 19:          0          0          0          0   IO-APIC-level  usb-uhci
 24:     247647          0          0          0   IO-APIC-level  eth3
 27:          6          0          0          0   IO-APIC-level  eth2
 31:       1509          0          0          0   IO-APIC-level  eth0
 48:       2663          0          0          0   IO-APIC-level  dpti0
NMI:          0          0          0          0
LOC:      22996      22929      22995      22994
ERR:          0
MIS:          0

However, if I use Ingo's irq_balance patch, I get this:

# cat /proc/interrupts on 2.4.23 w/ irq-balancing patch

CPU0 CPU1 CPU2 CPU3

  0:      67879      87331      67921      86498    IO-APIC-edge  timer
  1:          1          0          0          1    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  8:          1          0          0          0    IO-APIC-edge  rtc
 15:          4          0          0          1    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level  usb-uhci
 19:          0          0          0          0   IO-APIC-level  usb-uhci
 24:   14524668   17990352   14614712   17901660   IO-APIC-level  eth3
 27:        341        431        341        421   IO-APIC-level  eth2
 31:      10656      13555      10724      14244   IO-APIC-level  eth0
 48:       2483       2997       2603       2874   IO-APIC-level  dpti0
NMI:          0          0          0          0
LOC:     309474     309472     309472     309405
ERR:          0
MIS:          0

It is my understanding that Ingo's patch implements a "Brownian motion" of the interrupts. While I'm unqualified to comment on the mathematical theory, I will confirm that I perceive a real improvement in performance. The CPU activity reported by 'top' definitely shows better balance across logical CPUs. The dmesg output for both scenarios is also attached.

I get similar results on all my Dell 2650 servers which have the Server Works chip set (specs at support.dell.com). Also, there doesn't seem to
be any difference in behavior by selecting/deselecting full/ht/off
ACPI support, if that is relevant. I know that ACPI is used to scan the tables regardless -- you'll note proper detection and sibling mapping in both. Furthermore, I note that if I disable HT in BIOS, I can get the two CPU balancing of interrupts seen on any non-HT SMP box, e.g., dual PIII on i440GX chip set.

But, as of 2.4.23 Ingo's patch doesn't apply cleanly, so I modified it. It is attached for your review -- NOTA BENE: it works and I'm running it in production with moderate to heavy loads, but I don't know if I've introduced a bug somewhere else. I've only tested it on the hardware mentioned above.

I have attempted user space alternatives -- irq_balance-0.06 and smp_affinity via sysctrl. The former seems to "blindly" affine an IRQ to a single logical CPU, which in my case, puts the timer and eth3 on CPU0 and it gets "overloaded" while the others are mostly idle. Using a mask with the latter results in the low number sibling handling the interrupts and the other sibling doing nothing/little. Specifically, I use a mask of 0x03 on, say, IRQ 24 in an effort to use logical CPUs 0-1 and only CPU0 shows activity via /proc/interrupts. Likewise, a mask of 0x0C on, say, IRQ 48 in an effort to use logical CPUs 2-3 and only CPU2 shows activity. The perceived performance in both cases is much less than using the patch.

Also, I noticed that Nitin Kamble had submitted a possible alternative patch to 2.5. In offline discussion with him, he preferred a discussion here prior to doing a back port.

I ask that Ingo's, Nitin's, or somebody's work be considered for inclusion in the 2.4 mainline kernel to address this issue. It doesn't seem intrusive, it doesn't break anything I can find, and it "fixes" part of an existing "feature". I'm willing to test what I can or provide what I can. I don't have the skills to be a kernel programmer, but I can follow instructions.

Thank you for your time and effort on this issue. Please reply by CC and post since I am not subscribed. I follow LKML by archive and Kernel Traffic (thanks - KT).

Regards,
Ken Beaty

# dmesg output from 2.4.23 (essentially vanilla)

Linux version 2.4.23 (root@scythe) (gcc version 3.2.2) #2 SMP Tue Dec 9 10:30:24 EST 2003 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009a800 (usable)
BIOS-e820: 000000000009a800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
BIOS-e820: 000000007fff0000 - 000000007ffff000 (ACPI data)
BIOS-e820: 000000007ffff000 - 0000000080000000 (ACPI NVS)
BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
ACPI: have wakeup address 0xc0002000
found SMP MP-table at 000ff780
hm, page 000ff000 reserved twice.
hm, page 00100000 reserved twice.
hm, page 000f1000 reserved twice.
hm, page 000f2000 reserved twice.
On node 0 totalpages: 524272
zone(0): 4096 pages.
zone(1): 225280 pages.
zone(2): 294896 pages.
ACPI: RSDP (v000 INTEL ) @ 0x000ff9b0
ACPI: RSDT (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff0000
ACPI: FADT (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff0030
ACPI: MADT (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff00b0
ACPI: OEMR (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff0140
ACPI: DSDT (v001 INTEL SWV20 0x00000100 MSFT 0x0100000d) @ 0x00000000
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
Processor #6 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x07] enabled)
Processor #7 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC_NMI (acpi_id[0x00] polarity[0x1] trigger[0x3] lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] polarity[0x1] trigger[0x3] lint[0x1])
Using ACPI for processor (LAPIC) configuration information
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: INTEL Product ID: SWV20 APIC at: 0xFEE00000
I/O APIC #8 Version 32 at 0xFEC00000.
I/O APIC #9 Version 32 at 0xFEC81000.
I/O APIC #10 Version 32 at 0xFEC81400.
Enabling APIC mode: Flat. Using 3 I/O APICs
Processors: 4
Kernel command line: auto BOOT_IMAGE=BBFKernel ro root=801 idebus=33 acpi=ht
ide_setup: idebus=33
Initializing CPU#0
Detected 2392.994 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 4771.02 BogoMIPS
Memory: 2068636k/2097088k available (2045k kernel code, 28044k reserved, 646k data, 116k init, 1179584k highmem) Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode cache hash table entries: 131072 (order: 8, 1048576 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 131072 (order: 7, 524288 bytes)
Page-cache hash table entries: 524288 (order: 9, 2097152 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000

Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.40 (20010327) Richard Gooch ([email protected])
mtrr: detected mtrr type: Intel
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
Intel machine check reporting enabled on CPU#0.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU0: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07
per-CPU timeslice cutoff: 1462.89 usecs.

enabled ExtINT on CPU#0
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Booting processor 1/1 eip 3000
Initializing CPU#1
masked ExtINT on CPU#1
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 4784.12 BogoMIPS
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
Intel machine check reporting enabled on CPU#1.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU1: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07

Booting processor 2/6 eip 3000
Initializing CPU#2
masked ExtINT on CPU#2
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 4784.12 BogoMIPS
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
Intel machine check reporting enabled on CPU#2.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU2: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07

Booting processor 3/7 eip 3000
Initializing CPU#3
masked ExtINT on CPU#3
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 4784.12 BogoMIPS
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
Intel machine check reporting enabled on CPU#3.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU3: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07

Total of 4 processors activated (19123.40 BogoMIPS).
cpu_sibling_map[0] = 1
cpu_sibling_map[1] = 0
cpu_sibling_map[2] = 3
cpu_sibling_map[3] = 2
ENABLING IO-APIC IRQs
Setting 8 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 8 ... ok.
Setting 9 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 9 ... ok.
Setting 10 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 10 ... ok.
init IO_APIC IRQs
IO-APIC (apicid-pin) 8-0, 8-18, 8-20, 8-21, 8-22, 8-23, 9-1, 9-2, 9-4, 9-5, 9-8, 9-9, 9-10, 9-11, 9-12, 9-13, 9-14, 9-15, 9-16, 9-17, 9-18, 9-19, 9-20, 9-21, 9-22, 9-23, 10-1, 10-2, 10-3, 10-4, 10-5, 10-6, 10-7, 10-8, 10-9, 10-10, 10-11, 10-12, 10-13, 10-14, 10-15, 10-16, 10-17, 10-18, 10-19, 10-20, 10-21, 10-22, 10-23 not connected.
..TIMER: vector=0x31 pin1=2 pin2=0
number of MP IRQ sources: 25.
number of IO-APIC #8 registers: 24.
number of IO-APIC #9 registers: 24.
number of IO-APIC #10 registers: 24.
testing the IO APIC.......................

IO APIC #8......
.... register #00: 08000000

.......    : physical APIC id: 08
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00178020
.......     : max redirection entries: 0017
.......     : PRQ implemented: 1
.......     : IO APIC version: 0020

.... register #02: 00000000
....... : arbitration: 00
.... register #03: 00000001
....... : Boot DT : 1
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:

 00 000 00  1    0    0   0   0    0    0    00
 01 00F 0F  0    0    0   0   0    1    1    39
 02 00F 0F  0    0    0   0   0    1    1    31
 03 00F 0F  0    0    0   0   0    1    1    41
 04 00F 0F  0    0    0   0   0    1    1    49
 05 00F 0F  0    0    0   0   0    1    1    51
 06 00F 0F  0    0    0   0   0    1    1    59
 07 00F 0F  0    0    0   0   0    1    1    61
 08 00F 0F  0    0    0   0   0    1    1    69
 09 00F 0F  0    0    0   0   0    1    1    71
 0a 00F 0F  0    0    0   0   0    1    1    79
 0b 00F 0F  0    0    0   0   0    1    1    81
 0c 00F 0F  0    0    0   0   0    1    1    89
 0d 00F 0F  0    0    0   0   0    1    1    91
 0e 00F 0F  0    0    0   0   0    1    1    99
 0f 00F 0F  0    0    0   0   0    1    1    A1
 10 00F 0F  1    1    0   1   0    1    1    A9
 11 00F 0F  1    1    0   1   0    1    1    B1
 12 000 00  1    0    0   0   0    0    0    00
 13 00F 0F  1    1    0   1   0    1    1    B9
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00

IO APIC #9......
.... register #00: 09000000

.......    : physical APIC id: 09
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00178020
.......     : max redirection entries: 0017
.......     : PRQ implemented: 1
.......     : IO APIC version: 0020

.... register #02: 09000000
....... : arbitration: 09
.... register #03: 00000001
....... : Boot DT : 1
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:

 00 00F 0F  1    1    0   1   0    1    1    C1
 01 000 00  1    0    0   0   0    0    0    00
 02 000 00  1    0    0   0   0    0    0    00
 03 00F 0F  1    1    0   1   0    1    1    C9
 04 000 00  1    0    0   0   0    0    0    00
 05 000 00  1    0    0   0   0    0    0    00
 06 00F 0F  1    1    0   1   0    1    1    D1
 07 00F 0F  1    1    0   1   0    1    1    D9
 08 000 00  1    0    0   0   0    0    0    00
 09 000 00  1    0    0   0   0    0    0    00
 0a 000 00  1    0    0   0   0    0    0    00
 0b 000 00  1    0    0   0   0    0    0    00
 0c 000 00  1    0    0   0   0    0    0    00
 0d 000 00  1    0    0   0   0    0    0    00
 0e 000 00  1    0    0   0   0    0    0    00
 0f 000 00  1    0    0   0   0    0    0    00
 10 000 00  1    0    0   0   0    0    0    00
 11 000 00  1    0    0   0   0    0    0    00
 12 000 00  1    0    0   0   0    0    0    00
 13 000 00  1    0    0   0   0    0    0    00
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00

IO APIC #10......
.... register #00: 0A000000

.......    : physical APIC id: 0A
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00178020
.......     : max redirection entries: 0017
.......     : PRQ implemented: 1
.......     : IO APIC version: 0020

.... register #02: 0A000000
....... : arbitration: 0A
.... register #03: 00000001
....... : Boot DT : 1
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:

 00 00F 0F  1    1    0   1   0    1    1    E1
 01 000 00  1    0    0   0   0    0    0    00
 02 000 00  1    0    0   0   0    0    0    00
 03 000 00  1    0    0   0   0    0    0    00
 04 000 00  1    0    0   0   0    0    0    00
 05 000 00  1    0    0   0   0    0    0    00
 06 000 00  1    0    0   0   0    0    0    00
 07 000 00  1    0    0   0   0    0    0    00
 08 000 00  1    0    0   0   0    0    0    00
 09 000 00  1    0    0   0   0    0    0    00
 0a 000 00  1    0    0   0   0    0    0    00
 0b 000 00  1    0    0   0   0    0    0    00
 0c 000 00  1    0    0   0   0    0    0    00
 0d 000 00  1    0    0   0   0    0    0    00
 0e 000 00  1    0    0   0   0    0    0    00
 0f 000 00  1    0    0   0   0    0    0    00
 10 000 00  1    0    0   0   0    0    0    00
 11 000 00  1    0    0   0   0    0    0    00
 12 000 00  1    0    0   0   0    0    0    00
 13 000 00  1    0    0   0   0    0    0    00
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00

IRQ to pin mappings:
IRQ0 -> 0:2
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ5 -> 0:5
IRQ6 -> 0:6
IRQ7 -> 0:7
IRQ8 -> 0:8
IRQ9 -> 0:9
IRQ10 -> 0:10
IRQ11 -> 0:11
IRQ12 -> 0:12
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
IRQ16 -> 0:16
IRQ17 -> 0:17
IRQ19 -> 0:19
IRQ24 -> 1:0
IRQ27 -> 1:3
IRQ30 -> 1:6
IRQ31 -> 1:7
IRQ48 -> 2:0
.................................... done.
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 2393.0461 MHz.
..... host bus clock speed is 99.7100 MHz.
cpu: 0, clocks: 997100, slice: 199420 CPU0<T0:997088,T1:797664,D:4,S:199420,C:997100>
cpu: 1, clocks: 997100, slice: 199420
cpu: 2, clocks: 997100, slice: 199420
cpu: 3, clocks: 997100, slice: 199420 CPU2<T0:997088,T1:398800,D:28,S:199420,C:997100> CPU3<T0:997088,T1:199408,D:0,S:199420,C:997100> CPU1<T0:997088,T1:598240,D:8,S:199420,C:997100>
checking TSC synchronization across CPUs: passed.
Waiting on wait_init_idle (map = 0xe)
All processors have done init_idle
ACPI: Subsystem revision 20031002
ACPI: Interpreter disabled.
PCI: PCI BIOS revision 2.10 entry at 0xfdb85, last bus=4
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: ACPI tables contain no PCI IRQ routing entries
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 00:1f.1
PCI: Unable to handle 64-bit address space for
Transparent bridge - Intel Corp. 82801BA/CA/DB/EB PCI Bridge
PCI: Using IRQ router PIIX/ICH [8086/2480] at 00:1f.0
PCI->APIC IRQ transform: (B0,I29,P0) -> 16
PCI->APIC IRQ transform: (B0,I29,P1) -> 19
PCI->APIC IRQ transform: (B0,I31,P0) -> 17
PCI->APIC IRQ transform: (B0,I31,P1) -> 17
PCI->APIC IRQ transform: (B4,I8,P0) -> 48
PCI->APIC IRQ transform: (B3,I7,P0) -> 30
PCI->APIC IRQ transform: (B3,I7,P1) -> 31
PCI->APIC IRQ transform: (B3,I8,P0) -> 24
PCI->APIC IRQ transform: (B3,I9,P0) -> 27
PCI->APIC IRQ transform: (B1,I12,P0) -> 17
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
allocated 32 pages and 32 bhs reserved for the highmem bounces
VFS: Disk quotas vdquot_6.5.1
Journalled Block Device driver loaded
pty: 512 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.10e
Floppy drive(s): fd0 is 1.44M
keyboard: Timeout - AT keyboard not present?(ed)
keyboard: Timeout - AT keyboard not present?(f4)
FDC 0 is a National Semiconductor PC87306
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 5.2.20-k1
Copyright (c) 1999-2003 Intel Corporation.
eth0: Intel(R) PRO/1000 Network Connection
eth1: Intel(R) PRO/1000 Network Connection
eth2: Intel(R) PRO/1000 Network Connection
eth3: Intel(R) PRO/1000 Network Connection
Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
ide: Assuming 33MHz system bus speed for PIO modes
ICH3: IDE controller at PCI slot 00:1f.1
PCI: Enabling device 00:1f.1 (0005 -> 0007)
ICH3: chipset revision 2
ICH3: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0x03a0-0x03a7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0x03a8-0x03af, BIOS settings: hdc:DMA, hdd:pio

hdc: MATSHITADVD-ROM SR-8177, ATAPI CD/DVD-ROM drive ide1 at 0x170-0x177,0x376 on irq 15
hdc: attached ide-cdrom driver.
hdc: ATAPI 24X DVD-ROM drive, 256kB Cache, UDMA(33) Uniform CD-ROM driver Revision: 3.12
SCSI subsystem driver Revision: 1.00
Loading Adaptec I2O RAID: Version 2.4 Build 5 Detecting Adaptec I2O RAID controllers... Adaptec I2O RAID controller 0 irq=48

     BAR0 f8891000 - size= 100000
     BAR1 f8992000 - size= 1000000

dpti: If you have a lot of devices this could take a few minutes.
dpti0: Reading the hardware resource table.
TID 008 Vendor: ADAPTEC Device: AIC-7899 Rev: 00000001
TID 009 Vendor: ADAPTEC Device: AIC-7899 Rev: 00000001
TID 517 Vendor: ESG-SHV S Device: SCA HSBP M20 Rev: 0.050
TID 522 Vendor: ADAPTEC R Device: RAID-5 Rev: 380ED
scsi0 : Vendor: Adaptec Model: 2000S FW:380E

  Vendor: ADAPTEC   Model: RAID-5            Rev: 380E
  Type:   Direct-Access                      ANSI SCSI revision: 02
  Vendor: ESG-SHV   Model: SCA HSBP M20      Rev: 0.05
  Type:   Processor                          ANSI SCSI revision: 02

Attached scsi disk sda at scsi0, channel 1, id 0, lun 0 SCSI device sda: 215494656 512-byte hdwr sectors (110333 MB) Partition check:
sda: sda1 sda2 sda3 sda4
Attached scsi generic sg1 at scsi0, channel 1, id 6, lun 0, type 3 usb.c: registered new driver usbdevfs
usb.c: registered new driver hub
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 16384 buckets, 128Kbytes
TCP: Hash tables configured (established 262144 bind 65536)
ip_tables: (C) 2000-2002 Netfilter core team
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0. Ebtables v2.0 registered
NET4: Ethernet Bridge 008 for NET4.0
Bridge firewalling registered
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 116k freed
Adding Swap: 2097136k swap-space (priority -1)
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,3), internal journal
EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,4), internal journal
EXT3-fs: mounted filesystem with ordered data mode. scsi1 : SCSI host adapter emulation for IDE ATAPI devices
e1000: eth0 NIC Link is Up 100 Mbps Full Duplex usb-uhci.c: $Revision: 1.275 $ time 14:18:13 Dec 2 2003 usb-uhci.c: High bandwidth mode enabled
PCI: Setting latency timer of device 00:1d.0 to 64 usb-uhci.c: USB UHCI at I/O 0x3020, IRQ 16
usb-uhci.c: Detected 2 ports
usb.c: new USB bus registered, assigned bus number 1 hub.c: USB hub found
hub.c: 2 ports detected
PCI: Setting latency timer of device 00:1d.1 to 64 usb-uhci.c: USB UHCI at I/O 0x3000, IRQ 19
usb-uhci.c: Detected 2 ports
usb.c: new USB bus registered, assigned bus number 2 hub.c: USB hub found
hub.c: 2 ports detected
usb-uhci.c: v1.275:USB Universal Host Controller Interface driver i2c-core.o: i2c core module version 2.8.0 (20030714) i2c-i801 version 2.8.0 (20030714)
i2c-proc.o version 2.8.0 (20030714)
adm1021.o version 2.8.0 (20030714)

# dmesg output from 2.4.23 w/ irq-balancing patch

Linux version 2.4.23 (root@scythe) (gcc version 3.2.2) #1 SMP Tue Dec 9 11:13:31 EST 2003 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009a800 (usable)
BIOS-e820: 000000000009a800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
BIOS-e820: 000000007fff0000 - 000000007ffff000 (ACPI data)
BIOS-e820: 000000007ffff000 - 0000000080000000 (ACPI NVS)
BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000ff780
hm, page 000ff000 reserved twice.
hm, page 00100000 reserved twice.
hm, page 000f1000 reserved twice.
hm, page 000f2000 reserved twice.
On node 0 totalpages: 524272
zone(0): 4096 pages.
zone(1): 225280 pages.
zone(2): 294896 pages.
ACPI: RSDP (v000 INTEL ) @ 0x000ff9b0
ACPI: RSDT (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff0000
ACPI: FADT (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff0030
ACPI: MADT (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff00b0
ACPI: OEMR (v001 INTEL SWV20 0x00000001 MSFT 0x01000000) @ 0x7fff0140
ACPI: DSDT (v001 INTEL SWV20 0x00000100 MSFT 0x0100000d) @ 0x00000000
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
Processor #6 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x07] enabled)
Processor #7 Pentium 4(tm) XEON(tm) APIC version 20
ACPI: LAPIC_NMI (acpi_id[0x00] polarity[0x1] trigger[0x3] lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] polarity[0x1] trigger[0x3] lint[0x1])
Using ACPI for processor (LAPIC) configuration information
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: INTEL Product ID: SWV20 APIC at: 0xFEE00000
I/O APIC #8 Version 32 at 0xFEC00000.
I/O APIC #9 Version 32 at 0xFEC81000.
I/O APIC #10 Version 32 at 0xFEC81400.
Enabling APIC mode: Flat. Using 3 I/O APICs
Processors: 4
Kernel command line: auto BOOT_IMAGE=BBFKernel ro root=801 idebus=33 acpi=ht
ide_setup: idebus=33
Initializing CPU#0
Detected 2392.984 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 4771.02 BogoMIPS
Memory: 2068756k/2097088k available (1935k kernel code, 27924k reserved, 618k data, 112k init, 1179584k highmem) Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode cache hash table entries: 131072 (order: 8, 1048576 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 131072 (order: 7, 524288 bytes)
Page-cache hash table entries: 524288 (order: 9, 2097152 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000

Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.40 (20010327) Richard Gooch ([email protected])
mtrr: detected mtrr type: Intel
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
Intel machine check reporting enabled on CPU#0.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU0: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07
per-CPU timeslice cutoff: 1462.89 usecs.

enabled ExtINT on CPU#0
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Booting processor 1/1 eip 2000
Initializing CPU#1
masked ExtINT on CPU#1
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 4784.12 BogoMIPS
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
Intel machine check reporting enabled on CPU#1.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU1: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07

Booting processor 2/6 eip 2000
Initializing CPU#2
masked ExtINT on CPU#2
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 4784.12 BogoMIPS
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
Intel machine check reporting enabled on CPU#2.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU2: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07

Booting processor 3/7 eip 2000
Initializing CPU#3
masked ExtINT on CPU#3
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 4784.12 BogoMIPS
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
Intel machine check reporting enabled on CPU#3.

CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU3: Intel(R) Xeon(TM) CPU 2.40GHz stepping 07

Total of 4 processors activated (19123.40 BogoMIPS).
cpu_sibling_map[0] = 1
cpu_sibling_map[1] = 0
cpu_sibling_map[2] = 3
cpu_sibling_map[3] = 2
ENABLING IO-APIC IRQs
Setting 8 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 8 ... ok.
Setting 9 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 9 ... ok.
Setting 10 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 10 ... ok.
init IO_APIC IRQs
IO-APIC (apicid-pin) 8-0, 8-18, 8-20, 8-21, 8-22, 8-23, 9-1, 9-2, 9-4, 9-5, 9-8, 9-9, 9-10, 9-11, 9-12, 9-13, 9-14, 9-15, 9-16, 9-17, 9-18, 9-19, 9-20, 9-21, 9-22, 9-23, 10-1, 10-2, 10-3, 10-4, 10-5, 10-6, 10-7, 10-8, 10-9, 10-10, 10-11, 10-12, 10-13, 10-14, 10-15, 10-16, 10-17, 10-18, 10-19, 10-20, 10-21, 10-22, 10-23 not connected.
..TIMER: vector=0x31 pin1=2 pin2=0
number of MP IRQ sources: 25.
number of IO-APIC #8 registers: 24.
number of IO-APIC #9 registers: 24.
number of IO-APIC #10 registers: 24.
testing the IO APIC.......................

IO APIC #8......
.... register #00: 08000000

.......    : physical APIC id: 08
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00178020
.......     : max redirection entries: 0017
.......     : PRQ implemented: 1
.......     : IO APIC version: 0020

.... register #02: 00000000
....... : arbitration: 00
.... register #03: 00000001
....... : Boot DT : 1
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:

 00 000 00  1    0    0   0   0    0    0    00
 01 00F 0F  0    0    0   0   0    1    1    39
 02 001 01  0    0    0   0   0    1    1    31
 03 00F 0F  0    0    0   0   0    1    1    41
 04 00F 0F  0    0    0   0   0    1    1    49
 05 00F 0F  0    0    0   0   0    1    1    51
 06 00F 0F  0    0    0   0   0    1    1    59
 07 00F 0F  0    0    0   0   0    1    1    61
 08 00F 0F  0    0    0   0   0    1    1    69
 09 00F 0F  0    0    0   0   0    1    1    71
 0a 00F 0F  0    0    0   0   0    1    1    79
 0b 00F 0F  0    0    0   0   0    1    1    81
 0c 00F 0F  0    0    0   0   0    1    1    89
 0d 00F 0F  0    0    0   0   0    1    1    91
 0e 00F 0F  0    0    0   0   0    1    1    99
 0f 00F 0F  0    0    0   0   0    1    1    A1
 10 00F 0F  1    1    0   1   0    1    1    A9
 11 00F 0F  1    1    0   1   0    1    1    B1
 12 000 00  1    0    0   0   0    0    0    00
 13 00F 0F  1    1    0   1   0    1    1    B9
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00

IO APIC #9......
.... register #00: 09000000

.......    : physical APIC id: 09
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00178020
.......     : max redirection entries: 0017
.......     : PRQ implemented: 1
.......     : IO APIC version: 0020

.... register #02: 09000000
....... : arbitration: 09
.... register #03: 00000001
....... : Boot DT : 1
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:

 00 00F 0F  1    1    0   1   0    1    1    C1
 01 000 00  1    0    0   0   0    0    0    00
 02 000 00  1    0    0   0   0    0    0    00
 03 00F 0F  1    1    0   1   0    1    1    C9
 04 000 00  1    0    0   0   0    0    0    00
 05 000 00  1    0    0   0   0    0    0    00
 06 00F 0F  1    1    0   1   0    1    1    D1
 07 00F 0F  1    1    0   1   0    1    1    D9
 08 000 00  1    0    0   0   0    0    0    00
 09 000 00  1    0    0   0   0    0    0    00
 0a 000 00  1    0    0   0   0    0    0    00
 0b 000 00  1    0    0   0   0    0    0    00
 0c 000 00  1    0    0   0   0    0    0    00
 0d 000 00  1    0    0   0   0    0    0    00
 0e 000 00  1    0    0   0   0    0    0    00
 0f 000 00  1    0    0   0   0    0    0    00
 10 000 00  1    0    0   0   0    0    0    00
 11 000 00  1    0    0   0   0    0    0    00
 12 000 00  1    0    0   0   0    0    0    00
 13 000 00  1    0    0   0   0    0    0    00
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00

IO APIC #10......
.... register #00: 0A000000

.......    : physical APIC id: 0A
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00178020
.......     : max redirection entries: 0017
.......     : PRQ implemented: 1
.......     : IO APIC version: 0020

.... register #02: 0A000000
....... : arbitration: 0A
.... register #03: 00000001
....... : Boot DT : 1
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:

 00 00F 0F  1    1    0   1   0    1    1    E1
 01 000 00  1    0    0   0   0    0    0    00
 02 000 00  1    0    0   0   0    0    0    00
 03 000 00  1    0    0   0   0    0    0    00
 04 000 00  1    0    0   0   0    0    0    00
 05 000 00  1    0    0   0   0    0    0    00
 06 000 00  1    0    0   0   0    0    0    00
 07 000 00  1    0    0   0   0    0    0    00
 08 000 00  1    0    0   0   0    0    0    00
 09 000 00  1    0    0   0   0    0    0    00
 0a 000 00  1    0    0   0   0    0    0    00
 0b 000 00  1    0    0   0   0    0    0    00
 0c 000 00  1    0    0   0   0    0    0    00
 0d 000 00  1    0    0   0   0    0    0    00
 0e 000 00  1    0    0   0   0    0    0    00
 0f 000 00  1    0    0   0   0    0    0    00
 10 000 00  1    0    0   0   0    0    0    00
 11 000 00  1    0    0   0   0    0    0    00
 12 000 00  1    0    0   0   0    0    0    00
 13 000 00  1    0    0   0   0    0    0    00
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00

IRQ to pin mappings:
IRQ0 -> 0:2
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ5 -> 0:5
IRQ6 -> 0:6
IRQ7 -> 0:7
IRQ8 -> 0:8
IRQ9 -> 0:9
IRQ10 -> 0:10
IRQ11 -> 0:11
IRQ12 -> 0:12
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
IRQ16 -> 0:16
IRQ17 -> 0:17
IRQ19 -> 0:19
IRQ24 -> 1:0
IRQ27 -> 1:3
IRQ30 -> 1:6
IRQ31 -> 1:7
IRQ48 -> 2:0
.................................... done.
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 2392.9287 MHz.
..... host bus clock speed is 99.7052 MHz.
cpu: 0, clocks: 997052, slice: 199410 CPU0<T0:997040,T1:797616,D:14,S:199410,C:997052>
cpu: 1, clocks: 997052, slice: 199410
cpu: 2, clocks: 997052, slice: 199410
cpu: 3, clocks: 997052, slice: 199410 CPU1<T0:997040,T1:598208,D:12,S:199410,C:997052> CPU2<T0:997040,T1:398800,D:10,S:199410,C:997052> CPU3<T0:997040,T1:199392,D:8,S:199410,C:997052>
checking TSC synchronization across CPUs: passed.
Waiting on wait_init_idle (map = 0xe)
All processors have done init_idle
PCI: PCI BIOS revision 2.10 entry at 0xfdb85, last bus=4
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 00:1f.1
PCI: Unable to handle 64-bit address space for
Transparent bridge - Intel Corp. 82801BA/CA/DB/EB PCI Bridge
PCI: Using IRQ router PIIX/ICH [8086/2480] at 00:1f.0
PCI->APIC IRQ transform: (B0,I29,P0) -> 16
PCI->APIC IRQ transform: (B0,I29,P1) -> 19
PCI->APIC IRQ transform: (B0,I31,P0) -> 17
PCI->APIC IRQ transform: (B0,I31,P1) -> 17
PCI->APIC IRQ transform: (B4,I8,P0) -> 48
PCI->APIC IRQ transform: (B3,I7,P0) -> 30
PCI->APIC IRQ transform: (B3,I7,P1) -> 31
PCI->APIC IRQ transform: (B3,I8,P0) -> 24
PCI->APIC IRQ transform: (B3,I9,P0) -> 27
PCI->APIC IRQ transform: (B1,I12,P0) -> 17
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
allocated 32 pages and 32 bhs reserved for the highmem bounces
VFS: Disk quotas vdquot_6.5.1
Journalled Block Device driver loaded
pty: 512 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.10e
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 5.2.20-k1
Copyright (c) 1999-2003 Intel Corporation.
eth0: Intel(R) PRO/1000 Network Connection
eth1: Intel(R) PRO/1000 Network Connection
eth2: Intel(R) PRO/1000 Network Connection
eth3: Intel(R) PRO/1000 Network Connection
Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
ide: Assuming 33MHz system bus speed for PIO modes
ICH3: IDE controller at PCI slot 00:1f.1
PCI: Enabling device 00:1f.1 (0005 -> 0007)
ICH3: chipset revision 2
ICH3: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0x03a0-0x03a7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0x03a8-0x03af, BIOS settings: hdc:DMA, hdd:pio

hdc: MATSHITADVD-ROM SR-8177, ATAPI CD/DVD-ROM drive ide1 at 0x170-0x177,0x376 on irq 15
hdc: attached ide-cdrom driver.
hdc: ATAPI 24X DVD-ROM drive, 256kB Cache, UDMA(33) Uniform CD-ROM driver Revision: 3.12
SCSI subsystem driver Revision: 1.00
Loading Adaptec I2O RAID: Version 2.4 Build 5 Detecting Adaptec I2O RAID controllers... Adaptec I2O RAID controller 0 irq=48

     BAR0 f8891000 - size= 100000
     BAR1 f8992000 - size= 1000000

dpti: If you have a lot of devices this could take a few minutes.
dpti0: Reading the hardware resource table.
TID 008 Vendor: ADAPTEC Device: AIC-7899 Rev: 00000001
TID 009 Vendor: ADAPTEC Device: AIC-7899 Rev: 00000001
TID 517 Vendor: ESG-SHV S Device: SCA HSBP M20 Rev: 0.050
TID 522 Vendor: ADAPTEC R Device: RAID-5 Rev: 380ED
scsi0 : Vendor: Adaptec Model: 2000S FW:380E

  Vendor: ADAPTEC   Model: RAID-5            Rev: 380E
  Type:   Direct-Access                      ANSI SCSI revision: 02
  Vendor: ESG-SHV   Model: SCA HSBP M20      Rev: 0.05
  Type:   Processor                          ANSI SCSI revision: 02

Attached scsi disk sda at scsi0, channel 1, id 0, lun 0 SCSI device sda: 215494656 512-byte hdwr sectors (110333 MB) Partition check:
sda: sda1 sda2 sda3 sda4
Attached scsi generic sg1 at scsi0, channel 1, id 6, lun 0, type 3 usb.c: registered new driver usbdevfs
usb.c: registered new driver hub
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 16384 buckets, 128Kbytes
TCP: Hash tables configured (established 262144 bind 65536)
ip_tables: (C) 2000-2002 Netfilter core team
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0. Ebtables v2.0 registered
NET4: Ethernet Bridge 008 for NET4.0
Bridge firewalling registered
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 112k freed
Adding Swap: 2097136k swap-space (priority -1)
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,3), internal journal
EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,4), internal journal
EXT3-fs: mounted filesystem with ordered data mode. scsi1 : SCSI host adapter emulation for IDE ATAPI devices
e1000: eth0 NIC Link is Up 100 Mbps Full Duplex usb-uhci.c: $Revision: 1.275 $ time 14:18:13 Dec 2 2003 usb-uhci.c: High bandwidth mode enabled
PCI: Setting latency timer of device 00:1d.0 to 64 usb-uhci.c: USB UHCI at I/O 0x3020, IRQ 16
usb-uhci.c: Detected 2 ports
usb.c: new USB bus registered, assigned bus number 1 hub.c: USB hub found
hub.c: 2 ports detected
PCI: Setting latency timer of device 00:1d.1 to 64 usb-uhci.c: USB UHCI at I/O 0x3000, IRQ 19
usb-uhci.c: Detected 2 ports
usb.c: new USB bus registered, assigned bus number 2 hub.c: USB hub found
hub.c: 2 ports detected
usb-uhci.c: v1.275:USB Universal Host Controller Interface driver i2c-core.o: i2c core module version 2.8.0 (20030714) i2c-i801 version 2.8.0 (20030714)
i2c-proc.o version 2.8.0 (20030714)
adm1021.o version 2.8.0 (20030714)

#1 lore raw
Quoted 4 lines · Ken

I have attempted user space alternatives -- irq_balance-0.06 and smp_affinity via sysctrl. The former seems to "blindly" affine an IRQ to a single logical CPU, which in my case, puts the timer and eth3 on CPU0 and it gets "overloaded" while the others are mostly idle.

1) This got fixed in version 0.07
2) You are talking a whopping 100 irq's per second, which is like about no interrupts... I doubt you can find a scenario where 100 irq's per second matter.... ;)

author
#2 lore raw
Quoted 1 line · Arjan van de Ven

1) This got fixed in version 0.07

Thank you, Arjan -- let's see how it does. Hmm, I see you added a
ROTATE policy for timer. Ok, I edited your classify.c to add my storage controller -- just to ensure everything is properly classified.

I get this result:

CPU0 CPU1 CPU2 CPU3

  0:       7502      48699      48048      49595    IO-APIC-edge  timer
  1:          2          0          0          0    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  8:          1          0          0          0    IO-APIC-edge  rtc
 15:          4          0          0          1    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level  usb-uhci
 19:          0          0          0          0   IO-APIC-level  usb-uhci
 24:   21061496          0          0          0   IO-APIC-level  eth3
 27:         21          0        732          0   IO-APIC-level  eth2
 31:        374      11363          0          0   IO-APIC-level  eth0
 48:       2716          0       1328        566   IO-APIC-level  dpti0
NMI:          0          0          0          0
LOC:     153702     153699     153699     153635
ERR:          0
MIS:          0

So, the timer now migrates, but IRQ 24 isn't even shared by the
sibling, let alone the other "pair". Well, all these interfaces are GigE -- 2 fiber, 2 copper -- so, I re-classified "eth" as GIGE and didn't see any improvement; however, if I follow your policy correctly, shouldn't the sibling get some usage in either case?

In any case, I consider the above behavior undesirable over time. CPU0
is handling most of the system while CPU1 contributes little. For example, this snapshot:
top - 23:11:16 up 12 min, 1 user, load average: 0.26, 0.35, 0.19

Tasks:  49 total,   3 running,  46 sleeping,   0 stopped,   0 zombie
Cpu0 :   0.8% user,  34.8% system,   0.0% nice,  64.4% idle
Cpu1 :   0.0% user,   0.0% system,   0.0% nice, 100.0% idle
Cpu2 :  15.5% user,   4.1% system,   0.0% nice,  80.5% idle
Cpu3 :   6.7% user,   2.4% system,   0.0% nice,  90.9% idle
Mem:   2068752k total,   477636k used,  1591116k free,     5156k buffers
Swap:  2097136k total,        0k used,  2097136k free,    36720k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND
    3 root      19  19     0    0    0 R  0.0  0.0   0:00.21  0 
ksoftirqd_CPU0
 1206 nobody    15   0  359m 359m 1724 R 29.3 17.8   3:33.23  3 ntop
 1221 root      12   0   992  992  816 R  0.4  0.0   0:00.94  2 top
    1 root       9   0   228  228  196 S  0.0  0.0   0:09.40  3 init

I appreciate your help, Arjan. Am I missing something?

Quoted 3 lines

2) You are talking a whopping 100 irq's per second, which is like about no interrupts... I doubt you can find a scenario where 100 irq's per second matter.... ;)

If I understand you correctly, I agree 100/s is negligible. But, I
think I see more than that on this lightly loaded machine:
'vmstat 1'
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
1 0 0 1569188 5708 36728 0 0 11 3 3464 4614 6
11 83 0
0 0 0 1569184 5708 36728 0 0 0 0 14560 23163 6
10 84 0
0 0 0 1569184 5708 36728 0 0 0 0 14284 23734 8
11 81 0
1 0 0 1569184 5708 36728 0 0 0 0 14292 23617 5
10 85 0
0 0 0 1569160 5732 36728 0 0 0 64 14426 23771 6
12 82 0

Again, am I missing something?

Thanks again,
Ken Beaty

Signature · 4 lines
[email protected]                                   /     /  /|   /  /  / | /
GPG KeyID: 09209FA2 (on KeyServer)            /     /  / |  /  /  /  |/
http://members.nova.org/~ken/index.html      /     /  /  | /  /  /  /|
Running Slackware 8.0 & SMP kernel 2.4.23.../___ _/_ /   |/  /__/  / |
#3 lore raw
Quoted 8 lines · Ken
Quoted 1 line · Arjan van de Ven

1) This got fixed in version 0.07

So, the timer now migrates, but IRQ 24 isn't even shared by the
sibling, let alone the other "pair". Well, all these interfaces are GigE -- 2 fiber, 2 copper -- so, I re-classified "eth" as GIGE and didn't see any improvement; however, if I follow your policy correctly, shouldn't the sibling get some usage in either case?

No. Ethernet (and esp gigabit) has significant performance benefits from being (and staying) on one cpu. What matters are lots of cacheline bounces, and (even worse) packet(fragment) reordering that can happen if it moves around.

Quoted 10 lines

In any case, I consider the above behavior undesirable over time.
CPU0 is handling most of the system while CPU1 contributes little. For
example, this snapshot:
top - 23:11:16 up 12 min, 1 user, load average: 0.26, 0.35, 0.19

Tasks:  49 total,   3 running,  46 sleeping,   0 stopped,   0 zombie
Cpu0 :   0.8% user,  34.8% system,   0.0% nice,  64.4% idle
Cpu1 :   0.0% user,   0.0% system,   0.0% nice, 100.0% idle
Cpu2 :  15.5% user,   4.1% system,   0.0% nice,  80.5% idle
Cpu3 :   6.7% user,   2.4% system,   0.0% nice,  90.9% idle

your machine is very very idle. Also remember that with modern machines, even if a machine is 0% you probably can keep adding load to it for a while, because then you are increasing "batching" which is very very cache efficient operation, with the current differences between L1/L2 cache speed and main memory speed it's not unlikely you can do 2x or 3x more work than you do the moment idle hits 0%.

Quoted 3 lines

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 0  0      0 1569184   5708  36728    0    0     0     0 14560 23163  6 10 84  0
 0  0      0 1569184   5708  36728    0    0     0     0 14284 23734  8 11 81  0

this is a sign you may want to enable IRQ mitigation in your network driver to increase batching (unless your application is very latency bound)

Greetings,
Arjan van de Ven

#4 lore raw
In reply to Ken
Quoted 52 lines

I am a long time Linux user who has observed a behavior in recent 2.4.x kernels that I consider undesirable. I have read the FAQ, 'googled'/searched the LKML archives, "lurked" on the list via archives, and contacted the Maintainer directly prior to posting. Since Marcelo has announced the freezing of 2.4, I believe this is the appropriate time/forum to raise this issue. This issue may be related to other recent threads with similar subjects, but has not been addressed IMHO.

I have found that each kernel in 2.4.[19|20|21|22|23] detects my Dual (2) Xeon HT CPUs and maps hardware IRQs, but does not distribute interrupts from the IRQs across the 4 logical CPUs. All interrupts seem to be handled by CPU0 except for LOC, which is per CPU. In each case, I have corrected the behavior by applying Ingo's "linux-2.4.18-irqbalance.patch" which I found in an archive. Thank you, Ingo, for posting it -- it has been a lifesaver. ;-) I've found that it now exists by other names in various places.

For example, I have an Intel SR2300 based on an Intel SE7501-WV2 MB with two Xeon 2.4G (HT) CPUs and E7500 (Plumas) chip set. Booting the vanilla kernel results in this:

# cat /proc/interrupts from 2.4.23 (essentially vanilla)

CPU0 CPU1 CPU2 CPU3

   0:      23147          0          0          0    IO-APIC-edge  timer
   1:         49          0          0          0    IO-APIC-edge  keyboard
   2:          0          0          0          0          XT-PIC  cascade
   8:          1          0          0          0    IO-APIC-edge  rtc
  15:          4          1          0          0    IO-APIC-edge  ide1
  16:          0          0          0          0   IO-APIC-level  usb-uhci
  19:          0          0          0          0   IO-APIC-level  usb-uhci
  24:     247647          0          0          0   IO-APIC-level  eth3
  27:          6          0          0          0   IO-APIC-level  eth2
  31:       1509          0          0          0   IO-APIC-level  eth0
  48:       2663          0          0          0   IO-APIC-level  dpti0
NMI:          0          0          0          0
LOC:      22996      22929      22995      22994
ERR:          0
MIS:          0

However, if I use Ingo's irq_balance patch, I get this:

# cat /proc/interrupts on 2.4.23 w/ irq-balancing patch

CPU0 CPU1 CPU2 CPU3

   0:      67879      87331      67921      86498    IO-APIC-edge  timer
   1:          1          0          0          1    IO-APIC-edge  keyboard
   2:          0          0          0          0          XT-PIC  cascade
   8:          1          0          0          0    IO-APIC-edge  rtc
  15:          4          0          0          1    IO-APIC-edge  ide1
  16:          0          0          0          0   IO-APIC-level  usb-uhci
  19:          0          0          0          0   IO-APIC-level  usb-uhci
  24:   14524668   17990352   14614712   17901660   IO-APIC-level  eth3
  27:        341        431        341        421   IO-APIC-level  eth2
  31:      10656      13555      10724      14244   IO-APIC-level  eth0
  48:       2483       2997       2603       2874   IO-APIC-level  dpti0
NMI:          0          0          0          0
LOC:     309474     309472     309472     309405
ERR:          0
MIS:          0

It is my understanding that Ingo's patch implements a "Brownian motion" of the interrupts. While I'm unqualified to comment on the mathematical theory, I will confirm that I perceive a real improvement in performance. The CPU activity reported by 'top' definitely shows better balance across logical CPUs. The dmesg output for both scenarios is also attached.

Quoted 7 lines

I get similar results on all my Dell 2650 servers which have the Server Works chip set (specs at support.dell.com). Also, there doesn't seem to be any difference in behavior by selecting/deselecting full/ht/off ACPI support, if that is relevant. I know that ACPI is used to scan the tables regardless -- you'll note proper detection and sibling mapping in both. Furthermore, I note that if I disable HT in BIOS, I can get the two CPU balancing of interrupts seen on any non-HT SMP box, e.g., dual PIII on i440GX chip set.

But, as of 2.4.23 Ingo's patch doesn't apply cleanly, so I modified it. It is attached for your review -- NOTA BENE: it works and I'm running it in production with moderate to heavy loads, but I don't know if I've introduced a bug somewhere else. I've only tested it on the hardware mentioned above.

I have attempted user space alternatives -- irq_balance-0.06 and smp_affinity via sysctrl. The former seems to "blindly" affine an IRQ to a single logical CPU, which in my case, puts the timer and eth3 on CPU0 and it gets "overloaded" while the others are mostly idle. Using a mask with the latter results in the low number sibling handling the interrupts and the other sibling doing nothing/little. Specifically, I use a mask of 0x03 on, say, IRQ 24 in an effort to use logical CPUs 0-1 and only CPU0 shows activity via /proc/interrupts. Likewise, a mask of 0x0C on, say, IRQ 48 in an effort to use logical CPUs 2-3 and only CPU2 shows activity. The perceived performance in both cases is much less than using the patch.

Also, I noticed that Nitin Kamble had submitted a possible alternative patch to 2.5. In offline discussion with him, he preferred a discussion here prior to doing a back port.

Nitin's patch is in 2.6 - does that work OK for you?

M.

author
#5 lore raw
Quoted 3 lines · Martin J. Bligh

Nitin's patch is in 2.6 - does that work OK for you?

Thanks, Martin -- I wish I could test it. As of -test11, the
Adaptec/DPT I2O driver still doesn't build and I need that on this box.
I think I understand the error messages, but fixing them is beyond me. It's obviously a known issue, so I don't think I can provide any new
info on it.

Too bad for me -- I'm really looking forward to trying it and stuff
like the NAPI for the e1000!

Regards,
Ken Beaty

#6 lore raw
Quoted 6 lines
Quoted 2 lines

Nitin's patch is in 2.6 - does that work OK for you?

Thanks, Martin -- I wish I could test it. As of -test11, the Adaptec/DPT I2O driver still doesn't build and I need that on this box. I think I understand the error messages, but fixing them is beyond me. It's obviously a known issue, so I don't think I can provide any new info on it.

Too bad for me -- I'm really looking forward to trying it and stuff like the NAPI for the e1000!

The code has been in for ages - so you should be able to use test9 or test10 just fine, if that works better from a driver point of view.

M.