P4 SMP load balancing
I posted this a while back in linux-smp (which seems like a dead list?)
I have several P4 Xeon SMP systems (Supermicro P4DCE, Intel i860 chipset)
ovendev:~# cat /proc/interrupts
CPU0 CPU1
0: 6348212 0 IO-APIC-edge timer
1: 2 0 IO-APIC-edge keyboard
2: 0 0 XT-PIC cascade
8: 1 0 IO-APIC-edge rtc
9: 0 0 IO-APIC-edge acpi
16: 92620 0 IO-APIC-level eth0
18: 5085 0 IO-APIC-level aic7xxx, aic7xxx
NMI: 0 0
LOC: 6348388 6348427
ERR: 0
MIS: 0 How much of a problem is this really? The program's I am
running on these systems (I have 9 of them) seem do ok right now. Currently the jobs running on them are heavily CPU bound and don't do any I/O, but this is going to change when I link them up over a private network so they can work together on some distributable jobs). I am running 2.4.10 on most of them, and 2.4.10-ac10 on my developer system in the farm. The only difference this newer kernel seems to have made from older ones is that there is only one 'warning unexpected IO-APIC' message in my startup instead of two.
Snippet from dmesg:
CPU1: Intel(R) Xeon(TM) CPU 1700MHz stepping 0a
Total of 2 processors activated (6723.99 BogoMIPS).
ENABLING IO-APIC IRQs
...changing IO-APIC physical APIC ID to 2 ... ok.
init IO_APIC IRQs
IO-APIC (apicid-pin) 2-0, 2-5, 2-10, 2-11, 2-12, 2-17, 2-20, 2-21, 2-22
not connected.
..TIMER: vector=0x31 pin1=2 pin2=0
number of MP IRQ sources: 18.
number of IO-APIC #2 registers: 24.
testing the IO APIC.......................
IO APIC #2......
.... register #00: 02000000
....... : physical APIC id: 02
.... register #01: 00178020
....... : max redirection entries: 0017 ....... : PRQ implemented: 1 ....... : IO APIC version: 0020
WARNING: unexpected IO-APIC, please mail
to [email protected]
.... register #02: 00000000
....... : arbitration: 00
.... IRQ redirection table:
<snip>
- Sean
Quoted 5 lines
ovendev:~# cat /proc/interrupts
CPU0 CPU1
0: 6348212 0 IO-APIC-edge timer
1: 2 0 IO-APIC-edge keyboard
2: 0 0 XT-PIC cascadeQuoted 8 lines
8: 1 0 IO-APIC-edge rtc 9: 0 0 IO-APIC-edge acpi 16: 92620 0 IO-APIC-level eth0 18: 5085 0 IO-APIC-level aic7xxx, aic7xxx NMI: 0 0 LOC: 6348388 6348427 ERR: 0 MIS: 0
I don't think this should happen. In the event of both procs having equal priority (linux never changes them, so they always do), we should fall back to the arbitration priority of the lapic. Whether you have 1 or 2 I/O apics working shouldn't make a difference.
The arb priority of the local apic should change whenever a message is sent (see the Intel docs on developer.intel.com), so we effectively get round robin. For instance, a 4 way looks like this:
CPU0 CPU1 CPU2 CPU3
0: 1608606 1595657 2168078 1575546 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge keyboard 2: 0 0 0 0 XT-PIC cascade 4: 76 52 62 48 IO-APIC-edge serial 23: 7983 8263 8286 8306 IO-APIC-level qlogicisp 39: 0 0 0 0 IO-APIC-level eth1 40: 6247 6216 6894 6325 IO-APIC-level eth0 NMI: 0 0 0 0 LOC: 6947876 6947859 6947873 6947874 ERR: 0 MIS: 0
Which isn't perfectly balanced, but it looks a damned sight better than yours does ;-) Do you have something in the log that looks like this?
Oct 11 15:35:04 elm3b76 kernel: IO APIC #13...... Oct 11 15:35:04 elm3b76 kernel: .... register #00: 0D000000 Oct 11 15:35:04 elm3b76 kernel: ....... : physical APIC id: 0D Oct 11 15:35:04 elm3b76 kernel: .... register #01: 00170011 Oct 11 15:35:04 elm3b76 kernel: ....... : max redirection entries: 0017 Oct 11 15:35:04 elm3b76 kernel: ....... : IO APIC version: 0011 Oct 11 15:35:04 elm3b76 kernel: .... register #02: 00000000 Oct 11 15:35:04 elm3b76 kernel: ....... : arbitration: 00 Oct 11 15:35:04 elm3b76 kernel: .... IRQ redirection table: Oct 11 15:35:04 elm3b76 kernel: NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect: Oct 11 15:35:04 elm3b76 kernel: 00 000 00 1 0 0 0 0 0 0 00 Oct 11 15:35:05 elm3b76 kernel: 01 00F 0F 0 0 0 0 0 0 1 39 Oct 11 15:35:05 elm3b76 kernel: 02 00F 0F 0 0 0 0 0 0 1 31 Oct 11 15:35:05 elm3b76 kernel: 03 00F 0F 0 0 0 0 0 0 1 41 Oct 11 15:35:05 elm3b76 kernel: 04 00F 0F 0 0 0 0 0 0 1 49 Oct 11 15:35:05 elm3b76 kernel: 05 00F 0F 0 0 0 0 0 0 1 51 Oct 11 15:35:05 elm3b76 kernel: 06 00F 0F 0 0 0 0 0 0 1 59 Oct 11 15:35:05 elm3b76 kernel: 07 00F 0F 1 1 0 1 0 0 1 61 Oct 11 15:35:05 elm3b76 kernel: 08 00F 0F 1 1 0 0 0 0 1 69 Oct 11 15:35:05 elm3b76 kernel: 09 00F 0F 0 0 0 0 0 0 1 71 Oct 11 15:35:05 elm3b76 kernel: 0a 00F 0F 0 0 0 0 0 0 1 79 Oct 11 15:35:05 elm3b76 kernel: 0b 00F 0F 1 1 0 1 0 0 1 81 Oct 11 15:35:05 elm3b76 kernel: 0c 00F 0F 0 0 0 0 0 0 1 89 Oct 11 15:35:05 elm3b76 kernel: 0d 00F 0F 1 1 0 1 0 0 1 91 Oct 11 15:35:05 elm3b76 kernel: 0e 00F 0F 0 0 0 0 0 0 1 99 Oct 11 15:35:05 elm3b76 kernel: 0f 00F 0F 1 1 0 1 0 0 1 A1 Oct 11 15:35:05 elm3b76 kernel: 10 00F 0F 1 1 0 1 0 0 1 A9 Oct 11 15:35:05 elm3b76 kernel: 11 00F 0F 1 1 0 1 0 0 1 B1 Oct 11 15:35:05 elm3b76 kernel: 12 00F 0F 1 1 0 1 0 0 1 B9 Oct 11 15:35:05 elm3b76 kernel: 13 00F 0F 1 1 0 1 0 0 1 C1 Oct 11 15:35:05 elm3b76 kernel: 14 00F 0F 1 1 0 1 0 0 1 C9 Oct 11 15:35:05 elm3b76 kernel: 15 00F 0F 1 1 0 1 0 0 1 D1 Oct 11 15:35:05 elm3b76 kernel: 16 00F 0F 1 1 0 1 0 0 1 D9 Oct 11 15:35:05 elm3b76 kernel: 17 00F 0F 1 1 0 1 0 0 1 E1
You might have to tweak syslog.conf to log the debug messages. And possibly increase LOG_BUF_LEN in kernel/printk.c to something sensible (63356?)
M.
Quoted 8 lines
From: Martin J. Bligh [mailto:[email protected]]
Sent: Friday, October 12, 2001 1:00 PM
To: Sean Cavanaugh
Cc: [email protected]
Subject: Re: P4 SMP load balancingWhich isn't perfectly balanced, but it looks a damned sight better
than yours does ;-) Do you have something in the log that looks like this?
Quoted 6 lines
Oct 11 15:35:04 elm3b76 kernel: IO APIC #13...... Oct 11 15:35:04 elm3b76 kernel: .... register #00: 0D000000 Oct 11 15:35:04 elm3b76 kernel: ....... : physical APIC id: 0D Oct 11 15:35:04 elm3b76 kernel: .... register #01: 00170011 Oct 11 15:35:04 elm3b76 kernel: ....... : max redirection entries:
0017
Quoted 5 lines
Oct 11 15:35:04 elm3b76 kernel: ....... : IO APIC version: 0011 Oct 11 15:35:04 elm3b76 kernel: .... register #02: 00000000 Oct 11 15:35:04 elm3b76 kernel: ....... : arbitration: 00 Oct 11 15:35:04 elm3b76 kernel: .... IRQ redirection table: Oct 11 15:35:04 elm3b76 kernel: NR Log Phy Mask Trig IRR Pol Stat
Dest Deli Vect:
Quoted 1 line
Oct 11 15:35:04 elm3b76 kernel: 00 000 00 1 0 0 0 0 0
0 00
Quoted 1 line
Oct 11 15:35:05 elm3b76 kernel: 01 00F 0F 0 0 0 0 0 0
1 39
<snip>
Relevent: APIC info from dmesg (with some lead-in/lead-out):
Calibrating delay loop... 3368.55 BogoMIPS
CPU: Before vendor init, caps: 3febfbff 00000000 00000000, vendor = 0
CPU: L1 I cache: 12K, L1 D cache: 8K
CPU: L2 cache: 256K
CPU: After vendor init, caps: 3febfbff 00000000 00000000 00000000 Intel machine check reporting enabled on CPU#1.
CPU: After generic, caps: 3febfbff 00000000 00000000 00000000 CPU: Common caps: 3febfbff 00000000 00000000 00000000 CPU1: Intel(R) Xeon(TM) CPU 1700MHz stepping 0a
Total of 2 processors activated (6723.99 BogoMIPS).
ENABLING IO-APIC IRQs
...changing IO-APIC physical APIC ID to 2 ... ok.
init IO_APIC IRQs
IO-APIC (apicid-pin) 2-0, 2-5, 2-10, 2-11, 2-12, 2-17, 2-20, 2-21, 2-22
not connected.
..TIMER: vector=0x31 pin1=2 pin2=0
number of MP IRQ sources: 18.
number of IO-APIC #2 registers: 24.
testing the IO APIC.......................
IO APIC #2......
.... register #00: 02000000
....... : physical APIC id: 02
.... register #01: 00178020
....... : max redirection entries: 0017 ....... : PRQ implemented: 1 ....... : IO APIC version: 0020
WARNING: unexpected IO-APIC, please mail
to [email protected]
.... register #02: 00000000
....... : arbitration: 00
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
00 000 00 1 0 0 0 0 0 0 00 01 003 03 0 0 0 0 0 1 1 39 02 003 03 0 0 0 0 0 1 1 31 03 003 03 0 0 0 0 0 1 1 41 04 003 03 0 0 0 0 0 1 1 49 05 000 00 1 0 0 0 0 0 0 00 06 003 03 0 0 0 0 0 1 1 51 07 003 03 0 0 0 0 0 1 1 59 08 003 03 0 0 0 0 0 1 1 61 09 003 03 0 0 0 0 0 1 1 69 0a 000 00 1 0 0 0 0 0 0 00 0b 000 00 1 0 0 0 0 0 0 00 0c 000 00 1 0 0 0 0 0 0 00 0d 003 03 0 0 0 0 0 1 1 71 0e 003 03 0 0 0 0 0 1 1 79 0f 003 03 0 0 0 0 0 1 1 81 10 003 03 1 1 0 1 0 1 1 89 11 000 00 1 0 0 0 0 0 0 00 12 003 03 1 1 0 1 0 1 1 91 13 003 03 1 1 0 1 0 1 1 99 14 000 00 1 0 0 0 0 0 0 00 15 000 00 1 0 0 0 0 0 0 00 16 000 00 1 0 0 0 0 0 0 00 17 003 03 1 1 0 1 0 1 1 A1
IRQ to pin mappings:
IRQ0 -> 0:2
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ6 -> 0:6
IRQ7 -> 0:7
IRQ8 -> 0:8
IRQ9 -> 0:9
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
IRQ16 -> 0:16
IRQ18 -> 0:18
IRQ19 -> 0:19
IRQ23 -> 0:23 .................................... done. Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 1685.2574 MHz.
..... host bus clock speed is 99.1326 MHz.
cpu: 0, clocks: 991326, slice: 330442 CPU0<T0:991312,T1:660864,D:6,S:330442,C:991326>
cpu: 1, clocks: 991326, slice: 330442 CPU1<T0:991312,T1:330416,D:12,S:330442,C:991326> checking TSC synchronization across CPUs: passed. Waiting on wait_init_idle (map = 0x2)
All processors have done init_idle
PCI: PCI BIOS revision 2.10 entry at 0xfb3e0, last bus=4
PCI: Using configuration type 1
PCI: Probing PCI hardware
Unknown bridge resource 0: assuming transparent Unknown bridge resource 1: assuming transparent Unknown bridge resource 2: assuming transparent Unknown bridge resource 2: assuming transparent Unknown bridge resource 2: assuming transparent Unknown bridge resource 2: assuming transparent
PCI: Using IRQ router PIIX [8086/2440] at 00:1f.0 PCI->APIC IRQ transform: (B0,I31,P3) -> 19 PCI->APIC IRQ transform: (B0,I31,P1) -> 19 PCI->APIC IRQ transform: (B0,I31,P2) -> 23 PCI->APIC IRQ transform: (B3,I4,P0) -> 18 PCI->APIC IRQ transform: (B3,I4,P1) -> 18 PCI->APIC IRQ transform: (B4,I4,P0) -> 16 PCI->APIC IRQ transform: (B4,I7,P0) -> 16
isapnp: Scanning for PnP cards...
- Sean