NMI problems with Dell SMP Xeons
Summary: sending an IPI as NMI will reboot a Dell SMP Xeon box. This
looks like a problem with setting up the I/O APICs.
Debugging, crash dump etc. need to send an IPI to other processors to bring them to a stopped state. If the other cpus are spinning disabled then the IPI needs to be sent as an NMI to override the disabled state.
Sending an IPI as NMI works fine on almost every bit of i386 and x86_64 SMP hardware that I have tried. But not on some Dell Xeons, specifically not on an SC1425 server. On SC1425, when any IPI is sent as NMI then it gets an instant reset, AFAICT it never gets to the NMI routine. Changing CONFIG_ACPI or booting with a different maxcpus will change the behaviour.
Ugly test patch against 2.6.17-rc4 follows, plus .config and some boot logs. The problem exists in both 32 and 64 bit mode on the SC1425, but this is an i386 build so it can be used on all my test boxes.
modprobe test_nmi sends four different IPIs, with a short pause between them :-
(1) IPI 2, not marked as NMI. This does _not_ call into the do_nmi() routine.
People have been telling me (hi, Andi:) that sending interrupt 2 as an IPI automatically sends it as an NMI. Every piece of hardware I have tested (including the Dell SC1425) shows that this is not true; instead IPI 2 routes as a normal interrupt, which agrees with Intel's documentation[1].
Something strange, this IPI just disappears. Not only is it not being delivered as an NMI, it does not seem to be delivered at all. This is true on all the hardware that I have tested. Some of the hardware (but not all of it) reports APIC errors when sending IPI 2.
APIC error on CPU0: 00(20)
APIC error on CPU1: 00(40)
(2) IPI 20, not marked as NMI. A test IPI which is used to check that other IPIs are getting through, and they are. IOW, the test module is working fine, as long as it does not try to send IPI 2.
(3) IPI 2, marked as NMI (APIC_DM_NMI in smp.c). If the machine stays up then this IPI does call into the do_nmi() routine. The NMI handler gets dazed and confused, but that is expected.
(4) IPI 20, marked as NMI. This also calls into the do_nmi() routine, as expected.
Typical log on an Intel motherboard, with 2 Xeon cpus.
test_nmi_init: 1 cpu 0
kaos_send_IPI_allbutself: ipi 2
APIC error on CPU0: 00(20)
APIC error on CPU1: 00(40)
test_nmi_init: 2 cpu 0
kaos_send_IPI_allbutself: ipi 20
test_nmi_init: 3 cpu 0
do_ipi_20: cpu 1
kaos_send_IPI_allbutself: ipi 2 __send_IPI_shortcut: cpu 0 cfg 0xc0c00
test_nmi_init: 4 cpu 0
do_nmi: cpu 1
Uhhuh. NMI received for unknown reason 00 on CPU 1. Dazed and confused, but trying to continue Do you have a strange power saving mode enabled?
kaos_send_IPI_allbutself: ipi 20 __send_IPI_shortcut: cpu 0 cfg 0xc0c00
do_nmi: cpu 1
Uhhuh. NMI received for unknown reason 00 on CPU 1. Dazed and confused, but trying to continue Do you have a strange power saving mode enabled?
Typical log on Dell SC1425 system with 4 cpus, CONFIG_ACPI=y, booted with maxcpus=2.
test_nmi_init: 1 cpu 0
kaos_send_IPI_allbutself: ipi 2
APIC error on CPU0: 00(20)
APIC error on CPU1: 00(40)
test_nmi_init: 2 cpu 0
kaos_send_IPI_allbutself: ipi 20
test_nmi_init: 3 cpu 0
do_ipi_20: cpu 1
kaos_send_IPI_allbutself: ipi 2
__send_IPI_shortcut: cpu 0 cfg 0xc0c00
=== Instant reset here ===
Swapping the order of the NMI IPIs in test_nmi() (send 20 before 2) makes no difference on the SC1425. If the problem exists then the SC1425 always gets an instant reset on the first IPI that is sent as NMI.
Typical log on SC1425 with 4 cpus, CONFIG_ACPI=y, booted without maxcpus= or with maxcpus=4.
test_nmi_init: 1 cpu 1
kaos_send_IPI_allbutself: ipi 2
APIC error on CPU1: 00(20)
APIC error on CPU3: 00(40)
APIC error on CPU2: 00(40)
APIC error on CPU0: 00(40)
test_nmi_init: 2 cpu 1
kaos_send_IPI_allbutself: ipi 20
test_nmi_init: 3 cpu 1
do_ipi_20: cpu 3
do_ipi_20: cpu 2
do_ipi_20: cpu 0
kaos_send_IPI_allbutself: ipi 2 __send_IPI_shortcut: cpu 1 cfg 0xc0c00
test_nmi_init: 4 cpu 1
Uhhuh. NMI received for unknown reason 00 on CPU 2. Uhhuh. NMI received for unknown reason 00 on CPU 3.
do_nmi: cpu 0
Dazed and confused, but trying to continue Do you have a strange power saving mode enabled? Uhhuh. NMI received for unknown reason 30 on CPU 0. Dazed and confused, but trying to continue Dazed and confused, but trying to continue Do you have a strange power saving mode enabled? Do you have a strange power saving mode enabled?
kaos_send_IPI_allbutself: ipi 20 __send_IPI_shortcut: cpu 1 cfg 0xc0c00
do_nmi: cpu 0
Uhhuh. NMI received for unknown reason 00 on CPU 2. Uhhuh. NMI received for unknown reason 20 on CPU 0. Dazed and confused, but trying to continue Dazed and confused, but trying to continue
do_nmi: cpu 3
Do you have a strange power saving mode enabled? Do you have a strange power saving mode enabled? Uhhuh. NMI received for unknown reason 00 on CPU 3. Dazed and confused, but trying to continue Do you have a strange power saving mode enabled?
The system keeps going.
The .config below has CONFIG_ACPI=y. If the SC1425 is run with CONFIG_ACPI=y and with no maxcpus= on the boot line then it runs fine. Compile with CONFIG_ACPI=n or with CONFIG_ACPI=y and maxcpus= some value less than the actual number of cpus and the SC1425 dies on the first NMI IPI.
It is interesting that with CONFIG_ACPI=y, SC1425 has 4 cpus. With CONFIG_ACPI=n it only has 2.
My best guess is that this reset problem is being caused by an incorrect or incomplete setup of the I/O APICs when ACPI is off or not all the cpus are processed by the kernel at boot time.
The SC1425 is running BIOS 1.10 A03, which is the latest that I could find on http://support.dell.com.
####################################################################
Stripped .config.
X86_32=y
SEMAPHORE_SLEEPERS=y
X86=y
MMU=y
GENERIC_ISA_DMA=y
GENERIC_IOMAP=y
GENERIC_HWEIGHT=y
ARCH_MAY_HAVE_PC_FDC=y
DMI=y
EXPERIMENTAL=y
LOCK_KERNEL=y
INIT_ENV_ARG_LIMIT=32
LOCALVERSION="-kaos"
SWAP=y
SYSVIPC=y
SYSCTL=y
RELAY=y
INITRAMFS_SOURCE=""
UID16=y
VM86=y
CC_OPTIMIZE_FOR_SIZE=y
KALLSYMS=y
KALLSYMS_ALL=y
HOTPLUG=y
PRINTK=y
BUG=y
ELF_CORE=y
BASE_FULL=y
FUTEX=y
EPOLL=y
SHMEM=y
SLAB=y
BASE_SMALL=0
MODULES=y
MODULE_UNLOAD=y
KMOD=y
STOP_MACHINE=y
LBD=y
IOSCHED_NOOP=y
DEFAULT_NOOP=y
DEFAULT_IOSCHED="noop"
SMP=y
X86_PC=y
M686=y
X86_CMPXCHG=y
X86_XADD=y
X86_L1_CACHE_SHIFT=5
RWSEM_XCHGADD_ALGORITHM=y
GENERIC_CALIBRATE_DELAY=y
X86_PPRO_FENCE=y
X86_WP_WORKS_OK=y
X86_INVLPG=y
X86_BSWAP=y
X86_POPAD_OK=y
X86_CMPXCHG64=y
X86_GOOD_APIC=y
X86_USE_PPRO_CHECKSUM=y
X86_TSC=y
HPET_TIMER=y
HPET_EMULATE_RTC=y
NR_CPUS=8
SCHED_SMT=y
SCHED_MC=y
PREEMPT=y
PREEMPT_BKL=y
X86_LOCAL_APIC=y
X86_IO_APIC=y
X86_MCE=y
X86_MCE_NONFATAL=y
X86_MCE_P4THERMAL=y
MICROCODE=m
X86_MSR=m
X86_CPUID=m
HIGHMEM4G=y
PAGE_OFFSET=0xC0000000
HIGHMEM=y
ARCH_FLATMEM_ENABLE=y
ARCH_SPARSEMEM_ENABLE=y
ARCH_SELECT_MEMORY_MODEL=y
SELECT_MEMORY_MODEL=y
FLATMEM_MANUAL=y
FLATMEM=y
FLAT_NODE_MEM_MAP=y
SPARSEMEM_STATIC=y
SPLIT_PTLOCK_CPUS=4
MTRR=y
IRQBALANCE=y
REGPARM=y
SECCOMP=y
HZ_250=y
HZ=250
PHYSICAL_START=0x100000
PM=y
ACPI=y
ACPI_AC=y
ACPI_BATTERY=y
ACPI_BUTTON=y
ACPI_VIDEO=y
ACPI_FAN=y
ACPI_PROCESSOR=y
ACPI_THERMAL=y
ACPI_BLACKLIST_YEAR=0
ACPI_EC=y
ACPI_POWER=y
ACPI_SYSTEM=y
X86_PM_TIMER=y
PCI=y
PCI_GOANY=y
PCI_BIOS=y
PCI_DIRECT=y
PCI_MMCONFIG=y
ISA_DMA_API=y
BINFMT_ELF=y
BINFMT_MISC=m
NET=y
PACKET=m
PACKET_MMAP=y
UNIX=y
INET=y
IP_FIB_HASH=y
SYN_COOKIES=y
INET_DIAG=y
INET_TCP_DIAG=y
TCP_CONG_BIC=y
NETFILTER=y
NETFILTER_NETLINK=m
NETFILTER_XTABLES=m
NETFILTER_XT_TARGET_CLASSIFY=m
NETFILTER_XT_TARGET_MARK=m
NETFILTER_XT_TARGET_NFQUEUE=m
NETFILTER_XT_TARGET_NOTRACK=m
NETFILTER_XT_MATCH_COMMENT=m
NETFILTER_XT_MATCH_CONNTRACK=m
NETFILTER_XT_MATCH_DCCP=m
NETFILTER_XT_MATCH_ESP=m
NETFILTER_XT_MATCH_HELPER=m
NETFILTER_XT_MATCH_LENGTH=m
NETFILTER_XT_MATCH_LIMIT=m
NETFILTER_XT_MATCH_MAC=m
NETFILTER_XT_MATCH_MARK=m
NETFILTER_XT_MATCH_MULTIPORT=m
NETFILTER_XT_MATCH_PKTTYPE=m
NETFILTER_XT_MATCH_REALM=m
NETFILTER_XT_MATCH_SCTP=m
NETFILTER_XT_MATCH_STATE=m
NETFILTER_XT_MATCH_STRING=m
NETFILTER_XT_MATCH_TCPMSS=m
IP_NF_CONNTRACK=m
IP_NF_CONNTRACK_NETLINK=m
IP_NF_FTP=m
IP_NF_IRC=m
IP_NF_H323=m
IP_NF_QUEUE=m
IP_NF_IPTABLES=m
IP_NF_MATCH_IPRANGE=m
IP_NF_MATCH_TOS=m
IP_NF_MATCH_RECENT=m
IP_NF_MATCH_ECN=m
IP_NF_MATCH_DSCP=m
IP_NF_MATCH_AH=m
IP_NF_MATCH_TTL=m
IP_NF_MATCH_OWNER=m
IP_NF_MATCH_ADDRTYPE=m
IP_NF_MATCH_HASHLIMIT=m
IP_NF_FILTER=m
IP_NF_TARGET_REJECT=m
IP_NF_TARGET_LOG=m
IP_NF_TARGET_ULOG=m
IP_NF_TARGET_TCPMSS=m
IP_NF_NAT=m
IP_NF_NAT_NEEDED=y
IP_NF_TARGET_MASQUERADE=m
IP_NF_TARGET_REDIRECT=m
IP_NF_TARGET_NETMAP=m
IP_NF_TARGET_SAME=m
IP_NF_NAT_SNMP_BASIC=m
IP_NF_NAT_IRC=m
IP_NF_NAT_FTP=m
IP_NF_NAT_H323=m
IP_NF_MANGLE=m
IP_NF_TARGET_TOS=m
IP_NF_TARGET_ECN=m
IP_NF_TARGET_DSCP=m
IP_NF_TARGET_TTL=m
IP_NF_RAW=m
IP_NF_ARPTABLES=m
IP_NF_ARPFILTER=m
IP_NF_ARP_MANGLE=m
NET_CLS_ROUTE=y
STANDALONE=y
PREVENT_FIRMWARE_BUILD=y
PARPORT=m
PARPORT_PC=m
BLK_DEV_FD=y
BLK_DEV_LOOP=m
BLK_DEV_RAM=y
BLK_DEV_RAM_COUNT=16
BLK_DEV_RAM_SIZE=4096
IDE=y
BLK_DEV_IDE=y
BLK_DEV_IDEDISK=y
IDEDISK_MULTI_MODE=y
BLK_DEV_IDECD=m
BLK_DEV_IDESCSI=m
IDE_GENERIC=y
BLK_DEV_IDEPCI=y
IDEPCI_SHARE_IRQ=y
BLK_DEV_GENERIC=y
BLK_DEV_IDEDMA_PCI=y
IDEDMA_PCI_AUTO=y
BLK_DEV_PIIX=y
BLK_DEV_IDEDMA=y
IDEDMA_AUTO=y
SCSI=y
SCSI_PROC_FS=y
BLK_DEV_SD=y
BLK_DEV_SR=m
BLK_DEV_SR_VENDOR=y
CHR_DEV_SG=m
SCSI_CONSTANTS=y
SCSI_LOGGING=y
SCSI_SPI_ATTRS=y
SCSI_SATA=y
SCSI_ATA_PIIX=y
SCSI_SATA_INTEL_COMBINED=y
SCSI_SYM53C8XX_2=y
SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
SCSI_SYM53C8XX_DEFAULT_TAGS=16
SCSI_SYM53C8XX_MAX_TAGS=64
SCSI_SYM53C8XX_MMIO=y
MD=y
BLK_DEV_MD=m
MD_LINEAR=m
MD_RAID0=m
MD_RAID1=m
MD_RAID10=m
MD_RAID5=m
MD_RAID6=m
MD_MULTIPATH=m
BLK_DEV_DM=m
DM_CRYPT=m
DM_SNAPSHOT=m
DM_MIRROR=m
DM_ZERO=m
NETDEVICES=y
DUMMY=m
TUN=m
NET_ETHERNET=y
MII=y
NET_PCI=y
E100=y
E1000=m
TIGON3=m
PPP=m
PPP_ASYNC=m
PPP_DEFLATE=m
PPP_BSDCOMP=m
INPUT=y
INPUT_MOUSEDEV=y
INPUT_MOUSEDEV_PSAUX=y
INPUT_MOUSEDEV_SCREEN_X=1024
INPUT_MOUSEDEV_SCREEN_Y=768
INPUT_KEYBOARD=y
KEYBOARD_ATKBD=y
INPUT_MOUSE=y
MOUSE_PS2=m
MOUSE_SERIAL=m
SERIO=y
SERIO_I8042=y
SERIO_PCIPS2=m
SERIO_LIBPS2=y
VT=y
VT_CONSOLE=y
HW_CONSOLE=y
SERIAL_8250=y
SERIAL_8250_CONSOLE=y
SERIAL_8250_PCI=y
SERIAL_8250_NR_UARTS=4
SERIAL_8250_RUNTIME_UARTS=4
SERIAL_CORE=y
SERIAL_CORE_CONSOLE=y
UNIX98_PTYS=y
LEGACY_PTYS=y
LEGACY_PTY_COUNT=256
PRINTER=m
WATCHDOG=y
SOFT_WATCHDOG=m
RTC=y
HWMON=y
VGA_CONSOLE=y
DUMMY_CONSOLE=y
USB_ARCH_HAS_HCD=y
USB_ARCH_HAS_OHCI=y
USB_ARCH_HAS_EHCI=y
EDAC=y
EDAC_DEBUG=y
EDAC_MM_EDAC=y
EDAC_E7XXX=m
EDAC_E752X=y
EDAC_I82875P=m
EDAC_I82860=m
EDAC_POLL=y
EXT2_FS=y
FS_POSIX_ACL=y
XFS_FS=y
XFS_EXPORT=y
XFS_QUOTA=y
ROMFS_FS=m
INOTIFY=y
QUOTACTL=y
DNOTIFY=y
AUTOFS_FS=m
AUTOFS4_FS=m
FUSE_FS=m
ISO9660_FS=m
JOLIET=y
ZISOFS=y
ZISOFS_FS=m
UDF_FS=m
UDF_NLS=y
FAT_FS=m
MSDOS_FS=m
VFAT_FS=m
FAT_DEFAULT_CODEPAGE=437
FAT_DEFAULT_IOCHARSET="iso8859-1"
NTFS_FS=m
PROC_FS=y
PROC_KCORE=y
SYSFS=y
TMPFS=y
RAMFS=y
EFS_FS=m
HPFS_FS=m
UFS_FS=m
NFS_FS=m
NFS_V3=y
NFS_V4=y
NFSD=m
NFSD_V3=y
NFSD_V4=y
NFSD_TCP=y
LOCKD=m
LOCKD_V4=y
EXPORTFS=y
NFS_COMMON=y
SUNRPC=m
SUNRPC_GSS=m
RPCSEC_GSS_KRB5=m
SMB_FS=m
MSDOS_PARTITION=y
NLS=y
NLS_DEFAULT="iso8859-1"
NLS_CODEPAGE_437=m
NLS_CODEPAGE_850=y
NLS_ISO8859_1=y
NLS_UTF8=m
MAGIC_SYSRQ=y
DEBUG_KERNEL=y
LOG_BUF_SHIFT=16
DEBUG_PREEMPT=y
DEBUG_MUTEXES=y
DEBUG_BUGVERBOSE=y
DEBUG_FS=y
FRAME_POINTER=y
FORCED_INLINING=y
EARLY_PRINTK=y
STACK_BACKTRACE_COLS=2
DEBUG_RODATA=y
4KSTACKS=y
X86_FIND_SMP_CONFIG=y
X86_MPPARSE=y
DOUBLEFAULT=y
CRYPTO=y
CRYPTO_MD5=y
CRYPTO_DES=m
CRC_CCITT=m
CRC32=m
ZLIB_INFLATE=m
ZLIB_DEFLATE=m
TEXTSEARCH=y
TEXTSEARCH_KMP=m
TEXTSEARCH_BM=m
TEXTSEARCH_FSM=m
GENERIC_HARDIRQS=y
GENERIC_IRQ_PROBE=y
GENERIC_PENDING_IRQ=y
X86_SMP=y
X86_HT=y
X86_BIOS_REBOOT=y
X86_TRAMPOLINE=y
KTIME_SCALAR=y
####################################################################
Ugly test_nmi patch, against 2.6.17-rc4.
arch/i386/kernel/smp.c +19 -0
@@ -122,6 +122,8 @@ static inline int __prepare_ICR2 (unsignreturn SET_APIC_DEST_FIELD (mask); } +extern int kaos_expect_nmi; +void __send_IPI_shortcut (unsigned int shortcut,int vector) {/*
@@ -142,6 +144,11 @@ void __send_IPI_shortcut(unsigned int sh* No need to touch the target chip field */ cfg =__prepare_ICR (shortcut, vector); +if (vector == kaos_expect_nmi) { + cfg = (cfg&~APIC_VECTOR_MASK )|APIC_DM_NMI ; + kaos_expect_nmi =-1 ; +printk ("%s: cpu %d cfg 0x%x\n" , __FUNCTION__,smp_processor_id (), cfg); + }/* * Send the IPI. The write to APIC_ICR fires this off.
@@ -220,6 +227,11 @@ void send_IPI_mask_sequence(cpumask_t ma* program the ICR */ cfg =__prepare_ICR (0 , vector); +if (vector == kaos_expect_nmi) { + cfg = (cfg&~APIC_VECTOR_MASK )|APIC_DM_NMI ; + kaos_expect_nmi =-1 ; +printk ("%s: cpu %d cfg 0x%lx\n" , __FUNCTION__,smp_processor_id (), cfg); + }/* * Send the IPI. The write to APIC_ICR fires this off.
@@ -562,6 +574,13 @@ int smp_call_function (void (*func) (voi }EXPORT_SYMBOL (smp_call_function); +void kaos_send_IPI_allbutself (int ipi) +{ +printk ("%s: ipi %d\n" , __FUNCTION__, ipi); +send_IPI_allbutself (ipi); +} +EXPORT_SYMBOL (kaos_send_IPI_allbutself); +static void stop_this_cpu (void * dummy) {/*
arch/i386/kernel/traps.c +14 -0
@@ -87,6 +87,7 @@ asmlinkage void general_protection(void) asmlinkage void page_fault(void); asmlinkage void coprocessor_error(void); asmlinkage void simd_coprocessor_error(void); +asmlinkage void ipi_20(void); asmlinkage void alignment_check(void); asmlinkage void spurious_interrupt_bug(void); asmlinkage void machine_check(void);
@@ -717,10 +718,17 @@ static int dummy_nmi_callback(struct pt_ }static nmi_callback_t nmi_callback = dummy_nmi_callback; + +int kaos_expect_nmi; +EXPORT_SYMBOL (kaos_expect_nmi); fastcall void do_nmi(struct pt_regs * regs, long error_code) {int cpu; +if (kaos_expect_nmi) { + kaos_expect_nmi =0 ; +printk ("%s: cpu %d\n" , __FUNCTION__,smp_processor_id ()); + }nmi_enter ();
@@ -734,6 +742,11 @@ fastcall void do_nmi(struct pt_regs * renmi_exit (); } +fastcall void do_ipi_20(struct pt_regs * regs, long error_code) +{ +printk ("%s: cpu %d\n" , __FUNCTION__,raw_smp_processor_id ()); +} +void set_nmi_callback (nmi_callback_t callback) {vmalloc_sync_all ();
@@ -1171,6 +1184,7 @@ void __init trap_init(void)set_trap_gate (18 ,&machine_check);#endif set_trap_gate (19 ,&simd_coprocessor_error); +set_trap_gate (20 ,&ipi_20);if (cpu_has_fxsr) {/*
kernel/Makefile +2 -0
@@ -39,6 +39,8 @@ obj-$(CONFIG_SECCOMP) += seccomp.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o obj-$(CONFIG_RELAY) += relay.o +obj-m += test_nmi.o + ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y) # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is # needed for x86 only. Why this used to be enabled for all architectures is beyond
kernel/test_nmi.c new file +40 -0
@@ -0,0 +1,40 @@ +#include <linux/config.h> +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/delay.h> + +MODULE_LICENSE ("GPL" ); + +extern void kaos_send_IPI_allbutself (int ); +extern int kaos_expect_nmi; + +static int __init +test_nmi_init (void ) +{ +int cpu =get_cpu (); + kaos_expect_nmi =0 ; +printk ("%s: 1 cpu %d\n" , __FUNCTION__, cpu); +mdelay (1000 ); +kaos_send_IPI_allbutself (2 ); +printk ("%s: 2 cpu %d\n" , __FUNCTION__, cpu); +mdelay (1000 ); +kaos_send_IPI_allbutself (20 ); +printk ("%s: 3 cpu %d\n" , __FUNCTION__, cpu); +mdelay (1000 ); + kaos_expect_nmi =2 ; +kaos_send_IPI_allbutself (2 ); +printk ("%s: 4 cpu %d\n" , __FUNCTION__, cpu); +mdelay (1000 ); + kaos_expect_nmi =20 ; +kaos_send_IPI_allbutself (20 ); +put_cpu (); +return 0 ; +} + +static void __exit +test_nmi_exit (void ) +{ +} + +module_init (test_nmi_init) +module_exit (test_nmi_exit)
arch/i386/kernel/entry.S +5 -0
@@ -476,6 +476,11 @@ ENTRY(simd_coprocessor_error) pushl $do_simd_coprocessor_error jmp error_code +ENTRY(ipi_20) + pushl $0 + pushl $do_ipi_20 + jmp error_code + ENTRY(device_not_available) pushl $-1 # mark this as an int SAVE_ALL
####################################################################
Boot log of SC1425 with CONFIG_ACPI=y, maxcpus=4. This test works.
Linux version 2.6.17-rc4-kaos (kaos@linuxbuild) (gcc version 3.3.3 (SuSE Linux)) #53 SMP PREEMPT Mon May 22 17:29:17 EST 2006 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
BIOS-e820: 0000000000100000 - 000000007ffc0000 (usable)
BIOS-e820: 000000007ffc0000 - 000000007ffcfc00 (ACPI data)
BIOS-e820: 000000007ffcfc00 - 000000007ffff000 (reserved)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000fe710
DMI 2.3 present.
ACPI: PM-Timer IO Port: 0x808
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
Processor #6 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
Processor #1 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
Processor #7 15:4 APIC version 20
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[32])
IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 32-55
ACPI: IOAPIC (id[0x0a] address[0xfec80800] gsi_base[64])
IOAPIC[2]: apic_id 10, version 32, address 0xfec80800, GSI 64-87
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Enabling APIC mode: Flat. Using 3 I/O APICs
ACPI: HPET id: 0xffffffff base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 80000000 (gap: 7ffff000:60001000)
Built 1 zonelists
Kernel command line: root=/dev/sdb2 ro selinux=0 console=tty console=ttyS0,9600 maxcpus=4
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c03d3000 soft=c03cb000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2075288k/2096896k available (1828k kernel code, 20524k reserved, 813k data, 180k init, 1179392k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok.
Using HPET for base-timer
Using HPET for gettimeofday
Detected 3000.240 MHz processor.
Using hpet for high-res timesource
Calibrating delay using timer specific routine.. 6005.07 BogoMIPS (lpj=12010151)
Mount-cache hash table entries: 512
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
CPU0: Thermal monitoring enabled
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 12k freed
CPU0: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Booting processor 1/1 eip 2000
CPU 1 irqstacks, hard=c03d4000 soft=c03cc000
Initializing CPU#1
Calibrating delay using timer specific routine.. 6000.31 BogoMIPS (lpj=12000634)
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Booting processor 2/6 eip 2000
CPU 2 irqstacks, hard=c03d5000 soft=c03cd000
Initializing CPU#2
Calibrating delay using timer specific routine.. 6000.41 BogoMIPS (lpj=12000826)
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 3
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#2.
CPU2: Intel P4/Xeon Extended MCE MSRs (24) available
CPU2: Thermal monitoring enabled
CPU2: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Booting processor 3/7 eip 2000
CPU 3 irqstacks, hard=c03d6000 soft=c03ce000
Initializing CPU#3
Calibrating delay using timer specific routine.. 6000.33 BogoMIPS (lpj=12000673)
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 3
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#3.
CPU3: Intel P4/Xeon Extended MCE MSRs (24) available
CPU3: Thermal monitoring enabled
CPU3: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Total of 4 processors activated (24006.14 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 4 CPUs: passed.
Brought up 4 CPUs
migration_cost=4000,4000
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using MMCONFIG
Setting up standard PCI resources
ACPI: Subsystem revision 20060127
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 *11 12)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 7 10 11 12)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *10 11 12)
ACPI: PCI Interrupt Link [LNKE] (IRQs *3 4 5 6 7 10 11 12)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 *6 7 10 11 12)
SCSI subsystem initialized
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
PCI: Bridge: 0000:01:00.0
IO window: e000-efff
MEM window: fe900000-feafffff
PREFETCH window: disabled.
PCI: Bridge: 0000:01:00.2
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
IO window: e000-efff
MEM window: fe700000-feafffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: d000-dfff
MEM window: fe500000-fe6fffff
PREFETCH window: f0000000-f7ffffff
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 16
NET: Registered protocol family 2
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 3145728 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
SGI XFS with large block numbers, no debug enabled
SGI XFS Quota Management subsystem
Initializing Cryptographic API
io scheduler noop registered (default)
Intel E7520/7320/7525 detected.<6>ACPI: Power Button (FF) [PWRF]
ACPI: Video Device [EVGA] (multi-head: no rom: yes post: no)
Real Time Clock Driver v1.12ac
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
floppy0: no floppy controllers found
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
e100: Intel(R) PRO/100 Network Driver, 3.5.10-k2-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH5: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 17
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
hda: TEAC CD-ROM CD-224E, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14ata_piix 0000:00:1f.2: MAP [ P0 -- P1 -- ]
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 17
ata1: SATA max UDMA/133 cmd 0xCCB8 ctl 0xCCB2 bmdma 0xCC80 irq 17
ata2: SATA max UDMA/133 cmd 0xCCA0 ctl 0xCC9A bmdma 0xCC88 irq 17
ata1: dev 0 ATA-6, max UDMA/100, 156250000 sectors: LBA
ata1: dev 0 configured for UDMA/100
scsi0 : ata_piix
ata2: dev 0 ATA-7, max UDMA/133, 586072368 sectors: LBA48
ata2: dev 0 configured for UDMA/133
scsi1 : ata_piix
Vendor: ATA Model: WDC WD800JD-75JN Rev: 06.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: ATA Model: ST3300622AS Rev: 3.AA Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: sda1 sda2 < sda5 sda6 sda7 >
sd 0:0:0:0: Attached scsi disk sda
SCSI device sdb: 586072368 512-byte hdwr sectors (300069 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
SCSI device sdb: 586072368 512-byte hdwr sectors (300069 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
sdb: sdb1 sdb2
sd 1:0:0:0: Attached scsi disk sdb
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
EDAC MC: Ver: 2.0.0 May 22 2006
Contact your BIOS vendor to see if the E752x error registers can be safely un-hidden TCP bic registered
NET: Registered protocol family 1
Starting balanced_irq
Using IPI Shortcut mode
input: AT Translated Set 2 keyboard as /class/input/input0
XFS mounting filesystem sdb2
####################################################################
Boot log of SC1425 with CONFIG_ACPI=y, maxcpus=2. This test fails.
Linux version 2.6.17-rc4-kaos (kaos@linuxbuild) (gcc version 3.3.3 (SuSE Linux)) #52 SMP PREEMPT Mon May 22 17:17:11 EST 2006 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
BIOS-e820: 0000000000100000 - 000000007ffc0000 (usable)
BIOS-e820: 000000007ffc0000 - 000000007ffcfc00 (ACPI data)
BIOS-e820: 000000007ffcfc00 - 000000007ffff000 (reserved)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000fe710
DMI 2.3 present.
ACPI: PM-Timer IO Port: 0x808
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
Processor #6 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
Processor #1 15:4 APIC version 20
WARNING: maxcpus limit of 2 reached. Processor ignored.
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
Processor #7 15:4 APIC version 20
WARNING: maxcpus limit of 2 reached. Processor ignored.
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[32])
IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 32-55
ACPI: IOAPIC (id[0x0a] address[0xfec80800] gsi_base[64])
IOAPIC[2]: apic_id 10, version 32, address 0xfec80800, GSI 64-87
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Enabling APIC mode: Flat. Using 3 I/O APICs
ACPI: HPET id: 0xffffffff base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 80000000 (gap: 7ffff000:60001000)
Built 1 zonelists
Kernel command line: root=/dev/sdb2 ro selinux=0 console=tty console=ttyS0,9600 maxcpus=2
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c03d3000 soft=c03cb000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2075288k/2096896k available (1828k kernel code, 20460k reserved, 813k data, 180k init, 1179392k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok.
Using HPET for base-timer
Using HPET for gettimeofday
Detected 3000.213 MHz processor.
Using hpet for high-res timesource
Calibrating delay using timer specific routine.. 6005.06 BogoMIPS (lpj=12010131)
Mount-cache hash table entries: 512
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
CPU0: Thermal monitoring enabled
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 12k freed
CPU0: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Booting processor 1/1 eip 2000
CPU 1 irqstacks, hard=c03d4000 soft=c03cc000
Initializing CPU#1
Calibrating delay using timer specific routine.. 6000.25 BogoMIPS (lpj=12000510)
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Total of 2 processors activated (12005.32 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 2 CPUs: passed.
Brought up 2 CPUs
migration_cost=8000
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using MMCONFIG
Setting up standard PCI resources
ACPI: Subsystem revision 20060127
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 *11 12)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 7 10 11 12)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *10 11 12)
ACPI: PCI Interrupt Link [LNKE] (IRQs *3 4 5 6 7 10 11 12)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 *6 7 10 11 12)
SCSI subsystem initialized
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
PCI: Bridge: 0000:01:00.0
IO window: e000-efff
MEM window: fe900000-feafffff
PREFETCH window: disabled.
PCI: Bridge: 0000:01:00.2
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
IO window: e000-efff
MEM window: fe700000-feafffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: d000-dfff
MEM window: fe500000-fe6fffff
PREFETCH window: f0000000-f7ffffff
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 16
NET: Registered protocol family 2
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 3145728 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
SGI XFS with large block numbers, no debug enabled
SGI XFS Quota Management subsystem
Initializing Cryptographic API
io scheduler noop registered (default)
Intel E7520/7320/7525 detected.<6>ACPI: Power Button (FF) [PWRF]
ACPI: Video Device [EVGA] (multi-head: no rom: yes post: no)
Real Time Clock Driver v1.12ac
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
floppy0: no floppy controllers found
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
e100: Intel(R) PRO/100 Network Driver, 3.5.10-k2-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH5: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 17
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
hda: TEAC CD-ROM CD-224E, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14ata_piix 0000:00:1f.2: MAP [ P0 -- P1 -- ]
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 17
ata1: SATA max UDMA/133 cmd 0xCCB8 ctl 0xCCB2 bmdma 0xCC80 irq 17
ata2: SATA max UDMA/133 cmd 0xCCA0 ctl 0xCC9A bmdma 0xCC88 irq 17
ata1: dev 0 ATA-6, max UDMA/100, 156250000 sectors: LBA
ata1: dev 0 configured for UDMA/100
scsi0 : ata_piix
ata2: dev 0 ATA-7, max UDMA/133, 586072368 sectors: LBA48
ata2: dev 0 configured for UDMA/133
scsi1 : ata_piix
Vendor: ATA Model: WDC WD800JD-75JN Rev: 06.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: ATA Model: ST3300622AS Rev: 3.AA Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: sda1 sda2 < sda5 sda6 sda7 >
sd 0:0:0:0: Attached scsi disk sda
SCSI device sdb: 586072368 512-byte hdwr sectors (300069 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
SCSI device sdb: 586072368 512-byte hdwr sectors (300069 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
sdb: sdb1 sdb2
sd 1:0:0:0: Attached scsi disk sdb
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
EDAC MC: Ver: 2.0.0 May 22 2006
Contact your BIOS vendor to see if the E752x error registers can be safely un-hidden
input: AT Translated Set 2 keyboard as /class/input/input0
TCP bic registered
NET: Registered protocol family 1
Starting balanced_irq
Using IPI Shortcut mode
XFS mounting filesystem sdb2
####################################################################
Boot log of SC1425 with CONFIG_ACPI=n, maxcpus not specified. This test fails.
Linux version 2.6.17-rc4-kaos (kaos@linuxbuild) (gcc version 3.3.3 (SuSE Linux)) #54 SMP PREEMPT Mon May 22 17:47:45 EST 2006 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
BIOS-e820: 0000000000100000 - 000000007ffc0000 (usable)
BIOS-e820: 000000007ffc0000 - 000000007ffcfc00 (ACPI data)
BIOS-e820: 000000007ffcfc00 - 000000007ffff000 (reserved)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000fe710
DMI 2.3 present.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: DELL Product ID: PE SC1425 APIC at: 0xFEE00000
Processor #0 15:4 APIC version 20
Processor #6 15:4 APIC version 20
I/O APIC #8 Version 32 at 0xFEC00000.
I/O APIC #9 Version 32 at 0xFEC80000.
I/O APIC #10 Version 32 at 0xFEC80800.
Enabling APIC mode: Flat. Using 3 I/O APICs
Processors: 2
Allocating PCI resources starting at 80000000 (gap: 7ffff000:60001000)
Built 1 zonelists
Kernel command line: root=/dev/sdb2 ro selinux=0 console=tty console=ttyS0,9600 maxcpus=2
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0397000 soft=c038f000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Detected 3000.854 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2075552k/2096896k available (1689k kernel code, 20196k reserved, 728k data, 164k init, 1179392k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 6009.37 BogoMIPS (lpj=12018756)
Mount-cache hash table entries: 512
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
CPU0: Thermal monitoring enabled
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 12k freed
CPU0: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Booting processor 1/6 eip 2000
CPU 1 irqstacks, hard=c0398000 soft=c0390000
Initializing CPU#1
Calibrating delay using timer specific routine.. 6000.77 BogoMIPS (lpj=12001555)
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 3
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 3.00GHz stepping 03
Total of 2 processors activated (12010.15 BogoMIPS).
ExtINT not setup in hardware but reported by MP table
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=0 pin2=0
checking TSC synchronization across 2 CPUs: passed.
Brought up 2 CPUs
migration_cost=1327
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfc3de, last bus=4
Setting up standard PCI resources
SCSI subsystem initialized
PCI: Probing PCI hardware
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: Transparent bridge - 0000:00:1e.0
PCI: Using IRQ router PIIX/ICH [8086/24d0] at 0000:00:1f.0PCI->APIC IRQ transform: 0000:00:02.0[A] -> IRQ 16PCI->APIC IRQ transform: 0000:00:1d.0[A] -> IRQ 16PCI->APIC IRQ transform: 0000:00:1d.1[B] -> IRQ 19PCI->APIC IRQ transform: 0000:00:1d.7[D] -> IRQ 23PCI->APIC IRQ transform: 0000:00:1f.1[A] -> IRQ 18PCI->APIC IRQ transform: 0000:00:1f.2[A] -> IRQ 18PCI->APIC IRQ transform: 0000:02:04.0[A] -> IRQ 24PCI->APIC IRQ transform: 0000:04:03.0[A] -> IRQ 20PCI->APIC IRQ transform: 0000:04:0d.0[A] -> IRQ 17
PCI: Bridge: 0000:01:00.0
IO window: e000-efff
MEM window: fe900000-feafffff
PREFETCH window: disabled.
PCI: Bridge: 0000:01:00.2
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
IO window: e000-efff
MEM window: fe700000-feafffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: d000-dfff
MEM window: fe500000-fe6fffff
PREFETCH window: f0000000-f7ffffff
NET: Registered protocol family 2
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 3145728 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
SGI XFS with large block numbers, no debug enabled
SGI XFS Quota Management subsystem
Initializing Cryptographic API
io scheduler noop registered (default)
Intel E7520/7320/7525 detected.<6>Real Time Clock Driver v1.12ac
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
floppy0: no floppy controllers found
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
e100: Intel(R) PRO/100 Network Driver, 3.5.10-k2-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH5: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
hda: TEAC CD-ROM CD-224E, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14ata_piix 0000:00:1f.2: MAP [ P0 -- P1 -- ]
ata1: SATA max UDMA/133 cmd 0xCCB8 ctl 0xCCB2 bmdma 0xCC80 irq 18
ata2: SATA max UDMA/133 cmd 0xCCA0 ctl 0xCC9A bmdma 0xCC88 irq 18
ata1: dev 0 ATA-6, max UDMA/100, 156250000 sectors: LBA
ata1: dev 0 configured for UDMA/100
scsi0 : ata_piix
ata2: dev 0 ATA-7, max UDMA/133, 586072368 sectors: LBA48
ata2: dev 0 configured for UDMA/133
scsi1 : ata_piix
Vendor: ATA Model: WDC WD800JD-75JN Rev: 06.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: ATA Model: ST3300622AS Rev: 3.AA Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
SCSI device sda: 156250000 512-byte hdwr sectors (80000 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: sda1 sda2 < sda5 sda6 sda7 >
sd 0:0:0:0: Attached scsi disk sda
SCSI device sdb: 586072368 512-byte hdwr sectors (300069 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
SCSI device sdb: 586072368 512-byte hdwr sectors (300069 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
sdb: sdb1 sdb2
sd 1:0:0:0: Attached scsi disk sdb
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
EDAC MC: Ver: 2.0.0 May 22 2006
Contact your BIOS vendor to see if the E752x error registers can be safely un-hidden TCP bic registered
NET: Registered protocol family 1
Starting balanced_irq
Using IPI Shortcut mode
input: AT Translated Set 2 keyboard as /class/input/input0
####################################################################
[1] Extract of 25366819.pdf (IA-32 Intel Architecture Software Developer’s Manual Volume 3A: System Programming Guide, Part 1)
5.3.3 Software-Generated Interrupts
The INT n instruction permits interrupts to be generated from within software by supplying an interrupt vector number as an operand. For example, the INT 35 instruction forces an implicit call to the interrupt handler for interrupt 35.
Any of the interrupt vectors from 0 to 255 can be used as a parameter in this instruction. If the processor ’s predefined NMI vector is used, however, the response of the processor will not be the same as it would be from an NMI interrupt generated in the normal manner. If vector number 2 (the NMI vector) is used in this instruction, the NMI interrupt handler is called, but the processor’s NMI-handling hardware is not activated.
5.7 NONMASKABLE INTERRUPT (NMI)
The nonmaskable interrupt (NMI) can be generated in either of two ways:
* External hardware asserts the NMI pin.
* The processor receives a message on the system bus (Pentium 4 and Intel Xeon processors) or the APIC serial bus (P6 family and Pentium processors) with a delivery mode NMI.
When the processor receives a NMI from either of these sources, the processor handles it immediately by calling the NMI handler pointed to by interrupt vector number 2. The processor also invokes certain hardware conditions to insure that no other interrupts, including NMI interrupts, are received until the NMI handler has completed executing (see Section 5.7.1, “Handling Multiple NMIs�).
Also, when an NMI is received from either of the above sources, it cannot be masked by the IF flag in the EFLAGS register.
It is possible to issue a maskable hardware interrupt (through the INTR pin) to vector 2 to invoke the NMI interrupt handler; however, this interrupt will not truly be an NMI interrupt. A true NMI interrupt that activates the processor’s NMI-handling hardware can only be delivered through one of the mechanisms listed above.
Quoted 6 lines
(1) IPI 2, not marked as NMI. This does _not_ call into the
do_nmi()routine.People have been telling me (hi, Andi:) that sending interrupt 2 as an IPI automatically sends it as an NMI.
I can't remember ever saying that. I said that sending anything with the NMI bit set will end up at the NMI vector, not the original vector you specified. Or at least that is what the Intel manual specify. That is why it is useless to hook the original vector like you did and add special cases just to get an NMI send with different vector.
-Andi
Quoted 4 lines · Andi Kleen (on Tue, 23 May 2006 01:56:39 +0200) wrote:
Quoted 4 lines
(1) IPI 2, not marked as NMI. This does _not_ call into the
do_nmi()routine.
Quoted 8 lines
Quoted 2 lines
People have been telling me (hi, Andi:) that sending interrupt 2 as an IPI automatically sends it as an NMI.
I can't remember ever saying that. I said that sending anything with the NMI bit set will end up at the NMI vector, not the original vector you specified. Or at least that is what the Intel manual specify. That is why it is useless to hook the original vector like you did and add special cases just to get an NMI send with different vector.
I have never disagreed that all NMIs will end up on the NMI vector (2). But there still has to be code in arch/*/kernel to detect that the IPI being sent is to be marked as NMI, IOW you still need the code that sets APIC_DM_NMI. Whether that is done by using a special vector number (i386 does) or by defining a separate routine for sending NMI (x86_64 does) is a matter for debate.
Unfortunately the way that you changed the x86_64 kdb code, it now does neither. Your hack to kdb sends an IPI using NMI_VECTOR (2) which is
(a) not actually sent as an NMI and
(b) on most of the hardware I have tested, it does not even get through to the other cpus, instead it generates APIC errors.
FWIW, kdb on ia64 first sends a normal maskable IPI using its own KDB_VECTOR and waits for the other cpus to rendezvous. Only if some cpus have not rendezvoused does ia64 kdb resort to using a non-maskable interrupt. I have found that this gives much better backtracing and is more reliable. It is a sad fact of life that NMIs can be delivered in the middle of code that sets up the kernel stack, when that happens it is impossible to backtrace. I am changing kdb on i386 to do the same two step process, try a normal interrupt first, wait a bit then resort to NMI.
nd add special cases just to get an NMI send with different vector.
Quoted 2 lines
I have never disagreed that all NMIs will end up on the NMI vector (2).
The problem was that KDB had an own handler for its debug vector, although that was only ever called as NMI.
Quoted 6 lines
Unfortunately the way that you changed the
x86_64kdb code, it now does neither. Your hack to kdb sends an IPI usingNMI_VECTOR(2) which is(a) not actually sent as an NMI and
(b) on most of the hardware I have tested, it does not even get through to the other cpus, instead it generates APIC errors.
Ok fine we can use some other vector and just the NMI bit. Feel free to submit a patch to do that conversion for NMI_VECTOR
-Andi
Quoted 6 lines · Andi Kleen (on Tue, 23 May 2006 03:55:48 +0200) wrote:
nd add special cases just to get an NMI send with different vector.
Quoted 2 lines
I have never disagreed that all NMIs will end up on the NMI vector (2).
The problem was that KDB had an own handler for its debug vector, although that was only ever called as NMI.
You are confusing KDB_ENTER (instruction code 'int 0x81') with KDB_VECTOR (IPI). KDB_ENTER needs its own int handler which is not an NMI, KDB_VECTOR does not need (and does not have) its own handler, it is handled by the NMI vector.
Quoted 5 lines · Keith Owens (on Tue, 23 May 2006 12:02:10 +1000) wrote:
Quoted 4 lines · Andi Kleen (on Tue, 23 May 2006 03:55:48 +0200) wrote:
nd add special cases just to get an NMI send with different vector.
Quoted 2 lines
I have never disagreed that all NMIs will end up on the NMI vector (2).
Quoted 7 lines
Quoted 2 lines
The problem was that KDB had an own handler for its debug vector, although that was only ever called as NMI.
You are confusing
KDB_ENTER(instruction code 'int0x81') withKDB_VECTOR(IPI).KDB_ENTERneeds its own int handler which is not an NMI,KDB_VECTORdoes not need (and does not have) its own handler, it is handled by the NMI vector.
Or you might be thinking of kdb-v0.6-2.2.13 and earlier, which did have a spurious handler for KDB_VECTOR. But that was fixed in kdb-v1.0-2.3.29, back in January 2000. You are not going to hold that against me are you, especially since I was not maintaining KDB then?
Quoted 2 lines · Keith Owens (on Mon, 22 May 2006 19:08:04 +1000) wrote:
Summary: sending an IPI as NMI will reboot a Dell SMP Xeon box. This
looks like a problem with setting up the I/O APICs.
Some more data points, which make this problem even less understandable. My previous mail described an i386 build, so I tried an x86_64 kernel on two boxes that support 64 bit. Both of them used flat APIC mode.
Intel M/B, CONFIG_ACPI=y. Sending IPI 2 without APIC_DM_NMI is unexpectedly delivered as an NMI, and gets the 'Dazed and confused message'. Contrast this with an i386 kernel with CONFIG_ACPI=y on the same box, where IPI 2 without APIC_DM_NMI is not delivered as NMI, instead it gets APIC errors.
Dell SC1425 M/B, CONFIG_ACPI=y. Sending IPI 2 without APIC_DM_NMI is not delivered at all, but there are no APIC errors. Contrast this with an i386 kernel with CONFIG_ACPI=y on the same box, where IPI 2 without APIC_DM_NMI is also not delivered, but it gets APIC errors.
Dell SC1425 M/B, CONFIG_ACPI=y. Sending IPI 2 with APIC_DM_NMI is delivered as NMI, but it does not get the 'Dazed and confused' message. Contrast this with an i386 kernel with CONFIG_ACPI=y on the same box, where IPI 2 with APIC_DM_NMI is also delivered as NMI, but with the 'Dazed and confused' message.
Dell SC1425 M/B, CONFIG_ACPI=n. Sending IPI 2 without APIC_DM_NMI causes an instant reset. Contrast this with an i386 kernel with CONFIG_ACPI=n on the same box, which got APIC errors but no reset, unless APIC_DM_NMI was set.
i386 vs. x86_64 differences. Intel M/B vs. Dell M/B differences. Colour me confused.
Stripped x86_64 .config.
X86_64=y
64BIT=y
X86=y
SEMAPHORE_SLEEPERS=y
MMU=y
RWSEM_GENERIC_SPINLOCK=y
GENERIC_HWEIGHT=y
GENERIC_CALIBRATE_DELAY=y
X86_CMPXCHG=y
EARLY_PRINTK=y
GENERIC_ISA_DMA=y
GENERIC_IOMAP=y
ARCH_MAY_HAVE_PC_FDC=y
DMI=y
EXPERIMENTAL=y
LOCK_KERNEL=y
INIT_ENV_ARG_LIMIT=32
LOCALVERSION="-kaos"
SWAP=y
SYSVIPC=y
SYSCTL=y
RELAY=y
INITRAMFS_SOURCE=""
UID16=y
VM86=y
CC_OPTIMIZE_FOR_SIZE=y
KALLSYMS=y
KALLSYMS_ALL=y
HOTPLUG=y
PRINTK=y
BUG=y
ELF_CORE=y
BASE_FULL=y
FUTEX=y
EPOLL=y
SHMEM=y
SLAB=y
BASE_SMALL=0
MODULES=y
MODULE_UNLOAD=y
MODULE_FORCE_UNLOAD=y
STOP_MACHINE=y
LBD=y
IOSCHED_NOOP=y
DEFAULT_NOOP=y
DEFAULT_IOSCHED="noop"
X86_PC=y
GENERIC_CPU=y
X86_L1_CACHE_BYTES=128
X86_L1_CACHE_SHIFT=7
X86_INTERNODE_CACHE_BYTES=128
X86_TSC=y
X86_GOOD_APIC=y
X86_MSR=y
X86_CPUID=y
X86_HT=y
X86_IO_APIC=y
X86_LOCAL_APIC=y
MTRR=y
SMP=y
SCHED_SMT=y
SCHED_MC=y
PREEMPT=y
PREEMPT_BKL=y
ARCH_SPARSEMEM_ENABLE=y
ARCH_FLATMEM_ENABLE=y
SELECT_MEMORY_MODEL=y
FLATMEM_MANUAL=y
FLATMEM=y
FLAT_NODE_MEM_MAP=y
SPLIT_PTLOCK_CPUS=4
NR_CPUS=8
HPET_TIMER=y
GART_IOMMU=y
SWIOTLB=y
X86_MCE=y
X86_MCE_INTEL=y
X86_MCE_AMD=y
PHYSICAL_START=0x200000
SECCOMP=y
HZ_250=y
HZ=250
GENERIC_HARDIRQS=y
GENERIC_IRQ_PROBE=y
ISA_DMA_API=y
GENERIC_PENDING_IRQ=y
PM=y
ACPI=y
ACPI_AC=y
ACPI_BATTERY=y
ACPI_BUTTON=y
ACPI_VIDEO=y
ACPI_FAN=y
ACPI_PROCESSOR=y
ACPI_THERMAL=y
ACPI_BLACKLIST_YEAR=0
ACPI_EC=y
ACPI_POWER=y
ACPI_SYSTEM=y
X86_PM_TIMER=y
PCI=y
PCI_DIRECT=y
PCI_MSI=y
BINFMT_ELF=y
BINFMT_MISC=y
IA32_EMULATION=y
COMPAT=y
SYSVIPC_COMPAT=y
NET=y
PACKET=y
PACKET_MMAP=y
UNIX=y
INET=y
IP_FIB_HASH=y
SYN_COOKIES=y
INET_DIAG=y
INET_TCP_DIAG=y
TCP_CONG_BIC=y
NETFILTER=y
NETFILTER_NETLINK=y
NETFILTER_XTABLES=y
NETFILTER_XT_TARGET_CLASSIFY=y
NETFILTER_XT_TARGET_MARK=y
NETFILTER_XT_TARGET_NFQUEUE=y
NETFILTER_XT_TARGET_NOTRACK=y
NETFILTER_XT_MATCH_COMMENT=y
NETFILTER_XT_MATCH_CONNTRACK=y
NETFILTER_XT_MATCH_DCCP=y
NETFILTER_XT_MATCH_HELPER=y
NETFILTER_XT_MATCH_LENGTH=y
NETFILTER_XT_MATCH_LIMIT=y
NETFILTER_XT_MATCH_MAC=y
NETFILTER_XT_MATCH_MARK=y
NETFILTER_XT_MATCH_PKTTYPE=y
NETFILTER_XT_MATCH_REALM=y
NETFILTER_XT_MATCH_SCTP=y
NETFILTER_XT_MATCH_STATE=y
NETFILTER_XT_MATCH_STRING=y
NETFILTER_XT_MATCH_TCPMSS=y
IP_NF_CONNTRACK=y
IP_NF_CONNTRACK_NETLINK=y
IP_NF_FTP=y
IP_NF_IRC=y
IP_NF_QUEUE=y
IP_NF_IPTABLES=y
IP_NF_MATCH_IPRANGE=y
IP_NF_MATCH_TOS=y
IP_NF_MATCH_RECENT=y
IP_NF_MATCH_ECN=y
IP_NF_MATCH_DSCP=y
IP_NF_MATCH_TTL=y
IP_NF_MATCH_OWNER=y
IP_NF_MATCH_ADDRTYPE=y
IP_NF_MATCH_HASHLIMIT=y
IP_NF_FILTER=y
IP_NF_TARGET_REJECT=y
IP_NF_TARGET_LOG=y
IP_NF_TARGET_ULOG=y
IP_NF_TARGET_TCPMSS=y
IP_NF_NAT=y
IP_NF_NAT_NEEDED=y
IP_NF_TARGET_MASQUERADE=y
IP_NF_TARGET_REDIRECT=y
IP_NF_TARGET_NETMAP=y
IP_NF_TARGET_SAME=y
IP_NF_NAT_SNMP_BASIC=y
IP_NF_NAT_IRC=y
IP_NF_NAT_FTP=y
IP_NF_MANGLE=y
IP_NF_TARGET_TOS=y
IP_NF_TARGET_ECN=y
IP_NF_TARGET_DSCP=y
IP_NF_TARGET_TTL=y
IP_NF_RAW=y
IP_NF_ARPTABLES=y
IP_NF_ARPFILTER=y
IP_NF_ARP_MANGLE=y
NET_CLS_ROUTE=y
STANDALONE=y
PREVENT_FIRMWARE_BUILD=y
PARPORT=y
PARPORT_PC=y
BLK_DEV_FD=y
BLK_DEV_LOOP=y
BLK_DEV_RAM=y
BLK_DEV_RAM_COUNT=16
BLK_DEV_RAM_SIZE=4096
IDE=y
BLK_DEV_IDE=y
BLK_DEV_IDEDISK=y
IDEDISK_MULTI_MODE=y
BLK_DEV_IDECD=y
BLK_DEV_IDESCSI=y
IDE_GENERIC=y
BLK_DEV_IDEPCI=y
IDEPCI_SHARE_IRQ=y
BLK_DEV_GENERIC=y
BLK_DEV_IDEDMA_PCI=y
IDEDMA_PCI_AUTO=y
BLK_DEV_PIIX=y
BLK_DEV_IDEDMA=y
IDEDMA_AUTO=y
SCSI=y
SCSI_PROC_FS=y
BLK_DEV_SD=y
BLK_DEV_SR=y
BLK_DEV_SR_VENDOR=y
CHR_DEV_SG=y
SCSI_CONSTANTS=y
SCSI_LOGGING=y
SCSI_SATA=y
SCSI_ATA_PIIX=y
SCSI_SATA_INTEL_COMBINED=y
MD=y
BLK_DEV_MD=y
MD_LINEAR=y
MD_RAID0=y
MD_RAID1=y
MD_RAID10=y
MD_RAID5=y
MD_RAID6=y
MD_MULTIPATH=y
BLK_DEV_DM=y
DM_CRYPT=y
DM_SNAPSHOT=y
DM_MIRROR=y
DM_ZERO=y
NETDEVICES=y
DUMMY=y
TUN=y
NET_ETHERNET=y
MII=y
NET_PCI=y
E100=y
E1000=y
TIGON3=y
PPP=y
PPP_ASYNC=y
PPP_DEFLATE=y
PPP_BSDCOMP=y
INPUT=y
INPUT_MOUSEDEV=y
INPUT_MOUSEDEV_PSAUX=y
INPUT_MOUSEDEV_SCREEN_X=1024
INPUT_MOUSEDEV_SCREEN_Y=768
INPUT_KEYBOARD=y
KEYBOARD_ATKBD=y
INPUT_MOUSE=y
MOUSE_PS2=y
MOUSE_SERIAL=y
SERIO=y
SERIO_I8042=y
SERIO_SERPORT=y
SERIO_PCIPS2=y
SERIO_LIBPS2=y
VT=y
VT_CONSOLE=y
HW_CONSOLE=y
SERIAL_8250=y
SERIAL_8250_CONSOLE=y
SERIAL_8250_PCI=y
SERIAL_8250_NR_UARTS=4
SERIAL_8250_RUNTIME_UARTS=4
SERIAL_8250_EXTENDED=y
SERIAL_CORE=y
SERIAL_CORE_CONSOLE=y
UNIX98_PTYS=y
LEGACY_PTYS=y
LEGACY_PTY_COUNT=256
PRINTER=y
RTC=y
AGP=y
AGP_AMD64=y
AGP_INTEL=y
HWMON=y
VGA_CONSOLE=y
DUMMY_CONSOLE=y
USB_ARCH_HAS_HCD=y
USB_ARCH_HAS_OHCI=y
USB_ARCH_HAS_EHCI=y
EDAC=y
EDAC_DEBUG=y
EDAC_MM_EDAC=y
EDAC_E752X=y
EDAC_POLL=y
EXT2_FS=y
FS_POSIX_ACL=y
XFS_FS=y
XFS_EXPORT=y
XFS_QUOTA=y
ROMFS_FS=y
INOTIFY=y
QUOTACTL=y
DNOTIFY=y
AUTOFS_FS=y
AUTOFS4_FS=y
FUSE_FS=y
ISO9660_FS=y
JOLIET=y
ZISOFS=y
ZISOFS_FS=y
UDF_FS=y
UDF_NLS=y
FAT_FS=y
MSDOS_FS=y
VFAT_FS=y
FAT_DEFAULT_CODEPAGE=437
FAT_DEFAULT_IOCHARSET="iso8859-1"
NTFS_FS=y
PROC_FS=y
PROC_KCORE=y
SYSFS=y
TMPFS=y
RAMFS=y
EFS_FS=y
HPFS_FS=y
UFS_FS=y
NFS_FS=y
NFS_V3=y
NFS_V4=y
NFSD=y
NFSD_V3=y
NFSD_V4=y
NFSD_TCP=y
LOCKD=y
LOCKD_V4=y
EXPORTFS=y
NFS_COMMON=y
SUNRPC=y
SUNRPC_GSS=y
RPCSEC_GSS_KRB5=y
SMB_FS=y
MSDOS_PARTITION=y
NLS=y
NLS_DEFAULT="iso8859-1"
NLS_CODEPAGE_437=y
NLS_CODEPAGE_850=y
NLS_ISO8859_1=y
NLS_UTF8=y
MAGIC_SYSRQ=y
DEBUG_KERNEL=y
LOG_BUF_SHIFT=16
DEBUG_PREEMPT=y
DEBUG_MUTEXES=y
DEBUG_INFO=y
DEBUG_FS=y
FRAME_POINTER=y
FORCED_INLINING=y
DEBUG_RODATA=y
CRYPTO=y
CRYPTO_MD5=y
CRYPTO_DES=y
CRYPTO_AES_X86_64=y
CRC_CCITT=y
CRC32=y
ZLIB_INFLATE=y
ZLIB_DEFLATE=y
TEXTSEARCH=y
TEXTSEARCH_KMP=y
TEXTSEARCH_BM=y
TEXTSEARCH_FSM=y
Following a suggestion by Brendan Trotter, I ran some more tests to track down the problem with sending NMI IPI on Dell Xeons.
BIOS Logical OS ACPI Cpus IPI 2 NMI IPI
Processor BIOS OS (APIC_DM_NMI)
Enabled Enabled 4 4 Not delivered Delivered as NMI Enabled Disabled 4 2 Machine reset Machine reset Disabled Enabled 2 2 Not delivered Delivered as NMI Disabled Disabled 2 2 Not delivered Delivered as NMI
So the killer combination with this motherboard is when the BIOS knows about logical processors but the OS does not. Sending IPI 2 or NMI IPI with that combination kills the machine. Brendan suggested that the BIOS is seeing the broadcast NMI on the logical processors which are not under OS control and that the BIOS cannot cope.
Should we change the x86_64 send_IPI_allbutself() so it is only delivered to cpus that the OS knows about, instead of doing a general broadcast. That would prevent offline or hidden cpus being sent an interrupt that they are not expecting. The failing case is __send_IPI_shortcut, with a cfg of 0xc0c00.
Quoted 11 lines · Keith Owens
Following a suggestion by Brendan Trotter, I ran some more tests to track down the problem with sending NMI IPI on Dell Xeons.
BIOS Logical OS ACPI Cpus IPI 2 NMI IPI
Processor BIOS OS (APIC_DM_NMI)Enabled Enabled 4 4 Not delivered Delivered as NMI Enabled Disabled 4 2 Machine reset Machine reset Disabled Enabled 2 2 Not delivered Delivered as NMI Disabled Disabled 2 2 Not delivered Delivered as NMI
Quoted 5 lines
So the killer combination with this motherboard is when the BIOS knows about logical processors but the OS does not. Sending IPI 2 or NMI IPI with that combination kills the machine. Brendan suggested that the BIOS is seeing the broadcast NMI on the logical processors which are not under OS control and that the BIOS cannot cope.
How did you manage that? Normally the OS should use all CPUs known to BIOS. Or did you boot with special boot options to limit it?
Quoted 3 lines
Should we change the
x86_64send_IPI_allbutself()so it is only delivered to cpus that the OS knows about, instead of doing a general broadcast.
Hmm, we should be doing that already to avoid races for CPU hotplug. But maybe it's not working correctly for KDB. Does it go away when you enable CPU hotplug? Anyways, should be a SMOP to force it. I wouldn't have a problem to use sequence ipis always and get rid of the broadcasts. There were benchmarks at some point and there wasn't a noticeable difference.
-Andi
Quoted 12 lines · Andi Kleen (on Wed, 7 Jun 2006 09:20:23 +0200) wrote:
Quoted 11 lines · Keith Owens
Following a suggestion by Brendan Trotter, I ran some more tests to track down the problem with sending NMI IPI on Dell Xeons.
BIOS Logical OS ACPI Cpus IPI 2 NMI IPI
Processor BIOS OS (APIC_DM_NMI)Enabled Enabled 4 4 Not delivered Delivered as NMI Enabled Disabled 4 2 Machine reset Machine reset Disabled Enabled 2 2 Not delivered Delivered as NMI Disabled Disabled 2 2 Not delivered Delivered as NMI
Quoted 8 lines
Quoted 5 lines
So the killer combination with this motherboard is when the BIOS knows about logical processors but the OS does not. Sending IPI 2 or NMI IPI with that combination kills the machine. Brendan suggested that the BIOS is seeing the broadcast NMI on the logical processors which are not under OS control and that the BIOS cannot cope.
How did you manage that? Normally the OS should use all CPUs known to BIOS. Or did you boot with special boot options to limit it?
Two ways:
(1) Boot with a kernel with CONFIG_ACPI=n, so the OS only finds 2 cpus in the MPT instead of the 4 listed by ACPI.
(2) The kernel has ACPI=y, but is booted with maxcpus=2.
In both cases, send_IPI_allbutself() with IPI 2 or an NMI will result in a hard reset.
Quoted 6 lines
Quoted 3 lines
Should we change the
x86_64send_IPI_allbutself()so it is only delivered to cpus that the OS knows about, instead of doing a general broadcast.Hmm, we should be doing that already to avoid races for CPU hotplug. But maybe it's not working correctly for KDB.
This problem is not KDB specific, although that is where it was first noticed. Any code that sends a broadcast IPI 2 or an NMI IPI will crash these Dell boxes when there is a mismatch between the cpus known to the BIOS and the cpus known to the OS.
Quoted 2 lines
Does it go away when you
enable CPU hotplug?
HOTPLUG_CPU was already on in all of my test kernels.
Quoted 4 lines
Anyways, should be a SMOP to force it. I wouldn't
have a problem to use sequence ipis always and get rid of the broadcasts. There were benchmarks at some point and there wasn't a noticeable difference.
I will try forcing send_IPI_allbutself() to use the mask version rather than the broadcast shortcut. Later tonight ...
Quoted 11 lines
Two ways:
(1) Boot with a kernel with
CONFIG_ACPI=n, so the OS only finds 2 cpus in the MPT instead of the 4 listed by ACPI.(2) The kernel has ACPI=y, but is booted with maxcpus=2.
In both cases,
send_IPI_allbutself()with IPI 2 or an NMI will result in a hard reset.
Sounds both like a "Don't do that when it hurts" . I know some people have religious issues with ACPI, but it's simple a fact that many modern boxes don't work correctly in obvious or subtle ways without it.
-Andi
Quoted 7 lines · Andi Kleen (on Wed, 7 Jun 2006 10:01:40 +0200) wrote:
Quoted 7 lines
Two ways:
(1) Boot with a kernel with
CONFIG_ACPI=n, so the OS only finds 2 cpus in the MPT instead of the 4 listed by ACPI.
Quoted 8 lines
Quoted 4 lines
(2) The kernel has ACPI=y, but is booted with maxcpus=2.
In both cases,
send_IPI_allbutself()with IPI 2 or an NMI will result in a hard reset.Sounds both like a "Don't do that when it hurts" . I know some people have religious issues with ACPI, but it's simple a fact that many modern boxes don't work correctly in obvious or subtle ways without it.
Building a kernel without ACPI is silly nowadays. But even with ACPI, booting with a restricted maxcpus and sending IPI 2 or NMI as broadcast will kill these boxes. maxcpus is a valid option.
Quoted 13 lines · Keith Owens
Quoted 12 lines · Andi Kleen (on Wed, 7 Jun 2006 10:01:40 +0200) wrote:
Quoted 11 lines
Two ways:
(1) Boot with a kernel with
CONFIG_ACPI=n, so the OS only finds 2 cpus in the MPT instead of the 4 listed by ACPI.(2) The kernel has ACPI=y, but is booted with maxcpus=2.
In both cases,
send_IPI_allbutself()with IPI 2 or an NMI will result in a hard reset.
Quoted 7 lines
Quoted 3 lines
Sounds both like a "Don't do that when it hurts" . I know some people have religious issues with ACPI, but it's simple a fact that many modern boxes don't work correctly in obvious or subtle ways without it.
Building a kernel without ACPI is silly nowadays. But even with ACPI, booting with a restricted maxcpus and sending IPI 2 or NMI as broadcast will kill these boxes. maxcpus is a valid option.
Yes, I'm not opposed to using *_sequence on cpu_online_map here. Please send a patch.
-Andi
Hi,
Quoted 4 lines · Keith Owens
This problem is not KDB specific, although that is where it was first noticed. Any code that sends a broadcast IPI 2 or an NMI IPI will crash these Dell boxes when there is a mismatch between the cpus known to the BIOS and the cpus known to the OS.
I missed this (broadcast IPI 2 causing problems)...
Before the OS starts all AP CPUs should be in a special halt state (specifically "cli; hlt"), where the only interrupt sources that reach the CPU are NMI, SMI and INIT. This is why broadcast NMI is unsafe for a variety of reasons (not just on Dells).
If the Dell machines allow an IPI 2 to get through then unused CPUs aren't in the "cli; hlt" state, and they would also respond to any other broadcast IPIs. This would imply that any code that uses send_IPI_allbutself(), send_IPI_all() or anything else that sends broadcast IPIs would/could cause problems on these Dell machines. Depending on whether Linux leaves the real mode IVT intact or not and which interrupt vector/s are used, the result could be nothing at all, erratic behaviour, triple faults/resets, or executing unintended BIOS functions (which could also trash memory or in some cases cause device failure).
The easiest solution here would be for Dell to release an updated BIOS that fixes this bug (i.e. puts the AP CPUs into a "cli; hlt" state, as per Intel's notes in section 7.5.4.2 "Typical AP Initialization Sequence" of the system programming guide). This still won't/shouldn't fix the "broadcast NMI" problem though.
Quoted 2 lines
I will try forcing
send_IPI_allbutself()to use the mask version rather than the broadcast shortcut. Later tonight ...
I've been looking into this a little - it appears that Linux tries to use one bit per CPU in the local APIC's "logical destination register" and that in all cases at least one bit is set for each valid CPU. As far as I can tell sending an NMI (or any other broadcast IPI) in logical mode with no shorthand and "destination = 0xFF" should work fine for both "cluster mode" and "flat mode". In this case I'd also suggest that "clear_local_APIC()" clear the destination format register (DFR) and the logical destination register (LDR) so that it doesn't receive broadcast NMIs if the CPU is taken offline.
The "send_IPI_mask_sequence()" function also seems like a perfectly valid option (except for the comments, which are easy enough to change). IMHO it's just not as pretty, especially if it's to be used for all broadcast IPIs (rather than just for broadcast NMIs) - I'd be tempted to do an "#ifdef BORKED_DELL" around it in that case. :-)
Cheers,
Brendan
Quoted 6 lines
The "
send_IPI_mask_sequence()" function also seems like a perfectly valid option (except for the comments, which are easy enough to change). IMHO it's just not as pretty, especially if it's to be used for all broadcast IPIs (rather than just for broadcast NMIs) - I'd be tempted to do an "#ifdefBORKED_DELL" around it in that case. :-)
It's not only that Dell that has trouble with broadcasts. On newer platforms which support CPU hotplug there are races at the hardware level with broadcasting. That is why we're moving away from it.
Not sure why this particular case got missed. Ashok might want to comment - he did the hotplug work.
-Andi
Quoted 4 lines · Brendan Trotter
Quoted 3 lines
I will try forcing
send_IPI_allbutself()to use the mask version rather than the broadcast shortcut. Later tonight ...
Quoted 10 lines
I've been looking into this a little - it appears that Linux tries to use one bit per CPU in the local APIC's "logical destination register" and that in all cases at least one bit is set for each valid CPU. As far as I can tell sending an NMI (or any other broadcast IPI) in logical mode with no shorthand and "destination =
0xFF" should work fine for both "cluster mode" and "flat mode". In this case I'd also suggest that "clear_local_APIC()" clear the destination format register (DFR) and the logical destination register (LDR) so that it doesn't receive broadcast NMIs if the CPU is taken offline.
I will double-check with the CPU folks, but I don't think this will do what you want it to do. If you send an NMI IPI with destination set to 0xff, I think it will interrupt all CPUs, including the non-started CPUs in the cli;hlt state. If I remember right, the destination shorthand method simply sends the interrupt on the bus with destination set to 0xff..
Rajesh
Quoted 15 lines · Rajesh Shah
Quoted 4 lines · Brendan Trotter
Quoted 3 lines
I will try forcing
send_IPI_allbutself()to use the mask version rather than the broadcast shortcut. Later tonight ...Quoted 10 lines
I've been looking into this a little - it appears that Linux tries to use one bit per CPU in the local APIC's "logical destination register" and that in all cases at least one bit is set for each valid CPU. As far as I can tell sending an NMI (or any other broadcast IPI) in logical mode with no shorthand and "destination =
0xFF" should work fine for both "cluster mode" and "flat mode". In this case I'd also suggest that "clear_local_APIC()" clear the destination format register (DFR) and the logical destination register (LDR) so that it doesn't receive broadcast NMIs if the CPU is taken offline.
Quoted 3 lines
I will double-check with the CPU folks, but I don't think this will do what you want it to do. If you send an NMI IPI with destination set to
0xff, I think it will interrupt all CPUs,
I was wrong - a (reasonably modern) CPU will indeed not take the NMI IPI if its LDR is clear and the IPI was sent in logical mode. However, the CPU folks weren't expecting it to be used this way, so they aren't sure yet if this is guaranteed to work this way on all CPUs.
Rajesh
Quoted 15 lines · Brendan Trotter
Hi,
Quoted 4 lines · Keith Owens
This problem is not KDB specific, although that is where it was first noticed. Any code that sends a broadcast IPI 2 or an NMI IPI will crash these Dell boxes when there is a mismatch between the cpus known to the BIOS and the cpus known to the OS.
I missed this (broadcast IPI 2 causing problems)...
Before the OS starts all AP CPUs should be in a special halt state (specifically "cli; hlt"), where the only interrupt sources that reach the CPU are NMI, SMI and INIT. This is why broadcast NMI is unsafe for a variety of reasons (not just on Dells).
Quoted 5 lines
If the Dell machines allow an IPI 2 to get through then unused CPUs aren't in the "cli; hlt" state, and they would also respond to any other broadcast IPIs. This would imply that any code that uses
send_IPI_allbutself(),send_IPI_all()or anything else that sends broadcast IPIs would/could cause problems on these Dell machines.
The other possibility is that these CPUs are in the cli;hlt state, but BIOS didn't install a proper NMI handler for them. This would explain why other "normal" broadcast IPIs didn't cause problems while NMI broadcast killed the machine.
Rajesh
Quoted 9 lines · "Brendan Trotter" (on Wed, 7 Jun 2006 15:18:57 +0000) wrote:
Hi,
Quoted 4 lines · Keith Owens
This problem is not KDB specific, although that is where it was first noticed. Any code that sends a broadcast IPI 2 or an NMI IPI will crash these Dell boxes when there is a mismatch between the cpus known to the BIOS and the cpus known to the OS.
I missed this (broadcast IPI 2 causing problems)...
It turns out that there is an uncommented hack in x86_64's version of __prepare_ICR().
case NMI_VECTOR:
icr |= APIC_DM_NMI;
break;That hack is not in i386, which means that IPI 2 does different things in i386 vs. x86_64 mode on the same hardware. I am still tracking down all the differences before doing a clean up patch that makes i386 and x86_64 behave the same.