Where do the "Machine Check Exceptions" come from?
Hello!
We have a Server runing here, with a very strange behavior.
It all started, that the machine crashed in two-day-intervalls with the following message in log:
Jan 6 22:39:01 CPU 0: Machine Check Exception: 0000000000000004 Jan 6 22:39:01 Bank 4: b200000000040151 Jan 6 22:39:01 Kernel panic: CPU context corrupt
So we took the machine out of productivity and started to search for the problem. We first thought it must be some Hardware error, so we did a memtest86 for a long time (over 10 passes) without any errors there. We then booted from a knoppix CD and did a burnMMX and burnP4. Nothing happend, all ran smooth.
So we thought maybe it is some other system component, so we removed everything not needed (network card, scsi-controller) and changed the video-card. We then bootet again from knoppix and did a lot of kernel compiles over night (on the harddisk, not on a ramdisk) --> all went smooth.
We then bootet the original system (without all unneeded hardware), started kernel compiling and it crashed after a day. This was strange. So we looked in out changelog and then realized, that the crashing started, when we changed the running kernel from a vanilla 2.4.19 to a vanilla 2.4.23.
We thought it could be something in the new kernel. So we took a new 2.4.24 with the config from 2.4.23 (make oldconfig) and tested --> system crashed after compiling kernels for a day.
So there must be something else. Next step was to take the config from the 2.4.19 kernel and do a "make oldconfig" with the 2.4.24. The system is now running for two days without a crash. So it must be something that has changed between the two configs.
So I took the config from the faulty 2.4.23 kernel, and did a "make oldconfig" with the running config from 2.4.19 on the 2.4.23 kernel.
I will attach what a "diff faulty_config running_config" showed at the end of the mail.
Any ideas what option new option made the kernel crash? I will try the three options directly compiled into the kernel (not as a module) the next few days and will give an, if I can find out what causes this behavior.
Best regards
Kai Militzer
++++output of diff+++++
153c153
< CONFIG_BLK_STATS=y
---
Quoted 1 line
#
CONFIG_BLK_STATSis not set
194c194
< CONFIG_IP_NF_TFTP=m
---
Quoted 1 line
#
CONFIG_IP_NF_TFTPis not set
200c200
< CONFIG_IP_NF_MATCH_PKTTYPE=m
---
Quoted 1 line
#
CONFIG_IP_NF_MATCH_PKTTYPEis not set
204,206c204,206
< CONFIG_IP_NF_MATCH_RECENT=m
< CONFIG_IP_NF_MATCH_ECN=m
< CONFIG_IP_NF_MATCH_DSCP=m
---
Quoted 3 lines
#
CONFIG_IP_NF_MATCH_RECENTis not set
#CONFIG_IP_NF_MATCH_ECNis not set
#CONFIG_IP_NF_MATCH_DSCPis not set
211c211
< CONFIG_IP_NF_MATCH_HELPER=m
---
Quoted 1 line
#
CONFIG_IP_NF_MATCH_HELPERis not set
213c213
< CONFIG_IP_NF_MATCH_CONNTRACK=m
---
Quoted 1 line
#
CONFIG_IP_NF_MATCH_CONNTRACKis not set
227d226
< CONFIG_IP_NF_NAT_TFTP=m
230,231c229,230
< CONFIG_IP_NF_TARGET_ECN=m
< CONFIG_IP_NF_TARGET_DSCP=m
---
Quoted 2 lines
#
CONFIG_IP_NF_TARGET_ECNis not set
#CONFIG_IP_NF_TARGET_DSCPis not set
238c237
< CONFIG_IP_NF_ARP_MANGLE=m
---
Quoted 1 line
#
CONFIG_IP_NF_ARP_MANGLEis not set
329c328
< CONFIG_BLK_DEV_GENERIC=y
---
Quoted 1 line
#
CONFIG_BLK_DEV_GENERICis not set
557c556
< CONFIG_B44=m
---
Quoted 1 line
#
CONFIG_B44is not set
565c564
< CONFIG_E100=m
---
Quoted 1 line
#
CONFIG_E100is not set
593,594c592
< CONFIG_E1000=m
< # CONFIG_E1000_NAPI is not set
---
Quoted 1 line
#
CONFIG_E1000is not set
599c597
< CONFIG_R8169=m
---
Quoted 1 line
#
CONFIG_R8169is not set
712c710
< CONFIG_HW_RANDOM=m
---
Quoted 1 line
#
CONFIG_HW_RANDOMis not set
910c908
< CONFIG_DEBUG_STACKOVERFLOW=y
---
Quoted 1 line
#
CONFIG_DEBUG_STACKOVERFLOWis not set
927c925
< CONFIG_CRC32=m
---
Quoted 1 line
#
CONFIG_CRC32is not set
929c927
< CONFIG_ZLIB_DEFLATE=m
---
Quoted 1 line
#
CONFIG_ZLIB_DEFLATEis not set
+++++end output of diff++++
Signature · 4 lines
Kai Militzer WESTEND GmbH | Internet-Business-Provider
Technik CISCO Systems Partner - Authorized Reseller
Lütticher Straße 10 Tel 0241/701333-11
[email protected] D-52064 Aachen Fax 0241/911879Re: Where do the "Machine Check Exceptions" come from? [update]
Hello everyone!
I have an update on the reproduction of the strange kernel oopses on an 2.4.24 kernel.
Quoted 6 lines
It all started, that the machine crashed in two-day-intervalls with the following message in log:
Jan 6 22:39:01 CPU 0: Machine Check Exception: 0000000000000004 Jan 6 22:39:01 Bank 4: b200000000040151 Jan 6 22:39:01 Kernel panic: CPU context corrupt
That's the message, that always appears.
We then tested around as described in my original mail.
Quoted 4 lines
So there must be something else. Next step was to take the config from the 2.4.19 kernel and do a "make oldconfig" with the 2.4.24. The system is now running for two days without a crash. So it must be something that has changed between the two configs.
The kernel ran for four days without crashing. So I tried to activate some options, that were activeted in the crashing kernel.
I started with this option, just by a foresought.
Quoted 3 lines
<
CONFIG_DEBUG_STACKOVERFLOW=y
---Quoted 1 line
#
CONFIG_DEBUG_STACKOVERFLOWis not set
It was not set in the kernel running for four days, but in the one, crashing. After I activated it (means: CONFIG_DEBUG_STACKOVERFLOW=y), compiled the kernel and let it run under work for the weekend (starting on friday). This morning (monday) it crashed. So I would say, it was the CONFIG_DEBUG_STACKOVERFLOW.
Does anyone have an idea, why this options makes the kernel crash? Shouldn't this option prevent the kernel from crashing?
If more information is needed (i.e. full kernel config, hardware specs, etc.) please let me know.
Regards
Kai
Signature · 4 lines
Kai Militzer WESTEND GmbH | Internet-Business-Provider
Technik CISCO Systems Partner - Authorized Reseller
Lütticher Straße 10 Tel 0241/701333-11
[email protected] D-52064 Aachen Fax 0241/911879Re[3]: 2.6.2 Compile Failure - Redhat 7.3 Distro
Quoted 9 lines
Hello Robert,
Sighs...I guess I have to look at making new set of RPM packages
for 7.3 Distros \
to upgrade the glibc, gcc and few other packages to have it updated to be able to \
compile the kernel.Thanks.
Saturday, February 7, 2004, 4:31:25 PM, you wrote:
Quoted 19 lines · RFM> Elikster wrote:
Quoted 19 lines
fs/proc/array.c: In function `proc_pid_stat':fs/proc/array.c:398: Unrecognizable insn:
(insn/i 721 1009 1003 (parallel[
(set (reg:SI 0 eax)
(asm_operands("") ("=a") 0[
(reg:DI 1 edx)
]
[
(asm_input:DI ("A"))
] ("include/linux/times.h") 38))
(set (reg:SI 1 edx)
(asm_operands("") ("=d") 1[
(reg:DI 1 edx)
]
[
(asm_input:DI ("A"))
] ("include/linux/times.h") 38))
(clobber (reg:QI 19 dirflag))
(clobber (reg:QI 18 fpsr))
Quoted 8 lines
Quoted 8 lines
(clobber (reg:QI 17 flags))
] ) -1 (insn_list715 (nil))
(nil))fs/proc/array.c:398: confused by earlier errors, bailing out make[2]: *** [fs/proc/array.o] Error 1
make[1]: *** [fs/proc] Error 2
make: *** [fs] Error 2
root@longmont [/usr/src/linux-2.6.2]#
I was getting all sorts of funny reportedly code errors with RH GCC:
gcc version 2.96 20000731 (Red Hat Linux 7.2 2.96-112.7.1)
It used to build ok, then report an error somewhere.. but never in the same place/file. Funnily enough, a kernel built with 5 or 6 of these errors (i.e. just 'make'ing again from where it 'broke' to carry on the build) is OK.
Well, for me it turned out GCC gets very _flakey_ if CPU (K6 233) gets too hot...
...I just put on a bigger heatsink, and it fixed it!
The machine had/has never shown any other symptoms of overheating until/unless I build a kernel.
Nick