AMD FX CPU bug, not fixed by latest microcode?
Hi,
I have an AMD FX-8120 boxed CPU in an ASUS M5A99X-EVO mainboard with 32GB DDR3/1600 memory, running Fedora 17, upgraded from 16. memtest86+ show no problems.
Still, I get occasional crashes and signal 11 during kernel compilation even with single-job make. Sometimes the compiler jumps out with a strange error message, like "stray \NNN character in the source". When re-running make, the error doesn't happen in the same file and the source file doesn't contain the character being complained about when inspecting with an editor or hexdump.
Now, a few minutes ago I was able to catch this bug when I copied the kernel GIT tree to apply a patch manually and did "git commit -a". Strangely, the commit contained one extra file that I didn't touch. git diff showed this for the extra file:
==============================
drivers/usb/gadget/fsl_usb2_udc.h +1 -1
@@ -427,7 +427,7 @@ struct ep_td_struct {#define DTD_ADDR_MASK 0xFFFFFFE0#define DTD_PACKET_SIZE 0x7FFF0000#define DTD_LENGTH_BIT_POS 16 -#define DTD_ERROR_MASK (DTD_STATUS_HALTED | \ +#define DTD_ERROR_MASK (DTD_STATUS_HALTED | ^ZDTD_STATUS_DATA_BUFF_ERR | \ DTD_STATUS_TRANSACTION_ERR)/* Alignment requirements; must be a power of two */
==============================
The "^Z" is a 0-character in the file and is not present in the original source tree, only in the copy.
Similar errors happened during copying large files on the same machine but it seems it's enough to trigger if the total amount of data read is large enough.
The mainboard has the latest (UEFI) firmware flashed which contains the latest AMD microcode, so microcode_ctl doesn't need to apply it anymore. Previously, I used amd-ucode-2012-01-17.tar from www.amd64.org/support/microcode.html which is now part of microcode_ctl in Fedora.
Since the error happens during compiling a source file and not only copying, the bug seems to happens during *reading* data.
Does anyone know whether it's a known problem in AMD FX CPUs? Does AMD have a newer microcode to fix this bug, or should I apply for warranty?
Thanks in advance,
Zoltán Böszörményi
Quoted 29 lines · Boszormenyi Zoltan
I have an AMD FX-8120 boxed CPU in an ASUS M5A99X-EVO mainboard with 32GB DDR3/1600 memory, running Fedora 17, upgraded from 16.
I get occasional crashes and signal 11 during kernel compilation even with single-job make. Sometimes the compiler jumps out with a strange error message, like "stray \NNN character in the source". When re-running make, the error doesn't happen in the same file and the source file doesn't contain the character being complained about when inspecting with an editor or hexdump.
Now, a few minutes ago I was able to catch this bug when I copied the kernel GIT tree to apply a patch manually and did "git commit -a". Strangely, the commit contained one extra file that I didn't touch. git diff showed this for the extra file:
==============================
drivers/usb/gadget/fsl_usb2_udc.h +1
-1@@ -427,7 +427,7 @@ struct ep_td_struct {#define DTD_ADDR_MASK 0xFFFFFFE0#define DTD_PACKET_SIZE 0x7FFF0000#define DTD_LENGTH_BIT_POS 16 -#define DTD_ERROR_MASK (DTD_STATUS_HALTED | \ +#define DTD_ERROR_MASK (DTD_STATUS_HALTED | ^ZDTD_STATUS_DATA_BUFF_ERR | \ DTD_STATUS_TRANSACTION_ERR)/* Alignment requirements; must be a power of two */ ==============================
Quoted 2 lines
The "^Z" is a 0-character in the file and is not present in the original source tree, only in the copy.
Is it always a zero, or other invalids characters? (The (number of) changed bits might tell something.)
Quoted 3 lines
Similar errors happened during copying large files on the same machine but it seems it's enough to trigger if the total amount of data read is large enough.
Does "large enough" mean "large enough so that they are not in the file cache"?
All caches and your memory are ECC protected, so I think it is unlikely that the problem is with these. If I had to guess, I'd point to your disk (firmware) or the SATA controller. (A bad or loose SATA cable would throw CRC errors into the kernel log. Are there any?)
What is the exact offset of the changed byte in the file? (It might be at a cacheline, sector, or page boundary.)
Quoted 1 line
Does anyone know whether it's a known problem in AMD FX CPUs?
http://support.amd.com/us/Processor_TechDocs/48063_15h_Mod_00h-0Fh_Rev_Guide.pdf
Regards,
Clemens
2012-06-11 09:52 keltezéssel, Clemens Ladisch írta:
Quoted 30 lines
Quoted 29 lines · Boszormenyi Zoltan
I have an AMD FX-8120 boxed CPU in an ASUS M5A99X-EVO mainboard with 32GB DDR3/1600 memory, running Fedora 17, upgraded from 16.
I get occasional crashes and signal 11 during kernel compilation even with single-job make. Sometimes the compiler jumps out with a strange error message, like "stray \NNN character in the source". When re-running make, the error doesn't happen in the same file and the source file doesn't contain the character being complained about when inspecting with an editor or hexdump.
Now, a few minutes ago I was able to catch this bug when I copied the kernel GIT tree to apply a patch manually and did "git commit -a". Strangely, the commit contained one extra file that I didn't touch. git diff showed this for the extra file:
==============================
drivers/usb/gadget/fsl_usb2_udc.h +1
-1@@ -427,7 +427,7 @@ struct ep_td_struct {#define DTD_ADDR_MASK 0xFFFFFFE0#define DTD_PACKET_SIZE 0x7FFF0000#define DTD_LENGTH_BIT_POS 16 -#define DTD_ERROR_MASK (DTD_STATUS_HALTED | \ +#define DTD_ERROR_MASK (DTD_STATUS_HALTED | ^ZDTD_STATUS_DATA_BUFF_ERR | \ DTD_STATUS_TRANSACTION_ERR)/* Alignment requirements; must be a power of two */ ==============================
Quoted 2 lines
Quoted 2 lines
The "^Z" is a 0-character in the file and is not present in the original source tree, only in the copy.
Actually, the "^Z" there is 0x1a. It should be 0x5c, the backslash character.
Quoted 2 lines
Is it always a zero, or other invalids characters? (The (number of) changed bits might tell something.)
IIRC, GCC has a different error for a 0-character and "stray \NNN character" (that's not inside a string literal) and both happened at some time. Sorry, I didn't bother to make a note of the error messages.
Quoted 8 lines
Quoted 4 lines
Similar errors happened during copying large files on the same machine but it seems it's enough to trigger if the total amount of data read is large enough.
Does "large enough" mean "large enough so that they are not in the file cache"?
All caches and your memory are ECC protected,
Unfortunately the memory is not with ECC. "Large enough" means it's usually not in file system cache
Quoted 4 lines
so I think it is unlikely
that the problem is with these. If I had to guess, I'd point to your disk (firmware) or the SATA controller. (A bad or loose SATA cable would throw CRC errors into the kernel log. Are there any?)
The disks (8 of them) are attached to 3ware 9650SE-8LPML in RAID10. tw_cli reports no problems.
Quoted 2 lines
What is the exact offset of the changed byte in the file? (It might be at a cacheline, sector, or page boundary.)
The bad character is at offset 0x4b74.
Quoted 2 lines
Quoted 1 line
Does anyone know whether it's a known problem in AMD FX CPUs?
http://support.amd.com/us/Processor_TechDocs/48063_15h_Mod_00h-0Fh_Rev_Guide.pdf
Thanks but I have seen this file already. The "no fix planned" for every errata is saddening...
Quoted 5 lines
Regards,
Clemens
(leaving in the full text)
Quoted 5 lines · Boszormenyi Zoltan
Hi,
I have an AMD FX-8120 boxed CPU in an ASUS M5A99X-EVO mainboard with 32GB DDR3/1600 memory, running Fedora 17, upgraded from 16. memtest86+ show no problems.
Did you have the same issue with Fedora 16? Also, could you test with another distro whether the same thing happens?
Quoted 4 lines
Still, I get occasional crashes and signal 11 during kernel compilation even with single-job make. Sometimes the compiler jumps out with a strange error message, like "stray \NNN character in the source".
Is that the same ^Z character as below? Is that character with ascii number \026? Or do you get different characters each time?
Quoted 31 lines
When re-running
make, the error doesn't happen in the same file and the source file doesn't contain the character being complained about when inspecting with an editor or hexdump.Now, a few minutes ago I was able to catch this bug when I copied the kernel GIT tree to apply a patch manually and did "git commit -a". Strangely, the commit contained one extra file that I didn't touch. git diff showed this for the extra file:
==============================
drivers/usb/gadget/fsl_usb2_udc.h +1
-1@@ -427,7 +427,7 @@ struct ep_td_struct {#define DTD_ADDR_MASK 0xFFFFFFE0#define DTD_PACKET_SIZE 0x7FFF0000#define DTD_LENGTH_BIT_POS 16 -#define DTD_ERROR_MASK (DTD_STATUS_HALTED | \ +#define DTD_ERROR_MASK (DTD_STATUS_HALTED | ^ZDTD_STATUS_DATA_BUFF_ERR | \ DTD_STATUS_TRANSACTION_ERR)/* Alignment requirements; must be a power of two */ ==============================
The "^Z" is a 0-character in the file and is not present in the original source tree, only in the copy.
Similar errors happened during copying large files on the same machine but it seems it's enough to trigger if the total amount of data read is large enough.
Quoted 5 lines
The mainboard has the latest (UEFI) firmware flashed which contains the latest AMD microcode, so
microcode_ctldoesn't need to apply it anymore. Previously, I used amd-ucode-2012-01-17.tar from www.amd64.org/support/microcode.html which is now part ofmicrocode_ctlin Fedora.
Can you send /proc/cpuinfo?
Also, a dmesg from a recent kernel?
Quoted 6 lines
Since the error happens during compiling a source file and not only copying, the bug seems to happens during *reading* data.
Does anyone know whether it's a known problem in AMD FX CPUs? Does AMD have a newer microcode to fix this bug, or should I apply for warranty?
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 4 lines · Borislav Petkov
Quoted 3 lines · Boszormenyi Zoltan
I have an AMD FX-8120 boxed CPU in an ASUS M5A99X-EVO mainboard with 32GB DDR3/1600 memory, running Fedora 17, upgraded from 16. memtest86+ show no problems.
Ohe other thing: if there's an option in the BIOS to disable the IOMMU, can you do that and try reproducing the issue with IOMMU disabled?
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 9 lines · Boszormenyi Zoltan
2012-06-11 09:52 keltezéssel, Clemens Ladisch írta:
Quoted 7 lines
Quoted 3 lines
Similar errors happened during copying large files on the same machine but it seems it's enough to trigger if the total amount of data read is large enough.
Does "large enough" mean "large enough so that they are not in the file cache"?
"Large enough" means it's usually not in file system cache
If you could see a change while it's in the cache, you could rule out the disks.
Quoted 3 lines
Quoted 1 line
All caches and your memory are ECC protected,
Unfortunately the memory is not with ECC.
Sorry, I misread your mail.
This means that you cannot rule out bad memory.
Quoted 7 lines
Quoted 4 lines
so I think it is unlikely
that the problem is with these. If I had to guess, I'd point to your disk (firmware) or the SATA controller. (A bad or loose SATA cable would throw CRC errors into the kernel log. Are there any?)The disks (8 of them) are attached to 3ware 9650SE-8LPML in RAID10.
tw_clireports no problems.
Could you check whether the same happens with some disk connected to the on-board SATA controller? Or while copying around lots of data inside a RAM disk?
Quoted 4 lines
Quoted 2 lines
What is the exact offset of the changed byte in the file? (It might be at a cacheline, sector, or page boundary.)
The bad character is at offset
0x4b74.
That's completely random, i.e., probably an hardware error.
Quoted 3 lines
The "no fix planned" for every errata is saddening...
It's good news, because none of them actually matter.
Regards,
Clemens
2012-06-11 12:21 keltezéssel, Clemens Ladisch írta:
Quoted 5 lines
All caches and your memory are ECC protected,
Quoted 1 line
Unfortunately the memory is not with ECC.
Sorry, I misread your mail.
This means that you cannot rule out bad memory.
And there were two storms recently with lightings that struck nearby. :-( I will retest with memtest86+.
Thanks for everyone who replied.
Best regards,
Zoltán Böszörményi
Quoted 8 lines · Borislav Petkov
Quoted 4 lines · Borislav Petkov
Quoted 3 lines · Boszormenyi Zoltan
I have an AMD FX-8120 boxed CPU in an ASUS M5A99X-EVO mainboard with 32GB DDR3/1600 memory, running Fedora 17, upgraded from 16. memtest86+ show no problems.
Ohe other thing: if there's an option in the BIOS to disable the IOMMU, can you do that and try reproducing the issue with IOMMU disabled?
Maybe not related, but I had bad memory in my Intel Core-i5 based system some months ago which resulted in rare crashes, usually manifested itself as g++ ICEs when compiling a mid-sized C++ project -- compiling a kernel with make -p4 showed no problem. Also memtest86+ didn't show the issue,
so I tried memtest86-4.0a which claims to find more errors due to SMP support. An overnight run left me with a screen full of garbage and a crashed memtest86-4.0. I replaced the RAM anyway and the box was stable since then.
memtest86-4.0a is at
http://memtest86.com/
The page claims:
With a single CPU it is not possible to drive multi-channel memory controllers at full speed making it impossible to detect some types of errors
Maybe someone knowledgable could comment if this is true.
Johannes
2012-06-11 13:05 keltezéssel, Johannes Stezenbach írta:
Quoted 18 lines · Borislav Petkov
Quoted 7 lines · Borislav Petkov
Quoted 4 lines · Boszormenyi Zoltan
Quoted 3 lines
I have an AMD FX-8120 boxed CPU in an ASUS M5A99X-EVO mainboard with 32GB DDR3/1600 memory, running Fedora 17, upgraded from 16. memtest86+ show no problems.
Ohe other thing: if there's an option in the BIOS to disable the IOMMU, can you do that and try reproducing the issue with IOMMU disabled?
Maybe not related, but I had bad memory in my Intel Core-i5 based system some months ago which resulted in rare crashes, usually manifested itself as g++ ICEs when compiling a mid-sized C++ project -- compiling a kernel with make -p4 showed no problem. Also memtest86+ didn't show the issue,
so I tried memtest86-4.0a which claims to find more errors due to SMP support. An overnight run left me with a screen full of garbage and a crashed memtest86-4.0. I replaced the RAM anyway and the box was stable since then.
Quoted 8 lines
memtest86-4.0a is at
http://memtest86.com/The page claims:
With a single CPU it is not possible to drive multi-channel memory controllers at full speed making it impossible to detect some types of errorsMaybe someone knowledgable could comment if this is true.
This one locked up on my machine but memtest86+ 4.20 detected 12 different addresses with faulty bits in the lower 16GB. Applying for warranty.
With only two modules, "make -j8" succeeded a lot of times.
Thanks for everyone who tried to help.
Best regards,
Zoltán Böszörményi
Quoted 3 lines · Boszormenyi Zoltan
This one locked up on my machine but memtest86+ 4.20 detected 12 different addresses with faulty bits in the lower 16GB. Applying for warranty.
If you have multiple DIMMs, you can also take out the DIMM which contains the 16GB and rerun memtest to confirm it really is the culprit.
Signature · 2 lines
Regards/Gruss,
Boris.2012-06-13 17:57 keltezéssel, Borislav Petkov írta:
Quoted 7 lines · Boszormenyi Zoltan
Quoted 3 lines
This one locked up on my machine but memtest86+ 4.20 detected 12 different addresses with faulty bits in the lower 16GB. Applying for warranty.
If you have multiple DIMMs, you can also take out the DIMM which contains the 16GB and rerun memtest to confirm it really is the culprit.
I did exactly that, the remaining two 8GB modules don't have faults according to memtest86+ and the machine is stable with "make -j8" in the kernel tree. Neither thunderbird nor firefox crashed for a day, these are the usual victims when hitting the bad memory address.
Quoted 4 lines · Boszormenyi Zoltan
I did exactly that, the remaining two 8GB modules don't have faults according to memtest86+ and the machine is stable with "make -j8" in the kernel tree. Neither thunderbird nor firefox crashed for a day, these are the usual victims when hitting the bad memory address.
Cool.
I guess you could try to limit it even further by taking a known-good 8GB module and pairing it with one of the "bad" ones to see whether one of the "bad" 8GB modules is faulty or both of them are.
Or you could stop wasting time and go buy two new ones :-)
Signature · 2 lines
Regards/Gruss,
Boris.2012-06-14 00:06 keltezéssel, Borislav Petkov írta:
Quoted 5 lines · Boszormenyi Zoltan
Quoted 4 lines
I did exactly that, the remaining two 8GB modules don't have faults according to memtest86+ and the machine is stable with "make -j8" in the kernel tree. Neither thunderbird nor firefox crashed for a day, these are the usual victims when hitting the bad memory address.
Quoted 7 lines
Cool.
I guess you could try to limit it even further by taking a known-good 8GB module and pairing it with one of the "bad" ones to see whether one of the "bad" 8GB modules is faulty or both of them are.
Or you could stop wasting time and go buy two new ones :-)
Yesterday I took the bad pair back to the shop with a screenshot showing memtest86+ results and they accepted it for warranty. I will get a new pair or RAMs for no extra fee.