AMD64 Machine hardlocks when using memset
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Philip Lawatsch
I've now tried the most conservative settings available. The 32 bit kernel now hangs after about 150000 Iterations (compared to about 16000 before) but the 64 bit kernel still hangs after about 5000.
I'm still seeing this on my system as well, using the most conservative timings possible (DDR200, all delay parameters except the refresh time set to the largest possible value) as well as DDR333 with the same timings and DDR400 with everything set to auto. I also tried the kernel on the Fedora Core 3 rescue disc (same crash) and in single user mode (same crash).
So far, the crashes have consisted of either a hang, reboot or panic. One panic was a "spinlock already locked at kernel/module.c:2022" error. The other one is below, for what it's worth:
Mar 31 18:55:43 Newcastle kernel: Unable to handle kernel paging request at ffff8100588f5000 RIP:
Mar 31 18:55:43 Newcastle kernel: <ffffffff80236ac7>{clear_page+7}
Mar 31 18:55:43 Newcastle kernel: PGD 8063 PUD a063 PMD 0
Mar 31 18:55:43 Newcastle kernel: Oops: 0002 [1]
Mar 31 18:55:43 Newcastle kernel: CPU 0
Mar 31 18:55:43 Newcastle kernel: Modules linked in: md5(U) ipv6(U) parport_pc(U) lp(U) parport(U) autofs4(U) it87(U) i2c_sensor(U) i2c_isa(U) i2c_dev(U) i2c_core(U) sunrpc(U) pcmcia(U) yenta_socket(U) rsrc_nonstatic(U) pcmcia_core(U) joydev(U) nls_utf8(U) ntfs(U) vfat(U) fat(U) dm_mod(U) video(U) button(U) battery(U) ac(U) usb_storage(U) ohci1394(U) ieee1394(U) ohci_hcd(U) ehci_hcd(U) snd_ice1724(U) snd_ice17xx_ak4xxx(U) snd_ac97_codec(U) snd_pcm_oss(U) snd_mixer_oss(U) snd_pcm(U) snd_timer(U) snd_page_alloc(U) snd_ak4xxx_adda(U) snd_mpu401_uart(U) snd_rawmidi(U) snd_seq_device(U) snd(U) soundcore(U) forcedeth(U) floppy(U) ext3(U) jbd(U) sata_nv(U) libata(U) sd_mod(U) scsi_mod(U)
Mar 31 18:55:43 Newcastle kernel: Pid: 4928, comm: crashtest Not tainted 2.6.11-1.7_FC3custom
Mar 31 18:55:43 Newcastle kernel: RIP: 0010:[<ffffffff80236ac7>] <ffffffff80236ac7>{clear_page+7}
Mar 31 18:55:43 Newcastle kernel: RSP: 0000:ffff810078299ca0 EFLAGS: 00010246
Mar 31 18:55:43 Newcastle kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000200
Mar 31 18:55:43 Newcastle kernel: RDX: ffffffff80478940 RSI: 0000000000000000 RDI: ffff8100588f5000
Mar 31 18:55:43 Newcastle kernel: RBP: ffff81000235f5d0 R08: 0000000000000000 R09: 0000000000000000
Mar 31 18:55:43 Newcastle kernel: R10: 00000000000552fa R11: 0000000000000000 R12: ffff810000000000
Mar 31 18:55:43 Newcastle kernel: R13: ffff81000235f598 R14: 6db6db6db6db6db7 R15: 0000000000000000
Mar 31 18:55:43 Newcastle kernel: FS: 00002aaaaaabeb00(0000) GS:ffffffff80552300(0000) knlGS:0000000000000000
Mar 31 18:55:43 Newcastle kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:55:43 Newcastle kernel: CR2: ffff8100588f5000 CR3: 00000000791b0000 CR4: 00000000000006e0
Mar 31 18:55:43 Newcastle kernel: Process crashtest (pid: 4928, threadinfo ffff810078298000, task ffff8100788d67e0)
Mar 31 18:55:43 Newcastle kernel: Stack: ffffffff80170bc2 0000000000000019 0000000000000286 0000000000000000
Mar 31 18:55:43 Newcastle kernel: 000000000000000a 000080d20000000a 0000000000000286 0000000000000256
Mar 31 18:55:43 Newcastle kernel: ffffffff80478bc0 0000000000000000
Mar 31 18:55:43 Newcastle kernel: Call Trace:<ffffffff80170bc2>{buffered_rmqueue+1154} <ffffffff80170dac>{__alloc_pages+220}
Mar 31 18:55:43 Newcastle kernel: <ffffffff80181c52>{do_no_page+370} <ffffffff801825c0>{handle_mm_fault+560}
Mar 31 18:55:43 Newcastle kernel: <ffffffff80284f9c>{write_chan+860} <ffffffff80123834>{do_page_fault+1044}
Mar 31 18:55:43 Newcastle kernel: <ffffffff803a3699>{thread_return+41} <ffffffff8010f58d>{error_exit+0}
Mar 31 18:55:43 Newcastle kernel:
Mar 31 18:55:43 Newcastle kernel:
Mar 31 18:55:43 Newcastle kernel: Code: f3 48 ab c3 66 66 66 90 66 66 66 90 66 66 66 90 66 66 66 90
Mar 31 18:55:43 Newcastle kernel: RIP <ffffffff80236ac7>{clear_page+7} RSP <ffff810078299ca0>
Mar 31 18:55:43 Newcastle kernel: CR2: ffff8100588f5000Signature · 3 lines
Robert Hancock Saskatoon, SK, Canada To email, remove "nospam" from [email protected] Home Page: http://www.roberthancock.com/
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Ray Lee
Quoted 1 line · Robert Hancock
This is getting pretty ridiculous.. I've tried memory timings down to the slowest possible, ran Memtest86 for 4 passes with no errors, and it's been stable in Windows for a few months now. Still something is blowing up in Linux with this test though..
Have you run the same memset test under windows?
I've traced a lot of oddball problems down to bad or marginal power supplies.
I've now built a similar test program for Windows. I've let it run over 2000 iterations of 512MB memsets with no problems. On Linux it usually blew up with under 200 iterations. It does run visibly slower than the Linux version though - this is after all 32 bit Windows and it was compiled with crufty old Visual C++ 6.0 so it is probably not that optimized for this CPU. I will see if I can get a more optimized build of this to try in Mingw32 or something.. after all if it's related to some instruction combination or something it may not show up in the build I have.
Signature · 3 lines
Robert Hancock Saskatoon, SK, Canada To email, remove "nospam" from [email protected] Home Page: http://www.roberthancock.com/
Quoted 1 line · Robert
It does run visibly slower
The x86_64 memset(), both in user space and the kernel, for whatever gcc I have, and for a current kernel, uses the "repz stos" or "rep stosq" prefixed instruction for the bulk of the copy. This combination is a long running, interruptible Intel string instruction that loops on itself until the CX register decrements to zero.
Was your windows app using "stos"?
I'll wager a nickel that the actual crash you see comes when the processor has to handle an interrupt while in the middle of this instruction.
I'll wager a dime it's hardware, though interrupt activity may be required to provoke it.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373, 1.925.600.0401Quoted 6 lines · Paul Jackson
The
x86_64memset(), both in user space and the kernel, for whatever gcc I have, and for a current kernel, uses the "repz stos" or "rep stosq" prefixed instruction for the bulk of the copy. This combination is a long running, interruptible Intel string instruction that loops on itself until the CX register decrements to zero.
Quoted 8 lines
Was your windows app using "stos"?
I'll wager a nickel that the actual crash you see comes when the processor has to handle an interrupt while in the middle of this instruction.
I'll wager a dime it's hardware, though interrupt activity may be required to provoke it.
I ended up making a test program which essentially did the same thing except not using memset (just moving an int* up repeatedly and setting the value there to 0). That worked fine on both Windows and Linux. I then tried such a program using a long* compiled as 64-bit on Linux, that also worked fine. It seems like I can only reproduce it when memset is actually used..
I don't remember exactly what the Windows memset was using, that was on my work machine - it was inline assembly though, and I do know that it had only one instruction for the whole set, so it was likely "repz stos" or something similar to that.
As it turns out, the memset in my version of glibc x86_64 is not using such a string instruction though - it seems to be using two different sets of instructions depending on the size of the memset (not sure exactly how they're calculating the threshold between these..) For sizes below the treshold, this is the inner loop - it's using normal mov instructions:
3: /* Copy 64 bytes. */
mov %r8,(%rcx) mov %r8,0x8(%rcx) mov %r8,0x10(%rcx) mov %r8,0x18(%rcx) mov %r8,0x20(%rcx) mov %r8,0x28(%rcx) mov %r8,0x30(%rcx) mov %r8,0x38(%rcx) add $0x40,%rcx dec %rax jne 3b
For sizes above the threshold though, this is the inner loop. It's using movnti which is an SSE cache-bypasssing store:
11: /* Copy 64 bytes without polluting the cache. */
/* We could use movntdq %xmm0,(%rcx) here to further speed up for large cases but let's not use XMM registers. */ movnti %r8,(%rcx) movnti %r8,0x8(%rcx) movnti %r8,0x10(%rcx) movnti %r8,0x18(%rcx) movnti %r8,0x20(%rcx) movnti %r8,0x28(%rcx) movnti %r8,0x30(%rcx) movnti %r8,0x38(%rcx) add $0x40,%rcx dec %rax jne 11b
I'm wondering if one does a ton of these cache-bypassing stores whether something gets hosed because of that. Not sure what that could be though. I don't imagine the chipset is involved with any of that on the Athlon 64 - either the CPU or RAM seems the most likely suspect to me
Signature · 3 lines
Robert Hancock Saskatoon, SK, Canada To email, remove "nospam" from [email protected] Home Page: http://www.roberthancock.com/
Quoted 4 lines · Robert Hancock
I'm wondering if one does a ton of these cache-bypassing stores whether something gets hosed because of that. Not sure what that could be though. I don't imagine the chipset is involved with any of that on the Athlon 64 - either the CPU or RAM seems the most likely suspect to me
The glibc version is essentially the "perfect" copy function for the CPU. If you have any bus/memory problems or chipset bugs it will bite you.
Alan
[disregard my previous mail. I should have read the whole thread first]
Quoted 24 lines · Robert Hancock
As it turns out, the memset in my version of glibc
x86_64is not using such a string instruction though - it seems to be using two different sets of instructions depending on the size of the memset (not sure exactly how they're calculating the threshold between these..) For sizes below the treshold, this is the inner loop - it's using normal mov instructions:3: /* Copy 64 bytes. */
mov %r8,(%rcx) mov %r8,0x8(%rcx) mov %r8,0x10(%rcx) mov %r8,0x18(%rcx) mov %r8,0x20(%rcx) mov %r8,0x28(%rcx) mov %r8,0x30(%rcx) mov %r8,0x38(%rcx) add $0x40,%rcx dec %rax jne 3bFor sizes above the threshold though, this is the inner loop. It's using movnti which is an SSE cache-bypasssing store:
11: /* Copy 64 bytes without polluting the cache. */
Quoted 13 lines
/* We could use movntdq %xmm0,(%rcx) here to further speed up for large cases but let's not use XMM registers. */ movnti %r8,(%rcx) movnti %r8,0x8(%rcx) movnti %r8,0x10(%rcx) movnti %r8,0x18(%rcx) movnti %r8,0x20(%rcx) movnti %r8,0x28(%rcx) movnti %r8,0x30(%rcx) movnti %r8,0x38(%rcx) add $0x40,%rcx dec %rax jne 11b
This is a very rarely used instruction. People either do plain old rep stosl or do 3DNOW or SSE2 non-temporal stores.
Maybe movnti is different (buggy?) in subtle way.
Does it blow up if you use 3DNOW or SSE2 non-temporal stores?
If yes, then try different BIOS (not nesessarily latest is best). BTW, 'Athlon bug' was tracked down similarly. New BIOS enabled buggy chipset feature - BOOM! non-temporals killed the box (took several months to figure it out back then).
--
vda