data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!
Hi.
Perhaps some of you have read my older two threads:
http://marc.theaimsgroup.com/?t=116312440000001&r=1&w=2 and the even older http://marc.theaimsgroup.com/?t=116291314500001&r=1&w=2
The issue was basically the following:
I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following: I have about 30GB of testing data on my harddisk,... I repeat verifying sha512 sums on these files and check if errors occur.
One test pass verifies the 30GB 50 times,... about one to four differences are found in each pass.
The corrupted data is not one single completely wrong block of data or so,.. but if you look at the area of the file where differences are found,.. than some bytes are ok,.. some are wrong,.. and so on (seems to be randomly).
Also, there seems to be no event that triggers the corruption,.. it seems to be randomly, too.
It is really definitely not a harware issue (see my old threads my emails to Tyan/Hitachi and my "workaround" below. My system isn't overclocked.
My System:
Mainboard: Tyan S2895
Chipsets: Nvidia nforce professional 2200 and 2050 and AMD 8131
CPU: 2x DualCore Opterons model 275
RAM: 4GB Kingston Registered/ECC
Diskdrives: IBM/Hitachi: 1 PATA, 2 SATA
The data corruption error occurs on all drives.
You might have a look at the emails between me and Tyan and Hitachi,.. they contain probalby lots of valuable information (especially my different tests).
Some days ago,.. an engineer of Tyan suggested me to boot the kernel with mem=3072M.
When doing this,.. the issue did not occur (I don't want to say it was solved. Why? See my last emails to Tyan!)
Then he suggested me to disable the memory hole mapping in the BIOS,... When doing so,.. the error doesn't occur, too.
But I loose about 2GB RAM,.. and,.. more important,.. I cant believe that this is responsible for the whole issue. I don't consider it a solution but more a poor workaround which perhaps only by fortune solves the issue (Why? See my last eMails to Tyan ;) )
So I'd like to ask you if you perhaps could read the current information in this and previous mails,.. and tell me your opinions.
It is very likely that a large number of users suffer from this error (namely all Nvidia chipset users) but only few (there are some,.. I found most of them in the Nvidia forums,.. and they have exactly the same issue) identify this as an error because it's so rare.
Perhaps someone have an idea why disabling the memhole mapping solves it. I've always thought that memhole mapping just moves some address space to higher addreses to avoid the conflict between address space for PCI devices and address space for pyhsical memory.
But this should be just a simple addition and not solve this obviously complex error.
Lots of thanks in advance.
Best wishes,
Chris.
#########################################################################
### email #1 to Tyan/Hitachi ###
#########################################################################
(sorry for reposting but the AMD support system requires to add some keywords in the subject, and I wanted to have the correct subject for all other parties (Tyan and Hitachi) too, so that CC'ing would be possible for all.
Hi.
I provide this information to:
- Tyan ([email protected]) - Mr. Rodger Dusatko
- Hitachi ([email protected] , please add the GST Support Request #627-602-082-5 in the subject) Mr. Schledz)
- and with this email for the first time to AMD [email protected] (for the AMD people: please have a look at the information at the very end of this email first,... there you'll find links where you can read the introduction and description about the whole issue).
It might be useful if you contact each other (and especially nvidia which I wasn't able to contact myself),.. but please CC me in all you communications.
Also, please forward my emails/information to your responsible technical engineers and developers.
Please do not ignore this problem:
- it existing,
- several users are experiencing it (thus this is not a single failure of my system),
- it can cause severe data corruption (which is even more grave, as the a user won't notice it throught error messages) and
- it happens with different Operating Systems (at least Linux and Windows).
This is my current state of testing. For further information,.. please do not hesitate to ask.
You'll find old information (included in my previous mails or found at the linux-kernel mailinglist thread I've included in my mails) at the end.
- In the meantime I do not use diff any longer for my tests, simply because it takes much longer than to use sha512sums to verify dataintegrity (but this has not effect on the testing or the issue itself, it just proves that the error is not in the diff program).
- I always test 30GB insteat of 15
- As before I'm still very sure, that the following components are fully working and not damaged (see my old mails or lkml):
CPUs => due to extensive gimps/mprime torture tests
memory => due to extensive memtest86+ tests
harddisks => because I use three different disks (SATA-II and PATA) (but all from IBM/Hitachi or Hitachi) and I did extensive badblock scans temperature should be ok in my system => proper chassis (not any of the chep ones) with several fans, CPUs between 38 °C an 45°C, System ambient about 46°C, videocard, between 55° and 88°C (when under full 3D use),... the chipsetS (!) don't have temperature monitoring,.. and seem to be quite hot, but according Tyan this is normal.
Ok now my current state:
- I found (although it was difficult) a lot of resource in the internet where users report about the same or a very similar problem using the same hardware components. Some of them:
http://forums.nvidia.com/index.php?showtopic=8171&st=0
http://forums.nvidia.com/index.php?showtopic=18923
http://lkml.org/lkml/2006/8/14/42 (see http://lkml.org/lkml/2006/8/15/109)
Note that I've opened a thread at the nvidia forums myself:
http://forums.nvidia.com/index.php?showtopic=21576
All of them have in common, that the issue is/was not a hardware failure and it seems that none of them was able to reproduce the failure.
- As far as I understand the Tyan S2895 mainboard manual ftp://ftp.tyan.com/manuals/m_s2895_101.pdf on page 9,... both the IDE and SATA are connected to the Nvidia nforce professional 2200,.. so this may be nvidia related
(If anyone of you has the ability to contact nvidia,.. please do so and send them all my information (also the old one). It seems that it's not easily possible to contact them for "end-users")
- I tried different cable routings in my chassis (as far as this was possible) which did not solve the problem.
I also switched of all other devices in my rooms that might produce electro-magnetic disturbances....
thus electro-magnetic disturbances are unlikely.
- I found the errata for the AMD 8131 (which is on of my chipsets):
www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/26310.pdf Please have a look at it (all the issues) as some might be responsible for the whole issue.
- I tried to use older BIOS versions (1.02 and 1.03) but all of them gave me an OPROM error (at bus 12 device 06 function 1) and despite of that booting,.. the problem still exists.
According to Linux's dmesg this is:
12:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
12:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
- I tried with 8131 Errata 56 PCLK (which actualy disables the AMD Errata 56 fix) but the issue still exists
- I activated my BIOS's Spread Spectrum Option (although it is not described what it does).
Is this just for harddisks? And would I have to activate SpredSpectrum at the Hitachi Feature Tools, too, for having an effect?
- I tried everything with the lowest possible BIOS settings,.. which solved nothing.
- According to the information found in the threads at the nvidia boards (see the links above)... this may be a nvidia-Hitachi nvidia-Maxtor (and even other manufracturers of HDDs) related problem.
Some even claimed that deactivation of Native Command Queueing (NCQ) helped,.. BUT only for a limited time.
But as far as I know, Linux doesn't support NCQ at all (at the moment).
Thank you very much for now.
Best wishes,
Christoph Anton Mitterer.
----------------------------
Old information:
As a reminder my software/hardware data:
CPU: 2x DualCore Opteron 275
Mainboard: Tyan Thunder K8WE (S2895)
Chipsets: Nvidia nForce professional 2200, Nvidia nForce professional 2050, AMD 8131
Memory: Kingston ValueRAM 4x 1GB Registered ECC
Harddisks: 1x PATA IBM/Hitachi, 2x SATA IBM/Hitachi Additional Devices/Drives: Plextor PX760A DVD/CD, TerraTec Aureon 7.1 Universe soundcard, Hauppage Nova 500T DualDVB-T card.
Distribution: Debian sid
Kernel: self-compiled 2.6.18.2 (see below for .config) with applied EDAC patches
The system should be cooled enough so I don't think that this comes from overheating issues. Nothing is overclocked.
The issue was:
For an in depth description of the problem please have a look at the linux-kernel mailing list.
The full, current thread:
http://marc.theaimsgroup.com/?t=116312440000001&r=1&w=2
(You'll find there my system specs, too.)
An older thread with the same problem, but where I thougt the problem is FAT32 related (anyway, might be interesting, too):
http://marc.theaimsgroup.com/?t=116291314500001&r=1&w=2
#########################################################################
### email #2 to Tyan/Hitachi ###
#########################################################################
Hi Mr. Dusatko, hi Mr. Ebner.
Quoted 2 lines · Rodger Dusatko - Tyan
Quoted 1 line
Thanks so much for your e-mail.
Well I have to thank you for your support, too :-)
Quoted 3 lines
Quoted 2 lines
You seem to have tried many different tests.
Dozens ^^
Quoted 4 lines
Quoted 3 lines
If I understand the problem correctly, when you use the on-board SATA, you are receiving corrupted data.
It happens with the onboard IDE, too !!!! This sounds reasonable as both IDE and SATA are connected to the nforce 2200.
If you read the links to pages where other people report about the same problem (especially at the Nvidia forums) you'll see that others think that this is nforce related, too.
I established contact with some of them and most think that this may (but of course there is no definite proof for this) related to a nforce/disk manufracturer combination. So all of them report for example that the error occurs with nforce/Hitachi.
Some of them think that it might be NCQ related (did you read the NCQ related parts of my last email? As far as I can see NCQ will be added in the kernel for sata_nv in 2.6.19).
Both of this sounds somewhat strange to me...
For my understanding of computer technology I would assume that this should be a general harddisk error,.. and not only Hitachi or e.g. Maxtor related.
(I didn't test it with the onboard SCSI, as I don't have any SCSI drives.) Not that
Quoted 3 lines
Quoted 2 lines
Sometimes we have solved this problem by simply readjusting bios settings.
Does this mean that you were able to reproduce the problem?
Quoted 8 lines
Quoted 7 lines
Please try the following:
in the Linux boot prompt, please try (mem=3072M). This will show whether it might be a problem related to the memory hole.
or use only 2gb of memory.
I'm going to test this in a few minutes (althoug I think I did already a similar test)...
Anyway from a theoretical point of view it sounds very unlikely to me, that this is a memory related issue at all. Not only because of my memtest86+ test,.. but also because of the way the linux kernel works in that area.
Quoted 4 lines
Quoted 3 lines
If it is a memory hole problem, you should have (with Linux) the following settings:
My current memhole seetings are these (the ones that I use under "normal" production):
IOMMU -> enabled
IOMMU -> 64 MB
Memhole -> AUTO
mapping -> HARDWARE
Other memory settings
-> Node Memory Interleave -> enabled
-> Dram Bank Interleave -> enabled
-> MTTR Mapping -> discrete
-> Memory Hole
-> Memory Hole mapping -> enabled
-> Memory Config
-> Memory Clock DDR400
-> Swizzle memory banks enabled
Quoted 5 lines
Quoted 5 lines
CMOS reset (press CMOS Clear button for 20 seconds). Go into Bios -> Set all values to default (F9) Main -> Installed O/S -> Linux
Advanced -> Hammer Config
-> Node Memory Interleave -> disabled
Quoted 9 lines
Quoted 8 lines
-> Dram Bank Interleave -> disabled
-> MTTR Mapping -> discrete
-> Memory Hole
-> Memory Hole mapping -> Disabled
-> Memory Config
-> Memory Clock DDR333
-> Swizzle memory banks disabled
I've already checked excatly this setting ;) expect that I used DDR400,... could that make any difference?
Quoted 3 lines
Quoted 2 lines
You might try SATA HDDs from another manufacturer.
I'm already trying to do so but currently none of my friends was able to borrow me any devices,... I'm also going to check the issue with other operating systems (at least if I find any that support the Nvidia chipsets at all),.. maybe some *BSD or OpenSolaris.
Quoted 6 lines
Quoted 5 lines
Also, I have a newer beta bios version available.
ftp://ftp.tech2.de/boards/28xx/2895/BIOS/ -> 2895_1047.zip you might want to try.
Please don't understand me wrong,... I still would like you to help and investigate in that issue... but currently I think (although I may be wrong) that this could be harddisk firmware related.
So what _excatly_ did you change in that version,.. or is it just a crappy solution or workaround,...?
Any idea about that spread spectrum option?:
Quoted 6 lines
Quoted 5 lines
- I activated my BIOS's Spread Spectrum Option (although it is not described what it does).
Is this just for harddisks? And would I have to activate SpredSpectrum at the Hitachi Feature Tools, too, for having an effect?
Thanks so far.
Chris.
#########################################################################
### email #3 to Tyan/Hitachi ###
#########################################################################
Quoted 9 lines · Rodger Dusatko - Tyan
Quoted 8 lines
Hello Christoph,
another customer having data corruption problems said by entering the command mem=3072M he no longer has data corruption problems.
Please let me know as soon as possible, that I might know how to help further.
I just finished my test....
Used my "old" BIOS settings (not the one included in you mail)... but set mem=3072M.
It seems (although I'm not yet fully convinced as I've already had cases where an error occured after lots of sha512-passes) that with mem=3072M _no error occures_
But of course I get only 2GB ram (of my 4GB which I wanted to upgrad to even more memory in the next months).
So just to use mem=3072M is not acceptable.
And I must admit that I have strong concerns about the fact that memhole settings are a proper fix for that.
Of course I'd be glad if I could fix that... but from my own large system programming experience I know that there are many cases where a fix isn't really a fix for a problem,... but solves the problem in conjunction with other errors (that are not yet found).
I'd be glad if you could give me better explanation of the memhole-solution (and especially how to solve it without mem=3072M because I'd like to have my full memory) ... because I'd like to fully understand the issue to secure that it is really fixed or not.
I'll test you beta BIOS tomorrow and report my results.
If you whish I could also call you via phone (just give me your phone-#).
Thanks in advance,
Chris.
#########################################################################
### email #4 to Tyan/Hitachi ###
#########################################################################
One thing I forgot,...
Although using it very very rarely,.. there are some cases where I have to use M$ Windows.... and afaik,.. you cannot tell windows something like mem=3072M
So it wouldn't solve that for Windows.
Chris.
#########################################################################
### email #5 to Tyan/Hitachi ###
#########################################################################
Dear Mr. Dusatko, Mr. Ebner and dear sir at the Hitachi GST Support.
I'd like to give you my current status of the problem.
First of all AMD didn't even answer until now, the same applies for my request at Nvidias knowledge base,... says something about these companies I think.
For the people at Hitachi: With the advice of Mr. Dusatko from Tyan I was able to workaround the problem:
Quoted 4 lines · Rodger Dusatko - Tyan
Quoted 3 lines
as I mentioned earlier, you can do some of these memory hole settings : (for
Linux)
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Go into Bios -> Set all values to default (F9)
Main -> Installed O/S -> Linux
Advanced -> Hammer Config
-> Node Memory Interleave -> disabledQuoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-> Dram Bank Interleave -> disabled
-> MTTR Mapping -> discrete
-> Memory Hole
-> Memory Hole mapping -> Disabled
-> Memory Config
-> Memory Clock DDR333
-> Swizzle memory banks disabled
The above settings for the BIOS actually lead to a system that did not make any errors during one of my complete tests (that is verifying sha512sums 50 times on 30 GB of data).
Actually I seems to depend only on one of the above settings: Memory hole mapping.
Currently I'm using the following:
Main -> Installed O/S -> Linux
Advanced -> Hammer Config
-> Node Memory Interleave -> Auto
-> Dram Bank Interleave -> Auto
-> MTTR Mapping -> discrete
-> Memory Hole
-> Memory Hole mapping -> Disabled
-> Memory Config
-> Memory Clock->DDR400
->Swizzle memory banks -> Enabled
And still no error occurs.
But as soon as I set Memory Hole mapping to one of the other values (Auto, Hardware or Software),.. the error occurs. (Especially for Tyan: Note that when using Software Node Memory Interleave is always automatically set to Disabled after reboot, while when using Harware, Auto works - perhaps a bug?)
Ok,.. now you might think,... problem solved,.. but it is defenitely not:
1) Memory Hole mapping costs me 2GB of my 4GB RAM (which are unusable because of the memory hole),.. this is not really acceptable. The beta BIOS Mr. Dusatko from Tyan gave might solve this, but I wasn't able to test this yet.
2) But even it this would solve the problem I'm still very concerned and encourage especially the people at Hitachi to try to find another reason. Why? Because I cannot imagine how the memory hole leads to the wole issue:
- The memory hole is a quite simple process where the BIOS / Hardware remaps to some portions of physical RAM to higher areas,.. to give the lower areas to PCI devices that make uses of mmap.
Even if there would be an error,... that would not only affect IDE/SATA but also CD/DVD/SCSI drives and any other memory operations at all. AND there would be complete block that would be corrupted,.. not only several bytes (remember: I've reportet that in a currupted block some bytes are ok,.. some are note,... and so on).
-If you look at the board description (ftp://ftp.tyan.com/manuals/m_s2895_101.pdf page 9) you see that both IDE and SATA are connected to the nforce professional 2200, right? Why should the memhole settings affect only the IDE/SATA drives? If there was an error in the memory controller it would affect every memory operation in the system (see above) because the memory controller is not onboard,.. but integrated in the Operton CPUs. (This is also the reason why, if the memory controller would have design errors, not only people using nvidia chipsets have this problem,.. which is apparently the case.)
-Last but not least,.. (as also noted above) the errors are always like the following: not a complete block is corrupted but just perhaps half of all its bytes (in any order). Could this come from the simple memory hole remapping???? In my opinion, definitely not.
So I think "we" are not yet finished with work.
- I ask the Hitachi people to continue their work (or start with it ;) ) in taking a special look at their firmware and how it interoperates with nforce chipsets.
I found (really) lots of reports where people tells that this issue has been resolved by firmware upgrades of their vendor (especially for Maxtor devices).
Nvidia itself suggests this:
http://nvidia.custhelp.com/cgi-bin/nvidia.cfg/php/enduser/std_adp.php?p_faqid=768&p_created=1138923863&p_sid=9qSJ8Yni&p_accessibility=0&p_redirect=&p_lva=&p_sp=cF9zcmNoPSZwX3NvcnRfYnk9JnBfZ3JpZHNvcnQ9JnBfcm93X2NudD00MzImcF9wcm9kcz0mcF9jYXRzPSZwX3B2PSZwX2N2PSZwX3NlYXJjaF90eXBlPWFuc3dlcnMuc2VhcmNoX2ZubCZwX3BhZ2U9MQ**&p_li=&p_topview=1
(although they think that the issue appears only on SATA which is definitely not true)
Please have a detailed look on the NCQ of the drives:
This would be (according to how NCQ works) the most likely reason for the error,... and some people say that deactivating it under Windows, solved the issue. Anyway,... if NCQ was responsible for the error,.. it would not appear on the IDE drives (but it does).
And I'm not even sure if Linux/libata (until kernel 2.6.18.x) even uses NCQ. I always thought it would not but I might be wrong. See this part of my dmesg:sata_nv 0000:00:07.0: version 2.0
ACPI: PCI Interrupt Link [LTID] enabled at IRQ 22
GSI 18 sharing vector 0xD9 and IRQ 18
ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [LTID] -> GSI 22 (level, high) -> IRQ 217
PCI: Setting latency timer of device 0000:00:07.0 to 64
ata1: SATA max UDMA/133 cmd 0x1C40 ctl 0x1C36 bmdma 0x1C10 irq 217
ata2: SATA max UDMA/133 cmd 0x1C38 ctl 0x1C32 bmdma 0x1C18 irq 217 scsi0 : sata_nv
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata1.00: ATA-7, max UDMA/133, 488397168 sectors: LBA48 NCQ (depth 0/32) ata1.00: ata1: dev 0 multi count 16
ata1.00: configured for UDMA/133
scsi1 : sata_nv
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata2.00: ATA-7, max UDMA/133, 488397168 sectors: LBA48 NCQ (depth 0/32) ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/133
Vendor: ATA Model: HDT722525DLA380 Rev: V44O Type: Direct-Access ANSI SCSI revision: 05 Vendor: ATA Model: HDT722525DLA380 Rev: V44O Type: Direct-Access ANSI SCSI revision: 05
It says something about NCQ...
-It would also be great if Hitachi could inform me about their current progress or how long it will take until their engineers start to have a look at my issue.
Especially for Mr. Dusatko at tyan:
Quoted 9 lines
Quoted 8 lines
Just because memtest86 works it doesn't mean that the memory you are using is compatible memory. That is why we have a recommended list.
Each of the modules on our recommended list have been thoroughly tested. Most memories pass the memtest86 test, yet many of these do not past our tests.
my tel-nr.
My memory modules are actually on your compatible list (they're the Kingston KVR400D8R3A/1G) so this cannot be the point.
I still was not able to test your beta BIOS but I'll do so as soon as possible an report the results. And I'm going to call you this or next week (have to work at the Leibniz-Supercomputing Centre today and tomorrow,.. so don't know when I have enough time).
Thanks for now.
Best wishes,
Chris.
#########################################################################
### email #6 to Tyan/Hitachi ###
#########################################################################
Quoted 13 lines · Rodger Dusatko - Tyan
Quoted 13 lines
you mention:
Quoted 3 lines
Quoted 3 lines
My memory modules are actually on your compatible list (they're the Kingston KVR400D8R3A/1G) so this cannot be the point.
I have talked with so many customers about this very problem. Just because the part-nr. of the Kingston modules is correct, this means absolutely nothing.
Quoted 7 lines
Quoted 6 lines
You need to also have the same chips as on our recommended website. The chips being used are even more important than the kingston part-nr.
The chips on the KVR400D8R3A/1G must be Micron, having chip part-nr. MT46V64M8TG-5B D as shown on our recommended memory page.
I'll check this these days and inform you about the exact chips on the DIMMs
Anyway...
What do you say to the reasons why I don't think that the memhole stuff is a real solution but more a poor workaround (see my last email,.. which is attached below).
You didn't comment on my ideas in your last answer.
Quoted 5 lines
Quoted 4 lines
This is a grave problem with Kingston memory and why I would only recommend Kingston memory when your supplier is willing to help you to get the exact modules which we have tested.
Well are you absolutely sure that this is memory related? (See also my comments in my last email)
Note that lots of users were able to solve this via disk drive firmware upgrades and many of them didn't have Kingston RAMs. Also,... all RAMs "shoudl" be usable as all "should" follow the SDRAM standard...
If there would be a Kingston error,.. that data corruption issue should appear everywhere, shouldn't it? And not only on hard disk accesses.
In all doing respect, and please believe me that I truely respect your knowledge and so (because you surely know more about hardware because my computer science study goes more about theoretical stuff)... but I cannot believe that this is the simple reason,... "wrong RAMs wrong BIOS settings and you cannot use your full RAM" (see my reasons in my last email)...
I'd say that there is somewhere a real and perhaps grave error.... either on the board itself ot the nvidia chipset (which I suspect as the evil here ;-) ).
And I think the error is severe enought that there should be made a considerable effort to solve it, or at least, exactly locate where there error is, and why the memhole disabled solves it.
And remember,... it may be the case that the data corruption doesn't appear when UDMA (at PATA drives) is disabled,.. but this shouldn't have to do anything with memory vendor or memhole settings,... so why would this solve the issue, too (if it actually does which I cannot proove)?
I'm also going to start my test with changing the following BIOS settings:
SCSI Bus master from my current setting Enabled to Disabled Disk Access Mode (don't recall the actual name) from Other to DOS.
I'm going to report you the results next week,.. and I'll probably going to call you again.
Quoted 8 lines
Quoted 7 lines
Wiith ATP or other vendors, they stick usually to the same chips as long as the vendor part-nr is the same. In such a case, you probably would have been right when the vendor part-nr matches your part-nr.
The problems you are having, as I mentioned before, may disappear if you use memory on our recommended memory list.
Is it possible for Tyan to borrow me such memory for testing? I live in Munich and Tyan Germany is in Munich too, if I recall correctly.
Thanks in adc
Best wishes,
Chris.
#########################################################################
### email #7 to Tyan/Hitachi ###
#########################################################################
Sorry I forgot one thing:
The beta BIOS you gave me did not change anything.
As soon as I activate memhole mapping (either to software, hardware or auto),.. data corruption occurs.
Chris.
#########################################################################
### reply to #1 from Tyan ###
#########################################################################
Hello Chris,
there are often problems which are not really so easy to understand.
As I understand it, the hard disk uses DMA (Direct Memory Access), which is supported by the chipset.
The processor uses the DMA access to the DIMMs through the chipset to write to the disks.
Now, I really am not an expert on this, but normally the DMA is not used by the processor when communicating with the memory, but rather the hypertransport connection.
This may be an explanation of what is causing the problem. Because a driver for HDDs also exists, there may be different links where the problem is occuring.
The driver may be able to solve problems which can make it that even using the hardware setting for memory hole causes no problems. However, there are many different amd cpu steppings, all different in how they manage memory (and in this case, the memory hole). If the drivers take all of these considerations, they may be able to adjust according to the processor being used. But I am not sure if the people who write these drivers get involved with this.
Rodger
, the DMA s supported from the chipset uses the DMA access for
communicating with the processor, the memory
----- Original Message -----
...
...
...
That were all (important) emails so until now.
Quoted 7 lines · Christoph Anton Mitterer
The issue was basically the following:
I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following: I have about 30GB of testing data on my harddisk,... I repeat verifying sha512 sums on these files and check if errors occur.
One test pass verifies the 30GB 50 times,... about one to four differences are found in each pass.
Doh! I have a Tyan S2895 in my system, and I've been pulling my hair out trying to track down the cause of a similar somewhat rare failure for the pre-computer sha1 of a block of data to actually match the calculated sha1. I'd been hunting in vain the past few days trying to find a cause -- looking for buffer overflows, non thread safe code, or similar usual suspects.
It is a relief to see I am not alone!
-Erik
--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
Quoted 9 lines · Erik Andersen
Doh! I have a Tyan S2895 in my system, and I've been pulling my hair out trying to track down the cause of a similar somewhat rare failure for the pre-computer sha1 of a block of data to actually match the calculated sha1. I'd been hunting in vain the past few days trying to find a cause -- looking for buffer overflows, non thread safe code, or similar usual suspects.
It is a relief to see I am not alone!
^^
You might read my email and all links in it, etc. throughly,.. than you can try what I did.
Please inform me about all your results, and about your specific hardware (i.e. CPU type (with stepping and exact model), which harddisks and so on).
Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[email protected]
x-mozilla-html:TRUE
version:2.1
end:vcard
Quoted 4 lines · Christoph Anton Mitterer
I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following: I have about 30GB of testing data on my harddisk,... I repeat verifying sha512 sums on these files and check if errors occur.
Heh, I see this also with an Tyan S2866 (nforce4 chipset). I've been aware something is a miss for a while because if I transfer about 40GB of data from one machine to another there are checksum mismatches and some files have to be transfered again.
I've kept quite about it so far because it's not been clear what the cause is and because i can mostly ignore it now that I checksum all my data and check after xfer that it's sane (so I have 2+ copies of all this stuff everywhere).
Quoted 2 lines
One test pass verifies the 30GB 50 times,... about one to four differences are found in each pass.
Sounds about the same occurance rate I see, 30-40GB xfer, one or two pages (4K) are wrong.
Quoted 4 lines
The corrupted data is not one single completely wrong block of data or so,.. but if you look at the area of the file where differences are found,.. than some bytes are ok,.. some are wrong,.. and so on (seems to be randomly).
For me it seems that a single block in the file would be bad and the rest OK --- we I'm talking about 2 random blocks in 30BG or so.
Quoted 5 lines · Chris Wedgwood
Heh, I see this also with an Tyan S2866 (nforce4 chipset). I've been aware something is a miss for a while because if I transfer about 40GB of data from one machine to another there are checksum mismatches and some files have to be transfered again.
It seems that this may be occur on _all_ nvidia chipsets (of course I'm talking about mainboard chips ;) )
Which harddisk types to you use (vendor an interface type)
Quoted 5 lines
I've kept quite about it so far because it's not been clear what the cause is and because i can mostly ignore it now that I checksum all my data and check after xfer that it's sane (so I have 2+ copies of all this stuff everywhere).
I assume that a large number of users actually experience this error,.. but as it's so rare only few correctly identify it.
Most of them might think that its filesystem related or so.
Quoted 9 lines
Quoted 5 lines
The corrupted data is not one single completely wrong block of data or so,.. but if you look at the area of the file where differences are found,.. than some bytes are ok,.. some are wrong,.. and so on (seems to be randomly).
For me it seems that a single block in the file would be bad and the rest OK --- we I'm talking about 2 random blocks in 30BG or so.
Did you check this with an hex editor? I did it an while the errors were restricted to one "region" of a file.... it was not so that that region was completely corrupted but only some single bytes. Actually it was that mostly one bit was wrong,..
Chris.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Ville Herva
I saw something very similar with Via KT133 years ago. Then the culprit was botched PCI implementation that sometimes corrupted PCI transfers when there was heavy PCI I/O going on. Usually than meant running two disk transfers at the same time. Doing heavy network I/O at the time made it more likely happen.
Hm I do only on concurrent test,... and network is not used very much during the tests.
Quoted 3 lines
I used this crude hack:
http://v.iki.fi/~vherva/tmp/wrchk.c
I'll have a look at it :)
Quoted 7 lines
If the problem in your case is that the PCI transfer gets corrupted when it happens to a certain memory area, I guess you could try to binary search for the bad spot with the kernel BadRam patches
http://www.linuxjournal.com/article/4489 (I seem to recall it was possible to turn off memory areas with vanilla kernel boot params without a patch, but I can't find a reference.)
I know badram,.. but the thing is,.. that it's highly unlikely that my RAMs are damaged. Many hours of memtest86+ runs did not show any error (not even ECC errors),...
And why should memhol mapping disabled solve the issue if memory was damaged? That could only be if the badblocks would be in the address space used by the memhole....
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[email protected]
x-mozilla-html:TRUE
version:2.1
end:vcard
Hello Christoph!
Quoted 2 lines · Christoph Anton Mitterer
I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following: I have about 30GB of testing data on
This sounds very familiar! One of the Linux compute clusters I administer at work is a 336 node system consisting of the following components:
* 2x Dual-Core AMD Opteron 275
* Tyan S2891 mainboard
* Hitachi HDS728080PLA380 harddisk
* 4 GB RAM (some nodes have 8 GB) - intensively tested with memtest86+
* SUSE 9.3 x86_64 (kernel 2.6.11.4-21.14-smp) - But I've also e.g. tried the latest openSUSE 10.2 RC1+ kernel 2.6.18.2-33 which makes no difference.
We are running LS-Dyna on these machines and discovered a testcase which shows a similar data corruption. So I can confirm that the problem is for real an not a hardware defect of a single machine!
Here's a diff of a corrupted and a good file written during our testcase:
("-" == corrupted file, "+" == good file)
...
009f2ff0 67 2a 4c c4 6d 9d 34 44 ad e6 3c 45 05 9a 4d c4 |g*L.m.4D..<E..M.| -009f3000 39 60 e6 44 20 ab 46 44 56 aa 46 44 c2 35 e6 44 |9.D .FDV.FD.5.D| -009f3010 45 e1 48 44 88 3d 47 44 f3 81 e6 44 93 0b 46 44 |E.HD.=GD...D..FD| -009f3020 d4 eb 48 44 22 57 e6 44 3d 3d 48 44 ac 89 49 44 |..HD"W.D==HD..ID| -009f3030 00 8c e9 44 39 af 2d 44 e7 1b 8d 44 a8 6e e9 44 |...D9.-D...D.n.D| -009f3040 16 d4 2e 44 5e 12 8c 44 78 51 e9 44 c0 f5 2f 44 |...D^..DxQ.D../D| ... -009f3fd0 22 ae 4e 44 81 b5 ee 43 0c 8a df 44 8d e2 6b 44 |".ND...C...D..kD| -009f3fe0 6c a0 e8 43 b6 8f e9 44 22 ae 4e 44 55 e9 ed 43 |l..C...D".NDU..C| -009f3ff0 a8 b2 e0 44 78 7c 69 44 56 6f e8 43 5e b2 e0 44 |...Dx|iDVo.C^..D| +009f3000 1b 32 30 44 50 59 3d 45 a2 79 4e c4 66 6e 2f 44 |.20DPY=E.yN.fn/D| +009f3010 40 91 3d 45 d1 b6 4e c4 a1 6c 31 44 1b cb 3d 45 |@.=E..N..l1D..=E| +009f3020 0d f6 4e c4 57 7c 33 44 bf cb 3c 45 88 9a 4d c4 |..N.W|3D..<E..M.| +009f3030 79 e9 29 44 3a 10 3d 45 d3 e1 4d c4 17 28 2c 44 |y.)D:.=E..M..(,D| +009f3040 f6 50 3d 45 dc 25 4e c4 b6 50 2e 44 b3 4f 3c 45 |.P=E.%N..P.D.O<E| ... +009f3fd0 9c 70 6c 45 04 be 9f c3 fe fc 8f 44 ce 65 6c 45 |.plE.......D.elE| +009f3fe0 fc 56 9c c3 32 f7 90 44 e5 3c 6c 45 cd 79 9c c3 |.V..2..D.<lE.y..| +009f3ff0 f3 55 92 44 c1 10 6c 45 5e 12 a0 c3 60 31 93 44 |.U.D..lE^...1.D| 009f4000 88 cd 6b 45 c1 6d cd c3 00 a5 8b 44 f2 ac 6b 45 |..kE.m.....D..kE|
...
Please notice:
a) the corruption begins at a page boundary
b) the corrupted byte range is a single memory page and
c) almost every fourth byte is set to 0x44 in the corrupted case (but the other bytes changed, too)
To me this looks as if a wrong memory page got written into the file.
Quoted 1 line
From our testing I can also tell that the data corruption does
*not* appear at all when we are booting the nodes with mem=2G. However, when we are using all the 4GB the data corruption shows up - but not everytime and thus not on all nodes. Sometimes a node runs for ours without any problem. That's why we are testing on 32 nodes in parallel most of the time. I have the impression that it has something to do with physical memory layout of the running processes.
Please also notice that this is a silent data corruption. I.e. there are no error or warning messages in the kernel log or the mce log at all.
Christoph, I will carefully re-read your entire posting and the included links on Monday and will also try the memory hole setting.
If somebody has an explanation for this problem I can offer some of our compute nodes+time for testing because we really want to get this fixed as soon as possible.
Best regards,
Karsten
Signature · 1 line
Dipl.-Inf. Karsten Weiss - http://www.machineroom.de/knweiss
Quoted 10 lines · On Sat, 2 Dec 2006 12:00:36 +0100 (CET) Karsten Weiss <[email protected]> wrote:
Hello Christoph!
Quoted 2 lines · Christoph Anton Mitterer
I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following: I have about 30GB of testing data on
This sounds very familiar! One of the Linux compute clusters I administer at work is a 336 node system consisting of the following components:
See the thread http://lkml.org/lkml/2006/8/16/305
Quoted 2 lines · Alan
See the thread http://lkml.org/lkml/2006/8/16/305
Hi Alan.
Thanks for your reply. I've read this thread already some weeks ago.... but from my limited knowledge I understood, that this was an issue related to a SCSI adapter or so. Or did I understand this wrong. And as soon as he removed the card everything was fine.
I don't have any PCI SCSI cards,... (but I have an onboard LSI53C1030 controller).
The only cards I have are:
PCIe bus (two slots):
Asus Nividia 7800GTX based card
PCI bus: no card (one slot):
no card
PCI-X bus A (100MHz) (two slots):
Hauppauge Nova T 500 Dual DVB-T card (which is actually a "normal" PCI card,.. but should be compatible with PCI-X)
TerraTec Aureon 7.1 Universe Soundcard (which is actually a "normal" PCI card,.. but should be compatible with PCI-X)
PCI-X bus B (133MHz) (one slots):
no card
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[email protected]
x-mozilla-html:TRUE
version:2.1
end:vcard
Quoted 11 lines · Karsten Weiss
Here's a diff of a corrupted and a good file written during our testcase:
("-" == corrupted file, "+" == good file)
...
009f2ff067 2a 4c c4 6d 9d 34 44 ad e6 3c 45 05 9a 4d c4 |g*L.m.4D..<E..M.|
-009f3000 39 60 e6 44 20 ab 46 44 56 aa 46 44 c2 35 e6 44 |9.D .FDV.FD.5.D|
....
+009f3ff0 f3 55 92 44 c1 10 6c 45 5e 12 a0 c3 60 31 93 44 |.U.D..lE^...1.D|
009f400088 cd 6b 45 c1 6d cd c3 00 a5 8b 44 f2 ac 6b 45 |..kE.m.....D..kE|
Well as I told in my mails to the list I made the experience that not all bytes of the corrupted area are invalid,.. but only some,.. while it seems that in you diff ALL the bytes are wrong, right?
Quoted 10 lines
Please notice:
a) the corruption begins at a page boundary
b) the corrupted byte range is a single memory page and
c) almost every fourth byte is set to0x44in the corrupted case (but the other bytes changed, too)To me this looks as if a wrong memory page got written into the file.
Hmm and do you have any ideas what's the reason for all this? Defect in the nforce chipset? Or even in the CPU (the Opterons do have integrated memory controllers).
Quoted 9 lines
Quoted 1 line
From our testing I can also tell that the data corruption does
*not* appear at all when we are booting the nodes with mem=2G. However, when we are using all the 4GB the data corruption shows up - but not everytime and thus not on all nodes. Sometimes a node runs for ours without any problem. That's why we are testing on 32 nodes in parallel most of the time. I have the impression that it has something to do with physical memory layout of the running processes.
Hmm maybe,.. but I have absolutely no idea ;)
Quoted 4 lines
Please also notice that this is a silent data corruption. I.e. there are no error or warning messages in the kernel log or the mce log at all.
Yes I can confirm that.
Quoted 4 lines
Christoph, I will carefully re-read your entire posting and the included links on Monday and will also try the memory hole setting.
And did you get out anything new?
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[email protected]
x-mozilla-html:TRUE
version:2.1
end:vcard
Quoted 5 lines · Christoph Anton Mitterer
Quoted 4 lines
Christoph, I will carefully re-read your entire posting and the included links on Monday and will also try the memory hole setting.
And did you get out anything new?
As I already mentioned the kernel parameter "iommu=soft" fixes the data corruption for me. We saw no more data corruption during a test on 48 machines over the last week-end. Chris Wedgewood already confirmed that this setting fixed the data corruption for him, too.
Of course, the big question "Why does the hardware iommu *not* work on those machines?" still remains.
I have also tried setting "memory hole mapping" to "disabled" instead of "hardware" on some of the machines and this *seems* to work stable, too. However, I did only test it on about a dozen machines because this bios setting costs us 1 GB memory (and iommu=soft does not).
BTW: Maybe I should also mention that other machines types (e.g. the HP xw9300 dual opteron workstations) which also use a NVIDIA chipset and Opterons never had this problem as far as I know.
Best regards,
Karsten
Signature · 1 line
Dipl.-Inf. Karsten Weiss - http://www.machineroom.de/knweiss
Quoted 3 lines · Karsten Weiss
Of course, the big question "Why does the hardware iommu *not* work on those machines?" still remains.
I'm going to check AMDs errata docs these days,.. perhaps I find something that relates. But I'd ask you to do the same as I don't consider myself as an expert in these issues ;-)
Chris Wedgwood said that iommu isn't used unter windows at all,.. so I think the following three solutions would be possible:
- error in the Opteron (memory controller)
- error in the Nvidia chipsets
- error in the kernel
Quoted 6 lines
I have also tried setting "memory hole mapping" to "disabled" instead of "hardware" on some of the machines and this *seems* to work stable, too. However, I did only test it on about a dozen machines because this bios setting costs us 1 GB memory (and iommu=soft does not).
Yes... loosing so much memory is a big drawback,.. anyway it would be great if you can make some more extensive tests that we'd be able to say if memholemapping=disabled in the BIOS really solves that issue, too, or not.
Does anyone know how memhole mapping in the BIOS relates to the iommu stuff? Is it likely or explainable that both would sovle the issue?
Quoted 5 lines
BTW: Maybe I should also mention that other machines types (e.g. the HP xw9300 dual opteron workstations) which also use a NVIDIA chipset and Opterons never had this problem as far as I know.
Uhm,.. that's really strange,... I would have thought that this would affect all systems that uses either the (mayby) buggy nforce chipset,.. or the (mayby) buggy Opteron.
Did those systems have exactly the same Nvidia-Type? Same question for the CPU (perhaps the issue only occurs for a speciffic stepping) Again I have:
nforce professional 2200
nforce professional 2050
Opteron model 275 (stepping E6)
btw: I think that is already clear but again:
Both "solutions" solve the problem for me:
Either
- memhole mapping=disabled in the BIOS (but you loose some memory)
- without any iommu= option for the kernel
or
- memhole mapping=hardware in the BIOS (I suppuse it will work with software too)
- with iommu=soft for the kernel
Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[email protected]
x-mozilla-html:TRUE
version:2.1
end:vcard
Quoted 3 lines · Christoph Anton Mitterer
- error in the Opteron (memory controller)
- error in the Nvidia chipsets
- error in the kernel
My guess without further information would be that some, but not all BIOSes are doing some work to avoid this.
Does anyone have an amd64 with an nforce4 chipset and >4GB that does NOT have this problem? If so it might be worth chasing the BIOS vendors to see what errata they are dealing with.
Hi.
Some days ago I received the following message from "Sunny Days". I think he did not send it lkml so I forward it now:
Quoted 25 lines · Sunny Days
hello,
i have done some extensive testing on this.
various opterons, always single socket
various dimms 1 and 2gb modules
and hitachi+seagate disks with various firmwares and sizes but i am getting a diferent pattern in the corruption. My test file was 10gb.I have mapped the earliest corruption as low as 10mb in the written data. i have also monitor the adress range used from the cp /md5sum proccess under
/proc//$PID/mapsto see if i could find a pattern but i was unable to.i also tested ext2 and lvm with similar results aka corruption. later on the week i should get a pci promise controller and test on that one.
Things i have not tested is the patch that linus released 10 days ago and reiserfs3/4
my nvidia chipset was ck804 (a3)
Hope somehow we get to the bottom of this.
Quoted 9 lines
Hope this helps
btw amd erratas that could possible influence this are
115, 123, 156 with the latter been fascinating as it the workaround suggested is
0x0page entry.
Does anyone has any opinions about this? Could you please read the mentioned erratas and tell me what you think?
Best wishes,
Chris.
@ Sunny Days: Thanks for you mail.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[email protected]
x-mozilla-html:TRUE
version:2.1
end:vcard
Re: data corruption with nvidia chipsets and IDE/SATA drives
In reply to Christoph Anton MittererQuoted 6 lines · Christoph Anton Mitterer
The issue was basically the following: I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following:
I have about 30GB of testing data on my harddisk,... I repeat verifying sha512 sums on these files and check if errors occur. One test pass verifies the 30GB 50 times,... about one to four differences are found in each pass.
I'm also experiencing silent data corruption on writes to SATA disks connected to a Nvidia controller (nForce 4 chipset). The problem is 100% reproducible. Details of my configuration (mainboard model, lspci, etc.) are near the bottom of this message. What follows is a summation of my findings.
I have confirmed the corruption is occurring on the writes and not the reads. Furthermore, if I compare the original and copy while both are still cached in memory no corruption is found. But as soon as I flush the pagecache (by reading another file larger than memory) to force the copy of the file to be read from disk the corruption is seen. The corruption occurs with direct I/O and normal buffered filesystem I/O (ext3).
Booting with "mem=1g" (system has 4 GiB installed) makes no difference. So it isn't due to remapping memory above the 4 GiB boundary. Booting to single user and ensuring no unnecessary modules (video, etc.) are loaded also makes no difference.
The problem affects both disks attached to the nVidia SATA controller but not the two disks attached to the PATA side of the same controller. All four disks are different models. The same SATA disks attached to the Silicon Image 3114 SATA RAID controller (on the same mainboard) experiences the same corruption but at a lower probability. The same disks attached to a Promise TX2 SATA controller (in the same system) experience no corruption.
The system has run memtest86 for 24 hours with no errors.
The corruption occurs with a 32-bit kernel.org 2.6.12 kernel (from a Knoppix CD), 64-bit kernel.org 2.6.18.1, and 64-bit kernel.org 2.6.19.
The only pattern I can discern is that the corruption only affects the second four byte block of a sixteen byte aligned region. That is, the offset of the corrupted bytes always falls in the range 0x...4 to 0x...7. For example, here are a few representative offsets of corrupted bytes from one test of many:
0x020f4554
0x020f4555
0x020f4556
0x020f4557
0x020f4555
0x1597f1d4
0x23034ee5
0x2dfd08d4
0x33690b14
0x33690b15
0x33690b16
0x33690b17Approximately half the corruption involves all four bytes of the second four byte word of the 16 by aligned region. The remaining instances show one, two or three bytes being corrupted. There is no pattern that I can discern to the corruption. It is definitely not anything as simple as the correct bytes being replaced with zeros or specific bits being forced to a zero or one state or flipped. Copying a 2 GiB file typically results in between five and thirty bytes being corrupted. Back to back tests copying the same file results in different bytes being corrupted.
The problem appears to be sensitive to the data pattern. Some files can be copied repeatedly without corruption. But others will exhibit corruption 100% of the time when attached to the nVidia SATA controller and over 50% of the time when connected to the Sil 3114 controller.
I have tried direct I/O with varying transfer sizes from 1 KiB to 128 KiB (I have not tried single sector I/O). Corruption occurs with all block sizes. If I execute an I/O loop like this the corruption does not appear to occur:
while direct read 64 KiB original
direct write copy
direct read + verify copy
done
But that is tentative as I've only done two such tests at this time.
A coworker has hypothesized that this may be a consequence of bus crosstalk. That it doesn't happen when using the Promise TX2 controller suggests that if it is a HW bus problem it isn't a hypertransport or memory bus problem.
Configuration Details
=====================
Mainboard: ASUS A8N-SLI Deluxe with nForce 4 chipset
BIOS: 1016 (original) and 1805 (upgraded to in attempt to resolve)
CPU: AMD Athlon 64 X2 Dual Core Processor 4400+ stepping 02
Memory: 4 x 1 GiB Kingston matched pairs lifetime warranty
Kernel: 2.6.19 kernel.org (as well as others)
Disks: Western Digital WD740GD-00FL (10000 rpm "Raptor" disk)
Western Digital WD1600JD-32H (7200 rpm)00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
Subsystem: ASUSTeK Computer Inc. Unknown device 815a
Flags: bus master, 66MHz, fast devsel, latency 0
Capabilities: [44] HyperTransport: Slave or Primary Interface
Capabilities: [e0] HyperTransport: MSI Mapping
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev f3)
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard Flags: 66MHz, fast devsel, IRQ 3 I/O ports at e000 [size=32] I/O ports at 4c00 [size=64] I/O ports at 4c40 [size=64] Capabilities: [44] Power Management version 2
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2) (prog-if 10 [OHCI])
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 5
Memory at d4003000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Power Management version 2
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3) (prog-if 20 [EHCI])
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 7
Memory at feb00000 (32-bit, non-prefetchable) [size=256]
Capabilities: [44] Debug port
Capabilities: [80] Power Management version 2
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio Controller (rev a2)
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 11 I/O ports at d800 [size=256] I/O ports at dc00 [size=256] Memory at d4002000 (32-bit, non-prefetchable) [size=4K] Capabilities: [44] Power Management version 2
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2) (prog-if 8a [Master SecP PriP])
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0
I/O ports at f000 [size=16]
Capabilities: [44] Power Management version 2
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3) (prog-if 85 [Master SecO PriO])
Subsystem: ASUSTeK Computer Inc. Unknown device 815a Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 11 I/O ports at 09f0 [size=8] I/O ports at 0bf0 [size=4] I/O ports at 0970 [size=8] I/O ports at 0b70 [size=4] I/O ports at d400 [size=16] Memory at d4001000 (32-bit, non-prefetchable) [size=4K] Capabilities: [44] Power Management version 2
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3) (prog-if 85 [Master SecO PriO])
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 5 I/O ports at 09e0 [size=8] I/O ports at 0be0 [size=4] I/O ports at 0960 [size=8] I/O ports at 0b60 [size=4] I/O ports at c000 [size=16] Memory at d4000000 (32-bit, non-prefetchable) [size=4K] Capabilities: [44] Power Management version 2
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev f2) (prog-if 01 [Subtractive decode])
Flags: bus master, 66MHz, fast devsel, latency 0 Bus: primary=00, secondary=05, subordinate=05, sec-latency=128 I/O behind bridge: 00007000-00009fff Memory behind bridge: d2000000-d3ffffff Prefetchable memory behind bridge: d4100000-d42fffff
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000a000-0000afff
Memory behind bridge: d0000000-d1ffffff
Prefetchable memory behind bridge: 00000000c0000000-00000000cff00000
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
Flags: fast devsel
Capabilities: [80] HyperTransport: Host or Secondary Interface
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
Flags: fast devsel
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
Flags: fast devsel
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
Flags: fast devsel
01:00.0 VGA compatible controller: ATI Technologies Inc RV530 [Radeon X1600] (prog-if 00 [VGA])
Subsystem: VISIONTEK Unknown device 1890
Flags: bus master, fast devsel, latency 0, IRQ 3
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d1000000 (64-bit, non-prefetchable) [size=64K]
I/O ports at a000 [size=256]
Expansion ROM at d0000000 [disabled] [size=128K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Express Endpoint IRQ 0
Capabilities: [80] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
01:00.1 Display controller: ATI Technologies Inc RV530 [Radeon X1600] (Secondary)
Subsystem: VISIONTEK Unknown device 1891
Flags: fast devsel
Memory at d1010000 (64-bit, non-prefetchable) [disabled] [size=64K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Express Endpoint IRQ 0
05:07.0 Mass storage controller: Promise Technology, Inc. PDC20375 (SATA150 TX2plus) (rev 02)
Subsystem: Promise Technology, Inc. PDC20375 (SATA150 TX2plus) Flags: bus master, 66MHz, medium devsel, latency 96, IRQ 5 I/O ports at 7000 [size=64] I/O ports at 7400 [size=16] I/O ports at 7800 [size=128] Memory at d3124000 (32-bit, non-prefetchable) [size=4K] Memory at d3100000 (32-bit, non-prefetchable) [size=128K] Expansion ROM at d4280000 [disabled] [size=16K] Capabilities: [60] Power Management version 2
05:08.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 08)
Subsystem: IBM Netfinity 10/100 Flags: bus master, medium devsel, latency 32, IRQ 3 Memory at d3127000 (32-bit, non-prefetchable) [size=4K] I/O ports at 7c00 [size=64] Memory at d3000000 (32-bit, non-prefetchable) [size=1M] Expansion ROM at d4100000 [disabled] [size=1M] Capabilities: [dc] Power Management version 2
05:0a.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
Subsystem: ASUSTeK Computer Inc. Unknown device 8167 Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 7 I/O ports at 8000 [size=8] I/O ports at 8400 [size=4] I/O ports at 8800 [size=8] I/O ports at 8c00 [size=4] I/O ports at 9000 [size=16] Memory at d3125000 (32-bit, non-prefetchable) [size=1K] Expansion ROM at d4200000 [disabled] [size=512K] Capabilities: [60] Power Management version 2
05:0b.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link) (prog-if 10 [OHCI])
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, medium devsel, latency 32, IRQ 3
Memory at d3126000 (32-bit, non-prefetchable) [size=2K]
Memory at d3120000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [44] Power Management version 2
Signature · 4 lines
Kurtis D. Rader, Linux level 3 support email: [email protected] IBM Integrated Technology Services DID: +1 503-578-3714 15300 SW Koll Pkwy, MS RHE2-O2 service: 800-IBM-SERV Beaverton, OR 97006-6063 http://www.ibm.com
Re: data corruption with nvidia chipsets and IDE/SATA drives
Quoted 2 lines · Kurtis D. Rader
The same disks attached to a Promise TX2 SATA controller (in the same system) experience no corruption.
I spoke too soon. Corruption is occurring with the disks attached to the Promise TX2 SATA controller but much less frequently. With the drives attached to the nVidia controller copying certain 2 GiB files would result in at least five bytes, and as many as thirty, being corrupted every single time. On the Promise controller a given copy is likely to be good. And when corruption does occur fewer bytes are being affected --- as little as a single byte in a 2 GiB file. But still, some files never show corruption while others do.
The Promise controller in a PCI slot is measurably slower than the nVidia on the baseboard so the speed of the transfers appears to be a factor. In addition to the pattern of data. My hunch is this is a nVidia nForce 4 chipset design defect involving buss crosstalk or something similar. Which may be why I'm not seeing it when writing to my relatively slow PATA disks.
Signature · 4 lines
Kurtis D. Rader, Linux level 3 support email: [email protected] IBM Integrated Technology Services DID: +1 503-578-3714 15300 SW Koll Pkwy, MS RHE2-O2 service: 800-IBM-SERV Beaverton, OR 97006-6063 http://www.ibm.com
Re: data corruption with nvidia chipsets and IDE/SATA drives
In reply to Kurtis D. RaderHi!
Quoted 34 lines · * On Sat, Dec 02, 2006 at 05:17 PM (-0800), Kurtis D. Rader wrote:
Quoted 6 lines · Christoph Anton Mitterer
The issue was basically the following: I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following:
I have about 30GB of testing data on my harddisk,... I repeat verifying sha512 sums on these files and check if errors occur. One test pass verifies the 30GB 50 times,... about one to four differences are found in each pass.I'm also experiencing silent data corruption on writes to SATA disks connected to a Nvidia controller (nForce 4 chipset). The problem is 100% reproducible. Details of my configuration (mainboard model, lspci, etc.) are near the bottom of this message. What follows is a summation of my findings.
I have confirmed the corruption is occurring on the writes and not the reads. Furthermore, if I compare the original and copy while both are still cached in memory no corruption is found. But as soon as I flush the pagecache (by reading another file larger than memory) to force the copy of the file to be read from disk the corruption is seen. The corruption occurs with direct I/O and normal buffered filesystem I/O (ext3).
Booting with "mem=1g" (system has 4 GiB installed) makes no difference. So it isn't due to remapping memory above the 4 GiB boundary. Booting to single user and ensuring no unnecessary modules (video, etc.) are loaded also makes no difference.
The problem affects both disks attached to the nVidia SATA controller but not the two disks attached to the PATA side of the same controller. All four disks are different models. The same SATA disks attached to the Silicon Image 3114 SATA RAID controller (on the same mainboard) experiences the same corruption but at a lower probability. The same disks attached to a Promise TX2 SATA controller (in the same system) experience no corruption.
Quoted 1 line
The system has run memtest86 for 24 hours with no errors.
Although your problem report seems rather clearly to be related to the disk sub-system (e.g. as it only seems to appear at writings), I would just like to point out that running "memtest86" for some time without getting any errors does not necessarily state that the memory is faultless.
I recently had one case where a machine (Athlon XP 2200+) crashed irregularly. "memtest86", running several days, didn't find anything, but running the "stress test" of "Prime95" [1] for a few minutes clearly showed that the machine just miscalculated (Prime95's stress test stops in this case).
Just removing and reinserting the two memory modules (2 x 256 MB DDR-RAM) fixed it. The machine is now stable and "Prime95" hasn't stopped due to computational errors anymore since then. I suppose that one of the modules wasn't seated in its socket correctly, but I don't know why "memtest86" (and "memtest86+") didn't find it.
Bye,
Steffen
Re: data corruption with nvidia nForce 4 chipsets and IDE/SATA drives
In reply to Kurtis D. RaderQuoted 5 lines · Kurtis D. Rader
I'm also experiencing silent data corruption on writes to SATA disks connected to a Nvidia controller (nForce 4 chipset). The problem is 100% reproducible. Details of my configuration (mainboard model, lspci, etc.) are near the bottom of this message. What follows is a summation of my findings.
I ran more tests today. This is definitely not due to faulty memory. Also, clearly this is not a problem with the nVidia SATA controller that is part of the nForce 4 chipset since the problem can be reproduced with a Promise TX2 controller in a PCI slot.
The key question is whether this is a HW quirk of the nForce 4 chipset that the kernel can and should be working around? What tests can I run that will help narrow the field of investigation or provide more useful data?
I put one disk on my Promise TX2 SATA controller and the other on the onboard nVidia controller. As reported before corruption occurs when writing to either disk but at a lower probability when writing to the disk on the Promise TX2. Also, if I use a PATA disk as the source of the copy the probabability of corruption is also greatly reduced (the PATA disk has about 1/3 the throughput of the SATA disks).
I removed half the memory (two 1 GiB DIMMs from the second bank). Corruption still occurs. I Replaced the DIMMs in the first bank with the two removed from the second bank (leaving the second bank unpopulated as before). Corruption still occurs. I verified by inspection of the e820 map that all memory is mapped below the 4 GiB boundary. I've also been running prime95 all day with the options "-B2 -t". No errors have been reported. Coupled with previous clean runs of memtest86 and the symptoms there seems no reason to believe that faulty memory is the cause of the corruption.
I should stress that my system is not overclocked. The memory is top of the line matched pairs of Corsair CMX1024-3200PT DDR2 400 Mhz. The power supply is an Antec True 380S (380 watts). According to the BIOS temperature and voltage monitoring everthing is well within operational limits. All components are less than a year old. I buy the best components and am very conservative when building a system that I depend upon for doing my job.
I also performed some additional copies of the problematic files using a program whose core is a direct read, write, verify loop:
ifd = open(source_path, O_RDONLY | O_DIRECT);
ofd = open(dest_path, O_RDWR | O_DIRECT);
while (1) {
if (read( ifd, buf1, blocksize ) != blocksize) exit(0);
again:
write( ofd, buf1, blocksize );
lseek( ofd, -blocksize, SEEK_CUR );
read( ofd, buf2, blocksize );
for (i = 0; i < blocksize; i++) {
if (buf1[i] != buf2[i]) {
fprintf( stderr, "blk %6d offset 0x%04x good %02x bad %02x\n",
blk, i, buf1[i], buf2[i] );
lseek( ofd, -blocksize, SEEK_CUR );
goto again;
}
}
}It reports corruption at a very low rate (a single block out of 15 GiB). Rewriting the corrupted block always succeeds on the first try. Note that the test involves seven 2 GiB and one 1 GiB file (VMware Windows XP guest image split on 2 GiB boundaries). Of the eight files four have never been corrupted. Those four are mostly free space (i.e., blocks of nulls). The four files which consistently show corruption have few free blocks. Which is further evidence that this involves some subtle HW design fault that requires a specific pattern of data and bus transactions.
It's interesting to note that running prime95 at the same time as the disk write test reduces the number of corrupted bytes.
The test loop computes a md5sum for each copied file and compares that to the known correct md5sum. If any md5sums don't match it then performs a "cmp -l" of the original file and the copy.
In the "cmp -l" output below the middle column is the good value and the right-hand column is the bad value from the just created copy of the file. As reported before all corruption involves the second 32-bit word of a 16-byte aligned region. Note that the offsets reported by cmp(1) start at one so you need to subtract one to get a proper offset from the start of the file. So subtract one then convert to hex.
Below are the results of one test with 2 GiB of memory. I'll buy a beer for the person who can find a pattern to the corruption. Results from other test runs can be provided upon request. Most corruption involves only a few bytes in a given 2 GiB file. But I've had a couple of runs where hundreds, and in one case thousands, of bytes have been corrupted.
iteration 1
1c1
< 748b7ad615a62e41a88a6b5d47bb5581 Windows XP-f001.vmdk
---
Quoted 1 line
8aff2d3a23e4f08d9a3145d011368e93Windows XP-f001.vmdk
3,5c3,5
< 6bbe96c7da14487adab7e0c13b7e54f6 Windows XP-f003.vmdk < bf7c45c7a6c24bda251a34e73b0cbe9c Windows XP-f004.vmdk < 241c45aa023556d0bb0b864b3a83a800 Windows XP-f005.vmdk
---
Quoted 3 lines
e08d9b8194a1aac41de611a3ba782e03 Windows XP-f003.vmdk 8aa16afaf6bae8dcdeb2bb5595cbc76f Windows XP-f004.vmdk e65f5de0bd69d76d40d23593f9221f36 Windows XP-f005.vmdk
Windows XP-f001.vmdk
327748952 115 117
657644309 327 100
657644310 105 0 657644311 221 0 657644312 127 0
778889238 145 45
778889240 350 312
Windows XP-f002.vmdk
Windows XP-f003.vmdk
1025312597 70 60
1622579125 164 174
1622579126 135 125
1622579128 170 140
Windows XP-f004.vmdk
1129237493 50 104
1129237494 1 15
1922382310 14 4
1922382312 321 333
1936442328 236 224
2004252949 37 215
2004252950 0 4
2004252951 200 26
2004252952 371 212
2004430229 200 270
2004430230 16 107
2004430231 0 340
2004430232 242 317
2056253589 164 160
2056253590 340 200
2056253592 151 101
Windows XP-f005.vmdk
113235864 71 73 536394981 1 24 536394982 203 13
536394983 310 376
536394984 2 161
764048760 2 0
Windows XP-f006.vmdk
Windows XP-f007.vmdk
Windows XP-f008.vmdk
Signature · 4 lines
Kurtis D. Rader, Linux level 3 support email: [email protected] IBM Integrated Technology Services DID: +1 503-578-3714 15300 SW Koll Pkwy, MS RHE2-O2 service: 800-IBM-SERV Beaverton, OR 97006-6063 http://www.ibm.com
Re: data corruption with nvidia nForce 4 chipsets and IDE/SATA drives
Quoted 3 lines
The key question is whether this is a HW quirk of the nForce 4 chipset that the kernel can and should be working around? What tests can I run that will help narrow the field of investigation or provide more useful data?
Really it would need information from Nvidia on the problem, non-problem, possible errata and/or chipset flaws. In the absence of that I don't see a good way to debug it further than you have already.
Re: data corruption with nvidia chipsets and IDE/SATA drives
In reply to Kurtis D. RaderQuoted 5 lines · Kurtis D. Rader
I'm also experiencing silent data corruption on writes to SATA disks connected to a Nvidia controller (nForce 4 chipset). The problem is 100% reproducible. Details of my configuration (mainboard model, lspci, etc.) are near the bottom of this message. What follows is a summation of my findings.
Various suggestions (e.g., booting with "acpi=off") have either not helped or have resulted in a system which won't boot.
Today I replaced the ASUS A8N (nVidia nForce 4 chipset) mainboard and AMD Athlon 64 CPU with a Intel DP965LT (Intel 965 chipset) and E6600 Duo Core 2 CPU. The SATA disks and cables are unchanged. The case, power supply, and video card are also unchanged. Not one of the previous tests now results in corruption.
If anyone (e.g., a nVidia employee) wants to pursue this and can provide a meaningful action plan I'll be happy to install the problem components in another case and attempt to gather additional diagnostic data.
Signature · 4 lines
Kurtis D. Rader, Linux level 3 support email: [email protected] IBM Integrated Technology Services DID: +1 503-578-3714 15300 SW Koll Pkwy, MS RHE2-O2 service: 800-IBM-SERV Beaverton, OR 97006-6063 http://www.ibm.com
Re: data corruption with nvidia chipsets and IDE/SATA drives
In reply to Kurtis D. RaderAm Sonntag, 3. Dezember 2006 02:17 schrieb Kurtis D. Rader:
Quoted 279 lines · Christoph Anton Mitterer
Quoted 6 lines
The issue was basically the following: I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following:
I have about 30GB of testing data on my harddisk,... I repeat verifying sha512 sums on these files and check if errors occur. One test pass verifies the 30GB 50 times,... about one to four differences are found in each pass.I'm also experiencing silent data corruption on writes to SATA disks connected to a Nvidia controller (nForce 4 chipset). The problem is 100% reproducible. Details of my configuration (mainboard model, lspci, etc.) are near the bottom of this message. What follows is a summation of my findings.
I have confirmed the corruption is occurring on the writes and not the reads. Furthermore, if I compare the original and copy while both are still cached in memory no corruption is found. But as soon as I flush the pagecache (by reading another file larger than memory) to force the copy of the file to be read from disk the corruption is seen. The corruption occurs with direct I/O and normal buffered filesystem I/O (ext3).
Booting with "mem=1g" (system has 4 GiB installed) makes no difference. So it isn't due to remapping memory above the 4 GiB boundary. Booting to single user and ensuring no unnecessary modules (video, etc.) are loaded also makes no difference.
The problem affects both disks attached to the nVidia SATA controller but not the two disks attached to the PATA side of the same controller. All four disks are different models. The same SATA disks attached to the Silicon Image 3114 SATA RAID controller (on the same mainboard) experiences the same corruption but at a lower probability. The same disks attached to a Promise TX2 SATA controller (in the same system) experience no corruption.
The system has run memtest86 for 24 hours with no errors.
The corruption occurs with a 32-bit kernel.org 2.6.12 kernel (from a Knoppix CD), 64-bit kernel.org 2.6.18.1, and 64-bit kernel.org 2.6.19.
The only pattern I can discern is that the corruption only affects the second four byte block of a sixteen byte aligned region. That is, the offset of the corrupted bytes always falls in the range 0x...4 to 0x...7. For example, here are a few representative offsets of corrupted bytes from one test of many:
0x020f4554 0x020f4555 0x020f4556 0x020f4557 0x020f4555 0x1597f1d4 0x23034ee5 0x2dfd08d4 0x33690b14 0x33690b15 0x33690b16 0x33690b17Approximately half the corruption involves all four bytes of the second four byte word of the 16 by aligned region. The remaining instances show one, two or three bytes being corrupted. There is no pattern that I can discern to the corruption. It is definitely not anything as simple as the correct bytes being replaced with zeros or specific bits being forced to a zero or one state or flipped. Copying a 2 GiB file typically results in between five and thirty bytes being corrupted. Back to back tests copying the same file results in different bytes being corrupted.
The problem appears to be sensitive to the data pattern. Some files can be copied repeatedly without corruption. But others will exhibit corruption 100% of the time when attached to the nVidia SATA controller and over 50% of the time when connected to the Sil 3114 controller.
I have tried direct I/O with varying transfer sizes from 1 KiB to 128 KiB (I have not tried single sector I/O). Corruption occurs with all block sizes. If I execute an I/O loop like this the corruption does not appear to occur:
while direct read 64 KiB original
direct write copy
direct read + verify copy
doneBut that is tentative as I've only done two such tests at this time.
A coworker has hypothesized that this may be a consequence of bus crosstalk. That it doesn't happen when using the Promise TX2 controller suggests that if it is a HW bus problem it isn't a hypertransport or memory bus problem.
Configuration Details
=====================Mainboard: ASUS A8N-SLI Deluxe with nForce 4 chipset BIOS: 1016 (original) and 1805 (upgraded to in attempt to resolve) CPU: AMD Athlon 64 X2 Dual Core Processor 4400+ stepping 02 Memory: 4 x 1 GiB Kingston matched pairs lifetime warranty Kernel: 2.6.19 kernel.org (as well as others) Disks: Western Digital WD740GD-00FL (10000 rpm "Raptor" disk) Western Digital WD1600JD-32H (7200 rpm)00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3) Subsystem: ASUSTeK Computer Inc. Unknown device 815a
Flags: bus master, 66MHz, fast devsel, latency 0
Capabilities: [44] HyperTransport: Slave or Primary Interface
Capabilities: [e0] HyperTransport: MSI Mapping00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev f3)
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 000:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard Flags: 66MHz, fast devsel, IRQ 3 I/O ports at e000 [size=32] I/O ports at 4c00 [size=64] I/O ports at 4c40 [size=64] Capabilities: [44] Power Management version 200:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2) (prog-if 10 [OHCI]) Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 5 Memory atd4003000(32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Power Management version 200:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3) (prog-if 20 [EHCI]) Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 7 Memory atfeb00000(32-bit, non-prefetchable) [size=256]
Capabilities: [44] Debug port
Capabilities: [80] Power Management version 200:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio Controller (rev a2) Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 11 I/O ports at d800 [size=256] I/O ports at dc00 [size=256] Memory at d4002000 (32-bit, non-prefetchable) [size=4K] Capabilities: [44] Power Management version 200:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2) (prog-if 8a [Master SecP PriP]) Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0
I/O ports at f000 [size=16]
Capabilities: [44] Power Management version 200:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3) (prog-if 85 [Master SecO PriO]) Subsystem: ASUSTeK Computer Inc. Unknown device 815a
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 11 I/O ports at 09f0 [size=8] I/O ports at 0bf0 [size=4] I/O ports at 0970 [size=8] I/O ports at 0b70 [size=4] I/O ports at d400 [size=16] Memory at d4001000 (32-bit, non-prefetchable) [size=4K] Capabilities: [44] Power Management version 200:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3) (prog-if 85 [Master SecO PriO]) Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 5 I/O ports at 09e0 [size=8] I/O ports at 0be0 [size=4] I/O ports at 0960 [size=8] I/O ports at 0b60 [size=4] I/O ports at c000 [size=16] Memory at d4000000 (32-bit, non-prefetchable) [size=4K] Capabilities: [44] Power Management version 200:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev f2) (prog-if 01 [Subtractive decode]) Flags: bus master, 66MHz, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=128 I/O behind bridge: 00007000-00009fff Memory behind bridge: d2000000-d3ffffff Prefetchable memory behind bridge: d4100000-d42fffff00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000a000-0000afff
Memory behind bridge: d0000000-d1ffffff
Prefetchable memory behind bridge: 00000000c0000000-00000000cff00000
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Capabilities: [58] HyperTransport: MSI Mapping
Capabilities: [80] Express Root Port (Slot+) IRQ 0
Capabilities: [100] Virtual Channel00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration Flags: fast devsel
Capabilities: [80] HyperTransport: Host or Secondary Interface00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map Flags: fast devsel
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller Flags: fast devsel
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control Flags: fast devsel
01:00.0 VGA compatible controller: ATI Technologies Inc RV530 [Radeon X1600] (prog-if 00 [VGA]) Subsystem: VISIONTEK Unknown device 1890
Flags: bus master, fast devsel, latency 0, IRQ 3 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at d1000000 (64-bit, non-prefetchable) [size=64K] I/O ports at a000 [size=256] Expansion ROM at d0000000 [disabled] [size=128K] Capabilities: [50] Power Management version 2 Capabilities: [58] Express Endpoint IRQ 0 Capabilities: [80] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-01:00.1 Display controller: ATI Technologies Inc RV530 [Radeon X1600] (Secondary) Subsystem: VISIONTEK Unknown device 1891
Flags: fast devsel
Memory atd1010000(64-bit, non-prefetchable) [disabled] [size=64K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Express Endpoint IRQ 005:07.0 Mass storage controller: Promise Technology, Inc. PDC20375 (SATA150 TX2plus) (rev 02) Subsystem: Promise Technology, Inc. PDC20375 (SATA150 TX2plus)
Flags: bus master, 66MHz, medium devsel, latency 96, IRQ 5 I/O ports at 7000 [size=64] I/O ports at 7400 [size=16] I/O ports at 7800 [size=128] Memory at d3124000 (32-bit, non-prefetchable) [size=4K] Memory at d3100000 (32-bit, non-prefetchable) [size=128K] Expansion ROM at d4280000 [disabled] [size=16K] Capabilities: [60] Power Management version 205:08.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 08) Subsystem: IBM Netfinity 10/100
Flags: bus master, medium devsel, latency 32, IRQ 3 Memory at d3127000 (32-bit, non-prefetchable) [size=4K] I/O ports at 7c00 [size=64] Memory at d3000000 (32-bit, non-prefetchable) [size=1M] Expansion ROM at d4100000 [disabled] [size=1M] Capabilities: [dc] Power Management version 205:0a.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02) Subsystem: ASUSTeK Computer Inc. Unknown device 8167
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 7 I/O ports at 8000 [size=8] I/O ports at 8400 [size=4] I/O ports at 8800 [size=8] I/O ports at 8c00 [size=4] I/O ports at 9000 [size=16] Memory at d3125000 (32-bit, non-prefetchable) [size=1K] Expansion ROM at d4200000 [disabled] [size=512K] Capabilities: [60] Power Management version 2
Quoted 7 lines
05:0b.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link) (prog-if 10 [OHCI]) Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard
Flags: bus master, medium devsel, latency 32, IRQ 3 Memory at d3126000 (32-bit, non-prefetchable) [size=2K] Memory at d3120000 (32-bit, non-prefetchable) [size=16K] Capabilities: [44] Power Management version 2
Hello lkml,
I also have an nForce4 based mainboard and tried hard to reproduce your testcase but I found no errors so far!
Asus A8N-SLI Deluxe (nForce4)
AMD Athlon 64x2 3800 (overclocked)
2x Samsung SpinPoint 250GB (SATA2)
1x Samsung SpinPoint 400GB (SATA2)
2GB DDR1-Ram
Linux 2.6.19-rc6-mm1 #2 SMP PREEMPT Thu Nov 23 15:34:58 CET 2006 x86_64 GNU/Linux
Filesystem is XFS on LVM on dmraid (nForce4 fakeraid)
I copied a massive amount of data (more than 500GB) several times between the HDDs and ran md5sum each time, but it found no errors.
Maybe it's a temperature problem? I found my board to be very vulnerable to higher temperatures. If the nForce4 chipset reaches a temp. over 42°C it tends to malfunction. First symptoms I've had was that USB ceased to work then came SATA hangs and finally a kernel panic or bluescreen on Windows.
lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3) 00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3) 00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2) 00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3) 00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2) 00:07.0 RAID bus controller: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 RAID bus controller: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2) 00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3) 00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3) 00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3) 00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3) 00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: nVidia Corporation G70 [GeForce 7600 GT] (rev a1)
05:0b.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
05:0c.0 Ethernet controller: Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller (rev 13)
-Christian
Re: data corruption with nvidia chipsets and IDE/SATA drives
Quoted 2 lines · Christian
I copied a massive amount of data (more than 500GB) several times between the HDDs and ran md5sum each time, but it found no errors.
It might be a known problem that your BIOS addresses already, or maybe it's restricted to some revisions of the chip(s)?
Quoted 6 lines · Christoph Anton Mitterer
Hi.
Perhaps some of you have read my older two threads:
http://marc.theaimsgroup.com/?t=116312440000001&r=1&w=2 and the even older http://marc.theaimsgroup.com/?t=116291314500001&r=1&w=2
Quoted 7 lines
The issue was basically the following:
I found a severe bug mainly by fortune because it occurs very rarely. My test looks like the following: I have about 30GB of testing data on my harddisk,... I repeat verifying sha512 sums on these files and check if errors occur.
One test pass verifies the 30GB 50 times,... about one to four differences are found in each pass.
This sounds very similar to a corruption issue I was experiencing on my nforce4 based system. After replacing most of my hardware to no avail, I discovered that if increased the voltage for my RAM chips the corruption went away. Note that I was not overclocking at all.
Worth a try.
Dax Kelson