October 30, 2002
Quoted 1 line
o in 2.5.35 Serial ATA support (Andre Hedrick)
Erm, really ?
Quoted 1 line
o Post-freeze Add hardware sensors drivers (
lm_sensorsteam)
Something else I took a look at in the last few days was the ECC drivers. These are also zero impact, and could go in after the freeze (assuming the authors want them merged). They could do with a small amount of cleanup, but otherwise look ok.
Quoted 1 line
o Started Reorder x86 initialization (Dave Jones, Randy Dunlap)
I've jiggled a bunch of this (Randy didnt have time to play here) around as much as its probably going to be for 2.6. It's in -dj, has been sent for -ac, and will likely go to Linus post-freeze as its all cleanups, and one-liners.
Dave
Signature · 1 line
| Dave Jones. http://www.codemonkey.org.uk
On Wed, 30 Oct 2002, Dave Jones wrote:
| > o Started Reorder x86 initialization (Dave Jones, Randy Dunlap) | | I've jiggled a bunch of this (Randy didnt have time to play here) | around as much as its probably going to be for 2.6. It's in -dj, | has been sent for -ac, and will likely go to Linus post-freeze | as its all cleanups, and one-liners.
Right. Please remove my name from that item.
Signature · 1 line
~Randy
Quoted 4 lines · Dave Jones
Something else I took a look at in the last few days was the ECC drivers. These are also zero impact, and could go in after the freeze (assuming the authors want them merged). They could do with a small amount of cleanup, but otherwise look ok.
Assuming they work. No offense to the guys who got the ball rolling, but the architecture is lousy, and every driver I have messed with does not work correctly, and I wind up reimplementing it before I can use it.
I actually like the idea of ECC drivers, and routinely make certain there is a working ECC driver on the systems I ship. It is so much very easier to catch memory errors with good ECC error reporting. But unless I have slept soundly through a fundamental change, the linux-ecc project currently does not ship quality drivers. The infrastructure is bad, and the code is not quite correct.
If you want I can dig up the drivers I am currently using and send them to you.
I even have a working memory scrub routine.
Eric
Quoted 9 lines · Eric W. Biederman
I actually like the idea of ECC drivers, and routinely make certain there is a working ECC driver on the systems I ship. It is so much very easier to catch memory errors with good ECC error reporting. But unless I have slept soundly through a fundamental change, the linux-ecc project currently does not ship quality drivers. The infrastructure is bad, and the code is not quite correct.
If you want I can dig up the drivers I am currently using and send them to you.
That would be really cool
On 31 Oct 2002, Alan Cox wrote:
| On Thu, 2002-10-31 at 06:22, Eric W. Biederman wrote: | > I actually like the idea of ECC drivers, and routinely make certain | > there is a working ECC driver on the systems I ship. It is so much | > very easier to catch memory errors with good ECC error reporting. But | > unless I have slept soundly through a fundamental change, the | > linux-ecc project currently does not ship quality drivers. The | > infrastructure is bad, and the code is not quite correct. | > | > If you want I can dig up the drivers I am currently using and send | > them to you. | | That would be really cool
Ditto.
Signature · 1 line
~Randy
Quoted 9 lines · Eric W. Biederman
I actually like the idea of ECC drivers, and routinely make certain there is a working ECC driver on the systems I ship. It is so much very easier to catch memory errors with good ECC error reporting. But unless I have slept soundly through a fundamental change, the linux-ecc project currently does not ship quality drivers. The infrastructure is bad, and the code is not quite correct.
If you want I can dig up the drivers I am currently using and send them to you.
Go wild..
Dave
Signature · 1 line
| Dave Jones. http://www.codemonkey.org.uk
Hi!
Quoted 4 lines
If you want I can dig up the drivers I am currently using and send them to you.
I even have a working memory scrub routine.
What is "memory scrubbing" good for?
Pavel
Signature · 1 line
When do you have heart between your knees?
Quoted 8 lines · Pavel Machek
Hi!
Quoted 4 lines
If you want I can dig up the drivers I am currently using and send them to you.
I even have a working memory scrub routine.
What is "memory scrubbing" good for?
When you have a correctable ECC error on a page you need to rewrite the memory to remove the error. This prevents the correctable error from becoming an uncorrectable error if another bit goes bad. Also if you have a working software memory scrub routine you can be certain multiple errors from the same address are actually distinct. As opposed to multiple reports of the same error.
Eric
Quoted 6 lines · Eric W. Biederman
When you have a correctable ECC error on a page you need to rewrite the memory to remove the error. This prevents the correctable error from becoming an uncorrectable error if another bit goes bad. Also if you have a working software memory scrub routine you can be certain multiple errors from the same address are actually distinct. As opposed to multiple reports of the same error.
Note that this area has some extremely "interesting" properties. For one you have to be very careful what operation you use to scrub and its platform specific. On x86 for example you want to do something like lock addl $0, mem. A simple read/write isnt safe because if the memory area is a DMA target your read then write just corrupted data and made the problem worse not better!
Quoted 8 lines · Alan Cox
Quoted 6 lines · Eric W. Biederman
When you have a correctable ECC error on a page you need to rewrite the memory to remove the error. This prevents the correctable error from becoming an uncorrectable error if another bit goes bad. Also if you have a working software memory scrub routine you can be certain multiple errors from the same address are actually distinct. As opposed to multiple reports of the same error.
Quoted 7 lines
Note that this area has some extremely "interesting" properties. For one you have to be very careful what operation you use to scrub and its platform specific. On x86 for example you want to do something like lock addl $0, mem. A simple read/write isnt safe because if the memory area is a DMA target your read then write just corrupted data and made the problem worse not better!
The correctable ECC is supposed to be just that (correctable). It's supposed to be entirely transparent to the CPU/Software. An additional read of the affected error produces the same correction so the CPU will never even know. The x86 CPU/Software is only notified on an uncorrectable error. I don't know of any SDRAM controller that generates an interrupt upon a correctable error. Some store "logging" information internally, very difficult to get at on a running system.
Given that, "scrubbing" RAM seems to be somewhat useless on a running system. The next write to the affected area will fix the ECC bits, that't what is supposed to clear up the condition.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
Quoted 18 lines · Richard B. Johnson
Quoted 3 lines · Alan Cox
Quoted 2 lines · Eric W. Biederman
When you have a correctable ECC error on a page you need to rewrite the memory to remove the error. This prevents the correctable error from
becoming
Quoted 11 lines
Quoted 4 lines
an uncorrectable error if another bit goes bad. Also if you have a working software memory scrub routine you can be certain multiple errors from the same address are actually distinct. As opposed to multiple reports of the same error.
Note that this area has some extremely "interesting" properties. For one you have to be very careful what operation you use to scrub and its platform specific. On x86 for example you want to do something like lock addl $0, mem. A simple read/write isnt safe because if the memory area is a DMA target your read then write just corrupted data and made the problem worse not better!
yep lock addl $0, mem with the appropriate kmaps so it will work on any system I use. It isn't rocket science but since it is using kmap_atomic that function at least should probably get in the kernel.
Quoted 7 lines
The correctable ECC is supposed to be just that (correctable). It's supposed to be entirely transparent to the CPU/Software. An additional read of the affected error produces the same correction so the CPU will never even know. The x86 CPU/Software is only notified on an uncorrectable error. I don't know of any SDRAM controller that generates an interrupt upon a correctable error. Some store "logging" information internally, very difficult to get at on a running system.
Polling the memory controller periodically isn't hard, and you can usually get an interrupt as well. Though I have not explored the whole interrupt territory. Finding out when you have a corrected error is extremely useful as it gives a warning that your memory is going bad. Just like with a disk getting a bunch of errors means it is time to be replaced, but you still have a little time left.
Quoted 3 lines
Given that, "scrubbing" RAM seems to be somewhat useless on a running system. The next write to the affected area will fix the ECC bits, that't what is supposed to clear up the condition.
If it is your kernel text space that is getting the error there will be no next write.
Beyond that if you are trying to see if the multiple correctable errors you have are a single error, or an actual problem software scrubbing helps. Because then you know the second report was because the problem reoccured. Making it likely you have a bad bit in your DIMM.
Eric
Quoted 47 lines · Eric W. Biederman
Quoted 18 lines · Richard B. Johnson
Quoted 3 lines · Alan Cox
Quoted 2 lines · Eric W. Biederman
When you have a correctable ECC error on a page you need to rewrite the memory to remove the error. This prevents the correctable error from
becoming
Quoted 11 lines
Quoted 4 lines
an uncorrectable error if another bit goes bad. Also if you have a working software memory scrub routine you can be certain multiple errors from the same address are actually distinct. As opposed to multiple reports of the same error.
Note that this area has some extremely "interesting" properties. For one you have to be very careful what operation you use to scrub and its platform specific. On x86 for example you want to do something like lock addl $0, mem. A simple read/write isnt safe because if the memory area is a DMA target your read then write just corrupted data and made the problem worse not better!
yep lock addl $0, mem with the appropriate kmaps so it will work on any system I use. It isn't rocket science but since it is using
kmap_atomicthat function at least should probably get in the kernel.Quoted 7 lines
The correctable ECC is supposed to be just that (correctable). It's supposed to be entirely transparent to the CPU/Software. An additional read of the affected error produces the same correction so the CPU will never even know. The x86 CPU/Software is only notified on an uncorrectable error. I don't know of any SDRAM controller that generates an interrupt upon a correctable error. Some store "logging" information internally, very difficult to get at on a running system.
Polling the memory controller periodically isn't hard, and you can usually get an interrupt as well. Though I have not explored the whole interrupt territory. Finding out when you have a corrected error is extremely useful as it gives a warning that your memory is going bad. Just like with a disk getting a bunch of errors means it is time to be replaced, but you still have a little time left.
Quoted 3 lines
Given that, "scrubbing" RAM seems to be somewhat useless on a running system. The next write to the affected area will fix the ECC bits, that't what is supposed to clear up the condition.
If it is your kernel text space that is getting the error there will be no next write.
Quoted 7 lines
Beyond that if you are trying to see if the multiple correctable errors you have are a single error, or an actual problem software scrubbing helps. Because then you know the second report was because the problem reoccured. Making it likely you have a bad bit in your DIMM.
Eric
The initial premise is fundamentally flawed. That being that the first error you get will be a single-bit error.
Memory is not a bunch of randomly spaced bits that get coalesced into bites/shorts/longs when accessed. Instead, all the bits in a word are in the same general area. This means that a nuclear event will alter several. In fact, a nuclear event will likely put an "electronic hole" in a physical area of memory. This area may cross several memory "block" boundaries. These blocks are not usually related to physical pages at all. These blocks have different bit-densities depending upon the type and manufacturer. Typical bit-densities are 16, 64, 128, and 256 megabits. They are organized into banks so they can be addressed in rows and columns, minimizing the hardware. The result of a nuclear event may look like this:
Base
_______________________________________
0x1000 | Bank 1 | Bank 2 | Bank 3 | Bank 4 |
0x8000 | | | / | |
0x10000 |--------|---------/---------|--------|
0x18000 | Bank 5 | Bank / | Bank 7 | Bank 8 |
0x20000 | | / | | |
0x0000 -------------/-------------------------
/Particle trail---->
In this case, the event altered bits in bank 6 and bank 3. It may have altered bits in 2 and 7 also. The hits altered bits at many memory addresses as the diagram shows. The bits that got altered are in the hundreds of thousands (of bits). If you read these areas, without disabling ECC, you will get a NMI. If you read these areas with modern ECC hardware, the read, just like a write, will correct the ECC bits. Therefore, you have "fixed" corrupt memory data. This is not good.
Isolating a bad bit in RAM caused by bad RAM is not done by memory "scrubbing", it is done by having the NMI handler disable access to the bad RAM. In an ix86 machine, that task is very difficult because the handler, unlike a page-fault handler, has no direct knowledge of the page being accessed when the NMI occurred. One could "inspect" the code leading up to the fault, and guess what memory access occurred but that access is quite likely in the .text segment which means the code isn't even correct to inspect. This stuff is possible to do, and now that gigabytes of RAM are commonplace, it would probably be a welcome addition to the kernel because the probability of a single-bit error in ba-zillions of bits is quite high.
Any "memory scrubbing" routines are worthless and simply eat CPU cycles. Further, because of the well-established principle of locality-of-action, you can have multiple pages of trashed data in RAM, owned by all those sleeping processes, that won't be accessed until the next boot. If you want a reliable system, it's better to let sleeping dogs lie and not access that RAM. You certainly don't want to "scrub" it. That's like picking a scab. It will bleed.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
Quoted 2 lines · Richard B. Johnson
The initial premise is fundamentally flawed. That being that the first error you get will be a single-bit error.
I did not say a single bit error I said a correctable error. Which can recover if a single chip on a pair of DIMMs goes bad.
What I have seen in practice is that during manufacturing it is pretty random weather the first error from bad memory will be correctable or uncorrectable. Once the memory is running error free it is quite likely the first error will be a correctable error. Especially when it is the RAM that is going bad.
Quoted 4 lines
Isolating a bad bit in RAM caused by bad RAM is not done by memory "scrubbing", it is done by having the NMI handler disable access to the bad RAM.
Scrubbing is for making certain the correction is written back to the RAM. Many chipsets will correct the data going to the processor, but will leave it corrupted in RAM. Allowing the possibility of errors to accumulate, and making it hard to tell if multiple reports are from the same error or a different error.
Quoted 4 lines
In an ix86
machine, that task is very difficult because the handler, unlike a page-fault handler, has no direct knowledge of the page being accessed when the NMI occurred.
We are obviously working with quite different hardware. Intel chipsets routinely report an ECC error on the page level granularity.
Quoted 5 lines
One could
"inspect" the code leading up to the fault, and guess what memory access occurred but that access is quite likely in the .text segment which means the code isn't even correct to inspect.
I have seen no NMI error that ever trigger a cpu exception to be synchronous with the code, though that may be possible with an Athlon, which does the ECC correction in the CPU. In general the errors come in asynchronously at some point after the error occured. So even killing the task that is using the bad RAM is unreliable. If the error is not correctable, on a server I panic the machine.
Quoted 8 lines
This stuff is possible to do, and
now that gigabytes of RAM are commonplace, it would probably be a welcome addition to the kernel because the probability of a single-bit error in ba-zillions of bits is quite high.Any "memory scrubbing" routines are worthless and simply eat CPU cycles.
Functional memory in practice does not have ECC errors, so ECC code does not run. I only run the scrub routine on memory that has reported a correctable error. And I think
1200 machines with 4GB each, running processor intensive tasks is a reasonable sample to make this conclusion with.
Quoted 7 lines
Further, because of the well-established principle of locality-of-action, you can have multiple pages of trashed data in RAM, owned by all those sleeping processes, that won't be accessed until the next boot. If you want a reliable system, it's better to let sleeping dogs lie and not access that RAM. You certainly don't want to "scrub" it. That's like picking a scab. It will bleed.
I do not randomly scrub memory, though for the hardware that does not do that I am not be opposed to the idea of a daemon that does. The biggest problem with doing that in the cpu is that you are likely to trash your cache.
One of the bigger challenges to work through is that frequently leaves a few ECC error after setting up RAM. So a cpu scrubber might trigger those. Replacing the BIOS is a good way to be certain that doesn't happen :)
Eric