Skip to contentSkip to conversation

marching through all physical memory in software

DISCUSSION Eric W. Biederman 8 participants 14 messages Jan 30, 2009 – Feb 4, 2009 1 message not in the archive Memory management

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 18 lines · Doug Thompson

Nigel Cunningham <[email protected]> wrote:

Hi again.

On Fri, 2009-01-30 at 10:13 +0100, Pavel Machek wrote:
> > Hi.
> >
> > On Wed, 2009-01-28 at 20:38 +0100, Pavel Machek wrote:
> > > You can do the scrubbing today by echo reboot > /sys/power/disk; echo > > > disk > /sys/power/state :-)... or using uswsusp APIs.
> >
> > That won't work. The RAM retains its contents across a reboot, and even > > for a little while after powering off.
>
> Yes, and the original goal was to rewrite all the memory with same > contents so that parity errors don't accumulate. SO scrubbing here != > trying to clear it.

Quoted 5 lines

Sorry - I think I missed something.

AFAICS, hibernating is going to be a noop as far as doing anything to memory that's not touched by the process of hibernating goes. It won't clear it or scrub it or anything else.

A background software scrubber simply has the job of rewritting memory to it's current content so that the data and the ecc check bits are guaranteed to be in sync keeping correctable ecc errors caused by environmental factors from accumulating.

Pavel's original comment was that the hibernation code has to walk all of memory to save it to disk so it would be a good place to look to figure out how to walk all of memory. And incidentally hibernation would serve as a crud way of rewritting all of memory.

Quoted 3 lines

Even if hibernating worked, it does not touch the issue of scrubbing memory that doesn't have hardware support AND the requirement of thousands of nodes in cluster who MUST remain operational for days on end.

But it may still serve as an example of how to walk through all of memory.

Quoted 10 lines

Sicortex's MIPS based system fits that exactly. When I did their EDAC driver they wanted to have a software scrubber at a NICE run level to scan memory and do this operation without shutting down the system.

We never got to it, but it would be a nice for some to have a background software scrubber. But I would need help from the memory guys on a proper interface.

The goal would be have a "loose" target of attempting to all most memory if not all. Sometime of iteration over the memory set.

Thinking about it. We only care about memory the kernel is using so the memory maps the BIOS supplies the kernel should be sufficient. We have weird corner cases like ACPI but not handling those in the first pass and getting something working should be fine.

There are other silly things such as wanting to only scrub memory on it's native NUMA node (if possible) for both performance and scalability.

Eric

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 3 lines · Eric W. Biederman

A background software scrubber simply has the job of rewritting memory to it's current content so that the data and the ecc check bits are guaranteed to be in sync

Don't you just need to READ memory? The memory controller hardware takes care of the rest in the vast majority of cases.

You only need to rewrite RAM if a correctable error occurs, and the chipset doesn't support automatic write-back of the corrected value (a different problem altogether...). The actual memory bits themselves are refreshed by the hardware quite frequently (max of every 64ms for DDR2, I believe)...

Cheers,

Tim.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 5 lines · Tim Small
Quoted 3 lines · Eric W. Biederman

A background software scrubber simply has the job of rewritting memory to it's current content so that the data and the ecc check bits are guaranteed to be in sync

Quoted 7 lines

Don't you just need to READ memory? The memory controller hardware takes care of the rest in the vast majority of cases.

You only need to rewrite RAM if a correctable error occurs, and the chipset doesn't support automatic write-back of the corrected value (a different problem altogether...). The actual memory bits themselves are refreshed by the hardware quite frequently (max of every 64ms for DDR2, I believe)...

At the point we are talking about software scrubbing it makes sense to assume a least common denominator memory controller, one that does not do automatic write-back of the corrected value, as all of the recent memory controllers do scrubbing in hardware.

Once you handle the stupidest hardware all other cases are just software optimizations on that, and we already have the tricky code that does a read-modify-write without changing the contents of memory, so guarantees everything it touches will be written back.

Eric

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 5 lines · Eric W. Biederman

At the point we are talking about software scrubbing it makes sense to assume a least common denominator memory controller, one that does not do automatic write-back of the corrected value, as all of the recent memory controllers do scrubbing in hardware.

I was just trying to clarify the distinction between the two processes which have similar names, but aren't (IMO) actually that similar:

"Software Scrubbing"

Triggering a read, and subsequent rewrite of a particular RAM location which has suffered a correctable ECC error(s) i.e. hardware detects an error, then the OS takes care of the rewrite to "scrub" the error in the case that the hardware doesn't handle this automatically.

This should be a very-occasional error-path process, and performance is probably not critical..

"Background Scrubbing"

. This is a poor name, IMO (scrub infers some kind of write to me), which applies to a process whereby you ensure that the ECC check-bits are verified periodically for the whole of physical RAM, so that single bit errors in a given ECC block don't accumulate and turn into uncorrectable errors. It may also lead to improved data collection for some failure modes. Again, many memory controllers implement this feature in hardware, so we shouldn't do it twice where this is supported.

There is (AFAIK) no need to do any writes here, and in fact doing so is only likely to hurt performance, I think.... The design which springs to mind is of a background thread which (possibly at idle priority) reads RAM at a user-configurable rate (e.g. consume a max of n% of memory bandwidth, or read all of RAM at least once every x minutes). Possible design issues:

. There will be some trade off between reducing impact on the system as a whole, and making firm guarantees about how often memory is checked. Difficult to know what the default would be, but probably no-firm-guarantee of minimum time (idle processing only) is likely to cause least problems for most users.
. An eye will need to be kept on the impact that this reading has on the performance of the rest of the system (e.g. cache pollution, and NUMA, as you previously mentioned), but my gut feeling is that for the majority of systems it shouldn't be significant. If practical mechanisms are available on some CPUs to read RAM without populating the CPU cache, we should use those (but I've no idea if they exist or not).

Perhaps a good default would be to benchmark memory read bandwidth when the feature is turned on, and then operate at (e.g.) 0.5% of that bandwidth.

Cheers,

Tim.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw
Quoted 23 lines · Tim Small
Quoted 5 lines · Eric W. Biederman

At the point we are talking about software scrubbing it makes sense to assume a least common denominator memory controller, one that does not do automatic write-back of the corrected value, as all of the recent memory controllers do scrubbing in hardware.

I was just trying to clarify the distinction between the two processes which have similar names, but aren't (IMO) actually that similar:

"Software Scrubbing"

Triggering a read, and subsequent rewrite of a particular RAM location which has suffered a correctable ECC error(s) i.e. hardware detects an error, then the OS takes care of the rewrite to "scrub" the error in the case that the hardware doesn't handle this automatically.

This should be a very-occasional error-path process, and performance is probably not critical..

"Background Scrubbing"

Quoted 7 lines

. This is a poor name, IMO (scrub infers some kind of write to me), which applies to a process whereby you ensure that the ECC check-bits are verified periodically for the whole of physical RAM, so that single bit errors in a given ECC block don't accumulate and turn into uncorrectable errors. It may also lead to improved data collection for some failure modes. Again, many memory controllers implement this feature in hardware, so we shouldn't do it twice where this is supported.

It is implined in the background scrubbing, that if a background scrub page read causes an ECC correctable error to be flagged, the normal "fix through scrub" behaviour of the memory controller will be triggered (possibly, the software scrubbing described above).

And if an uncorretable error is detected during the scrub, we have to do something about it as well. And that won't be that easy: locate whatever process is using that page, and so something smart to it... or do some emergency evasive actions if it is one of the kernel's data scructures, etc.

So, as you said, "background scrubbing" and "software scrubbing" really are very different things, and one has to expect that background scrubbing will eventually trigger software scrubbing, major system emergency handling (uncorrectable errors in kernel memory) or minor system emergency handling (uncorrectable errors in process memory).

Quoted 1 line

There is (AFAIK) no need to do any writes here, and in fact doing so is

One might want the possibility of doing inconditional writes, because it helps with memory bitrot on crappy hardware where the refresh cycles aren't enough to avoid bitrot. But you definately won't want it most of the time.

You can also implement software-based ECC using a background scrubber and setting aside pages to store the ECC information. Now, THAT is probably not worth bothering with due to the performance impact, but who knows...

Signature · 10 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#6 lore raw

Hi!

Quoted 19 lines

And if an uncorretable error is detected during the scrub, we have to do something about it as well. And that won't be that easy: locate whatever process is using that page, and so something smart to it... or do some emergency evasive actions if it is one of the kernel's data scructures, etc.

So, as you said, "background scrubbing" and "software scrubbing" really are very different things, and one has to expect that background scrubbing will eventually trigger software scrubbing, major system emergency handling (uncorrectable errors in kernel memory) or minor system emergency handling (uncorrectable errors in process memory).

Quoted 1 line

There is (AFAIK) no need to do any writes here, and in fact doing so is

One might want the possibility of doing inconditional writes, because it helps with memory bitrot on crappy hardware where the refresh cycles aren't enough to avoid bitrot. But you definately won't want it most of the time.

Quoted 4 lines

You can also implement software-based ECC using a background scrubber and setting aside pages to store the ECC information. Now, THAT is probably not worth bothering with due to the performance impact, but who knows...

Actually, that would be quite cool. a) I suspect memory in my zaurus bitrots and b) bitroting memory over s2ram is apprently quite common.

Pavel

Signature · 8 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#7 lore raw
Quoted 7 lines · Pavel Machek
Quoted 4 lines

You can also implement software-based ECC using a background scrubber and setting aside pages to store the ECC information. Now, THAT is probably not worth bothering with due to the performance impact, but who knows...

Actually, that would be quite cool. a) I suspect memory in my zaurus bitrots and b) bitroting memory over s2ram is apprently quite common.

Well, software-based ECC for s2ram (calculate right before s2ram, check-and-fix right after waking up) is certainly doable and a LOT easier than my crazy idea of sofware-based generic ECC (which requires some sort of trick to detect pages that were written to, so that you can update their ECC information)...

Signature · 10 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#8 lore raw
In reply to Eric W. Biederman

Hi.

Quoted 31 lines · Eric W. Biederman
Quoted 18 lines · Doug Thompson

Nigel Cunningham <[email protected]> wrote:

Hi again.

On Fri, 2009-01-30 at 10:13 +0100, Pavel Machek wrote:
> > Hi.
> >
> > On Wed, 2009-01-28 at 20:38 +0100, Pavel Machek wrote:
> > > You can do the scrubbing today by echo reboot > /sys/power/disk; echo > > > disk > /sys/power/state :-)... or using uswsusp APIs.
> >
> > That won't work. The RAM retains its contents across a reboot, and even > > for a little while after powering off.
>
> Yes, and the original goal was to rewrite all the memory with same > contents so that parity errors don't accumulate. SO scrubbing here != > trying to clear it.

Quoted 5 lines

Sorry - I think I missed something.

AFAICS, hibernating is going to be a noop as far as doing anything to memory that's not touched by the process of hibernating goes. It won't clear it or scrub it or anything else.

A background software scrubber simply has the job of rewritting memory to it's current content so that the data and the ecc check bits are guaranteed to be in sync keeping correctable ecc errors caused by environmental factors from accumulating.

Quoted 4 lines

Pavel's original comment was that the hibernation code has to walk all of memory to save it to disk so it would be a good place to look to figure out how to walk all of memory. And incidentally hibernation would serve as a crud way of rewritting all of memory.

Thanks. Now I get it :)

Nigel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
In reply to Eric W. Biederman
Quoted 4 lines · Eric W. Biederman

Thinking about it. We only care about memory the kernel is using so the memory maps the BIOS supplies the kernel should be sufficient. We have weird corner cases like ACPI but not handling those in the first pass and getting something working should be fine.

Agreed.

The next question is who handles the conversion of the various different arch-specific BIOS mappings to a standard format that we can feed to the background "scrub" code. Is this something that belongs in the edac memory controller code, or would it live in /arch/foo somewhere?

Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw
Quoted 4 lines · Chris Friesen

The next question is who handles the conversion of the various different arch-specific BIOS mappings to a standard format that we can feed to the background "scrub" code. Is this something that belongs in the edac memory controller code, or would it live in /arch/foo somewhere?

If it's intended to be something basically stand-alone that doesn't require an actual EDAC chipset, it should probably live elsewhere. Otherwise, you get into the case of people who don't enable it because they "know" their hardware doesn't have an EDAC ability, even if they *could* benefit from the function.

On the other hand, if it's an EDAC-only thing, maybe under drivers/edac/$ARCH?

#11 lore raw
Quoted 11 lines · [email protected] wrote:
Quoted 1 line · Chris Friesen

The next question is who handles the conversion of the various different arch-specific BIOS mappings to a standard format that we can feed to the background "scrub" code. Is this something that belongs in the edac memory controller code, or would it live in /arch/foo somewhere?

If it's intended to be something basically stand-alone that doesn't require an actual EDAC chipset, it should probably live elsewhere. Otherwise, you get into the case of people who don't enable it because they "know" their hardware doesn't have an EDAC ability, even if they *could* benefit from the function.

On the other hand, if it's an EDAC-only thing, maybe under drivers/edac/$ARCH?

I don't see anything in the name of EDAC that implies hardware only...a software memory scrub could be considered "error detection and correction". Might have to update the config help text though.

Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#12 lore raw

Hi!

Quoted 12 lines
Quoted 12 lines
Quoted 5 lines

The next question is who handles the conversion of the various different arch-specific BIOS mappings to a standard format that we can feed to the background "scrub" code. Is this something that belongs in the edac memory controller code, or would it live in /arch/foo somewhere?

If it's intended to be something basically stand-alone that doesn't require an actual EDAC chipset, it should probably live elsewhere. Otherwise, you get into the case of people who don't enable it because they "know" their hardware doesn't have an EDAC ability, even if they *could* benefit from the function.

Quoted 5 lines
Quoted 1 line

On the other hand, if it's an EDAC-only thing, maybe under drivers/edac/$ARCH?

I don't see anything in the name of EDAC that implies hardware only...a software memory scrub could be considered "error detection and correction". Might have to update the config help text though.

Software memory scrub would no longer be a "driver" :-). So it should go into kernel/scrub or mm/scrub or maybe mm/edac or something.

Signature · 8 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#13 lore raw
Quoted 5 lines · Pavel Machek
Quoted 1 line

I don't see anything in the name of EDAC that implies hardware only...a software memory scrub could be considered "error detection and correction". Might have to update the config help text though.

Software memory scrub would no longer be a "driver" :-). So it should go into kernel/scrub or mm/scrub or maybe mm/edac or something.

True enough. In that case, something under "mm" makes more sense to me than something under "kernel".

Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw
In reply to Pavel Machek

There may be generic code, but the actual scrubbing can be architecture specific. You have to atomically read and write back. And each arch has different way of handling that. See arch/X/include/asm/edac.h

Quoted 3 lines · Pavel Machek

Software memory scrub would no longer be a "driver" :-). So it should go into kernel/scrub or mm/scrub or maybe mm/edac or something.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>