support for broken memory modules (BadRAM)

Following the RFC for the BadRAM feature here's the updated version with spelling fixes, thanks go to Randy Dunlap. Also the code is now less verbose, as requested by Andi Kleen.
v2 with even more spelling fixes suggested by Randy.
Patches are against vanilla 2.6.39.

The idea is to allow the user to specify RAM addresses that shouldn't be touched by the OS, because they are broken in some way. Not all machines have hardware support for hwpoison, ECC RAM, etc, so here's a solution that allows to use bitmasks to mask address patterns with the new "badram" kernel command line parameter.
Memtest86 has an option to generate these patterns since v2.3 so the only thing for the user to do should be:
- run Memtest86
- note down the pattern
- add badram=<pattern> to the kernel command line

The concerning pages are then marked with the hwpoison flag and thus won't be used by the memory managment system.

I have a couple of thoughts here:

- If this patchset is merged and a major user such as google is unable to use it and has to continue to carry a separate patch then that's a regrettable situation for the upstream kernel.

- Google's is, afaik, the largest use case we know of: zillions of machines for a number of years.  And this real-world experience tells us that the badram patchset has shortcomings.  Shortcomings which we can expect other users to experience.

So.  What are your thoughts on these issues?

Quoted 17 lines · On Wed, 22 Jun 2011 13:18:51 +0200 Stefan Assmann <[email protected]> wrote:

Following the RFC for the BadRAM feature here's the updated version with spelling fixes, thanks go to Randy Dunlap. Also the code is now less verbose, as requested by Andi Kleen.
v2 with even more spelling fixes suggested by Randy.
Patches are against vanilla 2.6.39.

The idea is to allow the user to specify RAM addresses that shouldn't be touched by the OS, because they are broken in some way. Not all machines have hardware support for hwpoison, ECC RAM, etc, so here's a solution that allows to use bitmasks to mask address patterns with the new "badram" kernel command line parameter.
Memtest86 has an option to generate these patterns since v2.3 so the only thing for the user to do should be:
- run Memtest86
- note down the pattern
- add badram=<pattern> to the kernel command line

Quoted 2 lines

The concerning pages are then marked with the hwpoison flag and thus won't be used by the memory managment system.

The google kernel has a similar capability. I asked Nancy to comment on these patches and she said:

: One, the bad addresses are passed via the kernel command line, which : has a limited length. It's okay if the addresses can be fit into a : pattern, but that's not necessarily the case in the google kernel. And : even with patterns, the limit on the command line length limits the : number of patterns that user can specify. Instead we use lilo to pass : a file containing the bad pages in e820 format to the kernel. :
: Second, the BadRAM patch expands the address patterns from the command : line into individual entries in the kernel's e820 table. The e820 : table is a fixed buffer that supports a very small, hard coded number : of entries (128). We require a much larger number of entries (on : the order of a few thousand), so much of the google kernel patch deals : with expanding the e820 table. Also, with the BadRAM patch, entries : that don't fit in the table are silently dropped and this isn't : appropriate for us.
:
: Another caveat of mapping out too much bad memory in general. If too : much memory is removed from low memory, a system may not boot. We : solve this by generating good maps. Our userspace tools do not map out : memory below a certain limit, and it verifies against a system's iomap : that only addresses from memory is mapped out.

I have a couple of thoughts here:

- If this patchset is merged and a major user such as google is unable to use it and has to continue to carry a separate patch then that's a regrettable situation for the upstream kernel.

- Google's is, afaik, the largest use case we know of: zillions of machines for a number of years. And this real-world experience tells us that the badram patchset has shortcomings. Shortcomings which we can expect other users to experience.

So. What are your thoughts on these issues?

Quoted 58 lines · On Wed, 22 Jun 2011 11:00:34 -0700 Andrew Morton wrote:

Quoted 17 lines · On Wed, 22 Jun 2011 13:18:51 +0200 Stefan Assmann <[email protected]> wrote:

Following the RFC for the BadRAM feature here's the updated version with spelling fixes, thanks go to Randy Dunlap. Also the code is now less verbose, as requested by Andi Kleen.
v2 with even more spelling fixes suggested by Randy.
Patches are against vanilla 2.6.39.

The idea is to allow the user to specify RAM addresses that shouldn't be touched by the OS, because they are broken in some way. Not all machines have hardware support for hwpoison, ECC RAM, etc, so here's a solution that allows to use bitmasks to mask address patterns with the new "badram" kernel command line parameter.
Memtest86 has an option to generate these patterns since v2.3 so the only thing for the user to do should be:
- run Memtest86
- note down the pattern
- add badram=<pattern> to the kernel command line

Quoted 2 lines

The concerning pages are then marked with the hwpoison flag and thus won't be used by the memory managment system.

The google kernel has a similar capability.  I asked Nancy to comment on these patches and she said:

: One, the bad addresses are passed via the kernel command line, which : has a limited length.  It's okay if the addresses can be fit into a : pattern, but that's not necessarily the case in the google kernel.  And : even with patterns, the limit on the command line length limits the : number of patterns that user can specify.  Instead we use lilo to pass : a file containing the bad pages in e820 format to the kernel. :
: Second, the BadRAM patch expands the address patterns from the command : line into individual entries in the kernel's e820 table.  The e820 : table is a fixed buffer that supports a very small, hard coded number : of entries (128).  We require a much larger number of entries (on : the order of a few thousand), so much of the google kernel patch deals : with expanding the e820 table. Also, with the BadRAM patch, entries : that don't fit in the table are silently dropped and this isn't : appropriate for us.
:
: Another caveat of mapping out too much bad memory in general.  If too : much memory is removed from low memory, a system may not boot.  We : solve this by generating good maps.  Our userspace tools do not map out : memory below a certain limit, and it verifies against a system's iomap : that only addresses from memory is mapped out.

I have a couple of thoughts here:

- If this patchset is merged and a major user such as google is unable to use it and has to continue to carry a separate patch then that's a regrettable situation for the upstream kernel.

- Google's is, afaik, the largest use case we know of: zillions of machines for a number of years.  And this real-world experience tells us that the badram patchset has shortcomings.  Shortcomings which we can expect other users to experience.

Quoted 1 line

So.  What are your thoughts on these issues?

Good comments, so where is google's patch submittal?

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

:
: Second, the BadRAM patch expands the address patterns from the command : line into individual entries in the kernel's e820 table. The e820 : table is a fixed buffer that supports a very small, hard coded number : of entries (128). We require a much larger number of entries (on : the order of a few thousand), so much of the google kernel patch deals : with expanding the e820 table.

I have a couple of thoughts here:

- If this patchset is merged and a major user such as google is unable to use it and has to continue to carry a separate patch then that's a regrettable situation for the upstream kernel.

- Google's is, afaik, the largest use case we know of: zillions of machines for a number of years. And this real-world experience tells us that the badram patchset has shortcomings. Shortcomings which we can expect other users to experience.

So. What are your thoughts on these issues?

Quoted 7 lines · Andrew Morton

:
: Second, the BadRAM patch expands the address patterns from the command : line into individual entries in the kernel's e820 table.  The e820 : table is a fixed buffer that supports a very small, hard coded number : of entries (128).  We require a much larger number of entries (on : the order of a few thousand), so much of the google kernel patch deals : with expanding the e820 table.

This has not been true for a long time.

Quoted 7 lines

I have a couple of thoughts here:

- If this patchset is merged and a major user such as google is unable to use it and has to continue to carry a separate patch then that's a regrettable situation for the upstream kernel.

Quoted 6 lines

- Google's is, afaik, the largest use case we know of: zillions of machines for a number of years.  And this real-world experience tells us that the badram patchset has shortcomings.  Shortcomings which we can expect other users to experience.

So.  What are your thoughts on these issues?

I think a binary structure fed as a linked list data object makes a lot more sense.  We already support feeding e820 entries in this way, bypassing the 128-entry limitation of the fixed table in the zeropage.

The main issue then is priority; in particular memory marked UNUSABLE (type 5) in the fed-in e820 map will of course overlap entries with normal RAM (type 1) information in the native map; we need to make sure that the type 5 information takes priority.

       -hpa

Good point. There's the MAX_NODES that expands it, though it's still hard coded, and as I understand, intended for NUMA node entries. We need anywhere from 8K to 64K 'bad' entries. This creates holes and translates to twice as many entries in the e820. We only want to allow this memory if it's needed, instead of hard coding it.

So. What are your thoughts on these issues?

Quoted 1 line

So. What are your thoughts on these issues?

Sounds orthogonal to me. You have to crawl before you walk.

A better way to pass in the data would be nice, but can be always added on top (e.g. some EFI environment variable)

For a first try a command line argument is quite
appropiate and simple enough.

A check for removing too much memory would be nice though, although it's just a choice between panicing early or later.

If something can be grafted on later then that's of course all good. I do think we should have some sort of plan in which we work out how that will be done. If we want to do it, that is.

However if we go this way then there's a risk that we'll end up with two different ways of configuring the feature and we'll need to

You'll always have multiple ways. Whatever magic you come up for the google BIOS or for EFI won't help the majority of users with old crufty legacy BIOS.

Quoted 5 lines · Andi Kleen

You'll always have multiple ways. Whatever magic you come up for the google BIOS or for EFI won't help the majority of users with old crufty legacy BIOS.

I don't think this has anything to do with this.

Quoted 7 lines · H. Peter Anvin

Quoted 5 lines · Andi Kleen

You'll always have multiple ways. Whatever magic you come up for the google BIOS or for EFI won't help the majority of users with old crufty legacy BIOS.

Quoted 1 line

I don't think this has anything to do with this.

Please elaborate.

How would you pass the bad page information instead in a fully backwards compatible way?

The fully backward compatible way is "memmap=<address>$<length>".

On Wed, 22 Jun 2011 13:18:51 +0200 Stefan Assmann <[email protected]> wrote:

Quoted 11 lines

The idea is to allow the user to specify RAM addresses that shouldn't be touched by the OS, because they are broken in some way. Not all machines have hardware support for hwpoison, ECC RAM, etc, so here's a solution that allows to use bitmasks to mask address patterns with the new "badram" kernel command line parameter.
Memtest86 has an option to generate these patterns since v2.3 so the only thing for the user to do should be:
- run Memtest86
- note down the pattern
- add badram=<pattern> to the kernel command line

Quoted 2 lines

The concerning pages are then marked with the hwpoison flag and thus won't be used by the memory managment system.

The google kernel has a similar capability. I asked Nancy to comment on these patches and she said:

: One, the bad addresses are passed via the kernel command line, which : has a limited length. It's okay if the addresses can be fit into a : pattern, but that's not necessarily the case in the google kernel. And : even with patterns, the limit on the command line length limits the : number of patterns that user can specify. Instead we use lilo to pass : a file containing the bad pages in e820 format to the kernel.

:
: Second, the BadRAM patch expands the address patterns from the command : line into individual entries in the kernel's e820 table. The e820 : table is a fixed buffer that supports a very small, hard coded number : of entries (128). We require a much larger number of entries (on : the order of a few thousand), so much of the google kernel patch deals : with expanding the e820 table. Also, with the BadRAM patch, entries : that don't fit in the table are silently dropped and this isn't : appropriate for us.

:
: Another caveat of mapping out too much bad memory in general. If too : much memory is removed from low memory, a system may not boot. We : solve this by generating good maps. Our userspace tools do not map out : memory below a certain limit, and it verifies against a system's iomap : that only addresses from memory is mapped out.

I have a couple of thoughts here:

- If this patchset is merged and a major user such as google is unable to use it and has to continue to carry a separate patch then that's a regrettable situation for the upstream kernel.

- Google's is, afaik, the largest use case we know of: zillions of machines for a number of years. And this real-world experience tells us that the badram patchset has shortcomings. Shortcomings which we can expect other users to experience.

So. What are your thoughts on these issues?

Quoted 2 lines

The concerning pages are then marked with the hwpoison flag and thus won't be used by the memory managment system.

The google kernel has a similar capability. I asked Nancy to comment on these patches and she said:

: One, the bad addresses are passed via the kernel command line, which : has a limited length. It's okay if the addresses can be fit into a : pattern, but that's not necessarily the case in the google kernel.

: And
: even with patterns, the limit on the command line length limits the : number of patterns that user can specify. Instead we use lilo to pass : a file containing the bad pages in e820 format to the kernel.

: Second, the BadRAM patch expands the address patterns from the command : line into individual entries in the kernel's e820 table. The e820 : table is a fixed buffer [...]

: We require a much larger number of entries (on
: the order of a few thousand), so much of the google kernel patch deals : with expanding the e820 table.

: Also, with the BadRAM patch, entries
: that don't fit in the table are silently dropped and this isn't : appropriate for us.

: Another caveat of mapping out too much bad memory in general.

: If too
: much memory is removed from low memory, a system may not boot. We : solve this by generating good maps. Our userspace tools do not map out : memory below a certain limit, and it verifies against a system's iomap : that only addresses from memory is mapped out.

- If this patchset is merged and a major user such as google is unable to use it and has to continue to carry a separate patch then that's a regrettable situation for the upstream kernel.

- Google's is, afaik, the largest use case we know of: zillions of machines for a number of years. And this real-world experience tells us that the badram patchset has shortcomings. Shortcomings which we can expect other users to experience.

The idea is to allow the user to specify RAM addresses that shouldn't be touched by the OS, because they are broken in some way. Not all machines have hardware support for hwpoison, ECC RAM, etc, so here's a solution that allows to use bitmasks to mask address patterns with the new "badram" kernel command line parameter.
Memtest86 has an option to generate these patterns since v2.3 so the only thing for the user to do should be:
- run Memtest86
- note down the pattern
- add badram=<pattern> to the kernel command line

Quoted 12 lines · Stefan Assmann

The idea is to allow the user to specify RAM addresses that shouldn't be touched by the OS, because they are broken in some way. Not all machines have hardware support for hwpoison, ECC RAM, etc, so here's a solution that allows to use bitmasks to mask address patterns with the new "badram" kernel command line parameter.
Memtest86 has an option to generate these patterns since v2.3 so the only thing for the user to do should be:
- run Memtest86
- note down the pattern
- add badram=<pattern> to the kernel command line

We already support the equivalent functionality with memmap=<address>$<length> for those with only a few ranges... this has been supported for ages, literally. For those with a lot of ranges, like Google, the command line is insufficient.

Quoted 14 lines · H. Peter Anvin

Quoted 12 lines · Stefan Assmann

The idea is to allow the user to specify RAM addresses that shouldn't be touched by the OS, because they are broken in some way. Not all machines have hardware support for hwpoison, ECC RAM, etc, so here's a solution that allows to use bitmasks to mask address patterns with the new "badram" kernel command line parameter.
Memtest86 has an option to generate these patterns since v2.3 so the only thing for the user to do should be:
- run Memtest86
- note down the pattern
- add badram=<pattern> to the kernel command line

Quoted 4 lines

We already support the equivalent functionality with memmap=<address>$<length> for those with only a few ranges... this has been supported for ages, literally. For those with a lot of ranges, like Google, the command line is insufficient.

Right, I think this has been discussed a while ago. So the advantages I see in this approach are. It allows to break down memory exclusion to the page level with a pattern of non-consecutive pages. So if every other page would be considered bad that's a bit tough to deal with using memmap.
Secondly patterns can be easily generated by running Memtest86 and thus easily be fed to the kernel by command line. Making it much more feasible for the average user to take advantage of it.

We already support the equivalent functionality with memmap=<address>$<length> for those with only a few ranges...

For those with a lot of ranges,
like Google, the command line is insufficient.

So far the use case I had in mind wasn't "thousands of entries". However expanding the e820 table is probably an issue that could be dealt with separately ?

Well if too much low memory is bad, you're screwed anyway, not? :)

At the moment I don't see any arguments why this patchset couldn't play along nicely or get enhanced to support what Google needs, but I don't know Googles patches yet.

How common are nontrivial patterns on real hardware? This would be interesting to hear from Google or another large user.

I have to say I think Google's point that truncating the list is unacceptable...

that would mean running in a known-bad configuration, and even a hard crash would be better.

If you plug 10 DIMMs into your machine, and each has a faulty row somewhere, then you will get into trouble if you stick to 5 patterns.

But if you happen to run into a faulty DIMM from time to time, the patterns should be your way out.

Quoted 2 lines

that would mean running in a known-bad configuration, and even a hard crash would be better.

...which is so sensible that it was of course taken into account in the BadRAM design!

Following the RFC for the BadRAM feature here's the updated version with spelling fixes, thanks go to Randy Dunlap. Also the code is now less verbose, as requested by Andi Kleen.
v2 with even more spelling fixes suggested by Randy.
Patches are against vanilla 2.6.39.
Repost with LKML in Cc as suggested by Andrew Morton.

Quoted 6 lines · Stefan Assmann

Following the RFC for the BadRAM feature here's the updated version with spelling fixes, thanks go to Randy Dunlap. Also the code is now less verbose, as requested by Andi Kleen.
v2 with even more spelling fixes suggested by Randy.
Patches are against vanilla 2.6.39.
Repost with LKML in Cc as suggested by Andrew Morton.

Would it be more reasonable to do this in the bootloader? You'd ideally want this to be done as early as possible in order to avoid awkward situations like your ramdisk ending up in the bad RAM area.

Quoted 3 lines · Matthew Garrett

Would it be more reasonable to do this in the bootloader? You'd ideally want this to be done as early as possible in order to avoid awkward situations like your ramdisk ending up in the bad RAM area.

Not sure what exactly you are suggesting here. The kernel somehow needs to know what memory areas to avoid so we supply this information via kernel command line.
What the bootloader could do is to allow the kernel/initrd to be loaded at an alternative address. That's briefly mentioned in the BadRAM Documentation as well. Is that what you mean or am I missing something?

Quoted 11 lines · Stefan Assmann

Quoted 3 lines · Matthew Garrett

Would it be more reasonable to do this in the bootloader? You'd ideally want this to be done as early as possible in order to avoid awkward situations like your ramdisk ending up in the bad RAM area.

Not sure what exactly you are suggesting here. The kernel somehow needs to know what memory areas to avoid so we supply this information via kernel command line.
What the bootloader could do is to allow the kernel/initrd to be loaded at an alternative address. That's briefly mentioned in the BadRAM Documentation as well. Is that what you mean or am I missing something?

For EFI booting we just hand an e820 map to the kernel. It ought to be easy enough to add support for that to the 16-bit entry point as well. Then the bootloader just needs to construct an e820 map of its own. I think grub2 actually already has some support for this. The advantage of this approach is that the knowledge of bad memory only has to exist in one place (ie, the bootloader) - the kernel can remain blisfully unaware.

According to Rick's reply in this thread a damaged row in a DIMM can easily cause a few thousand entries in the e820 table because it doesn't handle patterns. So the question I'm asking is, is it acceptable to have an e820 table with thousands maybe ten-thousands of entries? I really have no idea of the implications, maybe somebody else can comment on that.

Quoted 8 lines · Stefan Assmann

According to Rick's reply in this thread a damaged row in a DIMM can easily cause a few thousand entries in the e820 table because it doesn't handle patterns. So the question I'm asking is, is it acceptable to have an e820 table with thousands maybe ten-thousands of entries? I really have no idea of the implications, maybe somebody else can comment on that.

Given that that is what actually ends up happening in the kernel at some point anyway,

According to Rick's reply in this thread a damaged row in a DIMM can easily cause a few thousand entries in the e820 table because it doesn't handle patterns. So the question I'm asking is, is it acceptable to have an e820 table with thousands maybe ten-thousands of entries? I really have no idea of the implications, maybe somebody else can comment on that.

I don't think it makes sense to handle something like that with a list. The compact representation currently in badram is great for that.

Quoted 2 lines

I don't think it makes sense to handle something like that with a list. The compact representation currently in badram is great for that.

I'd tend to agree here. Rick has made a convincing argument that there are significant numbers of real world cases where a defective row/column in a DIMM results in a predictable pattern of errors. The ball is now in Google's court to take a look at their systems that have high numbers of errors to see if they can actually be described by a small number of BadRAM patterns as Rick has claimed.

-Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

We (Google) are working on a data-driven answer for this question. I know that there has been some analysis on this topic on the past, but I don't want to speculate until we've had some time to put all the pieces together.
Stay tuned for specifics.

Quoted 4 lines

We (Google) are working on a data-driven answer for this question.  I know that there has been some analysis on this topic on the past, but I don't want to speculate until we've had some time to put all the pieces together. Stay tuned for specifics.

It would be also good if you posted your kernel patches.

It's highly unusual -- to say the least -- to let someone's openly developed and posted patchkit compete with someone's else secret internal solution for review purposes.

-Andi

We (Google) are working on a data-driven answer for this question. I know that there has been some analysis on this topic on the past, but I don't want to speculate until we've had some time to put all the pieces together.

Hi Craig,

Quoted 1 line

We (Google) are working on a data-driven answer for this question. I

know

Quoted 2 lines

that there has been some analysis on this topic on the past, but I don't want to speculate until we've had some time to put all the pieces

together.

The easiest way to do this could be to take the algorithm from Memtest86 and apply it to your data, to see if it finds suitable patterns for the cases tried.

By counting bits set to zero in the masks, you could then determine how 'tight' they are. A mask with all-ones covers one memory page; each zero bit in the mask (outside of the CPU's page size) doubles the number of pages covered.

You can ignore the address over which the mask is applied, although you would then be assuming that all the pages covered by the mask are indeed filled with RAM.

You would want to add the figures for the different masks.

I am very curious about your findings. Independently of those, I am in favour of a patch that enables longer e820 tables if it has no further impact on speed or space.

Cheers,
-Rick

Hi Craig,

Quoted 3 lines

We (Google) are working on a data-driven answer for this question.  I know that there has been some analysis on this topic on the past, but I don't want to speculate until we've had some time to put all the pieces together.

The easiest way to do this could be to take the algorithm from Memtest86 and apply it to your data, to see if it finds suitable patterns for the cases tried.

By counting bits set to zero in the masks, you could then determine how 'tight' they are.  A mask with all-ones covers one memory page; each zero bit in the mask (outside of the CPU's page size) doubles the number of pages covered.

You can ignore the address over which the mask is applied, although you would then be assuming that all the pages covered by the mask are indeed filled with RAM.

You would want to add the figures for the different masks.

I am very curious about your findings.  Independently of those, I am in favour of a patch that enables longer e820 tables if it has no further impact on speed or space.

Cheers,
 -Rick

I am very curious about your findings. Independently of those, I am in favour of a patch that enables longer e820 tables if it has no further impact on speed or space.

Quoted 4 lines

I am very curious about your findings. Independently of those, I am in favour of a patch that enables longer e820 tables if it has no further impact on speed or space.

That is already in the mainline kernel, although only if fed from the boot loader (it was developed in the context of mega-NUMA machines); the stub fetching from INT 15h doesn't use this at the moment.

Does it scale? [...] Perhaps we may end up with a composite solution.

Hello,

Quoted 1 line

Does it scale? [...] Perhaps we may end up with a composite solution.

If I had my way, there would be an extension to the e820 format to allow the BadRAM patterns to be specified. Since the extension with bad page information is specific to boot loader interaction, this would work in exactly those cases that are covered by the current situation.

Quoted 2 lines

I don't think it makes sense to handle something like that with a list. The compact representation currently in badram is great for that.

I'd tend to agree here.  Rick has made a convincing argument that there are significant numbers of real world cases where a defective row/column in a DIMM results in a predictable pattern of errors.  The ball is now in Google's court to take a look at their systems that have high numbers of errors to see if they can actually be described by a small number of BadRAM patterns as Rick has claimed.

-Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/