Linux MM Alignment Session on Hwpoison on Wednesday
Hi everybody,
We host a biweekly series, the Linux MM Alignment Session, on Wednesdays. We'd like to invite MM developers to attend and will announce the topic for the next instance on the Monday prior to the next meeting.
Our next Linux MM Alignment Session is scheduled for Wednesday. The details:
Wednesday, September 23 * 9:00 - 10:00am PDT (UTC-7)
https://meet.google.com/csb-wcds-xya
backup: https://tel.meet/csb-wcds-xya?pin=1301132214803
doc:
https://docs.google.com/document/d/1QQfZFPHa-pGEGf4A06cSqS0jdJwuZ8ro9JU5H0YSUQ0/view
This week's topic will be Hwpoison, see the discussion in
https://marc.info/?l=linux-kernel&m=178796007385067&w=2
No specific agenda other than to gather core MM developers with hwpoison developers and discuss:
- where validation should live, esp with regard to the page allocator, or strict invariants in the page allocator
+ implications for the fastpath
- handling for folios of arbitary orders
- handling partial folio poisoning and split failures
- race conditions between memory_failure() and concurrent alloc/free/compaction
- soft offline for pages in the pcp lists
We might even want to be bold and discuss a holistic design overhaul if it's warranted.
If anybody has ideas for future topics, please let me know and I'll try to organize them. We'd love to have volunteers to lead future topics as well as requests for MM topics to be presented.
Looking forward to seeing all of you on Wednesday!
Time zones
PDT (UTC-7) 9:00am
MDT (UTC-6) 10:00am
CDT (UTC-5) 11:00am
EDT (UTC-4) 12:00pm
Rio de Janeiro (UTC-3) 1:00pm
London (UTC+1) 5:00pm
Berlin (UTC+2) 6:00pm
Moscow (UTC+3) 7:00pm
Dubai (UTC+4) 8:00pm
Mumbai (UTC+5:30) 9:30pm
Singapore (UTC+8) 12:00am Thursday
Beijing (UTC+8) 12:00am Thursday
Tokyo (UTC+9) 1:00am Thursday
Sydney (UTC+10) 2:00am Thursday
Auckland (UTC+12) 4:00am Thursday
Quoted 18 lines · David Rientjes
Hi everybody,
We host a biweekly series, the Linux MM Alignment Session, on Wednesdays. We'd like to invite MM developers to attend and will announce the topic for the next instance on the Monday prior to the next meeting.
Our next Linux MM Alignment Session is scheduled for Wednesday. The details:
Wednesday, September 23 * 9:00 - 10:00am PDT (UTC-7)
https://meet.google.com/csb-wcds-xya
backup: https://tel.meet/csb-wcds-xya?pin=1301132214803
doc:
https://docs.google.com/document/d/1QQfZFPHa-pGEGf4A06cSqS0jdJwuZ8ro9JU5H0YSUQ0/viewThis week's topic will be Hwpoison, see the discussion in
https://marc.info/?l=linux-kernel&m=178796007385067&w=2
Quoted 10 lines
No specific agenda other than to gather core MM developers with hwpoison developers and discuss:
- where validation should live, esp with regard to the page allocator, or strict invariants in the page allocator
+ implications for the fastpath
- handling for folios of arbitary orders
- handling partial folio poisoning and split failures
- race conditions betweenmemory_failure()and concurrent alloc/free/compaction
- soft offline for pages in the pcp lists
[+Cc Breno]
This might be worth some attention:
[PATCH v5 0/9] mm/memory-failure: keep hardware-poisoned pages out of the next kexec
https://lore.kernel.org/linux-mm/[email protected]/
- The question: How best can we pass information about poisoned pages across kexec when the kernel accesses a poisoned page, panics and calls kexec, so that the next kernel avoids hitting the poisoned pages again?
One challenge here is introducing another source (a new EFI bitmap that represents poisoned memory, in addition to existing PG_hwpoison) of poisoned pages adds complexity and confusion.
David Hildenbrand suggested that we should simplify this as much as possible in a way that we don't have two sources of information w/ inconsistency between them and using memmap as the only source.
e.g.) By consuming the EFI bitmap only once when initializing memmap (to propagate poison information to not just free pages through __free_pages_core(), but to the all pages on memmap).
Another challenge here is that we don't have functionality to poison pages early in the boot process before memmap and buddy are ready. So any allocation before initializing them might still allocate poisoned pages.
Quoted 26 lines
We might even want to be bold and discuss a holistic design overhaul if it's warranted.
If anybody has ideas for future topics, please let me know and I'll try to organize them. We'd love to have volunteers to lead future topics as well as requests for MM topics to be presented.
Looking forward to seeing all of you on Wednesday!
Time zones
PDT (UTC-7) 9:00am
MDT (UTC-6) 10:00am
CDT (UTC-5) 11:00am
EDT (UTC-4) 12:00pm
Rio de Janeiro (UTC-3) 1:00pm
London (UTC+1) 5:00pm
Berlin (UTC+2) 6:00pm
Moscow (UTC+3) 7:00pm
Dubai (UTC+4) 8:00pm
Mumbai (UTC+5:30) 9:30pm
Singapore (UTC+8) 12:00am Thursday
Beijing (UTC+8) 12:00am Thursday
Tokyo (UTC+9) 1:00am Thursday
Sydney (UTC+10) 2:00am Thursday
Auckland (UTC+12) 4:00am Thursday
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 50 lines · Harry Yoo
Quoted 18 lines · David Rientjes
Hi everybody,
We host a biweekly series, the Linux MM Alignment Session, on Wednesdays. We'd like to invite MM developers to attend and will announce the topic for the next instance on the Monday prior to the next meeting.
Our next Linux MM Alignment Session is scheduled for Wednesday. The details:
Wednesday, September 23 * 9:00 - 10:00am PDT (UTC-7)
https://meet.google.com/csb-wcds-xya
backup: https://tel.meet/csb-wcds-xya?pin=1301132214803
doc:
https://docs.google.com/document/d/1QQfZFPHa-pGEGf4A06cSqS0jdJwuZ8ro9JU5H0YSUQ0/viewThis week's topic will be Hwpoison, see the discussion in
https://marc.info/?l=linux-kernel&m=178796007385067&w=2Quoted 10 lines
No specific agenda other than to gather core MM developers with hwpoison developers and discuss:
- where validation should live, esp with regard to the page allocator, or strict invariants in the page allocator
+ implications for the fastpath
- handling for folios of arbitary orders
- handling partial folio poisoning and split failures
- race conditions betweenmemory_failure()and concurrent alloc/free/compaction
- soft offline for pages in the pcp lists[+Cc Breno]
This might be worth some attention:
[PATCH v5 0/9]
mm/memory-failure: keep hardware-poisoned pages out of the next kexec
https://lore.kernel.org/linux-mm/[email protected]/- The question: How best can we pass information about poisoned pages across kexec when the kernel accesses a poisoned page, panics and calls kexec, so that the next kernel avoids hitting the poisoned pages again?
One challenge here is introducing another source (a new EFI bitmap that represents poisoned memory, in addition to existing
PG_hwpoison) of poisoned pages adds complexity and confusion.David Hildenbrand suggested that we should simplify this as much as possible in a way that we don't have two sources of information w/ inconsistency between them and using memmap as the only source.
Quoted 8 lines
e.g.) By consuming the EFI bitmap only once when initializing memmap (to propagate poison information to not just free pages through
__free_pages_core(), but to the all pages on memmap).Another challenge here is that we don't have functionality to poison pages early in the boot process before memmap and buddy are ready. So any allocation before initializing them might still allocate poisoned pages.
I've been thinking some more (and will reply in detail to the series), but I do wonder whether memblock should actually be thought about poisoned ranges and refuse to hand them out (reserved/allocated).
So we'd use the bitmap only to feed memblock, but not for anything else later (if possible).
Signature · 3 lines
Cheers, David
Quoted 55 lines · David Hildenbrand (Arm)
Quoted 54 lines · Harry Yoo
Quoted 18 lines · David Rientjes
Hi everybody,
We host a biweekly series, the Linux MM Alignment Session, on Wednesdays. We'd like to invite MM developers to attend and will announce the topic for the next instance on the Monday prior to the next meeting.
Our next Linux MM Alignment Session is scheduled for Wednesday. The details:
Wednesday, September 23 * 9:00 - 10:00am PDT (UTC-7)
https://meet.google.com/csb-wcds-xya
backup: https://tel.meet/csb-wcds-xya?pin=1301132214803
doc:
https://docs.google.com/document/d/1QQfZFPHa-pGEGf4A06cSqS0jdJwuZ8ro9JU5H0YSUQ0/viewThis week's topic will be Hwpoison, see the discussion in
https://marc.info/?l=linux-kernel&m=178796007385067&w=2Quoted 10 lines
No specific agenda other than to gather core MM developers with hwpoison developers and discuss:
- where validation should live, esp with regard to the page allocator, or strict invariants in the page allocator
+ implications for the fastpath
- handling for folios of arbitary orders
- handling partial folio poisoning and split failures
- race conditions betweenmemory_failure()and concurrent alloc/free/compaction
- soft offline for pages in the pcp lists[+Cc Breno]
This might be worth some attention:
[PATCH v5 0/9]
mm/memory-failure: keep hardware-poisoned pages out of the next kexec
https://lore.kernel.org/linux-mm/[email protected]/- The question: How best can we pass information about poisoned pages across kexec when the kernel accesses a poisoned page, panics and calls kexec, so that the next kernel avoids hitting the poisoned pages again?
One challenge here is introducing another source (a new EFI bitmap that represents poisoned memory, in addition to existing
PG_hwpoison) of poisoned pages adds complexity and confusion.David Hildenbrand suggested that we should simplify this as much as possible in a way that we don't have two sources of information w/ inconsistency between them and using memmap as the only source.
e.g.) By consuming the EFI bitmap only once when initializing memmap (to propagate poison information to not just free pages through
__free_pages_core(), but to the all pages on memmap).
Quoted 8 lines
Quoted 4 lines
Another challenge here is that we don't have functionality to poison pages early in the boot process before memmap and buddy are ready. So any allocation before initializing them might still allocate poisoned pages.
I've been thinking some more (and will reply in detail to the series), but I do wonder whether memblock should actually be thought about poisoned ranges and refuse to hand them out (reserved/allocated).
I believe this is what the patchset actually did in v2.
That way we'll have to either
1) make any architecture that supports kexec select ARCH_KEEP_MEMBLOCK, or 2) make kexec scan the bitmap when allocating the memory.
Quoted 2 lines
So we'd use the bitmap only to feed memblock, but not for anything else later (if possible).
We can try not to use it for anything later, but still we'd have to pass a proper bitmap (through the EFI table) to the next kernel. At least the kernel should update the bitmap for newly poisoned pages at some point.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 36 lines · Harry Yoo
Quoted 26 lines · David Hildenbrand (Arm)
Quoted 25 lines · Harry Yoo
[+Cc Breno]
This might be worth some attention:
[PATCH v5 0/9]
mm/memory-failure: keep hardware-poisoned pages out of the next kexec
https://lore.kernel.org/linux-mm/[email protected]/- The question: How best can we pass information about poisoned pages across kexec when the kernel accesses a poisoned page, panics and calls kexec, so that the next kernel avoids hitting the poisoned pages again?
One challenge here is introducing another source (a new EFI bitmap that represents poisoned memory, in addition to existing
PG_hwpoison) of poisoned pages adds complexity and confusion.David Hildenbrand suggested that we should simplify this as much as possible in a way that we don't have two sources of information w/ inconsistency between them and using memmap as the only source.
e.g.) By consuming the EFI bitmap only once when initializing memmap (to propagate poison information to not just free pages through
__free_pages_core(), but to the all pages on memmap).Quoted 8 lines
Quoted 4 lines
Another challenge here is that we don't have functionality to poison pages early in the boot process before memmap and buddy are ready. So any allocation before initializing them might still allocate poisoned pages.
I've been thinking some more (and will reply in detail to the series), but I do wonder whether memblock should actually be thought about poisoned ranges and refuse to hand them out (reserved/allocated).
Quoted 5 lines
I believe this is what the patchset actually did in v2.
That way we'll have to either
1) make any architecture that supports kexec selectARCH_KEEP_MEMBLOCK, or 2) make kexec scan the bitmap when allocating the memory.
My naive design would be:
1) Teach memblock early about poisoned memory ranges. Don't let it hand them out.
2) Memblock will naturally skip them when freeing pages to the buddy. No need to hook bitmap scans into other code path.
3) Teach memblock to hwpoison the memmap.
4) Don't update memblock (ARCH_KEEP_MEMBLOCK) when poisoning occurs later during boot. Any hwpoison happening after boot will be reflected in the memmap (+ synced to the bitmap).
What you are describing is "how to keep kexec from placing data on hwpoisoned ranges". If we want to avoid scanning a bitmap (which I really think we want to avoid), the memmap is our ultimate source of truth for hwpoisoned. I would just consult that. (that's likely what v2 did?)
One complication might be that a single hwpoisoned page is represented as a bigger chunk in the bitmap. So we might place kexec data on healthy pages within a poisoned (per bitmap) block. That's fine (pages are healthy after all), we just have to teach the new kernel to deal with that.
Quoted 8 lines
Quoted 3 lines
So we'd use the bitmap only to feed memblock, but not for anything else later (if possible).
We can try not to use it for anything later, but still we'd have to pass a proper bitmap (through the EFI table) to the next kernel. At least the kernel should update the bitmap for newly poisoned pages at some point.
Yes. I think it's fine to update the bitmap passively through some callback mechanism from hwpoison code.
Signature · 3 lines
Cheers, David
Quoted 39 lines · David Hildenbrand (Arm)
Quoted 38 lines · Harry Yoo
Quoted 26 lines · David Hildenbrand (Arm)
Quoted 25 lines · Harry Yoo
[+Cc Breno]
This might be worth some attention:
[PATCH v5 0/9]
mm/memory-failure: keep hardware-poisoned pages out of the next kexec
https://lore.kernel.org/linux-mm/[email protected]/- The question: How best can we pass information about poisoned pages across kexec when the kernel accesses a poisoned page, panics and calls kexec, so that the next kernel avoids hitting the poisoned pages again?
One challenge here is introducing another source (a new EFI bitmap that represents poisoned memory, in addition to existing
PG_hwpoison) of poisoned pages adds complexity and confusion.David Hildenbrand suggested that we should simplify this as much as possible in a way that we don't have two sources of information w/ inconsistency between them and using memmap as the only source.
e.g.) By consuming the EFI bitmap only once when initializing memmap (to propagate poison information to not just free pages through
__free_pages_core(), but to the all pages on memmap).Quoted 8 lines
Quoted 4 lines
Another challenge here is that we don't have functionality to poison pages early in the boot process before memmap and buddy are ready. So any allocation before initializing them might still allocate poisoned pages.
I've been thinking some more (and will reply in detail to the series), but I do wonder whether memblock should actually be thought about poisoned ranges and refuse to hand them out (reserved/allocated).
I believe this is what the patchset actually did in v2.
Quoted 7 lines
Quoted 3 lines
That way we'll have to either
1) make any architecture that supports kexec selectARCH_KEEP_MEMBLOCK, or 2) make kexec scan the bitmap when allocating the memory.My naive design would be:
1) Teach memblock early about poisoned memory ranges. Don't let it hand them out.
I suggested using a bitmap because ranges are not scalable. Some memory failure modes generate errors repeated across the physical address space (think of a column failure, for instance). It would produce too many ranges to be viable.
We already consult a bitmap in memblock for unaccepted memory. We *can* do the same for poisoned memory.
But I am not convinced it is needed for the initial implementation. Memblock is a small portion of kernel allocations. If we step on broken memory there, the machine is dead and requires repair. It can be improved later if the rate is too high.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 9 lines · Kiryl Shutsemau
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 6 lines · Harry Yoo
I believe this is what the patchset actually did in v2.
That way we'll have to either
1) make any architecture that supports kexec selectARCH_KEEP_MEMBLOCK, or 2) make kexec scan the bitmap when allocating the memory.
Quoted 8 lines
Quoted 3 lines
My naive design would be:
1) Teach memblock early about poisoned memory ranges. Don't let it hand them out.
I suggested using a bitmap because ranges are not scalable. Some memory failure modes generate errors repeated across the physical address space (think of a column failure, for instance). It would produce too many ranges to be viable.
These are in 2M granularity. Is that a real problem?
Quoted 4 lines
We already consult a bitmap in memblock for unaccepted memory. We *can* do the same for poisoned memory.
We can do many things. I don't want it.
Quoted 1 line
But I am not convinced it is needed for the initial implementation.
As I expressed yesterday, I won't tolerate anything in hwpoison code that looks like a hack.
Signature · 3 lines
Cheers, David
Quoted 12 lines · David Hildenbrand (Arm)
Quoted 11 lines · Kiryl Shutsemau
Quoted 10 lines · David Hildenbrand (Arm)
Quoted 6 lines · Harry Yoo
I believe this is what the patchset actually did in v2.
That way we'll have to either
1) make any architecture that supports kexec selectARCH_KEEP_MEMBLOCK, or 2) make kexec scan the bitmap when allocating the memory.My naive design would be:
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
1) Teach memblock early about poisoned memory ranges. Don't let it hand them out.
I suggested using a bitmap because ranges are not scalable. Some memory failure modes generate errors repeated across the physical address space (think of a column failure, for instance). It would produce too many ranges to be viable.
These are in 2M granularity. Is that a real problem?
2M is arbitrary. We can trade finer granularity for the size of the bitmap.
Quoted 5 lines
Quoted 3 lines
We already consult a bitmap in memblock for unaccepted memory. We *can* do the same for poisoned memory.
We can do many things. I don't want it.
What's wrong with it? The last step on allocation from memblock check if the memory is poisoned in bitmap. If it is, try again next range, starting from the next non-poisoned address.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 11 lines · Kiryl Shutsemau
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 5 lines · Kiryl Shutsemau
I suggested using a bitmap because ranges are not scalable. Some memory failure modes generate errors repeated across the physical address space (think of a column failure, for instance). It would produce too many ranges to be viable.
These are in 2M granularity. Is that a real problem?
2M is arbitrary. We can trade finer granularity for the size of the bitmap.
Sure. And if you run into scalability problems you can go to coarse granularity.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
We already consult a bitmap in memblock for unaccepted memory. We *can* do the same for poisoned memory.
We can do many things. I don't want it.
What's wrong with it? The last step on allocation from memblock check if the memory is poisoned in bitmap. If it is, try again next range, starting from the next non-poisoned address.
See my reply to Breno on the other thread.
Signature · 3 lines
Cheers, David