mm/memory-failure: Do not call action_result() on already poisoned pages
Subsystems (5)
PATCH 1/1 mm/memory-failure: Do not call action_result() on already poisoned pages
Calling action_result() on already poisoned pages causes issues:
* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.
Do not call action_result() on already poisoned pages and drop unused MF_MSG_ALREADY_POISONED.
--- include/linux/mm.h | 1 - include/ras/ras_event.h | 1 - mm/memory-failure.c | 3 --- 3 files changed, 5 deletions(-)
include/linux/mm.h +0 -1
@@ -4005,7 +4005,6 @@ enum mf_action_page_type {MF_MSG_BUDDY ,MF_MSG_DAX ,MF_MSG_UNSPLIT_THP , -MF_MSG_ALREADY_POISONED , MF_MSG_UNKNOWN, };
include/ras/ras_event.h +0 -1
@@ -374,7 +374,6 @@ TRACE_EVENT(aer_event,EM (MF_MSG_BUDDY ,"free buddy page" ) \ EM ( MF_MSG_DAX, "dax page" ) \ EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" ) \ - EM ( MF_MSG_ALREADY_POISONED, "already poisoned" ) \ EMe ( MF_MSG_UNKNOWN, "unknown page" )/*
mm/memory-failure.c +0 -3
@@ -948,7 +948,6 @@ static const char * const action_page_types[] = { [MF_MSG_BUDDY] = "free buddy page", [MF_MSG_DAX] = "dax page", [MF_MSG_UNSPLIT_THP] = "unsplit thp", - [MF_MSG_ALREADY_POISONED] = "already poisoned", [MF_MSG_UNKNOWN] = "unknown page", };
@@ -2090,7 +2089,6 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlbif (flags &MF_ACTION_REQUIRED ) { folio =page_folio (p); res =kill_accessing_process (current,folio_pfn (folio), flags); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); }return res; }else if (res == -EBUSY ) {
@@ -2283,7 +2281,6 @@ int memory_failure(unsigned long pfn, int flags) res =kill_accessing_process (current, pfn, flags);if (flags &MF_COUNT_INCREASED )put_page (p); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); goto unlock_mutex; }
Signature · 1 line
2.50.1
Quoted 6 lines · Kyle Meyer
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.
All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.
Hi Kyle,
Patch looks great to me, just one thought...
Alternatively, have you thought about keeping MF_MSG_ALREADY_POISONED but changing action_result for MF_MSG_ALREADY_POISONED?
- don't num_poisoned_pages_inc(pfn)
- don't update_per_node_mf_stats(pfn, result)
- still pr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" in memory_failure and try_memory_failure_hugetlb
This way, all the MF recovery result kernel logs out will be sitting in one place, action_result, instead of scattering around all over the place.
Quoted 65 lines
Fixes b8b9488d50b7 ("mm/memory-failure: improve memory failure action_result messages")Signed-off-by Kyle Meyer <[email protected]>--- include/linux/mm.h | 1 - include/ras/ras_event.h | 1 - mm/memory-failure.c | 3 --- 3 files changed, 5 deletions(-)include/linux/mm.h +0
-1@@ -4005,7 +4005,6 @@ enum mf_action_page_type {MF_MSG_BUDDY ,MF_MSG_DAX ,MF_MSG_UNSPLIT_THP , -MF_MSG_ALREADY_POISONED , MF_MSG_UNKNOWN, };include/ras/ras_event.h +0
-1@@ -374,7 +374,6 @@ TRACE_EVENT(aer_event,EM (MF_MSG_BUDDY ,"free buddy page" ) \ EM ( MF_MSG_DAX, "dax page" ) \ EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" ) \ - EM ( MF_MSG_ALREADY_POISONED, "already poisoned" ) \ EMe ( MF_MSG_UNKNOWN, "unknown page" )/* mm/memory-failure.c +0
-4@@ -948,7 +948,6 @@ static const char * const action_page_types[] = { [MF_MSG_BUDDY] = "free buddy page", [MF_MSG_DAX] = "dax page", [MF_MSG_UNSPLIT_THP] = "unsplit thp", - [MF_MSG_ALREADY_POISONED] = "already poisoned", [MF_MSG_UNKNOWN] = "unknown page", };@@ -2090,7 +2089,6 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlbif (flags &MF_ACTION_REQUIRED ) { folio =page_folio (p); res =kill_accessing_process (current,folio_pfn (folio), flags); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); }return res; }else if (res == -EBUSY ) {@@ -2283,7 +2281,6 @@ int memory_failure(unsigned long pfn, int flags) res =kill_accessing_process (current, pfn, flags);if (flags &MF_COUNT_INCREASED )put_page (p); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); goto unlock_mutex; } --2.50.1
Quoted 18 lines · Jiaqi Yan
Quoted 6 lines · Kyle Meyer
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Quoted 7 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlb
I did consider that approach but I was concerned about passing MF_MSG_ALREADY_POISONED to action_result() with MF_FAILED. The message is a bit misleading.
How about introducing a new MF action result? Maybe MF_NONE? The message could look something like:
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
Quoted 3 lines
This way, all the MF recovery result kernel logs out will be sitting in one place,
action_result, instead of scattering around all over the place.
That sounds better to me.
Quoted 65 lines
Quoted 65 lines
Fixes b8b9488d50b7 ("mm/memory-failure: improve memory failure action_result messages")Signed-off-by Kyle Meyer <[email protected]>--- include/linux/mm.h | 1 - include/ras/ras_event.h | 1 - mm/memory-failure.c | 3 --- 3 files changed, 5 deletions(-)include/linux/mm.h +0
-1@@ -4005,7 +4005,6 @@ enum mf_action_page_type {MF_MSG_BUDDY ,MF_MSG_DAX ,MF_MSG_UNSPLIT_THP , -MF_MSG_ALREADY_POISONED , MF_MSG_UNKNOWN, };include/ras/ras_event.h +0
-1@@ -374,7 +374,6 @@ TRACE_EVENT(aer_event,EM (MF_MSG_BUDDY ,"free buddy page" ) \ EM ( MF_MSG_DAX, "dax page" ) \ EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" ) \ - EM ( MF_MSG_ALREADY_POISONED, "already poisoned" ) \ EMe ( MF_MSG_UNKNOWN, "unknown page" )/* mm/memory-failure.c +0
-4@@ -948,7 +948,6 @@ static const char * const action_page_types[] = { [MF_MSG_BUDDY] = "free buddy page", [MF_MSG_DAX] = "dax page", [MF_MSG_UNSPLIT_THP] = "unsplit thp", - [MF_MSG_ALREADY_POISONED] = "already poisoned", [MF_MSG_UNKNOWN] = "unknown page", };@@ -2090,7 +2089,6 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlbif (flags &MF_ACTION_REQUIRED ) { folio =page_folio (p); res =kill_accessing_process (current,folio_pfn (folio), flags); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); }return res; }else if (res == -EBUSY ) {@@ -2283,7 +2281,6 @@ int memory_failure(unsigned long pfn, int flags) res =kill_accessing_process (current, pfn, flags);if (flags &MF_COUNT_INCREASED )put_page (p); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); goto unlock_mutex; } --2.50.1
Quoted 28 lines · Kyle Meyer
Quoted 18 lines · Jiaqi Yan
Quoted 6 lines · Kyle Meyer
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Quoted 7 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlb
Quoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.
Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence MF_FAILED.
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Quoted 3 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:
Adding MF_NONE sounds fine to me, as long as we correctly document its meaning, which can be subtle.
Let's see what Miaohe's thoughts are.
Quoted 74 lines
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
Quoted 3 lines
This way, all the MF recovery result kernel logs out will be sitting in one place,
action_result, instead of scattering around all over the place.That sounds better to me.
Quoted 65 lines
Quoted 65 lines
Fixes b8b9488d50b7 ("mm/memory-failure: improve memory failure action_result messages")Signed-off-by Kyle Meyer <[email protected]>--- include/linux/mm.h | 1 - include/ras/ras_event.h | 1 - mm/memory-failure.c | 3 --- 3 files changed, 5 deletions(-)include/linux/mm.h +0
-1@@ -4005,7 +4005,6 @@ enum mf_action_page_type {MF_MSG_BUDDY ,MF_MSG_DAX ,MF_MSG_UNSPLIT_THP , -MF_MSG_ALREADY_POISONED , MF_MSG_UNKNOWN, };include/ras/ras_event.h +0
-1@@ -374,7 +374,6 @@ TRACE_EVENT(aer_event,EM (MF_MSG_BUDDY ,"free buddy page" ) \ EM ( MF_MSG_DAX, "dax page" ) \ EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" ) \ - EM ( MF_MSG_ALREADY_POISONED, "already poisoned" ) \ EMe ( MF_MSG_UNKNOWN, "unknown page" )/* mm/memory-failure.c +0
-4@@ -948,7 +948,6 @@ static const char * const action_page_types[] = { [MF_MSG_BUDDY] = "free buddy page", [MF_MSG_DAX] = "dax page", [MF_MSG_UNSPLIT_THP] = "unsplit thp", - [MF_MSG_ALREADY_POISONED] = "already poisoned", [MF_MSG_UNKNOWN] = "unknown page", };@@ -2090,7 +2089,6 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlbif (flags &MF_ACTION_REQUIRED ) { folio =page_folio (p); res =kill_accessing_process (current,folio_pfn (folio), flags); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); }return res; }else if (res == -EBUSY ) {@@ -2283,7 +2281,6 @@ int memory_failure(unsigned long pfn, int flags) res =kill_accessing_process (current, pfn, flags);if (flags &MF_COUNT_INCREASED )put_page (p); -action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); goto unlock_mutex; } --2.50.1
Miaohe Lin
maintainerQuoted 14 lines · Jiaqi Yan
Quoted 13 lines · Kyle Meyer
Quoted 11 lines · Jiaqi Yan
Quoted 6 lines · Kyle Meyer
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.All agreed.
Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.
Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.
There should be other cases that memory_failure failed to isolate the hwpoisoned pages at first time due to various reasons.
Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 7 lines
Quoted 4 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.
Adding a new MF action result sounds good to me. But IMHO MF_NONE might not be that suitable as kill_accessing_process might be called to kill proc in this case, so it's not "NONE".
Thanks.
.
Quoted 62 lines · Miaohe Lin
Quoted 14 lines · Jiaqi Yan
Quoted 13 lines · Kyle Meyer
Quoted 11 lines · Jiaqi Yan
Quoted 6 lines · Kyle Meyer
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.All agreed.
Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.There should be other cases that
memory_failurefailed to isolate the hwpoisoned pages at first time due to various reasons.Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.Adding a new MF action result sounds good to me. But IMHO
MF_NONEmight not be that suitable askill_accessing_processmight be called to kill proc in this case, so it's not "NONE".
OK, would you like a separate MF action result for each case? Maybe MF_ALREADY_POISONED and MF_ALREADY_POISONED_KILLED?
MF_ALREADY_POISONED can be the default and MF_ALREADY_POISONED_KILLED can be used when kill_accessing_process() returns -EHWPOISON.
The log messages could look like...
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
and
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: Process killed
Thanks,
Kyle Meyer
Quoted 4 lines · Kyle Meyer
Quoted 3 lines · Miaohe Lin
Quoted 2 lines · Jiaqi Yan
Quoted 1 line · Kyle Meyer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Jiaqi Yan
Quoted 7 lines · Kyle Meyer
Quoted 6 lines
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.
Assuming this means that the numbers reported from
/sys/devices/system/node/node*/memory_failure/*
do not match certain expectation, right?
If so, could you clarify what is the expectation?
Quoted 63 lines
Quoted 51 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.There should be other cases that
memory_failurefailed to isolate the hwpoisoned pages at first time due to various reasons.Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.Adding a new MF action result sounds good to me. But IMHO
MF_NONEmight not be that suitable askill_accessing_processmight be called to kill proc in this case, so it's not "NONE".OK, would you like a separate MF action result for each case? Maybe
MF_ALREADY_POISONEDandMF_ALREADY_POISONED_KILLED?
Quoted 8 lines
MF_ALREADY_POISONEDcan be the default andMF_ALREADY_POISONED_KILLEDcan be used whenkill_accessing_process()returns-EHWPOISON.The log messages could look like...
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
and
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: Process killed
Agreed with Miaohe that "None" won't work.
"Process killed" sounds okay for MF_MSG_ALREADY_POISONED, but
we need to understand why "Failed" doesn't work for your usecase.
"Failed" means process is killed but page is not successfully isolated which applies to MF_MSG_ALREADY_POISONED case as well.
thanks!
-jane
Quoted 3 lines
Thanks,
Kyle Meyer
Quoted 14 lines · On Mon, Aug 25, 2025 at 03:36:54PM -0700, [email protected] wrote:
Quoted 4 lines · Kyle Meyer
Quoted 3 lines · Miaohe Lin
Quoted 2 lines · Jiaqi Yan
Quoted 1 line · Kyle Meyer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Jiaqi Yan
Quoted 7 lines · Kyle Meyer
Quoted 6 lines
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.
Quoted 5 lines
Assuming this means that the numbers reported from
/sys/devices/system/node/node*/memory_failure/*
do not match certain expectation, right?If so, could you clarify what is the expectation?
Sure, and please let me know if I'm mistaken.
Here's the description of total:
What: /sys/devices/system/node/nodeX/memory_failure/total
Date: January 2023
Contact: Jiaqi Yan <[email protected]>
Description:
The total number of raw poisoned pages (pages containing corrupted data due to memory errors) on a NUMA node.
That should emit the number of poisoned pages on NUMA node X. That's incremented each time update_per_node_mf_stats() is called.
Here's the description of failed:
What: /sys/devices/system/node/nodeX/memory_failure/failed
Date: January 2023
Contact: Jiaqi Yan <[email protected]>
Description:
Of the raw poisoned pages on a NUMA node, how many pages are failed by memory error recovery attempt. This usually means a key recovery operation failed.
That should emit the number of poisoned pages on NUMA node X that could not be recovered because the attempt failed. That's incremented each time update_per_node_mf_stats() is called with MF_FAILED.
We're currently calling action_result() with MF_FAILED each time we encounter a poisoned page (note: the huge page path is a bit different, we only call action_result() with MF_FAILED when MF_ACTION_REQUIRED is set). That, IMO, breaks the descriptions. We already incremented the per NUMA node MF statistics to account for that poisoned page.
Quoted 66 lines
Quoted 66 lines
Quoted 51 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.There should be other cases that
memory_failurefailed to isolate the hwpoisoned pages at first time due to various reasons.Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.Adding a new MF action result sounds good to me. But IMHO
MF_NONEmight not be that suitable askill_accessing_processmight be called to kill proc in this case, so it's not "NONE".OK, would you like a separate MF action result for each case? Maybe
MF_ALREADY_POISONEDandMF_ALREADY_POISONED_KILLED?
MF_ALREADY_POISONEDcan be the default andMF_ALREADY_POISONED_KILLEDcan be used whenkill_accessing_process()returns-EHWPOISON.
Quoted 7 lines
Quoted 5 lines
The log messages could look like...
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
and
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: Process killedAgreed with Miaohe that "None" won't work.
What action is M-F() taking to recover already poisoned pages that don't have MF_ACTION_REQUIRED set?
Quoted 4 lines
"Process killed" sounds okay for
MF_MSG_ALREADY_POISONED, but
we need to understand why "Failed" doesn't work for your usecase. "Failed" means process is killed but page is not successfully isolated which applies toMF_MSG_ALREADY_POISONEDcase as well.
So that accessing process is killed. Why call action_result() with MF_FAILED? Doesn't that indicate we poisoned another page and the recovery attempt failed?
Thanks,
Kyle Meyer
Quoted 49 lines · Kyle Meyer
Quoted 14 lines · On Mon, Aug 25, 2025 at 03:36:54PM -0700, [email protected] wrote:
Quoted 4 lines · Kyle Meyer
Quoted 3 lines · Miaohe Lin
Quoted 2 lines · Jiaqi Yan
Quoted 1 line · Kyle Meyer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Jiaqi Yan
Quoted 7 lines · Kyle Meyer
Quoted 6 lines
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.Quoted 5 lines
Assuming this means that the numbers reported from
/sys/devices/system/node/node*/memory_failure/*
do not match certain expectation, right?If so, could you clarify what is the expectation?
Sure, and please let me know if I'm mistaken.
Here's the description of total:
What:
/sys/devices/system/node/nodeX/memory_failure/total
Date: January 2023
Contact: Jiaqi Yan <[email protected]>
Description:
The total number of raw poisoned pages (pages containing corrupted data due to memory errors) on a NUMA node.That should emit the number of poisoned pages on NUMA node X. That's incremented each time
update_per_node_mf_stats()is called.Here's the description of failed:
What:
/sys/devices/system/node/nodeX/memory_failure/failed
Date: January 2023
Contact: Jiaqi Yan <[email protected]>
Description:
Of the raw poisoned pages on a NUMA node, how many pages are failed by memory error recovery attempt. This usually means a key recovery operation failed.That should emit the number of poisoned pages on NUMA node X that could not be recovered because the attempt failed. That's incremented each time
update_per_node_mf_stats()is called withMF_FAILED.
Quoted 5 lines
We're currently calling
action_result()withMF_FAILEDeach time we encounter a poisoned page (note: the huge page path is a bit different, we only callaction_result()withMF_FAILEDwhenMF_ACTION_REQUIREDis set). That, IMO, breaks the descriptions. We already incremented the per NUMA node MF statistics to account for that poisoned page.
Thanks! My reading is that these numbers are best as hints, I won't take them literately. As you alluded below, kill_accessing_process is applied only if MF_ACTION_REQUIRED is set, though the page is already marked poisoned. Besides, there can be bug that renders a poisoned page not being properly isolated nor being properly categorized. If you're looking for something precise, is there another way? maybe from firmware?
Quoted 69 lines
Quoted 69 lines
Quoted 69 lines
Quoted 52 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.There should be other cases that
memory_failurefailed to isolate the hwpoisoned pages at first time due to various reasons.Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.Adding a new MF action result sounds good to me. But IMHO
MF_NONEmight not be that suitable askill_accessing_processmight be called to kill proc in this case, so it's not "NONE".OK, would you like a separate MF action result for each case? Maybe
MF_ALREADY_POISONEDandMF_ALREADY_POISONED_KILLED?
MF_ALREADY_POISONEDcan be the default andMF_ALREADY_POISONED_KILLEDcan be used whenkill_accessing_process()returns-EHWPOISON.The log messages could look like...
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
and
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: Process killedAgreed with Miaohe that "None" won't work.
What action is M-F() taking to recover already poisoned pages that don't have
MF_ACTION_REQUIREDset?
The action taken toward poisoned page not under MF_ACTION_REQUIRED is typically isolation, that is, remove the pte or mark the pte as poisoned special swap entry, so that subsequent page fault is given a chance to deliver a SIGBUS. That said, things might fail during the process, like
encountering GUP pinned THP page.>
Quoted 7 lines
Quoted 4 lines
"Process killed" sounds okay for
MF_MSG_ALREADY_POISONED, but
we need to understand why "Failed" doesn't work for your usecase. "Failed" means process is killed but page is not successfully isolated which applies toMF_MSG_ALREADY_POISONEDcase as well.So that accessing process is killed. Why call
action_result()withMF_FAILED? Doesn't that indicate we poisoned another page and the recovery attempt failed?
What I recall is that, "recovery" is reserved for page that is clean, isolated, and even by chance, unmapped. "failed" is reserved for page that has been(or might not?) removed from the page table, page might be dirty, certainly mapped, etc. A SIGBUS doesn't make recovery an automatic success.
Others please correct me if I'm mistaken.
thanks,
-jane
Quoted 3 lines
Thanks,
Kyle Meyer
Quoted 57 lines · On Tue, Aug 26, 2025 at 10:24:07AM -0700, [email protected] wrote:
Quoted 49 lines · Kyle Meyer
Quoted 14 lines · On Mon, Aug 25, 2025 at 03:36:54PM -0700, [email protected] wrote:
Quoted 4 lines · Kyle Meyer
Quoted 3 lines · Miaohe Lin
Quoted 2 lines · Jiaqi Yan
Quoted 1 line · Kyle Meyer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Jiaqi Yan
Quoted 7 lines · Kyle Meyer
Quoted 6 lines
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.Quoted 5 lines
Assuming this means that the numbers reported from
/sys/devices/system/node/node*/memory_failure/*
do not match certain expectation, right?If so, could you clarify what is the expectation?
Sure, and please let me know if I'm mistaken.
Here's the description of total:
What:
/sys/devices/system/node/nodeX/memory_failure/total
Date: January 2023
Contact: Jiaqi Yan <[email protected]>
Description:
The total number of raw poisoned pages (pages containing corrupted data due to memory errors) on a NUMA node.That should emit the number of poisoned pages on NUMA node X. That's incremented each time
update_per_node_mf_stats()is called.Here's the description of failed:
What:
/sys/devices/system/node/nodeX/memory_failure/failed
Date: January 2023
Contact: Jiaqi Yan <[email protected]>
Description:
Of the raw poisoned pages on a NUMA node, how many pages are failed by memory error recovery attempt. This usually means a key recovery operation failed.That should emit the number of poisoned pages on NUMA node X that could not be recovered because the attempt failed. That's incremented each time
update_per_node_mf_stats()is called withMF_FAILED.Quoted 5 lines
We're currently calling
action_result()withMF_FAILEDeach time we encounter a poisoned page (note: the huge page path is a bit different, we only callaction_result()withMF_FAILEDwhenMF_ACTION_REQUIREDis set). That, IMO, breaks the descriptions. We already incremented the per NUMA node MF statistics to account for that poisoned page.
Quoted 6 lines
Thanks! My reading is that these numbers are best as hints, I won't take them literately. As you alluded below,
kill_accessing_processis applied only ifMF_ACTION_REQUIREDis set, though the page is already marked poisoned. Besides, there can be bug that renders a poisoned page not being properly isolated nor being properly categorized. If you're looking for something precise, is there another way? maybe from firmware?
Firmware records the number of memory errors that have been detected and reported, but it doesn't record how Linux responded to those memory errors.
Checking the ring buffer, the amount of hardware corrupted memory, and the per NUMA node memory failure statistics is a simple way to determine how Linux responded.
Since commit b8b9488d50b7, that has become unreliable. The same memory error may be reported by multiple sources and now each report increments the amount of hardware corrupted memory and the per NUMA node memory failure statistics. Isn't that a regression?
The per NUMA node memory failure statistics might not always be 100% accurate, but this issue seems preventable.
Quoted 90 lines
Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
Quoted 51 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.There should be other cases that
memory_failurefailed to isolate the hwpoisoned pages at first time due to various reasons.Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.Adding a new MF action result sounds good to me. But IMHO
MF_NONEmight not be that suitable askill_accessing_processmight be called to kill proc in this case, so it's not "NONE".OK, would you like a separate MF action result for each case? Maybe
MF_ALREADY_POISONEDandMF_ALREADY_POISONED_KILLED?
MF_ALREADY_POISONEDcan be the default andMF_ALREADY_POISONED_KILLEDcan be used whenkill_accessing_process()returns-EHWPOISON.The log messages could look like...
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
and
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: Process killedAgreed with Miaohe that "None" won't work.
What action is M-F() taking to recover already poisoned pages that don't have
MF_ACTION_REQUIREDset?The action taken toward poisoned page not under
MF_ACTION_REQUIREDis typically isolation, that is, remove the pte or mark the pte as poisoned special swap entry, so that subsequent page fault is given a chance to deliver a SIGBUS. That said, things might fail during the process, like encountering GUP pinned THP page.>Quoted 7 lines
Quoted 4 lines
"Process killed" sounds okay for
MF_MSG_ALREADY_POISONED, but
we need to understand why "Failed" doesn't work for your usecase. "Failed" means process is killed but page is not successfully isolated which applies toMF_MSG_ALREADY_POISONEDcase as well.So that accessing process is killed. Why call
action_result()withMF_FAILED? Doesn't that indicate we poisoned another page and the recovery attempt failed?
Quoted 6 lines
What I recall is that, "recovery" is reserved for page that is clean, isolated, and even by chance, unmapped. "failed" is reserved for page that has been(or might not?) removed from the page table, page might be dirty, certainly mapped, etc. A SIGBUS doesn't make recovery an automatic success.
Others please correct me if I'm mistaken.
Thank you very much for taking the time to explain everything.
Thanks,
Kyle Meyer
Quoted 69 lines · Kyle Meyer
Quoted 57 lines · On Tue, Aug 26, 2025 at 10:24:07AM -0700, [email protected] wrote:
Quoted 49 lines · Kyle Meyer
Quoted 14 lines · On Mon, Aug 25, 2025 at 03:36:54PM -0700, [email protected] wrote:
Quoted 4 lines · Kyle Meyer
Quoted 3 lines · Miaohe Lin
Quoted 2 lines · Jiaqi Yan
Quoted 1 line · Kyle Meyer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Jiaqi Yan
Quoted 7 lines · Kyle Meyer
Quoted 6 lines
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.Quoted 5 lines
Assuming this means that the numbers reported from
/sys/devices/system/node/node*/memory_failure/*
do not match certain expectation, right?If so, could you clarify what is the expectation?
Sure, and please let me know if I'm mistaken.
Here's the description of total:
What: /sys/devices/system/node/nodeX/memory_failure/total Date: January 2023 Contact: Jiaqi Yan <[email protected]>Description:
The total number of raw poisoned pages (pages containing corrupted data due to memory errors) on a NUMA node.That should emit the number of poisoned pages on NUMA node X. That's incremented each time
update_per_node_mf_stats()is called.Here's the description of failed:
What: /sys/devices/system/node/nodeX/memory_failure/failed Date: January 2023 Contact: Jiaqi Yan <[email protected]>Description:
Of the raw poisoned pages on a NUMA node, how many pages are failed by memory error recovery attempt. This usually means a key recovery operation failed.That should emit the number of poisoned pages on NUMA node X that could not be recovered because the attempt failed. That's incremented each time
update_per_node_mf_stats()is called withMF_FAILED.Quoted 5 lines
We're currently calling
action_result()withMF_FAILEDeach time we encounter a poisoned page (note: the huge page path is a bit different, we only callaction_result()withMF_FAILEDwhenMF_ACTION_REQUIREDis set). That, IMO, breaks the descriptions. We already incremented the per NUMA node MF statistics to account for that poisoned page.Quoted 6 lines
Thanks! My reading is that these numbers are best as hints, I won't take them literately. As you alluded below,
kill_accessing_processis applied only ifMF_ACTION_REQUIREDis set, though the page is already marked poisoned. Besides, there can be bug that renders a poisoned page not being properly isolated nor being properly categorized. If you're looking for something precise, is there another way? maybe from firmware?Firmware records the number of memory errors that have been detected and reported, but it doesn't record how Linux responded to those memory errors.
Quoted 6 lines
Checking the ring buffer, the amount of hardware corrupted memory, and the per NUMA node memory failure statistics is a simple way to determine how Linux responded.
Since commit
b8b9488d50b7, that has become unreliable. The same memory error may be reported by multiple sources and now each report increments the amount of
To me this multiple counting is an existing issue, and what Kyle originally targeted. It seems to me that as long as the mf_action_page_type is MF_MSG_ALREADY_POISONED (or HWPoison flag is set), action_result shouldn't invoke statistic related ops, and mf_results just become irrelevant (wrt /sys/devices/system/node/nodeX/memory_failure/*), right? IWO, we can decouple the two issues:
1. how to update statistics correctly?
2. what log msg to show for recovery result, MF_FAILED or something else/new?
While Kyle fixing issue #1, can we just keep MF_FAILED as is?
Quoted 107 lines
hardware corrupted memory and the per NUMA node memory failure statistics. Isn't that a regression?
The per NUMA node memory failure statistics might not always be 100% accurate, but this issue seems preventable.
Quoted 90 lines
Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
Quoted 51 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.There should be other cases that
memory_failurefailed to isolate the hwpoisoned pages at first time due to various reasons.Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.Adding a new MF action result sounds good to me. But IMHO
MF_NONEmight not be that suitable askill_accessing_processmight be called to kill proc in this case, so it's not "NONE".OK, would you like a separate MF action result for each case? Maybe
MF_ALREADY_POISONEDandMF_ALREADY_POISONED_KILLED?
MF_ALREADY_POISONEDcan be the default andMF_ALREADY_POISONED_KILLEDcan be used whenkill_accessing_process()returns-EHWPOISON.The log messages could look like...
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
and
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: Process killedAgreed with Miaohe that "None" won't work.
What action is M-F() taking to recover already poisoned pages that don't have
MF_ACTION_REQUIREDset?The action taken toward poisoned page not under
MF_ACTION_REQUIREDis typically isolation, that is, remove the pte or mark the pte as poisoned special swap entry, so that subsequent page fault is given a chance to deliver a SIGBUS. That said, things might fail during the process, like encountering GUP pinned THP page.>Quoted 7 lines
Quoted 4 lines
"Process killed" sounds okay for
MF_MSG_ALREADY_POISONED, but
we need to understand why "Failed" doesn't work for your usecase. "Failed" means process is killed but page is not successfully isolated which applies toMF_MSG_ALREADY_POISONEDcase as well.So that accessing process is killed. Why call
action_result()withMF_FAILED? Doesn't that indicate we poisoned another page and the recovery attempt failed?Quoted 6 lines
What I recall is that, "recovery" is reserved for page that is clean, isolated, and even by chance, unmapped. "failed" is reserved for page that has been(or might not?) removed from the page table, page might be dirty, certainly mapped, etc. A SIGBUS doesn't make recovery an automatic success.
Others please correct me if I'm mistaken.
Thank you very much for taking the time to explain everything.
Thanks,
Kyle Meyer
Quoted 87 lines · Jiaqi Yan
Quoted 69 lines · Kyle Meyer
Quoted 57 lines · On Tue, Aug 26, 2025 at 10:24:07AM -0700, [email protected] wrote:
Quoted 49 lines · Kyle Meyer
Quoted 14 lines · On Mon, Aug 25, 2025 at 03:36:54PM -0700, [email protected] wrote:
Quoted 4 lines · Kyle Meyer
Quoted 3 lines · Miaohe Lin
Quoted 2 lines · Jiaqi Yan
Quoted 1 line · Kyle Meyer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Jiaqi Yan
Quoted 7 lines · Kyle Meyer
Quoted 6 lines
Calling
action_result()on already poisoned pages causes issues:* The amount of hardware corrupted memory is incorrectly incremented.
* NUMA node memory failure statistics are incorrectly updated.
* Redundant "already poisoned" messages are printed.Quoted 5 lines
Assuming this means that the numbers reported from
/sys/devices/system/node/node*/memory_failure/*
do not match certain expectation, right?If so, could you clarify what is the expectation?
Sure, and please let me know if I'm mistaken.
Here's the description of total:
What: /sys/devices/system/node/nodeX/memory_failure/total Date: January 2023 Contact: Jiaqi Yan <[email protected]>Description:
The total number of raw poisoned pages (pages containing corrupted data due to memory errors) on a NUMA node.That should emit the number of poisoned pages on NUMA node X. That's incremented each time
update_per_node_mf_stats()is called.Here's the description of failed:
What: /sys/devices/system/node/nodeX/memory_failure/failed Date: January 2023 Contact: Jiaqi Yan <[email protected]>Description:
Of the raw poisoned pages on a NUMA node, how many pages are failed by memory error recovery attempt. This usually means a key recovery operation failed.That should emit the number of poisoned pages on NUMA node X that could not be recovered because the attempt failed. That's incremented each time
update_per_node_mf_stats()is called withMF_FAILED.Quoted 5 lines
We're currently calling
action_result()withMF_FAILEDeach time we encounter a poisoned page (note: the huge page path is a bit different, we only callaction_result()withMF_FAILEDwhenMF_ACTION_REQUIREDis set). That, IMO, breaks the descriptions. We already incremented the per NUMA node MF statistics to account for that poisoned page.Quoted 6 lines
Thanks! My reading is that these numbers are best as hints, I won't take them literately. As you alluded below,
kill_accessing_processis applied only ifMF_ACTION_REQUIREDis set, though the page is already marked poisoned. Besides, there can be bug that renders a poisoned page not being properly isolated nor being properly categorized. If you're looking for something precise, is there another way? maybe from firmware?Firmware records the number of memory errors that have been detected and reported, but it doesn't record how Linux responded to those memory errors.
Quoted 6 lines
Checking the ring buffer, the amount of hardware corrupted memory, and the per NUMA node memory failure statistics is a simple way to determine how Linux responded.
Since commit
b8b9488d50b7, that has become unreliable. The same memory error may be reported by multiple sources and now each report increments the amount ofTo me this multiple counting is an existing issue, and what Kyle originally targeted. It seems to me that as long as the
mf_action_page_typeisMF_MSG_ALREADY_POISONED(or HWPoison flag is set),action_resultshouldn't invoke statistic related ops, andmf_resultsjust become irrelevant (wrt/sys/devices/system/node/nodeX/memory_failure/*), right? IWO, we can decouple the two issues:
1. how to update statistics correctly?
2. what log msg to show for recovery result,MF_FAILEDor something else/new?
Quoted 2 lines
While Kyle fixing issue #1, can we just keep
MF_FAILEDas is?
Sounds good to me.
thanks,
-jane
Quoted 107 lines
Quoted 107 lines
hardware corrupted memory and the per NUMA node memory failure statistics. Isn't that a regression?
The per NUMA node memory failure statistics might not always be 100% accurate, but this issue seems preventable.
Quoted 90 lines
Quoted 68 lines
Quoted 68 lines
Quoted 68 lines
Quoted 51 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
All agreed.
Quoted 3 lines
Do not call
action_result()on already poisoned pages and drop unusedMF_MSG_ALREADY_POISONED.Hi Kyle,
Patch looks great to me, just one thought...
Thanks both.
Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
Alternatively, have you thought about keeping
MF_MSG_ALREADY_POISONEDbut changingaction_resultforMF_MSG_ALREADY_POISONED?
- don'tnum_poisoned_pages_inc(pfn)
- don'tupdate_per_node_mf_stats(pfn, result)
- stillpr_err("%#lx: recovery action for %s: %s\n", ...)
- meanwhile remove "pr_err("%#lx: already hardware poisoned\n", pfn)" inmemory_failureandtry_memory_failure_hugetlbQuoted 3 lines
I did consider that approach but I was concerned about passing
MF_MSG_ALREADY_POISONEDtoaction_result()withMF_FAILED. The message is a bit misleading.Based on my reading the documentation for MF_* in static const char *action_name[]...
Yeah, for file mapped pages, kernel may not have hole-punched or truncated it from the file mapping (shmem and hugetlbfs for example) but that still considered as
MF_RECOVERED, so touching a page with HWPoison flag doesn't mean that page was failed to be recovered previously.Quoted 3 lines
For pages intended to be taken out of the buddy system, touching a page with HWPoison flag does imply it isn't isolated and hence
MF_FAILED.There should be other cases that
memory_failurefailed to isolate the hwpoisoned pages at first time due to various reasons.Quoted 4 lines
In summary, seeing the HWPoison flag again doesn't necessarily indicate what the recovery result was previously; it only indicate kernel won't re-attempt to recover?
Yes, kernel won't re-attempt to or just cannot recover.
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How about introducing a new MF action result? Maybe
MF_NONE? The message could look something like:Adding
MF_NONEsounds fine to me, as long as we correctly document its meaning, which can be subtle.Adding a new MF action result sounds good to me. But IMHO
MF_NONEmight not be that suitable askill_accessing_processmight be called to kill proc in this case, so it's not "NONE".OK, would you like a separate MF action result for each case? Maybe
MF_ALREADY_POISONEDandMF_ALREADY_POISONED_KILLED?
MF_ALREADY_POISONEDcan be the default andMF_ALREADY_POISONED_KILLEDcan be used whenkill_accessing_process()returns-EHWPOISON.The log messages could look like...
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: None
and
Memory failure: 0xXXXXXXXX: recovery action for already poisoned page: Process killedAgreed with Miaohe that "None" won't work.
What action is M-F() taking to recover already poisoned pages that don't have
MF_ACTION_REQUIREDset?The action taken toward poisoned page not under
MF_ACTION_REQUIREDis typically isolation, that is, remove the pte or mark the pte as poisoned special swap entry, so that subsequent page fault is given a chance to deliver a SIGBUS. That said, things might fail during the process, like encountering GUP pinned THP page.>Quoted 7 lines
Quoted 4 lines
"Process killed" sounds okay for
MF_MSG_ALREADY_POISONED, but
we need to understand why "Failed" doesn't work for your usecase. "Failed" means process is killed but page is not successfully isolated which applies toMF_MSG_ALREADY_POISONEDcase as well.So that accessing process is killed. Why call
action_result()withMF_FAILED? Doesn't that indicate we poisoned another page and the recovery attempt failed?Quoted 6 lines
What I recall is that, "recovery" is reserved for page that is clean, isolated, and even by chance, unmapped. "failed" is reserved for page that has been(or might not?) removed from the page table, page might be dirty, certainly mapped, etc. A SIGBUS doesn't make recovery an automatic success.
Others please correct me if I'm mistaken.
Thank you very much for taking the time to explain everything.
Thanks,
Kyle Meyer