RFC: Implement hwpoison on free for soft offlining
Here's a somewhat experimental patch to improve soft offlining in hwpoison, but allowing hwpoison on free for not directly freeable page types. It should work for nearly all left over page types that get eventually freed, so this makes soft offlining nearly universal. The only non handleable page types are now pages that never get freed.
Drawback: It needs an additional page flag. Cannot set hwpoison directly because that would not be "soft" and cause errors.
Since the flags are scarce on 32bit I only enabled it on 64bit.
Comments?
-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 HWPOISON: Implement hwpoison-on-free for soft offlining
From: Andi Kleen <[email protected]>
Hard offlining was always able to offline most kernel pages by setting the HWPoison flag on it and waiting for the next free. This works on all pages that get eventually freed.
This didn't work for soft offlining unfortunately, because it cannot safely set the HWPoison flag on a still alive page. Handle this by introducing a second page flag HWPoisonOnFree that is handled by page free and enables HWPoison then. This does not add any overhead to page free because it can be handled as a bad flag which are already checked for.
Since page flags are scarce on 32bit architectures this is only done on 64bit .
--- include/linux/mm.h | 1 + include/linux/page-flags.h | 15 +++++++++++++- mm/Kconfig | 4 +++ mm/memory-failure.c | 46 +++++++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 2 + 5 files changed, 66 insertions(+), 2 deletions(-)
include/linux/mm.h +1 -0
@@ -1481,6 +1481,7 @@ enum mf_flags {extern void memory_failure (unsigned long pfn,int trapno);extern int __memory_failure (unsigned long pfn,int trapno,int flags);extern int unpoison_memory (unsigned long pfn); +extern void hwpoison_page_on_free (struct page *p);extern int sysctl_memory_failure_early_kill;extern int sysctl_memory_failure_recovery;extern void shake_page (struct page *p,int access);
include/linux/page-flags.h +14 -1
@@ -107,6 +107,9 @@ enum pageflags {#endif #ifdef CONFIG_MEMORY_FAILURE PG_hwpoison, /* hardware poisoned page. Don't touch */ +#ifdef CONFIG_HWPOISON_ON_FREE + PG_hwpoison_on_free, /* hwpoison on free */ +#endif #endif __NR_PAGEFLAGS,
@@ -280,6 +283,15 @@ PAGEFLAG_FALSE(HWPoison)#define __PG_HWPOISON 0#endif +#if defined(CONFIG_MEMORY_FAILURE ) &&BITS_PER_LONG ==64 +PAGEFLAG(HWPoisonOnFree, hwpoison_on_free) +TESTSCFLAG(HWPoisonOnFree, hwpoison_on_free) +#define __PG_HWPOISON_ON_FREE (1UL << PG_hwpoison_on_free) +#else +PAGEFLAG_FALSE(HWPoisonOnFree) +#define __PG_HWPOISON_ON_FREE 0 +#endif +u64 stable_page_flags (struct page *page);static inline int PageUptodate (struct page *page)
@@ -406,7 +418,8 @@ static inline void __ClearPageTail(struct page *page)1 << PG_private |1 << PG_private_2 | \1 << PG_buddy |1 << PG_writeback |1 << PG_reserved | \1 << PG_slab |1 << PG_swapcache |1 << PG_active | \ - 1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON) +1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON | \ + __PG_HWPOISON_ON_FREE)/* * Flags checked when a page is prepped for return by the page allocator.
mm/Kconfig +4 -0
@@ -269,6 +269,10 @@ config MEMORY_FAILURE even when some of its memory has uncorrected errors. This requires special hardware support and typically ECC memory. +config HWPOISON_ON_FREE + depends on MEMORY_FAILURE && 64BIT + def_bool y + config HWPOISON_INJECT tristate "HWPoison pages injector" depends on MEMORY_FAILURE && DEBUG_KERNEL && PROC_FS
mm/memory-failure.c +45 -1
73 lines not on this page · raw message
mm/page_alloc.c +2 -0
@@ -564,6 +564,8 @@ static inline int free_pages_check(struct page *page) (page->mapping != NULL) | (atomic_read(&page->_count) != 0) | (page->flags & PAGE_FLAGS_CHECK_AT_FREE))) { +if (PageHWPoisonOnFree (page)) +hwpoison_page_on_free (page);bad_page (page);return 1 ; }
Signature · 7 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] HWPOISON: Implement hwpoison-on-free for soft offlining
Quoted 13 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Hard offlining was always able to offline most kernel pages by setting the HWPoison flag on it and waiting for the next free. This works on all pages that get eventually freed.
This didn't work for soft offlining unfortunately, because it cannot safely set the HWPoison flag on a still alive page. Handle this by introducing a second page flag HWPoisonOnFree that is handled by page free and enables HWPoison then. This does not add any overhead to page free because it can be handled as a bad flag which are already checked for.
Quoted 2 lines
Since page flags are scarce on 32bit architectures this is only done on 64bit .
I have no problem with the 64bit only page flag.
Quoted 36 lines
Signed-off-by Andi Kleen <[email protected]>--- include/linux/mm.h | 1 + include/linux/page-flags.h | 15 +++++++++++++- mm/Kconfig | 4 +++ mm/memory-failure.c | 46 +++++++++++++++++++++++++++++++++++++++++++- mm/page_alloc.c | 2 + 5 files changed, 66 insertions(+), 2 deletions(-)include/linux/mm.h +1
-0@@ -1481,6 +1481,7 @@ enum mf_flags {extern void memory_failure (unsigned long pfn,int trapno);extern int __memory_failure (unsigned long pfn,int trapno,int flags);extern int unpoison_memory (unsigned long pfn); +extern void hwpoison_page_on_free (struct page *p);extern int sysctl_memory_failure_early_kill;extern int sysctl_memory_failure_recovery;extern void shake_page (struct page *p,int access);include/linux/page-flags.h +3
-0@@ -107,6 +107,9 @@ enum pageflags {#endif #ifdef CONFIG_MEMORY_FAILURE PG_hwpoison, /* hardware poisoned page. Don't touch */ +#ifdef CONFIG_HWPOISON_ON_FREE + PG_hwpoison_on_free, /* hwpoison on free */ +#endif #endif __NR_PAGEFLAGS,
include/linux/page-flags.h PAGEFLAG_FALSE(HWPoison) · line 286
Quoted 4 lines
include/linux/page-flags.h +1
-0@@ -280,6 +283,15 @@ PAGEFLAG_FALSE(HWPoison)#define __PG_HWPOISON 0#endif +#if defined(CONFIG_MEMORY_FAILURE ) &&BITS_PER_LONG ==64
We have the simpler CONFIG_HWPOISON_ON_FREE :)
include/linux/page-flags.h PAGEFLAG_FALSE(HWPoison) · line 291
Quoted 5 lines
include/linux/page-flags.h +5
-0+PAGEFLAG(HWPoisonOnFree, hwpoison_on_free) +TESTSCFLAG(HWPoisonOnFree, hwpoison_on_free) +#define __PG_HWPOISON_ON_FREE (1UL << PG_hwpoison_on_free) +#else +PAGEFLAG_FALSE (HWPoisonOnFree)
Could define SETPAGEFLAG_NOOP(HWPoisonOnFree) too, for eliminating an #ifdef in the .c file.
Quoted 68 lines
include/linux/page-flags.h +5
-1+#define __PG_HWPOISON_ON_FREE 0 +#endif +u64 stable_page_flags (struct page *page);static inline int PageUptodate (struct page *page)@@ -406,7 +418,8 @@ static inline void __ClearPageTail(struct page *page)1 << PG_private |1 << PG_private_2 | \1 << PG_buddy |1 << PG_writeback |1 << PG_reserved | \1 << PG_slab |1 << PG_swapcache |1 << PG_active | \ - 1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON) +1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON | \ + __PG_HWPOISON_ON_FREE)/* * Flags checked when a page is prepped for return by the page allocator. mm/Kconfig +4
-0@@ -269,6 +269,10 @@ config MEMORY_FAILURE even when some of its memory has uncorrected errors. This requires special hardware support and typically ECC memory. +config HWPOISON_ON_FREE + depends on MEMORY_FAILURE && 64BIT + def_bool y + config HWPOISON_INJECT tristate "HWPoison pages injector" depends on MEMORY_FAILURE && DEBUG_KERNEL && PROC_FSmm/memory-failure.c +17
-1@@ -1268,6 +1268,14 @@ int soft_offline_page(struct page *page, int flags)if (ret ==0 ) goto done; +#ifdef CONFIG_HWPOISON_ON_FREE +/* +* When the page gets freed make sure to poison +* it immediately. +*/ +SetPageHWPoisonOnFree (page); +#endif +/* * Page cache page we can handle? */ @@ -1279,7 +1287,15 @@ int soft_offline_page(struct page *page, int flags)shake_page (page,1 );/* -* Did it turn free? +* Did it get poisoned on free? +*/ +if (PageHWPoison (page)) { +pr_info ("soft_offline: %#lx: Page freed through shaking\n" , pfn); +return 0 ; + } + +/* +* Try to grab it as a free page again. */ ret =get_any_page (page, pfn,0 );if (ret <0 )
mm/memory-failure.c int soft_offline_page(struct page *page, int flags) · line 1310
Quoted 8 lines
mm/memory-failure.c +5
-0@@ -1287,12 +1303,23 @@ int soft_offline_page(struct page *page, int flags)if (ret ==0 ) goto done; } + +if (PageHWPoisonOnFree (page)) { +pr_info ("soft_offline: %#lx: Delaying poision of unknown page %lx to free\n" , + pfn, page->flags); +return -EIO ;/* or 0? */
-EIO looks safer because HWPoisonOnFree does not guarantee success.
Quoted 41 lines
mm/memory-failure.c +23
-0+ } +if (!PageLRU (page)) {pr_debug ("soft_offline: %#lx: unknown non LRU page type %lx\n" , pfn, page->flags);return -EIO ; } +/* +* Normal page cache page here. +*/ +lock_page (page);wait_on_page_writeback (page);@@ -1389,3 +1416,20 @@ int is_hwpoison_address(unsigned long addr)return is_hwpoison_entry (entry); }EXPORT_SYMBOL_GPL (is_hwpoison_address); + +#ifdef CONFIG_HWPOISON_ON_FREE + +/* +* HWPoison a page after freeing. +* This is the fallback path for most pages we cannot free early. +*/ +void hwpoison_page_on_free (struct page *p) +{ +get_page (p); +SetPageHWPoison (p); +ClearPageHWPoisonOnFree (p); +pr_info ("MCE: %#lx: Delayed poisioning of page after free\n" , +page_to_pfn (p)); +atomic_long_inc (&mce_bad_pages); +} +#endif mm/page_alloc.c +0
-0
mm/page_alloc.c static inline int free_pages_check(struct page *page) · line 568
Quoted 5 lines
mm/page_alloc.c +2
-0@@ -564,6 +564,8 @@ static inline int free_pages_check(struct page *page) (page->mapping != NULL) | (atomic_read(&page->_count) != 0) | (page->flags & PAGE_FLAGS_CHECK_AT_FREE))) { +if (PageHWPoisonOnFree (page)) +hwpoison_page_on_free (page);
hwpoison_page_on_free() seems to be undefined when CONFIG_HWPOISON_ON_FREE is not defined.
mm/page_alloc.c static inline int free_pages_check(struct page *page) · line 571
Quoted 3 lines
mm/page_alloc.c +0
-0bad_page (page);return 1 ; }
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH] HWPOISON: Implement hwpoison-on-free for soft offlining
Quoted 5 lines · Wu Fengguang
Quoted 3 lines
+#if defined(CONFIG_MEMORY_FAILURE)&&
BITS_PER_LONG== 64We have the simpler
CONFIG_HWPOISON_ON_FREE:)
Leftover from when I didn't have that. Fixed.
Quoted 7 lines
include/linux/page-flags.h PAGEFLAG_FALSE(HWPoison) · line 291
Quoted 5 lines
include/linux/page-flags.h +5
-0+PAGEFLAG(HWPoisonOnFree, hwpoison_on_free) +TESTSCFLAG(HWPoisonOnFree, hwpoison_on_free) +#define __PG_HWPOISON_ON_FREE (1UL<< PG_hwpoison_on_free) +#else +PAGEFLAG_FALSE (HWPoisonOnFree)Could define
SETPAGEFLAG_NOOP(HWPoisonOnFree)too, for eliminating an #ifdef in the .c file.
Ok.
Quoted 9 lines
mm/memory-failure.c int soft_offline_page(struct page *page, int flags) · line 1310
Quoted 7 lines
}mm/memory-failure.c +5
-0+ +if (PageHWPoisonOnFree (page)) { +pr_info ("soft_offline: %#lx: Delaying poision of unknown page %lx to free\n" , + pfn, page->flags); +return -EIO ;/* or 0? */
-EIOlooks safer because HWPoisonOnFree does not guarantee success.
Ok.
Quoted 13 lines
mm/page_alloc.c static inline int free_pages_check(struct page *page) · line 568
Quoted 11 lines
mm/page_alloc.c +2
-0@@ -564,6 +564,8 @@ static inline int free_pages_check(struct page *page) (page->mapping != NULL) | (atomic_read(&page->_count) != 0) | (page->flags& PAGE_FLAGS_CHECK_AT_FREE))) { +if (PageHWPoisonOnFree (page)) +hwpoison_page_on_free (page);
hwpoison_page_on_free()seems to be undefined whenCONFIG_HWPOISON_ON_FREEis not defined.
Yes, but I rely on the compiler never generating the call in this case because the test is zero.
It would fail on a unoptimized build, but the kernel doesn't support that anyways.
Thanks for the review.
-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Wu Fengguang
Reviewed-byRe: [PATCH] HWPOISON: Implement hwpoison-on-free for soft offlining
Quoted 11 lines
Quoted 4 lines
mm/page_alloc.c static inline int free_pages_check(struct page *page) · line 568
Quoted 2 lines
mm/page_alloc.c +2
-0+if (PageHWPoisonOnFree (page)) +hwpoison_page_on_free (page);
hwpoison_page_on_free()seems to be undefined whenCONFIG_HWPOISON_ON_FREEis not defined.Yes, but I rely on the compiler never generating the call in this case because
the test is zero.It would fail on a unoptimized build, but the kernel doesn't support that anyways.
Fair enough.
Quoted 1 line
Thanks for the review.
:)
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Andi Kleen
Here's a somewhat experimental patch to improve soft offlining in hwpoison, but allowing hwpoison on free for not directly freeable page types. It should work for nearly all left over page types that get eventually freed, so this makes soft offlining nearly universal. The only non handleable page types are now pages that never get freed.
Drawback: It needs an additional page flag. Cannot set hwpoison directly because that would not be "soft" and cause errors.
Ping? Any comments on this patch?
Except for the page flag use I think it's nearly a no brainer. A lot of new soft hwpoison capability for very little additional code.
Has anyone a problem using up a 64bit page flag for that?
Thanks,
-Andi
Signature · 7 lines
[email protected] -- Speaking for myself only. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines
Quoted 9 lines · Andi Kleen
Here's a somewhat experimental patch to improve soft offlining in hwpoison, but allowing hwpoison on free for not directly freeable page types. It should work for nearly all left over page types that get eventually freed, so this makes soft offlining nearly universal. The only non handleable page types are now pages that never get freed.
Drawback: It needs an additional page flag. Cannot set hwpoison directly because that would not be "soft" and cause errors.
Quoted 6 lines
Ping? Any comments on this patch?
Except for the page flag use I think it's nearly a no brainer. A lot of new soft hwpoison capability for very little additional code.
Has anyone a problem using up a 64bit page flag for that?
To me, it's no problem if this keep 64bit only. IOW, I only dislike to add 32bit page flags.
Yeah, memory corruption is very crap and i think your effort has a lot of worth :)
offtopic, I don't think CONFIG_MEMORY_FAILURE and CONFIG_HWPOISON_ON_FREE are symmetric nor easy understandable. can you please consider naming change? (example, CONFIG_HWPOISON/CONFIG_HWPOISON_ON_FREE, CONFIG_MEMORY_FAILURE/CONFIG_MEMORY_FAILURE_SOFT_OFFLINE)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines
To me, it's no problem if this keep 64bit only. IOW, I only dislike to add 32bit page flags.
Yeah, memory corruption is very crap and i think your effort has a lot of worth :)
Thanks.
Quoted 6 lines
offtopic, I don't think
CONFIG_MEMORY_FAILUREandCONFIG_HWPOISON_ON_FREEare symmetric nor easy understandable. can you please consider naming change? (example, CONFIG_HWPOISON/CONFIG_HWPOISON_ON_FREE, CONFIG_MEMORY_FAILURE/CONFIG_MEMORY_FAILURE_SOFT_OFFLINE)
memory-failure was the old name before hwpoison as a term was invented by Andrew.
In theory it would make sense to rename everything to "hwpoison" now. But I decided so far the disadvantages from breaking user configurations and the impact from renaming files far outweight the small benefits in clarity.
So right now I prefer to keep the status quo, but name everything new hwpoison.
-Andi
Signature · 7 lines
[email protected] -- Speaking for myself only. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 18 lines
Quoted 5 lines
To me, it's no problem if this keep 64bit only. IOW, I only dislike to add 32bit page flags.
Yeah, memory corruption is very crap and i think your effort has a lot of worth :)
Thanks.
Quoted 6 lines
offtopic, I don't think
CONFIG_MEMORY_FAILUREandCONFIG_HWPOISON_ON_FREEare symmetric nor easy understandable. can you please consider naming change? (example, CONFIG_HWPOISON/CONFIG_HWPOISON_ON_FREE, CONFIG_MEMORY_FAILURE/CONFIG_MEMORY_FAILURE_SOFT_OFFLINE)memory-failure was the old name before hwpoison as a term was invented by Andrew.
Quoted 7 lines
In theory it would make sense to rename everything to "hwpoison" now. But I decided so far the disadvantages from breaking user configurations and the impact from renaming files far outweight the small benefits in clarity.
So right now I prefer to keep the status quo, but name everything new hwpoison.
ok. I've got it. thanks :)
Quoted 7 lines · Andi Kleen
Here's a somewhat experimental patch to improve soft offlining in hwpoison, but allowing hwpoison on free for not directly freeable page types. It should work for nearly all left over page types that get eventually freed, so this makes soft offlining nearly universal. The only non handleable page types are now pages that never get freed.
Quoted 6 lines
Drawback: It needs an additional page flag. Cannot set hwpoison directly because that would not be "soft" and cause errors.
Since the flags are scarce on 32bit I only enabled it on 64bit.
Comments?
I got a couple of positive comments and reviews and no negative comments.
So I assume noone has a problem with using up a 64bit page flag for this. I plan to push this into linux-next after some delay and then prepare it for merge later.
If there are any objections please speak up now.
Thanks,
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.