HWPOISON: Intro
Subsystems (28)
cover HWPOISON: Intro
This the version of hwpoison I intend to submit for 2.6.32.
Only some very minor fixes compared to the last version posted. I integrated one patch from Fengguang that has been reviewed separately.
Passes the hwpoison specific parts of the mce-test test suite
(git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git)
Also available as git tree fromgit://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-mce-2.6.git hwpoison
-Andi
---
Upcoming Intel CPUs have support for recovering from some memory errors (``MCA recovery''). This requires the OS to declare a page "poisoned", kill the processes associated with it and avoid using it in the future.
This patchkit implements the necessary infrastructure in the VM.
To quote the overview comment:
* High level machine check handler. Handles pages reported by the
* hardware as being corrupted usually due to a 2bit ECC memory or cache
* failure.
*
* This focusses on pages detected as corrupted in the background.
* When the current CPU tries to consume corruption the currently
* running process can just be killed directly instead. This implies
* that if the error cannot be handled for some reason it's safe to
* just ignore it because no corruption has been consumed yet. Instead
* when that happens another machine check will happen.
*
* Handles page cache pages in various states. The tricky part
* here is that we can access any page asynchronous to other VM
* users, because memory failures could happen anytime and anywhere,
* possibly violating some of their assumptions. This is why this code
* has to be extremely careful. Generally it tries to use normal locking
* rules, as in get the standard locks, even if that means the
* error handling takes potentially a long time.
*
* Some of the operations here are somewhat inefficient and have non
* linear algorithmic complexity, because the data structures have not
* been optimized for this case. This is in particular the case
* for the mapping from a vma to a process. Since this case is expected
* to be rare we hope we can get away with this.
The code consists of a the high level handler in mm/memory-failure.c, a new page poison bit and various checks in the VM to handle poisoned pages.
The main target right now is KVM guests, but it works for all kinds of applications.
For the KVM use there was need for a new signal type so that KVM can inject the machine check into the guest with the proper address. This in theory allows other applications to handle memory failures too. The expection is that near all applications won't do that, but some very specialized ones might.
This is not fully complete yet, in particular there are still ways to access poison through various ways (crash dump, /proc/kcore etc.) that need to be plugged too.
-Andi
PATCH 1/21 HWPOISON: Add page flag for poisoned pages
Hardware poisoned pages need special handling in the VM and shouldn't be touched again. This requires a new page flag. Define it here.
The page flags wars seem to be over, so it shouldn't be a problem to get a new one.
v2: Add TestSetHWPoison (suggested by Johannes Weiner)
--- include/linux/page-flags.h | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
include/linux/page-flags.h +16 -1
@@ -51,6 +51,9 @@* PG_buddy is set to indicate that the page is free and in the buddy system * (see mm/page_alloc.c). * +* PG_hwpoison indicates that a page got corrupted in hardware and contains +* data with incorrect ECC bits that triggered a machine check. Accessing is +* not safe since it may cause another machine check. Don't touch! */ /*
@@ -102,6 +105,9 @@ enum pageflags {#ifdef CONFIG_IA64_UNCACHED_ALLOCATOR PG_uncached, /* Page has been mapped as uncached */#endif +#ifdef CONFIG_MEMORY_FAILURE + PG_hwpoison,/* hardware poisoned page. Don't touch */ +#endif __NR_PAGEFLAGS,/* Filesystems */
@@ -263,6 +269,15 @@ PAGEFLAG(Uncached, uncached) PAGEFLAG_FALSE(Uncached)#endif +#ifdef CONFIG_MEMORY_FAILURE +PAGEFLAG(HWPoison, hwpoison) +TESTSETFLAG(HWPoison, hwpoison) +#define __PG_HWPOISON (1UL << PG_hwpoison) +#else +PAGEFLAG_FALSE(HWPoison) +#define __PG_HWPOISON 0 +#endif +static inline int PageUptodate (struct page *page) {int ret =test_bit (PG_uptodate, &(page)->flags);
@@ -387,7 +402,7 @@ static inline void __ClearPageTail(struc1 << PG_private |1 << PG_private_2 | \1 << PG_buddy |1 << PG_writeback |1 << PG_reserved | \1 << PG_slab |1 << PG_swapcache |1 << PG_active | \ - 1 << PG_unevictable | __PG_MLOCKED) + 1 << PG_unevictable | __PG_MLOCKED | __PG_HWPOISON)/* * Flags checked when a page is prepped for return by the page allocator.
PATCH 2/21 HWPOISON: Export some rmap vma locking to outside world
In reply to Andi KleenNeeded for later patch that walks rmap entries on its own.
This used to be very frowned upon, but memory-failure.c does some rather specialized rmap walking and rmap has been stable for quite some time, so I think it's ok now to export it.
--- include/linux/rmap.h | 6 ++++++ mm/rmap.c | 4 ++-- 2 files changed, 8 insertions(+), 2 deletions(-)
include/linux/rmap.h +6 -0
@@ -112,6 +112,12 @@ int page_mkclean(struct page *);*/ int try_to_munlock (struct page *); +/* +* Called by memory-failure.c to kill processes. +*/ +struct anon_vma *page_lock_anon_vma (struct page *page); +void page_unlock_anon_vma (struct anon_vma *anon_vma); +#else /* !CONFIG_MMU */ #define anon_vma_init () do {} while (0)
mm/rmap.c +2 -2
@@ -191,7 +191,7 @@ void __init anon_vma_init(void)* Getting a lock on a stable anon_vma from a page off the LRU is * tricky: page_lock_anon_vma rely on RCU to guard against the races. */ -static struct anon_vma *page_lock_anon_vma (struct page *page) +struct anon_vma *page_lock_anon_vma (struct page *page) {struct anon_vma *anon_vma;unsigned long anon_mapping;
@@ -211,7 +211,7 @@ out:return NULL ; } -static void page_unlock_anon_vma (struct anon_vma *anon_vma) +void page_unlock_anon_vma (struct anon_vma *anon_vma) {spin_unlock (&anon_vma->lock);rcu_read_unlock ();
PATCH 3/21 HWPOISON: Add support for poison swap entries v2
In reply to Andi KleenMemory migration uses special swap entry types to trigger special actions on page faults. Extend this mechanism to also support poisoned swap entries, to trigger poison handling on page faults. This allows follow-on patches to prevent processes from faulting in poisoned pages again.
v2: Fix overflow in MAX_SWAPFILES (Fengguang Wu)
v3: Better overflow fix (Hidehiro Kawai)
--- include/linux/swap.h | 34 ++++++++++++++++++++++++++++------ include/linux/swapops.h | 38 ++++++++++++++++++++++++++++++++++++++ mm/swapfile.c | 4 ++-- 3 files changed, 68 insertions(+), 8 deletions(-)
include/linux/swap.h +28 -6
@@ -34,16 +34,38 @@ static inline int current_is_kswapd(void* the type/offset into the pte as 5/27 as well. */ #define MAX_SWAPFILES_SHIFT 5 -#ifndef CONFIG_MIGRATION -#define MAX_SWAPFILES (1 << MAX_SWAPFILES_SHIFT) + +/* +* Use some of the swap files numbers for other purposes. This +* is a convenient way to hook into the VM to trigger special +* actions on faults. +*/ + +/* +* NUMA node memory migration support +*/ +#ifdef CONFIG_MIGRATION +#define SWP_MIGRATION_NUM 2 +#define SWP_MIGRATION_READ (MAX_SWAPFILES + SWP_HWPOISON_NUM) +#define SWP_MIGRATION_WRITE (MAX_SWAPFILES + SWP_HWPOISON_NUM + 1)#else -/* Use last two entries for page migration swap entries */ -#define MAX_SWAPFILES ((1 << MAX_SWAPFILES_SHIFT)-2) -#define SWP_MIGRATION_READ MAX_SWAPFILES -#define SWP_MIGRATION_WRITE (MAX_SWAPFILES + 1) +#define SWP_MIGRATION_NUM 0#endif /* +* Handling of hardware poisoned pages with memory corruption. +*/ +#ifdef CONFIG_MEMORY_FAILURE +#define SWP_HWPOISON_NUM 1 +#define SWP_HWPOISON MAX_SWAPFILES +#else +#define SWP_HWPOISON_NUM 0 +#endif + +#define MAX_SWAPFILES \ + ((1 << MAX_SWAPFILES_SHIFT) - SWP_MIGRATION_NUM - SWP_HWPOISON_NUM) + +/* * Magic header for a swap area. The first part of the union is * what the swap magic looks like for the old (limited to 128MB) * swap area format, the second part of the union adds - in the
include/linux/swapops.h +38 -0
@@ -131,3 +131,41 @@ static inline int is_write_migration_ent#endif +#ifdef CONFIG_MEMORY_FAILURE +/* +* Support for hardware poisoned pages +*/ +static inline swp_entry_t make_hwpoison_entry (struct page *page) +{ +BUG_ON (!PageLocked (page)); +return swp_entry (SWP_HWPOISON ,page_to_pfn (page)); +} + +static inline int is_hwpoison_entry (swp_entry_t entry) +{ +return swp_type (entry) ==SWP_HWPOISON ; +} +#else + +static inline swp_entry_t make_hwpoison_entry (struct page *page) +{ +return swp_entry (0 ,0 ); +} + +static inline int is_hwpoison_entry (swp_entry_t swp) +{ +return 0 ; +} +#endif + +#if defined(CONFIG_MEMORY_FAILURE ) || defined(CONFIG_MIGRATION ) +static inline int non_swap_entry (swp_entry_t entry) +{ +return swp_type (entry) >=MAX_SWAPFILES ; +} +#else +static inline int non_swap_entry (swp_entry_t entry) +{ +return 0 ; +} +#endif
mm/swapfile.c +2 -2
@@ -697,7 +697,7 @@ int free_swap_and_cache(swp_entry_t entrstruct swap_info_struct *p;struct page *page =NULL ; -if (is_migration_entry (entry)) +if (non_swap_entry (entry))return 1 ; p =swap_info_get (entry);
@@ -2083,7 +2083,7 @@ static int __swap_duplicate(swp_entry_tint count;bool has_cache; -if (is_migration_entry (entry)) +if (non_swap_entry (entry))return -EINVAL ; type =swp_type (entry);
PATCH 4/21 HWPOISON: Add new SIGBUS error codes for hardware poison signals
In reply to Andi KleenAdd new SIGBUS codes for reporting machine checks as signals. When the hardware detects an uncorrected ECC error it can trigger these signals.
This is needed for telling KVM's qemu about machine checks that happen to guests, so that it can inject them, but might be also useful for other programs. I find it useful in my test programs.
This patch merely defines the new types.
- Define two new si_codes for SIGBUS. BUS_MCEERR_AO and BUS_MCEERR_AR
* BUS_MCEERR_AO is for "Action Optional" machine checks, which means that some corruption has been detected in the background, but nothing has been consumed so far. The program can ignore those if it wants (but most programs would already get killed)
* BUS_MCEERR_AR is for "Action Required" machine checks. This happens when corrupted data is consumed or the application ran into an area which has been known to be corrupted earlier. These require immediate action and cannot just returned to. Most programs would kill themselves.
- They report the address of the corruption in the user address space in si_addr.
- Define a new si_addr_lsb field that reports the extent of the corruption to user space. That's currently always a (small) page. The user application cannot tell where in this page the corruption happened.
AK: I plan to write a man page update before anyone asks.
--- include/asm-generic/siginfo.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
include/asm-generic/siginfo.h +7 -1
@@ -82,6 +82,7 @@ typedef struct siginfo {#ifdef __ARCH_SI_TRAPNOint _trapno;/* TRAP # which caused the signal */ #endif +short _addr_lsb;/* LSB of the reported address */ } _sigfault;/* SIGPOLL */
@@ -112,6 +113,7 @@ typedef struct siginfo {#ifdef __ARCH_SI_TRAPNO#define si_trapno _sifields._sigfault._trapno#endif +#define si_addr_lsb _sifields._sigfault._addr_lsb#define si_band _sifields._sigpoll._band#define si_fd _sifields._sigpoll._fd
@@ -192,7 +194,11 @@ typedef struct siginfo {#define BUS_ADRALN (__SI_FAULT|1)/* invalid address alignment */ #define BUS_ADRERR (__SI_FAULT|2)/* non-existant physical address */ #define BUS_OBJERR (__SI_FAULT|3)/* object specific hardware error */ -#define NSIGBUS 3 +/* hardware memory error consumed on a machine check: action required */ +#define BUS_MCEERR_AR (__SI_FAULT|4) +/* hardware memory error detected in process but not consumed: action optional*/ +#define BUS_MCEERR_AO (__SI_FAULT|5) +#define NSIGBUS 5/* * SIGTRAP si_codes
PATCH 5/21 HWPOISON: Add basic support for poisoned pages in fault handler v3
In reply to Andi Kleen- Add a new VM_FAULT_HWPOISON error code to handle_mm_fault. Right now architectures have to explicitely enable poison page support, so this is forward compatible to all architectures. They only need to add it when they enable poison page support.
- Add poison page handling in swap in fault code
v2: Add missing delayacct_clear_flag (Hidehiro Kawai)
v3: Really use delayacct_clear_flag (Hidehiro Kawai)
--- include/linux/mm.h | 3 ++- mm/memory.c | 18 +++++++++++++++--- 2 files changed, 17 insertions(+), 4 deletions(-)
mm/memory.c +15 -3
@@ -1319,7 +1319,8 @@ int __get_user_pages(struct task_structif (ret &VM_FAULT_ERROR ) {if (ret &VM_FAULT_OOM )return i ? i : -ENOMEM ; -else if (ret &VM_FAULT_SIGBUS ) +if (ret & + (VM_FAULT_HWPOISON |VM_FAULT_SIGBUS ))return i ? i : -EFAULT ;BUG (); }
@@ -2511,8 +2512,15 @@ static int do_swap_page(struct mm_struct goto out; entry =pte_to_swp_entry (orig_pte); -if (is_migration_entry (entry)) { -migration_entry_wait (mm, pmd, address); +if (unlikely (non_swap_entry (entry))) { +if (is_migration_entry (entry)) { +migration_entry_wait (mm, pmd, address); + }else if (is_hwpoison_entry (entry)) { + ret =VM_FAULT_HWPOISON ; + }else { +print_bad_pte (vma, address, pte,NULL ); + ret =VM_FAULT_OOM ; + } goto out; }delayacct_set_flag (DELAYACCT_PF_SWAPIN );
@@ -2536,6 +2544,10 @@ static int do_swap_page(struct mm_struct/* Had to read the page from swap area: Major fault */ ret =VM_FAULT_MAJOR ;count_vm_event (PGMAJFAULT ); + }else if (PageHWPoison (page)) { + ret =VM_FAULT_HWPOISON ; +delayacct_clear_flag (DELAYACCT_PF_SWAPIN ); + goto out; }lock_page (page);
include/linux/mm.h +2 -1
@@ -685,11 +685,12 @@ static inline int page_mapped(struct pag#define VM_FAULT_SIGBUS 0x0002#define VM_FAULT_MAJOR 0x0004#define VM_FAULT_WRITE 0x0008/* Special case for get_user_pages */ +#define VM_FAULT_HWPOISON 0x0010/* Hit poisoned page */ #define VM_FAULT_NOPAGE 0x0100/* ->fault installed the pte, not return page */ #define VM_FAULT_LOCKED 0x0200/* ->fault locked the returned page */ -#define VM_FAULT_ERROR (VM_FAULT_OOM | VM_FAULT_SIGBUS) +#define VM_FAULT_ERROR (VM_FAULT_OOM | VM_FAULT_SIGBUS | VM_FAULT_HWPOISON)/* * Can be called by the pagefault handler when it gets a VM_FAULT_OOM.
PATCH 6/21 HWPOISON: Add various poison checks in mm/memory.c v2
In reply to Andi KleenBail out early when hardware poisoned pages are found in page fault handling. Since they are poisoned they should not be mapped freshly into processes, because that would cause another (potentially deadly) machine check
This is generally handled in the same way as OOM, just a different error code is returned to the architecture code.
v2: Do a page unlock if needed (Fengguang Wu)
--- mm/memory.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/memory.c +6 -0
@@ -2711,6 +2711,12 @@ static int __do_fault(struct mm_struct *if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE )))return ret; +if (unlikely (PageHWPoison (vmf.page))) { +if (ret &VM_FAULT_LOCKED ) +unlock_page (vmf.page); +return VM_FAULT_HWPOISON ; + } +/* * For consistency in subsequent calls, make the faulted page always * locked.
PATCH 7/21 HWPOISON: x86: Add VM_FAULT_HWPOISON handling to x86 page fault handler v2
In reply to Andi KleenAdd VM_FAULT_HWPOISON handling to the x86 page fault handler. This is very similar to VM_FAULT_OOM, the only difference is that a different si_code is passed to user space and the new addr_lsb field is initialized.
v2: Make the printk more verbose/unique
--- arch/x86/mm/fault.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
arch/x86/mm/fault.c +15 -4
@@ -167,6 +167,7 @@ force_sig_info_fault(int si_signo, int s info.si_errno =0 ; info.si_code = si_code; info.si_addr = (void __user *)address; + info.si_addr_lsb = si_code ==BUS_MCEERR_AR ?PAGE_SHIFT :0 ;force_sig_info (si_signo, &info, tsk); }
@@ -799,10 +800,12 @@ out_of_memory(struct pt_regs *regs, unsi }static void -do_sigbus (struct pt_regs *regs,unsigned long error_code,unsigned long address) +do_sigbus (struct pt_regs *regs,unsigned long error_code,unsigned long address, +unsigned int fault) {struct task_struct *tsk = current;struct mm_struct *mm = tsk->mm; +int code =BUS_ADRERR ;up_read (&mm->mmap_sem);
@@ -818,7 +821,15 @@ do_sigbus(struct pt_regs *regs, unsigned tsk->thread.error_code = error_code; tsk->thread.trap_no =14 ; -force_sig_info_fault (SIGBUS ,BUS_ADRERR , address, tsk); +#ifdef CONFIG_MEMORY_FAILURE +if (fault &VM_FAULT_HWPOISON ) { +printk (KERN_ERR +"MCE: Killing %s:%d due to hardware memory corruption fault at %lx\n" , + tsk->comm, tsk->pid, address); + code =BUS_MCEERR_AR ; + } +#endif +force_sig_info_fault (SIGBUS , code, address, tsk); }static noinline void
@@ -828,8 +839,8 @@ mm_fault_error(struct pt_regs *regs, unsif (fault &VM_FAULT_OOM ) {out_of_memory (regs, error_code, address); }else { -if (fault &VM_FAULT_SIGBUS ) -do_sigbus (regs, error_code, address); +if (fault & (VM_FAULT_SIGBUS |VM_FAULT_HWPOISON )) +do_sigbus (regs, error_code, address, fault);else BUG (); }
PATCH 8/21 HWPOISON: Use bitmask/action code for try_to_unmap behaviour
In reply to Andi Kleentry_to_unmap currently has multiple modi (migration, munlock, normal unmap) which are selected by magic flag variables. The logic is not very straight forward, because each of these flag change multiple behaviours (e.g. migration turns off aging, not only sets up migration ptes etc.) Also the different flags interact in magic ways.
A later patch in this series adds another mode to try_to_unmap, so this becomes quickly unmanageable.
Replace the different flags with a action code (migration, munlock, munmap) and some additional flags as modifiers (ignore mlock, ignore aging). This makes the logic more straight forward and allows easier extension to new behaviours. Change all the caller to declare what they want to do.
This patch is supposed to be a nop in behaviour. If anyone can prove it is not that would be a bug.
--- include/linux/rmap.h | 13 ++++++++++++- mm/migrate.c | 2 +- mm/rmap.c | 40 ++++++++++++++++++++++------------------ mm/vmscan.c | 2 +- 4 files changed, 36 insertions(+), 21 deletions(-)
include/linux/rmap.h +12 -1
@@ -85,7 +85,18 @@ static inline void page_dup_rmap(struct*/ int page_referenced (struct page *,int is_locked,struct mem_cgroup *cnt,unsigned long *vm_flags); -int try_to_unmap (struct page *,int ignore_refs); +enum ttu_flags { +TTU_UNMAP =0 ,/* unmap mode */ +TTU_MIGRATION =1 ,/* migration mode */ +TTU_MUNLOCK =2 ,/* munlock mode */ +TTU_ACTION_MASK =0xff , + +TTU_IGNORE_MLOCK = (1 <<8 ),/* ignore mlock */ +TTU_IGNORE_ACCESS = (1 <<9 ),/* don't age */ +}; +#define TTU_ACTION (x) ((x) & TTU_ACTION_MASK) + +int try_to_unmap (struct page *,enum ttu_flags flags);/* * Called from mm/filemap_xip.c to unmap empty zero page
mm/rmap.c +22 -18
@@ -774,7 +774,7 @@ void page_remove_rmap(struct page *page)* repeatedly from either try_to_unmap_anon or try_to_unmap_file. */ static int try_to_unmap_one (struct page *page,struct vm_area_struct *vma, -int migration) +enum ttu_flags flags) {struct mm_struct *mm = vma->vm_mm;unsigned long address;
@@ -796,11 +796,13 @@ static int try_to_unmap_one(struct page* If it's recently referenced (perhaps page_referenced * skipped over this mm) then we should reactivate it. */ -if (!migration) { +if (!(flags &TTU_IGNORE_MLOCK )) {if (vma->vm_flags &VM_LOCKED ) { ret =SWAP_MLOCK ; goto out_unmap; } + } +if (!(flags &TTU_IGNORE_ACCESS )) {if (ptep_clear_flush_young_notify (vma, address, pte)) { ret =SWAP_FAIL ; goto out_unmap;
@@ -840,12 +842,12 @@ static int try_to_unmap_one(struct page* pte. do_swap_page() will wait until the migration * pte is removed and then restart fault handling. */ -BUG_ON (!migration); +BUG_ON (TTU_ACTION (flags) !=TTU_MIGRATION ); entry =make_migration_entry (page,pte_write (pteval)); }set_pte_at (mm, address, pte,swp_entry_to_pte (entry));BUG_ON (pte_file (*pte)); - }else if (PAGE_MIGRATION && migration) { + }else if (PAGE_MIGRATION && (TTU_ACTION (flags) ==TTU_MIGRATION )) {/* Establish migration entry for a file page */ swp_entry_t entry; entry =make_migration_entry (page,pte_write (pteval));
@@ -1014,12 +1016,13 @@ static int try_to_mlock_page(struct page* vm_flags for that VMA. That should be OK, because that vma shouldn't be * 'LOCKED. */ -static int try_to_unmap_anon (struct page *page,int unlock,int migration) +static int try_to_unmap_anon (struct page *page,enum ttu_flags flags) {struct anon_vma *anon_vma;struct vm_area_struct *vma;unsigned int mlocked =0 ;int ret =SWAP_AGAIN ; +int unlock =TTU_ACTION (flags) ==TTU_MUNLOCK ;if (MLOCK_PAGES &&unlikely (unlock)) ret =SWAP_SUCCESS ;/* default for try_to_munlock() */
@@ -1035,7 +1038,7 @@ static int try_to_unmap_anon(struct pagecontinue ;/* must visit all unlocked vmas */ ret =SWAP_MLOCK ;/* saw at least one mlocked vma */ }else { - ret =try_to_unmap_one (page, vma, migration); + ret =try_to_unmap_one (page, vma, flags);if (ret ==SWAP_FAIL || !page_mapped (page))break ; }
@@ -1059,8 +1062,7 @@ static int try_to_unmap_anon(struct page/** * try_to_unmap_file - unmap/unlock file page using the object-based rmap method * @page: the page to unmap/unlock -* @unlock: request for unlock rather than unmap [unlikely] -* @migration: unmapping for migration - ignored if @unlock +* @flags: action and flags * * Find all the mappings of a page using the mapping pointer and the vma chains * contained in the address_space struct it points to.
@@ -1072,7 +1074,7 @@ static int try_to_unmap_anon(struct page* vm_flags for that VMA. That should be OK, because that vma shouldn't be * 'LOCKED. */ -static int try_to_unmap_file (struct page *page,int unlock,int migration) +static int try_to_unmap_file (struct page *page,enum ttu_flags flags) {struct address_space *mapping = page->mapping;pgoff_t pgoff = page->index << (PAGE_CACHE_SHIFT -PAGE_SHIFT );
@@ -1084,6 +1086,7 @@ static int try_to_unmap_file(struct pageunsigned long max_nl_size =0 ;unsigned int mapcount;unsigned int mlocked =0 ; +int unlock =TTU_ACTION (flags) ==TTU_MUNLOCK ;if (MLOCK_PAGES &&unlikely (unlock)) ret =SWAP_SUCCESS ;/* default for try_to_munlock() */
@@ -1096,7 +1099,7 @@ static int try_to_unmap_file(struct pagecontinue ;/* must visit all vmas */ ret =SWAP_MLOCK ; }else { - ret =try_to_unmap_one (page, vma, migration); + ret =try_to_unmap_one (page, vma, flags);if (ret ==SWAP_FAIL || !page_mapped (page)) goto out; }
@@ -1121,7 +1124,8 @@ static int try_to_unmap_file(struct page ret =SWAP_MLOCK ;/* leave mlocked == 0 */ goto out;/* no need to look further */ } -if (!MLOCK_PAGES && !migration && (vma->vm_flags &VM_LOCKED )) +if (!MLOCK_PAGES && !(flags &TTU_IGNORE_MLOCK ) && + (vma->vm_flags &VM_LOCKED ))continue ; cursor = (unsigned long ) vma->vm_private_data;if (cursor > max_nl_cursor)
@@ -1155,7 +1159,7 @@ static int try_to_unmap_file(struct page do { list_for_each_entry(vma, &mapping->i_mmap_nonlinear, shared.vm_set.list) { - if (!MLOCK_PAGES && !migration && + if (!MLOCK_PAGES && !(flags & TTU_IGNORE_MLOCK) && (vma->vm_flags & VM_LOCKED)) continue; cursor = (unsigned long) vma->vm_private_data;
@@ -1195,7 +1199,7 @@ out:/** * try_to_unmap - try to remove all page table mappings to a page * @page: the page to get unmapped -* @migration: migration flag +* @flags: action and flags * * Tries to remove all the page table entries which are mapping this * page, used in the pageout path. Caller must hold the page lock.
@@ -1206,16 +1210,16 @@ out:* SWAP_FAIL - the page is unswappable * SWAP_MLOCK - page is mlocked. */ -int try_to_unmap (struct page *page,int migration) +int try_to_unmap (struct page *page,enum ttu_flags flags) {int ret;BUG_ON (!PageLocked (page));if (PageAnon (page)) - ret =try_to_unmap_anon (page,0 , migration); + ret =try_to_unmap_anon (page, flags);else - ret =try_to_unmap_file (page,0 , migration); + ret =try_to_unmap_file (page, flags);if (ret !=SWAP_MLOCK && !page_mapped (page)) ret =SWAP_SUCCESS ;return ret;
@@ -1240,8 +1244,8 @@ int try_to_munlock(struct page *page)VM_BUG_ON (!PageLocked (page) ||PageLRU (page));if (PageAnon (page)) -return try_to_unmap_anon (page,1 ,0 ); +return try_to_unmap_anon (page,TTU_MUNLOCK );else -return try_to_unmap_file (page,1 ,0 ); +return try_to_unmap_file (page,TTU_MUNLOCK ); }
mm/vmscan.c +1 -1
@@ -659,7 +659,7 @@ static unsigned long shrink_page_list(st* processes. Try to unmap it here. */ if (page_mapped (page) && mapping) { -switch (try_to_unmap (page,0 )) { +switch (try_to_unmap (page,TTU_UNMAP )) {case SWAP_FAIL : goto activate_locked;case SWAP_AGAIN :
mm/migrate.c +1 -1
@@ -669,7 +669,7 @@ static int unmap_and_move(new_page_t get }/* Establish migration ptes or remove ptes */ -try_to_unmap (page,1 ); +try_to_unmap (page,TTU_MIGRATION |TTU_IGNORE_MLOCK |TTU_IGNORE_ACCESS );if (!page_mapped (page)) rc =move_to_new_page (newpage, page);
PATCH 9/21 HWPOISON: Handle hardware poisoned pages in try_to_unmap
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
When a page has the poison bit set replace the PTE with a poison entry. This causes the right error handling to be done later when a process runs into it.
v2: add a new flag to not do that (needed for the memory-failure handler later) (Fengguang)
v3: remove unnecessary is_migration_entry() test (Fengguang, Minchan)
--- include/linux/rmap.h | 1 + mm/rmap.c | 9 ++++++++- 2 files changed, 9 insertions(+), 1 deletion(-)
mm/rmap.c +8 -1
@@ -820,7 +820,14 @@ static int try_to_unmap_one(struct page/* Update high watermark before we lower rss */ update_hiwater_rss (mm); -if (PageAnon (page)) { +if (PageHWPoison (page) && !(flags &TTU_IGNORE_HWPOISON )) { +if (PageAnon (page)) +dec_mm_counter (mm, anon_rss); +else +dec_mm_counter (mm, file_rss); +set_pte_at (mm, address, pte, +swp_entry_to_pte (make_hwpoison_entry (page))); + }else if (PageAnon (page)) {swp_entry_t entry = { .val =page_private (page) };if (PageSwapCache (page)) {
include/linux/rmap.h +1 -0
@@ -93,6 +93,7 @@ enum ttu_flags {TTU_IGNORE_MLOCK = (1 <<8 ),/* ignore mlock */ TTU_IGNORE_ACCESS = (1 <<9 ),/* don't age */ + TTU_IGNORE_HWPOISON = (1 << 10),/* corrupted page is recoverable */ };#define TTU_ACTION (x) ((x) & TTU_ACTION_MASK)
PATCH 10/21 HWPOISON: check and isolate corrupted free pages v2
In reply to Andi KleenFrom: Wu Fengguang <[email protected]>
If memory corruption hits the free buddy pages, we can safely ignore them. No one will access them until page allocation time, then prep_new_page() will automatically check and isolate PG_hwpoison page for us (for 0-order allocation).
This patch expands prep_new_page() to check every component page in a high order page allocation, in order to completely stop PG_hwpoison pages from being recirculated.
Note that the common case -- only allocating a single page, doesn't do any more work than before. Allocating > order 0 does a bit more work, but that's relatively uncommon.
This simple implementation may drop some innocent neighbor pages, hopefully it is not a big problem because the event should be rare enough.
This patch adds some runtime costs to high order page users.
[AK: Improved description]
v2: Andi Kleen:
Port to -mm code
Move check into separate function.
Don't dump stack in bad_pages for hwpoisoned pages.
--- mm/page_alloc.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
mm/page_alloc.c +19 -1
@@ -234,6 +234,12 @@ static void bad_page(struct page *page)static unsigned long nr_shown;static unsigned long nr_unshown; +/* Don't complain about poisoned pages */ +if (PageHWPoison (page)) { +__ClearPageBuddy (page); +return ; + } +/* * Allow a burst of 60 reports, then keep quiet for that minute; * or allow a steady drip of one report per second.
@@ -646,7 +652,7 @@ static inline void expand(struct zone *z/* * This page is about to be returned from the page allocator */ -static int prep_new_page (struct page *page,int order,gfp_t gfp_flags) +static inline int check_new_page (struct page *page) { if (unlikely(page_mapcount(page) | (page->mapping != NULL) |
@@ -655,6 +661,18 @@ static int prep_new_page(struct page *pabad_page (page);return 1 ; } +return 0 ; +} + +static int prep_new_page (struct page *page,int order,gfp_t gfp_flags) +{ +int i; + +for (i =0 ; i < (1 << order); i++) { +struct page *p = page + i; +if (unlikely (check_new_page (p))) +return 1 ; + }set_page_private (page,0 );set_page_refcounted (page);
PATCH 11/21 HWPOISON: Refactor truncate to allow direct truncating of page v2
In reply to Andi KleenFrom: Nick Piggin <[email protected]>
Extract out truncate_inode_page() out of the truncate path so that it can be used by memory-failure.c
[AK: description, headers, fix typos]
v2: Some white space changes from Fengguang Wu
--- include/linux/mm.h | 2 ++ mm/truncate.c | 29 +++++++++++++++-------------- 2 files changed, 17 insertions(+), 14 deletions(-)
mm/truncate.c +15 -14
@@ -93,11 +93,11 @@ EXPORT_SYMBOL(cancel_dirty_page);* its lock, b) when a concurrent invalidate_mapping_pages got there first and * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space. */ -static void +static int truncate_complete_page (struct address_space *mapping,struct page *page) {if (page->mapping != mapping) -return ; +return -EIO ;if (page_has_private (page))do_invalidatepage (page,0 );
@@ -108,6 +108,7 @@ truncate_complete_page(struct address_spremove_from_page_cache (page);ClearPageMappedToDisk (page);page_cache_release (page);/* pagecache ref */ +return 0 ; }/*
@@ -135,6 +136,16 @@ invalidate_complete_page(struct address_return ret; } +int truncate_inode_page (struct address_space *mapping,struct page *page) +{ +if (page_mapped (page)) { +unmap_mapping_range (mapping, + (loff_t )page->index <<PAGE_CACHE_SHIFT , +PAGE_CACHE_SIZE ,0 ); + } +return truncate_complete_page (mapping, page); +} +/** * truncate_inode_pages - truncate range of pages specified by start & end byte offsets * @mapping: mapping to truncate
@@ -196,12 +207,7 @@ void truncate_inode_pages_range(struct aunlock_page (page);continue ; } -if (page_mapped (page)) { -unmap_mapping_range (mapping, - (loff_t )page_index<<PAGE_CACHE_SHIFT , -PAGE_CACHE_SIZE ,0 ); - } -truncate_complete_page (mapping, page); +truncate_inode_page (mapping, page);unlock_page (page); }pagevec_release (&pvec);
@@ -238,15 +244,10 @@ void truncate_inode_pages_range(struct abreak ;lock_page (page);wait_on_page_writeback (page); -if (page_mapped (page)) { -unmap_mapping_range (mapping, - (loff_t )page->index<<PAGE_CACHE_SHIFT , -PAGE_CACHE_SIZE ,0 ); - } +truncate_inode_page (mapping, page);if (page->index > next) next = page->index; next++; -truncate_complete_page (mapping, page);unlock_page (page); }pagevec_release (&pvec);
include/linux/mm.h +2 -0
@@ -794,6 +794,8 @@ static inline void unmap_shared_mapping_extern int vmtruncate (struct inode * inode,loff_t offset);extern int vmtruncate_range (struct inode * inode,loff_t offset,loff_t end); +int truncate_inode_page (struct address_space *mapping,struct page *page); +#ifdef CONFIG_MMU extern int handle_mm_fault (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,unsigned int flags);
PATCH 12/21 HWPOISON: Add invalidate_inode_page
In reply to Andi KleenFrom: Wu Fengguang <[email protected]>
Add a simple way to invalidate a single page This is just a refactoring of the truncate.c code. Originally from Fengguang, modified by Andi Kleen.
--- include/linux/mm.h | 2 ++ mm/truncate.c | 26 ++++++++++++++++++++------ 2 files changed, 22 insertions(+), 6 deletions(-)
include/linux/mm.h +2 -0
@@ -796,6 +796,8 @@ extern int vmtruncate_range(struct inodeint truncate_inode_page (struct address_space *mapping,struct page *page); +int invalidate_inode_page (struct page *page); +#ifdef CONFIG_MMU extern int handle_mm_fault (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,unsigned int flags);
mm/truncate.c +20 -6
@@ -146,6 +146,24 @@ int truncate_inode_page(struct address_sreturn truncate_complete_page (mapping, page); } +/* +* Safely invalidate one page from its pagecache mapping. +* It only drops clean, unused pages. The page must be locked. +* +* Returns 1 if the page is successfully invalidated, otherwise 0. +*/ +int invalidate_inode_page (struct page *page) +{ +struct address_space *mapping =page_mapping (page); +if (!mapping) +return 0 ; +if (PageDirty (page) ||PageWriteback (page)) +return 0 ; +if (page_mapped (page)) +return 0 ; +return invalidate_complete_page (mapping, page); +} +/** * truncate_inode_pages - truncate range of pages specified by start & end byte offsets * @mapping: mapping to truncate
@@ -312,12 +330,8 @@ unsigned long invalidate_mapping_pages(sif (lock_failed)continue ; -if (PageDirty (page) ||PageWriteback (page)) - goto unlock; -if (page_mapped (page)) - goto unlock; - ret +=invalidate_complete_page (mapping, page); -unlock: + ret +=invalidate_inode_page (page); +unlock_page (page);if (next > end)break ;
PATCH 13/21 HWPOISON: Define a new error_remove_page address space op for async truncation
In reply to Andi KleenTruncating metadata pages is not safe right now before we haven't audited all file systems.
To enable truncation only for data address space define a new address_space callback error_remove_page.
This is used for memory_failure.c memory error handling.
This can be then set to truncate_inode_page()
This patch just defines the new operation and adds documentation.
Callers and users come in followon patches.
--- Documentation/filesystems/vfs.txt | 7 +++++++ include/linux/fs.h | 1 + include/linux/mm.h | 1 + mm/truncate.c | 17 +++++++++++++++++ 4 files changed, 26 insertions(+)
include/linux/fs.h +1 -0
@@ -595,6 +595,7 @@ struct address_space_operations {int (*launder_page) (struct page *);int (*is_partially_uptodate) (struct page *,read_descriptor_t *,unsigned long ); +int (*error_remove_page)(struct address_space *,struct page *); };/*
Documentation/filesystems/vfs.txt +7 -0
@@ -536,6 +536,7 @@ struct address_space_operations { /* migrate the contents of a page to the specified target */ int (*migratepage) (struct page *, struct page *); int (*launder_page) (struct page *); + int (*error_remove_page) (struct mapping *mapping, struct page *page); }; writepage: called by the VM to write a dirty page to backing store.
@@ -694,6 +695,12 @@ struct address_space_operations { prevent redirtying the page, it is kept locked during the whole operation. + error_remove_page: normally set to generic_error_remove_page if truncation + is ok for this address space. Used for memory failure handling. + Setting this implies you deal with pages going away under you, + unless you have them locked or reference counts increased. + + The File Object ===============
mm/truncate.c +17 -0
@@ -147,6 +147,23 @@ int truncate_inode_page(struct address_s }/* +* Used to get rid of pages on hardware memory corruption. +*/ +int generic_error_remove_page (struct address_space *mapping,struct page *page) +{ +if (!mapping) +return -EINVAL ; +/* +* Only punch for normal data pages for now. +* Handling other types like directories would need more auditing. +*/ +if (!S_ISREG (mapping->host->i_mode)) +return -EIO ; +return truncate_inode_page (mapping, page); +} +EXPORT_SYMBOL (generic_error_remove_page); + +/* * Safely invalidate one page from its pagecache mapping. * It only drops clean, unused pages. The page must be locked. *
include/linux/mm.h +1 -0
@@ -795,6 +795,7 @@ extern int vmtruncate(struct inode * inoextern int vmtruncate_range (struct inode * inode,loff_t offset,loff_t end);int truncate_inode_page (struct address_space *mapping,struct page *page); +int generic_error_remove_page (struct address_space *mapping,struct page *page);int invalidate_inode_page (struct page *page);
PATCH 14/21 HWPOISON: shmem: call set_page_dirty() with locked page
In reply to Andi KleenFrom: Wu Fengguang <[email protected]>
The dirtying of page and set_page_dirty() can be moved into the page lock.
- In shmem_write_end(), the page was dirtied while the page lock was held, but it's being marked dirty just after dropping the page lock.
- In shmem_symlink(), both dirtying and marking can be moved into page lock.
It's valuable for the hwpoison code to know whether one bad page can be dropped without losing data. It mainly judges by testing the PG_dirty bit after taking the page lock. So it becomes important that the dirtying of page and the marking of dirtiness are both done inside the page lock. Which is a common practice, but sadly not a rule.
The noticeable exceptions are
- mapped pages
- pages with buffer_heads
The above pages could go dirty at any time. Fortunately the hwpoison will unmap the page and release the buffer_heads beforehand anyway.
Many other types of pages (eg. metadata pages) can also be dirtied at will by their owners, the hwpoison code cannot do meaningful things to them anyway. Only the dirtiness of pagecache pages owned by regular files are interested.
--- mm/shmem.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/shmem.c +2 -2
@@ -1630,8 +1630,8 @@ shmem_write_end(struct file *file, strucif (pos + copied > inode->i_size)i_size_write (inode, pos + copied); -unlock_page (page);set_page_dirty (page); +unlock_page (page);page_cache_release (page);return copied;
@@ -1968,13 +1968,13 @@ static int shmem_symlink(struct inode *diput (inode);return error; } -unlock_page (page); inode->i_mapping->a_ops = &shmem_aops; inode->i_op = &shmem_symlink_inode_operations; kaddr =kmap_atomic (page,KM_USER0 );memcpy (kaddr, symname, len);kunmap_atomic (kaddr,KM_USER0 );set_page_dirty (page); +unlock_page (page);page_cache_release (page); }if (dir->i_mode &S_ISGID )
Peter Zijlstra
maintainerRe: [PATCH] [14/21] HWPOISON: shmem: call set_page_dirty() with locked page
Quoted 20 lines · Andi Kleen
From: Wu Fengguang <[email protected]>
The dirtying of page and
set_page_dirty()can be moved into the page lock.- In
shmem_write_end(), the page was dirtied while the page lock was held, but it's being marked dirty just after dropping the page lock.
- Inshmem_symlink(), both dirtying and marking can be moved into page lock.It's valuable for the hwpoison code to know whether one bad page can be dropped without losing data. It mainly judges by testing the
PG_dirtybit after taking the page lock. So it becomes important that the dirtying of page and the marking of dirtiness are both done inside the page lock. Which is a common practice, but sadly not a rule.The noticeable exceptions are
- mapped pages
- pages withbuffer_heads
The above pages could go dirty at any time. Fortunately the hwpoison will unmap the page and release thebuffer_headsbeforehand anyway.
Quoted 3 lines
Many other types of pages (eg. metadata pages) can also be dirtied at will by their owners, the hwpoison code cannot do meaningful things to them anyway. Only the dirtiness of pagecache pages owned by regular files are interested.
Its not clear to me what this patch achieves.
Will it, from now on, be mandatory to dirty pages under PG_lock, and does this patch correct the last few such cases that didn't yet do that, or does it still leave holes?
Can we document the requirements and or holes to it someplace near set_page_dirty()?
Quoted 37 lines
Acked-by Hugh Dickins <[email protected]>Signed-off-by Andi Kleen <[email protected]>Reviewed-by WANG Cong <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- mm/shmem.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Index:
linux/mm/shmem.c
===================================================================mm/shmem.c +2
-2@@ -1630,8 +1630,8 @@ shmem_write_end(struct file *file, strucif (pos + copied > inode->i_size)i_size_write (inode, pos + copied); -unlock_page (page);set_page_dirty (page); +unlock_page (page);page_cache_release (page);return copied;@@ -1968,13 +1968,13 @@ static int shmem_symlink(struct inode *diput (inode);return error; } -unlock_page (page); inode->i_mapping->a_ops = &shmem_aops; inode->i_op = &shmem_symlink_inode_operations; kaddr =kmap_atomic (page,KM_USER0 );memcpy (kaddr, symname, len);kunmap_atomic (kaddr,KM_USER0 );set_page_dirty (page); +unlock_page (page);page_cache_release (page); }if (dir->i_mode &S_ISGID )
Re: [PATCH] [14/21] HWPOISON: shmem: call set_page_dirty() with locked page
Quoted 1 line
Will it, from now on, be mandatory to dirty pages under
PG_lock, and
Only data pages with mapping.
It's not a strict requirement, but if it's not done then in theory you could not propagate an error if the page gets an memory error at exactly the wrong time. It's probably only worth caring if you expect to have a lot of pages like this.
Quoted 2 lines
does this patch correct the last few such cases that didn't yet do that, or does it still leave holes?
I believe in metadata it's still widely there, but hwpoison ignores those pages anyways.
Quoted 3 lines
Can we document the requirements and or holes to it someplace near
set_page_dirty()?
Ok I'll add a comment there.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
PATCH 15/21 HWPOISON: Add PR_MCE_KILL prctl to control early kill behaviour per process
In reply to Andi KleenThis allows processes to override their early/late kill behaviour on hardware memory errors.
Typically applications which are memory error aware is better of with early kill (see the error as soon as possible), all others with late kill (only
see the error when the error is really impacting execution)
There's a global sysctl, but this way an application can set its specific policy.
We're using two bits, one to signify that the process stated its intention and that
I also made the prctl future proof by enforcing the unused arguments are 0.
The state is inherited to children for now. I've been considering to reset it on exec, but not done for now (TBD).
Note this makes us officially run out of process flags on 32bit, but the next patch can easily add another field.
Manpage patch will be supplied separately.
--- include/linux/prctl.h | 2 ++ include/linux/sched.h | 2 ++ kernel/sys.c | 22 ++++++++++++++++++++++ 3 files changed, 26 insertions(+)
include/linux/sched.h +2 -0
@@ -1675,6 +1675,7 @@ extern cputime_t task_gtime(struct task_#define PF_EXITPIDONE 0x00000008/* pi exit done on shut down */ #define PF_VCPU 0x00000010/* I'm a virtual CPU */ #define PF_FORKNOEXEC 0x00000040/* forked but didn't exec */ +#define PF_MCE_PROCESS 0x00000080/* process policy on mce errors */ #define PF_SUPERPRIV 0x00000100/* used super-user privileges */ #define PF_DUMPCORE 0x00000200/* dumped core */ #define PF_SIGNALED 0x00000400/* killed by a signal */
@@ -1694,6 +1695,7 @@ extern cputime_t task_gtime(struct task_#define PF_SPREAD_PAGE 0x01000000/* Spread page cache over cpuset */ #define PF_SPREAD_SLAB 0x02000000/* Spread some slab caches over cpuset */ #define PF_THREAD_BOUND 0x04000000/* Thread bound to specific cpu */ +#define PF_MCE_EARLY 0x08000000/* Early kill for mce process policy */ #define PF_MEMPOLICY 0x10000000/* Non-default NUMA mempolicy */ #define PF_MUTEX_TESTER 0x20000000/* Thread belongs to the rt mutex tester */ #define PF_FREEZER_SKIP 0x40000000/* Freezer should not count it as freezeable */
kernel/sys.c +22 -0
@@ -1528,6 +1528,28 @@ SYSCALL_DEFINE5(prctl, int, option, unsi current->timer_slack_ns = arg2; error =0 ;break ; +case PR_MCE_KILL : +if (arg4 | arg5) +return -EINVAL ; +switch (arg2) { +case 0 : +if (arg3 !=0 ) +return -EINVAL ; + current->flags &= ~PF_MCE_PROCESS ; +break ; +case 1 : + current->flags |=PF_MCE_PROCESS ; +if (arg3 !=0 ) + current->flags |=PF_MCE_EARLY ; +else + current->flags &= ~PF_MCE_EARLY ; +break ; +default : +return -EINVAL ; + } + error =0 ; +break ; +default : error = -EINVAL ;break ;
include/linux/prctl.h +2 -0
@@ -88,4 +88,6 @@#define PR_TASK_PERF_COUNTERS_DISABLE 31#define PR_TASK_PERF_COUNTERS_ENABLE 32 +#define PR_MCE_KILL 33 +#endif /* _LINUX_PRCTL_H */
PATCH 16/21 HWPOISON: The high level memory error handler in the VM v7
In reply to Andi KleenAdd the high level memory handler that poisons pages
that got corrupted by hardware (typically by a two bit flip in a DIMM or a cache) on the Linux level. The goal is to prevent everyone from accessing these pages in the future.
This done at the VM level by marking a page hwpoisoned and doing the appropriate action based on the type of page it is.
The code that does this is portable and lives in mm/memory-failure.c
To quote the overview comment:
* High level machine check handler. Handles pages reported by the
* hardware as being corrupted usually due to a 2bit ECC memory or cache
* failure.
*
* This focuses on pages detected as corrupted in the background.
* When the current CPU tries to consume corruption the currently
* running process can just be killed directly instead. This implies
* that if the error cannot be handled for some reason it's safe to
* just ignore it because no corruption has been consumed yet. Instead
* when that happens another machine check will happen.
*
* Handles page cache pages in various states. The tricky part
* here is that we can access any page asynchronous to other VM
* users, because memory failures could happen anytime and anywhere,
* possibly violating some of their assumptions. This is why this code
* has to be extremely careful. Generally it tries to use normal locking
* rules, as in get the standard locks, even if that means the
* error handling takes potentially a long time.
*
* Some of the operations here are somewhat inefficient and have non
* linear algorithmic complexity, because the data structures have not
* been optimized for this case. This is in particular the case
* for the mapping from a vma to a process. Since this case is expected
* to be rare we hope we can get away with this.
There are in principle two strategies to kill processes on poison:
- just unmap the data and wait for an actual reference before killing
- kill as soon as corruption is detected.
Both have advantages and disadvantages and should be used in different situations. Right now both are implemented and can be switched with a new sysctl vm.memory_failure_early_kill The default is early kill.
The patch does some rmap data structure walking on its own to collect processes to kill. This is unusual because normally all rmap data structure knowledge is in rmap.c only. I put it here for now to keep everything together and rmap knowledge has been seeping out anyways
v2: Fix anon vma unlock crash (noticed by Johannes Weiner <[email protected]>) Handle pages on free list correctly (also noticed by Johannes)
Fix inverted try_to_release_page check (found by Chris Mason)
Add documentation for the new sysctl.
Various other cleanups/comment fixes.
v3: Use blockable signal for AO SIGBUS for better qemu handling. Numerous fixes from Fengguang Wu:
New code layout for the table (redone by AK)
Move the hwpoison bit setting before the lock (Fengguang Wu)
Some code cleanups (Fengguang Wu, AK)
Add missing lru_drain (Fengguang Wu)
Do more checks for valid mappings (inspired by patch from Fengguang) Handle free pages and fixes for clean pages (Fengguang)
Removed swap cache handling for now, needs more work
Better mapping checks to avoid races (Fengguang)
Fix swapcache (Fengguang)
Handle private2 pages too (Fengguang)
v4: Various fixes based on review comments from Nick Piggin Document locking order.
Improved comments.
Slightly improved description
Remove bogus hunk.
Wait properly for writeback pages (Nick Piggin)
v5: Improve various comments
Handle page_address_in_vma() failure better by SIGKILL and also
make message debugging only
Clean up printks
Remove redundant PageWriteback check (Nick Piggin)
Add missing clear_page_mlock
Reformat state table to be <80 columns again
Use truncate helper instead of manual truncate in me_pagecache_* Check for metadata buffer pages and reject them.
A few cleanups.
v6:
Fix a printk broken in the last round of cleanups.
More minor cleanups and fixes based on comments from Fengguang Wu. Rename /proc/meminfo Header to "HardwareCorrupted"
Add a printk for the failed mapping case (Fengguang Wu)
Better clean page check (Fengguang Wu)
v7:
Use lru_add_drain_all
Use isolate_lru result in final status
[Merge with Fengguang's version. Only cherry picked changes.]
Fix bad pfn handling in action_result (Fengguang)
Reverse tasklist lock locking order (Nick Piggin)
Check if VMA has really page mapped (Fengguang)
Add a missing page->mapping test (Fengguang)
Account all bad pages (Fengguang)
Misc comment fixes (Fengguang, Andi)
Add sysctl to force panic on memory failure
Only truncate if address space has it enabled, otherwise invalidate Now selectable in Kconfig by the user.
Allow elevated reference count and return status
Misc changes
--- Documentation/sysctl/vm.txt | 41 ++ fs/proc/meminfo.c | 9 include/linux/mm.h | 7 include/linux/rmap.h | 1 kernel/sysctl.c | 25 + mm/Kconfig | 9 mm/Makefile | 1 mm/filemap.c | 4 mm/memory-failure.c | 828 ++++++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 7 10 files changed, 929 insertions(+), 3 deletions(-)
mm/Makefile +1 -0
@@ -40,5 +40,6 @@ obj-$(CONFIG_SMP) += allocpercpu.o endif obj-$(CONFIG_QUICKLIST) += quicklist.o obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o page_cgroup.o +obj-$(CONFIG_MEMORY_FAILURE) += memory-failure.o obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
mm/memory-failure.c new file +828 -0
828 lines not on this page · raw message
include/linux/mm.h +7 -0
@@ -1309,5 +1309,12 @@ void vmemmap_populate_print_last(void);extern int account_locked_memory (struct mm_struct *mm,struct rlimit *rlim,size_t size);extern void refund_locked_memory (struct mm_struct *mm,size_t size); + +extern void memory_failure (unsigned long pfn,int trapno); +extern int __memory_failure (unsigned long pfn,int trapno,int ref); +extern int sysctl_memory_failure_early_kill; +extern int sysctl_memory_failure_recovery; +extern atomic_long_t mce_bad_pages; +#endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */
kernel/sysctl.c +25 -0
@@ -1357,6 +1357,31 @@ static struct ctl_table vm_table[] = { .mode = 0644, .proc_handler = &scan_unevictable_handler, }, +#ifdef CONFIG_MEMORY_FAILURE + { + .ctl_name = CTL_UNNUMBERED, + .procname = "memory_failure_early_kill", + .data = &sysctl_memory_failure_early_kill, + .maxlen = sizeof(sysctl_memory_failure_early_kill), + .mode = 0644, + .proc_handler = &proc_dointvec_minmax, + .strategy = &sysctl_intvec, + .extra1 = &zero, + .extra2 = &one, + }, + { + .ctl_name = CTL_UNNUMBERED, + .procname = "memory_failure_recovery", + .data = &sysctl_memory_failure_recovery, + .maxlen = sizeof(sysctl_memory_failure_recovery), + .mode = 0644, + .proc_handler = &proc_dointvec_minmax, + .strategy = &sysctl_intvec, + .extra1 = &zero, + .extra2 = &one, + }, +#endif + /* * NOTE: do not add new entries to this table unless you have read * Documentation/sysctl/ctl_unnumbered.txt
fs/proc/meminfo.c +8 -1
@@ -95,7 +95,11 @@ static int meminfo_proc_show(struct seq_"Committed_AS: %8lu kB\n" "VmallocTotal: %8lu kB\n" "VmallocUsed: %8lu kB\n" -"VmallocChunk: %8lu kB\n" , +"VmallocChunk: %8lu kB\n" +#ifdef CONFIG_MEMORY_FAILURE +"HardwareCorrupted: %8lu kB\n" +#endif + ,K (i.totalram),K (i.freeram), K(i.bufferram),
@@ -140,6 +144,9 @@ static int meminfo_proc_show(struct seq_ (unsigned long )VMALLOC_TOTAL >>10 , vmi.used >>10 , vmi.largest_chunk >> 10 +#ifdef CONFIG_MEMORY_FAILURE + ,atomic_long_read (&mce_bad_pages) << (PAGE_SHIFT -10 ) +#endif );hugetlb_report_meminfo (m);
mm/Kconfig +9 -0
@@ -233,6 +233,15 @@ config DEFAULT_MMAP_MIN_ADDR /proc/sys/vm/mmap_min_addr tunable. +config MEMORY_FAILURE + depends on MMU + depends on X86_MCE + bool "Enable memory failure recovery" + help + Enables code to recover from some memory failures on systems + with MCA recovery. This allows a system to continue running + even when some of its memory has uncorrected errors. + config NOMMU_INITIAL_TRIM_EXCESS int "Turn on mmap() excess space trimming before booting" depends on !MMU
Documentation/sysctl/vm.txt +40 -1
59 lines not on this page · raw message
mm/filemap.c +4 -0
@@ -105,6 +105,10 @@* * ->task->proc_lock * ->dcache_lock (proc_pid_lookup) +* +* (code doesn't rely on that order, so you could switch it around) +* ->tasklist_lock (memory_failure, collect_procs_ao) +* ->i_mmap_lock */ /*
mm/rmap.c +6 -1
@@ -36,6 +36,11 @@* mapping->tree_lock (widely used, in set_page_dirty, * in arch-dependent flush_dcache_mmap_lock, * within inode_lock in __sync_single_inode) +* +* (code doesn't rely on that order so it could be switched around) +* ->tasklist_lock +* anon_vma->lock (memory_failure, collect_procs_anon) +* pte map lock */ #include <linux/mm.h>
@@ -311,7 +316,7 @@ pte_t *page_check_address(struct page *p* if the page is not mapped into the page tables of this VMA. Only * valid for normal file or anonymous VMAs. */ -static int page_mapped_in_vma (struct page *page,struct vm_area_struct *vma) +int page_mapped_in_vma (struct page *page,struct vm_area_struct *vma) {unsigned long address;pte_t *pte;
include/linux/rmap.h +1 -0
@@ -129,6 +129,7 @@ int try_to_munlock(struct page *);*/ struct anon_vma *page_lock_anon_vma (struct page *page);void page_unlock_anon_vma (struct anon_vma *anon_vma); +int page_mapped_in_vma (struct page *page,struct vm_area_struct *vma);#else /* !CONFIG_MMU */
Re: [PATCH] [16/21] HWPOISON: The high level memory error handler in the VM v7
Quoted 10 lines · Andi Kleen
fs/proc/meminfo.c +5
-1-"VmallocChunk: %8lu kB\n" , +"VmallocChunk: %8lu kB\n" +#ifdef CONFIG_MEMORY_FAILURE +"HardwareCorrupted: %8lu kB\n" +#endif + ,K (i.totalram),K (i.freeram), K(i.bufferram),
fs/proc/meminfo.c static int meminfo_proc_show(struct seq_ · line 150
Quoted 7 lines
fs/proc/meminfo.c +3
-0@@ -140,6 +144,9 @@ static int meminfo_proc_show(struct seq_ (unsigned long )VMALLOC_TOTAL >>10 , vmi.used >>10 , vmi.largest_chunk >> 10 +#ifdef CONFIG_MEMORY_FAILURE + ,atomic_long_read (&mce_bad_pages) << (PAGE_SHIFT -10 ) +#endif );
The rest of your patches are totally clean, except this one .. It has some warning, but one error in the second ifdef block above ..
Daniel
Re: [PATCH] [16/21] HWPOISON: The high level memory error handler in the VM v7
Quoted 2 lines
The rest of your patches are totally clean, except this one .. It has some warning, but one error in the second ifdef block above ..
That's intentional, fixing it would require a bare , on a single line which would be far uglier what is there today. I refuse to make code ugly just to work around checkpatch.pl
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH] [16/21] HWPOISON: The high level memory error handler in the VM v7
Quoted 8 lines · Andi Kleen
Quoted 2 lines
The rest of your patches are totally clean, except this one .. It has some warning, but one error in the second ifdef block above ..
That's intentional, fixing it would require a bare , on a single line which would be far uglier what is there today. I refuse to make code ugly just to work around checkpatch.pl
-Andi
Here's the error,
ERROR: space required after that ',' (ctx:ExV)
#1086: FILE: fs/proc/meminfo.c:148:
+ ,atomic_long_read(&mce_bad_pages) << (PAGE_SHIFT - 10)
^
I think it just wants a space after the comma .. Couldn't you also break up these printk's so the two ifdef'ed parts are combined into a single printk that just adds on to the prior print out?
Daniel
Re: [PATCH] [16/21] HWPOISON: The high level memory error handler in the VM v7
In reply to Andi KleenQuoted 6 lines · Andi Kleen
Quoted 2 lines
The rest of your patches are totally clean, except this one .. It has some warning, but one error in the second ifdef block above ..
That's intentional, fixing it would require a bare , on a single line which would be far uglier what is there today. I refuse to make code ugly just to work around checkpatch.pl
Like the following is clean, and I don't see why it would work. Although I only compile tested it.
--
Order the seq_printfs so they can more easily (and cleanly) be added to and ifdef'ed.
--- fs/proc/meminfo.c | 83 +++++++++++++++++++++++++--------------------------- 1 files changed, 40 insertions(+), 43 deletions(-)
fs/proc/meminfo.c +40 -43
@@ -65,41 +65,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)"Active(file): %8lu kB\n" "Inactive(file): %8lu kB\n" "Unevictable: %8lu kB\n" - "Mlocked: %8lu kB\n" -#ifdef CONFIG_HIGHMEM - "HighTotal: %8lu kB\n" - "HighFree: %8lu kB\n" - "LowTotal: %8lu kB\n" - "LowFree: %8lu kB\n" -#endif -#ifndef CONFIG_MMU -"MmapCopy: %8lu kB\n" -#endif -"SwapTotal: %8lu kB\n" -"SwapFree: %8lu kB\n" -"Dirty: %8lu kB\n" -"Writeback: %8lu kB\n" -"AnonPages: %8lu kB\n" -"Mapped: %8lu kB\n" -"Slab: %8lu kB\n" -"SReclaimable: %8lu kB\n" -"SUnreclaim: %8lu kB\n" -"PageTables: %8lu kB\n" -#ifdef CONFIG_QUICKLIST - "Quicklists: %8lu kB\n" -#endif -"NFS_Unstable: %8lu kB\n" -"Bounce: %8lu kB\n" -"WritebackTmp: %8lu kB\n" -"CommitLimit: %8lu kB\n" -"Committed_AS: %8lu kB\n" -"VmallocTotal: %8lu kB\n" -"VmallocUsed: %8lu kB\n" -"VmallocChunk: %8lu kB\n" -#ifdef CONFIG_MEMORY_FAILURE -"HardwareCorrupted: %8lu kB\n" -#endif - , +"Mlocked: %8lu kB\n" ,K (i.totalram),K (i.freeram), K(i.bufferram),
@@ -112,16 +78,35 @@ static int meminfo_proc_show(struct seq_file *m, void *v)K (pages[LRU_ACTIVE_FILE ]),K (pages[LRU_INACTIVE_FILE ]),K (pages[LRU_UNEVICTABLE ]), -K (global_page_state (NR_MLOCK )), + K(global_page_state(NR_MLOCK))); +#ifdef CONFIG_HIGHMEM +seq_printf (m, +"HighTotal: %8lu kB\n" +"HighFree: %8lu kB\n" +"LowTotal: %8lu kB\n" +"LowFree: %8lu kB\n" ,K (i.totalhigh),K (i.freehigh),K (i.totalram-i.totalhigh), -K (i.freeram-i.freehigh), +K (i.freeram-i.freehigh));#endif #ifndef CONFIG_MMU -K ((unsigned long )atomic_long_read (&mmap_pages_allocated)), +seq_printf (m, +"MmapCopy: %8lu kB\n" , +K ((unsigned long )atomic_long_read (&mmap_pages_allocated)));#endif + seq_printf(m, + "SwapTotal: %8lu kB\n" + "SwapFree: %8lu kB\n" + "Dirty: %8lu kB\n" + "Writeback: %8lu kB\n" + "AnonPages: %8lu kB\n" + "Mapped: %8lu kB\n" + "Slab: %8lu kB\n" + "SReclaimable: %8lu kB\n" + "SUnreclaim: %8lu kB\n" + "PageTables: %8lu kB\n", K(i.totalswap), K(i.freeswap), K(global_page_state(NR_FILE_DIRTY)),
@@ -132,10 +117,21 @@ static int meminfo_proc_show(struct seq_file *m, void *v) global_page_state(NR_SLAB_UNRECLAIMABLE)),K (global_page_state (NR_SLAB_RECLAIMABLE )),K (global_page_state (NR_SLAB_UNRECLAIMABLE )), -K (global_page_state (NR_PAGETABLE )), + K(global_page_state(NR_PAGETABLE)));#ifdef CONFIG_QUICKLIST -K (quicklist_total_size ()), +seq_printf (m, +"Quicklists: %8lu kB\n" +K (quicklist_total_size ()));#endif + seq_printf(m, + "NFS_Unstable: %8lu kB\n" + "Bounce: %8lu kB\n" + "WritebackTmp: %8lu kB\n" + "CommitLimit: %8lu kB\n" + "Committed_AS: %8lu kB\n" + "VmallocTotal: %8lu kB\n" + "VmallocUsed: %8lu kB\n" + "VmallocChunk: %8lu kB\n", K(global_page_state(NR_UNSTABLE_NFS)), K(global_page_state(NR_BOUNCE)), K(global_page_state(NR_WRITEBACK_TEMP)),
@@ -143,11 +139,12 @@ static int meminfo_proc_show(struct seq_file *m, void *v)K (committed), (unsigned long )VMALLOC_TOTAL >>10 , vmi.used >>10 , - vmi.largest_chunk >> 10 + vmi.largest_chunk >> 10);#ifdef CONFIG_MEMORY_FAILURE - ,atomic_long_read (&mce_bad_pages) << (PAGE_SHIFT -10 ) +seq_printf (m, +"HardwareCorrupted: %8lu kB\n" , +atomic_long_read (&mce_bad_pages) << (PAGE_SHIFT -10 ));#endif - );hugetlb_report_meminfo (m);
Signature · 1 line
1.5.6.3
Re: [PATCH] [16/21] HWPOISON: The high level memory error handler in the VM v7
Quoted 11 lines · Daniel Walker
Quoted 6 lines · Andi Kleen
Quoted 2 lines
The rest of your patches are totally clean, except this one .. It has some warning, but one error in the second ifdef block above ..
That's intentional, fixing it would require a bare , on a single line which would be far uglier what is there today. I refuse to make code ugly just to work around checkpatch.pl
Like the following is clean, and I don't see why it would work. Although I only compile tested it.
That would turn a simple two liner change into a monstrosity of patch.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH] [16/21] HWPOISON: The high level memory error handler in the VM v7
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Daniel Walker
Quoted 6 lines · Andi Kleen
Quoted 2 lines
The rest of your patches are totally clean, except this one .. It has some warning, but one error in the second ifdef block above ..
That's intentional, fixing it would require a bare , on a single line which would be far uglier what is there today. I refuse to make code ugly just to work around checkpatch.pl
Quoted 4 lines
Quoted 2 lines
Like the following is clean, and I don't see why it would work. Although I only compile tested it.
That would turn a simple two liner change into a monstrosity of patch.
It's a dirty job, but someones got to do it..
Daniel
PATCH 17/21 HWPOISON: Enable .remove_error_page for migration aware file systems
In reply to Andi KleenEnable removing of corrupted pages through truncation for a bunch of file systems: ext*, xfs, gfs2, ocfs2, ntfs These should cover most server needs.
I chose the set of migration aware file systems for this for now, assuming they have been especially audited. But in general it should be safe for all file systems on the data area that support read/write and truncate.
Caveat: the hardware error handler does not take i_mutex for now before calling the truncate function. Is that ok?
--- fs/ext2/inode.c | 2 ++ fs/ext3/inode.c | 3 +++ fs/ext4/inode.c | 4 ++++ fs/gfs2/aops.c | 3 +++ fs/ntfs/aops.c | 2 ++ fs/ocfs2/aops.c | 1 + fs/xfs/linux-2.6/xfs_aops.c | 1 + mm/shmem.c | 1 + 8 files changed, 17 insertions(+)
fs/gfs2/aops.c +3 -0
@@ -1135,6 +1135,7 @@ static const struct address_space_operat .direct_IO = gfs2_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };static const struct address_space_operations gfs2_ordered_aops = {
@@ -1151,6 +1152,7 @@ static const struct address_space_operat .direct_IO = gfs2_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };static const struct address_space_operations gfs2_jdata_aops = {
@@ -1166,6 +1168,7 @@ static const struct address_space_operat .invalidatepage = gfs2_invalidatepage, .releasepage = gfs2_releasepage, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };void gfs2_set_aops (struct inode *inode)
fs/ntfs/aops.c +2 -0
@@ -1550,6 +1550,7 @@ const struct address_space_operations nt .migratepage = buffer_migrate_page, /* Move a page cache page fromone physical page to an other. */ + .error_remove_page = generic_error_remove_page, };/**
@@ -1569,6 +1570,7 @@ const struct address_space_operations nt .migratepage = buffer_migrate_page, /* Move a page cache page fromone physical page to an other. */ + .error_remove_page = generic_error_remove_page, };#ifdef NTFS_RW
fs/ocfs2/aops.c +1 -0
@@ -1997,4 +1997,5 @@ const struct address_space_operations oc .releasepage = ocfs2_releasepage, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };
fs/xfs/linux-2.6/xfs_aops.c +1 -0
@@ -1636,4 +1636,5 @@ const struct address_space_operations xf .direct_IO = xfs_vm_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };
mm/shmem.c +1 -0
@@ -2421,6 +2421,7 @@ static const struct address_space_operat .write_end = shmem_write_end, #endif .migratepage = migrate_page, + .error_remove_page = generic_error_remove_page, }; static const struct file_operations shmem_file_operations = {
fs/ext2/inode.c +2 -0
@@ -819,6 +819,7 @@ const struct address_space_operations ex .writepages = ext2_writepages, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };const struct address_space_operations ext2_aops_xip = {
@@ -837,6 +838,7 @@ const struct address_space_operations ex .direct_IO = ext2_direct_IO, .writepages = ext2_writepages, .migratepage = buffer_migrate_page, + .error_remove_page = generic_error_remove_page, };/*
fs/ext3/inode.c +3 -0
@@ -1819,6 +1819,7 @@ static const struct address_space_operat .direct_IO = ext3_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };static const struct address_space_operations ext3_writeback_aops = {
@@ -1834,6 +1835,7 @@ static const struct address_space_operat .direct_IO = ext3_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };static const struct address_space_operations ext3_journalled_aops = {
@@ -1848,6 +1850,7 @@ static const struct address_space_operat .invalidatepage = ext3_invalidatepage, .releasepage = ext3_releasepage, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };void ext3_set_aops (struct inode *inode)
fs/ext4/inode.c +4 -0
@@ -3373,6 +3373,7 @@ static const struct address_space_operat .direct_IO = ext4_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };static const struct address_space_operations ext4_writeback_aops = {
@@ -3388,6 +3389,7 @@ static const struct address_space_operat .direct_IO = ext4_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };static const struct address_space_operations ext4_journalled_aops = {
@@ -3402,6 +3404,7 @@ static const struct address_space_operat .invalidatepage = ext4_invalidatepage, .releasepage = ext4_releasepage, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };static const struct address_space_operations ext4_da_aops = {
@@ -3418,6 +3421,7 @@ static const struct address_space_operat .direct_IO = ext4_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, + .error_remove_page = generic_error_remove_page, };void ext4_set_aops (struct inode *inode)
PATCH 18/21 HWPOISON: Enable error_remove_page for NFS
In reply to Andi KleenEnable hardware memory error handling for NFS
Truncation of data pages at runtime should be safe in NFS, even when it doesn't support migration so far.
Trond tells me migration is also queued up for 2.6.32.
--- fs/nfs/file.c | 1 + 1 file changed, 1 insertion(+)
fs/nfs/file.c +1 -0
@@ -480,6 +480,7 @@ const struct address_space_operations nf .releasepage = nfs_release_page, .direct_IO = nfs_direct_IO, .launder_page = nfs_launder_page, + .error_remove_page = generic_error_remove_page, };/*
PATCH 19/21 HWPOISON: Add madvise() based injector for hardware poisoned pages v4
In reply to Andi KleenImpact: optional, useful for debugging
Add a new madvice sub command to inject poison for some pages in a process' address space. This is useful for testing the poison page handling.
This patch can allow root to tie up large amounts of memory. I got feedback from container developers and they didn't see any problem.
v2: Use write flag for get_user_pages to make sure to always get a fresh page
v3: Don't request write mapping (Fengguang Wu)
v4: Move MADV_* number to avoid conflict with KSM (Hugh Dickins)
--- include/asm-generic/mman-common.h | 1 + mm/madvise.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+)
mm/madvise.c +30 -0
@@ -207,6 +207,32 @@ static long madvise_remove(struct vm_arereturn error; } +#ifdef CONFIG_MEMORY_FAILURE +/* +* Error injection support for memory error handling. +*/ +static int madvise_hwpoison (unsigned long start,unsigned long end) +{ +int ret =0 ; + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; +for (; start < end; start +=PAGE_SIZE ) { +struct page *p; +int ret =get_user_pages (current, current->mm, start,1 , +0 ,0 , &p,NULL ); +if (ret !=1 ) +return ret; +printk (KERN_INFO "Injecting memory failure for page %lx at %lx\n" , +page_to_pfn (p), start); +/* Ignore return value for now */ +__memory_failure (page_to_pfn (p),0 ,1 ); +put_page (p); + } +return ret; +} +#endif +static long madvise_vma (struct vm_area_struct *vma,struct vm_area_struct **prev,unsigned long start,unsigned long end,int behavior)
@@ -307,6 +333,10 @@ SYSCALL_DEFINE3(madvise, unsigned long,int write;size_t len; +#ifdef CONFIG_MEMORY_FAILURE +if (behavior ==MADV_HWPOISON ) +return madvise_hwpoison (start, start+len_in); +#endif if (!madvise_behavior_valid (behavior))return error;
include/asm-generic/mman-common.h +1 -0
@@ -34,6 +34,7 @@#define MADV_REMOVE 9/* remove these pages & resources */ #define MADV_DONTFORK 10/* don't inherit across fork */ #define MADV_DOFORK 11/* do inherit across fork */ +#define MADV_HWPOISON 100/* poison a page for testing */ /* compatibility flags */ #define MAP_FILE 0
PATCH 20/21 HWPOISON: Add simple debugfs interface to inject hwpoison on arbitary PFNs
In reply to Andi KleenUseful for some testing scenarios, although specific testing is often done better through MADV_POISON
This can be done with the x86 level MCE injector too, but this interface allows it to do independently from low level x86 changes.
v2: Add module license (Haicheng Li)
--- mm/Kconfig | 6 +++++- mm/Makefile | 1 + mm/hwpoison-inject.c | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+), 1 deletion(-)
mm/hwpoison-inject.c new file +41 -0
@@ -0,0 +1,41 @@ +/* Inject a hwpoison memory failure on a arbitary pfn */ +#include <linux/module.h> +#include <linux/debugfs.h> +#include <linux/kernel.h> +#include <linux/mm.h> + +static struct dentry *hwpoison_dir, *corrupt_pfn; + +static int hwpoison_inject (void *data,u64 val) +{ +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; +printk (KERN_INFO "Injecting memory failure at pfn %Lx\n" , val); +return __memory_failure (val,18 ,0 ); +} + +DEFINE_SIMPLE_ATTRIBUTE (hwpoison_fops,NULL , hwpoison_inject,"%lli\n" ); + +static void pfn_inject_exit (void ) +{ +if (hwpoison_dir) +debugfs_remove_recursive (hwpoison_dir); +} + +static int pfn_inject_init (void ) +{ + hwpoison_dir =debugfs_create_dir ("hwpoison" ,NULL ); +if (hwpoison_dir ==NULL ) +return -ENOMEM ; + corrupt_pfn =debugfs_create_file ("corrupt-pfn" ,0600 , hwpoison_dir, +NULL , &hwpoison_fops); +if (corrupt_pfn ==NULL ) { +pfn_inject_exit (); +return -ENOMEM ; + } +return 0 ; +} + +module_init (pfn_inject_init); +module_exit (pfn_inject_exit); +MODULE_LICENSE ("GPL" );
mm/Kconfig +5 -1
@@ -236,12 +236,16 @@ config DEFAULT_MMAP_MIN_ADDR config MEMORY_FAILURE depends on MMU depends on X86_MCE - bool "Enable memory failure recovery" + bool "Enable recovery from hardware memory errors" help Enables code to recover from some memory failures on systems with MCA recovery. This allows a system to continue running even when some of its memory has uncorrected errors. +config HWPOISON_INJECT + tristate "Poison pages injector" + depends on MEMORY_FAILURE && DEBUG_KERNEL + config NOMMU_INITIAL_TRIM_EXCESS int "Turn on mmap() excess space trimming before booting" depends on !MMU
mm/Makefile +1 -0
@@ -41,5 +41,6 @@ endif obj-$(CONFIG_QUICKLIST) += quicklist.o obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o page_cgroup.o obj-$(CONFIG_MEMORY_FAILURE) += memory-failure.o +obj-$(CONFIG_HWPOISON_INJECT) += hwpoison-inject.o obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
PATCH 21/21 HWPOISON: Enable error_remove_page on btrfs
In reply to Andi Kleen--- fs/btrfs/inode.c | 1 + 1 file changed, 1 insertion(+)
fs/btrfs/inode.c +1 -0
@@ -5269,6 +5269,7 @@ static struct address_space_operations b .invalidatepage = btrfs_invalidatepage, .releasepage = btrfs_releasepage, .set_page_dirty = btrfs_set_page_dirty, + .error_remove_page = generic_error_remove_page, };static struct address_space_operations btrfs_symlink_aops = {