ia64: Migrate data off physical pages with correctable errors
cover ia64: Migrate data off physical pages with correctable errors
Migrate data off physical pages with corrected memory errors
Purpose:
Physical memory with corrected errors may decay over time into uncorrectable errors. The purpose of this patch is to move the data off pages with correctable memory errors before the memory goes bad.
The patches:
[1/2] page.discard: Avoid putting a bad page back on the LRU.
page.discard are the arch independent changes. It adds a new page flag (PG_memerror) to mark the page as bad and prevent it from being put back on the LRU. PG_memerror is bit 32 and only defined on 64 bit architectures. It adds "BadPages:" output to /proc/meminfo on 64 bit architectures.
[2/2] cpe.migrate: Call migration code on correctable errors
cpe.migrate are the IA64 specific changes. It connects the CPE handler to the page migration code. It is implemented as a kernel loadable module, similar to the mca recovery code (mca_recovery.ko), so that it can be removed to turn off the feature. It exports three symbols (migrate_prep, isolate_lru_page, and migrate_pages).
Comments:
Since page flags are a precious commodity on 32 bit architectures, the choice was made to implement PG_memerror only on 64 bit architectures, in the upper 32 bits. If there is interest in this feature on 32 bit, it only requires defining PG_memerror in one of the lower 32 page flage bits and removing the BITS_PER_LONG checks.
There is always an issue of how agressive the code should be on migrating pages. Should it migrate on the first correctable error, or wait for some threshold? Reasonable people may disagree on the threshold and the "right" answer may be hardware specific. The decision making is confined to the cpe_migrate.c code. It is currently set to migrate on the first correctable error.
Only pages that can be isolated on the LRU are migrated. Other pages, such as compound pages, are not migrated. That functionality could be a future enhancement.
Flow of the code description (while testing on IA64):
1) A user level application test program allocates memory and passes the virtual address of the memory to the error injection driver.
2) The error injection driver converts the virtual address to physical, functions the Altix hardware to modify the ECC for the physical page, creating a correctable error, and returns to the user application.
3) The user application reads the memory.
4) The Altix hardware detects the correctable error and interrupts prom. SAL builds a CPU error record, then sends a CPE interrupt to linux.
5) The linux CPE handler calls the cpe_migrate module (if installed).
6) cpe_migrate parses the physical address from the CPE record and adds the address to the migrate list (if not already on the list) and schedules the worker thread (cpe_enable_work).
7) cpe_enable_work calls ia64_mca_cpe_move_page.
8) ia64_mca_cpe_move_page validates the physical address, converts to page, sets PG_memerror flag and calls the migration code (migrate_prep(), isolate_lru_page(), and migrate_pages(). If the page migrates successfully, the bad page is added to badpagelist.
9) Because PG_memerror is set, the bad page is not added back on the LRU due to checks in lru_cache_add() and lru_cache_add_active().
Signature · 2 lines
Russ Anderson, OS RAS/Partitioning Project Lead SGI - Silicon Graphics Inc [email protected]
Quoted 6 lines · Russ Anderson
There is always an issue of how agressive the code should be on migrating pages. Should it migrate on the first correctable error, or wait for some threshold? Reasonable people may disagree on the threshold and the "right" answer may be hardware specific. The decision making is confined to the
cpe_migrate.ccode. It is currently set to migrate on the first correctable error.
I think the kernel code should do the migration ASAP. But I think we should have a list of 'bad' pages. We could then have a badram driver that userspace can talk to to find out which pages are bad, map those pages into a badram process, do various tests on them, and return the pages to the pool if they're determined to be 'good'.
I could also see badramd having a list of pages found to be bad in previous boots and asking the badram driver to take them out of circulation early in boot before they've been allocated.
Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · Russ Anderson
There is always an issue of how agressive the code should be on migrating pages. Should it migrate on the first correctable error, or wait for some threshold? Reasonable people may disagree on the threshold and the "right" answer may be hardware specific. The decision making is confined to the
cpe_migrate.ccode. It is currently set to migrate on the first correctable error.
Quoted 5 lines
I think the kernel code should do the migration ASAP. But I think we should have a list of 'bad' pages. We could then have a badram driver that userspace can talk to to find out which pages are bad, map those pages into a badram process, do various tests on them, and return the pages to the pool if they're determined to be 'good'.
Sure. The bad page list is badpagelist (defined in mca.c).
Quoted 3 lines
I could also see badramd having a list of pages found to be bad in previous boots and asking the badram driver to take them out of circulation early in boot before they've been allocated.
That would be one alternative. That type functionality would be useful. FWIW, some of my testing was on a system with a DIMM with solid single bits. It is a row/column problem so several meg of addresses are effected. Each boot it would migrate several meg worth of pages without any problem. It works surprisingly good.
Signature · 2 lines
Russ Anderson, OS RAS/Partitioning Project Lead SGI - Silicon Graphics Inc [email protected]