Direct Migration V8: Overview
Subsystems (12)
cover Direct Migration V8: Overview
Page migration allows the moving of the physical location of pages between nodes in a numa system while the process is running. This means that the virtual addresses that the process sees do not change. However, the system rearranges the physical location of those pages.
The main intend of page migration patches here is to reduce the latency of memory access by moving pages near to the processor where the process accessing that memory is running.
The migration patchsets allow a process to manually relocate the node on which its pages are located through the MF_MOVE and MF_MOVE_ALL options while setting a new memory policy. The pages of process can also be relocated from another process using the sys_migrate_pages() function call. The migrate_pages function call takes two sets of nodes and moves pages of a process that are located on the from nodes to the destination nodes.
Manual migration is very useful if for example the scheduler has relocated a process to a processor on a distant node. A batch scheduler or an administrator can detect the situation and move the pages of the process nearer to the new processor.
Larger installations usually partition the system using cpusets into sections of nodes. Paul Jackson has equipped cpusets with the ability to move pages when a task is moved to another cpuset. This allows automatic control over locality of a process. If a task is moved to a new cpuset then also all its pages are moved with it so that the performance of the process does not sink dramatically (as is the case today).
The swap migration patchset in 2.6.15-rc5-mm3 works by simply evicting the page. The pages must be faulted back in. The pages are then typically reallocated by the system near the node where the process is executing. For swap migration the destination of the move is controlled by the allocation policy. Cpusets set the allocation policy before calling sys_migrate_pages() in order to move the pages as intended.
No allocation policy changes are performed for sys_migrate_pages(). This means that the pages may not faulted in to the specified nodes if no allocation policy was set by other means. The pages will just end up near the node where the fault occurred. The direct migration patchset extends the migration functionality to avoid going through swap. The destination node of the relation is controllable during the actual moving of pages. The crutch of using the allocation policy to relocate is not necessary any and the pages are moved directly to the target.
The direct migration patchset allows the preservation of the relative location of pages within a group of nodes for all migration techniques which will preserve a particular memory allocation pattern generated even after migrating a process. This is necessary in order to preserve the memory latencies. Processes will run with similar performance after migration.
This patch makes sys_migrate_pages() finally work as intended but does not do any significant modifications to APIs.
Benefits over swap migration:
1. It makes migrates_pages() actually migrate pages instead of just swapping pages from a set of nodes out.
2. Its faster because the page does not need to be written to swap space.
3. It does not use swap space and therefore there is no danger of running out of swap space.
4. The need to write back a dirty page before migration is avoided through a file system specific method.
5. Direct migration allows the preservation of the relative location of a page within a set of nodes.
Many of the ideas for this code were originally developed in the memory hotplug project and we hope that this code also will allow the hotplug project to build on this patch in order to get to their goals.
The patchset consists of five patches (only the first two are necessary to have basic direct migration support):
1. SwapCache patch
SwapCache pages may have changed their type after lock_page() if the page was migrated. Check for this and retry lookup if the page is no longer a SwapCache page.
2. migrate_pages()
Basic direct migration with fallback to swap if all other attempts fail.
3. remove_from_swap()
Page migration installs swap ptes for anonymous pages in order to preserve the information contained in the page tables. This patch removes the swap ptes after migration and replaces them with regular ptes.
4. upgrade of MPOL_MF_MOVE and sys_migrate_pages()
Add logic to mm/mempolicy.c to allow the policy layer to control direct page migration. Thanks to Paul Jackson for the interative logic to move between sets of nodes.
5. buffer_migrate_pages() patch
Allow migration without writing back dirty pages. Add filesystem dependent migration support for ext2/ext3 and xfs. Use swapper space to setup a method to migrate anonymous pages without writeback.
Credits (also in mm/vmscan.c):
The idea for this scheme of page migration was first developed in the context of the memory hotplug project. The main authors of the migration code from the memory hotplug project are:
IWAMOTO Toshihiro <[email protected]>
Hirokazu Takahashi <[email protected]>
Dave Hansen <[email protected]>
Changes V7->V8:
- Patchset against 2.6.15-rc5-mm3
- Export more functions so that filesystems are able to implement their own migrate_page() function.
- Fix remove_from_swap() to remove the page from the swap cache in addition to replacing swap ptes. Call with the page lock on the new page.
- Fix copying of struct page {} field to avoid using the macros that process field information.
Changes V7->V7:
- Rediff against 2.6.14-rc5-mm2
Changes V6->V7:
- Patchset agsinst 2.6.15-rc5-mm1
- Fix one occurence of page->mapping in migrate_page_remove_references()
- Update description]
Changes V5->V6:
- Patchset against 2.6.15-rc3-mm1
- Remove checks for page count increases while migrating after Andrew assured me that this cannot happen. Revise documentation to reflect that. If this is the case then we will have no need to include the unwind code from the hotplug project in the future.
- Wrong reference while calling remove_from_swap to page instead of newpage fixed.
Changes V4->V5:
- Patchset against 2.6.15-rc2-mm1
- Update policy layer patch to use the generic check_range in 2.6.15-rc2-mm1.
- Remove try_to_unmap patch since VM_RESERVED vanished under us and therefore there is no point anymore to distinguish between permament and transitional failures.
Changes V3->V4:
- Patchset against 2.6.15-rc1-mm2 + two swap migration fixes posted today.
- Remove what is already in 2.6.14-rc1-mm2 which results in a significant cleanup of the code.
Changes V2->V3:
- Patchset against 2.6.14-mm2
- Fix single processor build and builds without CONFIG_MIGRATION
- export symbols for filesystems that are modules and for modules using migrate_pages().
- Paul Jackson's cpuset migration support is in 2.6.14-mm2 so this patchset can be easily applied to -mm2 to get from swap based to direct page migration.
Changes V1->V2:
- Call node_remap with the right parameters in do_migrate_pages().
- Take radix tree lock while examining page count to avoid races with find_get_page() and various *_get_pages based on it.
- Convert direct ptes to swap ptes before radix tree update to avoid more races.
- Fix problem if CONFIG_MIGRATION is off for buffer_migrate_page
- Add documentation about page migration
- Change migrate_pages() api so that the caller can decide what to do about the migrated pages (badmem handling and hotplug have to remove those pages for good).
- Drop config patch (already in mm)
- Add try_to_unmap patch
- Patchset now against 2.6.14-mm1 without requiring additional patches.
PATCH 1/5 Direct Migration V8: PageSwapCache checks
Check for PageSwapCache after looking up and locking a swap page.
The page migration code may change a swap pte to point to a different page under lock_page().
If that happens then the vm must retry the lookup operation in the swap space to find the correct page number. There are a couple of locations in the VM where a lock_page() is done on a swap page. In these locations we need to check afterwards if the page was migrated. If the page was migrated then the old page that was looked up before was freed and no longer has the PageSwapCache bit set.
mm/memory.c +7 -0
@@ -1870,6 +1870,7 @@ static int do_swap_page(struct mm_struct goto out; entry =pte_to_swp_entry (orig_pte); +again: page =lookup_swap_cache (entry);if (!page) {swapin_readahead (entry, address, vma);
@@ -1893,6 +1894,12 @@ static int do_swap_page(struct mm_structmark_page_accessed (page);lock_page (page); +if (!PageSwapCache (page)) { +/* Page migration has occured */ +unlock_page (page); +page_cache_release (page); + goto again; + }/* * Back out if somebody else already faulted in this pte.
mm/shmem.c +8 -0
@@ -1028,6 +1028,14 @@ repeat:page_cache_release (swappage); goto repeat; } +if (!PageSwapCache (swappage)) { +/* Page migration has occured */ +shmem_swp_unmap (entry); +spin_unlock (&info->lock); +unlock_page (swappage); +page_cache_release (swappage); + goto repeat; + }if (PageWriteback (swappage)) {shmem_swp_unmap (entry);spin_unlock (&info->lock);
mm/swapfile.c +7 -0
@@ -624,6 +624,7 @@ static int try_to_unuse(unsigned int typ*/ swap_map = &si->swap_map[i]; entry =swp_entry (type, i); +again: page =read_swap_cache_async (entry,NULL ,0 );if (!page) {/*
@@ -658,6 +659,12 @@ static int try_to_unuse(unsigned int typwait_on_page_locked (page);wait_on_page_writeback (page);lock_page (page); +if (!PageSwapCache (page)) { +/* Page migration has occured */ +unlock_page (page); +page_cache_release (page); + goto again; + }wait_on_page_writeback (page);/*
PATCH 2/5 Direct Migration V8: migrate_pages() extension
In reply to Christoph LameterAdd direct migration support with fall back to swap.
Direct migration support on top of the swap based page migration facility.
This allows the direct migration of anonymous pages and the migration of file backed pages by dropping the associated buffers (requires writeout).
Fall back to swap out if necessary.
The patch is based on lots of patches from the hotplug project but the code was restructured, documented and simplified as much as possible.
Note that an additional patch that defines the migrate_page() method for filesystems is necessary in order to avoid writeback for anonymous and file backed pages.
V7->V8:
- Fixed by KAMEZAWA Hiroyuki <[email protected]>:
copy raw page->index and page->mapping to new page.
V6->V7:
- Patch against 2.6.15-rc5-mm1.
- Replace one occurrence of page->mapping with page_mapping(page) in migrate_pages_remove_references()
V4->V5:
- Patch against 2.6.15-rc2-mm1 + double unlock fix + consolidation patch
V3->V4:
- Remove components already in the swap migration patch
V1->V2:
- Change migrate_pages() so that it can return pagelist for failed and moved pages. No longer free the old pages but allow caller to dispose of them.
- Unmap pages before changing reverse map under tree lock. Take a write_lock instead of a read_lock.
- Add documentation
include/linux/swap.h +3 -0
@@ -178,6 +178,9 @@ extern int vm_swappiness;#ifdef CONFIG_MIGRATION extern int isolate_lru_page (struct page *p);extern int putback_lru_pages (struct list_head *l); +extern int migrate_page (struct page *,struct page *); +extern int migrate_page_remove_references (struct page *,struct page *,int ); +extern void migrate_page_copy (struct page *,struct page *);extern int migrate_pages (struct list_head *l,struct list_head *t,struct list_head *moved,struct list_head *failed);#endif
Documentation/vm/page_migration new file +129 -0
129 lines not on this page · raw message
mm/vmscan.c +223 -9
271 lines not on this page · raw message
PATCH 3/5 Direct Migration V8: remove_from_swap() to remove swap ptes
In reply to Christoph LameterAdd remove_from_swap
remove_from_swap() allows the restoration of the pte entries that existed before page migration occurred for anonymous pages by walking the reverse maps. This reduces swap use and establishes regular pte's without the need for page faults.
V7->V8:
- Move the removing of the page from the swap entries and from the swap cache to migrate page so that it can be done while the page lock on the new page is held.
- Unlock anon_vma
- Remove the page from the page cache
V5->V6:
- Somehow V5 did a remove_from_swap for the old page. Changed to new page
V3->V4:
- Add new function remove_vma_swap in swapfile.c to encapsulate the functionality needed instead of exporting unuse_vma.
- Add #ifdef CONFIG_MIGRATION
include/linux/swap.h +3 -0
@@ -230,6 +230,9 @@ extern int remove_exclusive_swap_page(ststruct backing_dev_info ;extern spinlock_t swap_lock; +#ifdef CONFIG_MIGRATION +extern int remove_vma_swap (struct vm_area_struct *vma,struct page *page); +#endif /* linux/mm/thrash.c */ extern struct mm_struct * swap_token_mm;
mm/swapfile.c +10 -0
@@ -532,6 +532,16 @@ static int unuse_mm(struct mm_struct *mmreturn 0 ; } +#ifdef CONFIG_MIGRATION +int remove_vma_swap (struct vm_area_struct *vma,struct page *page) +{ +swp_entry_t entry = { .val =page_private (page) }; + +return unuse_vma (vma, entry, page); +} +#endif + +/* * Scan swap_map from current position to next entry still in use. * Recycle to start on reaching the end, returning 0 when empty.
mm/rmap.c +29 -0
@@ -205,6 +205,35 @@ out:return anon_vma; } +#ifdef CONFIG_MIGRATION +/* +* Remove an anonymous page from swap replacing the swap pte's +* through real pte's pointing to valid pages and then releasing +* the page from the swap cache. +* +* Must hold page lock on page. +*/ +void remove_from_swap (struct page *page) +{ +struct anon_vma *anon_vma; +struct vm_area_struct *vma; + +if (!PageAnon (page) || !PageSwapCache (page)) +return ; + + anon_vma =page_lock_anon_vma (page); +if (!anon_vma) +return ; + +list_for_each_entry (vma, &anon_vma->head, anon_vma_node) +remove_vma_swap (vma, page); + +spin_unlock (&anon_vma->lock); + +delete_from_swap_cache (page); +} +#endif +/* * At what user virtual address is page expected in vma? */
include/linux/rmap.h +3 -0
@@ -92,6 +92,9 @@ static inline void page_dup_rmap(struct*/ int page_referenced (struct page *,int is_locked);int try_to_unmap (struct page *); +#ifdef CONFIG_MIGRATION +void remove_from_swap (struct page *page); +#endif /* * Called from mm/filemap_xip.c to unmap empty zero page
mm/vmscan.c +9 -0
@@ -807,6 +807,15 @@ int migrate_page(struct page *newpage, smigrate_page_copy (newpage, page); +/* +* Remove auxiliary swap entries and replace +* them with real ptes. +* +* Note that a real pte entry will allow processes that are not +* waiting on the page lock to use the new page via the page tables +* before the new page is unlocked. +*/ +remove_from_swap (newpage);return 0 ; }
PATCH 4/5 Direct Migration V8: upgrade MPOL_MF_MOVE and sys_migrate_pages()
In reply to Christoph LameterModify policy layer to support direct page migration
- Add migrate_pages_to() allowing the migration of a list of pages to a a specified node or to vma with a specific allocation policy in sets of MIGRATE_CHUNK_SIZE pages
- Modify do_migrate_pages() to do a staged move of pages from the source nodes to the target nodes.
V3->V4: Fixed up to be based on the swap migration code in 2.6.15-rc1-mm2.
V1->V2:
- Migrate processes in chunks of MIGRATE_CHUNK_SIZE
mm/mempolicy.c +151 -21
@@ -95,6 +95,9 @@#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1)/* Invert check for nodemask */ #define MPOL_MF_STATS (MPOL_MF_INTERNAL << 2)/* Gather statistics */ +/* The number of pages to migrate per call to migrate_pages() */ +#define MIGRATE_CHUNK_SIZE 256 +static kmem_cache_t *policy_cache;static kmem_cache_t *sn_cache;
@@ -561,24 +564,96 @@ static void migrate_page_add(struct vm_a } } -static int swap_pages (struct list_head *pagelist) +/* +* Migrate the list 'pagelist' of pages to a certain destination. +* +* Specify destination with either non-NULL vma or dest_node >= 0 +* Return the number of pages not migrated or error code +*/ +static int migrate_pages_to (struct list_head *pagelist, +struct vm_area_struct *vma,int dest) { +LIST_HEAD (newlist);LIST_HEAD (moved);LIST_HEAD (failed); -int n; +int err =0 ; +int nr_pages; +struct page *page; +struct list_head *p; - n =migrate_pages (pagelist,NULL , &moved, &failed); -putback_lru_pages (&failed); -putback_lru_pages (&moved); +redo: + nr_pages =0 ; +list_for_each (p, pagelist) { +if (vma) + page =alloc_page_vma (GFP_HIGHUSER , vma, + vma->vm_start); +else + page =alloc_pages_node (dest,GFP_HIGHUSER ,0 ); -return n; +if (!page) { + err = -ENOMEM ; + goto out; + } +list_add (&page->lru, &newlist); + nr_pages++; +if (nr_pages >MIGRATE_CHUNK_SIZE ); +break ; + } + err =migrate_pages (pagelist, &newlist, &moved, &failed); + +putback_lru_pages (&moved);/* Call release pages instead ?? */ + +if (err >=0 &&list_empty (&newlist) && !list_empty (pagelist)) + goto redo; +out: +/* Return leftover allocated pages */ +while (!list_empty (&newlist)) { + page = list_entry(newlist.next, struct page, lru); +list_del (&page->lru); +__free_page (page); + } +list_splice (&failed, pagelist); +if (err <0 ) +return err; + +/* Calculate number of leftover pages */ + nr_pages =0 ; +list_for_each (p, pagelist) + nr_pages++; +return nr_pages; }/* -* For now migrate_pages simply swaps out the pages from nodes that are in -* the source set but not in the target set. In the future, we would -* want a function that moves pages between the two nodesets in such -* a way as to preserve the physical layout as much as possible. +* Migrate pages from one node to a target node. +* Returns error or the number of pages not migrated. +*/ +int migrate_to_node (struct mm_struct *mm,int source,int dest,int flags) +{ +nodemask_t nmask; +LIST_HEAD (pagelist); +int err =0 ; + +nodes_clear (nmask); +node_set (source, nmask); + +check_range (mm, mm->mmap->vm_start,TASK_SIZE , &nmask, + flags |MPOL_MF_DISCONTIG_OK , + &pagelist); + +if (!list_empty (&pagelist)) { + + err =migrate_pages_to (&pagelist,NULL , dest); + +if (!list_empty (&pagelist)) +putback_lru_pages (&pagelist); + + } +return err; +} + +/* +* Move pages between the two nodesets so as to preserve the physical +* layout as much as possible. * * Returns the number of page that could not be moved. */
@@ -586,22 +661,76 @@ int do_migrate_pages(struct mm_struct *m const nodemask_t *from_nodes, const nodemask_t *to_nodes, int flags) {LIST_HEAD (pagelist ); -int count =0 ; -nodemask_t nodes; +int busy =0 ; +int err =0 ; +nodemask_t tmp; -nodes_andnot (nodes, *from_nodes, *to_nodes); +down_read (&mm->mmap_sem); -down_read (&mm->mmap_sem); -check_range (mm, mm->mmap->vm_start,TASK_SIZE , &nodes, - flags |MPOL_MF_DISCONTIG_OK , &pagelist); +/* Find a 'source' bit set in 'tmp' whose corresponding 'dest' +* bit in 'to' is not also set in 'tmp'. Clear the found 'source' +* bit in 'tmp', and return that <source, dest> pair for migration. +* The pair of nodemasks 'to' and 'from' define the map. +* +* If no pair of bits is found that way, fallback to picking some +* pair of 'source' and 'dest' bits that are not the same. If the +* 'source' and 'dest' bits are the same, this represents a node +* that will be migrating to itself, so no pages need move. +* +* If no bits are left in 'tmp', or if all remaining bits left +* in 'tmp' correspond to the same bit in 'to', return false +* (nothing left to migrate). +* +* This lets us pick a pair of nodes to migrate between, such that +* if possible the dest node is not already occupied by some other +* source node, minimizing the risk of overloading the memory on a +* node that would happen if we migrated incoming memory to a node +* before migrating outgoing memory source that same node. +* +* A single scan of tmp is sufficient. As we go, we remember the +* most recent <s, d> pair that moved (s != d). If we find a pair +* that not only moved, but what's better, moved to an empty slot +* (d is not set in tmp), then we break out then, with that pair. +* Otherwise when we finish scannng from_tmp, we at least have the +* most recent <s, d> pair that moved. If we get all the way through +* the scan of tmp without finding any node that moved, much less +* moved to an empty node, then there is nothing left worth migrating. +*/ -if (!list_empty (&pagelist)) { - count =swap_pages (&pagelist); -putback_lru_pages (&pagelist); + tmp = *from_nodes; +while (!nodes_empty (tmp)) { +int s,d; +int source =-1 ; +int dest =0 ; + +for_each_node_mask (s, tmp) { + + d =node_remap (s, *from_nodes, *to_nodes); +if (s == d) +continue ; + + source = s;/* Node moved. Memorize */ + dest = d; + +/* dest not in remaining from nodes? */ +if (!node_isset (dest, tmp)) +break ; + } +if (source ==-1 ) +break ; + +node_clear (source, tmp); + err =migrate_to_node (mm, source, dest, flags); +if (err >0 ) + busy += err; +if (err <0 ) +break ; }up_read (&mm->mmap_sem); -return count; +if (err <0 ) +return err; +return busy; } long do_mbind(unsigned long start, unsigned long len,
@@ -661,8 +790,9 @@ long do_mbind(unsigned long start, unsigint nr_failed =0 ; err =mbind_range (vma, start, end, new); +if (!list_empty (&pagelist)) - nr_failed =swap_pages (&pagelist); + nr_failed =migrate_pages_to (&pagelist, vma,-1 );if (!err && nr_failed && (flags &MPOL_MF_STRICT )) err = -EIO ;
PATCH 5/5 Direct Migration V8: Avoid writeback / page_migrate() method
In reply to Christoph LameterMigrate a page with buffers without requiring writeback
This introduces a new address space operation migratepage() that may be used by a filesystem to implement its own version of page migration.
A version is provided that migrates buffers attached to pages. Some filesystems (ext2, ext3, xfs) are modified to utilize this feature.
The swapper address space operation are modified so that a regular migrate_page() will occur for anonymous pages without writeback (migrate_pages forces every anonymous page to have a swap entry).
V7->V8:
- Export more functions in order for loadable filesystems to be able to define their own migration function.
V2->V3:
- export functions for filesystems that are modules and for modules that perform migration by calling migrate_pages().
- Fix macro name clash. Fix build on UP and systems without CONFIG_MIGRATION
V1->V2:
- Fix CONFIG_MIGRATION handling
include/linux/fs.h +8 -0
@@ -366,6 +366,8 @@ struct address_space_operations { loff_t offset, unsigned long nr_segs);struct page * (*get_xip_page)(struct address_space *,sector_t ,int ); +/* migrate the contents of a page to the specified target */ +int (*migratepage) (struct page *,struct page *); };struct backing_dev_info ;
@@ -1720,6 +1722,12 @@ extern void simple_release_fs(struct vfsextern ssize_t simple_read_from_buffer (void __user *,size_t ,loff_t *,const void *,size_t ); +#ifdef CONFIG_MIGRATION +extern int buffer_migrate_page (struct page *,struct page *); +#else +#define buffer_migrate_page NULL +#endif +extern int inode_change_ok (struct inode *,struct iattr *);extern int __must_check inode_setattr (struct inode *,struct iattr *);
mm/swap_state.c +1 -0
@@ -27,6 +27,7 @@ static struct address_space_operations s .writepage = swap_writepage, .sync_page = block_sync_page, .set_page_dirty = __set_page_dirty_nobuffers, + .migratepage = migrate_page, };static struct backing_dev_info swap_backing_dev_info = {
fs/xfs/linux-2.6/xfs_aops.c +1 -0
@@ -1347,4 +1347,5 @@ struct address_space_operations linvfs_a .commit_write = generic_commit_write, .bmap = linvfs_bmap, .direct_IO = linvfs_direct_IO, + .migratepage = buffer_migrate_page, };
fs/buffer.c +65 -0
71 lines not on this page · raw message
fs/ext3/inode.c +2 -0
@@ -1559,6 +1559,7 @@ static struct address_space_operations e .invalidatepage = ext3_invalidatepage, .releasepage = ext3_releasepage, .direct_IO = ext3_direct_IO, + .migratepage = buffer_migrate_page, };static struct address_space_operations ext3_writeback_aops = {
@@ -1572,6 +1573,7 @@ static struct address_space_operations e .invalidatepage = ext3_invalidatepage, .releasepage = ext3_releasepage, .direct_IO = ext3_direct_IO, + .migratepage = buffer_migrate_page, };static struct address_space_operations ext3_journalled_aops = {
fs/ext2/inode.c +2 -0
@@ -706,6 +706,7 @@ struct address_space_operations ext2_aop .bmap = ext2_bmap, .direct_IO = ext2_direct_IO, .writepages = ext2_writepages, + .migratepage = buffer_migrate_page, };struct address_space_operations ext2_aops_xip = {
@@ -723,6 +724,7 @@ struct address_space_operations ext2_nob .bmap = ext2_bmap, .direct_IO = ext2_direct_IO, .writepages = ext2_writepages, + .migratepage = buffer_migrate_page, };/*
fs/xfs/linux-2.6/xfs_buf.c +1 -0
@@ -1568,6 +1568,7 @@ xfs_mapping_buftarg(struct address_space *mapping;static struct address_space_operations mapping_aops = { .sync_page = block_sync_page, + .migratepage = fail_migrate_page, }; inode =new_inode (bdev->bd_inode->i_sb);
mm/vmscan.c +21 -0
@@ -608,6 +608,15 @@ int putback_lru_pages(struct list_head * }/* +* Non migratable page +*/ +int fail_migrate_page (struct page *newpage,struct page *page) +{ +return -EIO ; +} +EXPORT_SYMBOL (fail_migrate_page); + +/* * swapout a single page * page is locked upon entry, unlocked on exit */
@@ -652,6 +661,8 @@ unlock_retry: retry:return -EAGAIN ; } +EXPORT_SYMBOL (swap_page); +/* * Page migration was first developed in the context of the memory hotplug * project. The main authors of the migration code are:
@@ -752,6 +763,7 @@ int migrate_page_remove_references(strucreturn 0 ; } +EXPORT_SYMBOL (migrate_page_remove_references);/* * Copy the page to its new location
@@ -791,6 +803,7 @@ void migrate_page_copy(struct page *newpif (PageWriteback (newpage))end_page_writeback (newpage); } +EXPORT_SYMBOL (migrate_page_copy);/* * Common logic to directly migrate a single page suitable for
@@ -818,6 +831,7 @@ int migrate_page(struct page *newpage, sremove_from_swap (newpage);return 0 ; } +EXPORT_SYMBOL (migrate_page);/* * migrate_pages
@@ -917,6 +931,11 @@ redo:if (!mapping) goto unlock_both; +if (mapping->a_ops->migratepage) { + rc = mapping->a_ops->migratepage (newpage, page); + goto unlock_both; + } +/* * Trigger writeout if page is dirty */
@@ -986,6 +1005,7 @@ next:return nr_failed + retry; } +EXPORT_SYMBOL (migrate_pages);static void lru_add_drain_per_cpu (void *dummy) {
@@ -1028,6 +1048,7 @@ redo: }return rc; } +EXPORT_SYMBOL (isolate_lru_page);#endif /*
include/linux/swap.h +5 -0
@@ -183,6 +183,11 @@ extern int migrate_page_remove_referenceextern void migrate_page_copy (struct page *,struct page *);extern int migrate_pages (struct list_head *l,struct list_head *t,struct list_head *moved,struct list_head *failed); +extern int fail_migrate_page (struct page *,struct page *); +#else +/* Possible settings for the migrate_page() method in address_operations */ +#define migrate_page NULL +#define fail_migrate_page NULL#endif #ifdef CONFIG_MMU
mm/rmap.c +1 -0
@@ -232,6 +232,7 @@ void remove_from_swap(struct page *page) delete_from_swap_cache(page); } +EXPORT_SYMBOL (remove_from_swap);#endif /*