mm: Implement Swap Prefetching v25
Subsystems (7)
PATCH 1/1 mm: Implement Swap Prefetching v25
Everything is working nicely now, surviving a serious thrashing. Overall the behaviour I would say has improved as a result of all the modifications.
Many thanks go to Nick and Andrew for suggestions.
Andrew please consider this version for -mm.
Cheers,
Con
---
This patch implements swap prefetching when the vm is relatively idle and there is free ram available. The code is based on some preliminary code by Thomas Schlichter.
This stores a list of swapped entries in a list ordered most recently used and a radix tree. It generates a low priority kernel thread running at nice 19 to do the prefetching at a later stage.
Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram and changes in free ram representing disk writes or reads also prevent prefetching.
It then checks that we have spare ram looking for at least 3* pages_high free per zone and if it succeeds that will prefetch pages from swap into the swap cache. The pages are added to the tail of the inactive list to preserve LRU ordering.
Pages are prefetched until the list is empty or the vm is seen as busy according to the previously described criteria. Node data on numa is stored with the entries and an appropriate zonelist based on this is used when allocating ram.
The pages are copied to swap cache and kept on backing store. This allows pressure on either physical ram or swap to readily find free pages without further I/O.
Prefetching can be enabled/disabled via the tunable in /proc/sys/vm/swap_prefetch initially set to 1 (enabled).
In testing on modern pc hardware this results in wall-clock time activation of the firefox browser to speed up 5 fold after a worst case complete swap-out of the browser on a static web page.
Documentation/sysctl/vm.txt | 11 include/linux/mm_inline.h | 7 include/linux/swap.h | 55 ++++ include/linux/sysctl.h | 1 init/Kconfig | 22 + kernel/sysctl.c | 10 mm/Makefile | 1 mm/swap.c | 42 +++ mm/swap_prefetch.c | 488 ++++++++++++++++++++++++++++++++++++++++++++ mm/swap_state.c | 10 mm/vmscan.c | 5 11 files changed, 651 insertions(+), 1 deletion(-)
Documentation/sysctl/vm.txt +11 -0
@@ -29,6 +29,7 @@ Currently, these files are in /proc/sys/ - drop-caches - zone_reclaim_mode - zone_reclaim_interval +- swap_prefetch ==============================================================
@@ -178,3 +179,13 @@ Time is set in seconds and set by defaul Reduce the interval if undesired off node allocations occur. However, too frequent scans will have a negative impact onoff node allocation performance. +============================================================== + +swap_prefetch + +This enables or disables the swap prefetching feature. When the virtual +memory subsystem has been extremely idle for at least 5 seconds it will start +copying back pages from swap into the swapcache and keep a copy in swap. In +practice it can take many minutes before the vm is idle enough. + +The default value is 1.
include/linux/swap.h +55 -0
79 lines not on this page · raw message
include/linux/sysctl.h +1 -0
@@ -184,6 +184,7 @@ enumVM_PERCPU_PAGELIST_FRACTION =30 ,/* int: fraction of pages in each percpu_pagelist */ VM_ZONE_RECLAIM_MODE =31 ,/* reclaim local zone memory before going off node */ VM_ZONE_RECLAIM_INTERVAL =32 ,/* time period to wait after reclaim failure */ + VM_SWAP_PREFETCH=33, /* swap prefetch */ };
init/Kconfig +22 -0
@@ -103,6 +103,28 @@ config SWAP used to provide more virtual memory than the actual RAM present in your computer. If unsure say Y. +config SWAP_PREFETCH + bool "Support for prefetching swapped memory" + depends on SWAP + default y + ---help--- + This option will allow the kernel to prefetch swapped memory pages + when idle. The pages will be kept on both swap and in swap_cache + thus avoiding the need for further I/O if either ram or swap space + is required. + + What this will do on workstations is slowly bring back applications + that have swapped out after memory intensive workloads back into + physical ram if you have free ram at a later stage and the machine + is relatively idle. This means that when you come back to your + computer after leaving it idle for a while, applications will come + to life faster. Note that your swap usage will appear to increase + but these are cached pages, can be dropped freely by the vm, and it + should stabilise around 50% swap usage maximum. + + Workstations and multiuser workstation servers will most likely want + to say Y. + config SYSVIPC bool "System V IPC" ---help---
kernel/sysctl.c +10 -0
@@ -891,6 +891,16 @@ static ctl_table vm_table[] = { .strategy = &sysctl_jiffies, },#endif +#ifdef CONFIG_SWAP_PREFETCH + { + .ctl_name = VM_SWAP_PREFETCH, + .procname = "swap_prefetch", + .data = &swap_prefetch, + .maxlen = sizeof(swap_prefetch), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, +#endif { .ctl_name = 0 } };
mm/Makefile +1 -0
@@ -13,6 +13,7 @@ obj-y := bootmem.o filemap.o mempool.o prio_tree.o util.o $(mmu-y) obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o thrash.o +obj-$(CONFIG_SWAP_PREFETCH) += swap_prefetch.o obj-$(CONFIG_HUGETLBFS) += hugetlb.o obj-$(CONFIG_NUMA) += mempolicy.o obj-$(CONFIG_SPARSEMEM) += sparse.o
mm/swap.c +42 -0
53 lines not on this page · raw message
mm/swap_prefetch.c new file +488 -0
488 lines not on this page · raw message
mm/swap_state.c +9 -1
@@ -81,6 +81,7 @@ static int __add_to_swap_cache(struct pa error =radix_tree_insert (&swapper_space.page_tree, entry.val, page);if (!error) { +remove_from_swapped_list (entry.val);page_cache_get (page);SetPageLocked (page);SetPageSwapCache (page);
@@ -94,11 +95,12 @@ static int __add_to_swap_cache(struct pareturn error; } -static int add_to_swap_cache (struct page *page,swp_entry_t entry) +int add_to_swap_cache (struct page *page,swp_entry_t entry) {int error;if (!swap_duplicate (entry)) { +remove_from_swapped_list (entry.val);INC_CACHE_INFO (noent_race);return -ENOENT ; }
@@ -147,6 +149,9 @@ int add_to_swap(struct page * page, gfp_swp_entry_t entry;int err; +/* Swap prefetching is delayed if we're swapping pages */ +delay_swap_prefetch (); +if (!PageLocked (page))BUG ();
@@ -320,6 +325,9 @@ struct page *read_swap_cache_async(swp_estruct page *found_page, *new_page =NULL ;int err; +/* Swap prefetching is delayed if we're already reading from swap */ +delay_swap_prefetch (); + do { /* * First check the swap cache. Since this is normally
mm/vmscan.c +5 -0
@@ -396,6 +396,7 @@ static int remove_mapping(struct addressif (PageSwapCache (page)) {swp_entry_t swap = { .val =page_private (page) }; +add_to_swapped_list (page);__delete_from_swap_cache (page);write_unlock_irq (&mapping->tree_lock);swap_free (swap);
@@ -1406,6 +1407,8 @@ int try_to_free_pages(struct zone **zoneunsigned long lru_pages =0 ;int i; +delay_swap_prefetch (); + sc.gfp_mask = gfp_mask; sc.may_writepage = !laptop_mode; sc.may_swap =1 ;
@@ -1758,6 +1761,8 @@ int shrink_all_memory(int nr_pages) .reclaimed_slab = 0, }; +delay_swap_prefetch (); + current->reclaim_state = &reclaim_state;for_each_pgdat (pgdat) {int freed;
include/linux/mm_inline.h +7 -0
@@ -14,6 +14,13 @@ add_page_to_inactive_list(struct zone *z }static inline void +add_page_to_inactive_list_tail (struct zone *zone,struct page *page) +{ +list_add_tail (&page->lru, &zone->inactive_list); + zone->nr_inactive++; +} + +static inline void del_page_from_active_list (struct zone *zone,struct page *page) {list_del (&page->lru);
Quoted 10 lines · Con Kolivas
Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram and changes in free ram representing disk writes or reads also prevent prefetching.
It then checks that we have spare ram looking for at least 3*
pages_highfree per zone and if it succeeds that will prefetch pages from swap into the swap cache. The pages are added to the tail of the inactive list to preserve LRU ordering.
spare ram when swapping??? We are already under memory pressure. Why make it worse by getting rid of the few bits of available memory? If a system swaps then we are per definition in the bad performance range. Add more memory.
Quoted 4 lines · Christoph Lameter
spare ram when swapping??? We are already under memory pressure. Why make it worse by getting rid of the few bits of available memory? If a system swaps then we are per definition in the bad performance range. Add more memory.
when a program exits it's memory is now free, rather then just waiting until something uses this memory up normally, this patch attempts to fill that memory with things that are expected to be useful (things that were swapped out)
this won't be a win in all cases by any means, but if it uses disk bandwidth that would otherwise be idle and memory that is empty (and is at the tail of the LRU list so it's the first to be thrown away) the cost of doing this should be close to zero.
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 9 lines · David Lang
Quoted 4 lines · Christoph Lameter
spare ram when swapping??? We are already under memory pressure. Why make it worse by getting rid of the few bits of available memory? If a system swaps then we are per definition in the bad performance range. Add more memory.
when a program exits it's memory is now free, rather then just waiting until something uses this memory up normally, this patch attempts to fill that memory with things that are expected to be useful (things that were swapped out)
Then trigger this action when a program exits and when you know there was enough freed up to justify such an action. However, this is still a heuristic. No one knows if the pages read from swap will be of any use at all. For all I know a process may want to allocate some memory and fail because the memory was needlessly spend to read in pages that no one needs.
Quoted 20 lines · Christoph Lameter
Date: Mon, 13 Feb 2006 13:16:00 -0800 (PST)
From: Christoph Lameter <[email protected]>
To: David Lang <[email protected]>
Cc: Con Kolivas <[email protected]>,
linux kernel mailing list <[email protected]>, Andrew Morton <[email protected]>, ck list <[email protected]>
Subject: Re: [PATCH] mm: Implement Swap Prefetching v25Quoted 9 lines · David Lang
Quoted 4 lines · Christoph Lameter
spare ram when swapping??? We are already under memory pressure. Why make it worse by getting rid of the few bits of available memory? If a system swaps then we are per definition in the bad performance range. Add more memory.
when a program exits it's memory is now free, rather then just waiting until something uses this memory up normally, this patch attempts to fill that memory with things that are expected to be useful (things that were swapped out)
Quoted 6 lines
Then trigger this action when a program exits and when you know there was enough freed up to justify such an action. However, this is still a heuristic. No one knows if the pages read from swap will be of any use at all. For all I know a process may want to allocate some memory and fail because the memory was needlessly spend to read in pages that no one needs.
this won't happen, becouse if a program requests memory the kernel will through away data from the swap cache that's unchanged on disk (like the memory just filled by this) to free up memory for the program.
you don't want to trigger this on a program exit becouse the system is likly to be busy doing other things at that time, you want to try and do this when the system is otherwise idle.
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 13 lines · Christoph Lameter
Quoted 11 lines · Con Kolivas
Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram and changes in free ram representing disk writes or reads also prevent prefetching.
It then checks that we have spare ram looking for at least 3*
pages_highfree per zone and if it succeeds that will prefetch pages from swap into the swap cache. The pages are added to the tail of the inactive list to preserve LRU ordering.
Quoted 4 lines
spare ram when swapping??? We are already under memory pressure. Why make it worse by getting rid of the few bits of available memory? If a system swaps then we are per definition in the bad performance range. Add more memory.
This patch only prefetches the swapped pages when idle and tacks them onto the end of the inactive list so they will be the first thing that is removed if we need ram in the future.
Ordinary desktops swap under ordinary workloads day to day on normal pcs. It is extraordinary common that one workload will swap out other applications and then when complete we have free ram, yet our applications are floundering on swapspace.
Your solution of add more memory is unrealistic. Desktops do have a limit on how much ram we are likely to have/can afford and our workloads have, do, and will for ever more outstrip all available ram. A huge chunk of linux in the wild is on these sort of desktops. This patch makes a substantial difference to ordinary desktops, is configurable on or off and it can be runtime enabled/disabled.
Cheers,
Con
Hello,
Quoted 10 lines · Con Kolivas
Everything is working nicely now, surviving a serious thrashing. Overall the behaviour I would say has improved as a result of all the modifications.
Many thanks go to Nick and Andrew for suggestions.
Andrew please consider this version for -mm.
Cheers,
Con
I'm what one would call a typical desktop user. I also very much a layman with kernels. The layman is very interested in believing, if not knowing, that the kernel is doing everything within its power to provide application responsiveness, especially during heavy application loads.
Swap Prefetch has quantifiable and visible results when an application is swapped out and then swapped back in after a idle period (a very logical time to do things). That /is/ an obvious interaction benefit.
I play World of Warcraft daily and it forces many applications into swap. That said, I now regularly experience desktop conditions that are _very_ favorable with the Swap Prefetch patch soon as WoW is cleared out and resources are regained for productive desktop use (browser, email, chat, media, etc). Hard disk activity, after an idle period, almost never get in the way of interactivity (likely due to swapping out) -- because they're back in physical memory. That is _very_ good use of time and resources.
I'm very interested in seeing this patch applied to MM for major testing, so that others with typical desktop demands can get an idea of how this patch is beneficial.
Thanks for your attention,
Best of regards,
Ryan M.