Skip to contentSkip to conversation

vm - swap_prefetch v12

PATCH In review Con Kolivas 2 participants 6 messages Sep 29, 2005

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vm - swap_prefetch v12

Latest version of swap prefetching also available in incremental form and for download here:
http://ck.kolivas.org/patches/swap-prefetch/

-One unlocking bug was fixed
-Pages are prefetched inactive now so they're the first thing that is swapped back out again under pressure
-Numerous tweaks and cleanups including:
-Acted on suggestion by John Corbet (thanks!)

Cheers,
Con

This patch implements swap prefetching when the vm is relatively idle and there is free ram available. The code is based on some early work by Thomas Schlichter.

This stores a list of swapped entries in a list ordered most recently used and a radix tree. It generates a low priority kernel thread running at nice 19 to do the prefetching at a later stage.

Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram also prevent prefetching. It then checks that we have spare ram looking for at least 3* pages_high free per zone and if it succeeds that will prefetch pages from swap. The pages are prefetched in 128kb groups every 1 second until the vm is busy for the tests above, the watermarks fail to detect adequate free ram or the list is emptied. The pages are copied to swap cache and kept on backing store. This allows pressure on either physical ram or swap to readily find free pages without further I/O.

In testing on modern pc hardware this results in wall-clock time activation of the firefox browser to speed up 5 fold after a worst case complete swap-out of the browser on an static web page.

 include/linux/swap.h |   32 ++++
 init/Kconfig         |   21 ++
 mm/Makefile          |    1
 mm/page_alloc.c      |   14 +
 mm/swap.c            |    3
 mm/swap_prefetch.c   |  408 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/swap_state.c      |   10 +
 mm/vmscan.c          |    5
 8 files changed, 489 insertions(+), 5 deletions(-)
include/linux/swap.h +32 -0
Index: linux-2.6.13-sp/include/linux/swap.h
===================================================================
@@ -185,6 +185,37 @@ extern int shmem_unuse(swp_entry_t entry
 
 extern void swap_unplug_io_fn(struct backing_dev_info *, struct page *);
 
+#ifdef CONFIG_SWAP_PREFETCH
+/* only used by prefetch externally */
+/*	mm/swap_prefetch.c */
+extern void prepare_prefetch(void);
+extern void add_to_swapped_list(unsigned long index);
+extern void remove_from_swapped_list(unsigned long index);
+extern void delay_prefetch(void);
+/* linux/mm/page_alloc.c */
+extern struct page *
+buffered_rmqueue(struct zone *zone, int order, unsigned int __nocast gfp_flags);
+extern void zone_statistics(struct zonelist *zonelist, struct zone *z);
+
+#else	/* CONFIG_SWAP_PREFETCH */
+static inline void add_to_swapped_list(unsigned long index)
+{
+}
+
+static inline void prepare_prefetch(void)
+{
+}
+
+static inline void remove_from_swapped_list(unsigned long index)
+{
+}
+
+static inline void delay_prefetch(void)
+{
+}
+
+#endif	/* CONFIG_SWAP_PREFETCH */
+
 #ifdef CONFIG_SWAP
 /* linux/mm/page_io.c */
 extern int swap_readpage(struct file *, struct page *);
@@ -206,6 +237,7 @@ extern void free_pages_and_swap_cache(st
 extern struct page * lookup_swap_cache(swp_entry_t);
 extern struct page * read_swap_cache_async(swp_entry_t, struct vm_area_struct *vma,
 					   unsigned long addr);
+extern int add_to_swap_cache(struct page *page, swp_entry_t entry);
 /* linux/mm/swapfile.c */
 extern long total_swap_pages;
 extern unsigned int nr_swapfiles;
init/Kconfig +21 -0
Index: linux-2.6.13-sp/init/Kconfig
===================================================================
@@ -87,6 +87,27 @@ config SWAP
 	  used to provide more virtual memory than the actual RAM present
 	  in your computer.  If unsure say Y.
 
+config SWAP_PREFETCH
+	bool "Support for prefetching swapped memory"
+	depends on SWAP && EXPERIMENTAL
+	default n
+	---help---
+	  This option will allow the kernel to prefetch swapped memory pages
+	  when idle. The pages will be kept on both swap and in swap_cache
+	  thus avoiding the need for further I/O if either ram or swap space
+	  is required.
+	  
+	  What this will do on workstations is slowly bring back applications
+	  that have swapped out after memory intensive workloads back into
+	  physical ram if you have free ram at a later stage and the machine
+	  is relatively idle. This means that when you come back to your
+	  computer after leaving it idle for a while, applications will come
+	  to life faster. Note that your swap usage will appear to increase
+	  but these are cached pages, can be dropped freely by the vm, and it
+	  should stabilise around 50% swap usage.
+	  
+	  Desktop users will most likely want to say Y.
+
 config SYSVIPC
 	bool "System V IPC"
 	depends on MMU
mm/Makefile +1 -0
Index: linux-2.6.13-sp/mm/Makefile
===================================================================
@@ -13,6 +13,7 @@ obj-y			:= bootmem.o filemap.o mempool.o
 			   prio_tree.o $(mmu-y)
 
 obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o thrash.o
+obj-$(CONFIG_SWAP_PREFETCH) += swap_prefetch.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o
 obj-$(CONFIG_NUMA) 	+= mempolicy.o
 obj-$(CONFIG_SPARSEMEM)	+= sparse.o
mm/page_alloc.c +10 -4
Index: linux-2.6.13-sp/mm/page_alloc.c
===================================================================
@@ -607,7 +607,7 @@ void drain_local_pages(void)
 }
 #endif /* CONFIG_PM */
 
-static void zone_statistics(struct zonelist *zonelist, struct zone *z)
+void zone_statistics(struct zonelist *zonelist, struct zone *z)
 {
 #ifdef CONFIG_NUMA
 	unsigned long flags;
@@ -684,7 +684,7 @@ static inline void prep_zero_page(struct
  * we cheat by calling it from here, in the order > 0 path.  Saves a branch
  * or two.
  */
-static struct page *
+struct page *
 buffered_rmqueue(struct zone *zone, int order, unsigned int __nocast gfp_flags)
 {
 	unsigned long flags;
@@ -745,7 +745,7 @@ int zone_watermark_ok(struct zone *z, in
 		min -= min / 4;
 
 	if (free_pages <= min + z->lowmem_reserve[classzone_idx])
-		return 0;
+		goto out_failed;
 	for (o = 0; o < order; o++) {
 		/* At the next order, this order's pages become unavailable */
 		free_pages -= z->free_area[o].nr_free << o;
@@ -754,9 +754,15 @@ int zone_watermark_ok(struct zone *z, in
 		min >>= 1;
 
 		if (free_pages <= min)
-			return 0;
+			goto out_failed;
 	}
+
 	return 1;
+out_failed:
+	/* Swap prefetching is delayed if any watermark is low */
+	delay_prefetch();
+
+	return 0;	
 }
 
 static inline int
mm/swap.c +3 -0
Index: linux-2.6.13-sp/mm/swap.c
===================================================================
@@ -481,5 +481,8 @@ void __init swap_setup(void)
 	 * Right now other parts of the system means that we
 	 * _really_ don't want to cluster much more
 	 */
+
+	prepare_prefetch();
+
 	hotcpu_notifier(cpu_swap_callback, 0);
 }
mm/swap_prefetch.c +408 -0

408 lines not on this page · raw message

mm/swap_state.c +9 -1
Index: linux-2.6.13-sp/mm/swap_state.c
===================================================================
@@ -80,6 +80,7 @@ static int __add_to_swap_cache(struct pa
 		error = radix_tree_insert(&swapper_space.page_tree,
 						entry.val, page);
 		if (!error) {
+			remove_from_swapped_list(entry.val);
 			page_cache_get(page);
 			SetPageLocked(page);
 			SetPageSwapCache(page);
@@ -93,11 +94,12 @@ static int __add_to_swap_cache(struct pa
 	return error;
 }
 
-static int add_to_swap_cache(struct page *page, swp_entry_t entry)
+int add_to_swap_cache(struct page *page, swp_entry_t entry)
 {
 	int error;
 
 	if (!swap_duplicate(entry)) {
+		remove_from_swapped_list(entry.val);
 		INC_CACHE_INFO(noent_race);
 		return -ENOENT;
 	}
@@ -145,6 +147,9 @@ int add_to_swap(struct page * page)
 	swp_entry_t entry;
 	int err;
 
+	/* Swap prefetching is delayed if we're swapping pages */
+	delay_prefetch();
+
 	if (!PageLocked(page))
 		BUG();
 
@@ -325,6 +330,9 @@ struct page *read_swap_cache_async(swp_e
 	struct page *found_page, *new_page = NULL;
 	int err;
 
+	/* Swap prefetching is delayed if we're already reading from swap */
+	delay_prefetch();
+
 	do {
 		/*
 		 * First check the swap cache.  Since this is normally
mm/vmscan.c +5 -0
Index: linux-2.6.13-sp/mm/vmscan.c
===================================================================
@@ -519,6 +519,7 @@ static int shrink_list(struct list_head 
 #ifdef CONFIG_SWAP
 		if (PageSwapCache(page)) {
 			swp_entry_t swap = { .val = page->private };
+			add_to_swapped_list(swap.val);
 			__delete_from_swap_cache(page);
 			write_unlock_irq(&mapping->tree_lock);
 			swap_free(swap);
@@ -929,6 +930,8 @@ int try_to_free_pages(struct zone **zone
 	unsigned long lru_pages = 0;
 	int i;
 
+	delay_prefetch();
+
 	sc.gfp_mask = gfp_mask;
 	sc.may_writepage = 0;
 	sc.may_swap = 1;
@@ -1275,6 +1278,8 @@ int shrink_all_memory(int nr_pages)
 		.reclaimed_slab = 0,
 	};
 
+	delay_prefetch();
+
 	current->reclaim_state = &reclaim_state;
 	for_each_pgdat(pgdat) {
 		int freed;
#1 lore raw
Quoted 8 lines · Con Kolivas

Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram also prevent prefetching. It then checks that we have spare ram looking for at least 3* pages_high free per zone and if it succeeds that will prefetch pages from swap.

Did you consider poking around in gendisk.disk_stats to determine whether the swap disk(s) are idleish?

#2 lore raw
Quoted 11 lines · Andrew Morton
Quoted 7 lines · Con Kolivas

Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram also prevent prefetching. It then checks that we have spare ram looking for at least 3* pages_high free per zone and if it succeeds that will prefetch pages from swap.

Did you consider poking around in gendisk.disk_stats to determine whether the swap disk(s) are idleish?

I didn't know where to look for that info. Thanks! I'm open to *any* suggestions and I'll look into it as I can't take this code much further without outside help.

Cheers,
Con

#3 lore raw
Quoted 11 lines · Con Kolivas
Quoted 9 lines · Andrew Morton
Quoted 7 lines · Con Kolivas

Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram also prevent prefetching. It then checks that we have spare ram looking for at least 3* pages_high free per zone and if it succeeds that will prefetch pages from swap.

Quoted 7 lines
Quoted 2 lines

Did you consider poking around in gendisk.disk_stats to determine whether the swap disk(s) are idleish?

I didn't know where to look for that info. Thanks! I'm open to *any* suggestions and I'll look into it as I can't take this code much further without outside help.

Might also be able to utilise CFQ's I/O priorities. That should be more efective than a heuristic based on disk_stats, however you'd probably need to work out whether the swapdev actually supports IO priorities and I'm not sure how one would query that (cleanly).

#4 lore raw
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Con Kolivas
Quoted 11 lines · Andrew Morton
Quoted 7 lines · Con Kolivas

Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram also prevent prefetching. It then checks that we have spare ram looking for at least 3* pages_high free per zone and if it succeeds that will prefetch pages from swap.

Did you consider poking around in gendisk.disk_stats to determine whether the swap disk(s) are idleish?

Quoted 8 lines
Quoted 3 lines

I didn't know where to look for that info. Thanks! I'm open to *any* suggestions and I'll look into it as I can't take this code much further without outside help.

Might also be able to utilise CFQ's I/O priorities. That should be more efective than a heuristic based on disk_stats, however you'd probably need to work out whether the swapdev actually supports IO priorities and I'm not sure how one would query that (cleanly).

Good idea. It should be easy to simply add the ioprio value to the kprefetchd thread and it either supports it or it doesn't depending on what iosched is being used.

Further to your original suggestion, prefetching is already delayed when any non-prefetch related swap in or out is occurring already so checking diskstats is probably not going to add to that for swap activity. However I assume it (diskstats) could also be used for all disk i/o as well? Currently a lot of dirty data can tell me that lots of writing is happening but I have no way of checking that lots of reads are occurring.

Cheers,
Con

#5 lore raw
Quoted 20 lines · Con Kolivas
Quoted 18 lines · Andrew Morton
Quoted 16 lines · Con Kolivas
Quoted 11 lines · Andrew Morton
Quoted 7 lines · Con Kolivas

Once pages have been added to the swapped list, a timer is started, testing for conditions suitable to prefetch swap pages every 5 seconds. Suitable conditions are defined as lack of swapping out or in any pages, and no watermark tests failing. Significant amounts of dirtied ram also prevent prefetching. It then checks that we have spare ram looking for at least 3* pages_high free per zone and if it succeeds that will prefetch pages from swap.

Did you consider poking around in gendisk.disk_stats to determine whether the swap disk(s) are idleish?

I didn't know where to look for that info. Thanks! I'm open to *any* suggestions and I'll look into it as I can't take this code much further without outside help.

Quoted 8 lines
Quoted 4 lines

Might also be able to utilise CFQ's I/O priorities. That should be more efective than a heuristic based on disk_stats, however you'd probably need to work out whether the swapdev actually supports IO priorities and I'm not sure how one would query that (cleanly).

Good idea. It should be easy to simply add the ioprio value to the kprefetchd thread and it either supports it or it doesn't depending on what iosched is being used.

Right. But if the queue doesn't support io priorities then one might want to fall back to monitoring recent queue activity.

Quoted 4 lines

Further to your original suggestion, prefetching is already delayed when any non-prefetch related swap in or out is occurring already so checking diskstats is probably not going to add to that for swap activity. However I assume it (diskstats) could also be used for all disk i/o as well?

Precisely. Few people would set an entire disk aside for swap.

Quoted 3 lines

Currently
a lot of dirty data can tell me that lots of writing is happening but I have no way of checking that lots of reads are occurring.

diskstats will tell that. Also one could peek at the current disk queue levels (request_list.count[]).