Guest page hinting: volatile page cache.

[patch 3/9] Guest page hinting: volatile page cache.

A new page state "volatile" is introduced that is used for clean, uptodate page cache pages. The host can choose to discard volatile pages as part of its vmscan operation instead of writing them to the hosts paging device. This greatly reduces the i/o needed by the host if it gets under memory pressure. The guest system doesn't notice that a volatile page is gone until it tries to access the page or tries to make it stable again. Then the guest needs to remove the page from the cache. A guest access to a discarded page causes the host to deliver a new kind of fault to the guest - the discard fault. After the guest has removed the page it is reloaded from the backing device.

The code added by this patch uses the volatile state for all page cache pages, even for pages which are referenced by writable ptes. The host needs to be able to check the dirty state of the pages. Since the host doesn't know where the page table entries of the guest are located, the volatile state as introduced by this patch is only usable on architectures with per-page dirty bits (s390 only). For per-pte dirty bit architectures some additional code is needed.

The interesting question is where to put the state transitions between the volatile and the stable state. The simple solution is the make a page stable whenever a lookup is done or a page reference is derived from a page table entry. Attempts to make pages volatile are added at strategic points. Now what are the conditions that prevent a page from being made volatile? There are 10 conditions:
1) The page is reserved. Some sort of special page.
2) The page is marked dirty in the struct page. The page content is more recent than the data on the backing device. The host cannot access the linux internal dirty bit so the page needs to be stable.
3) The page is in writeback. The page content is needed for i/o.
4) The page is locked. Someone has exclusive access to the page.
5) The page is anonymous. Swap cache support needs additional code.
6) The page has no mapping. No backing, the page cannot be recreated.
7) The page is not uptodate.
8) The page has private information. try_to_release_page can fail, e.g. in case the private information is journaling data. The discard fault need to be able to remove the page.
9) The page is already discarded.
10) The page map count is not equal to the page reference count - 1. The discard fault handler can remove the page cache reference and all mappers of a page. It cannot remove the page reference for any other user of the page.

The transitions to stable are done by find_get_pages() and its variants, in get_user_pages if called with a pages parameter, by copy-on-write in do_wp_page, and by the early copy-on-write in do_no_page. To make enough pages discardable by the host an attempt to do the transition to volatile state is done when a page gets unlocked (unlock_page), when writeback has finished (test_clear_page_dirty), when the page reference counter is decreased (put_page_testzero), and when the page map counter is increased (page_add_file_rmap).

All page references acquired with find_get_page and friends can be used to access the page frame content. A page reference grabbed from a page table cannot be used to access the page content. A page_make_stable needs to be done first and if that fails the page needs to get discarded. That removes the page table entry as well.

Two new page flags are added. To guard against concurrent page state updates the PG_state_change flag is used. It prevents that a transition to the stable state can "overtake" a transition to volatile state. If page_make_volatile has already done the 10 checks it will issue the state change primitive. If in the meantime on of the conditions has changed the user that requires the page in stable state will have to wait until the make volatile operation has finished. The make volatile operation does not wait for the PG_state_change if make stable has set the bit. Instead the attempt to make the page volatile fails. The reason is that test_clear_page_dirty is called from interrupt context and waiting for the bit might dead-lock.

The second new page flag is the PG_discarded bit. This bit is set for discarded pages. It prevents multiple removes of a page from the page cache due to concurrent discard faults and/or normal page removals. It also prevents the re-add of isolated pages to the lru list in vmscan if the page has been discarded while it was not on the lru list.

Another noticable change is that the first few lines of code in try_to_unmap_one that calculates the address from the page and the vma is moved out of try_to_unmap_one to the callers. This is done to make try_to_unmap_one usable for the removal of discarded pages in page_unmap_all.

---

 fs/buffer.c                 |    1 
 include/linux/mm.h          |    8 +
 include/linux/page-flags.h  |   13 +++
 include/linux/page-states.h |   55 +++++++++++++
 include/linux/pagemap.h     |    1 
 mm/Makefile                 |    1 
 mm/filemap.c                |   73 +++++++++++++++++-
 mm/memory.c                 |   44 ++++++++++
 mm/page-discard.c           |  178 ++++++++++++++++++++++++++++++++++++++++++++
 mm/page-writeback.c         |    3 
 mm/page_alloc.c             |    3 
 mm/rmap.c                   |   68 ++++++++++++++--
 mm/vmscan.c                 |   56 +++++++++----
 13 files changed, 471 insertions(+), 33 deletions(-)

diff -urpN linux-2.6/fs/buffer.c linux-2.6-patched/fs/buffer.c
@@ -105,6 +105,7 @@ __clear_page_buffers(struct page *page)
 {
 	ClearPagePrivate(page);
 	set_page_private(page, 0);
+	page_make_volatile(page, 2);
 	page_cache_release(page);
 }
 

diff -urpN linux-2.6/include/linux/mm.h linux-2.6-patched/include/linux/mm.h
@@ -308,11 +308,17 @@ struct page {
 
 /*
  * Drop a ref, return true if the refcount fell to zero (the page has no users)
+ * put_page_testzero checks if the page can be made volatile if the page
+ * still has users and guest page hinting is enabled.
  */
 static inline int put_page_testzero(struct page *page)
 {
+	int ret;
 	VM_BUG_ON(atomic_read(&page->_count) == 0);
-	return atomic_dec_and_test(&page->_count);
+	ret = atomic_dec_and_test(&page->_count);
+	if (!ret)
+		page_make_volatile(page, 1);
+	return ret;
 }
 
 /*

diff -urpN linux-2.6/include/linux/page-flags.h linux-2.6-patched/include/linux/page-flags.h
@@ -105,6 +105,9 @@
 #define PG_uncached		31	/* Page has been mapped as uncached */
 #endif
 
+#define PG_state_change		21	/* HV page state is changing. */
+#define PG_discarded		22	/* HV page has been discarded. */
+
 /*
  * Manipulation of page state flags
  */

@@ -254,6 +257,16 @@
 #define SetPageReadahead(page)	set_bit(PG_readahead, &(page)->flags)
 #define TestClearPageReadahead(page) test_and_clear_bit(PG_readahead, &(page)->flags)
 
+#define PageStateChange(page) test_bit(PG_state_change, &(page)->flags)
+#define ClearPageStateChange(page) clear_bit(PG_state_change, &(page)->flags)
+#define TestSetPageStateChange(page) \
+		test_and_set_bit(PG_state_change, &(page)->flags)
+
+#define PageDiscarded(page)	test_bit(PG_discarded, &(page)->flags)
+#define ClearPageDiscarded(page) clear_bit(PG_discarded, &(page)->flags)
+#define TestSetPageDiscarded(page) \
+		test_and_set_bit(PG_discarded, &(page)->flags)
+
 struct page;	/* forward declaration */
 
 int test_clear_page_dirty(struct page *page);

diff -urpN linux-2.6/include/linux/pagemap.h linux-2.6-patched/include/linux/pagemap.h
@@ -114,6 +114,7 @@ int add_to_page_cache_lru(struct page *p
 				unsigned long index, gfp_t gfp_mask);
 extern void remove_from_page_cache(struct page *page);
 extern void __remove_from_page_cache(struct page *page);
+extern void __remove_from_page_cache_nocheck(struct page *page);
 
 /*
  * Return byte-offset into filesystem object for page.

diff -urpN linux-2.6/mm/Makefile linux-2.6-patched/mm/Makefile
@@ -25,3 +25,4 @@ obj-$(CONFIG_MEMORY_HOTPLUG) += memory_h
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 
+obj-$(CONFIG_PAGE_STATES) += page-discard.o

diff -urpN linux-2.6/mm/page_alloc.c linux-2.6-patched/mm/page_alloc.c
@@ -185,7 +185,8 @@ static void bad_page(struct page *page)
 			1 << PG_slab    |
 			1 << PG_swapcache |
 			1 << PG_writeback |
-			1 << PG_buddy );
+			1 << PG_buddy |
+			1 << PG_discarded );
 	set_page_count(page, 0);
 	reset_page_mapcount(page);
 	page->mapping = NULL;

diff -urpN linux-2.6/mm/page-writeback.c linux-2.6-patched/mm/page-writeback.c
@@ -724,6 +724,7 @@ int test_clear_page_dirty(struct page *p
 			radix_tree_tag_clear(&mapping->page_tree,
 						page_index(page),
 						PAGECACHE_TAG_DIRTY);
+			page_make_volatile(page, 2);
 			write_unlock_irqrestore(&mapping->tree_lock, flags);
 			/*
 			 * We can continue to use `mapping' here because the

@@ -790,6 +791,8 @@ int test_clear_page_writeback(struct pag
 						page_index(page),
 						PAGECACHE_TAG_WRITEBACK);
 		write_unlock_irqrestore(&mapping->tree_lock, flags);
+		if (ret)
+			page_make_volatile(page, 2);
 	} else {
 		ret = TestClearPageWriteback(page);
 	}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>