Page host virtual assist: volatile page cache.

[patch 2/8] Page host virtual assist: volatile page cache.

A new page state "volatile" is introduced that is used for clean, uptodate page cache pages. The host can choose to discard volatile pages as part of its vmscan operation instead of writing them to the hosts paging device. This greatly reduces the i/o needed by the host if it gets under memory pressure. The guest system doesn't notice that a volatile page is gone until it tries to access the page or tries to make it stable again. Then the guest needs to remove the page from the cache. A guest access to a discarded page causes the host to deliver a new kind of fault to the guest - the discard fault. After the guest has removed the page it is reloaded from the backing device.

The code added by this patch uses the volatile state for all page cache pages, even for pages which are referenced by writable ptes. The host needs to be able to check the dirty state of the pages. Since the host doesn't know where the page table entries of the guest are located, the volatile state as introduced by this patch is only usable on architectures with per-page dirty bits (s390 only). For per-pte dirty bit architectures some additional code is needed.

The interesting question is where to put the state transitions between the volatile and the stable state. The simple solution is the make a page stable whenever a lookup is done or a page reference is derived from a page table entry. Attempts to make pages volatile are added at strategic points. Now what are the conditions that prevent a page from being made volatile? There are 10 conditions:
1) The page is reserved. Some sort of special page.
2) The page is marked dirty in the struct page. The page content is more recent than the data on the backing device. The host cannot access the linux internal dirty bit so the page needs to be stable.
3) The page is in writeback. The page content is needed for i/o.
4) The page is locked. Someone has exclusive access to the page.
5) The page is anonymous. Swap cache support needs additional code.
6) The page has no mapping. No backing, the page cannot be recreated.
7) The page is not uptodate.
8) The page has private information. try_to_release_page can fail, e.g. in case the private information is journaling data. The discard fault need to be able to remove the page.
9) The page is already discarded.
10) The page map count is not equal to the page reference count - 1. The discard fault handler can remove the page cache reference and all mappers of a page. It cannot remove the page reference for any other user of the page.

The transitions to stable are done by find_get_pages() and its variants, in get_user_pages if called with a pages parameter, by copy-on-write in do_wp_page, and by the early copy-on-write in do_no_page. To make enough pages discardable by the host an attempt to do the transition to volatile state is done when a page gets unlocked (unlock_page), when writeback has finished (test_clear_page_dirty), when the page reference counter is decreased (put_page_testzero), and when the page map counter is increased (page_add_file_rmap).

All page references acquired with find_get_page and friends can be used to access the page frame content. A page reference grabbed from a page table cannot be used to access the page content. A page_hva_make_stable needs to be done first and if that fails the page needs to get discarded. That removes the page table entry as well.

Two new page flags are added. To guard against concurrent page state updates the PG_state_change flag is used. It prevents that a transition to the stable state can "overtake" a transition to volatile state. If page_hva_make_volatile has already done the 1e checks it will issue the state change primitive. If in the meantime on of the conditions has changed the user that requires the page in stable state will have to wait until the make volatile operation has finished. The make volatile operation does not wait for the PG_state_change if make stable has set the bit. Instead the attempt to make the page volatile fails. The reason is that test_clear_page_dirty is called from interrupt context and waiting for the bit might dead-lock.

The second new page flag is the PG_discarded bit. This bit is set after a discarded page has been recognized. It is used to call the unmap function only for the first caller of the discard handler and it is used to do the late removal of page cache pages that have been isolated in vmscan. After an isolated page has been re-added to the lru list PG_discarded is tested and if it is set the discard handler is called.

Another noticable change is that the first few lines of code in try_to_unmap_one that calculates the address from the page and the vma is moved out of try_to_unmap_one to the callers. This is done to make try_to_unmap_one usable for the removal of discarded pages in page_hva_unmap_all.

---

 include/linux/mm.h         |   11 ++
 include/linux/page-flags.h |   14 +++
 include/linux/page_hva.h   |   29 +++++++
 mm/Makefile                |    1 
 mm/filemap.c               |   49 +++++++++++-
 mm/memory.c                |   45 ++++++++++-
 mm/page-writeback.c        |    1 
 mm/page_hva.c              |  174 +++++++++++++++++++++++++++++++++++++++++++++
 mm/rmap.c                  |   68 +++++++++++++++--
 mm/vmscan.c                |   18 +++-
 10 files changed, 391 insertions(+), 19 deletions(-)

diff -urpN linux-2.6/include/linux/mm.h linux-2.6-patched/include/linux/mm.h
@@ -302,12 +302,19 @@ struct page {
 
 /*
  * Drop a ref, return true if the logical refcount fell to zero (the page has
- * no users)
+ * no users).
+ *
+ * put_page_testzero checks if the page can be made volatile if the page
+ * still has users and the page host virtual assist is enabled.
  */
 static inline int put_page_testzero(struct page *page)
 {
+	int ret;
 	VM_BUG_ON(atomic_read(&page->_count) == 0);
-	return atomic_dec_and_test(&page->_count);
+	ret = atomic_dec_and_test(&page->_count);
+	if (!ret)
+		page_hva_make_volatile(page, 1);
+	return ret;
 }
 
 /*

diff -urpN linux-2.6/include/linux/page-flags.h linux-2.6-patched/include/linux/page-flags.h
@@ -102,6 +102,9 @@
 #define PG_uncached		31	/* Page has been mapped as uncached */
 #endif
 
+#define PG_state_change		20	/* HV page state is changing. */
+#define PG_discarded		21	/* HV page has been discarded. */
+
 /*
  * Global page accounting.  One instance per CPU.  Only unsigned longs are
  * allowed.

@@ -372,6 +375,17 @@ extern void __mod_page_state_offset(unsi
 #define SetPageUncached(page)	set_bit(PG_uncached, &(page)->flags)
 #define ClearPageUncached(page)	clear_bit(PG_uncached, &(page)->flags)
 
+#define PageStateChange(page) test_bit(PG_state_change, &(page)->flags)
+#define ClearPageStateChange(page) clear_bit(PG_state_change, &(page)->flags)
+#define TestSetPageStateChange(page) \
+		test_and_set_bit(PG_state_change, &(page)->flags)
+
+#define PageDiscarded(page)	test_bit(PG_discarded, &(page)->flags)
+#define SetPageDiscarded(page)	set_bit(PG_discarded, &(page)->flags)
+#define ClearPageDiscarded(page) clear_bit(PG_discarded, &(page)->flags)
+#define TestSetPageDiscarded(page) \
+		test_and_set_bit(PG_discarded, &(page)->flags)
+
 struct page;	/* forward declaration */
 
 int test_clear_page_dirty(struct page *page);

diff -urpN linux-2.6/include/linux/page_hva.h linux-2.6-patched/include/linux/page_hva.h
@@ -16,12 +16,41 @@
 
 #include <asm/page_hva.h>
 
+extern void page_hva_unmap_all(struct page *page);
+extern void page_hva_discard_page(struct page *page);
+
+extern int  __page_hva_make_stable(struct page *page);
+extern void __page_hva_make_volatile(struct page *page, unsigned int offset);
+
+static inline int page_hva_make_stable(struct page *page)
+{
+	if (!page_hva_enabled())
+		return 1;
+	return __page_hva_make_stable(page);
+}
+
+static inline void page_hva_make_volatile(struct page *page,
+					  unsigned int offset)
+{
+	if (!page_hva_enabled())
+		return;
+	if (likely(!test_bit(PG_discarded, &page->flags)))
+		__page_hva_make_volatile(page, offset);
+}
+
 #else
 
 #define page_hva_enabled()			(0)
 
 #define page_hva_set_unused(_page)		do { } while (0)
 #define page_hva_set_stable(_page)		do { } while (0)
+#define page_hva_set_volatile(_page)		do { } while (0)
+#define page_hva_cond_set_stable(_page)		(1)
+
+#define page_hva_make_stable(_page)		(1)
+#define page_hva_make_volatile(_page,_offset)	do { } while (0)
+
+#define page_hva_discard_page(_page)		do { } while (0)
 
 #endif
 

diff -urpN linux-2.6/mm/Makefile linux-2.6-patched/mm/Makefile
@@ -25,3 +25,4 @@ obj-$(CONFIG_MEMORY_HOTPLUG) += memory_h
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 
+obj-$(CONFIG_PAGE_HVA) 	+= page_hva.o

diff -urpN linux-2.6/mm/page-writeback.c linux-2.6-patched/mm/page-writeback.c
@@ -727,6 +727,7 @@ int test_clear_page_dirty(struct page *p
 			radix_tree_tag_clear(&mapping->page_tree,
 						page_index(page),
 						PAGECACHE_TAG_DIRTY);
+			page_hva_make_volatile(page, 1);
 			write_unlock_irqrestore(&mapping->tree_lock, flags);
 			if (mapping_cap_account_dirty(mapping))
 				dec_page_state(nr_dirty);

diff -urpN linux-2.6/mm/vmscan.c linux-2.6-patched/mm/vmscan.c
@@ -669,7 +669,11 @@ static unsigned long shrink_inactive_lis
 				add_page_to_active_list(zone, page);
 			else
 				add_page_to_inactive_list(zone, page);
-			if (!pagevec_add(&pvec, page)) {
+			if (page_hva_enabled() && unlikely(PageDiscarded(page))) {
+				spin_unlock_irq(&zone->lru_lock);
+				page_hva_discard_page(page);
+				spin_lock_irq(&zone->lru_lock);
+			} else if (!pagevec_add(&pvec, page)) {
 				spin_unlock_irq(&zone->lru_lock);
 				__pagevec_release(&pvec);
 				spin_lock_irq(&zone->lru_lock);

@@ -790,7 +794,11 @@ static void shrink_active_list(unsigned 
 
 		list_move(&page->lru, &zone->inactive_list);
 		pgmoved++;
-		if (!pagevec_add(&pvec, page)) {
+		if (page_hva_enabled() && unlikely(PageDiscarded(page))) {
+			spin_unlock_irq(&zone->lru_lock);
+			page_hva_discard_page(page);
+			spin_lock_irq(&zone->lru_lock);
+		} else if (!pagevec_add(&pvec, page)) {
 			zone->nr_inactive += pgmoved;
 			spin_unlock_irq(&zone->lru_lock);
 			pgdeactivate += pgmoved;

@@ -818,7 +826,11 @@ static void shrink_active_list(unsigned 
 		VM_BUG_ON(!PageActive(page));
 		list_move(&page->lru, &zone->active_list);
 		pgmoved++;
-		if (!pagevec_add(&pvec, page)) {
+		if (page_hva_enabled() && unlikely(PageDiscarded(page))) {
+			spin_unlock_irq(&zone->lru_lock);
+			page_hva_discard_page(page);
+			spin_lock_irq(&zone->lru_lock);
+		} else if (!pagevec_add(&pvec, page)) {
 			zone->nr_active += pgmoved;
 			pgmoved = 0;
 			spin_unlock_irq(&zone->lru_lock);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>