Skip to contentSkip to conversation

transcendent memory for Linux

RFC In review Dan Magenheimer 9 participants 29 messages Jun 19, 2009 – Jul 2, 2009

Subsystems (10)
Memory management x86 VFS MM core +6 more subsystems
lore
Share & tools
lore
#0 lore raw

RFC 1/1 transcendent memory for Linux

Normal memory is directly addressable by the kernel, of a known normally-fixed size, synchronously accessible, and persistent (though not across a reboot).

What if there was a class of memory that is of unknown and dynamically variable size, is addressable only indirectly by the kernel, can be configured either as persistent or as "ephemeral" (meaning it will be around for awhile, but might disappear without warning), and is still fast enough to be synchronously accessible?

We call this latter class "transcendent memory" and it provides an interesting opportunity to more efficiently utilize RAM in a virtualized environment. However this "memory but not really memory" may also have applications in NON-virtualized environments, such as hotplug-memory deletion, SSDs, and page cache compression. Others have suggested ideas such as allowing use of highmem memory without a highmem kernel, or use of spare video memory.

Transcendent memory, or "tmem" for short, provides a well-defined API to access this unusual class of memory. The basic operations are page-copy-based and use a flexible object-oriented addressing mechanism. Tmem assumes that some "privileged entity" is capable of executing tmem requests and storing pages of data; this entity is currently a hypervisor and operations are performed via hypercalls, but the entity could be a kernel policy, or perhaps a "memory node" in a cluster of blades connected by a high-speed interconnect such as hypertransport or QPI.

Since tmem is not directly accessible and because page copying is done to/from physical pageframes, it more suitable for in-kernel memory needs than for userland applications. However, there may be yet undiscovered userland possibilities.

With the tmem concept outlined vaguely and its broader potential hinted, we will overview two existing examples of how tmem can be used by the kernel. These examples are implemented in the attached (2.6.30-based) patches.

"Precache" can be thought of as a page-granularity victim cache for clean pages that the kernel's pageframe replacement algorithm (PFRA) would like to keep around, but can't since
there isn't enough memory. So when the PFRA "evicts" a page,
it first puts it into the precache via a call to tmem. And any time a filesystem reads a page from disk, it first attempts to get the page from precache. If it's there, a disk access is eliminated. If not, the filesystem just goes to the disk like normal. Precache is "ephemeral" so whether a page is kept in precache (between the "put" and the "get") is dependent on a number of factors that are invisible to the kernel.

"Preswap" IS persistent, but for various reasons may not always be available for use, again due to factors that may not be visible to the kernel (but, briefly, if the kernel is being "good" and has shared its resources nicely, then it will be able to use preswap, else it will not). Once a page is put, a get on the page will always succeed. So when the kernel finds itself in a situation where it needs to swap out a page, it first attempts to use preswap. If the put works, a disk write and (usually) a disk read are avoided. If it doesn't, the page is written to swap as usual. Unlike precache, whether a page is stored in preswap vs swap is recorded in kernel data structures, so when a page needs to be fetched, the kernel does a get if it is in preswap and reads from swap if it is not in preswap.

Both precache and preswap may be optionally compressed, trading off 2x space reduction vs 10x performance for access. Precache also has a sharing feature, which allows different nodes in a "virtual cluster" to share a local page cache.
(In the attached patch, precache is only implemented for ext3 and shared precache is only implemented for ocfs2.)

Tmem has some similarity to IBM's Collaborative Memory Management, but creates more of a partnership between the kernel and the "privileged entity" and is not very invasive. Tmem may be applicable for KVM and containers; there is some disagreement on the extent of its value. Tmem is highly complementary to ballooning (aka page granularity hot plug) and memory deduplication (aka transparent content-based page sharing) but still has value when neither are present.

Performance is difficult to quantify because some benchmarks respond very favorably to increases in memory and tmem may do quite well on those, depending on how much tmem is available which may vary widely and dynamically, depending on conditions completely outside of the system being measured. I'd appreciate ideas on how best to provide useful metrics.

Tmem is now supported in Xen's unstable tree and in Xen's 2.6.18-xen source tree. Again, Xen is not necessarily a requirement, but currently provides the only existing implementation of tmem.

Lots more information about tmem can be found at:
http://oss.oracle.com/projects/tmem and there will be a talk about it on the first day of Linux Symposium next month. Tmem is the result of a group effort, including Chris Mason, Dave McCracken, Kurt Hackel and Zhigang Wang, with helpful input from Jeremy Fitzhardinge, Keir Fraser, Ian Pratt, Sunil Mushran, and Joel Becker

Patches are as follows (organized for review, not for sequential application):
tmeminf.patch infrastructure for tmem layer and API
precache.patch precache implementation (layered on tmem)
preswap.patch preswap implementation (layered on tmem)
tmemxen.patch interface code for tmem on top of Xen

Diffstat below, reorganized to show changed vs new files, and core kernel vs xen. (Also attached in case the formatting gets messed up.)

Any feedback appreciated!

Thanks,
Dan Magenheimer

Changed core kernel files:

 fs/buffer.c                              |    5 +
 fs/ext3/super.c                          |    2 
 fs/mpage.c                               |    8 ++
 fs/ocfs2/super.c                         |    2 
 fs/super.c                               |    5 +
 include/linux/fs.h                       |    7 ++
 include/linux/swap.h                     |   57 +++++++++++++++++++++
 include/linux/sysctl.h                   |    1 
 kernel/sysctl.c                          |   12 ++++
 mm/Kconfig                               |   27 +++++++++
 mm/Makefile                              |    2 
 mm/filemap.c                             |   11 ++++
 mm/page_io.c                             |   12 ++++
 mm/swapfile.c                            |   41 ++++++++++++---
 mm/truncate.c                            |   10 +++
 15 files changed, 196 insertions(+), 6 deletions(-)

Newly added core kernel files:

 include/linux/tmem.h                     |   22 +
 mm/precache.c                            |  146 +++++++++++
 mm/preswap.c                             |  274 +++++++++++++++++++++
 3 files changed, 442 insertions(+)

Changed xen-specific files:

 arch/x86/include/asm/xen/hypercall.h     |    8 +++
 drivers/xen/Makefile                     |    1 
 include/xen/interface/tmem.h             |   43 +++++++++++++++++++++
 include/xen/interface/xen.h              |   22 ++++++++++
 4 files changed, 74 insertions(+)

Newly added xen-specific files:

 drivers/xen/tmem.c                       |  106 +++++++++++++++++++++
 include/xen/interface/tmem.h             |   43 ++++++++
 2 files changed, 149 insertions(+)
mm/Kconfig +27 -0
@@ -253,3 +253,30 @@
 	  of 1 says that all excess pages should be trimmed.
 
 	  See Documentation/nommu-mmap.txt for more information.
+
+#
+# support for transcendent memory
+#
+config TMEM
+	bool "Transcendent memory support"
+	depends on XEN # but in future may work without XEN
+	help
+	  In a virtualized environment, allows unused and underutilized
+	  system physical memory to be made accessible through a narrow
+	  well-defined page-copy-based API.  If unsure, say Y.
+
+config PRECACHE
+	bool "Cache clean pages in transcendent memory"
+	depends on TMEM
+	help
+	  Allows the transcendent memory pool to be used to store clean
+	  page-cache pages which, under some circumstances, will greatly
+	  reduce paging and thus improve performance.  If unsure, say Y.
+
+config PRESWAP
+	bool "Swap pages to transcendent memory"
+	depends on TMEM
+	help
+	  Allows the transcendent memory pool to be used as a pseudo-swap
+	  device which, under some circumstances, will greatly reduce
+	  swapping and thus improve performance.  If unsure, say Y.
mm/Makefile +2 -0
@@ -16,6 +16,8 @@
 obj-$(CONFIG_PROC_PAGE_MONITOR) += pagewalk.o
 obj-$(CONFIG_BOUNCE)	+= bounce.o
 obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o thrash.o
+obj-$(CONFIG_PRESWAP)	+= preswap.o
+obj-$(CONFIG_PRECACHE)	+= precache.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o
 obj-$(CONFIG_NUMA) 	+= mempolicy.o
include/linux/tmem.h +22 -0
@@ -0,0 +1,22 @@
+/*
+ * linux/tmem.h
+ *
+ * Interface to transcendent memory, used by mm/precache.c and mm/preswap.c
+ *
+ * Copyright (C) 2008,2009 Dan Magenheimer, Oracle Corp.
+ */
+
+struct tmem_ops {
+	int (*new_pool)(u64 uuid_lo, u64 uuid_hi, u32 flags);
+	int (*put_page)(u32 pool_id, u64 object, u32 index, unsigned long gmfn);
+	int (*get_page)(u32 pool_id, u64 object, u32 index, unsigned long gmfn);
+	int (*flush_page)(u32 pool_id, u64 object, u32 index);
+	int (*flush_object)(u32 pool_id, u64 object);
+	int (*destroy_pool)(u32 pool_id);
+};
+
+extern struct tmem_ops *tmem_ops;
+
+/* flags for tmem_ops.new_pool */
+#define TMEM_POOL_PERSIST          1
+#define TMEM_POOL_SHARED           2
fs/super.c +5 -0
@@ -39,6 +39,7 @@
 #include <linux/mutex.h>
 #include <linux/file.h>
 #include <linux/async.h>
+#include <linux/precache.h>
 #include <asm/uaccess.h>
 #include "internal.h"
 
@@ -110,6 +111,9 @@
 		s->s_qcop = sb_quotactl_ops;
 		s->s_op = &default_op;
 		s->s_time_gran = 1000000000;
+#ifdef CONFIG_PRECACHE
+		s->precache_poolid = -1;
+#endif
 	}
 out:
 	return s;
@@ -200,6 +204,7 @@
 		vfs_dq_off(s, 0);
 		down_write(&s->s_umount);
 		fs->kill_sb(s);
+		precache_flush_filesystem(s);
 		put_filesystem(fs);
 		put_super(s);
 	}
fs/ext3/super.c +2 -0
@@ -37,6 +37,7 @@
 #include <linux/quotaops.h>
 #include <linux/seq_file.h>
 #include <linux/log2.h>
+#include <linux/precache.h>
 
 #include <asm/uaccess.h>
 
@@ -1306,6 +1307,7 @@
 	} else {
 		printk("internal journal\n");
 	}
+	precache_init(sb);
 	return res;
 }
 
fs/ocfs2/super.c +2 -0
@@ -42,6 +42,7 @@
 #include <linux/mount.h>
 #include <linux/seq_file.h>
 #include <linux/quotaops.h>
+#include <linux/precache.h>
 
 #define MLOG_MASK_PREFIX ML_SUPER
 #include <cluster/masklog.h>
@@ -2162,6 +2163,7 @@
 		mlog_errno(status);
 		goto bail;
 	}
+	shared_precache_init(sb, &di->id2.i_super.s_uuid[0]);
 
 bail:
 	mlog_exit(status);
include/linux/fs.h +7 -0
@@ -1377,6 +1377,13 @@
 	 * storage for asynchronous operations
 	 */
 	struct list_head s_async_list;
+
+#ifdef CONFIG_PRECACHE
+	/*
+	 * saved pool identifier for precache (-1 means none)
+	 */
+	u32 precache_poolid;
+#endif
 };
 
 extern struct timespec current_fs_time(struct super_block *sb);
fs/buffer.c +5 -0
@@ -41,6 +41,7 @@
 #include <linux/bitops.h>
 #include <linux/mpage.h>
 #include <linux/bit_spinlock.h>
+#include <linux/precache.h>
 
 static int fsync_buffers_list(spinlock_t *lock, struct list_head *list);
 
@@ -271,6 +272,10 @@
 
 	invalidate_bh_lrus();
 	invalidate_mapping_pages(mapping, 0, -1);
+	/* 99% of the time, we don't need to flush the precache on the bdev.
+	 * But, for the strange corners, lets be cautious
+	 */
+	precache_flush_inode(mapping);
 }
 
 /*
fs/mpage.c +8 -0
@@ -26,6 +26,7 @@
 #include <linux/writeback.h>
 #include <linux/backing-dev.h>
 #include <linux/pagevec.h>
+#include <linux/precache.h>
 
 /*
  * I/O completion handler for multipage BIOs.
@@ -285,6 +286,13 @@
 		SetPageMappedToDisk(page);
 	}
 
+	if (fully_mapped &&
+	    blocks_per_page == 1 && !PageUptodate(page) &&
+	    precache_get(page->mapping, page->index, page) == 1) {
+		SetPageUptodate(page);
+		goto confused;
+	}
+
 	/*
 	 * This page will go to BIO.  Do we need to send this BIO off first?
 	 */
mm/truncate.c +10 -0
@@ -18,6 +18,7 @@
 #include <linux/task_io_accounting_ops.h>
 #include <linux/buffer_head.h>	/* grr. try_to_release_page,
 				   do_invalidatepage */
+#include <linux/precache.h>
 #include "internal.h"
 
 
@@ -50,6 +51,7 @@
 static inline void truncate_partial_page(struct page *page, unsigned partial)
 {
 	zero_user_segment(page, partial, PAGE_CACHE_SIZE);
+	precache_flush(page->mapping, page->index);
 	if (page_has_private(page))
 		do_invalidatepage(page, partial);
 }
@@ -107,6 +109,10 @@
 	clear_page_mlock(page);
 	remove_from_page_cache(page);
 	ClearPageMappedToDisk(page);
+	/* this must be after the remove_from_page_cache which
+	 * calls precache_put
+	 */
+	precache_flush(mapping, page->index);
 	page_cache_release(page);	/* pagecache ref */
 }
 
@@ -168,6 +174,7 @@
 	pgoff_t next;
 	int i;
 
+	precache_flush_inode(mapping);
 	if (mapping->nrpages == 0)
 		return;
 
@@ -251,6 +258,7 @@
 		}
 		pagevec_release(&pvec);
 	}
+	precache_flush_inode(mapping);
 }
 EXPORT_SYMBOL(truncate_inode_pages_range);
 
@@ -398,6 +406,7 @@
 	int did_range_unmap = 0;
 	int wrapped = 0;
 
+	precache_flush_inode(mapping);
 	pagevec_init(&pvec, 0);
 	next = start;
 	while (next <= end && !wrapped &&
@@ -454,6 +463,7 @@
 		pagevec_release(&pvec);
 		cond_resched();
 	}
+	precache_flush_inode(mapping);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(invalidate_inode_pages2_range);
mm/filemap.c +11 -0
@@ -34,6 +34,7 @@
 #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */
 #include <linux/memcontrol.h>
 #include <linux/mm_inline.h> /* for page_is_file_cache() */
+#include <linux/precache.h>
 #include "internal.h"
 
 /*
@@ -116,6 +117,16 @@
 {
 	struct address_space *mapping = page->mapping;
 
+	/*
+	 * if we're uptodate, flush out into the precache, otherwise
+	 * invalidate any existing precache entries.  We can't leave
+	 * stale data around in the precache once our page is gone
+	 */
+	if (PageUptodate(page))
+		precache_put(page->mapping, page->index, page);
+	else
+		precache_flush(page->mapping, page->index);
+
 	radix_tree_delete(&mapping->page_tree, page->index);
 	page->mapping = NULL;
 	mapping->nrpages--;
include/linux/precache.h +55 -0

55 lines not on this page · raw message

mm/precache.c +146 -0

146 lines not on this page · raw message

mm/page_io.c +12 -0
@@ -102,6 +102,12 @@
 		unlock_page(page);
 		goto out;
 	}
+	if (preswap_put(page) == 1) {
+		set_page_writeback(page);
+		unlock_page(page);
+		end_page_writeback(page);
+		goto out;
+	}
 	bio = get_swap_bio(GFP_NOIO, page_private(page), page,
 				end_swap_bio_write);
 	if (bio == NULL) {
@@ -134,6 +140,12 @@
 		ret = -ENOMEM;
 		goto out;
 	}
+	if (preswap_get(page) == 1) {
+		SetPageUptodate(page);
+		unlock_page(page);
+		bio_put(bio);
+		goto out;
+	}
 	count_vm_event(PSWPIN);
 	submit_bio(READ, bio);
 out:
mm/swapfile.c +35 -6

129 lines not on this page · raw message

include/linux/swap.h +57 -0

77 lines not on this page · raw message

mm/preswap.c +274 -0

274 lines not on this page · raw message

include/linux/sysctl.h +1 -0
@@ -205,6 +205,7 @@
 	VM_PANIC_ON_OOM=33,	/* panic at out-of-memory */
 	VM_VDSO_ENABLED=34,	/* map VDSO into new processes? */
 	VM_MIN_SLAB=35,		 /* Percent pages ignored by zone reclaim */
+	VM_PRESWAP_PAGES=36,	/* pages/target_pages in preswap */
 };
 
 
kernel/sysctl.c +12 -0
@@ -1282,6 +1282,18 @@
 		.proc_handler	= &scan_unevictable_handler,
 	},
 #endif
+#ifdef CONFIG_PRESWAP
+	{
+		.ctl_name	= VM_PRESWAP_PAGES,
+		.procname	= "preswap",
+		.data		= NULL,
+		.maxlen		= sizeof(unsigned long),
+		.mode		= 0644,
+		.proc_handler	= &preswap_sysctl_handler,
+		.extra1		= (void *)&preswap_zero,
+		.extra2		= (void *)&preswap_infinity,
+	},
+#endif
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt
arch/x86/include/asm/xen/hypercall.h +8 -0
@@ -45,6 +45,7 @@
 #include <xen/interface/xen.h>
 #include <xen/interface/sched.h>
 #include <xen/interface/physdev.h>
+#include <xen/interface/tmem.h>
 
 /*
  * The hypercall asms have to meet several constraints:
@@ -417,6 +418,13 @@
 	return _hypercall2(int, nmi_op, op, arg);
 }
 
+static inline int
+HYPERVISOR_tmem_op(
+	struct tmem_op *op)
+{
+	return _hypercall1(int, tmem_op, op);
+}
+
 static inline void
 MULTI_fpu_taskswitch(struct multicall_entry *mcl, int set)
 {
drivers/xen/Makefile +1 -0
@@ -3,5 +3,6 @@
 
 obj-$(CONFIG_HOTPLUG_CPU)	+= cpu_hotplug.o
 obj-$(CONFIG_XEN_XENCOMM)	+= xencomm.o
+obj-$(CONFIG_TMEM)		+= tmem.o
 obj-$(CONFIG_XEN_BALLOON)	+= balloon.o
 obj-$(CONFIG_XENFS)		+= xenfs/
\ No newline at end of file
include/xen/interface/tmem.h +43 -0

43 lines not on this page · raw message

include/xen/interface/xen.h +22 -0
@@ -58,6 +58,7 @@
 #define __HYPERVISOR_event_channel_op     32
 #define __HYPERVISOR_physdev_op           33
 #define __HYPERVISOR_hvm_op               34
+#define __HYPERVISOR_tmem_op              38
 
 /* Architecture-specific hypercall definitions. */
 #define __HYPERVISOR_arch_0               48
@@ -461,6 +462,27 @@
 #define __mk_unsigned_long(x) x ## UL
 #define mk_unsigned_long(x) __mk_unsigned_long(x)
 
+struct tmem_op {
+	uint32_t cmd;
+	int32_t pool_id; /* private > 0; shared < 0; 0 is invalid */
+	union {
+		struct {  /* for cmd == TMEM_NEW_POOL */
+			uint64_t uuid[2];
+			uint32_t flags;
+		} new;
+		struct {
+			uint64_t object;
+			uint32_t index;
+			uint32_t tmem_offset;
+			uint32_t pfn_offset;
+			uint32_t len;
+			GUEST_HANDLE(void) gmfn; /* guest machine page frame */
+		} gen;
+	} u;
+};
+typedef struct tmem_op tmem_op_t;
+DEFINE_GUEST_HANDLE_STRUCT(tmem_op_t);
+
 #else /* __ASSEMBLY__ */
 
 /* In assembly code we cannot use C numeric constant suffixes. */
drivers/xen/tmem.c +106 -0

106 lines not on this page · raw message

Changed core kernel files:

 fs/buffer.c                              |    5 +
 fs/ext3/super.c                          |    2 
 fs/mpage.c                               |    8 ++
 fs/ocfs2/super.c                         |    2 
 fs/super.c                               |    5 +
 include/linux/fs.h                       |    7 ++
 include/linux/swap.h                     |   57 +++++++++++++++++++++
 include/linux/sysctl.h                   |    1 
 kernel/sysctl.c                          |   12 ++++
 mm/Kconfig                               |   27 +++++++++
 mm/Makefile                              |    2 
 mm/filemap.c                             |   11 ++++
 mm/page_io.c                             |   12 ++++
 mm/swapfile.c                            |   41 ++++++++++++---
 mm/truncate.c                            |   10 +++
 15 files changed, 196 insertions(+), 6 deletions(-)

Newly added core kernel files:

 include/linux/tmem.h                     |   22 +
 mm/precache.c                            |  146 +++++++++++
 mm/preswap.c                             |  274 +++++++++++++++++++++
 3 files changed, 442 insertions(+)

Changed xen-specific files:

 arch/x86/include/asm/xen/hypercall.h     |    8 +++
 drivers/xen/Makefile                     |    1 
 include/xen/interface/tmem.h             |   43 +++++++++++++++++++++
 include/xen/interface/xen.h              |   22 ++++++++++
 4 files changed, 74 insertions(+)

Newly added xen-specific files:

 drivers/xen/tmem.c                       |  106 +++++++++++++++++++++
 include/xen/interface/tmem.h             |   43 ++++++++
 2 files changed, 149 insertions(+)
#1 lore raw

cover transcendent memory ("tmem") for Linux

Apologies for the breach of netiquette with attachments. Following up with inline patches in separate emails...

=====================

Normal memory is directly addressable by the kernel, of a known normally-fixed size, synchronously accessible, and persistent (though not across a reboot).

What if there was a class of memory that is of unknown and dynamically variable size, is addressable only indirectly by the kernel, can be configured either as persistent or as "ephemeral" (meaning it will be around for awhile, but might disappear without warning), and is still fast enough to be synchronously accessible?

We call this latter class "transcendent memory" and it provides an interesting opportunity to more efficiently utilize RAM in a virtualized environment. However this "memory but not really memory" may also have applications in NON-virtualized environments, such as hotplug-memory deletion, SSDs, and page cache compression. Others have suggested ideas such as allowing use of highmem memory without a highmem kernel, or use of spare video memory.

Transcendent memory, or "tmem" for short, provides a well-defined API to access this unusual class of memory. The basic operations are page-copy-based and use a flexible object-oriented addressing mechanism. Tmem assumes that some "privileged entity" is capable of executing tmem requests and storing pages of data; this entity is currently a hypervisor and operations are performed via hypercalls, but the entity could be a kernel policy, or perhaps a "memory node" in a cluster of blades connected by a high-speed interconnect such as hypertransport or QPI.

Since tmem is not directly accessible and because page copying is done to/from physical pageframes, it more suitable for in-kernel memory needs than for userland applications. However, there may be yet undiscovered userland possibilities.

With the tmem concept outlined vaguely and its broader potential hinted, we will overview two existing examples of how tmem can be used by the kernel. These examples are implemented in the attached (2.6.30-based) patches.

"Precache" can be thought of as a page-granularity victim cache for clean pages that the kernel's pageframe replacement algorithm (PFRA) would like to keep around, but can't since
there isn't enough memory. So when the PFRA "evicts" a page,
it first puts it into the precache via a call to tmem. And any time a filesystem reads a page from disk, it first attempts to get the page from precache. If it's there, a disk access is eliminated. If not, the filesystem just goes to the disk like normal. Precache is "ephemeral" so whether a page is kept in precache (between the "put" and the "get") is dependent on a number of factors that are invisible to the kernel.

"Preswap" IS persistent, but for various reasons may not always be available for use, again due to factors that may not be visible to the kernel (but, briefly, if the kernel is being "good" and has shared its resources nicely, then it will be able to use preswap, else it will not). Once a page is put, a get on the page will always succeed. So when the kernel finds itself in a situation where it needs to swap out a page, it first attempts to use preswap. If the put works, a disk write and (usually) a disk read are avoided. If it doesn't, the page is written to swap as usual. Unlike precache, whether a page is stored in preswap vs swap is recorded in kernel data structures, so when a page needs to be fetched, the kernel does a get if it is in preswap and reads from swap if it is not in preswap.

Both precache and preswap may be optionally compressed, trading off 2x space reduction vs 10x performance for access. Precache also has a sharing feature, which allows different nodes in a "virtual cluster" to share a local page cache.
(In the attached patch, precache is only implemented for ext3 and shared precache is only implemented for ocfs2.)

Tmem has some similarity to IBM's Collaborative Memory Management, but creates more of a partnership between the kernel and the "privileged entity" and is not very invasive. Tmem may be applicable for KVM and containers; there is some disagreement on the extent of its value. Tmem is highly complementary to ballooning (aka page granularity hot plug) and memory deduplication (aka transparent content-based page sharing) but still has value when neither are present.

Performance is difficult to quantify because some benchmarks respond very favorably to increases in memory and tmem may do quite well on those, depending on how much tmem is available which may vary widely and dynamically, depending on conditions completely outside of the system being measured. I'd appreciate ideas on how best to provide useful metrics.

Tmem is now supported in Xen's unstable tree and in Xen's 2.6.18-xen source tree. Again, Xen is not necessarily a requirement, but currently provides the only existing implementation of tmem.

Lots more information about tmem can be found at:
http://oss.oracle.com/projects/tmem and there will be a talk about it on the first day of Linux Symposium next month. Tmem is the result of a group effort, including Chris Mason, Dave McCracken, Kurt Hackel and Zhigang Wang, with helpful input from Jeremy Fitzhardinge, Keir Fraser, Ian Pratt, Sunil Mushran, and Joel Becker

Patches are as follows (organized for review, not for sequential application):
tmeminf.patch infrastructure for tmem layer and API
precache.patch precache implementation (layered on tmem)
preswap.patch preswap implementation (layered on tmem)
tmemxen.patch interface code for tmem on top of Xen

Diffstat below, reorganized to show changed vs new files, and core kernel vs xen. (Also attached in case the formatting gets messed up.)

Any feedback appreciated!

Thanks,
Dan Magenheimer

Changed core kernel files:

 fs/buffer.c                              |    5 +
 fs/ext3/super.c                          |    2 
 fs/mpage.c                               |    8 ++
 fs/ocfs2/super.c                         |    2 
 fs/super.c                               |    5 +
 include/linux/fs.h                       |    7 ++
 include/linux/swap.h                     |   57 +++++++++++++++++++++
 include/linux/sysctl.h                   |    1 
 kernel/sysctl.c                          |   12 ++++
 mm/Kconfig                               |   27 +++++++++
 mm/Makefile                              |    2 
 mm/filemap.c                             |   11 ++++
 mm/page_io.c                             |   12 ++++
 mm/swapfile.c                            |   41 ++++++++++++---
 mm/truncate.c                            |   10 +++
 15 files changed, 196 insertions(+), 6 deletions(-)

Newly added core kernel files:

 include/linux/tmem.h                     |   22 +
 mm/precache.c                            |  146 +++++++++++
 mm/preswap.c                             |  274 +++++++++++++++++++++
 3 files changed, 442 insertions(+)

Changed xen-specific files:

 arch/x86/include/asm/xen/hypercall.h     |    8 +++
 drivers/xen/Makefile                     |    1 
 include/xen/interface/tmem.h             |   43 +++++++++++++++++++++
 include/xen/interface/xen.h              |   22 ++++++++++
 4 files changed, 74 insertions(+)

Newly added xen-specific files:

 drivers/xen/tmem.c                       |  106 +++++++++++++++++++++
 include/xen/interface/tmem.h             |   43 ++++++++
 2 files changed, 149 insertions(+)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw

RFC PATCH 1/4 tmem: infrastructure for tmem layer

In reply to Dan Magenheimer
mm/Kconfig +27 -0
@@ -253,3 +253,30 @@
 	  of 1 says that all excess pages should be trimmed.
 
 	  See Documentation/nommu-mmap.txt for more information.
+
+#
+# support for transcendent memory
+#
+config TMEM
+	bool "Transcendent memory support"
+	depends on XEN # but in future may work without XEN
+	help
+	  In a virtualized environment, allows unused and underutilized
+	  system physical memory to be made accessible through a narrow
+	  well-defined page-copy-based API.  If unsure, say Y.
+
+config PRECACHE
+	bool "Cache clean pages in transcendent memory"
+	depends on TMEM
+	help
+	  Allows the transcendent memory pool to be used to store clean
+	  page-cache pages which, under some circumstances, will greatly
+	  reduce paging and thus improve performance.  If unsure, say Y.
+
+config PRESWAP
+	bool "Swap pages to transcendent memory"
+	depends on TMEM
+	help
+	  Allows the transcendent memory pool to be used as a pseudo-swap
+	  device which, under some circumstances, will greatly reduce
+	  swapping and thus improve performance.  If unsure, say Y.
mm/Makefile +2 -0
@@ -16,6 +16,8 @@
 obj-$(CONFIG_PROC_PAGE_MONITOR) += pagewalk.o
 obj-$(CONFIG_BOUNCE)	+= bounce.o
 obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o thrash.o
+obj-$(CONFIG_PRESWAP)	+= preswap.o
+obj-$(CONFIG_PRECACHE)	+= precache.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o
 obj-$(CONFIG_NUMA) 	+= mempolicy.o
include/linux/tmem.h +22 -0
@@ -0,0 +1,22 @@
+/*
+ * linux/tmem.h
+ *
+ * Interface to transcendent memory, used by mm/precache.c and mm/preswap.c
+ *
+ * Copyright (C) 2008,2009 Dan Magenheimer, Oracle Corp.
+ */
+
+struct tmem_ops {
+	int (*new_pool)(u64 uuid_lo, u64 uuid_hi, u32 flags);
+	int (*put_page)(u32 pool_id, u64 object, u32 index, unsigned long gmfn);
+	int (*get_page)(u32 pool_id, u64 object, u32 index, unsigned long gmfn);
+	int (*flush_page)(u32 pool_id, u64 object, u32 index);
+	int (*flush_object)(u32 pool_id, u64 object);
+	int (*destroy_pool)(u32 pool_id);
+};
+
+extern struct tmem_ops *tmem_ops;
+
+/* flags for tmem_ops.new_pool */
+#define TMEM_POOL_PERSIST          1
+#define TMEM_POOL_SHARED           2

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

Re: [RFC PATCH 1/4] tmem: infrastructure for tmem layer

mm/Makefile line 20

Quoted 8 lines · Dan Magenheimer
mm/Makefile +2 -0
@@ -16,6 +16,8 @@
 obj-$(CONFIG_PROC_PAGE_MONITOR) += pagewalk.o
 obj-$(CONFIG_BOUNCE)	+= bounce.o
 obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o thrash.o
+obj-$(CONFIG_PRESWAP)	+= preswap.o
+obj-$(CONFIG_PRECACHE)	+= precache.o

This patch does not actually add preswap.c or precache.c, so it would lead to an uncompilable changeset.

This in turn breaks git bisect.

Please make sure that every changeset that is applied results in a compilable and bootable kernel.

Signature · 7 lines
All rights reversed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw

RFC PATCH 2/4 tmem: precache implementation (layered on tmem)

In reply to Dan Magenheimer

--- linux-2.6.30/fs/super.c 2009-06-09 21:05:27.000000000 -0600
+++ linux-2.6.30-tmem/fs/super.c 2009-06-19 09:33:59.000000000 -0600
@@ -39,6 +39,7 @@
#include <linux/mutex.h>
#include <linux/file.h>
#include <linux/async.h>
+#include <linux/precache.h>
#include <asm/uaccess.h>
#include "internal.h"

@@ -110,6 +111,9 @@
s->s_qcop = sb_quotactl_ops;
s->s_op = &default_op;
s->s_time_gran = 1000000000;
+#ifdef CONFIG_PRECACHE
+ s->precache_poolid = -1;
+#endif
}
out:
return s;
@@ -200,6 +204,7 @@
vfs_dq_off(s, 0);
down_write(&s->s_umount);
fs->kill_sb(s);
+ precache_flush_filesystem(s);
put_filesystem(fs);
put_super(s);
}

fs/ext3/super.c +2 -0
@@ -37,6 +37,7 @@
 #include <linux/quotaops.h>
 #include <linux/seq_file.h>
 #include <linux/log2.h>
+#include <linux/precache.h>
 
 #include <asm/uaccess.h>
 
@@ -1306,6 +1307,7 @@
 	} else {
 		printk("internal journal\n");
 	}
+	precache_init(sb);
 	return res;
 }
 
fs/ocfs2/super.c +2 -0
@@ -42,6 +42,7 @@
 #include <linux/mount.h>
 #include <linux/seq_file.h>
 #include <linux/quotaops.h>
+#include <linux/precache.h>
 
 #define MLOG_MASK_PREFIX ML_SUPER
 #include <cluster/masklog.h>
@@ -2162,6 +2163,7 @@
 		mlog_errno(status);
 		goto bail;
 	}
+	shared_precache_init(sb, &di->id2.i_super.s_uuid[0]);
 
 bail:
 	mlog_exit(status);
include/linux/fs.h +7 -0
@@ -1377,6 +1377,13 @@
 	 * storage for asynchronous operations
 	 */
 	struct list_head s_async_list;
+
+#ifdef CONFIG_PRECACHE
+	/*
+	 * saved pool identifier for precache (-1 means none)
+	 */
+	u32 precache_poolid;
+#endif
 };
 
 extern struct timespec current_fs_time(struct super_block *sb);
fs/buffer.c +5 -0
@@ -41,6 +41,7 @@
 #include <linux/bitops.h>
 #include <linux/mpage.h>
 #include <linux/bit_spinlock.h>
+#include <linux/precache.h>
 
 static int fsync_buffers_list(spinlock_t *lock, struct list_head *list);
 
@@ -271,6 +272,10 @@
 
 	invalidate_bh_lrus();
 	invalidate_mapping_pages(mapping, 0, -1);
+	/* 99% of the time, we don't need to flush the precache on the bdev.
+	 * But, for the strange corners, lets be cautious
+	 */
+	precache_flush_inode(mapping);
 }
 
 /*
fs/mpage.c +8 -0
@@ -26,6 +26,7 @@
 #include <linux/writeback.h>
 #include <linux/backing-dev.h>
 #include <linux/pagevec.h>
+#include <linux/precache.h>
 
 /*
  * I/O completion handler for multipage BIOs.
@@ -285,6 +286,13 @@
 		SetPageMappedToDisk(page);
 	}
 
+	if (fully_mapped &&
+	    blocks_per_page == 1 && !PageUptodate(page) &&
+	    precache_get(page->mapping, page->index, page) == 1) {
+		SetPageUptodate(page);
+		goto confused;
+	}
+
 	/*
 	 * This page will go to BIO.  Do we need to send this BIO off first?
 	 */
mm/truncate.c +10 -0
@@ -18,6 +18,7 @@
 #include <linux/task_io_accounting_ops.h>
 #include <linux/buffer_head.h>	/* grr. try_to_release_page,
 				   do_invalidatepage */
+#include <linux/precache.h>
 #include "internal.h"
 
 
@@ -50,6 +51,7 @@
 static inline void truncate_partial_page(struct page *page, unsigned partial)
 {
 	zero_user_segment(page, partial, PAGE_CACHE_SIZE);
+	precache_flush(page->mapping, page->index);
 	if (page_has_private(page))
 		do_invalidatepage(page, partial);
 }
@@ -107,6 +109,10 @@
 	clear_page_mlock(page);
 	remove_from_page_cache(page);
 	ClearPageMappedToDisk(page);
+	/* this must be after the remove_from_page_cache which
+	 * calls precache_put
+	 */
+	precache_flush(mapping, page->index);
 	page_cache_release(page);	/* pagecache ref */
 }
 
@@ -168,6 +174,7 @@
 	pgoff_t next;
 	int i;
 
+	precache_flush_inode(mapping);
 	if (mapping->nrpages == 0)
 		return;
 
@@ -251,6 +258,7 @@
 		}
 		pagevec_release(&pvec);
 	}
+	precache_flush_inode(mapping);
 }
 EXPORT_SYMBOL(truncate_inode_pages_range);
 
@@ -398,6 +406,7 @@
 	int did_range_unmap = 0;
 	int wrapped = 0;
 
+	precache_flush_inode(mapping);
 	pagevec_init(&pvec, 0);
 	next = start;
 	while (next <= end && !wrapped &&
@@ -454,6 +463,7 @@
 		pagevec_release(&pvec);
 		cond_resched();
 	}
+	precache_flush_inode(mapping);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(invalidate_inode_pages2_range);
mm/filemap.c +11 -0
@@ -34,6 +34,7 @@
 #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */
 #include <linux/memcontrol.h>
 #include <linux/mm_inline.h> /* for page_is_file_cache() */
+#include <linux/precache.h>
 #include "internal.h"
 
 /*
@@ -116,6 +117,16 @@
 {
 	struct address_space *mapping = page->mapping;
 
+	/*
+	 * if we're uptodate, flush out into the precache, otherwise
+	 * invalidate any existing precache entries.  We can't leave
+	 * stale data around in the precache once our page is gone
+	 */
+	if (PageUptodate(page))
+		precache_put(page->mapping, page->index, page);
+	else
+		precache_flush(page->mapping, page->index);
+
 	radix_tree_delete(&mapping->page_tree, page->index);
 	page->mapping = NULL;
 	mapping->nrpages--;
include/linux/precache.h +55 -0

55 lines not on this page · raw message

mm/precache.c +146 -0

146 lines not on this page · raw message

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw

Re: [RFC PATCH 2/4] tmem: precache implementation (layered on tmem)

Review line 119

Quoted 10 lines · Dan Magenheimer
(quoted diff) +3 -0
@@ -110,6 +111,9 @@
 		s->s_qcop = sb_quotactl_ops;
 		s->s_op = &default_op;
 		s->s_time_gran = 1000000000;
+#ifdef CONFIG_PRECACHE
+		s->precache_poolid = -1;
+#endif
 	}
 out:
 	return s;

Please generate your patches with -up so we can see which functions are being modified by each patch hunk. That makes it a lot easier to find the context and see what you are trying to do.

Signature · 7 lines
All rights reversed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#6 lore raw

RFC PATCH 3/4 tmem: preswap implementation (layered on tmem)

In reply to Dan Magenheimer
mm/page_io.c +12 -0
@@ -102,6 +102,12 @@
 		unlock_page(page);
 		goto out;
 	}
+	if (preswap_put(page) == 1) {
+		set_page_writeback(page);
+		unlock_page(page);
+		end_page_writeback(page);
+		goto out;
+	}
 	bio = get_swap_bio(GFP_NOIO, page_private(page), page,
 				end_swap_bio_write);
 	if (bio == NULL) {
@@ -134,6 +140,12 @@
 		ret = -ENOMEM;
 		goto out;
 	}
+	if (preswap_get(page) == 1) {
+		SetPageUptodate(page);
+		unlock_page(page);
+		bio_put(bio);
+		goto out;
+	}
 	count_vm_event(PSWPIN);
 	submit_bio(READ, bio);
 out:
mm/swapfile.c +35 -6

129 lines not on this page · raw message

include/linux/swap.h +57 -0

77 lines not on this page · raw message

mm/preswap.c +274 -0

274 lines not on this page · raw message

include/linux/sysctl.h +1 -0
@@ -205,6 +205,7 @@
 	VM_PANIC_ON_OOM=33,	/* panic at out-of-memory */
 	VM_VDSO_ENABLED=34,	/* map VDSO into new processes? */
 	VM_MIN_SLAB=35,		 /* Percent pages ignored by zone reclaim */
+	VM_PRESWAP_PAGES=36,	/* pages/target_pages in preswap */
 };
 
 
kernel/sysctl.c +12 -0
@@ -1282,6 +1282,18 @@
 		.proc_handler	= &scan_unevictable_handler,
 	},
 #endif
+#ifdef CONFIG_PRESWAP
+	{
+		.ctl_name	= VM_PRESWAP_PAGES,
+		.procname	= "preswap",
+		.data		= NULL,
+		.maxlen		= sizeof(unsigned long),
+		.mode		= 0644,
+		.proc_handler	= &preswap_sysctl_handler,
+		.extra1		= (void *)&preswap_zero,
+		.extra2		= (void *)&preswap_infinity,
+	},
+#endif
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw

RFC PATCH 4/4 tmem: interface code for tmem on top of xen

In reply to Dan Magenheimer
arch/x86/include/asm/xen/hypercall.h +8 -0
@@ -45,6 +45,7 @@
 #include <xen/interface/xen.h>
 #include <xen/interface/sched.h>
 #include <xen/interface/physdev.h>
+#include <xen/interface/tmem.h>
 
 /*
  * The hypercall asms have to meet several constraints:
@@ -417,6 +418,13 @@
 	return _hypercall2(int, nmi_op, op, arg);
 }
 
+static inline int
+HYPERVISOR_tmem_op(
+	struct tmem_op *op)
+{
+	return _hypercall1(int, tmem_op, op);
+}
+
 static inline void
 MULTI_fpu_taskswitch(struct multicall_entry *mcl, int set)
 {
drivers/xen/Makefile +1 -0
@@ -3,5 +3,6 @@
 
 obj-$(CONFIG_HOTPLUG_CPU)	+= cpu_hotplug.o
 obj-$(CONFIG_XEN_XENCOMM)	+= xencomm.o
+obj-$(CONFIG_TMEM)		+= tmem.o
 obj-$(CONFIG_XEN_BALLOON)	+= balloon.o
 obj-$(CONFIG_XENFS)		+= xenfs/
\ No newline at end of file
include/xen/interface/tmem.h +43 -0

43 lines not on this page · raw message

include/xen/interface/xen.h +22 -0
@@ -58,6 +58,7 @@
 #define __HYPERVISOR_event_channel_op     32
 #define __HYPERVISOR_physdev_op           33
 #define __HYPERVISOR_hvm_op               34
+#define __HYPERVISOR_tmem_op              38
 
 /* Architecture-specific hypercall definitions. */
 #define __HYPERVISOR_arch_0               48
@@ -461,6 +462,27 @@
 #define __mk_unsigned_long(x) x ## UL
 #define mk_unsigned_long(x) __mk_unsigned_long(x)
 
+struct tmem_op {
+	uint32_t cmd;
+	int32_t pool_id; /* private > 0; shared < 0; 0 is invalid */
+	union {
+		struct {  /* for cmd == TMEM_NEW_POOL */
+			uint64_t uuid[2];
+			uint32_t flags;
+		} new;
+		struct {
+			uint64_t object;
+			uint32_t index;
+			uint32_t tmem_offset;
+			uint32_t pfn_offset;
+			uint32_t len;
+			GUEST_HANDLE(void) gmfn; /* guest machine page frame */
+		} gen;
+	} u;
+};
+typedef struct tmem_op tmem_op_t;
+DEFINE_GUEST_HANDLE_STRUCT(tmem_op_t);
+
 #else /* __ASSEMBLY__ */
 
 /* In assembly code we cannot use C numeric constant suffixes. */
drivers/xen/tmem.c +106 -0

106 lines not on this page · raw message

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
In reply to Dan Magenheimer
Quoted 8 lines · On Fri, 19 Jun 2009 16:53:45 -0700 (PDT) Dan Magenheimer…

Tmem has some similarity to IBM's Collaborative Memory Management, but creates more of a partnership between the kernel and the "privileged entity" and is not very invasive. Tmem may be applicable for KVM and containers; there is some disagreement on the extent of its value. Tmem is highly complementary to ballooning (aka page granularity hot plug) and memory deduplication (aka transparent content-based page sharing) but still has value when neither are present.

The basic idea seems to be that you reduce the amount of memory available to the guest and as a compensation give the guest some tmem, no? If that is the case then the effect of tmem is somewhat comparable to the volatile page cache pages.

The big advantage of this approach is its simplicity, but there are down sides as well:
1) You need to copy the data between the tmem pool and the page cache. At least temporarily there are two copies of the same page around. That increases the total amount of used memory.
2) The guest has a smaller memory size. Either the memory is large enough for the working set size in which case tmem is ineffective, or the working set does not fit which increases the memory pressure and the cpu cycles spent in the mm code.
3) There is an additional turning knob, the size of the tmem pool for the guest. I see the need for a clever algorithm to determine the size for the different tmem pools.

Overall I would say its worthwhile to investigate the performance impacts of the approach.

Signature · 10 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#9 lore raw
Quoted 8 lines
Quoted 8 lines

Tmem has some similarity to IBM's Collaborative Memory Management, but creates more of a partnership between the kernel and the "privileged entity" and is not very invasive. Tmem may be applicable for KVM and containers; there is some disagreement on the extent of its value. Tmem is highly complementary to ballooning (aka page granularity hot plug) and memory deduplication (aka transparent content-based page sharing) but still has value when neither are present.

Hi Martin --

Thanks much for taking the time to reply!

Quoted 3 lines

The basic idea seems to be that you reduce the amount of memory available to the guest and as a compensation give the guest some tmem, no?

That's mostly right. Tmem's primary role is to help with guests that have had their available memory reduced (via ballooning or hotplug or some future mechanism). However tmem additionally provides a way of providing otherwise unused-by-the-hypervisor ("fallow") memory to a guest, essentially expanding a guest kernel's page cache if no other guest is using the RAM anyway.

And "as a compensation GIVE the guest some tmem" is misleading, because tmem (at least ephemeral tmem) is never "given" to a guest. A better word might be "loaned" or "rented". The guest gets to use some tmem for awhile but if it doesn't use it effectively, the memory is "repossessed" (or the guest is "evicted" from using that memory) transparently so that it can be used more effectively elsewhere.

Quoted 2 lines

If that is the case then the effect of tmem is somewhat comparable to the volatile page cache pages.

There is definitely some similarity in that both are providing useful information to the hypervisor. In CMM's case, the guest is passively providing info; in tmem's case it is actively providing info and making use of the info within the kernel, not just in the hypervsior, which is why I described it as "more of a partnership".

Quoted 5 lines

The big advantage of this approach is its simplicity, but there are down sides as well:
1) You need to copy the data between the tmem pool and the page cache. At least temporarily there are two copies of the same page around. That increases the total amount of used memory.

Certainly this is theoretically true, but I think the increase is small and transient. The kernel only puts the page into precache when it has decided to use that page for another purpose (due to memory pressure). Until it actually "reprovisions" the page, the data is briefly duplicated.

On the other hand, copying eliminates the need for fancy games with virtual mappings and TLB entries. Copying appears to be getting much faster on recent CPUs; I'm not sure if this is also true of TLB operations.

Quoted 3 lines

2) The guest has a smaller memory size. Either the memory is large enough for the working set size in which case tmem is ineffective...

Yes, if the kernel has memory to "waste" (e.g. never refaults and never swaps), tmem is ineffective. The goal of tmem is to optimize memory usage across an environment where there is contention among multiple users (guests) for a limited resource (RAM). If your environment always has enough RAM for every guest and there's never any contention, you don't want tmem... but I'd assert you've wasted money in your data center by buying too much RAM!

Quoted 2 lines

or the working set does not fit which increases the memory pressure and the cpu cycles spent in the mm code.

True, this is where preswap is useful. Without tmem/preswap, "does not fit" means swap-to-disk or refaulting is required. Preswap alleviates the memory pressure by using tmem to essentially swap to "magic memory" and precache reduces the need for refaulting.

Quoted 3 lines

3) There is an additional turning knob, the size of the tmem pool for the guest. I see the need for a clever algorithm to determine the size for the different tmem pools.

Yes, some policy in the hypervisor is still required, essentially a "memory scheduler". The working implementation (in Xen) uses FIFO, but modified by admin-configurable "weight" values to allow QoS and avoid DoS.

Quoted 2 lines

Overall I would say its worthwhile to investigate the performance impacts of the approach.

Thanks. I'd appreciate any thoughts or experience you have in this area (onlist or offlist) as I don't think there are any adequate benchmarks that aren't either myopic for a complex environment or contrived (and thus misleading) to prove an isolated point.

I would also guess that tmem is more beneficial on recent multi-core processors, and more costly on older chips.

Thanks again,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw
In reply to Dan Magenheimer
Quoted 8 lines · Dan Magenheimer

What if there was a class of memory that is of unknown and dynamically variable size, is addressable only indirectly by the kernel, can be configured either as persistent or as "ephemeral" (meaning it will be around for awhile, but might disappear without warning), and is still fast enough to be synchronously accessible?

We call this latter class "transcendent memory"

While true that this memory is "exceeding usual limits", the more important criteria is that it may disappear.

It might be clearer to just call it "ephemeral memory".

There is going to be some overhead due to the extra copying, and at times there could be two copies of data in memory. It seems possible that certain apps right a the borderline could end up running slower because they can't fit in the regular+ephemeral memory due to the duplication, while the same amount of memory used normally could have been sufficient.

I suspect trying to optimize management of this could be difficult.

Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw
Quoted 8 lines
Quoted 8 lines

What if there was a class of memory that is of unknown and dynamically variable size, is addressable only indirectly by the kernel, can be configured either as persistent or as "ephemeral" (meaning it will be around for awhile, but might disappear without warning), and is still fast enough to be synchronously accessible?

Quoted 6 lines
Quoted 1 line

We call this latter class "transcendent memory"

While true that this memory is "exceeding usual limits", the more important criteria is that it may disappear.

It might be clearer to just call it "ephemeral memory".

Ephemeral tmem (precache) may be the most interesting, but there is persistent tmem (preswap) as well. Both are working today and both are included in the patches I posted.

Looking for a term encompassing both, I chose "transcendent".

Quoted 6 lines

There is going to be some overhead due to the extra copying, and at times there could be two copies of data in memory. It seems possible that certain apps right a the borderline could end up running slower because they can't fit in the regular+ephemeral memory due to the duplication, while the same amount of memory used normally could have been sufficient.

This is likely true, but I expect the duplicates to be few and transient and a very small fraction of the total memory cost for virtualization (and similar abstraction technologies).

Quoted 1 line

I suspect trying to optimize management of this could be difficult.

True. Optimizing the management of ANY resource across many consumers is difficult. But wasting the resource because its a pain to optimize doesn't seem to be a good answer either.

Thanks!
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#12 lore raw
In reply to Dan Magenheimer

Hi!

This description (whole mail) needs to go into Documentation/, somewhere.

Quoted 3 lines

Normal memory is directly addressable by the kernel, of a known normally-fixed size, synchronously accessible, and persistent (though not across a reboot).

...

Quoted 4 lines

Transcendent memory, or "tmem" for short, provides a well-defined API to access this unusual class of memory. The basic operations are page-copy-based and use a flexible object-oriented addressing mechanism. Tmem assumes

Should this API be documented, somewhere? Is it in-kernel API or does userland see it?

Quoted 10 lines

"Preswap" IS persistent, but for various reasons may not always be available for use, again due to factors that may not be visible to the kernel (but, briefly, if the kernel is being "good" and has shared its resources nicely, then it will be able to use preswap, else it will not). Once a page is put, a get on the page will always succeed. So when the kernel finds itself in a situation where it needs to swap out a page, it first attempts to use preswap. If the put works, a disk write and (usually) a disk read are avoided. If it doesn't, the page is written to swap as usual. Unlike precache, whether

Ok, how much slower this gets in the worst case? Single hypercall to find out that preswap is unavailable? I guess that compared to disk access that's lost in the noise?
Pavel

Signature · 8 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#13 lore raw

Hi Pavel --

Thanks for the feedback!

Quoted 2 lines

This description (whole mail) needs to go into
Documentation/, somewhere.

Good idea. I'll do that for the next time I post the patches.

Quoted 11 lines
Quoted 3 lines

Normal memory is directly addressable by the kernel, of a known normally-fixed size, synchronously accessible, and persistent (though not across a reboot).

...

Quoted 4 lines

Transcendent memory, or "tmem" for short, provides a well-defined API to access this unusual class of memory. The basic operations are page-copy-based and use a flexible object-oriented addressing mechanism. Tmem assumes

Should this API be documented, somewhere? Is it in-kernel API or does userland see it?

It is documented currently at:

http://oss.oracle.com/projects/tmem/documentation/api/

(just noticed I still haven't posted version 0.0.2 which has a few minor changes).

I will add a briefer description of this API in Documentation/

It is in-kernel only because some of the operations have a parameter that is a physical page frame number.

Quoted 11 lines
Quoted 10 lines

"Preswap" IS persistent, but for various reasons may not always be available for use, again due to factors that may not be visible to the kernel (but, briefly, if the kernel is being "good" and has shared its resources nicely, then it will be able to use preswap, else it will not). Once a page is put, a get on the page will always succeed. So when the kernel finds itself in a situation where it needs to swap out a page, it first attempts to use preswap. If the put works, a disk write and (usually) a disk read are avoided. If it doesn't, the page is written to swap as usual. Unlike precache, whether

Quoted 3 lines

Ok, how much slower this gets in the worst case? Single hypercall to find out that preswap is unavailable? I guess that compared to disk access that's lost in the noise?

Yes, the overhead of one hypercall per swap page is lost in the noise.

Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw
Quoted 9 lines

It is documented currently at:

http://oss.oracle.com/projects/tmem/documentation/api/

(just noticed I still haven't posted version 0.0.2 which has a few minor changes).

I will add a briefer description of this API in Documentation/

Please do.

At least TMEM_NEW_POOL() looks quite ugly. Why uuid? Mixing flags into size argument is strange.

Quoted 2 lines

It is in-kernel only because some of the operations have a parameter that is a physical page frame number.

In-kernel API is probably better described as function prototypes.
Pavel

Signature · 8 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#15 lore raw
Quoted 10 lines
Quoted 8 lines

It is documented currently at:

http://oss.oracle.com/projects/tmem/documentation/api/

(just noticed I still haven't posted version 0.0.2 which has a few minor changes).

I will add a briefer description of this API in Documentation/

Please do.

OK, will do.

Quoted 2 lines

At least TMEM_NEW_POOL() looks quite ugly. Why uuid? Mixing flags into size argument is strange.

The uuid is only used for shared pools. If two different "tmem clients" (guests) agree on a 128-bit "shared secret", they can share a tmem pool. For ocfs2, the 128-bit uuid in the on-disk superblock is used for this purpose to implement shared precache. (Pages evicted by one cluster node can be used by another cluster node that co-resides on the same physical system.)

The (page)size argument is always fixed (at PAGE_SIZE) for any given kernel. The underlying implementation can be capable of supporting multiple pagesizes.

So for the basic precache and preswap uses, "new pool" has a very simple interface.

Quoted 4 lines
Quoted 2 lines

It is in-kernel only because some of the operations have a parameter that is a physical page frame number.

In-kernel API is probably better described as function prototypes.

Good idea. I will do that.

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#16 lore raw
Quoted 8 lines · Dan Magenheimer

The uuid is only used for shared pools. If two different "tmem clients" (guests) agree on a 128-bit "shared secret", they can share a tmem pool. For ocfs2, the 128-bit uuid in the on-disk superblock is used for this purpose to implement shared precache. (Pages evicted by one cluster node can be used by another cluster node that co-resides on the same physical system.)

What are the implications of some third party VM guessing the "uuid" of a shared pool? Presumably they could view and modify the contents of the pool. Is there any security model beyond making UUIDs unguessable?

Quoted 4 lines

The (page)size argument is always fixed (at PAGE_SIZE) for any given kernel. The underlying implementation can be capable of supporting multiple pagesizes.

Pavel's other point was that merging the size field into the flags is a bit unusual/ugly. But you can workaround that by just defining the "flag" values for each plausible page size, since there's a pretty small
bound: TMEM_PAGESZ_4K, 8K, etc.

Also, having an "API version number" is a very bad idea. Such version numbers are very inflexible and basically don't work (esp if you're expecting to have multiple independent implementations of this API). Much better is to have feature flags; the caller asks for features on the new pool, and pool creation either succeeds or doesn't (a call to return the set of supported features is a good compliment).

J

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#17 lore raw
Quoted 11 lines

From: Jeremy Fitzhardinge [mailto:[email protected]]

Quoted 7 lines · Dan Magenheimer

The uuid is only used for shared pools. If two different "tmem clients" (guests) agree on a 128-bit "shared secret", they can share a tmem pool. For ocfs2, the 128-bit uuid in the on-disk superblock is used for this purpose to implement shared precache. (Pages evicted by one cluster node can be used by another cluster node that co-resides on the same physical system.)

Quoted 5 lines

What are the implications of some third party VM guessing the "uuid" of
a shared pool? Presumably they could view and modify the contents of the pool. Is there any security model beyond making UUIDs unguessable?

Interesting question. But, more than the 128-bit UUID must be guessed... a valid 64-bit object id and a valid 32-bit page index must also be guessed (though most instances of the page index are small numbers so easy to guess). Once 192 bits are guessed though, yes, the pages could be viewed and modified. I suspect there are much more easily targeted security holes in most data centers than guessing 192 (or even 128) bits.

Now this only affects shared pools, and shared-precache is still experimental and not really part of this patchset. Does "mount" of an accessible disk/filesystem have a better security model? Perhaps there are opportunities to leverage that?

Quoted 10 lines
Quoted 3 lines

The (page)size argument is always fixed (at PAGE_SIZE) for any given kernel. The underlying implementation can be capable of supporting multiple pagesizes.

Pavel's other point was that merging the size field into the flags is a
bit unusual/ugly. But you can workaround that by just defining the "flag" values for each plausible page size, since there's a pretty small
bound: TMEM_PAGESZ_4K, 8K, etc.

OK I see. Yes the point (and the workaround) are valid.

Quoted 6 lines

Also, having an "API version number" is a very bad idea. Such version numbers are very inflexible and basically don't work (esp if you're expecting to have multiple independent implementations of this API). Much better is to have feature flags; the caller asks for features on the new pool, and pool creation either succeeds or doesn't (a call to return the set of supported features is a good compliment).

Yes. Perhaps all the non-flag bits should just be reserved for future use. Today, the implementation just checks for (and implements) only zero anyway and nothing is defined anywhere except the 4K pagesize at the lowest levels of the (currently xen-only) API.

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#18 lore raw
Quoted 9 lines · Dan Magenheimer

Interesting question. But, more than the 128-bit UUID must be guessed... a valid 64-bit object id and a valid 32-bit page index must also be guessed (though most instances of the page index are small numbers so easy to guess). Once 192 bits are guessed though, yes, the pages could be viewed and modified. I suspect there are much more easily targeted security holes in most data centers than guessing 192 (or even 128) bits.

If its possible to verify the uuid is valid before trying to find a valid oid+page, then its much easier (since you can concentrate on the uuid first). If the uuid is derived from something like the filesystem's uuid - which wouldn't normally be considered sensitive information - then its not like its a search of the full 128-bit space. And even if it were secret, uuids are not generally 128 randomly chosen bits.

You also have to consider the case of a domain which was once part of the ocfs cluster, but now is not - it may still know the uuid, but not be otherwise allowed to use the cluster.

Quoted 5 lines

Now this only affects shared pools, and shared-precache is still experimental and not really part of this patchset. Does "mount" of an accessible disk/filesystem have a better security model? Perhaps there are opportunities to leverage that?

Well, a domain is allowed to access any block device you give it access to. I'm not sure what the equivalent model for tmem would be.

Anyway, it sounds like you need to think a fair bit more about shared tmem's security model before it can be considered for use.

Quoted 5 lines

Yes. Perhaps all the non-flag bits should just be reserved for future use. Today, the implementation just checks for (and implements) only zero anyway and nothing is defined anywhere except the 4K pagesize at the lowest levels of the (currently xen-only) API.

Yes. It should fail if it sees any unknown flags set in a guest request.

J

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#19 lore raw
Quoted 11 lines

From: Jeremy Fitzhardinge [mailto:[email protected]]

Quoted 8 lines · Dan Magenheimer

Interesting question. But, more than the 128-bit UUID must be guessed... a valid 64-bit object id and a valid 32-bit page index must also be guessed (though most instances of the page index are small numbers so easy to guess). Once 192 bits are guessed though, yes, the pages could be viewed and modified. I suspect there are much more easily targeted security holes in most data centers than guessing 192 (or even 128) bits.

Quoted 3 lines

If its possible to verify the uuid is valid before trying to find a valid oid+page, then its much easier (since you can concentrate on the uuid first).

No, the uuid can't be verified. Tmem gives no indication as to whether a newly-created pool is already in use (shared) by another guest. So without both the 128-bit uuid and an already-in-use 64-bit object id and 32-bit page index, no data is readable or writable by the attacker.

Quoted 9 lines

You also have to consider the case of a domain which was once part of the ocfs cluster, but now is not - it may still know the uuid, but not be otherwise allowed to use the cluster.
If the uuid is derived from something like the
filesystem's uuid - which wouldn't normally be considered sensitive information - then its not like its a search of the full 128-bit space.
And even if it were secret, uuids are not generally 128 randomly chosen bits.

Hmmm... that is definitely a thornier problem. I guess the security angle definitely deserves more design. But, again, this affects only shared precache which is not intended to part of the proposed initial tmem patchset, so this is a futures issue.)

Thanks again for the feedback!
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#20 lore raw
Quoted 6 lines · Dan Magenheimer

No, the uuid can't be verified. Tmem gives no indication as to whether a newly-created pool is already in use (shared) by another guest. So without both the 128-bit uuid and an already-in-use 64-bit object id and 32-bit page index, no data is readable or writable by the attacker.

You have to consider things like timing attacks as well (for example, a tmem hypercall might return faster if the uuid already exists).

Besides, you can tell whether a uuid exists, by at least a couple of mechanisms (from a quick read of the source, so I might have overlooked something):

1. You can create new shared pools until it starts failing as a result of hitting the MAX_GLOBAL_SHARED_POOLS limit with junk uuids. If you then successfully "create" a shared pool while searching, you know it already existed.
2. The returned pool id will increase unless the pool already exists, in which case you'll get a smaller id back (ignoring wraparound).

Quoted 5 lines

Hmmm... that is definitely a thornier problem. I guess the security angle definitely deserves more design. But, again, this affects only shared precache which is not intended to part of the proposed initial tmem patchset, so this is a futures issue.)

Yeah, a shared namespace of accessible objects is an entirely new thing in the Xen universe. I would also drop Xen support until there's a good security story about how they can be used.

J

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#21 lore raw
Quoted 8 lines

From: Jeremy Fitzhardinge [mailto:[email protected]]

Quoted 5 lines · Dan Magenheimer

No, the uuid can't be verified. Tmem gives no indication as to whether a newly-created pool is already in use (shared) by another guest. So without both the 128-bit uuid and an already-in-use 64-bit object id and 32-bit page index, no data is readable or writable by the attacker.

Quoted 7 lines

You have to consider things like timing attacks as well (for example, a
tmem hypercall might return faster if the uuid already exists).

Besides, you can tell whether a uuid exists, by at least a couple of mechanisms (from a quick read of the source, so I might have overlooked something):

All of these still require a large number of guesses across a 128-bit space of possible uuids, right?
It should be easy to implement "guess limits" in xen that disable tmem use by a guest if it fails too many guesses. I'm a bit more worried about:

Quoted 3 lines

You also have to consider the case of a domain which was once part of the ocfs cluster, but now is not - it may still know the uuid, but not be otherwise allowed to use the cluster.

But on the other hand, the security model here can be that if a trusted entity becomes untrusted, you have to change the locks.

Quoted 5 lines

Yeah, a shared namespace of accessible objects is an entirely new thing
in the Xen universe. I would also drop Xen support until there's a good
security story about how they can be used.

While I agree that the security is not bulletproof, I wonder if this position might be a bit extreme. Certainly, the NSA should not turn on tmem in a cluster, but that doesn't mean that nobody should be allowed to. I really suspect that there are less costly / more rewarding attack vectors at several layers in the hardware/software stack of most clusters.

Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#22 lore raw
Quoted 5 lines · Dan Magenheimer

All of these still require a large number of guesses across a 128-bit space of possible uuids, right?
It should be easy to implement "guess limits" in xen that disable tmem use by a guest if it fails too many guesses.

How does Xen distinguish between someone "guessing" uuids and a normal user which wants to create lots of pools?

Quoted 9 lines
Quoted 4 lines

You also have to consider the case of a domain which was once part of the ocfs cluster, but now is not - it may still know the uuid, but not be otherwise allowed to use the cluster.

But on the other hand, the security model here can be that if a trusted entity becomes untrusted, you have to change the locks.

Revocation is one of the big problems with capabilities-based systems.

Quoted 7 lines
Quoted 6 lines

Yeah, a shared namespace of accessible objects is an entirely new thing
in the Xen universe. I would also drop Xen support until there's a good
security story about how they can be used.

Quoted 7 lines

While I agree that the security is not bulletproof, I wonder if this position might be a bit extreme. Certainly, the NSA should not turn on tmem in a cluster, but that doesn't mean that nobody should be allowed to. I really suspect that there are less costly / more rewarding attack vectors at several layers in the hardware/software stack of most clusters.

Well, I think you can define any security model you like, but I think you need to have a defined security model before making it an available API. At the moment the model is defined by whatever you currently have implemented, and anyone using the API as-is - without special consideration of its security properties - is going to end up vulnerable.

In an earlier mail I said "a shared namespace of accessible objects is an entirely new thing in the Xen universe", which is obviously not true:
we have Xenbus.

It seems to me that a better approach to shared tmem pools should be moderated via Xenbus, which in turn allows dom0/xenstored/tmemd/etc to apply arbitrary policies to who gets to see what handles, revoke them, etc.

You don't need to deal with "uuids" at the tmem hypercall level. Instead, you have a well-defined xenbus path corresponding to the resource; reading it will return a handle number, which you can then use with your hypercalls. If your access is denied or revoked, then the read will fail (or the current handle will stop working if revoked). This requires some privileged hypercalls to establish and remove tmem handles for a particular domain.

I'm assuming that the job of managing and balancing tmem resources will need to happen in a tmem-domain rather than trying to build all that policy into Xen itself, so putting a bit more logic in there to manage shared access rules doesn't add much complexity to the system.

J

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#23 lore raw
In reply to Dan Magenheimer
Quoted 11 lines
Quoted 6 lines

Yeah, a shared namespace of accessible objects is an entirely new thing
in the Xen universe. I would also drop Xen support until there's a good
security story about how they can be used.

While I agree that the security is not bulletproof, I wonder if this position might be a bit extreme. Certainly, the NSA should not turn on tmem in a cluster, but that doesn't mean that nobody should be allowed to. I really suspect that there are

This has more problems than "just" security, and yes, security should
be really solved at design time...
Pavel

Signature · 8 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#24 lore raw

OK, OK, I give up. I will ensure all code for shared pools is removed from the next version of the patch.

Though for future reference, I am interested in what problems it has other than "just" security (offlist if you want).

Quoted 14 lines

-----Original Message-----
From: Pavel Machek [mailto:[email protected]]

Quoted 10 lines
Quoted 5 lines

Yeah, a shared namespace of accessible objects is an entirely new thing
in the Xen universe. I would also drop Xen support until there's a good
security story about how they can be used.

While I agree that the security is not bulletproof, I wonder if this position might be a bit extreme. Certainly, the NSA should not turn on tmem in a cluster, but that doesn't mean that nobody should be allowed to. I really suspect that there are

Quoted 4 lines

This has more problems than "just" security, and yes, security should be really solved at design time...

Pavel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#25 lore raw
In reply to Dan Magenheimer

2009/6/20 Dan Magenheimer <[email protected]>:

Quoted 8 lines

We call this latter class "transcendent memory" and it provides an interesting opportunity to more efficiently utilize RAM in a virtualized environment.  However this "memory but not really memory" may also have applications in NON-virtualized environments, such as hotplug-memory deletion, SSDs, and page cache compression.  Others have suggested ideas such as allowing use of highmem memory without a highmem kernel, or use of spare video memory.

Here is what I consider may be a use case from the embedded
world: we have to save power as much as possible, so we need to shut off entire banks of memory.

Currently people do things like put memory into self-refresh and then sleep, but for long lapses of time you would want to compress memory towards lower addresses and turn as many banks as possible off.

So we have something like 4x16MB banks of RAM = 64MB RAM, and the most necessary stuff easily fits in one of them. If we can shut down 3x16MB we save 3 x power supply of the RAMs.

However in embedded we don't have any swap, so we'd need some call that would attempt to remove a memory by paging out code and data that has been demand-paged in from the FS but no dirty pages, these should instead be moved down to memory which will be retained, and the call should fail if we didn't succeed to migrate all dirty pages.

Would this be possible with transcendent memory?

Yours,
Linus Walleij

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#26 lore raw
Quoted 35 lines · Linus Walleij

2009/6/20 Dan Magenheimer<[email protected]>:

Quoted 9 lines

We call this latter class "transcendent memory" and it provides an interesting opportunity to more efficiently utilize RAM in a virtualized environment. However this "memory but not really memory" may also have applications in NON-virtualized environments, such as hotplug-memory deletion, SSDs, and page cache compression. Others have suggested ideas such as allowing use of highmem memory without a highmem kernel, or use of spare video memory.

Here is what I consider may be a use case from the embedded
world: we have to save power as much as possible, so we need to shut off entire banks of memory.

Currently people do things like put memory into self-refresh and then sleep, but for long lapses of time you would want to compress memory towards lower addresses and turn as many banks as possible off.

So we have something like 4x16MB banks of RAM = 64MB RAM, and the most necessary stuff easily fits in one of them. If we can shut down 3x16MB we save 3 x power supply of the RAMs.

However in embedded we don't have any swap, so we'd need some call that would attempt to remove a memory by paging out code and data that has been demand-paged in from the FS but no dirty pages, these should instead be moved down to memory which will be retained, and the call should fail if we didn't succeed to migrate all dirty pages.

Quoted 2 lines

Would this be possible with transcendent memory?

You could do this with memory defragmentation, which is needed for things like memory hotunplug ayway.

Signature · 7 lines
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#27 lore raw
In reply to Linus Walleij
Quoted 37 lines

From: Linus Walleij [mailto:[email protected]]
Sent: Saturday, June 27, 2009 7:19 AM
Subject: Re: [RFC] transcendent memory for Linux

Quoted 8 lines

We call this latter class "transcendent memory" and it provides an interesting opportunity to more efficiently utilize RAM in a virtualized environment.  However this "memory but not really memory" may also have applications in NON-virtualized environments, such as hotplug-memory deletion, SSDs, and page cache compression.  Others have suggested ideas such as allowing use of highmem memory without a highmem kernel, or use of spare video memory.

Here is what I consider may be a use case from the embedded
world: we have to save power as much as possible, so we need to shut off entire banks of memory.

Currently people do things like put memory into self-refresh and then sleep, but for long lapses of time you would want to compress memory towards lower addresses and turn as many banks as possible off.

So we have something like 4x16MB banks of RAM = 64MB RAM, and the most necessary stuff easily fits in one of them. If we can shut down 3x16MB we save 3 x power supply of the RAMs.

However in embedded we don't have any swap, so we'd need some call that would attempt to remove a memory by paging out code and data that has been demand-paged in from the FS but no dirty pages, these should instead be moved down to memory which will be retained, and the call should fail if we didn't succeed to migrate all dirty pages.

Quoted 1 line

Would this be possible with transcendent memory?

Yes, I think this would work nicely as a use case for tmem.

As Avi points out, you could do this with memory defragmentation, but if you know in advance that you will be frequently powering on and off a bank of RAM, you could put only ephemeral memory into it (enforced by a kernel policy and the tmem API), then defragmentation (and compression towards lower addresses) would not be necessary, and you could power off a bank with no loss of data.

One issue though: I would guess that copying pages of memory could be very slow in an inexpensive embedded processor.

Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#28 lore raw
Quoted 3 lines

One issue though: I would guess that copying pages of memory could be very slow in an inexpensive embedded processor.

And copying memory could very easily burn enough power by keeping the CPU busy that you lose the incremental gain of turning the memory off vs. just going to self refresh. (And the copying latency would easily be as bad as the transition latency to/from self-refresh).

- R.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>