compcache: compressed caching v2
Subsystems (3)
cover compcache: compressed caching v2
Hi All,
(This revision contains all changes suggested in initial review).
This implements a RAM based block device which acts as swap disk. Pages swapped to this disk are compressed and stored in memory itself. This allows more applications to fit in given amount of memory. This is especially useful for embedded devices, OLPC and small desktops (aka virtual machines).
Project home: http://code.google.com/p/compcache/
It consists of following components:
- compcache.ko: Creates RAM based block device
- tlsf.ko: Two Level Segregate Fit (TLSF) allocator
- LZO de/compressor: (Already in mainline)
Project home contains some performance numbers for TLSF and LZO. For general desktop use, this is giving *significant* performance gain under memory pressure. For now, it has been tested on x86 and x86_64.
Thanks,
Nitin
PATCH 1/3 compcache: RAM based block device
This creates RAM based block device (called ramzswap0) which is used as swap disk.
On write (swap-out):
- compress page (using LZO)
- Allocate required amount of memory (using TLSF)
- Store reference to its location in simple array.
On read (swap-in):
- Get compressed page location from array
- Decompress this page.
--- drivers/block/Kconfig | 27 +++ drivers/block/Makefile | 1 + drivers/block/compcache.c | 432 +++++++++++++++++++++++++++++++++++++++++++++ drivers/block/compcache.h | 95 ++++++++++ 4 files changed, 555 insertions(+), 0 deletions(-)
drivers/block/Kconfig +27 -0
@@ -347,6 +347,33 @@ config BLK_DEV_RAM_SIZE The default value is 4096 kilobytes. Only change this if you know what you are doing. +config BLK_DEV_COMPCACHE + tristate "Compressed RAM based swap device" + select TLSF + select LZO_COMPRESS + select LZO_DECOMPRESS + help + This creates RAM based block device which acts as swap disk. Pages + swapped to this disk are compressed and stored in memory itself. + Project Home: http://code.google.com/p/compcache/ + +config BLK_DEV_COMPCACHE_DEBUG + default y + depends on BLK_DEV_COMPCACHE + bool "Enable debugging" + help + This causes negligible performance loss and size increase. + If unsure, say Y. + +config BLK_DEV_COMPCACHE_STATS + default y + depends on BLK_DEV_COMPCACHE + bool "Enable statistics" + help + Creates /proc/compcache to export various statistics. + This adds about 4K to size with negligible performance loss. + If unsure, say Y. + config BLK_DEV_XIP bool "Support XIP filesystems on RAM block device" depends on BLK_DEV_RAM
drivers/block/Makefile +1 -0
@@ -12,6 +12,7 @@ obj-$(CONFIG_PS3_DISK) += ps3disk.o obj-$(CONFIG_ATARI_FLOPPY) += ataflop.o obj-$(CONFIG_AMIGA_Z2RAM) += z2ram.o obj-$(CONFIG_BLK_DEV_RAM) += brd.o +obj-$(CONFIG_BLK_DEV_COMPCACHE) += compcache.o obj-$(CONFIG_BLK_DEV_LOOP) += loop.o obj-$(CONFIG_BLK_DEV_XD) += xd.o obj-$(CONFIG_BLK_CPQ_DA) += cpqarray.o
drivers/block/compcache.c new file +432 -0
432 lines not on this page · raw message
drivers/block/compcache.h new file +95 -0
95 lines not on this page · raw message
PATCH 2/3 compcache: TLSF Allocator
In reply to Nitin GuptaTwo Level Segregate Fit (TLSF) Allocator is used to allocate memory for variable size compressed pages. Its fast and gives low fragmentation. Following links give details on this allocator:
- http://rtportal.upv.es/rtmalloc/files/tlsf_paper_spe_2007.pdf
In tests, it shows ~43% less memory usage than SLUB!
See: http://code.google.com/p/compcache/wiki/AllocatorsComparison
This kernel port of TLSF (v2.3.2) introduces several changes but underlying algorithm remains the same.
Changelog TLSF v2.3.2 vs this kernel port
- Pool now dynamically expands/shrinks.
It is collection of contiguous memory regions.
- Changes to pool create interface as a result of above change.
- It is not scalable (lack of per-cpu data structures etc.)
- Collect and export stats (/proc/tlsfinfo)
- Cleanups: kernel coding style, added comments, macros -> static inline, etc.
--- include/linux/tlsf.h | 93 +++++++ init/Kconfig | 28 ++ mm/Makefile | 1 + mm/tlsf.c | 676 ++++++++++++++++++++++++++++++++++++++++++++++++++ mm/tlsf_int.h | 145 +++++++++++ 5 files changed, 943 insertions(+), 0 deletions(-)
include/linux/tlsf.h new file +93 -0
@@ -0,0 +1,93 @@ +/* +* Two Levels Segregate Fit memory allocator (TLSF) +* Version 2.3.2 +* +* Written by Miguel Masmano Tello <[email protected]> +* +* Thanks to Ismael Ripoll for his suggestions and reviews +* +* Copyright (C) 2007, 2006, 2005, 2004 +* +* This code is released using a dual license strategy: GPL/LGPL +* You can choose the licence that better fits your requirements. +* +* Released under the terms of the GNU General Public License Version 2.0 +* Released under the terms of the GNU Lesser General Public License Version 2.1 +* +* This is kernel port of TLSF allocator. +* Original code can be found at: http://rtportal.upv.es/rtmalloc/ +* - Nitin Gupta ([email protected]) +*/ + +#ifndef _TLSF_H_ +#define _TLSF_H_ + +typedef void * (get_memory )(size_t bytes); +typedef void (put_memory )(void *ptr); + +/** +* tlsf_create_memory_pool - create dynamic memory pool +* @name: name of the pool +* @get_mem: callback function used to expand pool +* @put_mem: callback function used to shrink pool +* @init_size: inital pool size (in bytes) +* @max_size: maximum pool size (in bytes) - set this as 0 for no limit +* @grow_size: amount of memory (in bytes) added to pool whenever required +* +* All size values are rounded up to next page boundary. +*/ +extern void *tlsf_create_memory_pool (const char *name, +get_memory get_mem, +put_memory put_mem, +size_t init_size, +size_t max_size, +size_t grow_size); +/** +* tlsf_destory_memory_pool - cleanup given pool +* @mem_pool: Pool to be destroyed +* +* Data structures associated with pool are freed. +* All memory allocated from pool must be freed before +* destorying it. +*/ +extern void tlsf_destroy_memory_pool (void *mem_pool); + +/** +* tlsf_malloc - allocate memory from given pool +* @size: no. of bytes +* @mem_pool: pool to allocate from +*/ +extern void *tlsf_malloc (size_t size,void *mem_pool); + +/** +* tlsf_calloc - allocate and zero-out memory from given pool +* @size: no. of bytes +* @mem_pool: pool to allocate from +*/ +extern void *tlsf_calloc (size_t nelem,size_t elem_size,void *mem_pool); + +/** +* tlsf_free - free memory from given pool +* @ptr: address of memory to be freed +* @mem_pool: pool to free from +*/ +extern void tlsf_free (void *ptr,void *mem_pool); + +/** +* tlsf_get_used_size - get memory currently used by given pool +* +* Used memory includes stored data + metadata + internal fragmentation +*/ +extern size_t tlsf_get_used_size (void *mem_pool); + +/** +* tlsf_get_total_size - get total memory currently allocated for given pool +* +* This is the total memory currently allocated for this pool which includes +* used size + free size. +* +* (Total - Used) is good indicator of memory efficiency of allocator. +*/ +extern size_t tlsf_get_total_size (void *mem_pool); + +#endif
init/Kconfig +28 -0
@@ -734,6 +734,34 @@ config SLOB endchoice +config TLSF + depends on EMBEDDED + tristate "TLSF Allocator" + help + Two Level Segregate Fit Allocator. Its fast and gives low + fragmentation. See: + http://rtportal.upv.es/rtmalloc/ + http://code.google.com/p/compcache/wiki/TLSFAllocator + for more information. + +config TLSF_DEBUG + default y + depends on TLSF + bool "Enable TLSF allocator debugging" if EMBEDDED + help + Enable TLSF debugging. + This causes negligible performance loss and size increase. + If unusure, say Y. + +config TLSF_STATS + default N + depends on TLSF + bool "Collect TLSF statistics" if EMBEDDED + help + Creates /proc/tlsfinfo to export various tlsf statistics. + This adds about 30K to size with significant performance loss. + If unsure, say N. + config PROFILING bool "Profiling support (EXPERIMENTAL)" help
mm/Makefile +1 -0
@@ -27,6 +27,7 @@ obj-$(CONFIG_TINY_SHMEM) += tiny-shmem.o obj-$(CONFIG_SLOB) += slob.o obj-$(CONFIG_SLAB) += slab.o obj-$(CONFIG_SLUB) += slub.o +obj-$(CONFIG_TLSF) += tlsf.o obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o obj-$(CONFIG_FS_XIP) += filemap_xip.o obj-$(CONFIG_MIGRATION) += migrate.o
mm/tlsf.c new file +676 -0
676 lines not on this page · raw message
mm/tlsf_int.h new file +145 -0
145 lines not on this page · raw message
PATCH 3/3 compcache: documentation
In reply to Nitin Guptacompcache usage documentation.
--- Documentation/compcache.txt | 29 +++++++++++++++++++++++++++++ 1 files changed, 29 insertions(+), 0 deletions(-)
Documentation/compcache.txt new file +29 -0
@@ -0,0 +1,29 @@ + +compcache: Compressed RAM based swap device +------------------------------------------- + +Project home: http://code.google.com/p/compcache + +* Introduction +This is a RAM based block device which acts as swap disk. +Pages swapped to this device are compressed and stored in +memory itself. + +It uses these components: + - TLSF: memory allocator + - LZO: de/compressor + +* Usage + - modprobe compcache compcache_size_kbytes=<size> + (If no size is specified, default size of 25% of RAM is taken). + - swapon /dev/ramzswap0 + +* Notes + - Allocator and compcache statistics are exported via /proc interface: + (if stats are enabled for corresponding modules) + - /proc/tlsfinfo (from tlsf.ko) + - /proc/compcache (from compcache.ko) + + +Nitin Gupta +([email protected])