compcache: in-memory compressed swapping v3
Subsystems (4)
cover compcache: in-memory compressed swapping v3
Project home: http://compcache.googlecode.com/
* Changelog: v3 vs v2
- All cleanups as suggested by Pekka.
- Move to staging (drivers/block/ramzswap/ -> drivers/staging/ramzswap/).
- Remove swap discard hooks -- swap notify support makes these redundant.
- Unify duplicate code between init_device() fail path and reset_device().
- Fix zero-page accounting.
- Do not accept backing swap with bad pages.
* Changelog: v2 vs initial revision
- Use 'struct page' instead of 32-bit PFNs in ramzswap driver and xvmalloc. This is to make these 64-bit safe.
- xvmalloc is no longer a separate module and does not export any symbols. Its compiled directly with ramzswap block driver. This is to avoid any last bit of confusion with any other allocator.
- set_swap_free_notify() now accepts block_device as parameter instead of swp_entry_t (interface cleanup).
- Fix: Make sure ramzswap disksize matches usable pages in backing swap file. This caused initialization error in case backing swap file had intra-page fragmentation.
It creates RAM based block devices which can be used (only) as swap disks. Pages swapped to these disks are compressed and stored in memory itself. This is a big win over swapping to slow hard-disk which are typically used as swap disk. For flash, these suffer from wear-leveling issues when used as swap disk
- so again its helpful. For swapless systems, it allows more apps to run for a given amount of memory.
It can create multiple ramzswap devices (/dev/ramzswapX, X = 0, 1, 2, ...). Each of these devices can have separate backing swap (file or disk partition) which is used when incompressible page is found or memory limit for device is reached.
A separate userspace utility called rzscontrol is used to manage individual ramzswap devices.
* Testing notes
Tested on x86, x64, ARM
ARM:
- Cortex-A8 (Beagleboard)
- ARM11 (Android G1)
- OMAP2420 (Nokia N810)
* Performance
All performance numbers/plots can be found at:
http://code.google.com/p/compcache/wiki/Performance
Below is a summary of this data:
General:
- Swap R/W times are reduced from milliseconds (in case of hard disks)
down to microseconds.
Positive cases:
- Shows 33% improvement in 'scan' benchmark which allocates given amount
of memory and linearly reads/writes to this region. This benchmark also exposes bottlenecks in ramzswap code (global mutex) due to which this gain is so small.
- On Linux thin clients, it gives the effect of nearly doubling the amount of
memory.
Negative cases:
Any workload that has active working set w.r.t. filesystem cache that is nearly equal to amount of RAM while has minimal anonymous memory requirement, is expected to suffer maximum loss in performance with ramzswap enabled.
Iozone filesystem benchmark can simulate exactly this kind of workload. As expected, this test shows performance loss of ~25% with ramzswap.
drivers/staging/Kconfig | 2 + drivers/staging/Makefile | 1 + drivers/staging/ramzswap/Kconfig | 21 + drivers/staging/ramzswap/Makefile | 3 + drivers/staging/ramzswap/ramzswap.txt | 51 + drivers/staging/ramzswap/ramzswap_drv.c | 1462 +++++++++++++++++++++++++++++ drivers/staging/ramzswap/ramzswap_drv.h | 173 ++++ drivers/staging/ramzswap/ramzswap_ioctl.h | 50 + drivers/staging/ramzswap/xvmalloc.c | 533 +++++++++++ drivers/staging/ramzswap/xvmalloc.h | 30 + drivers/staging/ramzswap/xvmalloc_int.h | 86 ++ include/linux/swap.h | 5 + mm/swapfile.c | 34 + 13 files changed, 2451 insertions(+), 0 deletions(-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/4 xvmalloc memory allocator
* Features:
- Low metadata overhead (just 4 bytes per object)
- O(1) Alloc/Free - except when we have to call system page allocator to get additional memory.
- Very low fragmentation: In all tests, xvmalloc memory usage is within 12% of "Ideal".
- Pool based allocator: Each pool can grow and shrink.
- It maps pages only when required. So, it does not hog vmalloc area which is very small on 32-bit systems.
SLUB allocator could not be used due to fragmentation issues:
http://code.google.com/p/compcache/wiki/AllocatorsComparison
Data here shows kmalloc using ~43% more memory than TLSF and xvMalloc is showed ~2% more space efficiency than TLSF (due to smaller metadata). Creating various kmem_caches can reduce space efficiency gap but still problem of being limited to low memory exists. Also, it depends on allocating higher order pages to reduce fragmentation - this is not acceptable for ramzswap as it is used under memory crunch (its a swap device!).
SLOB allocator could not be used do to reasons mentioned here:
http://lkml.org/lkml/2009/3/18/210
* Implementation:
It uses two-level bitmap search to find free list containing block of correct size. This idea is taken from TLSF (Two-Level Segregate Fit) allocator and is well explained in its paper (see [Links] below).
* Limitations:
- Poor scalability: No per-cpu data structures (work in progress).
[Links]
1. Details and Performance data:
http://code.google.com/p/compcache/wiki/xvMalloc
http://code.google.com/p/compcache/wiki/xvMallocPerformance
2. TLSF memory allocator:
home: http://rtportal.upv.es/rtmalloc/
paper: http://rtportal.upv.es/rtmalloc/files/MRBC_2008.pdf
--- drivers/staging/ramzswap/xvmalloc.c | 533 +++++++++++++++++++++++++++++++ drivers/staging/ramzswap/xvmalloc.h | 30 ++ drivers/staging/ramzswap/xvmalloc_int.h | 86 +++++ 3 files changed, 649 insertions(+), 0 deletions(-)
drivers/staging/ramzswap/xvmalloc.c new file +533 -0
@@ -0,0 +1,533 @@ +/* +* xvmalloc memory allocator +* +* Copyright (C) 2008, 2009 Nitin Gupta +* +* This code is released using a dual license strategy: BSD/GPL +* You can choose the licence that better fits your requirements. +* +* Released under the terms of 3-clause BSD License +* Released under the terms of GNU General Public License Version 2.0 +*/ + +#include <linux/bitops.h> +#include <linux/errno.h> +#include <linux/highmem.h> +#include <linux/init.h> +#include <linux/string.h> +#include <linux/slab.h> + +#include "xvmalloc.h" +#include "xvmalloc_int.h" + +static void stat_inc (u64 *value) +{ + *value = *value +1 ; +} + +static void stat_dec (u64 *value) +{ + *value = *value -1 ; +} + +static int test_flag (struct block_header *block,enum blockflags flag) +{ +return block->prev &BIT (flag); +} + +static void set_flag (struct block_header *block,enum blockflags flag) +{ + block->prev |=BIT (flag); +} + +static void clear_flag (struct block_header *block,enum blockflags flag) +{ + block->prev &= ~BIT (flag); +} + +/* +* Given <page, offset> pair, provide a derefrencable pointer. +* This is called from xv_malloc/xv_free path, so it +* needs to be fast. +*/ +static void *get_ptr_atomic (struct page *page,u16 offset,enum km_type type) +{ +unsigned char *base; + + base =kmap_atomic (page, type); +return base + offset; +} + +static void put_ptr_atomic (void *ptr,enum km_type type) +{ +kunmap_atomic (ptr, type); +} + +static u32 get_blockprev (struct block_header *block) +{ +return block->prev &PREV_MASK ; +} + +static void set_blockprev (struct block_header *block,u16 new_offset) +{ + block->prev = new_offset | (block->prev &FLAGS_MASK ); +} + +static struct block_header *BLOCK_NEXT (struct block_header *block) +{ +return (struct block_header *) + ((char *)block + block->size +XV_ALIGN ); +} + +/* +* Get index of free list containing blocks of maximum size +* which is less than or equal to given size. +*/ +static u32 get_index_for_insert (u32 size) +{ +if (unlikely (size >XV_MAX_ALLOC_SIZE )) + size =XV_MAX_ALLOC_SIZE ; + size &= ~FL_DELTA_MASK ; +return (size -XV_MIN_ALLOC_SIZE ) >>FL_DELTA_SHIFT ; +} + +/* +* Get index of free list having blocks of size greater than +* or equal to requested size. +*/ +static u32 get_index (u32 size) +{ +if (unlikely (size <XV_MIN_ALLOC_SIZE )) + size =XV_MIN_ALLOC_SIZE ; + size =ALIGN (size,FL_DELTA ); +return (size -XV_MIN_ALLOC_SIZE ) >>FL_DELTA_SHIFT ; +} + +/* +* Allocate a memory page. Called when a pool needs to grow. +*/ +static struct page *xv_alloc_page (gfp_t flags) +{ +struct page *page; + + page =alloc_page (flags); +if (unlikely (!page)) +return 0 ; + +return page; +} + +/* +* Called when all objects in a page are freed. +*/ +static void xv_free_page (struct page *page) +{ +__free_page (page); +} + +/** +* find_block - find block of at least given size +* @pool: memory pool to search from +* @size: size of block required +* @page: page containing required block +* @offset: offset within the page where block is located. +* +* Searches two level bitmap to locate block of at least +* the given size. If such a block is found, it provides +* <page, offset> to identify this block and returns index +* in freelist where we found this block. +* Otherwise, returns 0 and <page, offset> params are not touched. +*/ +static u32 find_block (struct xv_pool *pool,u32 size, +struct page **page,u32 *offset) +{ +ulong flbitmap, slbitmap; +u32 flindex, slindex, slbitstart; + +/* There are no free blocks in this pool */ +if (!pool->flbitmap) +return 0 ; + +/* Get freelist index correspoding to this size */ + slindex =get_index (size); + slbitmap = pool->slbitmap[slindex /BITS_PER_LONG ]; + slbitstart = slindex %BITS_PER_LONG ; + +/* +* If freelist is not empty at this index, we found the +* block - head of this list. This is approximate best-fit match. +*/ +if (test_bit (slbitstart, &slbitmap)) { + *page = pool->freelist[slindex].page; + *offset = pool->freelist[slindex].offset; +return slindex; + } + +/* +* No best-fit found. Search a bit further in bitmap for a free block. +* Second level bitmap consists of series of 32-bit chunks. Search +* further in the chunk where we expected a best-fit, starting from +* index location found above. +*/ + slbitstart++; + slbitmap >>= slbitstart; + +/* Skip this search if we were already at end of this bitmap chunk */ +if ((slbitstart !=BITS_PER_LONG ) && slbitmap) { + slindex +=__ffs (slbitmap) +1 ; + *page = pool->freelist[slindex].page; + *offset = pool->freelist[slindex].offset; +return slindex; + } + +/* Now do a full two-level bitmap search to find next nearest fit */ + flindex = slindex /BITS_PER_LONG ; + + flbitmap = (pool->flbitmap) >> (flindex +1 ); +if (!flbitmap) +return 0 ; + + flindex +=__ffs (flbitmap) +1 ; + slbitmap = pool->slbitmap[flindex]; + slindex = (flindex *BITS_PER_LONG ) +__ffs (slbitmap); + *page = pool->freelist[slindex].page; + *offset = pool->freelist[slindex].offset; + +return slindex; +} + +/* +* Insert block at <page, offset> in freelist of given pool. +* freelist used depends on block size. +*/ +static void insert_block (struct xv_pool *pool,struct page *page,u32 offset, +struct block_header *block) +{ +u32 flindex, slindex; +struct block_header *nextblock; + + slindex =get_index_for_insert (block->size); + flindex = slindex /BITS_PER_LONG ; + + block->link.prev_page =0 ; + block->link.prev_offset =0 ; + block->link.next_page = pool->freelist[slindex].page; + block->link.next_offset = pool->freelist[slindex].offset; + pool->freelist[slindex].page = page; + pool->freelist[slindex].offset = offset; + +if (block->link.next_page) { + nextblock =get_ptr_atomic (block->link.next_page, + block->link.next_offset,KM_USER1 ); + nextblock->link.prev_page = page; + nextblock->link.prev_offset = offset; +put_ptr_atomic (nextblock,KM_USER1 ); + } + +__set_bit (slindex %BITS_PER_LONG , &pool->slbitmap[flindex]); +__set_bit (flindex, &pool->flbitmap); +} + +/* +* Remove block from head of freelist. Index 'slindex' identifies the freelist. +*/ +static void remove_block_head (struct xv_pool *pool, +struct block_header *block,u32 slindex) +{ +struct block_header *tmpblock; +u32 flindex = slindex /BITS_PER_LONG ; + + pool->freelist[slindex].page = block->link.next_page; + pool->freelist[slindex].offset = block->link.next_offset; + block->link.prev_page =0 ; + block->link.prev_offset =0 ; + +if (!pool->freelist[slindex].page) { +__clear_bit (slindex %BITS_PER_LONG , &pool->slbitmap[flindex]); +if (!pool->slbitmap[flindex]) +__clear_bit (flindex, &pool->flbitmap); + }else { +/* +* DEBUG ONLY: We need not reinitialize freelist head previous +* pointer to 0 - we never depend on its value. But just for +* sanity, lets do it. +*/ + tmpblock =get_ptr_atomic (pool->freelist[slindex].page, + pool->freelist[slindex].offset,KM_USER1 ); + tmpblock->link.prev_page =0 ; + tmpblock->link.prev_offset =0 ; +put_ptr_atomic (tmpblock,KM_USER1 ); + } +} + +/* +* Remove block from freelist. Index 'slindex' identifies the freelist. +*/ +static void remove_block (struct xv_pool *pool,struct page *page,u32 offset, +struct block_header *block,u32 slindex) +{ +u32 flindex; +struct block_header *tmpblock; + +if (pool->freelist[slindex].page == page + && pool->freelist[slindex].offset == offset) { +remove_block_head (pool, block, slindex); +return ; + } + + flindex = slindex /BITS_PER_LONG ; + +if (block->link.prev_page) { + tmpblock =get_ptr_atomic (block->link.prev_page, + block->link.prev_offset,KM_USER1 ); + tmpblock->link.next_page = block->link.next_page; + tmpblock->link.next_offset = block->link.next_offset; +put_ptr_atomic (tmpblock,KM_USER1 ); + } + +if (block->link.next_page) { + tmpblock =get_ptr_atomic (block->link.next_page, + block->link.next_offset,KM_USER1 ); + tmpblock->link.prev_page = block->link.prev_page; + tmpblock->link.prev_offset = block->link.prev_offset; +put_ptr_atomic (tmpblock,KM_USER1 ); + } + +return ; +} + +/* +* Allocate a page and add it freelist of given pool. +*/ +static int grow_pool (struct xv_pool *pool,gfp_t flags) +{ +struct page *page; +struct block_header *block; + + page =xv_alloc_page (flags); +if (unlikely (!page)) +return -ENOMEM ; + +stat_inc (&pool->total_pages); + +spin_lock (&pool->lock); + block =get_ptr_atomic (page,0 ,KM_USER0 ); + + block->size =PAGE_SIZE -XV_ALIGN ; +set_flag (block,BLOCK_FREE ); +clear_flag (block,PREV_FREE ); +set_blockprev (block,0 ); + +insert_block (pool, page,0 , block); + +put_ptr_atomic (block,KM_USER0 ); +spin_unlock (&pool->lock); + +return 0 ; +} + +/* +* Create a memory pool. Allocates freelist, bitmaps and other +* per-pool metadata. +*/ +struct xv_pool *xv_create_pool (void ) +{ +u32 ovhd_size; +struct xv_pool *pool; + + ovhd_size =roundup (sizeof (*pool),PAGE_SIZE ); + pool =kzalloc (ovhd_size,GFP_KERNEL ); +if (!pool) +return NULL ; + +spin_lock_init (&pool->lock); + +return pool; +} + +void xv_destroy_pool (struct xv_pool *pool) +{ +kfree (pool); +} + +/** +* xv_malloc - Allocate block of given size from pool. +* @pool: pool to allocate from +* @size: size of block to allocate +* @page: page no. that holds the object +* @offset: location of object within page +* +* On success, <page, offset> identifies block allocated +* and 0 is returned. On failure, <page, offset> is set to +* 0 and -ENOMEM is returned. +* +* Allocation requests with size > XV_MAX_ALLOC_SIZE will fail. +*/ +int xv_malloc (struct xv_pool *pool,u32 size,struct page **page, +u32 *offset,gfp_t flags) +{ +int error; +u32 index, tmpsize, origsize, tmpoffset; +struct block_header *block, *tmpblock; + + *page =NULL ; + *offset =0 ; + origsize = size; + +if (unlikely (!size || size >XV_MAX_ALLOC_SIZE )) +return -ENOMEM ; + + size =ALIGN (size,XV_ALIGN ); + +spin_lock (&pool->lock); + + index =find_block (pool, size, page, offset); + +if (!*page) { +spin_unlock (&pool->lock); +if (flags &GFP_NOWAIT ) +return -ENOMEM ; + error =grow_pool (pool, flags); +if (unlikely (error)) +return -ENOMEM ; + +spin_lock (&pool->lock); + index =find_block (pool, size, page, offset); + } + +if (!*page) { +spin_unlock (&pool->lock); +return -ENOMEM ; + } + + block =get_ptr_atomic (*page, *offset,KM_USER0 ); + +remove_block_head (pool, block, index); + +/* Split the block if required */ + tmpoffset = *offset + size +XV_ALIGN ; + tmpsize = block->size - size; + tmpblock = (struct block_header *)((char *)block + size +XV_ALIGN ); +if (tmpsize) { + tmpblock->size = tmpsize -XV_ALIGN ; +set_flag (tmpblock,BLOCK_FREE ); +clear_flag (tmpblock,PREV_FREE ); + +set_blockprev (tmpblock, *offset); +if (tmpblock->size >=XV_MIN_ALLOC_SIZE ) +insert_block (pool, *page, tmpoffset, tmpblock); + +if (tmpoffset +XV_ALIGN + tmpblock->size !=PAGE_SIZE ) { + tmpblock =BLOCK_NEXT (tmpblock); +set_blockprev (tmpblock, tmpoffset); + } + }else { +/* This block is exact fit */ +if (tmpoffset !=PAGE_SIZE ) +clear_flag (tmpblock,PREV_FREE ); + } + + block->size = origsize; +clear_flag (block,BLOCK_FREE ); + +put_ptr_atomic (block,KM_USER0 ); +spin_unlock (&pool->lock); + + *offset +=XV_ALIGN ; + +return 0 ; +} + +/* +* Free block identified with <page, offset> +*/ +void xv_free (struct xv_pool *pool,struct page *page,u32 offset) +{ +void *page_start; +struct block_header *block, *tmpblock; + + offset -=XV_ALIGN ; + +spin_lock (&pool->lock); + + page_start =get_ptr_atomic (page,0 ,KM_USER0 ); + block = (struct block_header *)((char *)page_start + offset); + +/* Catch double free bugs */ +BUG_ON (test_flag (block,BLOCK_FREE )); + + block->size =ALIGN (block->size,XV_ALIGN ); + + tmpblock =BLOCK_NEXT (block); +if (offset + block->size +XV_ALIGN ==PAGE_SIZE ) + tmpblock =NULL ; + +/* Merge next block if its free */ +if (tmpblock &&test_flag (tmpblock,BLOCK_FREE )) { +/* +* Blocks smaller than XV_MIN_ALLOC_SIZE +* are not inserted in any free list. +*/ +if (tmpblock->size >=XV_MIN_ALLOC_SIZE ) { +remove_block (pool, page, + offset + block->size +XV_ALIGN , tmpblock, +get_index_for_insert (tmpblock->size)); + } + block->size += tmpblock->size +XV_ALIGN ; + } + +/* Merge previous block if its free */ +if (test_flag (block,PREV_FREE )) { + tmpblock = (struct block_header *)((char *)(page_start) + +get_blockprev (block)); + offset = offset - tmpblock->size -XV_ALIGN ; + +if (tmpblock->size >=XV_MIN_ALLOC_SIZE ) +remove_block (pool, page, offset, tmpblock, +get_index_for_insert (tmpblock->size)); + + tmpblock->size += block->size +XV_ALIGN ; + block = tmpblock; + } + +/* No used objects in this page. Free it. */ +if (block->size ==PAGE_SIZE -XV_ALIGN ) { +put_ptr_atomic (page_start,KM_USER0 ); +spin_unlock (&pool->lock); + +xv_free_page (page); +stat_dec (&pool->total_pages); +return ; + } + +set_flag (block,BLOCK_FREE ); +if (block->size >=XV_MIN_ALLOC_SIZE ) +insert_block (pool, page, offset, block); + +if (offset + block->size +XV_ALIGN !=PAGE_SIZE ) { + tmpblock =BLOCK_NEXT (block); +set_flag (tmpblock,PREV_FREE ); +set_blockprev (tmpblock, offset); + } + +put_ptr_atomic (page_start,KM_USER0 ); +spin_unlock (&pool->lock); + +return ; +} + +u32 xv_get_object_size (void *obj) +{ +struct block_header *blk; + + blk = (struct block_header *)((char *)(obj) -XV_ALIGN ); +return blk->size; +} + +/* +* Returns total memory used by allocator (userdata + metadata) +*/ +u64 xv_get_total_size_bytes (struct xv_pool *pool) +{ +return pool->total_pages <<PAGE_SHIFT ; +}
drivers/staging/ramzswap/xvmalloc.h new file +30 -0
@@ -0,0 +1,30 @@ +/* +* xvmalloc memory allocator +* +* Copyright (C) 2008, 2009 Nitin Gupta +* +* This code is released using a dual license strategy: BSD/GPL +* You can choose the licence that better fits your requirements. +* +* Released under the terms of 3-clause BSD License +* Released under the terms of GNU General Public License Version 2.0 +*/ + +#ifndef _XV_MALLOC_H_ +#define _XV_MALLOC_H_ + +#include <linux/types.h> + +struct xv_pool ; + +struct xv_pool *xv_create_pool (void ); +void xv_destroy_pool (struct xv_pool *pool); + +int xv_malloc (struct xv_pool *pool,u32 size,struct page **page, +u32 *offset,gfp_t flags); +void xv_free (struct xv_pool *pool,struct page *page,u32 offset); + +u32 xv_get_object_size (void *obj); +u64 xv_get_total_size_bytes (struct xv_pool *pool); + +#endif
drivers/staging/ramzswap/xvmalloc_int.h new file +86 -0
86 lines not on this page · raw message
Re: [PATCH 1/4] xvmalloc memory allocator
Quoted 5 lines · Nitin Gupta
(...)
drivers/staging/ramzswap/xvmalloc.c new file +4
-0+ +/* +* Allocate a memory page. Called when a pool needs to grow. +*/
drivers/staging/ramzswap/xvmalloc.c line 118
Quoted 10 lines
drivers/staging/ramzswap/xvmalloc.c new file +10
-0+static struct page *xv_alloc_page (gfp_t flags) +{ +struct page *page; + + page =alloc_page (flags); +if (unlikely (!page)) +return 0 ; + +return page; +}
When alloc_page returns 0 it returns 0, when not - it returns page. Why not call alloc_page directly?
Quoted 26 lines
(...)
drivers/staging/ramzswap/xvmalloc.c new file +25
-0+/* +* Remove block from freelist. Index 'slindex' identifies the freelist. +*/ +static void remove_block (struct xv_pool *pool,struct page *page,u32 offset, +struct block_header *block,u32 slindex) +{ +u32 flindex; +struct block_header *tmpblock; + +if (pool->freelist[slindex].page == page + && pool->freelist[slindex].offset == offset) { +remove_block_head (pool, block, slindex); +return ; + } + + flindex = slindex /BITS_PER_LONG ; + +if (block->link.prev_page) { + tmpblock =get_ptr_atomic (block->link.prev_page, + block->link.prev_offset,KM_USER1 ); + tmpblock->link.next_page = block->link.next_page; + tmpblock->link.next_offset = block->link.next_offset; +put_ptr_atomic (tmpblock,KM_USER1 ); + } +
drivers/staging/ramzswap/xvmalloc.c line 297
Quoted 10 lines
drivers/staging/ramzswap/xvmalloc.c new file +10
-0+if (block->link.next_page) { + tmpblock =get_ptr_atomic (block->link.next_page, + block->link.next_offset,KM_USER1 ); + tmpblock->link.prev_page = block->link.prev_page; + tmpblock->link.prev_offset = block->link.prev_offset; +put_ptr_atomic (tmpblock,KM_USER1 ); + } + +return ; +}
needless return
Quoted 62 lines
drivers/staging/ramzswap/xvmalloc.c new file +61
-0+ +/* +* Allocate a page and add it freelist of given pool. +*/ +static int grow_pool (struct xv_pool *pool,gfp_t flags) +{ +struct page *page; +struct block_header *block; + + page =xv_alloc_page (flags); +if (unlikely (!page)) +return -ENOMEM ; + +stat_inc (&pool->total_pages); + +spin_lock (&pool->lock); + block =get_ptr_atomic (page,0 ,KM_USER0 ); + + block->size =PAGE_SIZE -XV_ALIGN ; +set_flag (block,BLOCK_FREE ); +clear_flag (block,PREV_FREE ); +set_blockprev (block,0 ); + +insert_block (pool, page,0 , block); + +put_ptr_atomic (block,KM_USER0 ); +spin_unlock (&pool->lock); + +return 0 ; +} + +/** +* xv_malloc - Allocate block of given size from pool. +* @pool: pool to allocate from +* @size: size of block to allocate +* @page: page no. that holds the object +* @offset: location of object within page +* +* On success, <page, offset> identifies block allocated +* and 0 is returned. On failure, <page, offset> is set to +* 0 and -ENOMEM is returned. +* +* Allocation requests with size > XV_MAX_ALLOC_SIZE will fail. +*/ +int xv_malloc (struct xv_pool *pool,u32 size,struct page **page, +u32 *offset,gfp_t flags) +{ +int error; +u32 index, tmpsize, origsize, tmpoffset; +struct block_header *block, *tmpblock; + + *page =NULL ; + *offset =0 ; + origsize = size; + +if (unlikely (!size || size >XV_MAX_ALLOC_SIZE )) +return -ENOMEM ; + + size =ALIGN (size,XV_ALIGN ); + +spin_lock (&pool->lock);
drivers/staging/ramzswap/xvmalloc.c line 392
Quoted 10 lines
drivers/staging/ramzswap/xvmalloc.c new file +10
-0+ + index =find_block (pool, size, page, offset); + +if (!*page) { +spin_unlock (&pool->lock); +if (flags &GFP_NOWAIT ) +return -ENOMEM ; + error =grow_pool (pool, flags); +if (unlikely (error)) +return -ENOMEM ;
shouldn't it return error? (grow_pool returns 0 or -ENOMEM for now but...)
Quoted 115 lines
drivers/staging/ramzswap/xvmalloc.c new file +115
-0+ +spin_lock (&pool->lock); + index =find_block (pool, size, page, offset); + } + +if (!*page) { +spin_unlock (&pool->lock); +return -ENOMEM ; + } + + block =get_ptr_atomic (*page, *offset,KM_USER0 ); + +remove_block_head (pool, block, index); + +/* Split the block if required */ + tmpoffset = *offset + size +XV_ALIGN ; + tmpsize = block->size - size; + tmpblock = (struct block_header *)((char *)block + size +XV_ALIGN ); +if (tmpsize) { + tmpblock->size = tmpsize -XV_ALIGN ; +set_flag (tmpblock,BLOCK_FREE ); +clear_flag (tmpblock,PREV_FREE ); + +set_blockprev (tmpblock, *offset); +if (tmpblock->size >=XV_MIN_ALLOC_SIZE ) +insert_block (pool, *page, tmpoffset, tmpblock); + +if (tmpoffset +XV_ALIGN + tmpblock->size !=PAGE_SIZE ) { + tmpblock =BLOCK_NEXT (tmpblock); +set_blockprev (tmpblock, tmpoffset); + } + }else { +/* This block is exact fit */ +if (tmpoffset !=PAGE_SIZE ) +clear_flag (tmpblock,PREV_FREE ); + } + + block->size = origsize; +clear_flag (block,BLOCK_FREE ); + +put_ptr_atomic (block,KM_USER0 ); +spin_unlock (&pool->lock); + + *offset +=XV_ALIGN ; + +return 0 ; +} + +/* +* Free block identified with <page, offset> +*/ +void xv_free (struct xv_pool *pool,struct page *page,u32 offset) +{ +void *page_start; +struct block_header *block, *tmpblock; + + offset -=XV_ALIGN ; + +spin_lock (&pool->lock); + + page_start =get_ptr_atomic (page,0 ,KM_USER0 ); + block = (struct block_header *)((char *)page_start + offset); + +/* Catch double free bugs */ +BUG_ON (test_flag (block,BLOCK_FREE )); + + block->size =ALIGN (block->size,XV_ALIGN ); + + tmpblock =BLOCK_NEXT (block); +if (offset + block->size +XV_ALIGN ==PAGE_SIZE ) + tmpblock =NULL ; + +/* Merge next block if its free */ +if (tmpblock &&test_flag (tmpblock,BLOCK_FREE )) { +/* +* Blocks smaller than XV_MIN_ALLOC_SIZE +* are not inserted in any free list. +*/ +if (tmpblock->size >=XV_MIN_ALLOC_SIZE ) { +remove_block (pool, page, + offset + block->size +XV_ALIGN , tmpblock, +get_index_for_insert (tmpblock->size)); + } + block->size += tmpblock->size +XV_ALIGN ; + } + +/* Merge previous block if its free */ +if (test_flag (block,PREV_FREE )) { + tmpblock = (struct block_header *)((char *)(page_start) + +get_blockprev (block)); + offset = offset - tmpblock->size -XV_ALIGN ; + +if (tmpblock->size >=XV_MIN_ALLOC_SIZE ) +remove_block (pool, page, offset, tmpblock, +get_index_for_insert (tmpblock->size)); + + tmpblock->size += block->size +XV_ALIGN ; + block = tmpblock; + } + +/* No used objects in this page. Free it. */ +if (block->size ==PAGE_SIZE -XV_ALIGN ) { +put_ptr_atomic (page_start,KM_USER0 ); +spin_unlock (&pool->lock); + +xv_free_page (page); +stat_dec (&pool->total_pages); +return ; + } + +set_flag (block,BLOCK_FREE ); +if (block->size >=XV_MIN_ALLOC_SIZE ) +insert_block (pool, page, offset, block); + +if (offset + block->size +XV_ALIGN !=PAGE_SIZE ) {
drivers/staging/ramzswap/xvmalloc.c line 517
Quoted 10 lines
drivers/staging/ramzswap/xvmalloc.c new file +10
-0+ tmpblock =BLOCK_NEXT (block); +set_flag (tmpblock,PREV_FREE ); +set_blockprev (tmpblock, offset); + } + +put_ptr_atomic (page_start,KM_USER0 ); +spin_unlock (&pool->lock); + +return ; +}
needless return
Marcin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 1/4] xvmalloc memory allocator
Sorry for late reply. I nearly missed this mail. My comments inline.
Quoted 11 lines · Marcin Slusarz
Quoted 10 lines · Nitin Gupta
(...)
drivers/staging/ramzswap/xvmalloc.c new file +9
-0+ +/* +* Allocate a memory page. Called when a pool needs to grow. +*/ +static struct page *xv_alloc_page (gfp_t flags) +{ +struct page *page; + + page =alloc_page (flags);
Quoted 9 lines
drivers/staging/ramzswap/xvmalloc.c line 118
Quoted 5 lines
drivers/staging/ramzswap/xvmalloc.c new file +5
-0+if (unlikely (!page)) +return 0 ; + +return page; +}When
alloc_pagereturns 0 it returns 0, when not - it returns page. Why not callalloc_pagedirectly?
We now call alloc_page() and __free_page directly. Removed these wrappers.
Quoted 9 lines
Quoted 9 lines
(...)
drivers/staging/ramzswap/xvmalloc.c new file +8
-0+/* +* Remove block from freelist. Index 'slindex' identifies the freelist. +*/ +static void remove_block (struct xv_pool *pool,struct page *page,u32 offset, +struct block_header *block,u32 slindex) +{ +u32 flindex; +struct block_header *tmpblock;
<snip>
Quoted 6 lines
drivers/staging/ramzswap/xvmalloc.c line 276
Quoted 3 lines
drivers/staging/ramzswap/xvmalloc.c new file +3
-0+ +return ; +}needless return
Removed.
Quoted 5 lines
Quoted 5 lines
drivers/staging/ramzswap/xvmalloc.c new file +5
-0+int xv_malloc (struct xv_pool *pool,u32 size,struct page **page, +u32 *offset,gfp_t flags) +{ +int error; +
<snip>
Quoted 10 lines
drivers/staging/ramzswap/xvmalloc.c line 392
Quoted 7 lines
drivers/staging/ramzswap/xvmalloc.c new file +7
-0+if (!*page) { +spin_unlock (&pool->lock); +if (flags &GFP_NOWAIT ) +return -ENOMEM ; + error =grow_pool (pool, flags); +if (unlikely (error)) +return -ENOMEM ;shouldn't it return error? (
grow_poolreturns 0 or-ENOMEMfor now but...)
Yes, it should return error. Corrected.
Quoted 53 lines
Quoted 53 lines
drivers/staging/ramzswap/xvmalloc.c new file +53
-0+ +spin_lock (&pool->lock); + index =find_block (pool, size, page, offset); + } + +if (!*page) { +spin_unlock (&pool->lock); +return -ENOMEM ; + } + + block =get_ptr_atomic (*page, *offset,KM_USER0 ); + +remove_block_head (pool, block, index); + +/* Split the block if required */ + tmpoffset = *offset + size +XV_ALIGN ; + tmpsize = block->size - size; + tmpblock = (struct block_header *)((char *)block + size +XV_ALIGN ); +if (tmpsize) { + tmpblock->size = tmpsize -XV_ALIGN ; +set_flag (tmpblock,BLOCK_FREE ); +clear_flag (tmpblock,PREV_FREE ); + +set_blockprev (tmpblock, *offset); +if (tmpblock->size >=XV_MIN_ALLOC_SIZE ) +insert_block (pool, *page, tmpoffset, tmpblock); + +if (tmpoffset +XV_ALIGN + tmpblock->size !=PAGE_SIZE ) { + tmpblock =BLOCK_NEXT (tmpblock); +set_blockprev (tmpblock, tmpoffset); + } + }else { +/* This block is exact fit */ +if (tmpoffset !=PAGE_SIZE ) +clear_flag (tmpblock,PREV_FREE ); + } + + block->size = origsize; +clear_flag (block,BLOCK_FREE ); + +put_ptr_atomic (block,KM_USER0 ); +spin_unlock (&pool->lock); + + *offset +=XV_ALIGN ; + +return 0 ; +} + +/* +* Free block identified with <page, offset> +*/ +void xv_free (struct xv_pool *pool,struct page *page,u32 offset) +{
<snip>
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return; +}needless return
Removed.
Regarding your comments on page_zero_filled: I'm not sure if using unsigned long is better or just u64 irrespective of arch. I just changed it to ulong
-- some bechmarks can help decide which one is optimal. Maybe we need arch
specific optimized versions which means moving it to lib/ or something.
Thanks for your feedback.
Nitin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/4 send callback when swap slot is freed
In reply to Nitin GuptaCurrently, we have "swap discard" mechanism which sends a discard bio request when we find a free cluster during scan_swap_map(). This callback can come a long time after swap slots are actually freed.
This delay in callback is a great problem when (compressed) RAM [1] is used as a swap device. So, this change adds a callback which is called as soon as a swap slot becomes free. For above mentioned case of swapping over compressed RAM device, this is very useful since we can immediately free memory allocated for this swap page.
This callback does not replace swap discard support. It is called with swap_lock held, so it is meant to trigger action that finishes quickly. However, swap discard is an I/O request and can be used for taking longer actions.
It is preferred to use this callback for ramzswap case even if discard mechanism could be improved such that it can be called as often as required. This is because, allocation of 'bio'(s) is undesirable since ramzswap always operates under low memory conditions (its a swap device). Also, batching of discard bio requests is not optimal since stale data can accumulate very quickly in ramzswap devices, pushing system further into low memory state.
Links:
[1] http://compcache.googlecode.com/
--- include/linux/swap.h | 5 +++++ mm/swapfile.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 0 deletions(-)
include/linux/swap.h +5 -0
@@ -8,6 +8,7 @@#include <linux/memcontrol.h> #include <linux/sched.h> #include <linux/node.h> +#include <linux/blkdev.h> #include <asm/atomic.h> #include <asm/page.h>
@@ -20,6 +21,8 @@ struct bio;#define SWAP_FLAG_PRIO_MASK 0x7fff#define SWAP_FLAG_PRIO_SHIFT 0 +typedef void (swap_free_notify_fn ) (struct block_device *,unsigned long ); +static inline int current_is_kswapd (void ) {return current->flags &PF_KSWAPD ;
@@ -155,6 +158,7 @@ struct swap_info_struct {unsigned int max;unsigned int inuse_pages;unsigned int old_block_size; +swap_free_notify_fn *swap_free_notify_fn; };struct swap_list_t {
@@ -295,6 +299,7 @@ extern sector_t swapdev_block(int, pgoff_t);extern struct swap_info_struct *get_swap_info_struct (unsigned );extern int reuse_swap_page (struct page *);extern int try_to_free_swap (struct page *); +extern void set_swap_free_notify (struct block_device *,swap_free_notify_fn *);struct backing_dev_info ;/* linux/mm/thrash.c */
mm/swapfile.c +34 -0
@@ -554,6 +554,38 @@ out:return NULL ; } +/* +* Sets callback for event when swap_map[offset] == 0 +* i.e. page at this swap offset is no longer used. +*/ +void set_swap_free_notify (struct block_device *bdev, +swap_free_notify_fn *notify_fn) +{ +unsigned int i; +struct swap_info_struct *sis; + +spin_lock (&swap_lock); +for (i =0 ; i <= nr_swapfiles; i++) { + sis = &swap_info[i]; +if (!(sis->flags &SWP_USED )) +continue ; +if (sis->bdev == bdev) +break ; + } + +/* swap device not found */ +if (i > nr_swapfiles) { +spin_unlock (&swap_lock); +return ; + } + +BUG_ON (!sis || sis->swap_free_notify_fn); + sis->swap_free_notify_fn = notify_fn; +spin_unlock (&swap_lock); +return ; +} +EXPORT_SYMBOL_GPL (set_swap_free_notify); +static int swap_entry_free (struct swap_info_struct *p,swp_entry_t ent,int cache) {
@@ -585,6 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; +if (p->swap_free_notify_fn) + p->swap_free_notify_fn (p->bdev, offset); }if (!swap_count (count))mem_cgroup_uncharge_swap (ent);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 113 lines · Nitin Gupta
Currently, we have "swap discard" mechanism which sends a discard bio request when we find a free cluster during
scan_swap_map(). This callback can come a long time after swap slots are actually freed.This delay in callback is a great problem when (compressed) RAM [1] is used as a swap device. So, this change adds a callback which is called as soon as a swap slot becomes free. For above mentioned case of swapping over compressed RAM device, this is very useful since we can immediately free memory allocated for this swap page.
This callback does not replace swap discard support. It is called with
swap_lockheld, so it is meant to trigger action that finishes quickly. However, swap discard is an I/O request and can be used for taking longer actions.It is preferred to use this callback for ramzswap case even if discard mechanism could be improved such that it can be called as often as required. This is because, allocation of 'bio'(s) is undesirable since ramzswap always operates under low memory conditions (its a swap device). Also, batching of discard bio requests is not optimal since stale data can accumulate very quickly in ramzswap devices, pushing system further into low memory state.
Links:
[1] http://compcache.googlecode.com/Signed-off-by Nitin Gupta <[email protected]>--- include/linux/swap.h | 5 +++++ mm/swapfile.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 0 deletions(-)include/linux/swap.h +5
-0@@ -8,6 +8,7 @@#include <linux/memcontrol.h> #include <linux/sched.h> #include <linux/node.h> +#include <linux/blkdev.h> #include <asm/atomic.h> #include <asm/page.h> @@ -20,6 +21,8 @@ struct bio;#define SWAP_FLAG_PRIO_MASK 0x7fff#define SWAP_FLAG_PRIO_SHIFT 0 +typedef void (swap_free_notify_fn ) (struct block_device *,unsigned long ); +static inline int current_is_kswapd (void ) {return current->flags &PF_KSWAPD ;@@ -155,6 +158,7 @@ struct swap_info_struct {unsigned int max;unsigned int inuse_pages;unsigned int old_block_size; +swap_free_notify_fn *swap_free_notify_fn; };struct swap_list_t {@@ -295,6 +299,7 @@ extern sector_t swapdev_block(int, pgoff_t);extern struct swap_info_struct *get_swap_info_struct (unsigned );extern int reuse_swap_page (struct page *);extern int try_to_free_swap (struct page *); +extern void set_swap_free_notify (struct block_device *,swap_free_notify_fn *);struct backing_dev_info ;/* linux/mm/thrash.c */ mm/swapfile.c +32
-0@@ -554,6 +554,38 @@ out:return NULL ; } +/* +* Sets callback for event when swap_map[offset] == 0 +* i.e. page at this swap offset is no longer used. +*/ +void set_swap_free_notify (struct block_device *bdev, +swap_free_notify_fn *notify_fn) +{ +unsigned int i; +struct swap_info_struct *sis; + +spin_lock (&swap_lock); +for (i =0 ; i <= nr_swapfiles; i++) { + sis = &swap_info[i]; +if (!(sis->flags &SWP_USED )) +continue ; +if (sis->bdev == bdev) +break ; + } + +/* swap device not found */ +if (i > nr_swapfiles) { +spin_unlock (&swap_lock); +return ; + } + +BUG_ON (!sis || sis->swap_free_notify_fn); + sis->swap_free_notify_fn = notify_fn; +spin_unlock (&swap_lock); +return ; +} +EXPORT_SYMBOL_GPL (set_swap_free_notify); +static int swap_entry_free (struct swap_info_struct *p,swp_entry_t ent,int cache) {
mm/swapfile.c static int swap_entry_free(struct swap_info_struct *p, · line 624
Quoted 8 lines
mm/swapfile.c +2
-0@@ -585,6 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; +if (p->swap_free_notify_fn) + p->swap_free_notify_fn (p->bdev, offset); }if (!swap_count (count))mem_cgroup_uncharge_swap (ent);
OK, this hits core kernel code so we need to CC some more mm/swapfile.c people. The set_swap_free_notify() API looks strange to me. Hugh, I think you mentioned that you're okay with an explicit hook. Any suggestions how to do this cleanly?
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 12 lines · Pekka Enberg
Quoted 11 lines · Nitin Gupta
mm/swapfile.c +4
-0+EXPORT_SYMBOL_GPL (set_swap_free_notify); +static int swap_entry_free (struct swap_info_struct *p,swp_entry_t ent,int cache) {@@ -585,6 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; +if (p->swap_free_notify_fn) + p->swap_free_notify_fn (p->bdev, offset);
Quoted 8 lines
mm/swapfile.c static int swap_entry_free(struct swap_info_struct *p, · line 624
Quoted 3 lines
mm/swapfile.c +0
-0}if (!swap_count (count))mem_cgroup_uncharge_swap (ent);OK, this hits core kernel code so we need to CC some more
mm/swapfile.cpeople. Theset_swap_free_notify()API looks strange to me. Hugh, I think you mentioned that you're okay with an explicit hook. Any suggestions how to do this cleanly?
No, no better suggestion. I quite see Nitin's point that ramzswap would benefit significantly from a callback here, though it's not a place (holding swap_lock) where we'd like to offer a callback at all.
I think I would prefer the naming to make it absolutely clear that it's a special for ramzswap or compcache, rather than dressing it up in the grand generality of a swap_free_notify_fn: giving our hacks fancy names doesn't really make them better.
(Does the bdev matching work out if there are any regular swapfiles around? I've not checked, might or might not need refinement there.)
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Hi Hugh,
Quoted 26 lines · Hugh Dickins
Quoted 12 lines · Pekka Enberg
Quoted 11 lines · Nitin Gupta
mm/swapfile.c +4
-0+EXPORT_SYMBOL_GPL (set_swap_free_notify); +static int swap_entry_free (struct swap_info_struct *p,swp_entry_t ent,int cache) {@@ -585,6 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; +if (p->swap_free_notify_fn) + p->swap_free_notify_fn (p->bdev, offset);Quoted 8 lines
mm/swapfile.c static int swap_entry_free(struct swap_info_struct *p, · line 624
Quoted 3 lines
mm/swapfile.c +0
-0}if (!swap_count (count))mem_cgroup_uncharge_swap (ent);OK, this hits core kernel code so we need to CC some more
mm/swapfile.cpeople. Theset_swap_free_notify()API looks strange to me. Hugh, I think you mentioned that you're okay with an explicit hook. Any suggestions how to do this cleanly?No, no better suggestion. I quite see Nitin's point that ramzswap would benefit significantly from a callback here, though it's not a place (holding
swap_lock) where we'd like to offer a callback at all.
Quoted 7 lines
I think I would prefer the naming to make it absolutely clear that it's a special for ramzswap or compcache, rather than dressing it up in the grand generality of a
swap_free_notify_fn: giving our hacks fancy names doesn't really make them better.(Does the bdev matching work out if there are any regular swapfiles around? I've not checked, might or might not need refinement there.)
The *hook* looks OK to me but set_swap_free_notify() looks like an ugly hack. I don't understand why we're setting up the hook lazily in ramzswap_read() nor do I understand why we need to look up struct swap_info_struct with a bdev. Surely there's a cleaner way to do all this? Probably somewhere in sys_swapon()?
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 6 lines · Pekka Enberg
The *hook* looks OK to me but
set_swap_free_notify()looks like an ugly hack. I don't understand why we're setting up the hook lazily inramzswap_read()nor do I understand why we need to look up structswap_info_structwith a bdev. Surely there's a cleaner way to do all this? Probably somewhere insys_swapon()?
Sounds like you have something in mind, may well be better, but please show us a patch... (my mind is elsewhere!)
Hugh
Re: [PATCH 2/4] send callback when swap slot is freed
Hi Hugh,
Quoted 10 lines · Hugh Dickins
Quoted 6 lines · Pekka Enberg
The *hook* looks OK to me but
set_swap_free_notify()looks like an ugly hack. I don't understand why we're setting up the hook lazily inramzswap_read()nor do I understand why we need to look up structswap_info_structwith a bdev. Surely there's a cleaner way to do all this? Probably somewhere insys_swapon()?Sounds like you have something in mind, may well be better, but please show us a patch... (my mind is elsewhere!)
Hey, you tricked me into a world of pain! Here's a totally untested patch that adds a "swapon" funciton to struct block_device_operations that the ramzswap driver can implement to setup ->swap_free_notify_fn.
Pekka
Quoted 1 line
From
41827c57196f7eae66701a6ae6565c226c0b7674Mon Sep 17 00:00:00 2001
From: Nitin Gupta <[email protected]>
Date: Fri, 18 Sep 2009 04:13:30 +0530
Subject: [PATCH] mm: swap slot free notifier
Currently, we have "swap discard" mechanism which sends a discard bio request when we find a free cluster during scan_swap_map(). This callback can come a long time after swap slots are actually freed.
This delay in callback is a great problem when (compressed) RAM [1] is used as a swap device. So, this change adds a callback which is called as soon as a swap slot becomes free. For above mentioned case of swapping over compressed RAM device, this is very useful since we can immediately free memory allocated for this swap page.
This callback does not replace swap discard support. It is called with swap_lock held, so it is meant to trigger action that finishes quickly. However, swap discard is an I/O request and can be used for taking longer actions.
It is preferred to use this callback for ramzswap case even if discard mechanism could be improved such that it can be called as often as required. This is because, allocation of 'bio'(s) is undesirable since ramzswap always operates under low memory conditions (its a swap device). Also, batching of discard bio requests is not optimal since stale data can accumulate very quickly in ramzswap devices, pushing system further into low memory state.
--- include/linux/blkdev.h | 2 ++ include/linux/swap.h | 16 ++++++++++++++++ mm/swapfile.c | 5 +++++ 3 files changed, 23 insertions(+), 0 deletions(-)
include/linux/blkdev.h +2 -0
@@ -27,6 +27,7 @@ struct scsi_ioctl_command;struct request_queue ;struct elevator_queue ;struct request_pm_state ; +struct swap_info_struct ;struct blk_trace ;struct request ;struct sg_io_hdr ;
@@ -1239,6 +1240,7 @@ struct block_device_operations { unsigned long long);int (*revalidate_disk) (struct gendisk *);int (*getgeo)(struct block_device *,struct hd_geometry *); +int (*swapon)(struct block_device *,struct swap_info_struct *);struct module *owner; };
include/linux/swap.h +16 -0
@@ -8,6 +8,7 @@#include <linux/memcontrol.h> #include <linux/sched.h> #include <linux/node.h> +#include <linux/blkdev.h> #include <asm/atomic.h> #include <asm/page.h>
@@ -20,6 +21,8 @@ struct bio;#define SWAP_FLAG_PRIO_MASK 0x7fff#define SWAP_FLAG_PRIO_SHIFT 0 +typedef void (swap_free_notify_fn ) (struct block_device *,unsigned long ); +static inline int current_is_kswapd (void ) {return current->flags &PF_KSWAPD ;
@@ -155,6 +158,7 @@ struct swap_info_struct {unsigned int max;unsigned int inuse_pages;unsigned int old_block_size; +swap_free_notify_fn *swap_free_notify_fn; };struct swap_list_t {
@@ -297,6 +301,18 @@ extern int reuse_swap_page(struct page *);extern int try_to_free_swap (struct page *);struct backing_dev_info ; +static inline int +blkdev_swapon (struct block_device *bdev,struct swap_info_struct *sis) +{ +struct gendisk *disk = bdev->bd_disk; +int err =0 ; + +if (disk->fops->swapon) + err = disk->fops->swapon (bdev, sis); + +return err; +} +/* linux/mm/thrash.c */ extern struct mm_struct *swap_token_mm;extern void grab_swap_token (struct mm_struct *);
mm/swapfile.c +5 -0
@@ -585,6 +585,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; +if (p->swap_free_notify_fn) + p->swap_free_notify_fn (p->bdev, offset); }if (!swap_count (count))mem_cgroup_uncharge_swap (ent);
@@ -1845,6 +1847,9 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)if (error <0 ) goto bad_swap; p->bdev = bdev; + error =blkdev_swapon (bdev, p); +if (error <0 ) + goto bad_swap; }else if (S_ISREG (inode->i_mode)) { p->bdev = inode->i_sb->s_bdev;mutex_lock (&inode->i_mutex);
Signature · 9 lines
1.5.6.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Hi Pekka,
Quoted 10 lines · Pekka Enberg
Quoted 8 lines · Hugh Dickins
Quoted 6 lines · Pekka Enberg
The *hook* looks OK to me but
set_swap_free_notify()looks like an ugly hack. I don't understand why we're setting up the hook lazily inramzswap_read()nor do I understand why we need to look up structswap_info_structwith a bdev. Surely there's a cleaner way to do all this? Probably somewhere insys_swapon()?
Quoted 7 lines
Quoted 2 lines
Sounds like you have something in mind, may well be better, but please show us a patch... (my mind is elsewhere!)
Hey, you tricked me into a world of pain! Here's a totally untested patch that adds a "swapon" funciton to struct
block_device_operationsthat the ramzswap driver can implement to setup ->swap_free_notify_fn.
It is understood that this swap notify callback is bit of a hack. I think we will not gain much trying to beautify this hack. However, I agree with Hugh's suggestion to rename this notify callback related function/variables to make it explicit that its completely ramzswap related. I will send a path that affects these renames as reply to patch 0/4.
A solution that is generic while satisfying ramzswap demands may be a bit more involved. It will be easier to incrementally develop these additions.
Thanks,
Nitin
Quoted 118 lines
Quoted 2 lines
From
41827c57196f7eae66701a6ae6565c226c0b7674Mon Sep 17 00:00:00 2001From: Nitin Gupta<[email protected]>
Date: Fri, 18 Sep 2009 04:13:30 +0530
Subject: [PATCH] mm: swap slot free notifierCurrently, we have "swap discard" mechanism which sends a discard bio request when we find a free cluster during
scan_swap_map(). This callback can come a long time after swap slots are actually freed.This delay in callback is a great problem when (compressed) RAM [1] is used as a swap device. So, this change adds a callback which is called as soon as a swap slot becomes free. For above mentioned case of swapping over compressed RAM device, this is very useful since we can immediately free memory allocated for this swap page.
This callback does not replace swap discard support. It is called with
swap_lockheld, so it is meant to trigger action that finishes quickly. However, swap discard is an I/O request and can be used for taking longer actions.It is preferred to use this callback for ramzswap case even if discard mechanism could be improved such that it can be called as often as required. This is because, allocation of 'bio'(s) is undesirable since ramzswap always operates under low memory conditions (its a swap device). Also, batching of discard bio requests is not optimal since stale data can accumulate very quickly in ramzswap devices, pushing system further into low memory state.
Signed-off-by Nitin Gupta<[email protected]>Signed-off-by Pekka Enberg<[email protected]>--- include/linux/blkdev.h | 2 ++ include/linux/swap.h | 16 ++++++++++++++++ mm/swapfile.c | 5 +++++ 3 files changed, 23 insertions(+), 0 deletions(-)include/linux/blkdev.h +2
-0@@ -27,6 +27,7 @@ struct scsi_ioctl_command;struct request_queue ;struct elevator_queue ;struct request_pm_state ; +struct swap_info_struct ;struct blk_trace ;struct request ;struct sg_io_hdr ;@@ -1239,6 +1240,7 @@ struct block_device_operations { unsigned long long);int (*revalidate_disk) (struct gendisk *);int (*getgeo)(struct block_device *,struct hd_geometry *); +int (*swapon)(struct block_device *,struct swap_info_struct *);struct module *owner; };include/linux/swap.h +16
-0@@ -8,6 +8,7 @@#include <linux/memcontrol.h> #include <linux/sched.h> #include <linux/node.h> +#include <linux/blkdev.h> #include <asm/atomic.h> #include <asm/page.h> @@ -20,6 +21,8 @@ struct bio;#define SWAP_FLAG_PRIO_MASK 0x7fff#define SWAP_FLAG_PRIO_SHIFT 0 +typedef void (swap_free_notify_fn ) (struct block_device *,unsigned long ); +static inline int current_is_kswapd (void ) {return current->flags&PF_KSWAPD ;@@ -155,6 +158,7 @@ struct swap_info_struct {unsigned int max;unsigned int inuse_pages;unsigned int old_block_size; +swap_free_notify_fn *swap_free_notify_fn; };struct swap_list_t {@@ -297,6 +301,18 @@ extern int reuse_swap_page(struct page *);extern int try_to_free_swap (struct page *);struct backing_dev_info ; +static inline int +blkdev_swapon (struct block_device *bdev,struct swap_info_struct *sis) +{ +struct gendisk *disk = bdev->bd_disk; +int err =0 ; + +if (disk->fops->swapon) + err = disk->fops->swapon (bdev, sis); + +return err; +} +/* linux/mm/thrash.c */ extern struct mm_struct *swap_token_mm;extern void grab_swap_token (struct mm_struct *);mm/swapfile.c +2
-0@@ -585,6 +585,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; +if (p->swap_free_notify_fn) + p->swap_free_notify_fn (p->bdev, offset); }if (!swap_count (count))mem_cgroup_uncharge_swap (ent);
mm/swapfile.c SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · line 1855
Quoted 9 lines
mm/swapfile.c +3
-0@@ -1845,6 +1847,9 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)if (error<0 ) goto bad_swap; p->bdev = bdev; + error =blkdev_swapon (bdev, p); +if (error<0 ) + goto bad_swap; }else if (S_ISREG (inode->i_mode)) { p->bdev = inode->i_sb->s_bdev;mutex_lock (&inode->i_mutex);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Hi Nitin,
Quoted 4 lines · Nitin Gupta
It is understood that this swap notify callback is bit of a hack. I think
we will not gain much trying to beautify this hack. However, I agree with
Hugh's suggestion to rename this notify callback related function/variables
to make it explicit that its completely ramzswap related. I will send a path that affects these renames as reply to patch 0/4.
I don't quite agree and do think that my approach is a better long-term solution. That said, it's Hugh's call, not mine. Hugh?
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Hi Pekka,
Quoted 9 lines · Pekka Enberg
Quoted 6 lines · Nitin Gupta
It is understood that this swap notify callback is bit of a hack. I think we will not gain much trying to beautify this hack. However, I agree with Hugh's suggestion to rename this notify callback related function/variables
to make it explicit that its completely ramzswap related. I will send a path that affects these renames as reply to patch 0/4.
Quoted 3 lines
I don't quite agree and do think that my approach is a better long-term solution. That said, it's Hugh's call, not mine. Hugh?
Ok, lets discard all this. I will soon start working on a generic notifier based interface for various swap events: swapon, swapoff, swap slot free that I hope would be more acceptable. I will now surely miss this merge window but I hope the end result would be better.
The issue of swap_lock is still bugging me but I think atomic notifier list should be acceptable for swap slot free event, at least for the initial revision. If this particular event finds more users then we will have to work on reducing contention on swap_lock (per-swap file lock?).
Thanks,
Nitin
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 2 lines · Nitin Gupta
Ok, lets discard all this.
I think we don't have the right infrastructure even for discard yet ;)
Quoted 4 lines
I will soon start working on a generic notifier based
interface for various swap events: swapon, swapoff, swap slot free that I hope would be more acceptable. I will now surely miss this merge window but I hope the end result would be better.
Don't worry about missing the merge window, I think it was already missed. But beware of overdesign: who will be using these notifiers than you?
That's a rhetorical question: I just don't want you expending your time on something fancy, then see it rejected next time around. If you can convince that what you come up with will be generally useful, great, it'll go in; but I'm not interested in window dressing.
Quoted 5 lines
The issue of
swap_lockis still bugging me but I think atomic notifier list should be acceptable for swap slot free event, at least for the initial revision. If this particular event finds more users then we will have to work on reducing contention onswap_lock(per-swap file lock?).
The swap_lock in swap_free is troubling, and argues strongly against pretending a general interface; but I quite understand your difficulty in doing without it, it was awkward when I tried to discard near there.
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
In reply to Pekka EnbergQuoted 9 lines · Pekka Enberg
Quoted 5 lines · Nitin Gupta
It is understood that this swap notify callback is bit of a hack. I think we will not gain much trying to beautify this hack. However, I agree with Hugh's suggestion to rename this notify callback related function/variables to make it explicit that its completely ramzswap related. I will send a path that affects these renames as reply to patch 0/4.
I don't quite agree and do think that my approach is a better long-term solution. That said, it's Hugh's call, not mine. Hugh?
Sorry, Pekka, I do prefer Nitin's more explicit hackery.
Yours of course looks nicer: but again this method is actually just for the one single use, and it is "exporting" the swap_info_struct to the block device, whereas I'd prefer to move in the opposite direction, making that struct internal to swapfile.c. (I'd have done so already, but noticed TuxOnIce making use of it, and don't want to make life awkward there.)
Is the main basis for your disgust at the way that Nitin installs the callback, that loop down the swap_info_structs? I should point out that it was I who imposed that on Nitin: before that he was passing a swap entry (or was it a swap type extracted from a swap entry? I forget), which was the sole reference to a swp_entry_t in his driver - I advised a bdev interface.
Would a compromise be to extend the #ifdef CONFIG_HIBERNATION around swap_type_of() to cover ramzswap too, then Nitin use swap_type_of() on his bdev to get a swap type to use to install the notifier?
I'm not saying that would be better, haven't even thought through if it works: I'm just looking for a compromise, whereby you and I don't keep sending Nitin scurrying off in opposite directions.
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Hi Hugh,
Quoted 6 lines · Hugh Dickins
Is the main basis for your disgust at the way that Nitin installs the callback, that loop down the
swap_info_structs? I should point out that it was I who imposed that on Nitin: before that he was passing a swap entry (or was it a swap type extracted from a swap entry? I forget), which was the sole reference to aswp_entry_tin his driver - I advised a bdev interface.
The callback setup from ->read() just looks gross. However, it's your call Hugh so I'll just shut up now. ;-)
Pekka
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 10 lines · Pekka Enberg
Quoted 6 lines · Hugh Dickins
Is the main basis for your disgust at the way that Nitin installs the callback, that loop down the
swap_info_structs? I should point out that it was I who imposed that on Nitin: before that he was passing a swap entry (or was it a swap type extracted from a swap entry? I forget), which was the sole reference to aswp_entry_tin his driver - I advised a bdev interface.The callback setup from ->read() just looks gross. However, it's your call Hugh so I'll just shut up now. ;-)
Ah, no, please don't! So it's _that_ end of it that's upsetting you, and rightly so, I hadn't grasped that.
I must have glanced at that end, setting the notifier in ramzswap_read(), in a previous revision, to have spotted the swp_entry_t that was there; but haven't refreshed my memory of it recently.
(Nitin, your patch division is quite wrong: you should present a patch in which your driver works, albeit poorly, without the notifier; then a patch in which the notifier is added at the swapfile.c end and your driver end, so we can see how they fit together.)
I'd naively hoped when suggesting the bdev interface that it would then be usable at opening time, but I guess we don't know enough then.
I don't think installing the notifier at ramzswap_read() time quite covers all cases: though it's a exceptional, imagine writing some stuff out to swap, then swapoff called while all those pages are still in swapcache - no reads would be done, the swap would be freed, but the notifier never even installed, let alone called.
Well, it remains the case that I don't have time to review this at present. But when I get back here I ought to take another look at your patch (if it's not superseded by something obviously better from one or another).
Though exporting the swap_info_struct still bothers me, and it seems convoluted that the block device should have a method, so swapon can call the block device, so the block device can call swapfile.c to install a callout, so that swapfile.c can call the block device when freeing swap. I'm not saying there is a better way, just that I'd be glad of a better way.
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 5 lines · Hugh Dickins
Quoted 2 lines
The callback setup from ->read() just looks gross. However, it's your call Hugh so I'll just shut up now. ;-)
Ah, no, please don't! So it's _that_ end of it that's upsetting you, and rightly so, I hadn't grasped that.
Heh, right. :-)
Quoted 4 lines · Hugh Dickins
(Nitin, your patch division is quite wrong: you should present a patch in which your driver works, albeit poorly, without the notifier; then a patch in which the notifier is added at the
swapfile.cend and your driver end, so we can see how they fit together.)
Yes, please. Such standalone driver could go into drivers/staging for 2.6.32 probably.
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 4 lines · Pekka Enberg
On Mon, Sep 21, 2009 at 2:55 PM, Hugh Dickins
On Mon, Sep 21, 2009 at 2:55 PM, Hugh Dickins
<[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
(Nitin, your patch division is quite wrong: you should present a patch in which your driver works, albeit poorly, without the notifier; then a patch in which the notifier is added at the
swapfile.cend and your driver end, so we can see how they fit together.)Yes, please. Such standalone driver could go into
drivers/stagingfor 2.6.32 probably.
Ok, for now all I will remove all swap notifier bits from the patches so they reside completely in drivers/staging/. That should make it easier for GregKH to merge it. The problematic swap notifier part will be added later.
Thanks to you and Hugh for looking into these patches.
Nitin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
In reply to Hugh DickinsHi Hugh,
Quoted 6 lines · Hugh Dickins
Though exporting the
swap_info_structstill bothers me, and it seems convoluted that the block device should have a method, so swapon can call the block device, so the block device can callswapfile.cto install a callout, so thatswapfile.ccan call the block device when freeing swap. I'm not saying there is a better way, just that I'd be glad of a better way.
I guess we can combine my ->swapon() hook in struct block_device_operations with Nitin's set_swap_free_notify() function to avoid exporting struct swap_info_struct. Alternatively, we could add the ->swap_free() hook too struct block_device_operations. In any case, I don't think we can do the setup at sys_open() if we keep Nitin's hook as struct swap_info_struct is not set up until sys_swapon().
Pekka
Re: [PATCH 2/4] send callback when swap slot is freed
Quoted 11 lines · Pekka Enberg
Hi Hugh,
Quoted 6 lines · Hugh Dickins
Though exporting the
swap_info_structstill bothers me, and it seems convoluted that the block device should have a method, so swapon can call the block device, so the block device can callswapfile.cto install a callout, so thatswapfile.ccan call the block device when freeing swap. I'm not saying there is a better way, just that I'd be glad of a better way.
Quoted 9 lines
I guess we can combine my ->swapon() hook in struct
block_device_operationswith Nitin'sset_swap_free_notify()function to avoid exporting structswap_info_struct. Alternatively, we could add the ->swap_free() hook too structblock_device_operations. In any case, I don't think we can do the setup atsys_open()if we keep Nitin's hook as structswap_info_structis not set up untilsys_swapon().
I just converted all this to use more generic notifier interface and tested it by adding ramzswap notifier to list and it all works. I will shortly send RFC patches for swapfile.c changes and corresponding ramzswap changes to show its usage.
I'm still not sure if these notifiers will find more users, making all this look like a mere decoration. But, at least it looks better than whatever hack we currently have :)
(I will do better patch division when I post the next revision).
Thanks,
Nitin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
In reply to Hugh DickinsQuoted 22 lines · Hugh Dickins
Quoted 12 lines · Pekka Enberg
Quoted 11 lines · Nitin Gupta
mm/swapfile.c +2
-0+EXPORT_SYMBOL_GPL (set_swap_free_notify); +static int
swap_entry_free(struct swap_info_struct *p, swp_entry_t ent, int cache){mm/swapfile.c +2
-0@@ -585,6 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; + if (p->swap_free_notify_fn) + p->swap_free_notify_fn(p->bdev, offset);Quoted 8 lines
mm/swapfile.c static int swap_entry_free(struct swap_info_struct *p, · line 624
Quoted 3 lines
mm/swapfile.c +0
-0} if (!swap_count(count)) mem_cgroup_uncharge_swap(ent);OK, this hits core kernel code so we need to CC some more
mm/swapfile.cpeople. Theset_swap_free_notify()API looks strange to me. Hugh, I think you mentioned that you're okay with an explicit hook. Any suggestions how to do this cleanly?
Quoted 9 lines
No, no better suggestion. I quite see Nitin's point that ramzswap would benefit significantly from a callback here, though it's not a place (holding
swap_lock) where we'd like to offer a callback at all.I think I would prefer the naming to make it absolutely clear that it's a special for ramzswap or compcache, rather than dressing it up in the grand generality of a
swap_free_notify_fn: giving our hacks fancy names doesn't really make them better.
Yes, makes sense... Since we cannot afford to have a chain of callbacks within a spin lock, we have to keep it ramzswap specific (and rename functions/variables to reflect this).
set_ramzswap_free_notify_fn() -> set_ramzswap_free_notify_fn() andswap_free_notify_fn -> ramzswap_free_notify_fn
Now, this renaming exposes ugliness of this hack in its true sense. Currently, I don't have a cleaner solution but few points to consider:
- If we really have to do this within the lock then there cannot be
multiple callbacks.
It has to then remain ramzswap specific. In that case, current patch looks looks like
the simplest solution.
- Do we really have to have a callback within a spin lock? Things become
very complex in ramzswap driver if we try to do this outside the lock (I attempted
this but couldn't get it working). Still, we should think about it.
- If it can be done outside lock, we can afford to be chain of
callbacks attached
to this event. A nice generic solution. But if this means delaying callback for too long,
then it may be unacceptable for ramzswap (we come back to problem with discard approach).
Quoted 3 lines
(Does the bdev matching work out if there are any regular swapfiles around? I've not checked, might or might not need refinement there.)
Yes.
Thanks,
Nitin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
In reply to Hugh DickinsQuoted 22 lines · Hugh Dickins
Quoted 12 lines · Pekka Enberg
Quoted 11 lines · Nitin Gupta
(quoted diff) +4
-0+EXPORT_SYMBOL_GPL(set_swap_free_notify); + static int swap_entry_free(struct swap_info_struct *p, swp_entry_t ent, int cache) {@@ -585,6 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; + if (p->swap_free_notify_fn) + p->swap_free_notify_fn(p->bdev, offset);Quoted 8 lines
Review static int swap_entry_free(struct swap_info_struct *p, · line 624
Quoted 3 lines
(quoted diff) +0
-0} if (!swap_count(count)) mem_cgroup_uncharge_swap(ent);OK, this hits core kernel code so we need to CC some more
mm/swapfile.cpeople. Theset_swap_free_notify()API looks strange to me. Hugh, I think you mentioned that you're okay with an explicit hook. Any suggestions how to do this cleanly?
Quoted 9 lines
No, no better suggestion. I quite see Nitin's point that ramzswap would benefit significantly from a callback here, though it's not a place (holding
swap_lock) where we'd like to offer a callback at all.I think I would prefer the naming to make it absolutely clear that it's a special for ramzswap or compcache, rather than dressing it up in the grand generality of a
swap_free_notify_fn: giving our hacks fancy names doesn't really make them better.
One more thing. If this renaming is done, then I think this notify callback should no longer be unconditionally compiled. It should depend on some ramzswap specific symbol.
Do you think you will be able to Ack this swap notify patch if above things are done (rest of the driver is aimed at staging)? For your reference, below is the patch to do this. I think I will have to send "v4" patches that will include this one and related changes in ramzswap driver code.
(patch to make ramzswap notify callback conditional)
---
drivers/staging/ramzswap/Kconfig +5 -0
@@ -19,3 +19,8 @@ config RAMZSWAP_STATS help Enable statistics collection for ramzswap. This adds only a minimal overhead. In unsure, say Y. + +config RAMZSWAP_NOTIFY + bool + depends on RAMZSWAP + default y
include/linux/swap.h +7 -0
@@ -299,8 +299,15 @@ extern sector_t swapdev_block(int, pgoff_t);extern struct swap_info_struct *get_swap_info_struct (unsigned );extern int reuse_swap_page (struct page *);extern int try_to_free_swap (struct page *); +#ifdef CONFIG_RAMZSWAP_NOTIFY extern void set_ramzswap_free_notify (struct block_device *,ramzswap_free_notify_fn *); +#else +void set_ramzswap_free_notify (struct block_device *bdev, +ramzswap_free_notify_fn *notify_fn) +{ +} +#endif struct backing_dev_info ;/* linux/mm/thrash.c */
mm/swapfile.c +2 -0
@@ -554,6 +554,7 @@ out:return NULL ; } +#ifdef CONFIG_RAMZSWAP_NOTIFY /* * Sets callback for event when swap_map[offset] == 0 * i.e. page at this swap offset is no longer used.
@@ -585,6 +586,7 @@ void set_ramzswap_free_notify(struct block_device *bdev, return; }EXPORT_SYMBOL_GPL (set_ramzswap_free_notify); +#endif static int swap_entry_free (struct swap_info_struct *p,swp_entry_t ent,int cache)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/4] send callback when swap slot is freed
In reply to Nitin Guptamm/swapfile.c static int swap_entry_free(struct swap_info_struct *p, · line 624
Quoted 9 lines · On Fri, 18 Sep 2009 04:13:30 +0530 Nitin Gupta <[email protected]> wrote:
mm/swapfile.c +2
-0@@ -585,6 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; +if (p->swap_free_notify_fn) + p->swap_free_notify_fn (p->bdev, offset); }if (!swap_count (count))mem_cgroup_uncharge_swap (ent);
A nitpick but I feel I have to explain why mem_cgroup_ucharge_swap() is called here. (difference with p->swap_free_notify_fn)
if (!swap_count(count))
means "It seems no users for this swap entry but we're not sure there are
SwapCache for this entry or not."
In mem_cgroup_uncharge_swap(), swap_cgroup is checked and if there is a record (which means the SwapCache is not mapped anywhere), swap usage is uncharged.
This is for race window at freeing swap entry viazap_pte_range() => free_swap_and_cache().
(swap entry is not freed if the page is locked.)
I'll add some explanation in next series of memcg-cleanup patches.
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/4 virtual block device driver (ramzswap)
In reply to Nitin GuptaCreates RAM based block devices (/dev/ramzswapX) which can be used (only) as swap disks. Pages swapped to these are compressed and stored in memory itself.
The module is called ramzswap.ko. It depends on:
- xvmalloc memory allocator (compiled with this driver)
- lzo_compress.ko
- lzo_decompress.ko
See ramzswap.txt for usage details.
--- drivers/staging/Kconfig | 2 + drivers/staging/Makefile | 1 + drivers/staging/ramzswap/Kconfig | 21 + drivers/staging/ramzswap/Makefile | 3 + drivers/staging/ramzswap/ramzswap_drv.c | 1462 +++++++++++++++++++++++++++++ drivers/staging/ramzswap/ramzswap_drv.h | 173 ++++ drivers/staging/ramzswap/ramzswap_ioctl.h | 50 + 7 files changed, 1712 insertions(+), 0 deletions(-)
drivers/staging/Kconfig +2 -0
@@ -131,5 +131,7 @@ source "drivers/staging/iio/Kconfig" source "drivers/staging/cowloop/Kconfig" +source "drivers/staging/ramzswap/Kconfig" + endif # !STAGING_EXCLUDE_BUILD endif # STAGING
drivers/staging/Makefile +1 -0
@@ -47,3 +47,4 @@ obj-$(CONFIG_RAR_REGISTER) += rar/ obj-$(CONFIG_DX_SEP) += sep/ obj-$(CONFIG_IIO) += iio/ obj-$(CONFIG_COWLOOP) += cowloop/ +obj-$(CONFIG_RAMZSWAP) += ramzswap/
drivers/staging/ramzswap/Kconfig new file +21 -0
@@ -0,0 +1,21 @@ +config RAMZSWAP + tristate "Compressed in-memory swap device (ramzswap)" + depends on SWAP + select LZO_COMPRESS + select LZO_DECOMPRESS + default n + help + Creates virtual block devices which can be used (only) as a swap + disks. Pages swapped to these disks are compressed and stored in + memory itself. + + See ramzswap.txt for more information. + Project home: http://compcache.googlecode.com/ + +config RAMZSWAP_STATS + bool "Enable ramzswap stats" + depends on RAMZSWAP + default y + help + Enable statistics collection for ramzswap. This adds only a minimal + overhead. In unsure, say Y.
drivers/staging/ramzswap/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +ramzswap-objs := ramzswap_drv.o xvmalloc.o + +obj-$(CONFIG_RAMZSWAP) += ramzswap.o
drivers/staging/ramzswap/ramzswap_drv.c new file +1462 -0
1,462 lines not on this page · raw message
drivers/staging/ramzswap/ramzswap_drv.h new file +173 -0
173 lines not on this page · raw message
drivers/staging/ramzswap/ramzswap_ioctl.h new file +50 -0
50 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 3/4] virtual block device driver (ramzswap)
Quoted 6 lines · Nitin Gupta
(...)
drivers/staging/ramzswap/ramzswap_drv.c new file +5
-0+ +static int page_zero_filled (void *ptr) +{ +u32 pos; +u64 *page;
drivers/staging/ramzswap/ramzswap_drv.c line 81
Quoted 10 lines
drivers/staging/ramzswap/ramzswap_drv.c new file +10
-0+ + page = (u64 *)ptr; + +for (pos =0 ; pos !=PAGE_SIZE /sizeof (*page); pos++) { +if (page[pos]) +return 0 ; + } + +return 1 ; +}
Wouldn't unsigned long *page be better for both 32-bit and 64-bit machines?
(This function could return bool)
Quoted 33 lines
(...)
drivers/staging/ramzswap/ramzswap_drv.c new file +32
-0+static void create_device (struct ramzswap *rzs,int device_id) +{ +mutex_init (&rzs->lock); +INIT_LIST_HEAD (&rzs->backing_swap_extent_list); + + rzs->queue =blk_alloc_queue (GFP_KERNEL ); +if (!rzs->queue) { +pr_err ("Error allocating disk queue for device %d\n" , + device_id); +return ; + } + +blk_queue_make_request (rzs->queue, ramzswap_make_request); + rzs->queue->queuedata = rzs; + +/* gendisk structure */ + rzs->disk =alloc_disk (1 ); +if (!rzs->disk) { +blk_cleanup_queue (rzs->queue); +pr_warning ("Error allocating disk structure for device %d\n" , + device_id); +return ; + } + + rzs->disk->major = ramzswap_major; + rzs->disk->first_minor = device_id; + rzs->disk->fops = &ramzswap_devops; + rzs->disk->queue = rzs->queue; + rzs->disk->private_data = rzs; +snprintf (rzs->disk->disk_name,16 ,"ramzswap%d" , device_id); + +/*
drivers/staging/ramzswap/ramzswap_drv.c line 1385
Quoted 10 lines
drivers/staging/ramzswap/ramzswap_drv.c new file +10
-0+* Actual capacity set using RZSIO_SET_DISKSIZE_KB ioctl +* or set equal to backing swap device (if provided) +*/ +set_capacity (rzs->disk,0 ); +add_disk (rzs->disk); + + rzs->init_done =0 ; + +return ; +}
needless return
Marcin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/4 documentation
In reply to Nitin GuptaShort guide on how to setup and use ramzswap.
--- drivers/staging/ramzswap/ramzswap.txt | 51 +++++++++++++++++++++++++++++++++ 1 files changed, 51 insertions(+), 0 deletions(-)
drivers/staging/ramzswap/ramzswap.txt new file +51 -0
@@ -0,0 +1,51 @@ +ramzswap: Compressed RAM based swap device +------------------------------------------- + +Project home: http://compcache.googlecode.com/ + +* Introduction + +It creates RAM based block devices which can be used (only) as swap disks. +Pages swapped to these devices are compressed and stored in memory itself. +See project home for use cases, performance numbers and a lot more. + +Individual ramzswap devices are configured and initialized using rzscontrol +userspace utility as shown in examples below. See rzscontrol man page for more +details. + +* Usage + +Following shows a typical sequence of steps for using ramzswap. + +1) Load Modules: + modprobe ramzswap num_devices=4 + This creates 4 (uninitialized) devices: /dev/ramzswap{0,1,2,3} + (num_devices parameter is optional. Default: 1) + +2) Initialize: + Use rzscontrol utility to configure and initialize individual + ramzswap devices. Example: + rzscontrol /dev/ramzswap2 --init # uses default value of disksize_kb + + *See rzscontrol man page for more details and examples* + +3) Activate: + swapon /dev/ramzswap2 # or any other initialized ramzswap device + +4) Stats: + rzscontrol /dev/ramzswap2 --stats + +5) Deactivate: + swapoff /dev/ramzswap2 + +6) Reset: + rzscontrol /dev/ramzswap2 --reset + (This frees all the memory allocated for this device). + + +Please report any problems at: + - Mailing list: linux-mm-cc at laptop dot org + - Issue tracker: http://code.google.com/p/compcache/issues/list + +Nitin Gupta [email protected]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/1 ramzswap prefix for swap free callback
In reply to Nitin GuptaSwap free callback is used to inform the underlying block device that a swap page has been freed. Currently, ramzswap is the only user of this callback. So, rename swap notify interface to reflect this.
--- drivers/staging/ramzswap/ramzswap_drv.c | 4 ++-- include/linux/swap.h | 7 ++++--- mm/swapfile.c | 14 +++++++------- 3 files changed, 13 insertions(+), 12 deletions(-)
drivers/staging/ramzswap/ramzswap_drv.c +2 -2
@@ -761,7 +761,7 @@ static int ramzswap_read(struct ramzswap *rzs, struct bio *bio) index = bio->bi_sector >>SECTORS_PER_PAGE_SHIFT ;if (unlikely (!rzs->init_notify_callback) &&PageSwapCache (page)) { -set_swap_free_notify (bio->bi_bdev, ramzswap_free_notify); +set_ramzswap_free_notify (bio->bi_bdev, ramzswap_free_notify); rzs->init_notify_callback =1 ; }
@@ -1323,7 +1323,7 @@ static int ramzswap_ioctl(struct block_device *bdev, fmode_t mode,* can happen if another swapon is done before this reset. * TODO: A callback from swapoff() will solve this problem. */ -set_swap_free_notify (bdev,NULL ); +set_ramzswap_free_notify (bdev,NULL ); rzs->init_notify_callback =0 ;break ;
include/linux/swap.h +4 -3
@@ -21,7 +21,7 @@ struct bio;#define SWAP_FLAG_PRIO_MASK 0x7fff#define SWAP_FLAG_PRIO_SHIFT 0 -typedef void (swap_free_notify_fn ) (struct block_device *,unsigned long ); +typedef void (ramzswap_free_notify_fn ) (struct block_device *,unsigned long );static inline int current_is_kswapd (void ) {
@@ -158,7 +158,7 @@ struct swap_info_struct {unsigned int max;unsigned int inuse_pages;unsigned int old_block_size; -swap_free_notify_fn *swap_free_notify_fn; +ramzswap_free_notify_fn *ramzswap_free_notify_fn; };struct swap_list_t {
@@ -299,7 +299,8 @@ extern sector_t swapdev_block(int, pgoff_t);extern struct swap_info_struct *get_swap_info_struct (unsigned );extern int reuse_swap_page (struct page *);extern int try_to_free_swap (struct page *); -extern void set_swap_free_notify (struct block_device *,swap_free_notify_fn *); +extern void set_ramzswap_free_notify (struct block_device *, +ramzswap_free_notify_fn *);struct backing_dev_info ;/* linux/mm/thrash.c */
mm/swapfile.c +7 -7
@@ -558,8 +558,8 @@ out:* Sets callback for event when swap_map[offset] == 0 * i.e. page at this swap offset is no longer used. */ -void set_swap_free_notify (struct block_device *bdev, -swap_free_notify_fn *notify_fn) +void set_ramzswap_free_notify (struct block_device *bdev, +ramzswap_free_notify_fn *notify_fn) {unsigned int i;struct swap_info_struct *sis;
@@ -579,12 +579,12 @@ void set_swap_free_notify(struct block_device *bdev,return ; } -BUG_ON (!sis || sis->swap_free_notify_fn); - sis->swap_free_notify_fn = notify_fn; +BUG_ON (!sis || sis->ramzswap_free_notify_fn); + sis->ramzswap_free_notify_fn = notify_fn;spin_unlock (&swap_lock);return ; } -EXPORT_SYMBOL_GPL (set_swap_free_notify); +EXPORT_SYMBOL_GPL (set_ramzswap_free_notify);static int swap_entry_free (struct swap_info_struct *p,swp_entry_t ent,int cache)
@@ -617,8 +617,8 @@ static int swap_entry_free(struct swap_info_struct *p, swap_list.next = p - swap_info; nr_swap_pages++; p->inuse_pages--; -if (p->swap_free_notify_fn) - p->swap_free_notify_fn (p->bdev, offset); +if (p->ramzswap_free_notify_fn) + p->ramzswap_free_notify_fn (p->bdev, offset); }if (!swap_count (count))mem_cgroup_uncharge_swap (ent);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>