z3fold: the 3-fold allocator for compressed pages
Subsystems (2)
PATCH v2 1/1 z3fold: the 3-fold allocator for compressed pages
This patch introduces z3fold, a special purpose allocator for storing compressed pages. It is designed to store up to three compressed pages per physical page. It is a ZBUD derivative which allows for higher compression ratio keeping the simplicity and determinism of its predecessor.
The main differences between z3fold and zbud are:
* unlike zbud, z3fold allows for up to PAGE_SIZE allocations
* z3fold can hold up to 3 compressed pages in its page
This patch comes as a follow-up to the discussions at the Embedded Linux Conference in San-Diego related to the talk [1]. The outcome of these discussions was that it would be good to have a compressed page allocator as stable and deterministic as zbud with with higher compression ratio.
To keep the determinism and simplicity, z3fold, just like zbud, always stores an integral number of compressed pages per page, but it can store up to 3 pages unlike zbud which can store at most 2. Therefore the compression ratio goes to around 2.5x while zbud's one is around 1.7x.
The patch is based on the latest linux.git tree.
This version of the patch has updates related to various concurrency fixes made after intensive testing on SMP/HMP platforms.
--- mm/Kconfig | 9 + mm/Makefile | 1 + mm/z3fold.c | 806 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 816 insertions(+) create mode 100644 mm/z3fold.c
mm/Kconfig +9 -0
@@ -565,6 +565,15 @@ config ZBUD deterministic reclaim properties that make it preferable to a higher density approach when reclaim will be used. +config Z3FOLD + tristate "Low density storage for compressed pages" + default n + help + A special purpose allocator for storing compressed pages. + It is designed to store up to three compressed pages per physical + page. It is a ZBUD derivative so the simplicity and determinism are + still there. + config ZSMALLOC tristate "Memory allocator for compressed pages" depends on MMU
mm/Makefile +1 -0
@@ -89,6 +89,7 @@ obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o obj-$(CONFIG_ZPOOL) += zpool.o obj-$(CONFIG_ZBUD) += zbud.o obj-$(CONFIG_ZSMALLOC) += zsmalloc.o +obj-$(CONFIG_Z3FOLD) += z3fold.o obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += early_ioremap.o obj-$(CONFIG_CMA) += cma.o obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o
mm/z3fold.c new file +806 -0
806 lines not on this page · raw message
Signature · 7 lines
2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerQuoted 19 lines · On Tue, 19 Apr 2016 11:48:45 +0200 Vitaly Wool <[email protected]> wrote:
This patch introduces z3fold, a special purpose allocator for storing compressed pages. It is designed to store up to three compressed pages per physical page. It is a ZBUD derivative which allows for higher compression ratio keeping the simplicity and determinism of its predecessor.
The main differences between z3fold and zbud are:
* unlike zbud, z3fold allows for up toPAGE_SIZEallocations
* z3fold can hold up to 3 compressed pages in its pageThis patch comes as a follow-up to the discussions at the Embedded Linux Conference in San-Diego related to the talk [1]. The outcome of these discussions was that it would be good to have a compressed page allocator as stable and deterministic as zbud with with higher compression ratio.
To keep the determinism and simplicity, z3fold, just like zbud, always stores an integral number of compressed pages per page, but it can store up to 3 pages unlike zbud which can store at most 2. Therefore the compression ratio goes to around 2.5x while zbud's one is around 1.7x.
Quoted 7 lines
The patch is based on the latest linux.git tree.
This version of the patch has updates related to various concurrency fixes made after intensive testing on SMP/HMP platforms.
So... why don't we just replace zbud with z3fold? (Update the changelog to answer this rather obvious question, please!)
Are there performance (ie speed) differences? (Ditto).
There's no documentation. zbud is covered a bit in Documentation/vm/zswap.txt. Maybe there, if appropriate. Decent end-user documentation is notably absent.
The code does stuff whether or not CONFG_ZPOOL is enabled. Let's cover both scenarios in that documentation please.
mm/Kconfig config ZBUD · line 569
Quoted 8 lines
mm/Kconfig +2
-0@@ -565,6 +565,15 @@ config ZBUD deterministic reclaim properties that make it preferable to a higher density approach when reclaim will be used. +config Z3FOLD + tristate "Low density storage for compressed pages"
I don't really understand what "low density" means here. I'd have thought it was "high density" if anything.
Quoted 15 lines
mm/Kconfig +7
-0+ default n + help + A special purpose allocator for storing compressed pages. + It is designed to store up to three compressed pages per physical + page. It is a ZBUD derivative so the simplicity and determinism are + still there. + config ZSMALLOC tristate "Memory allocator for compressed pages" depends on MMUmm/Makefile +0
-0
mm/Makefile obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o · line 93
Quoted 5 lines
mm/Makefile +1
-0@@ -89,6 +89,7 @@ obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o obj-$(CONFIG_ZPOOL) += zpool.o obj-$(CONFIG_ZBUD) += zbud.o obj-$(CONFIG_ZSMALLOC) += zsmalloc.o +obj-$(CONFIG_Z3FOLD) += z3fold.o obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += early_ioremap.o
Something is going on with your email client. Space-stuffing.
Quoted 8 lines
mm/Makefile +0
-0obj-$(CONFIG_CMA) += cma.o obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.omm/z3fold.c new file +0
-0
mm/z3fold.c line 10
Quoted 10 lines
mm/z3fold.c new file +10
-0@@ -0,0 +1,806 @@ +/* +* z3fold.c +* +* Copyright (C) 2016, Vitaly Wool <[email protected]> +* +* This implementation is heavily based on zbud written by Seth Jennings. +* +* z3fold is an special purpose allocator for storing compressed pages. It +* can store up to three compressed pages per page which improves the +* compression ratio of zbud while pertaining its concept and simplicity.
s/pertaining/retaining/
mm/z3fold.c line 12
Quoted 2 lines
mm/z3fold.c new file +2
-0+* It still has simple and deterministic reclaim properties that make it +* preferable to a higher density approach when reclaim is used.
Again, what's does a "higher density approach" mean?
mm/z3fold.c line 15
Quoted 2 lines
mm/z3fold.c new file +2
-0+* As in zbud, pages are divided into "chunks". The size of the chunks is +* fixed at compile time and determined by NCHUNKS_ORDER below.
s/and/and is/
mm/z3fold.c line 29
Quoted 13 lines
mm/z3fold.c new file +13
-0+* The z3fold API doesn't differ from zbud API and zpool is also supported. +*/ + +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt + +#include <linux/atomic.h> +#include <linux/list.h> +#include <linux/mm.h> +#include <linux/module.h> +#include <linux/preempt.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <linux/zpool.h>
The code itself looks nice. It seems to compile so I'll put it in there for a bit of exposure.
checkpatch complains a lot - please take a look at that. These:
WARNING: externs should be avoided in .c files
#137: FILE: mm/z3fold.c:60:
+int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp,
WARNING: externs should be avoided in .c files
#139: FILE: mm/z3fold.c:62:
+void z3fold_free(struct z3fold_pool *pool, unsigned long handle);
WARNING: externs should be avoided in .c files
#140: FILE: mm/z3fold.c:63:
+int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries);
WARNING: externs should be avoided in .c files
#142: FILE: mm/z3fold.c:65:
+void z3fold_unmap(struct z3fold_pool *pool, unsigned long handle);
are weird. If the symbol is exported to other .c files then it should be declared in a header. If not, it should be static.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Tue, 19 Apr 2016 11:48:45 +0200 Vitaly Wool <[email protected]> wrote:
This patch introduces z3fold, a special purpose allocator for storing compressed pages. It is designed to store up to three compressed pages per physical page. It is a ZBUD derivative which allows for higher compression ratio keeping the simplicity and determinism of its predecessor.
The main differences between z3fold and zbud are:
* unlike zbud, z3fold allows for up toPAGE_SIZEallocations
* z3fold can hold up to 3 compressed pages in its pageThis patch comes as a follow-up to the discussions at the Embedded Linux Conference in San-Diego related to the talk [1]. The outcome of these discussions was that it would be good to have a compressed page allocator as stable and deterministic as zbud with with higher compression ratio.
To keep the determinism and simplicity, z3fold, just like zbud, always stores an integral number of compressed pages per page, but it can store up to 3 pages unlike zbud which can store at most 2. Therefore the compression ratio goes to around 2.5x while zbud's one is around 1.7x.
The patch is based on the latest linux.git tree.
Quoted 8 lines
Quoted 5 lines
This version of the patch has updates related to various concurrency fixes made after intensive testing on SMP/HMP platforms.
So... why don't we just replace zbud with z3fold? (Update the changelog to answer this rather obvious question, please!)
There was discussion between Seth and Vitaly on v1. Without me knowing the details myself, it looked like Seth's objections were addressed, but then the thread died. I think there should first be a more clear answer from Seth whether z3fold really looks like a clear win (i.e. not workload-dependent) over zbud, in which case zbud could be extended?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 40 lines · Vlastimil Babka
Quoted 32 lines · Andrew Morton
Quoted 28 lines · On Tue, 19 Apr 2016 11:48:45 +0200 Vitaly Wool <[email protected]> wrote:
This patch introduces z3fold, a special purpose allocator for storing compressed pages. It is designed to store up to three compressed pages per
physical page. It is a ZBUD derivative which allows for higher compression
ratio keeping the simplicity and determinism of its predecessor.The main differences between z3fold and zbud are:
* unlike zbud, z3fold allows for up toPAGE_SIZEallocations
* z3fold can hold up to 3 compressed pages in its pageThis patch comes as a follow-up to the discussions at the Embedded Linux Conference in San-Diego related to the talk [1]. The outcome of these discussions was that it would be good to have a compressed page allocator as stable and deterministic as zbud with with higher compression ratio.
To keep the determinism and simplicity, z3fold, just like zbud, always stores an integral number of compressed pages per page, but it can store up to 3 pages unlike zbud which can store at most 2. Therefore the compression ratio goes to around 2.5x while zbud's one is around 1.7x.
The patch is based on the latest linux.git tree.
This version of the patch has updates related to various concurrency fixes
made after intensive testing on SMP/HMP platforms.Quoted 5 lines
Quoted 2 lines
So... why don't we just replace zbud with z3fold? (Update the changelog to answer this rather obvious question, please!)
Quoted 5 lines
There was discussion between Seth and Vitaly on v1. Without me knowing the details myself, it looked like Seth's objections were addressed, but then the thread died. I think there should first be a more clear answer from Seth whether z3fold really looks like a clear win (i.e. not workload-dependent) over zbud, in which case zbud could be extended?
(sorry for the dup Vlastimil, didn't reply-to-all)
It seems like it could be in the case that most of the pages in your system compress to 1/3 their original size (on average). In my original research, I found that, using lzo, 1/2 a page was more typical. However, if you used deflate, you might be able to push the average down.
IMO I do think we should try to merge zbud and z3fold with zbud being the default mode (2 object per page) and have an option to enable the 3 objects per page logic. IIRC that 3rd object logic seemed to be fairly contained. Having the separate would duplicate a lot of very similar code.
However, if Andrew is ok with yet another z- allocator, it can just be another zpool backend. I'm fine either way. Just my two cents.
Seth
Quoted 1 line
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Vlastimil Babka
On 04/22/2016 01:22 AM, Andrew Morton wrote:
<snip>
Quoted 10 lines
Quoted 3 lines
So... why don't we just replace zbud with z3fold? (Update the changelog to answer this rather obvious question, please!)
There was discussion between Seth and Vitaly on v1. Without me knowing the details myself, it looked like Seth's objections were addressed, but then the thread died. I think there should first be a more clear answer from Seth whether z3fold really looks like a clear win (i.e. not workload-dependent) over zbud, in which case zbud could be extended?
I have tried to address this question in the changelog for v3 which came out today. Basically I'd like to play on the safe side and have z3fold co-existing with
zbud for a while, since zbud is a simple and proven solution which has less object code and can work without ZPOOL. The original zbud implementation doesn't use struct page's fields in any way, which z3fold can't get away without.
As a matter of fact, I don't think there's much of the similar code left between zbud and z3fold, other than the generic structure.
~vitaly
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>