zram/zsmalloc promotion

This patch moves zsmalloc under zram directory because there isn't any other user any more.

Before that, description will explain why we have needed custom allocator.

Zsmalloc is a new slab-based memory allocator for storing compressed pages. It is designed for low fragmentation and high allocation success rate on large object, but <= PAGE_SIZE allocations.

Quoted 10 lines · Minchan Kim

This patch moves zsmalloc under zram directory because there isn't any other user any more.

Before that, description will explain why we have needed custom allocator.

Zsmalloc is a new slab-based memory allocator for storing compressed pages. It is designed for low fragmentation and high allocation success rate on large object, but <= PAGE_SIZE allocations.

One things zsmalloc will probably have to address before Andrew deems it worthy is the "memmap peekers" issue. I had to make this change in zbud before Andrew would accept it and this is one of the reasons I have yet to implement zsmalloc support for zswap yet.

Basically, zsmalloc makes the assumption that once the kernel page allocator gives it a page for the pool, zsmalloc can stuff whatever metatdata it wants into the struct page. The problem comes when some parts of the kernel do not obtain the struct page pointer via the allocator but via walking the memmap. Those routines will make certain assumption about the state and structure of the data in the struct page, leading to issues.

My solution for zbud was to move the metadata into the pool pages themselves, using the first block of each page for metadata regarding that page.

Andrew might also have something to say about the placement of zsmalloc.c. IIRC, if it was going to be merged, he wanted it in mm/ if it was going to be messing around in the struct page.

Seth

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It's 6th trial of zram/zsmalloc promotion.
[patch 5, zram: promote zram from staging] explains why we need zram.

Main reason to block promotion is there was no review of zsmalloc part while Jens already acked zram part.

At that time, zsmalloc was used for zram, zcache and zswap so everybody wanted to make it general and at last, Mel reviewed it.
Most of review was related to zswap dumping mechanism which can pageout compressed page into swap in runtime and zswap gives up using zsmalloc and invented a new wheel, zbud. Other reviews were not major.
http://lkml.indiana.edu/hypermail/linux/kernel/1304.1/04334.html

Zcache don't use zsmalloc either so only zsmalloc user is zram now. So I think there is no concern any more.

Patch 1 adds new Kconfig for zram to use page table method instead of copy. Andrew suggested it.

Patch 2 adds lots of commnt for zsmalloc.

Patch 3 moves zsmalloc under driver/staging/zram because zram is only user for zram now.

Patch 4 makes unmap_kernel_range exportable function because zsmalloc have used map_vm_area which is already exported function but zsmalloc need to use unmap_kernel_range and it should be built with module.

Patch 5 moves zram from driver/staging to driver/blocks, finally.

It touches mm, staging, blocks so I am not sure who is right position maintainer so I will Cc Andrw, Jens and Greg.

This patch is based on next-20130813.

Thanks.

Minchan Kim (4):
zsmalloc: add Kconfig for enabling page table method
zsmalloc: move it under zram
mm: export unmap_kernel_range
zram: promote zram from staging

Nitin Cupta (1):
zsmalloc: add more comment

 drivers/block/Kconfig                    |    2 +
 drivers/block/Makefile                   |    1 +
 drivers/block/zram/Kconfig               |   37 +
 drivers/block/zram/Makefile              |    3 +
 drivers/block/zram/zram.txt              |   71 ++
 drivers/block/zram/zram_drv.c            |  987 +++++++++++++++++++++++++++
 drivers/block/zram/zsmalloc.c            | 1084 ++++++++++++++++++++++++++++++
 drivers/staging/Kconfig                  |    4 -
 drivers/staging/Makefile                 |    2 -
 drivers/staging/zram/Kconfig             |   25 -
 drivers/staging/zram/Makefile            |    3 -
 drivers/staging/zram/zram.txt            |   77 ---
 drivers/staging/zram/zram_drv.c          |  984 ---------------------------
 drivers/staging/zram/zram_drv.h          |  125 ----
 drivers/staging/zsmalloc/Kconfig         |   10 -
 drivers/staging/zsmalloc/Makefile        |    3 -
 drivers/staging/zsmalloc/zsmalloc-main.c | 1063 -----------------------------
 drivers/staging/zsmalloc/zsmalloc.h      |   43 --
 include/linux/zram.h                     |  123 ++++
 include/linux/zsmalloc.h                 |   52 ++
 mm/vmalloc.c                             |    1 +
 21 files changed, 2361 insertions(+), 2339 deletions(-)
 create mode 100644 drivers/block/zram/Kconfig
 create mode 100644 drivers/block/zram/Makefile
 create mode 100644 drivers/block/zram/zram.txt
 create mode 100644 drivers/block/zram/zram_drv.c
 create mode 100644 drivers/block/zram/zsmalloc.c
 delete mode 100644 drivers/staging/zram/Kconfig
 delete mode 100644 drivers/staging/zram/Makefile
 delete mode 100644 drivers/staging/zram/zram.txt
 delete mode 100644 drivers/staging/zram/zram_drv.c
 delete mode 100644 drivers/staging/zram/zram_drv.h
 delete mode 100644 drivers/staging/zsmalloc/Kconfig
 delete mode 100644 drivers/staging/zsmalloc/Makefile
 delete mode 100644 drivers/staging/zsmalloc/zsmalloc-main.c
 delete mode 100644 drivers/staging/zsmalloc/zsmalloc.h
 create mode 100644 include/linux/zram.h
 create mode 100644 include/linux/zsmalloc.h

--
1.7.9.5

During earlier discussions of zswap there was a plan to make it work with zsmalloc as an option instead of zbud. Does zbud work for

compression factors better than 2:1? I have the impression (maybe wrong) that it does not. In our use of zram (Chrome OS) typical

overall compression ratios are between 2.5:1 and 3:1. We would hate to waste that memory if we switch to zswap.

Hi Luigi,

Quoted 2 lines · Luigi Semenzato

During earlier discussions of zswap there was a plan to make it work with zsmalloc as an option instead of zbud. Does zbud work for

AFAIR, it was not an optoin but zsmalloc was must but there were several objections because zswap's notable feature is to dump compressed object to real swap storage. For that, zswap needs to store bounded objects in a zpage so that dumping could be bounded, too. Otherwise, it could encounter OOM easily.

Quoted 12 lines · Minchan Kim

Hi Luigi,

Quoted 2 lines · Luigi Semenzato

During earlier discussions of zswap there was a plan to make it work with zsmalloc as an option instead of zbud. Does zbud work for

AFAIR, it was not an optoin but zsmalloc was must but there were several objections because zswap's notable feature is to dump compressed object to real swap storage. For that, zswap needs to store bounded objects in a zpage so that dumping could be bounded, too. Otherwise, it could encounter OOM easily.

AFAIR, the next step of zswap should be have a modular allocation layer so that users can choose zsmalloc or zbud to use.

Seth?

Hi Luigi,

Quoted 2 lines · Luigi Semenzato

During earlier discussions of zswap there was a plan to make it work with zsmalloc as an option instead of zbud. Does zbud work for

AFAIR, it was not an optoin but zsmalloc was must but there were several objections because zswap's notable feature is to dump compressed object to real swap storage. For that, zswap needs to store bounded objects in a zpage so that dumping could be bounded, too. Otherwise, it could encounter OOM easily.

Quoted 2 lines

compression factors better than 2:1? I have the impression (maybe wrong) that it does not. In our use of zram (Chrome OS) typical

Since zswap changed allocator from zsmalloc to zbud, I didn't follow because I had no interest of low compressoin ratio allocator so I have no idea of status of zswap at a moment but I guess it would be still 2:1.

Quoted 2 lines

overall compression ratios are between 2.5:1 and 3:1. We would hate to waste that memory if we switch to zswap.

If you have real swap storage, zswap might be better although I have no number but real swap is money for embedded system and it has sudden garbage collection on firmware side if we use eMMC or SSD so that it could affect system latency. Morever, if we start to use real swap, maybe we should encrypt the data and it would be severe overhead(CPU and Power).

And what I am considering after promoting for zram feature is asynchronous I/O and it's possible because zram is block device.

Thanks!

Signature · 8 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Luigi,

Quoted 2 lines · Luigi Semenzato

During earlier discussions of zswap there was a plan to make it work with zsmalloc as an option instead of zbud. Does zbud work for

AFAIR, it was not an optoin but zsmalloc was must but there were several objections because zswap's notable feature is to dump compressed object to real swap storage. For that, zswap needs to store bounded objects in a zpage so that dumping could be bounded, too. Otherwise, it could encounter OOM easily.

Quoted 2 lines

compression factors better than 2:1? I have the impression (maybe wrong) that it does not. In our use of zram (Chrome OS) typical

Since zswap changed allocator from zsmalloc to zbud, I didn't follow because I had no interest of low compressoin ratio allocator so I have no idea of status of zswap at a moment but I guess it would be still 2:1.

Quoted 2 lines

overall compression ratios are between 2.5:1 and 3:1. We would hate to waste that memory if we switch to zswap.

If you have real swap storage, zswap might be better although I have no number but real swap is money for embedded system and it has sudden garbage collection on firmware side if we use eMMC or SSD so that it could affect system latency. Morever, if we start to use real swap, maybe we should encrypt the data and it would be severe overhead(CPU and Power).

And what I am considering after promoting for zram feature is asynchronous I/O and it's possible because zram is block device.

Thanks!

Signature · 8 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Minchan,

Quoted 23 lines · Minchan Kim

Hi Luigi,

Quoted 2 lines · Luigi Semenzato

During earlier discussions of zswap there was a plan to make it work with zsmalloc as an option instead of zbud. Does zbud work for

AFAIR, it was not an optoin but zsmalloc was must but there were several objections because zswap's notable feature is to dump compressed object to real swap storage. For that, zswap needs to store bounded objects in a zpage so that dumping could be bounded, too. Otherwise, it could encounter OOM easily.

Quoted 2 lines

compression factors better than 2:1? I have the impression (maybe wrong) that it does not. In our use of zram (Chrome OS) typical

Since zswap changed allocator from zsmalloc to zbud, I didn't follow because I had no interest of low compressoin ratio allocator so I have no idea of status of zswap at a moment but I guess it would be still 2:1.

Quoted 2 lines

overall compression ratios are between 2.5:1 and 3:1. We would hate to waste that memory if we switch to zswap.

Quoted 7 lines

If you have real swap storage, zswap might be better although I have no number but real swap is money for embedded system and it has sudden garbage collection on firmware side if we use eMMC or SSD so that it could affect system latency. Morever, if we start to use real swap, maybe we should encrypt the data and it would be severe overhead(CPU and Power).

Why real swap for embedded system need encrypt the data? I think there is no encrypt for data against server and desktop.

Quoted 14 lines

And what I am considering after promoting for zram feature is asynchronous I/O and it's possible because zram is block device.

Thanks!

Signature · 8 lines

Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi,

Quoted 13 lines · Wanpeng Li

Hi Minchan,

Quoted 10 lines · Minchan Kim

Hi Luigi,

On Wed, Aug 14, 2013 at 08:53:31AM -0700, Luigi Semenzato wrote:
If you have real swap storage, zswap might be better although I have no number but real swap is money for embedded system and it has sudden garbage collection on firmware side if we use eMMC or SSD so that it could affect system latency. Morever, if we start to use real swap, maybe we should encrypt the data and it would be severe overhead(CPU and Power).

Quoted 2 lines

Why real swap for embedded system need encrypt the data? I think there is no encrypt for data against server and desktop.

I have used some portable device but suddenly, I lost it or was stolen. A hacker can pick it up and read my swap and found my precious information. I don't want it. I guess it's one of reason ChromeOS don't want to use real swap.

https://groups.google.com/a/chromium.org/forum/#!msg/chromium-os-discuss/92Fvi4Ezego/ZvbrC3L2FG4J

It's 6th trial of zram/zsmalloc promotion.
[patch 5, zram: promote zram from staging] explains why we need zram.

Main reason to block promotion is there was no review of zsmalloc part while Jens already acked zram part.

At that time, zsmalloc was used for zram, zcache and zswap so everybody wanted to make it general and at last, Mel reviewed it.
Most of review was related to zswap dumping mechanism which can pageout compressed page into swap in runtime and zswap gives up using zsmalloc and invented a new wheel, zbud. Other reviews were not major.
http://lkml.indiana.edu/hypermail/linux/kernel/1304.1/04334.html

Quoted 6 lines · Minchan Kim

It's 6th trial of zram/zsmalloc promotion.
[patch 5, zram: promote zram from staging] explains why we need zram.

Main reason to block promotion is there was no review of zsmalloc part while Jens already acked zram part.

Quoted 7 lines

At that time, zsmalloc was used for zram, zcache and zswap so everybody wanted to make it general and at last, Mel reviewed it.
Most of review was related to zswap dumping mechanism which can pageout compressed page into swap in runtime and zswap gives up using zsmalloc and invented a new wheel, zbud. Other reviews were not major.
http://lkml.indiana.edu/hypermail/linux/kernel/1304.1/04334.html

zsmalloc has unpredictable performance characteristics when reclaiming a single page when it was used to back zswap. I felt the unpredictable performance characteristics would make it close to impossible to support for normal server workloads. It would appear to work well until there were massive stalls and I do not think this was ever properly investigated. At one point I would have been happy if zsmalloc could be tuned to store only store 2 compressed pages per physical page but cannot remember why that proposal was never implemented (or if it was and I missed it or forgot). I expected it would change over time but there were no follow-ups that I'm aware of.

I do not believe this is a problem for zram as such because I do not think it ever writes back to disk and is immune from the unpredictable performance characteristics problem. The problem for zram using zsmalloc is OOM killing. If it's used for swap then there is no guarantee that killing processes frees memory and that could result in an OOM storm. Of course there is no guarantee that memory is freed with zbud either but you are guaranteed that freeing 50%+1 of the compressed pages will free a single physical page. The characteristics for zsmalloc are much more severe. This might be managable in an applicance with very careful control of the applications that are running but not for general servers or desktops.

If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I do not know why this never happened but guess the zswap people never were interested and the zram people never tried.

Why was the pseudo device
to avoid writebacks never implemented? Why was the underlying allocator not made pluggable to optionally use zsmalloc when the user did not care that it had terrible writeback characteristics?

zswap cannot replicate zram+tmpfs but I also think that such a configuration is a bad idea anyway. As zram is already being deployed then it might get promoted anyway but personally I think compressed memory continues to be a tragic story.

--
Mel Gorman
SUSE Labs

Quoted 6 lines · Minchan Kim

It's 6th trial of zram/zsmalloc promotion.
[patch 5, zram: promote zram from staging] explains why we need zram.

Main reason to block promotion is there was no review of zsmalloc part while Jens already acked zram part.

Quoted 7 lines

At that time, zsmalloc was used for zram, zcache and zswap so everybody wanted to make it general and at last, Mel reviewed it.
Most of review was related to zswap dumping mechanism which can pageout compressed page into swap in runtime and zswap gives up using zsmalloc and invented a new wheel, zbud. Other reviews were not major.
http://lkml.indiana.edu/hypermail/linux/kernel/1304.1/04334.html

zsmalloc has unpredictable performance characteristics when reclaiming a single page when it was used to back zswap. I felt the unpredictable performance characteristics would make it close to impossible to support for normal server workloads. It would appear to work well until there were massive stalls and I do not think this was ever properly investigated. At one point I would have been happy if zsmalloc could be tuned to store only store 2 compressed pages per physical page but cannot remember why that proposal was never implemented (or if it was and I missed it or forgot). I expected it would change over time but there were no follow-ups that I'm aware of.

I do not believe this is a problem for zram as such because I do not think it ever writes back to disk and is immune from the unpredictable performance characteristics problem. The problem for zram using zsmalloc is OOM killing. If it's used for swap then there is no guarantee that killing processes frees memory and that could result in an OOM storm. Of course there is no guarantee that memory is freed with zbud either but you are guaranteed that freeing 50%+1 of the compressed pages will free a single physical page. The characteristics for zsmalloc are much more severe. This might be managable in an applicance with very careful control of the applications that are running but not for general servers or desktops.

If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

do not know why this never happened but guess the zswap people never were interested and the zram people never tried. Why was the pseudo device to avoid writebacks never implemented? Why was the underlying allocator not made pluggable to optionally use zsmalloc when the user did not care that it had terrible writeback characteristics?

zswap cannot replicate zram+tmpfs but I also think that such a configuration is a bad idea anyway. As zram is already being deployed then it might get

promoted anyway but personally I think compressed memory continues to be

a tragic story.

Signature · 8 lines

Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 12 lines

<SNIP>

I do not believe this is a problem for zram as such because I do not think it ever writes back to disk and is immune from the unpredictable performance characteristics problem. The problem for zram using zsmalloc is OOM killing. If it's used for swap then there is no guarantee that killing processes frees memory and that could result in an OOM storm. Of course there is no guarantee that memory is freed with zbud either but you are guaranteed that freeing 50%+1 of the compressed pages will free a single physical page. The characteristics for zsmalloc are much more severe. This might be managable in an applicance with very careful control of the applications that are running but not for general servers or desktops.

Fair enough but let's think of current usecase for zram.
As I said in description, most of user for zram are embedded products. So, most of them has no swap storage and hate OOM kill because OOM is already very very slow path so system slow response is really thing we want to avoid. We prefer early process kill to slow response. That's why custom low memory killer/notifier is popular in embedded side. so actually, OOM storm problem shouldn't be a big problem under well-control limited system.

Quoted 8 lines

If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 5 lines

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

One of difference between zswap and zram is asynchronous I/O support.

I guess frontswap is synchronous by semantic while zram could support asynchronous I/O.

Quoted 5 lines

do not know why this never happened but guess the zswap people never were interested and the zram people never tried. Why was the pseudo device to avoid writebacks never implemented? Why was the underlying allocator not made pluggable to optionally use zsmalloc when the user did not care that it had terrible writeback characteristics?

I remember you suggested to make zsmalloc with pluggable for zswap. But I don't know why zswap people didn't implement it.

Quoted 1 line

Quoted 2 lines

zswap cannot replicate zram+tmpfs but I also think that such a configuration is a bad idea anyway. As zram is already being deployed then it might get

It seems your big concern of zsmalloc is fragmentaion so if zsmalloc can support compaction, it would mitigate the concern.

Quoted 1 line

promoted anyway but personally I think compressed memory continues to be

I admit zram might have limitations but it has helped lots of people. It's not an imaginary scenario.

Please, let's not do get out of zram from kernel tree and stall it on staging forever with preventing new features.
Please, let's promote, expose it to more potential users, receive more complains from them, recruit more contributors and let's enhance.

Quoted 13 lines · Minchan Kim

Quoted 12 lines

<SNIP>

I do not believe this is a problem for zram as such because I do not think it ever writes back to disk and is immune from the unpredictable performance characteristics problem. The problem for zram using zsmalloc is OOM killing. If it's used for swap then there is no guarantee that killing processes frees memory and that could result in an OOM storm. Of course there is no guarantee that memory is freed with zbud either but you are guaranteed that freeing 50%+1 of the compressed pages will free a single physical page. The characteristics for zsmalloc are much more severe. This might be managable in an applicance with very careful control of the applications that are running but not for general servers or desktops.

Quoted 9 lines

Fair enough but let's think of current usecase for zram.
As I said in description, most of user for zram are embedded products. So, most of them has no swap storage and hate OOM kill because OOM is already very very slow path so system slow response is really thing we want to avoid. We prefer early process kill to slow response. That's why custom low memory killer/notifier is popular in embedded side. so actually, OOM storm problem shouldn't be a big problem under well-control limited system.

Which zswap could also do if

a) it had a pseudo block device that failed all writes
b) zsmalloc was pluggable

I recognise this sucks because zram is already in the field but if zram is promoted then zram and zswap will continue to diverge further with no reconcilation in sight.

Part of the point of using zswap was that potentially zcache could be implemented on top of it and so all file cache could be stored compressed in memory. AFAIK, it's not possible to do the same thing for zram because of the lack of writeback capabilities. Maybe it could be done if zram could be configured to write to an underlying storage device but it may be very clumsy to configure. I don't know as I never investigated it and to be honest, I'm struggling to remember how I got involved anywhere near zswap/zcache/zram/zwtf in the first place.

Quoted 9 lines

Quoted 8 lines

If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Quoted 7 lines

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

And is zsmalloc was pluggable for zswap then it would also benefit.

Quoted 7 lines

Quoted 5 lines

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

One of difference between zswap and zram is asynchronous I/O support.

As zram is not writing to disk, how compelling is asynchronous IO? If zswap was backed by the pseudo device is there a measurable bottleneck?

Quoted 13 lines

I guess frontswap is synchronous by semantic while zram could support asynchronous I/O.

Quoted 5 lines

do not know why this never happened but guess the zswap people never were interested and the zram people never tried. Why was the pseudo device to avoid writebacks never implemented? Why was the underlying allocator not made pluggable to optionally use zsmalloc when the user did not care that it had terrible writeback characteristics?

I remember you suggested to make zsmalloc with pluggable for zswap. But I don't know why zswap people didn't implement it.

Quoted 1 line

Quoted 6 lines

Quoted 2 lines

zswap cannot replicate zram+tmpfs but I also think that such a configuration is a bad idea anyway. As zram is already being deployed then it might get

It seems your big concern of zsmalloc is fragmentaion so if zsmalloc can support compaction, it would mitigate the concern.

Even if it supported zsmalloc I would still wonder why zswap is not using it as a pluggable option :(

Quoted 5 lines

Quoted 1 line

promoted anyway but personally I think compressed memory continues to be

I admit zram might have limitations but it has helped lots of people. It's not an imaginary scenario.

I know.

Quoted 5 lines

Please, let's not do get out of zram from kernel tree and stall it on staging forever with preventing new features.
Please, let's promote, expose it to more potential users, receive more complains from them, recruit more contributors and let's enhance.

As this is already used heavily in the field and I am not responsible for maintaining it I am not going to object to it being promoted. I can always push that it be disabled in distribution configs as it is not suitable for general workloads for reason already discussed.

However, I believe that the promotion will lead to zram and zswap diverging further from each other, both implementing similar functionality and ultimately cause greater maintenance headaches. There is a path that makes

zswap a functional replacement for zram and I've seen no good reason why that path was not taken. Zram cannot be a functional replacment for zswap as there is no obvious sane way writeback could be implemented. Continuing to diverge will ultimately bite someone in the ass.

Quoted 13 lines · Minchan Kim

Quoted 12 lines

<SNIP>

I do not believe this is a problem for zram as such because I do not think it ever writes back to disk and is immune from the unpredictable performance characteristics problem. The problem for zram using zsmalloc is OOM killing. If it's used for swap then there is no guarantee that killing processes frees memory and that could result in an OOM storm. Of course there is no guarantee that memory is freed with zbud either but you are guaranteed that freeing 50%+1 of the compressed pages will free a single physical page. The characteristics for zsmalloc are much more severe. This might be managable in an applicance with very careful control of the applications that are running but not for general servers or desktops.

Quoted 9 lines

Fair enough but let's think of current usecase for zram.
As I said in description, most of user for zram are embedded products. So, most of them has no swap storage and hate OOM kill because OOM is already very very slow path so system slow response is really thing we want to avoid. We prefer early process kill to slow response. That's why custom low memory killer/notifier is popular in embedded side. so actually, OOM storm problem shouldn't be a big problem under well-control limited system.

Which zswap could also do if

a) it had a pseudo block device that failed all writes
b) zsmalloc was pluggable

I recognise this sucks because zram is already in the field but if zram is promoted then zram and zswap will continue to diverge further with no reconcilation in sight.

Part of the point of using zswap was that potentially zcache could be implemented on top of it and so all file cache could be stored compressed in memory. AFAIK, it's not possible to do the same thing for zram because of the lack of writeback capabilities. Maybe it could be done if zram could be configured to write to an underlying storage device but it may be very clumsy to configure. I don't know as I never investigated it and to be honest, I'm struggling to remember how I got involved anywhere near zswap/zcache/zram/zwtf in the first place.

Quoted 9 lines

Quoted 8 lines

If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Quoted 7 lines

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

And is zsmalloc was pluggable for zswap then it would also benefit.

Quoted 7 lines

Quoted 5 lines

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

One of difference between zswap and zram is asynchronous I/O support.

As zram is not writing to disk, how compelling is asynchronous IO? If zswap was backed by the pseudo device is there a measurable bottleneck?

Quoted 13 lines

I guess frontswap is synchronous by semantic while zram could support asynchronous I/O.

Quoted 5 lines

do not know why this never happened but guess the zswap people never were interested and the zram people never tried. Why was the pseudo device to avoid writebacks never implemented? Why was the underlying allocator not made pluggable to optionally use zsmalloc when the user did not care that it had terrible writeback characteristics?

I remember you suggested to make zsmalloc with pluggable for zswap. But I don't know why zswap people didn't implement it.

Quoted 1 line

Quoted 6 lines

Quoted 2 lines

zswap cannot replicate zram+tmpfs but I also think that such a configuration is a bad idea anyway. As zram is already being deployed then it might get

It seems your big concern of zsmalloc is fragmentaion so if zsmalloc can support compaction, it would mitigate the concern.

Even if it supported zsmalloc I would still wonder why zswap is not using it as a pluggable option :(

Quoted 5 lines

Quoted 1 line

promoted anyway but personally I think compressed memory continues to be

I admit zram might have limitations but it has helped lots of people. It's not an imaginary scenario.

I know.

Quoted 5 lines

Please, let's not do get out of zram from kernel tree and stall it on staging forever with preventing new features.
Please, let's promote, expose it to more potential users, receive more complains from them, recruit more contributors and let's enhance.

As this is already used heavily in the field and I am not responsible for maintaining it I am not going to object to it being promoted. I can always push that it be disabled in distribution configs as it is not suitable for general workloads for reason already discussed.

However, I believe that the promotion will lead to zram and zswap diverging further from each other, both implementing similar functionality and ultimately cause greater maintenance headaches. There is a path that makes zswap a functional replacement for zram and I've seen no good reason why

that path was not taken. Zram cannot be a functional replacment for zswap as there is no obvious sane way writeback could be implemented. Continuing to diverge will ultimately bite someone in the ass.

Quoted 13 lines · Minchan Kim

Quoted 12 lines

<SNIP>

I do not believe this is a problem for zram as such because I do not think it ever writes back to disk and is immune from the unpredictable performance characteristics problem. The problem for zram using zsmalloc is OOM killing. If it's used for swap then there is no guarantee that killing processes frees memory and that could result in an OOM storm. Of course there is no guarantee that memory is freed with zbud either but you are guaranteed that freeing 50%+1 of the compressed pages will free a single physical page. The characteristics for zsmalloc are much more severe. This might be managable in an applicance with very careful control of the applications that are running but not for general servers or desktops.

Quoted 9 lines

Fair enough but let's think of current usecase for zram.
As I said in description, most of user for zram are embedded products. So, most of them has no swap storage and hate OOM kill because OOM is already very very slow path so system slow response is really thing we want to avoid. We prefer early process kill to slow response. That's why custom low memory killer/notifier is popular in embedded side. so actually, OOM storm problem shouldn't be a big problem under well-control limited system.

Which zswap could also do if

a) it had a pseudo block device that failed all writes
b) zsmalloc was pluggable

I recognise this sucks because zram is already in the field but if zram is promoted then zram and zswap will continue to diverge further with no reconcilation in sight.

Part of the point of using zswap was that potentially zcache could be implemented on top of it and so all file cache could be stored compressed in memory. AFAIK, it's not possible to do the same thing for zram because of the lack of writeback capabilities. Maybe it could be done if zram could be configured to write to an underlying storage device but it may be very clumsy to configure. I don't know as I never investigated it and to be honest, I'm struggling to remember how I got involved anywhere near zswap/zcache/zram/zwtf in the first place.

Quoted 9 lines

Quoted 8 lines

If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Quoted 7 lines

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

And is zsmalloc was pluggable for zswap then it would also benefit.

Quoted 8 lines

Quoted 6 lines

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

One of difference between zswap and zram is asynchronous I/O support.

As zram is not writing to disk, how compelling is asynchronous IO? If zswap was backed by the pseudo device is there a measurable bottleneck?

Quoted 14 lines

I guess frontswap is synchronous by semantic while zram could support asynchronous I/O.

Quoted 5 lines

do not know why this never happened but guess the zswap people never were interested and the zram people never tried. Why was the pseudo device to avoid writebacks never implemented? Why was the underlying allocator not made pluggable to optionally use zsmalloc when the user did not care that it had terrible writeback characteristics?

I remember you suggested to make zsmalloc with pluggable for zswap. But I don't know why zswap people didn't implement it.

Quoted 1 line

Quoted 6 lines

Quoted 2 lines

zswap cannot replicate zram+tmpfs but I also think that such a configuration is a bad idea anyway. As zram is already being deployed then it might get

It seems your big concern of zsmalloc is fragmentaion so if zsmalloc can support compaction, it would mitigate the concern.

Even if it supported zsmalloc I would still wonder why zswap is not using it as a pluggable option :(

Quoted 5 lines

Quoted 1 line

promoted anyway but personally I think compressed memory continues to be

I admit zram might have limitations but it has helped lots of people. It's not an imaginary scenario.

I know.

Quoted 5 lines

Please, let's not do get out of zram from kernel tree and stall it on staging forever with preventing new features.
Please, let's promote, expose it to more potential users, receive more complains from them, recruit more contributors and let's enhance.

As this is already used heavily in the field and I am not responsible for maintaining it I am not going to object to it being promoted. I can always push that it be disabled in distribution configs as it is not suitable for general workloads for reason already discussed.

However, I believe that the promotion will lead to zram and zswap diverging further from each other, both implementing similar functionality and ultimately cause greater maintenance headaches. There is a path that makes zswap a functional replacement for zram and I've seen no good reason why that path was not taken. Zram cannot be a functional replacment for zswap as there is no obvious sane way writeback could be implemented. Continuing

to diverge will ultimately bite someone in the ass.

Signature · 3 lines

-- 
Mel Gorman
SUSE Labs

Hi Mel,

Quoted 59 lines · Mel Gorman

Quoted 13 lines · Minchan Kim

Quoted 12 lines

<SNIP>

I do not believe this is a problem for zram as such because I do not think it ever writes back to disk and is immune from the unpredictable performance characteristics problem. The problem for zram using zsmalloc is OOM killing. If it's used for swap then there is no guarantee that killing processes frees memory and that could result in an OOM storm. Of course there is no guarantee that memory is freed with zbud either but you are guaranteed that freeing 50%+1 of the compressed pages will free a single physical page. The characteristics for zsmalloc are much more severe. This might be managable in an applicance with very careful control of the applications that are running but not for general servers or desktops.

Quoted 9 lines

Fair enough but let's think of current usecase for zram.
As I said in description, most of user for zram are embedded products. So, most of them has no swap storage and hate OOM kill because OOM is already very very slow path so system slow response is really thing we want to avoid. We prefer early process kill to slow response. That's why custom low memory killer/notifier is popular in embedded side. so actually, OOM storm problem shouldn't be a big problem under well-control limited system.

Which zswap could also do if

a) it had a pseudo block device that failed all writes
b) zsmalloc was pluggable

I recognise this sucks because zram is already in the field but if zram is promoted then zram and zswap will continue to diverge further with no reconcilation in sight.

Part of the point of using zswap was that potentially zcache could be implemented on top of it and so all file cache could be stored compressed in memory. AFAIK, it's not possible to do the same thing for zram because of the lack of writeback capabilities. Maybe it could be done if zram could be configured to write to an underlying storage device but it may be very clumsy to configure. I don't know as I never investigated it and to be honest, I'm struggling to remember how I got involved anywhere near zswap/zcache/zram/zwtf in the first place.

Quoted 16 lines

Quoted 8 lines

If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 1 line

And is zsmalloc was pluggable for zswap then it would also benefit.

But zswap isn't pseudo block device so it couldn't be used for block device. Let say one usecase for using zram-blk.

1) Many embedded system don't have swap so although tmpfs can support swapout it's pointless still so such systems should have sane configuration to limit memory space so it's not only zram problem.

2) Many embedded system don't have enough memory. Let's assume short-lived file growing up until half of system memory once in a while. We don't want to write it on flash by wear-leveing issue and very slowness so we want to use in-memory but if we uses tmpfs, it should evict half of working set to cover them when the size reach peak. zram would be better choice.

Quoted 11 lines

Quoted 8 lines

Quoted 6 lines

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

One of difference between zswap and zram is asynchronous I/O support.

As zram is not writing to disk, how compelling is asynchronous IO? If zswap was backed by the pseudo device is there a measurable bottleneck?

Compression. It was really bottlneck point. I had an internal patch which can make zram use various compressor, not only LZO.
The better good compressor was, the more bottlenck compressor was.

Quoted 43 lines

Quoted 14 lines

I guess frontswap is synchronous by semantic while zram could support asynchronous I/O.

Quoted 5 lines

do not know why this never happened but guess the zswap people never were interested and the zram people never tried. Why was the pseudo device to avoid writebacks never implemented? Why was the underlying allocator not made pluggable to optionally use zsmalloc when the user did not care that it had terrible writeback characteristics?

I remember you suggested to make zsmalloc with pluggable for zswap. But I don't know why zswap people didn't implement it.

Quoted 1 line

Quoted 6 lines

Quoted 2 lines

zswap cannot replicate zram+tmpfs but I also think that such a configuration is a bad idea anyway. As zram is already being deployed then it might get

It seems your big concern of zsmalloc is fragmentaion so if zsmalloc can support compaction, it would mitigate the concern.

Even if it supported zsmalloc I would still wonder why zswap is not using it as a pluggable option :(

Quoted 5 lines

Quoted 1 line

promoted anyway but personally I think compressed memory continues to be

I admit zram might have limitations but it has helped lots of people. It's not an imaginary scenario.

I know.

Quoted 5 lines

Please, let's not do get out of zram from kernel tree and stall it on staging forever with preventing new features.
Please, let's promote, expose it to more potential users, receive more complains from them, recruit more contributors and let's enhance.

As this is already used heavily in the field and I am not responsible for maintaining it I am not going to object to it being promoted. I can always push that it be disabled in distribution configs as it is not suitable for general workloads for reason already discussed.

Quoted 6 lines

However, I believe that the promotion will lead to zram and zswap diverging further from each other, both implementing similar functionality and ultimately cause greater maintenance headaches. There is a path that makes zswap a functional replacement for zram and I've seen no good reason why that path was not taken. Zram cannot be a functional replacment for zswap as there is no obvious sane way writeback could be implemented. Continuing

Then, do you think current zswap's writeback is sane way?
I didn't raise an issue because I didn't want to be a blocker when zswap was promoted. Actually, I didn't like that way because I thought swap-writeback feature should be implemented by VM itself rather than some hooked driver internal logic. VM alreay has a lot information so it would handle multipe heterogenous swap more efficenlty like cache hierachy without LRU inversing. It could solve current zswap LRU inversing problem generally and help others who want to configure multiple swap system as well as zram.

Quoted 1 line

to diverge will ultimately bite someone in the ass.

Mel, current zram situation is following as.

1) There are a lot users in the world.
2) So, many valuable contributions have been in there.
2) The new feature development of zram had stalled because Greg asserted he doesn't accept new feature until promote will be done and recently, he said he will remove zram in staging if anybody doesn't try to promote
3) You are saying zram shouldn't be promote. IOW, zram should go away.

Right? Then, What should we zram developers do?

What's next step for zram which is really perfect for embedded system? We should really lose a chance to enhance zram although fresh zswap couldn't replace old zram?

Mel, please consider embedded world although they are very little voice in this core subsystem.

Quoted 18 lines

Quoted 17 lines

Quoted 9 lines

<SNIP>
If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 1 line

And is zsmalloc was pluggable for zswap then it would also benefit.

But zswap isn't pseudo block device so it couldn't be used for block device.

Let say one usecase for using zram-blk.

1) Many embedded system don't have swap so although tmpfs can support swapout it's pointless still so such systems should have sane configuration to limit memory space so it's not only zram problem.

2) Many embedded system don't have enough memory. Let's assume short-lived file growing up until half of system memory once in a while. We don't want to write it on flash by wear-leveing issue and very slowness so we want to use in-memory but if we uses tmpfs, it should evict half of working set to cover them when the size reach peak. zram would be better choice.

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

Quoted 4 lines

Quoted 1 line

One of difference between zswap and zram is asynchronous I/O support.

As zram is not writing to disk, how compelling is asynchronous IO? If zswap was backed by the pseudo device is there a measurable bottleneck?

Compression. It was really bottlneck point. I had an internal patch which can make zram use various compressor, not only LZO.
The better good compressor was, the more bottlenck compressor was.

Quoted 6 lines

However, I believe that the promotion will lead to zram and zswap diverging further from each other, both implementing similar functionality and ultimately cause greater maintenance headaches. There is a path that makes zswap a functional replacement for zram and I've seen no good reason why that path was not taken. Zram cannot be a functional replacment for zswap as there is no obvious sane way writeback could be implemented. Continuing

Then, do you think current zswap's writeback is sane way?

I didn't raise an issue because I didn't want to be a blocker when zswap was promoted. Actually, I didn't like that way because I thought swap-writeback feature should be implemented by VM itself rather than some hooked driver internal logic.

VM alreay has a lot information so it would handle multipe heterogenous swap more efficenlty like cache hierachy without LRU inversing. It could solve current zswap LRU inversing problem generally and help others who want to configure multiple swap system as well as zram.

Quoted 1 line

to diverge will ultimately bite someone in the ass.

Mel, current zram situation is following as.

1) There are a lot users in the world.
2) So, many valuable contributions have been in there.
2) The new feature development of zram had stalled because Greg asserted he doesn't accept new feature until promote will be done and recently, he said he will remove zram in staging if anybody doesn't try to promote
3) You are saying zram shouldn't be promote. IOW, zram should go away.

Right? Then, What should we zram developers do?

What's next step for zram which is really perfect for embedded system? We should really lose a chance to enhance zram although fresh zswap couldn't replace old zram?

Mel, please consider embedded world although they are very little voice in this core subsystem.

On Fri, Aug 16, 2013 at 01:26:41PM +0900, Minchan Kim wrote:
<SNIP>
It'll get even more entertaining if/when someone ever tries to reimplement zcache although since Dan left I do not believe anyone is planning to try.

Quoted 5 lines · Mel Gorman

On Fri, Aug 16, 2013 at 01:26:41PM +0900, Minchan Kim wrote:
<SNIP>
It'll get even more entertaining if/when someone ever tries to reimplement zcache although since Dan left I do not believe anyone is planning to try.

I should mention that Bob Liu did some work with zcache recently but is now looking like it'll be dropped from staging. I did not look at the details and I have no idea if anything else is planned with it.

Hi Mel,

Quoted 7 lines · Mel Gorman

Quoted 5 lines · Mel Gorman

On Fri, Aug 16, 2013 at 01:26:41PM +0900, Minchan Kim wrote:
<SNIP>
It'll get even more entertaining if/when someone ever tries to reimplement zcache although since Dan left I do not believe anyone is planning to try.

Quoted 4 lines

I should mention that Bob Liu did some work with zcache recently but is now looking like it'll be dropped from staging. I did not look at the details and I have no idea if anything else is planned with it.

The plan is like this:
Zcache dropped from staging.

I already said I recognise it has a large number of users in the field and users count a lot more than me complaining. If it gets promoted then I expect it will be on those grounds.

My position is that I think it's a bad idea because it is clear there is no plan or intention of ever brining zram and zswap together. Instead we are to have two features providing similar functionality with zram diverging further from zswap. Ultimately I believe this will increase maintenance headaches. It'll get even more entertaining if/when someone ever tries to reimplement zcache although since Dan left I do not believe anyone is planning to try. I will not be acking this series but there many be enough

developers that are actually willing to maintain a duel zram/zswap mess to make it happen anyway.

Quoted 18 lines · Minchan Kim

Quoted 18 lines

Quoted 17 lines

Quoted 9 lines

<SNIP>
If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 3 lines

Quoted 1 line

And is zsmalloc was pluggable for zswap then it would also benefit.

But zswap isn't pseudo block device so it couldn't be used for block device.

It would not be impossible to write one. Taking a quick look it might even be doable by just providing a zbud_ops that does not have an evict handler and make sure the errors are handled correctly. i.e. does the following patch mean that zswap never writes back and instead just compresses pages in memory?

mm/zswap.c +0 -1

index deda2b6..99e41c8 100644
@@ -819,7 +819,6 @@ static void zswap_frontswap_invalidate_area(unsigned type)
 }
 
 static struct zbud_ops zswap_zbud_ops = {
-	.evict = zswap_writeback_entry
 };
 
 static void zswap_frontswap_init(unsigned type)

If so, it should be doable to link that up in a sane way so it can be configured at runtime.

Did you ever even try something like this?

Quoted 7 lines

Let say one usecase for using zram-blk.

1) Many embedded system don't have swap so although tmpfs can support swapout it's pointless still so such systems should have sane configuration to limit memory space so it's not only zram problem.

If zswap was backed by a pseudo device that failed all writes or an an ops with no evict handler then it would be functionally similar.

Quoted 6 lines

2) Many embedded system don't have enough memory. Let's assume short-lived file growing up until half of system memory once in a while. We don't want to write it on flash by wear-leveing issue and very slowness so we want to use in-memory but if we uses tmpfs, it should evict half of working set to cover them when the size reach peak. zram would be better choice.

Then back it by a pseudo device that fails all writes so it does not have to write to disk.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

These are the sort of reason why I feel that zram has limited cases where it is safe to use and zswap has a wider range of applications. At least I would be very unhappy to try supporting zram in the field for normal servers. zswap should be able to replace the functionality of zram+swap by backing zswap with a pseudo block device that rejects all writes. I

Quoted 9 lines

Quoted 4 lines

Quoted 1 line

One of difference between zswap and zram is asynchronous I/O support.

As zram is not writing to disk, how compelling is asynchronous IO? If zswap was backed by the pseudo device is there a measurable bottleneck?

Compression. It was really bottlneck point. I had an internal patch which can make zram use various compressor, not only LZO.
The better good compressor was, the more bottlenck compressor was.

There are two issues there. One that different compression algorithms should be optional with tradeoffs on speed vs compression ratio. There is no reason why that couldn't be hacked into zswap.

The second is that only one page can be compressed at a time. That would require further work to allow the frontswap API to asynchronously compress pages. It would be a lot more heavy lifting but it is not impossible.

Quoted 9 lines

Quoted 7 lines

However, I believe that the promotion will lead to zram and zswap diverging further from each other, both implementing similar functionality and ultimately cause greater maintenance headaches. There is a path that makes zswap a functional replacement for zram and I've seen no good reason why that path was not taken. Zram cannot be a functional replacment for zswap as there is no obvious sane way writeback could be implemented. Continuing

Then, do you think current zswap's writeback is sane way?

No, it's clunky as hell and the layering between zswap and zbud is twisty (e.g. zswap store -> zbud reclaim -> zswap writeback wtf?). I believe it used to be a lot worse but was ironed out a bit in preparation for merging. As bad as it is, general workloads cannot just consume unreclaimable pages with compressed data and writeback should be optionally handled at the very least.

How zswap currently implements it could be a whole lot better. It's silly that it is the allocator the directly performs synchronous writeback one page at a time because that will means the world stalls when zswap fills. On larger machines that is just going to be brick wall and considering that zswap was intended for virtualisation it is particularly hilarious. I think I brought up its stalling behaviour during review when it was being merged. It would have been preferable if writeback could be initiated in batches and then waited on at the very least. It's worse that it uses _swap_writepage directly instead of going through a writepage ops. It would have been better if zbud pages existed on the LRU and written back with an address space ops and properly handled asynchonous writeback.

Zswap could be massively improved, there is no denying that. I've seen no follow-up patches since which is a bit worrying but I'm not losing sleep over it.

Zram does not even try to do anything like this and from your description of the embedded use case there is no intention of ever trying.

Quoted 5 lines

I didn't raise an issue because I didn't want to be a blocker when zswap was promoted. Actually, I didn't like that way because I thought swap-writeback feature should be implemented by VM itself rather than some hooked driver internal logic.

You could also have brought it up any time since or pushed for it to be implemented with the view to making zswap functionally equivalent to zram.

Quoted 10 lines

VM alreay has a lot information so it would handle multipe heterogenous swap more efficenlty like cache hierachy without LRU inversing. It could solve current zswap LRU inversing problem generally and help others who want to configure multiple swap system as well as zram.

Quoted 1 line

to diverge will ultimately bite someone in the ass.

Mel, current zram situation is following as.

Quoted 8 lines

1) There are a lot users in the world.
2) So, many valuable contributions have been in there.
2) The new feature development of zram had stalled because Greg asserted he doesn't accept new feature until promote will be done and recently, he said he will remove zram in staging if anybody doesn't try to promote
3) You are saying zram shouldn't be promote. IOW, zram should go away.

Right? Then, What should we zram developers do?

I've already explained, more than once going at least as far back as April, how I thought zswap could be made functionally identical to zram and improved.

Quoted 7 lines

What's next step for zram which is really perfect for embedded system? We should really lose a chance to enhance zram although fresh zswap couldn't replace old zram?

Mel, please consider embedded world although they are very little voice in this core subsystem.

I already said I recognise it has a large number of users in the field and users count a lot more than me complaining. If it gets promoted then I expect it will be on those grounds.

My position is that I think it's a bad idea because it is clear there is no plan or intention of ever brining zram and zswap together. Instead we are to have two features providing similar functionality with zram diverging further from zswap. Ultimately I believe this will increase maintenance headaches. It'll get even more entertaining if/when someone ever tries to reimplement zcache although since Dan left I do not believe anyone is planning to try. I will not be acking this series but there many be enough developers that are actually willing to maintain a duel zram/zswap mess to make it happen anyway.

Hello Mel,

Quoted 42 lines · Mel Gorman

Quoted 18 lines · Minchan Kim

Quoted 18 lines

Quoted 17 lines

Quoted 9 lines

<SNIP>
If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 3 lines

Quoted 1 line

And is zsmalloc was pluggable for zswap then it would also benefit.

But zswap isn't pseudo block device so it couldn't be used for block device.

It would not be impossible to write one. Taking a quick look it might even be doable by just providing a zbud_ops that does not have an evict handler and make sure the errors are handled correctly. i.e. does the following patch mean that zswap never writes back and instead just compresses pages in memory?

mm/zswap.c +0 -1

index deda2b6..99e41c8 100644
@@ -819,7 +819,6 @@ static void zswap_frontswap_invalidate_area(unsigned type)
 }
 
 static struct zbud_ops zswap_zbud_ops = {
-	.evict = zswap_writeback_entry
 };
 
 static void zswap_frontswap_init(unsigned type)

Quoted 4 lines

If so, it should be doable to link that up in a sane way so it can be configured at runtime.

Did you ever even try something like this?

Never. Because I didn't have such requirement for zram.

Quoted 18 lines

Quoted 7 lines

Let say one usecase for using zram-blk.

1) Many embedded system don't have swap so although tmpfs can support swapout it's pointless still so such systems should have sane configuration to limit memory space so it's not only zram problem.

If zswap was backed by a pseudo device that failed all writes or an an ops with no evict handler then it would be functionally similar.

Quoted 6 lines

2) Many embedded system don't have enough memory. Let's assume short-lived file growing up until half of system memory once in a while. We don't want to write it on flash by wear-leveing issue and very slowness so we want to use in-memory but if we uses tmpfs, it should evict half of working set to cover them when the size reach peak. zram would be better choice.

Quoted 2 lines

Then back it by a pseudo device that fails all writes so it does not have to write to disk.

You mean "make pseudo block device and register make_request_fn
and prevent writeback". Bah, yes, it's doable but what is it different with below?

1) move zbud into zram
2) implement frontswap API in zram
3) implement writebazk in zram

The zram has been for a long time in staging to be promoted and have been maintained/deployed. Of course, I have asked the promotion several times for above a year.

Why can't zram include zswap functions if you really want to merge them? Is there any problem?

Hi Minchan,

Quoted 84 lines · Minchan Kim

Hello Mel,

Quoted 42 lines · Mel Gorman

Quoted 18 lines · Minchan Kim

Quoted 18 lines

Quoted 17 lines

Quoted 9 lines

<SNIP>
If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 3 lines

Quoted 1 line

And is zsmalloc was pluggable for zswap then it would also benefit.

But zswap isn't pseudo block device so it couldn't be used for block device.

It would not be impossible to write one. Taking a quick look it might even be doable by just providing a zbud_ops that does not have an evict handler and make sure the errors are handled correctly. i.e. does the following patch mean that zswap never writes back and instead just compresses pages in memory?

mm/zswap.c +0 -1

index deda2b6..99e41c8 100644
@@ -819,7 +819,6 @@ static void zswap_frontswap_invalidate_area(unsigned type)
 }
 
 static struct zbud_ops zswap_zbud_ops = {
-	.evict = zswap_writeback_entry
 };
 
 static void zswap_frontswap_init(unsigned type)

Quoted 4 lines

If so, it should be doable to link that up in a sane way so it can be configured at runtime.

Did you ever even try something like this?

Never. Because I didn't have such requirement for zram.

Quoted 18 lines

Quoted 7 lines

Let say one usecase for using zram-blk.

1) Many embedded system don't have swap so although tmpfs can support swapout it's pointless still so such systems should have sane configuration to limit memory space so it's not only zram problem.

If zswap was backed by a pseudo device that failed all writes or an an ops with no evict handler then it would be functionally similar.

Quoted 6 lines

2) Many embedded system don't have enough memory. Let's assume short-lived file growing up until half of system memory once in a while. We don't want to write it on flash by wear-leveing issue and very slowness so we want to use in-memory but if we uses tmpfs, it should evict half of working set to cover them when the size reach peak. zram would be better choice.

Quoted 2 lines

Then back it by a pseudo device that fails all writes so it does not have to write to disk.

You mean "make pseudo block device and register make_request_fn
and prevent writeback". Bah, yes, it's doable but what is it different with below?

1) move zbud into zram
2) implement frontswap API in zram
3) implement writebazk in zram

The zram has been for a long time in staging to be promoted and have been maintained/deployed. Of course, I have asked the promotion several times for above a year.

Quoted 2 lines

Why can't zram include zswap functions if you really want to merge them? Is there any problem?

I think merging zram into zswap or merging zswap into zram are the same thing. It's no difference.

Both way will result in a solution finally with zram block device, frontswap API etc.

The difference is just the name and the merging patch title, I think it's unimportant.

I've implemented a series [PATCH 0/4] mm: merge zram into zswap, I can change the tile to "merge zswap into zram" if you want and rename zswap to something like zhybrid.

Hello Bob,

Sorry for the late response. I was on holiday.

Quoted 87 lines · Bob Liu

Hi Minchan,

Quoted 84 lines · Minchan Kim

Hello Mel,

Quoted 42 lines · Mel Gorman

Quoted 18 lines · Minchan Kim

Quoted 18 lines

Quoted 17 lines

Quoted 9 lines

<SNIP>
If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 3 lines

Quoted 1 line

And is zsmalloc was pluggable for zswap then it would also benefit.

But zswap isn't pseudo block device so it couldn't be used for block device.

It would not be impossible to write one. Taking a quick look it might even be doable by just providing a zbud_ops that does not have an evict handler and make sure the errors are handled correctly. i.e. does the following patch mean that zswap never writes back and instead just compresses pages in memory?

mm/zswap.c +0 -1

index deda2b6..99e41c8 100644
@@ -819,7 +819,6 @@ static void zswap_frontswap_invalidate_area(unsigned type)
 }
 
 static struct zbud_ops zswap_zbud_ops = {
-  .evict = zswap_writeback_entry
 };
 
 static void zswap_frontswap_init(unsigned type)

Quoted 4 lines

If so, it should be doable to link that up in a sane way so it can be configured at runtime.

Did you ever even try something like this?

Never. Because I didn't have such requirement for zram.

Quoted 18 lines

Quoted 7 lines

Let say one usecase for using zram-blk.

1) Many embedded system don't have swap so although tmpfs can support swapout it's pointless still so such systems should have sane configuration to limit memory space so it's not only zram problem.

If zswap was backed by a pseudo device that failed all writes or an an ops with no evict handler then it would be functionally similar.

Quoted 6 lines

2) Many embedded system don't have enough memory. Let's assume short-lived file growing up until half of system memory once in a while. We don't want to write it on flash by wear-leveing issue and very slowness so we want to use in-memory but if we uses tmpfs, it should evict half of working set to cover them when the size reach peak. zram would be better choice.

Quoted 2 lines

Then back it by a pseudo device that fails all writes so it does not have to write to disk.

You mean "make pseudo block device and register make_request_fn
and prevent writeback". Bah, yes, it's doable but what is it different with below?

1) move zbud into zram
2) implement frontswap API in zram
3) implement writebazk in zram

The zram has been for a long time in staging to be promoted and have been maintained/deployed. Of course, I have asked the promotion several times for above a year.

Quoted 5 lines

Quoted 2 lines

Why can't zram include zswap functions if you really want to merge them? Is there any problem?

I think merging zram into zswap or merging zswap into zram are the same thing. It's no difference.

True but i'd like to merge zswap code into zram.
Because as you know, zram has already lots of users while zswap is almost new young so I'd like to keep backward compatibility for zram so moving zswap code into zram is more handy and could keep the git log as well.

Quoted 2 lines

Both way will result in a solution finally with zram block device, frontswap API etc.

Right but z* family people should discuss that zswap-writeback is really good solution for compressed swap. Firstly, I thought zswap is differnt with zram so there is no issue to promote zram so I and Nitin helped zsmalloc promotion for Seth and have reviewed at zswap inital phases but the situation is chainging. Let's discussion further points about compresssed swap solution. I raised issues as reply of Mel in my thread. Let's think of it.

Quoted 3 lines

The difference is just the name and the merging patch title, I think it's unimportant.

If we decide merging them, yes, module name would be important and we can't ignore copyright and maintainer part, either. Anyway,
I'd like to go that way as last resort afther enough thinking which can justify frontswap-based swap writeback is right approach.

Quoted 4 lines

I've implemented a series [PATCH 0/4] mm: merge zram into zswap, I can change the tile to "merge zswap into zram" if you want and rename zswap to something like zhybrid.

Hmm, I saw that roughly and you are ignoring zram's backward compatibility and zram-blk functionality which can be used for in-memory compressed block device without swap.

Thanks.

--
Kind regards,
Minchan Kim

We are gearing up to evaluate zswap, but we have only ported kernels up to 3.8 to our hardware, so we may be missing important patches.

In our experience, and with all due respect, the linux MM is a complex beast, and it's difficult to predict how hard it will be for us to switch to zswap. Even with the relatively simple zram, our load

triggered bugs in other parts of the MM that took a fair amount of work to resolve.

I may be wrong, but the in-memory compressed block device implemented by zram seems like a simple device which uses a well-established API to the rest of the kernel. If it is removed from the kernel, will it be difficult for us to carry our own patch? Because we may have to do that for a while. Of course we would prefer it if it stayed in, at least temporarily.

Also, could someone confirm or deny that the maximum compression ratio in zbud is 2? Because we easily achieve a 2.6-2.8 compression ratio with our loads using zram with zsmalloc and LZO or snappy. Losing that memory will cause a noticeable regression, which will encourage us to stick with zram.

I am hoping that our load is not so unusual that we are the only Linux users in this situation, and that zsmalloc (or other allocator-compressor with similar characteristics) will continue to exist, whether it is used by zram or zswap.

Thanks!

Quoted 131 lines · Minchan Kim

Hello Bob,

Sorry for the late response. I was on holiday.

Quoted 87 lines · Bob Liu

Hi Minchan,

Quoted 84 lines · Minchan Kim

Hello Mel,

Quoted 42 lines · Mel Gorman

Quoted 18 lines · Minchan Kim

Quoted 18 lines

Quoted 17 lines

Quoted 9 lines

<SNIP>
If it's used for something like tmpfs then it becomes much worse. Normal tmpfs without swap can lockup if tmpfs is allowed to fill memory. In a sane configuration, lockups will be avoided and deleting a tmpfs file is guaranteed to free memory. When zram is used to back tmpfs, there is no guarantee that any memory is freed due to fragmentation of the compressed pages. The only way to recover the memory may be to kill applications holding tmpfs files open and then delete them which is fairly drastic action in a normal server environment.

Indeed.
Actually, I had a plan to support zsmalloc compaction. The zsmalloc exposes handle instead of pure pointer so it could migrate some zpages to somewhere to pack in. Then, it could help above problem and OOM storm problem. Anyway, it's a totally new feature and requires many changes and experiement. Although we don't have such feature, zram is still good for many people.

Quoted 3 lines

Quoted 1 line

And is zsmalloc was pluggable for zswap then it would also benefit.

But zswap isn't pseudo block device so it couldn't be used for block device.

It would not be impossible to write one. Taking a quick look it might even be doable by just providing a zbud_ops that does not have an evict handler and make sure the errors are handled correctly. i.e. does the following patch mean that zswap never writes back and instead just compresses pages in memory?

mm/zswap.c +0 -1

index deda2b6..99e41c8 100644
@@ -819,7 +819,6 @@ static void zswap_frontswap_invalidate_area(unsigned type)
 }
 
 static struct zbud_ops zswap_zbud_ops = {
-  .evict = zswap_writeback_entry
 };
 
 static void zswap_frontswap_init(unsigned type)

Quoted 4 lines

If so, it should be doable to link that up in a sane way so it can be configured at runtime.

Did you ever even try something like this?

Never. Because I didn't have such requirement for zram.

Quoted 18 lines

Quoted 7 lines

Let say one usecase for using zram-blk.

1) Many embedded system don't have swap so although tmpfs can support swapout it's pointless still so such systems should have sane configuration to limit memory space so it's not only zram problem.

If zswap was backed by a pseudo device that failed all writes or an an ops with no evict handler then it would be functionally similar.

Quoted 6 lines

2) Many embedded system don't have enough memory. Let's assume short-lived file growing up until half of system memory once in a while. We don't want to write it on flash by wear-leveing issue and very slowness so we want to use in-memory but if we uses tmpfs, it should evict half of working set to cover them when the size reach peak. zram would be better choice.

Quoted 2 lines

Then back it by a pseudo device that fails all writes so it does not have to write to disk.

You mean "make pseudo block device and register make_request_fn
and prevent writeback". Bah, yes, it's doable but what is it different with below?

1) move zbud into zram
2) implement frontswap API in zram
3) implement writebazk in zram

The zram has been for a long time in staging to be promoted and have been maintained/deployed. Of course, I have asked the promotion several times for above a year.

Quoted 5 lines

Quoted 2 lines

Why can't zram include zswap functions if you really want to merge them? Is there any problem?

I think merging zram into zswap or merging zswap into zram are the same thing. It's no difference.

True but i'd like to merge zswap code into zram.
Because as you know, zram has already lots of users while zswap is almost new young so I'd like to keep backward compatibility for zram so moving zswap code into zram is more handy and could keep the git log as well.

Quoted 2 lines

Both way will result in a solution finally with zram block device, frontswap API etc.

Right but z* family people should discuss that zswap-writeback is really good solution for compressed swap. Firstly, I thought zswap is differnt with zram so there is no issue to promote zram so I and Nitin helped zsmalloc promotion for Seth and have reviewed at zswap inital phases but the situation is chainging. Let's discussion further points about compresssed swap solution. I raised issues as reply of Mel in my thread. Let's think of it.

Quoted 3 lines

The difference is just the name and the merging patch title, I think it's unimportant.

If we decide merging them, yes, module name would be important and we can't ignore copyright and maintainer part, either. Anyway,
I'd like to go that way as last resort afther enough thinking which can justify frontswap-based swap writeback is right approach.

Quoted 4 lines

I've implemented a series [PATCH 0/4] mm: merge zram into zswap, I can change the tile to "merge zswap into zram" if you want and rename zswap to something like zhybrid.

Hmm, I saw that roughly and you are ignoring zram's backward compatibility and zram-blk functionality which can be used for in-memory compressed block device without swap.

Quoted 5 lines

Thanks.

--
Kind regards,
Minchan Kim

We are gearing up to evaluate zswap, but we have only ported kernels up to 3.8 to our hardware, so we may be missing important patches.

In our experience, and with all due respect, the linux MM is a complex beast, and it's difficult to predict how hard it will be for us to switch to zswap. Even with the relatively simple zram, our load triggered bugs in other parts of the MM that took a fair amount of work to resolve.

I may be wrong, but the in-memory compressed block device implemented by zram seems like a simple device which uses a well-established API to the rest of the kernel. If it is removed from the kernel, will it

be difficult for us to carry our own patch? Because we may have to do that for a while. Of course we would prefer it if it stayed in, at least temporarily.

Also, could someone confirm or deny that the maximum compression ratio in zbud is 2? Because we easily achieve a 2.6-2.8 compression ratio with our loads using zram with zsmalloc and LZO or snappy. Losing that memory will cause a noticeable regression, which will encourage us to stick with zram.

I am hoping that our load is not so unusual that we are the only Linux users in this situation, and that zsmalloc (or other allocator-compressor with similar characteristics) will continue to exist, whether it is used by zram or zswap.

Thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>