minimize swapping on zswap store failure
Subsystems (6)
Nhat Pham
maintainercover minimize swapping on zswap store failure
Currently, when a zswap store attempt fails, the page is immediately swapped out. This could happen for a variety of reasons. For instance, the compression algorithm could fail (such as when the data is not compressible), or the backend allocator might not be able to find a suitable slot for the compressed page. If these pages are needed later on, users will incur IOs from swapins.
This issue prevents the adoption of zswap for potential users who cannot tolerate the latency associated with swapping. In many cases, these IOs are avoidable if we just keep in memory the pages that zswap fail to store.
This patch series add two new features for zswap that will alleviate the risk of swapping:
a) When a store attempt fail, keep the page untouched in memory instead of swapping it out.
b) If the store attempt fails at the compression step, allow the page to be stored in its uncompressed form in the zswap pool. This maintains the LRU ordering of pages, which will be helpful for accurate memory reclaim (zswap writeback in particular).
These features could be enabled independently via two new zswap module parameters.
Nhat Pham (2):
swap: allows swap bypassing on zswap store failure
zswap: store uncompressed pages when compression algorithm fails
Documentation/admin-guide/mm/zswap.rst | 16 +++++++ include/linux/zswap.h | 9 ++++ mm/page_io.c | 6 +++ mm/shmem.c | 8 +++- mm/zswap.c | 64 +++++++++++++++++++++++--- 5 files changed, 95 insertions(+), 8 deletions(-)
Signature · 1 line
2.34.1
Nhat Pham
maintainerPATCH 1/2 swap: allows swap bypassing on zswap store failure
During our experiment with zswap, we sometimes observe swap IOs even though the zswap pool limit is never hit. This is due to occasional zswap store failures, in which case the page will be written straight to the swapping device. This prevents many users who cannot tolerate swapping from adopting zswap to save memory where possible.
This patch adds the option to bypass swap when a zswap store fails. The feature is disabled by default (to preserve the existing behavior), and can be enabled via a new zswap module parameter. When enabled, swapping is all but prevented (except for when the zswap pool is full and have to write pages back to swap).
--- Documentation/admin-guide/mm/zswap.rst | 9 +++++++++ include/linux/zswap.h | 9 +++++++++ mm/page_io.c | 6 ++++++ mm/shmem.c | 8 ++++++-- mm/zswap.c | 4 ++++ 5 files changed, 34 insertions(+), 2 deletions(-)
Documentation/admin-guide/mm/zswap.rst +9 -0
@@ -153,6 +153,15 @@ attribute, e. g.:: Setting this parameter to 100 will disable the hysteresis. +Many users cannot tolerate the swapping that comes with zswap store failures, +due to the IO incurred if these pages are needed later on. In this scenario, +users can bypass swapping when zswap store attempts fail (and keep the pages +in memory) as follows: + + echo Y > /sys/module/zswap/parameters/bypass_swap_when_store_fail_enabled + +Note that swapping due to writeback is not disabled with this option. + When there is a sizable amount of cold memory residing in the zswap pool, it can be advantageous to proactively write these cold pages to swap and reclaim the memory for other use cases. By default, the zswap shrinker is disabled.
include/linux/zswap.h +9 -0
@@ -7,6 +7,7 @@extern u64 zswap_pool_total_size;extern atomic_t zswap_stored_pages; +extern bool zswap_bypass_swap_when_store_fail_enabled;#ifdef CONFIG_ZSWAP
@@ -18,6 +19,10 @@ void zswap_swapoff(int type);bool zswap_remove_swpentry_from_lru (swp_entry_t swpentry);void zswap_insert_swpentry_into_lru (swp_entry_t swpentry); +static inline bool zswap_bypass_swap_when_store_fail (void ) +{ +return zswap_bypass_swap_when_store_fail_enabled; +}#else static inline bool zswap_store (struct folio *folio)
@@ -41,6 +46,10 @@ static inline bool zswap_remove_swpentry_from_lru(swp_entry_t swpentry)static inline void zswap_insert_swpentry_into_lru (swp_entry_t swpentry) {} +static inline bool zswap_bypass_swap_when_store_fail (void ) +{ +return false; +}#endif #endif /* _LINUX_ZSWAP_H */
mm/page_io.c +6 -0
@@ -201,6 +201,12 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)folio_end_writeback (folio);return 0 ; } + +if (zswap_bypass_swap_when_store_fail ()) { +folio_mark_dirty (folio); +return AOP_WRITEPAGE_ACTIVATE ; + } +__swap_writepage (&folio->page, wbc);return 0 ; }
mm/shmem.c +6 -2
@@ -1514,8 +1514,12 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)mutex_unlock (&shmem_swaplist_mutex);BUG_ON (folio_mapped (folio)); -swap_writepage (&folio->page, wbc); -return 0 ; +/* +* Seeing AOP_WRITEPAGE_ACTIVATE here indicates swapping is disabled on +* zswap store failure. Note that in that case the folio is already +* re-marked dirty by swap_writepage() +*/ +return swap_writepage (&folio->page, wbc); }mutex_unlock (&shmem_swaplist_mutex);
mm/zswap.c +4 -0
@@ -138,6 +138,10 @@ static bool zswap_non_same_filled_pages_enabled = true;module_param_named (non_same_filled_pages_enabled, zswap_non_same_filled_pages_enabled, bool,0644 ); +bool zswap_bypass_swap_when_store_fail_enabled; +module_param_named (bypass_swap_when_store_fail_enabled, + zswap_bypass_swap_when_store_fail_enabled, bool,0644 ); +static bool zswap_exclusive_loads_enabled =IS_ENABLED (CONFIG_ZSWAP_EXCLUSIVE_LOADS_DEFAULT_ON );module_param_named (exclusive_loads, zswap_exclusive_loads_enabled, bool,0644 );
Signature · 1 line
2.34.1
Nhat Pham
maintainerPATCH 2/2 zswap: store uncompressed pages when compression algorithm fails
In reply to Nhat PhamIn a zswap store, the compression step could fail, leaving the pages out of the zswap pool. This could happen, for instance, when the pages hold random data (such as /dev/urandom). This behavior can potentially create unnecessary LRU inversion: warm but incompressible pages are swapped out, whereas cold compressible pages reside in the zswap pool. This could incur IO later when the warmer pages are indeed required again.
We can instead just store the original page in the zswap pool. Storing PAGE_SIZE objects is already supported by zsmalloc, zswap's default backend allocator. When these pages are needed (either for a zswap load or for writeback), we can simply copy the original content into the desired buffer.
This patch allows zswap to store uncompressed pages. The feature can be enabled at runtime via a new parameter. In addition, we add a counter to track the number of times the compression step fails for diagnostics.
--- Documentation/admin-guide/mm/zswap.rst | 7 +++ mm/zswap.c | 60 +++++++++++++++++++++++--- 2 files changed, 61 insertions(+), 6 deletions(-)
Documentation/admin-guide/mm/zswap.rst +7 -0
@@ -162,6 +162,13 @@ in memory) as follows: Note that swapping due to writeback is not disabled with this option. +Compression could also fail during a zswap store attempt. In many cases, it is +nevertheless beneficial to store the page in the zswap pool (in its +uncompressed form) for the sake of maintaining the LRU ordering, which will +be useful for reclaim. This can be enabled as follows: + + echo Y > /sys/module/zswap/parameters/uncompressed_pages_enabled + When there is a sizable amount of cold memory residing in the zswap pool, it can be advantageous to proactively write these cold pages to swap and reclaim the memory for other use cases. By default, the zswap shrinker is disabled.
mm/zswap.c +54 -6
140 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 13 lines · Nhat Pham
Currently, when a zswap store attempt fails, the page is immediately swapped out. This could happen for a variety of reasons. For instance, the compression algorithm could fail (such as when the data is not compressible), or the backend allocator might not be able to find a suitable slot for the compressed page. If these pages are needed later on, users will incur IOs from swapins.
This issue prevents the adoption of zswap for potential users who cannot tolerate the latency associated with swapping. In many cases, these IOs are avoidable if we just keep in memory the pages that zswap fail to store.
Quoted 5 lines
This patch series add two new features for zswap that will alleviate the risk of swapping:
a) When a store attempt fail, keep the page untouched in memory instead of swapping it out.
What about writeback when the zswap limit is hit? I understand the problem, but I am wondering if this is the correct way of fixing it. We really need to make zswap work without a backing swapfile, which I think is the correct way to fix all these problems. I was working on that, but unfortunately I had to pivot to something else before I had something that was working.
At Google, we have "ghost" swapfiles that we use just to use zswap without a swapfile. They are sparse files, and we have internal kernel patches to flag them and never try to actually write to them.
I am not sure how many bandaids we can afford before doing the right thing. I understand it's a much larger surgery, perhaps there is a way to get a short-term fix that is also a step towards the final state we want to reach instead?
Quoted 5 lines
b) If the store attempt fails at the compression step, allow the page to be stored in its uncompressed form in the zswap pool. This maintains the LRU ordering of pages, which will be helpful for accurate memory reclaim (zswap writeback in particular).
This is dangerous. Johannes and I discussed this before. This means that reclaim can end up allocating more memory instead of freeing. Allocations made in the reclaim path are made under the assumption that we will eventually free memory. In this case, we won't. In the worst case scenario, reclaim can leave the system/memcg in a worse state than before it started.
Perhaps there is a way we can do this without allocating a zswap entry?
I thought before about having a special list_head that allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and struct zswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
Quoted 17 lines
These features could be enabled independently via two new zswap module parameters.
Nhat Pham (2):
swap: allows swap bypassing on zswap store failure
zswap: store uncompressed pages when compression algorithm failsDocumentation/admin-guide/mm/zswap.rst | 16 +++++++ include/linux/zswap.h | 9 ++++ mm/page_io.c | 6 +++ mm/shmem.c | 8 +++- mm/zswap.c | 64 +++++++++++++++++++++++--- 5 files changed, 95 insertions(+), 8 deletions(-)--
2.34.1
Johannes Weiner
maintainerQuoted 27 lines · Yosry Ahmed
Quoted 13 lines · Nhat Pham
Currently, when a zswap store attempt fails, the page is immediately swapped out. This could happen for a variety of reasons. For instance, the compression algorithm could fail (such as when the data is not compressible), or the backend allocator might not be able to find a suitable slot for the compressed page. If these pages are needed later on, users will incur IOs from swapins.
This issue prevents the adoption of zswap for potential users who cannot tolerate the latency associated with swapping. In many cases, these IOs are avoidable if we just keep in memory the pages that zswap fail to store.
Quoted 5 lines
This patch series add two new features for zswap that will alleviate the risk of swapping:
a) When a store attempt fail, keep the page untouched in memory instead of swapping it out.
What about writeback when the zswap limit is hit? I understand the problem, but I am wondering if this is the correct way of fixing it. We really need to make zswap work without a backing swapfile, which I think is the correct way to fix all these problems. I was working on that, but unfortunately I had to pivot to something else before I had something that was working.
Quoted 8 lines
At Google, we have "ghost" swapfiles that we use just to use zswap without a swapfile. They are sparse files, and we have internal kernel patches to flag them and never try to actually write to them.
I am not sure how many bandaids we can afford before doing the right thing. I understand it's a much larger surgery, perhaps there is a way to get a short-term fix that is also a step towards the final state we want to reach instead?
I agree it should also stop writeback due to the limit.
Note that a knob like this is still useful even when zswap space is decoupled from disk swap slots. We still are using disk swap broadly in the fleet as well, so a static ghost file setup wouldn't be a good solution for us. Swapoff with common swapfile sizes is often not an option during runtime, due to how slow it is, and the destabilizing effect it can have on the system unless that's basically completely idle. As such, we expect to continue deploying swap files for physical use, and switch the zswap-is-terminal knob depending on the workload.
The other aspect to this is that workloads that do not want the swapout/swapin overhead for themselves are usually co-located with system management software, and/or can share the host with less latency sensitive workloads, that should continue to use disk swap.
Due to the latter case, I wonder if a global knob is actually enough. More likely we'd need per-cgroup control over this.
[ It's at this point where the historic coupling of zswap to disk
space is especially unfortunate. Because of it, zswap usage counts toward the memory.swap allowance. If these were separate, we could have easily set memory.zswap.max=max, memory.swap.max=0 to achieve the desired effect.
Alas, that ship has sailed. Even after a decoupling down the line it would be difficult to change established memory.swap semantics. ]
So I obviously agree that we still need to invest in decoupling zswap space from physical disk slots. It's insanely wasteful, especially with larger memory capacities. But while it would be a fantastic optimization, I don't see how it would be an automatic solution to the problem that inspired this proposal.
We still need some way to reasonably express desired workload policy in a setup that supports multiple, simultaneous modes of operation.
Quoted 11 lines
Quoted 4 lines
b) If the store attempt fails at the compression step, allow the page to be stored in its uncompressed form in the zswap pool. This maintains the LRU ordering of pages, which will be helpful for accurate memory reclaim (zswap writeback in particular).
This is dangerous. Johannes and I discussed this before. This means that reclaim can end up allocating more memory instead of freeing. Allocations made in the reclaim path are made under the assumption that we will eventually free memory. In this case, we won't. In the worst case scenario, reclaim can leave the system/memcg in a worse state than before it started.
Yeah, this is a concern. It's not such a big deal if it's only a few pages, and we're still shrinking the footprint on aggregate. But it's conceivable this can happen systematically with some datasets, in which case reclaim will drive up the memory consumption and cause OOMs, or potentially deplete the reserves with PF_MEMALLOC and cause memory deadlocks.
This isn't something we can reasonably accept as worst-case behavior.
Quoted 10 lines
Perhaps there is a way we can do this without allocating a zswap entry?
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
That should work. Once zswap has exclusive control over the page, it is free to muck with its lru linkage. A lower bit tag on the next or prev pointer should suffice to distinguish between struct page and struct zswap_entry when pulling stuff from the list.
We'd also have to teach vmscan.c to hand off the page. It currently expects that it either frees the page back to the allocator, or puts it back on the LRU. We'd need a compromise where it continues to tear down the page and remove the mapping, but then leaves it to zswap.
Neither of those sound impossible. But since it's a bigger complication than this proposal, it probably needs a new cost/benefit analysis, with potentially more data on the problem of LRU inversions.
Thanks for your insightful feedback, Yosry.
Quoted 49 lines · Johannes Weiner
Quoted 27 lines · Yosry Ahmed
Quoted 13 lines · Nhat Pham
Currently, when a zswap store attempt fails, the page is immediately swapped out. This could happen for a variety of reasons. For instance, the compression algorithm could fail (such as when the data is not compressible), or the backend allocator might not be able to find a suitable slot for the compressed page. If these pages are needed later on, users will incur IOs from swapins.
This issue prevents the adoption of zswap for potential users who cannot tolerate the latency associated with swapping. In many cases, these IOs are avoidable if we just keep in memory the pages that zswap fail to store.
Quoted 5 lines
This patch series add two new features for zswap that will alleviate the risk of swapping:
a) When a store attempt fail, keep the page untouched in memory instead of swapping it out.
What about writeback when the zswap limit is hit? I understand the problem, but I am wondering if this is the correct way of fixing it. We really need to make zswap work without a backing swapfile, which I think is the correct way to fix all these problems. I was working on that, but unfortunately I had to pivot to something else before I had something that was working.
Quoted 8 lines
At Google, we have "ghost" swapfiles that we use just to use zswap without a swapfile. They are sparse files, and we have internal kernel patches to flag them and never try to actually write to them.
I am not sure how many bandaids we can afford before doing the right thing. I understand it's a much larger surgery, perhaps there is a way to get a short-term fix that is also a step towards the final state we want to reach instead?
I agree it should also stop writeback due to the limit.
Note that a knob like this is still useful even when zswap space is decoupled from disk swap slots. We still are using disk swap broadly in the fleet as well, so a static ghost file setup wouldn't be a good solution for us. Swapoff with common swapfile sizes is often not an option during runtime, due to how slow it is, and the destabilizing effect it can have on the system unless that's basically completely idle. As such, we expect to continue deploying swap files for physical use, and switch the zswap-is-terminal knob depending on the workload.
Quoted 7 lines
The other aspect to this is that workloads that do not want the swapout/swapin overhead for themselves are usually co-located with system management software, and/or can share the host with less latency sensitive workloads, that should continue to use disk swap.
Due to the latter case, I wonder if a global knob is actually enough. More likely we'd need per-cgroup control over this.
In conjunction with ghost swapfiles, we have a knob to determine the type of swapfile to use for each cgroup (normal, ghost, either, or none). This achieves what you are describing, allowing different workloads on the same machine to use zswap only or disk swap, although in practice we only use zswap now.
I am not saying that's necessarily the correct way of doing it. Having a zswap-is-terminal knob per-cgroup is another way to achieve this. I will loop in folks maintaining this code internally to see what they think.
Quoted 9 lines
[ It's at this point where the historic coupling of zswap to disk
space is especially unfortunate. Because of it, zswap usage counts toward the memory.swap allowance. If these were separate, we could have easily set memory.zswap.max=max, memory.swap.max=0 to achieve the desired effect.Alas, that ship has sailed. Even after a decoupling down the line it would be difficult to change established memory.swap semantics. ]
Fully agree here. This is unfortunate.
Quoted 6 lines
So I obviously agree that we still need to invest in decoupling zswap space from physical disk slots. It's insanely wasteful, especially with larger memory capacities. But while it would be a fantastic optimization, I don't see how it would be an automatic solution to the problem that inspired this proposal.
Well, in my head, I imagine such a world where we have multiple separate swapping backends with cgroup knob(s) that control what backends are allowed for each cgroup. A zswap-is-terminal knob is hacky-ish way of doing that where the backends are only zswap and disk swap.
Quoted 16 lines
We still need some way to reasonably express desired workload policy in a setup that supports multiple, simultaneous modes of operation.
Quoted 11 lines
Quoted 4 lines
b) If the store attempt fails at the compression step, allow the page to be stored in its uncompressed form in the zswap pool. This maintains the LRU ordering of pages, which will be helpful for accurate memory reclaim (zswap writeback in particular).
This is dangerous. Johannes and I discussed this before. This means that reclaim can end up allocating more memory instead of freeing. Allocations made in the reclaim path are made under the assumption that we will eventually free memory. In this case, we won't. In the worst case scenario, reclaim can leave the system/memcg in a worse state than before it started.
Quoted 8 lines
Yeah, this is a concern. It's not such a big deal if it's only a few pages, and we're still shrinking the footprint on aggregate. But it's conceivable this can happen systematically with some datasets, in which case reclaim will drive up the memory consumption and cause OOMs, or potentially deplete the reserves with
PF_MEMALLOCand cause memory deadlocks.This isn't something we can reasonably accept as worst-case behavior.
Right.
Quoted 12 lines
Quoted 11 lines
Perhaps there is a way we can do this without allocating a zswap entry?
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
Quoted 4 lines
That should work. Once zswap has exclusive control over the page, it is free to muck with its lru linkage. A lower bit tag on the next or prev pointer should suffice to distinguish between struct page and struct
zswap_entrywhen pulling stuff from the list.
Right.
We handle incompressible memory internally in a different way, we put them back on the unevictable list with an incompressible page flag. This achieves a similar effect.
A missing point here is that those pages, if dirtied, may be compressible again. When we have them on the LRUs, we rely on periodic scanning (similar to the MGLRU-based periodic scanning we proposed before) to check the dirty bit and make those pages evictable again.
If we leave them on the zswap LRU, we will incur a fault instead to pull them back to the LRUs. For anon pages, that's probably fine, as in both cases by the time we reach zswap the page has been unmapped, and accessing it again incurs a fault anyway (whether it's in zswap LRUs or in the reclaim LRUs). For shmem though, we put the incompressible pages back in the page cache, preventing a page fault on the next access. This is a drawback of the zswap LRU approach AFAICT. Not sure how much it matters in practice.
Quoted 5 lines
We'd also have to teach
vmscan.cto hand off the page. It currently expects that it either frees the page back to the allocator, or puts it back on the LRU. We'd need a compromise where it continues to tear down the page and remove the mapping, but then leaves it to zswap.
Right.
Quoted 4 lines
Neither of those sound impossible. But since it's a bigger complication than this proposal, it probably needs a new cost/benefit analysis, with potentially more data on the problem of LRU inversions.
Makes sense.
Quoted 2 lines
Thanks for your insightful feedback, Yosry.
Johannes Weiner
maintainerQuoted 9 lines · Yosry Ahmed
Quoted 7 lines · Johannes Weiner
Quoted 1 line · Yosry Ahmed
On Mon, Oct 16, 2023 at 5:35 PM Nhat Pham <[email protected]> wrote:
So I obviously agree that we still need to invest in decoupling zswap space from physical disk slots. It's insanely wasteful, especially with larger memory capacities. But while it would be a fantastic optimization, I don't see how it would be an automatic solution to the problem that inspired this proposal.
Quoted 5 lines
Well, in my head, I imagine such a world where we have multiple separate swapping backends with cgroup knob(s) that control what backends are allowed for each cgroup. A zswap-is-terminal knob is hacky-ish way of doing that where the backends are only zswap and disk swap.
"I want compression" vs "I want disk offloading" is a more reasonable question to ask at the cgroup level. We've had historically a variety of swap configurations across the fleet. E.g. it's a lot easier to add another swapfile than it is to grow an existing one at runtime. In other cases, one storage config might have one swapfile, another machine model might want to spread it out over multiple disks etc.
This doesn't matter much with ghost files. But with conventional swapfiles this requires an unnecessary awareness of the backend topology in order to express container policy. That's no bueno.
Quoted 16 lines
Quoted 11 lines
Quoted 10 lines
Perhaps there is a way we can do this without allocating a zswap entry?
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.Quoted 4 lines
That should work. Once zswap has exclusive control over the page, it is free to muck with its lru linkage. A lower bit tag on the next or prev pointer should suffice to distinguish between struct page and struct
zswap_entrywhen pulling stuff from the list.
Quoted 5 lines
Right.
We handle incompressible memory internally in a different way, we put them back on the unevictable list with an incompressible page flag. This achieves a similar effect.
It doesn't. We want those incompressible pages to continue aging alongside their compressible peers, and eventually get written back to disk with them.
Quoted 24 lines · Johannes Weiner
Quoted 9 lines · Yosry Ahmed
Quoted 7 lines · Johannes Weiner
Quoted 1 line · Yosry Ahmed
On Mon, Oct 16, 2023 at 5:35 PM Nhat Pham <[email protected]> wrote:
So I obviously agree that we still need to invest in decoupling zswap space from physical disk slots. It's insanely wasteful, especially with larger memory capacities. But while it would be a fantastic optimization, I don't see how it would be an automatic solution to the problem that inspired this proposal.
Quoted 5 lines
Well, in my head, I imagine such a world where we have multiple separate swapping backends with cgroup knob(s) that control what backends are allowed for each cgroup. A zswap-is-terminal knob is hacky-ish way of doing that where the backends are only zswap and disk swap.
"I want compression" vs "I want disk offloading" is a more reasonable question to ask at the cgroup level. We've had historically a variety of swap configurations across the fleet. E.g. it's a lot easier to add another swapfile than it is to grow an existing one at runtime. In other cases, one storage config might have one swapfile, another machine model might want to spread it out over multiple disks etc.
Quoted 3 lines
This doesn't matter much with ghost files. But with conventional swapfiles this requires an unnecessary awareness of the backend topology in order to express container policy. That's no bueno.
Oh I didn't mean that cgroups would designate specific swapfiles, but rather swapping backends, which would be "zswap" or "disk" or both in this case. I just imagined an interface that is more generic and extensible rather than a specific zswap-is-terminal knob.
Quoted 19 lines
Quoted 19 lines
Quoted 12 lines
Quoted 11 lines
Perhaps there is a way we can do this without allocating a zswap entry?
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.Quoted 4 lines
That should work. Once zswap has exclusive control over the page, it is free to muck with its lru linkage. A lower bit tag on the next or prev pointer should suffice to distinguish between struct page and struct
zswap_entrywhen pulling stuff from the list.Right.
Quoted 7 lines
Quoted 3 lines
We handle incompressible memory internally in a different way, we put them back on the unevictable list with an incompressible page flag. This achieves a similar effect.
It doesn't. We want those incompressible pages to continue aging alongside their compressible peers, and eventually get written back to disk with them.
Sorry I wasn't clear, I was talking about the case where zswap is terminal. When zswap is not, in our approach we just skip zswap for incompressible pages and write them directly to disk. Aging them on the LRU is probably the better approach here.
For the case where zswap is terminal, making them unevictable incurs less page faults, at least for shmem.
Nhat Pham
maintainerQuoted 107 lines · Johannes Weiner
Quoted 27 lines · Yosry Ahmed
Quoted 13 lines · Nhat Pham
Currently, when a zswap store attempt fails, the page is immediately swapped out. This could happen for a variety of reasons. For instance, the compression algorithm could fail (such as when the data is not compressible), or the backend allocator might not be able to find a suitable slot for the compressed page. If these pages are needed later on, users will incur IOs from swapins.
This issue prevents the adoption of zswap for potential users who cannot tolerate the latency associated with swapping. In many cases, these IOs are avoidable if we just keep in memory the pages that zswap fail to store.
Quoted 5 lines
This patch series add two new features for zswap that will alleviate the risk of swapping:
a) When a store attempt fail, keep the page untouched in memory instead of swapping it out.
What about writeback when the zswap limit is hit? I understand the problem, but I am wondering if this is the correct way of fixing it. We really need to make zswap work without a backing swapfile, which I think is the correct way to fix all these problems. I was working on that, but unfortunately I had to pivot to something else before I had something that was working.
Quoted 8 lines
At Google, we have "ghost" swapfiles that we use just to use zswap without a swapfile. They are sparse files, and we have internal kernel patches to flag them and never try to actually write to them.
I am not sure how many bandaids we can afford before doing the right thing. I understand it's a much larger surgery, perhaps there is a way to get a short-term fix that is also a step towards the final state we want to reach instead?
I agree it should also stop writeback due to the limit.
Note that a knob like this is still useful even when zswap space is decoupled from disk swap slots. We still are using disk swap broadly in the fleet as well, so a static ghost file setup wouldn't be a good solution for us. Swapoff with common swapfile sizes is often not an option during runtime, due to how slow it is, and the destabilizing effect it can have on the system unless that's basically completely idle. As such, we expect to continue deploying swap files for physical use, and switch the zswap-is-terminal knob depending on the workload.
The other aspect to this is that workloads that do not want the swapout/swapin overhead for themselves are usually co-located with system management software, and/or can share the host with less latency sensitive workloads, that should continue to use disk swap.
Due to the latter case, I wonder if a global knob is actually enough. More likely we'd need per-cgroup control over this.
[ It's at this point where the historic coupling of zswap to disk
space is especially unfortunate. Because of it, zswap usage counts toward the memory.swap allowance. If these were separate, we could have easily set memory.zswap.max=max, memory.swap.max=0 to achieve the desired effect.Alas, that ship has sailed. Even after a decoupling down the line it would be difficult to change established memory.swap semantics. ]
So I obviously agree that we still need to invest in decoupling zswap space from physical disk slots. It's insanely wasteful, especially with larger memory capacities. But while it would be a fantastic optimization, I don't see how it would be an automatic solution to the problem that inspired this proposal.
We still need some way to reasonably express desired workload policy in a setup that supports multiple, simultaneous modes of operation.
Quoted 11 lines
Quoted 4 lines
b) If the store attempt fails at the compression step, allow the page to be stored in its uncompressed form in the zswap pool. This maintains the LRU ordering of pages, which will be helpful for accurate memory reclaim (zswap writeback in particular).
This is dangerous. Johannes and I discussed this before. This means that reclaim can end up allocating more memory instead of freeing. Allocations made in the reclaim path are made under the assumption that we will eventually free memory. In this case, we won't. In the worst case scenario, reclaim can leave the system/memcg in a worse state than before it started.
Yeah, this is a concern. It's not such a big deal if it's only a few pages, and we're still shrinking the footprint on aggregate. But it's conceivable this can happen systematically with some datasets, in which case reclaim will drive up the memory consumption and cause OOMs, or potentially deplete the reserves with
PF_MEMALLOCand cause memory deadlocks.This isn't something we can reasonably accept as worst-case behavior.
Quoted 10 lines
Perhaps there is a way we can do this without allocating a zswap entry?
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
Quoted 4 lines
That should work. Once zswap has exclusive control over the page, it is free to muck with its lru linkage. A lower bit tag on the next or prev pointer should suffice to distinguish between struct page and struct
zswap_entrywhen pulling stuff from the list.
Hmm I'm a bit iffy about pointers bit hacking, but I guess that's the least involved way to store the uncompressed page in the zswap LRU without allocating a zswap_entry for it.
Lemme give this a try. If it looks decently clean I'll send it out :)
Quoted 11 lines
We'd also have to teach
vmscan.cto hand off the page. It currently expects that it either frees the page back to the allocator, or puts it back on the LRU. We'd need a compromise where it continues to tear down the page and remove the mapping, but then leaves it to zswap.Neither of those sound impossible. But since it's a bigger complication than this proposal, it probably needs a new cost/benefit analysis, with potentially more data on the problem of LRU inversions.
Thanks for your insightful feedback, Yosry.
Nhat Pham
maintainerQuoted 28 lines · Yosry Ahmed
Quoted 13 lines · Nhat Pham
Currently, when a zswap store attempt fails, the page is immediately swapped out. This could happen for a variety of reasons. For instance, the compression algorithm could fail (such as when the data is not compressible), or the backend allocator might not be able to find a suitable slot for the compressed page. If these pages are needed later on, users will incur IOs from swapins.
This issue prevents the adoption of zswap for potential users who cannot tolerate the latency associated with swapping. In many cases, these IOs are avoidable if we just keep in memory the pages that zswap fail to store.
Quoted 5 lines
This patch series add two new features for zswap that will alleviate the risk of swapping:
a) When a store attempt fail, keep the page untouched in memory instead of swapping it out.
What about writeback when the zswap limit is hit? I understand the problem, but I am wondering if this is the correct way of fixing it. We really need to make zswap work without a backing swapfile, which I think is the correct way to fix all these problems. I was working on that, but unfortunately I had to pivot to something else before I had something that was working.
Quoted 8 lines
At Google, we have "ghost" swapfiles that we use just to use zswap without a swapfile. They are sparse files, and we have internal kernel patches to flag them and never try to actually write to them.
I am not sure how many bandaids we can afford before doing the right thing. I understand it's a much larger surgery, perhaps there is a way to get a short-term fix that is also a step towards the final state we want to reach instead?
Regarding the writeback - I'll make sure to also short-circuit writeback when the bypass_swap option is enabled in v2 :) I'll probably send out v2 after
I absolutely agree that we must decouple zswap and swap (and would be happy to help out in any capacity I could - we have heard similar concerns/complaints about swap wastage from internal parties as well).
However, as Johannes has pointed out, this feature still has its place, given our already existing swapfile deployments. I do agree that a global knob is insufficient tho. I'll add a per-cgroup knob in v2 so that we can enable/disable this feature on a per-workload basis.
Quoted 16 lines
Quoted 6 lines
b) If the store attempt fails at the compression step, allow the page to be stored in its uncompressed form in the zswap pool. This maintains the LRU ordering of pages, which will be helpful for accurate memory reclaim (zswap writeback in particular).
This is dangerous. Johannes and I discussed this before. This means that reclaim can end up allocating more memory instead of freeing. Allocations made in the reclaim path are made under the assumption that we will eventually free memory. In this case, we won't. In the worst case scenario, reclaim can leave the system/memcg in a worse state than before it started.
Perhaps there is a way we can do this without allocating a zswap entry?
Quoted 8 lines
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
Hmm yeah you're right about these concerns. That seems like a lot more involved than what I envisioned initially.
Let's put this aside for now. I'll just send the first patch in v2, and we can work on + discuss more about uncompressed pages storing later on.
Quoted 18 lines
Quoted 18 lines
These features could be enabled independently via two new zswap module parameters.
Nhat Pham (2):
swap: allows swap bypassing on zswap store failure
zswap: store uncompressed pages when compression algorithm failsDocumentation/admin-guide/mm/zswap.rst | 16 +++++++ include/linux/zswap.h | 9 ++++ mm/page_io.c | 6 +++ mm/shmem.c | 8 +++- mm/zswap.c | 64 +++++++++++++++++++++++--- 5 files changed, 95 insertions(+), 8 deletions(-)--
2.34.1
Nhat Pham
maintainerQuoted 34 lines · Nhat Pham
Quoted 32 lines · Yosry Ahmed
Quoted 13 lines · Nhat Pham
Currently, when a zswap store attempt fails, the page is immediately swapped out. This could happen for a variety of reasons. For instance, the compression algorithm could fail (such as when the data is not compressible), or the backend allocator might not be able to find a suitable slot for the compressed page. If these pages are needed later on, users will incur IOs from swapins.
This issue prevents the adoption of zswap for potential users who cannot tolerate the latency associated with swapping. In many cases, these IOs are avoidable if we just keep in memory the pages that zswap fail to store.
Quoted 5 lines
This patch series add two new features for zswap that will alleviate the risk of swapping:
a) When a store attempt fail, keep the page untouched in memory instead of swapping it out.
What about writeback when the zswap limit is hit? I understand the problem, but I am wondering if this is the correct way of fixing it. We really need to make zswap work without a backing swapfile, which I think is the correct way to fix all these problems. I was working on that, but unfortunately I had to pivot to something else before I had something that was working.
At Google, we have "ghost" swapfiles that we use just to use zswap without a swapfile. They are sparse files, and we have internal kernel patches to flag them and never try to actually write to them.
Quoted 8 lines
Quoted 4 lines
I am not sure how many bandaids we can afford before doing the right thing. I understand it's a much larger surgery, perhaps there is a way to get a short-term fix that is also a step towards the final state we want to reach instead?
Regarding the writeback - I'll make sure to also short-circuit writeback when the
bypass_swapoption is enabled in v2 :) I'll probably send out v2 after
... I gather all these feedbacks.
Quoted 58 lines
I absolutely agree that we must decouple zswap and swap (and would be happy to help out in any capacity I could - we have heard similar concerns/complaints about swap wastage from internal parties as well).
However, as Johannes has pointed out, this feature still has its place, given our already existing swapfile deployments. I do agree that a global knob is insufficient tho. I'll add a per-cgroup knob in v2 so that we can enable/disable this feature on a per-workload basis.
Quoted 16 lines
Quoted 6 lines
b) If the store attempt fails at the compression step, allow the page to be stored in its uncompressed form in the zswap pool. This maintains the LRU ordering of pages, which will be helpful for accurate memory reclaim (zswap writeback in particular).
This is dangerous. Johannes and I discussed this before. This means that reclaim can end up allocating more memory instead of freeing. Allocations made in the reclaim path are made under the assumption that we will eventually free memory. In this case, we won't. In the worst case scenario, reclaim can leave the system/memcg in a worse state than before it started.
Perhaps there is a way we can do this without allocating a zswap entry?
Quoted 8 lines
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.Hmm yeah you're right about these concerns. That seems like a lot more involved than what I envisioned initially.
Let's put this aside for now. I'll just send the first patch in v2, and we can work on + discuss more about uncompressed pages storing later on.
Quoted 18 lines
Quoted 18 lines
These features could be enabled independently via two new zswap module parameters.
Nhat Pham (2):
swap: allows swap bypassing on zswap store failure
zswap: store uncompressed pages when compression algorithm failsDocumentation/admin-guide/mm/zswap.rst | 16 +++++++ include/linux/zswap.h | 9 ++++ mm/page_io.c | 6 +++ mm/shmem.c | 8 +++- mm/zswap.c | 64 +++++++++++++++++++++++--- 5 files changed, 95 insertions(+), 8 deletions(-)--
2.34.1
Quoted 10 lines · On Mon, 16 Oct 2023 17:57:31 -0700 Yosry Ahmed <[email protected]> wrote:
On Mon, Oct 16, 2023 at 5:35 PM Nhat Pham <[email protected]> wrote:
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
Hi Yosry,
Apologies for reviving an old thread, but I wasn't sure whether opening an entirely new thread was a better choice : -)
So I've implemented your idea, using the lower 2 bits of the list_head's prev pointer (last bit indicates whether the list_head belongs to a page or a zswap_entry, and the second to last bit was repurposed for the second chance algorithm).
For a very high level overview what I did in the patch:
- When a page fails to compress, I remove the page mapping and tag both the xarray entry (tag == set lowest bit to 1) and the page's list_head prev ptr, then store the page directly into the zswap LRU.
- In zswap_load, we take the entry out of the xarray and check if it's tagged.
- If it is tagged, then instead of decompressing, we just copy the page's contents to the newly allocated page.
- (More details about how to teach vmscan / page_io / list iterators how to handle this, but we can gloss over those details for now)
I have a working version, but have been holding off because I have only been seeing regressions. I wasn't really sure where they were coming from, but after going through some perf traces with Nhat, found out that the regressions come from the associated page faults that come from initially unmapping the page, and then re-allocating it for every load. This causes (1) more memcg flushing, and (2) extra allocations ==> more pressure ==> more reclaim, even though we only temporarily keep the extra page.
Just wanted to put this here in case you were still thinking about this idea. What do you think? Ideally, there would be a way to keep the page around in the zswap LRU, but do not have to re-allocate a new page on a fault, but this seems like a bigger task.
Ultimately the goal is to prevent an incompressible page from hoarding the compression algorithm on multiple reclaim attempts, but if we are spending more time by allocating new pages... maybe this isn't the correct approach :(
Please let me know if you have any thoughts on this : -) Have a great day!
Joshua
Sent using hkml (https://github.com/sjp38/hackermail)
Nhat Pham
maintainerQuoted 34 lines · Joshua Hahn
Quoted 10 lines · On Mon, 16 Oct 2023 17:57:31 -0700 Yosry Ahmed <[email protected]> wrote:
On Mon, Oct 16, 2023 at 5:35 PM Nhat Pham <[email protected]> wrote:
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.Hi Yosry,
Apologies for reviving an old thread, but I wasn't sure whether opening an entirely new thread was a better choice : -)
So I've implemented your idea, using the lower 2 bits of the list_head's prev pointer (last bit indicates whether the
list_headbelongs to a page or azswap_entry, and the second to last bit was repurposed for the second chance algorithm).For a very high level overview what I did in the patch:
- When a page fails to compress, I remove the page mapping and tag both the xarray entry (tag == set lowest bit to 1) and the page'slist_headprev ptr, then store the page directly into the zswap LRU.
- Inzswap_load, we take the entry out of the xarray and check if it's tagged.
- If it is tagged, then instead of decompressing, we just copy the page's contents to the newly allocated page.
- (More details about how to teach vmscan /page_io/ list iterators how to handle this, but we can gloss over those details for now)
Quoted 7 lines
I have a working version, but have been holding off because I have only been seeing regressions. I wasn't really sure where they were coming from, but after going through some perf traces with Nhat, found out that the regressions come from the associated page faults that come from initially unmapping the page, and then re-allocating it for every load. This causes (1) more memcg flushing, and (2) extra allocations ==> more pressure ==> more reclaim, even though we only temporarily keep the extra page.
Thanks for your effort on this idea :)
Quoted 5 lines
Just wanted to put this here in case you were still thinking about this idea. What do you think? Ideally, there would be a way to keep the page around in the zswap LRU, but do not have to re-allocate a new page on a fault, but this seems like a bigger task.
I wonder if we can return the page in the event of a page fault. We'll need to keep it in the swap cache for this to work:
1. On reclaim, do the same thing as your prototype but keep the page in swap cache (i.e do not remove_mapping() it).
2. On page fault (do_swap_page), before returning check if the page is in zswap LRU. If it is, invalidate the zswap LRU linkage, and put it back to one of the proper LRUs.
Johannes, do you feel like this is possible?
Quoted 4 lines
Ultimately the goal is to prevent an incompressible page from hoarding the compression algorithm on multiple reclaim attempts, but if we are spending more time by allocating new pages... maybe this isn't the correct approach :(
Hmmm, IIUC this problem also exists with zram, since zram allocates a PAGE_SIZE sized buffer to hold the original page's content. I will note though that zram seems to favor these kinds of pages for writeback :) Maybe this is why...?
(+ Minchan)
Quoted 2 lines
Please let me know if you have any thoughts on this : -)
Well worst case scenario there is still the special incompressible LRU idea. We'll need some worker thread to check for write access to these pages to promote them though.
(+ Shakeel)
Quoted 5 lines
Have a great day!
JoshuaSent using hkml (https://github.com/sjp38/hackermail)
Quoted 8 lines · Nhat Pham
Quoted 3 lines
Ultimately the goal is to prevent an incompressible page from hoarding the compression algorithm on multiple reclaim attempts, but if we are spending more time by allocating new pages... maybe this isn't the correct approach :(
Hmmm, IIUC this problem also exists with zram, since zram allocates a
PAGE_SIZEsized buffer to hold the original page's content. I will note though that zram seems to favor these kinds of pages for writeback :) Maybe this is why...?
zram is a generic block device, it must store whatever comes in, compressible or incompressible. E.g. when we have, say, ext4 running atop of the zram device we cannot reject page stores.
And you are right, when we use zram for swap, there is some benefit in storing incompressible pages. First, those pages are candidates for zram writeback, which achieves the goal of removing the page from RAM after all, we give up on the incompressible page reclamation with "return it back to LRU" approach. Second, on some zram setups we do re-compression (with a slower and more efficient algorithm) and in certain number of cases what is incompressible with the primary (fast) algorithm is compressible with the secondary algorithm.
Quoted 14 lines · On Fri, 4 Apr 2025 10:46:22 +0900 Sergey Senozhatsky…
Quoted 8 lines · Nhat Pham
Quoted 3 lines
Ultimately the goal is to prevent an incompressible page from hoarding the compression algorithm on multiple reclaim attempts, but if we are spending more time by allocating new pages... maybe this isn't the correct approach :(
Hmmm, IIUC this problem also exists with zram, since zram allocates a
PAGE_SIZEsized buffer to hold the original page's content. I will note though that zram seems to favor these kinds of pages for writeback :) Maybe this is why...?zram is a generic block device, it must store whatever comes in, compressible or incompressible. E.g. when we have, say, ext4 running atop of the zram device we cannot reject page stores.
Quoted 8 lines
And you are right, when we use zram for swap, there is some benefit in storing incompressible pages. First, those pages are candidates for zram writeback, which achieves the goal of removing the page from RAM after all, we give up on the incompressible page reclamation with "return it back to LRU" approach. Second, on some zram setups we do re-compression (with a slower and more efficient algorithm) and in certain number of cases what is incompressible with the primary (fast) algorithm is compressible with the secondary algorithm.
Hello Sergey,
Thank you for your insight, I did not know this is how zram handled incompressible pages. In the case of this prototype, I expected to see the most gains from storing incompressible pages in the zswap LRU when writeback was disabled (if writeback is enabled, then we expect to see less differences with just writing the page back).
On the note of trying a second compression algorithm -- do you know how much of the initially incompressible pages get compressed later? I can certainly imagine that trying different compression algorithms makes a difference, I am wondering if zswap should attempt this as well, or if it is not worth spending even more CPU trying to re-comprses the page.
Thank you again for your response! Have a great day : -) Joshua
Sent using hkml (https://github.com/sjp38/hackermail)
Nhat Pham
maintainerQuoted 34 lines · Joshua Hahn
Quoted 14 lines · On Fri, 4 Apr 2025 10:46:22 +0900 Sergey Senozhatsky…
Quoted 8 lines · Nhat Pham
Quoted 3 lines
Ultimately the goal is to prevent an incompressible page from hoarding the compression algorithm on multiple reclaim attempts, but if we are spending more time by allocating new pages... maybe this isn't the correct approach :(
Hmmm, IIUC this problem also exists with zram, since zram allocates a
PAGE_SIZEsized buffer to hold the original page's content. I will note though that zram seems to favor these kinds of pages for writeback :) Maybe this is why...?zram is a generic block device, it must store whatever comes in, compressible or incompressible. E.g. when we have, say, ext4 running atop of the zram device we cannot reject page stores.
Quoted 8 lines
And you are right, when we use zram for swap, there is some benefit in storing incompressible pages. First, those pages are candidates for zram writeback, which achieves the goal of removing the page from RAM after all, we give up on the incompressible page reclamation with "return it back to LRU" approach. Second, on some zram setups we do re-compression (with a slower and more efficient algorithm) and in certain number of cases what is incompressible with the primary (fast) algorithm is compressible with the secondary algorithm.
Hello Sergey,
Thank you for your insight, I did not know this is how zram handled incompressible pages. In the case of this prototype, I expected to see the most gains from storing incompressible pages in the zswap LRU when writeback was disabled (if writeback is enabled, then we expect to see less differences with just writing the page back).
Quoted 5 lines
On the note of trying a second compression algorithm -- do you know how much of the initially incompressible pages get compressed later? I can certainly imagine that trying different compression algorithms makes a difference, I am wondering if zswap should attempt this as well, or if it is not worth spending even more CPU trying to re-comprses the page.
It wouldn't help us :) The algorithm we use, zstd, is usually already the slow algorithm in this context. We can try higher levels of zstd, but there are always data that are simply incompressible - think random values, or memory already compressed by userspace.
Yeah we can target them for writeback to swap in zswap as well. It wouldn't help your (micro)benchmark though, because IIRC you don't do writeback and/or do not writeback before it is faulted back in :)
Quoted 6 lines
Thank you again for your response! Have a great day : -) Joshua
Sent using hkml (https://github.com/sjp38/hackermail)
Hi,
Sorry for the delay
Quoted 25 lines · Joshua Hahn
Quoted 14 lines · On Fri, 4 Apr 2025 10:46:22 +0900 Sergey Senozhatsky…
Quoted 8 lines · Nhat Pham
Quoted 3 lines
Ultimately the goal is to prevent an incompressible page from hoarding the compression algorithm on multiple reclaim attempts, but if we are spending more time by allocating new pages... maybe this isn't the correct approach :(
Hmmm, IIUC this problem also exists with zram, since zram allocates a
PAGE_SIZEsized buffer to hold the original page's content. I will note though that zram seems to favor these kinds of pages for writeback :) Maybe this is why...?zram is a generic block device, it must store whatever comes in, compressible or incompressible. E.g. when we have, say, ext4 running atop of the zram device we cannot reject page stores.
Quoted 8 lines
And you are right, when we use zram for swap, there is some benefit in storing incompressible pages. First, those pages are candidates for zram writeback, which achieves the goal of removing the page from RAM after all, we give up on the incompressible page reclamation with "return it back to LRU" approach. Second, on some zram setups we do re-compression (with a slower and more efficient algorithm) and in certain number of cases what is incompressible with the primary (fast) algorithm is compressible with the secondary algorithm.
Quoted 4 lines
Hello Sergey,
Thank you for your insight, I did not know this is how zram handled incompressible pages.
Well, yes, zram doesn't have a freedom to reject writes, to the fs/vfs that would look like a block device error.
[..]
Quoted 2 lines
On the note of trying a second compression algorithm -- do you know how much of the initially incompressible pages get compressed later?
So I don't recall the exact numbers, but, if I'm not mistaken, in my tests (on chromeos) I think I saw something like 20+% (a little higher than just 20%) success rate (successful re-compression with a secondary algorithm), but like you said this is very data patterns specific.
Quoted 1 line
Thank you again for your response! Have a great day : -)
Thanks, you too!
Nhat Pham
maintainerQuoted 34 lines · Joshua Hahn
Quoted 10 lines · On Mon, 16 Oct 2023 17:57:31 -0700 Yosry Ahmed <[email protected]> wrote:
On Mon, Oct 16, 2023 at 5:35 PM Nhat Pham <[email protected]> wrote:
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.Hi Yosry,
Apologies for reviving an old thread, but I wasn't sure whether opening an entirely new thread was a better choice : -)
So I've implemented your idea, using the lower 2 bits of the list_head's prev pointer (last bit indicates whether the
list_headbelongs to a page or azswap_entry, and the second to last bit was repurposed for the second chance algorithm).For a very high level overview what I did in the patch:
- When a page fails to compress, I remove the page mapping and tag both the xarray entry (tag == set lowest bit to 1) and the page'slist_headprev ptr, then store the page directly into the zswap LRU.
- Inzswap_load, we take the entry out of the xarray and check if it's tagged.
- If it is tagged, then instead of decompressing, we just copy the page's contents to the newly allocated page.
- (More details about how to teach vmscan /page_io/ list iterators how to handle this, but we can gloss over those details for now)
Quoted 7 lines
I have a working version, but have been holding off because I have only been seeing regressions. I wasn't really sure where they were coming from, but after going through some perf traces with Nhat, found out that the regressions come from the associated page faults that come from initially unmapping the page, and then re-allocating it for every load. This causes (1) more memcg flushing, and (2) extra allocations ==> more pressure ==> more reclaim, even though we only temporarily keep the extra page.
Also, double check your benchmark :) If you only cycle through the pages only in the LRU, or if you will need the pages soon, by definition you cannot do better than the status quo.
The gains will only materialize when you have longer workload, which can (and often will) cycle the incompressible pages through the LRU multiple times, recompressing (and failing) them multiple times with zswap, etc.
Quoted 13 lines · Joshua Hahn
Quoted 10 lines · On Mon, 16 Oct 2023 17:57:31 -0700 Yosry Ahmed <[email protected]> wrote:
On Mon, Oct 16, 2023 at 5:35 PM Nhat Pham <[email protected]> wrote:
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
Quoted 4 lines
Hi Yosry,
Apologies for reviving an old thread, but I wasn't sure whether opening an entirely new thread was a better choice : -)
This seems like the right choice to me :)
Quoted 5 lines
So I've implemented your idea, using the lower 2 bits of the list_head's prev pointer (last bit indicates whether the
list_headbelongs to a page or azswap_entry, and the second to last bit was repurposed for the second chance algorithm).
Thanks a lot for spending time looking into this, and sorry for the delayed resposne (I am technically on leave right now).
Quoted 5 lines
For a very high level overview what I did in the patch:
- When a page fails to compress, I remove the page mapping and tag both the xarray entry (tag == set lowest bit to 1) and the page'slist_headprev ptr, then store the page directly into the zswap LRU.
What do you mean by 'remove the page mapping'? Do you mean __remove_mapping()?
This is already called by reclaim, so I assume vmscan code hands over ownership of the page to zswap and doesn't call __remove_mapping(), so you end up doing that in zswap instead.
Quoted 6 lines
- In
zswap_load, we take the entry out of the xarray and check if it's tagged.
- If it is tagged, then instead of decompressing, we just copy the page's contents to the newly allocated page.
- (More details about how to teach vmscan /page_io/ list iterators how to handle this, but we can gloss over those details for now)
Quoted 7 lines
I have a working version, but have been holding off because I have only been seeing regressions. I wasn't really sure where they were coming from, but after going through some perf traces with Nhat, found out that the regressions come from the associated page faults that come from initially unmapping the page, and then re-allocating it for every load. This causes (1) more memcg flushing, and (2) extra allocations ==> more pressure ==> more reclaim, even though we only temporarily keep the extra page.
Hmm how is this worse than the status quo though? IIUC currently incompressible pages will skip zswap and go to the backing swapfile. Surely reading them from disk is slower than copying them?
Unless of course, writeback is disabled, in which case these pages are not being reclaimed at all today. In this case, it makes sense that reclaiming them makes accessing them slower, even if we don't actually need to decompress them.
I have a few thoughts in mind:
- As Nhat said, if we can keep the pages in the swapcache, we can avoid making a new allocation and copying the page. We'd need to move it back from zswap LRUs to the reclaim LRUs though.
- One advantage of keeping incompressible pages in zswap is preserving LRU ordering. IOW, if some compressible pages go to zswap first (old), then some incompressible pages (new), then the old compressible pages should go to disk via writeback first. Otherwise, accessing the hotter incompressible pages will be slower than accessing the colder compressible pages. This happens today because incompressible pages go straight to disk.
The above will only materialize for a workload that has writeback enabled and a mixture of both incompressible and compressible workingset.
The other advantage, as you mention below, is preventing repeatedly sending incompressible pages to zswap when writeback is disabled, but that could be offset by the extra cost of allocations/copying.
- The above being said, we should not regress workloads that have writeback disabled, so we either need to keep the pages in the swapcache to avoid the extra allocations/copies -- or avoid storing the pages in zswap completely if writeback is disabled. If writeback is disabled and the page is incompressible, we could probably just put it in the unevictable LRU because that's what it really is. We'd need to make sure we remove it when it becomes compressible again. The first approach is probably simpler.
Quoted 17 lines
Just wanted to put this here in case you were still thinking about this idea. What do you think? Ideally, there would be a way to keep the page around in the zswap LRU, but do not have to re-allocate a new page on a fault, but this seems like a bigger task.
Ultimately the goal is to prevent an incompressible page from hoarding the compression algorithm on multiple reclaim attempts, but if we are spending more time by allocating new pages... maybe this isn't the correct approach :(
Please let me know if you have any thoughts on this : -) Have a great day!
JoshuaSent using hkml (https://github.com/sjp38/hackermail)
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
I thought before about having a special
list_headthat allows us to use the lower bits of the pointers as markers, similar to the xarray. The markers can be used to place different objects on the same list. We can have a list that is a mixture of struct page and structzswap_entry. I never pursued this idea, and I am sure someone will scream at me for suggesting it. Maybe there is a less convoluted way to keep the LRU ordering intact without allocating memory on the reclaim path.
Quoted 7 lines
Quoted 4 lines
So I've implemented your idea, using the lower 2 bits of the list_head's prev pointer (last bit indicates whether the
list_headbelongs to a page or azswap_entry, and the second to last bit was repurposed for the second chance algorithm).Thanks a lot for spending time looking into this, and sorry for the delayed resposne (I am technically on leave right now).
Hi Yosry,
Thank you for getting back to me! I hope you are enjoying your leave : -)
Quoted 7 lines
Quoted 4 lines
For a very high level overview what I did in the patch:
- When a page fails to compress, I remove the page mapping and tag both the xarray entry (tag == set lowest bit to 1) and the page'slist_headprev ptr, then store the page directly into the zswap LRU.What do you mean by 'remove the page mapping'? Do you mean
__remove_mapping()?
Yes -- but I am calling remove_mapping() to unfreeze with a refcount of 1 (zswap is now the sole owner of the page).
Quoted 3 lines
This is already called by reclaim, so I assume vmscan code hands over ownership of the page to zswap and doesn't call
__remove_mapping(), so you end up doing that in zswap instead.
Yes! I changed reclaim logic to be aware that zswap can do this, so there is a new switch case that simply continues through the folio list when zswap steals the incompressible page (but we don't want to drop the page).
Quoted 14 lines
Quoted 6 lines
- In
zswap_load, we take the entry out of the xarray and check if it's tagged.
- If it is tagged, then instead of decompressing, we just copy the page's contents to the newly allocated page.
- (More details about how to teach vmscan /page_io/ list iterators how to handle this, but we can gloss over those details for now)Quoted 7 lines
I have a working version, but have been holding off because I have only been seeing regressions. I wasn't really sure where they were coming from, but after going through some perf traces with Nhat, found out that the regressions come from the associated page faults that come from initially unmapping the page, and then re-allocating it for every load. This causes (1) more memcg flushing, and (2) extra allocations ==> more pressure ==> more reclaim, even though we only temporarily keep the extra page.
Quoted 8 lines
Hmm how is this worse than the status quo though? IIUC currently incompressible pages will skip zswap and go to the backing swapfile. Surely reading them from disk is slower than copying them?
Unless of course, writeback is disabled, in which case these pages are not being reclaimed at all today. In this case, it makes sense that reclaiming them makes accessing them slower, even if we don't actually need to decompress them.
Yes, sorry for the ambiguity -- this was specifically for the writeback disabled case. My focus currently is on reducing the amount of CPU cycles spent stuck on trying to compress incompressible pages.
Quoted 5 lines
I have a few thoughts in mind:
- As Nhat said, if we can keep the pages in the swapcache, we can avoid making a new allocation and copying the page. We'd need to move it back from zswap LRUs to the reclaim LRUs though.
Yes, Nhat and Shakeel have both offered the same perspective. I'm currently working on this approach with help from Nhat.
Quoted 11 lines
- One advantage of keeping incompressible pages in zswap is preserving LRU ordering. IOW, if some compressible pages go to zswap first (old), then some incompressible pages (new), then the old compressible pages should go to disk via writeback first. Otherwise, accessing the hotter incompressible pages will be slower than accessing the colder compressible pages. This happens today because incompressible pages go straight to disk.
The above will only materialize for a workload that has writeback enabled and a mixture of both incompressible and compressible workingset.
This makes sense to me. I'll take this into consideration when writing benchmarks for this patch!
Quoted 3 lines
The other advantage, as you mention below, is preventing repeatedly sending incompressible pages to zswap when writeback is disabled, but that could be offset by the extra cost of allocations/copying.
Yes -- hopefully, keeping it in swapcache allows us to reap the benefits of both worlds, minus the duplicated allocation / copying.
Quoted 8 lines
- The above being said, we should not regress workloads that have writeback disabled, so we either need to keep the pages in the swapcache to avoid the extra allocations/copies -- or avoid storing the pages in zswap completely if writeback is disabled. If writeback is disabled and the page is incompressible, we could probably just put it in the unevictable LRU because that's what it really is. We'd need to make sure we remove it when it becomes compressible again. The first approach is probably simpler.
This is a good point. While I don't have the numbers to back this up, I have an intuition that this patch really sees the most benefits from reducing CPU time spent trying to compress incompressible pages, rather than from maintaining a better LRU ordering. For that reason I also suspect that we see much better performance gains when writeback is disabled. Following that logic... maybe there is some future work to be done that just moves these incompressible pages to an unevictable LRU when writeback is disabled?
For now, I'm still experimenting with keeping the page in swapcache. I'll be sure to report back with cool findings! Thank you again for your review of this idea Yosry, I hope you have a great day!
Joshua