mm/zswap: fix inconsistency when zswap_store_page() fails
Subsystems (3)
PATCH v3 1/1 mm/zswap: fix inconsistency when zswap_store_page() fails
Commit b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()") skips charging any zswap entries when it failed to zswap the entire folio.
However, when some base pages are zswapped but it failed to zswap the entire folio, the zswap operation is rolled back.
When freeing zswap entries for those pages, zswap_entry_free() uncharges the zswap entries that were not previously charged, causing zswap charging to become inconsistent.
This inconsistency triggers two warnings with following steps:
# On a machine with 64GiB of RAM and 36GiB of zswap
$ stress-ng --bigheap 2 # wait until the OOM-killer kills stress-ng $ sudo reboot
The two warnings are:
in mm/memcontrol.c:163, function obj_cgroup_release():
WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));
in mm/page_counter.c:60, function page_counter_cancel():
if (WARN_ONCE(new < 0, "page_counter underflow: %ld nr_pages=%lu\n",
new, nr_pages))
zswap_stored_pages also becomes inconsistent in the same way.
As suggested by Kanchana, increment zswap_stored_pages and charge zswap entries within zswap_store_page() when it succeeds. This way, zswap_entry_free() will decrement the counter and uncharge the entries when it failed to zswap the entire folio.
While this could potentially be optimized by batching objcg charging and incrementing the counter, let's focus on fixing the bug this time and leave the optimization for later after some evaluation.
After resolving the inconsistency, the warnings disappear.
---
v2 -> v3:
- Adjusted Kanchana's feedback:
- Fixed inconsistency in zswap_stored_pages
- Now objcg charging and incrementing zswap_store_pages is done within zswap_stored_pages, one by one
mm/zswap.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
mm/zswap.c +4 -6
@@ -1504,11 +1504,14 @@ static ssize_t zswap_store_page(struct page *page, entry->pool = pool; entry->swpentry = page_swpentry; entry->objcg = objcg; +if (objcg) +obj_cgroup_charge_zswap (objcg, entry->length); entry->referenced = true;if (entry->length) {INIT_LIST_HEAD (&entry->lru);zswap_lru_add (&zswap_list_lru, entry); } +atomic_long_inc (&zswap_stored_pages);return entry->length;
@@ -1526,7 +1529,6 @@ bool zswap_store(struct folio *folio)struct obj_cgroup *objcg =NULL ;struct mem_cgroup *memcg =NULL ;struct zswap_pool *pool; -size_t compressed_bytes =0 ;bool ret = false;long index;
@@ -1569,15 +1571,11 @@ bool zswap_store(struct folio *folio) bytes =zswap_store_page (page, objcg, pool);if (bytes <0 ) goto put_pool; - compressed_bytes += bytes; } -if (objcg) { -obj_cgroup_charge_zswap (objcg, compressed_bytes); +if (objcg)count_objcg_events (objcg,ZSWPOUT , nr_pages); - } -atomic_long_add (nr_pages, &zswap_stored_pages);count_vm_events (ZSWPOUT , nr_pages); ret = true;
Signature · 1 line
2.47.1
Quoted 34 lines · Hyeonggon Yoo
Commit
b7c0ccdfbafd("mm: zswap: support large folios inzswap_store()") skips charging any zswap entries when it failed to zswap the entire folio.However, when some base pages are zswapped but it failed to zswap the entire folio, the zswap operation is rolled back.
When freeing zswap entries for those pages,zswap_entry_free()uncharges the zswap entries that were not previously charged, causing zswap charging to become inconsistent.This inconsistency triggers two warnings with following steps:
# On a machine with 64GiB of RAM and 36GiB of zswap$ stress-ng --bigheap 2 # wait until the OOM-killer kills stress-ng $ sudo rebootThe two warnings are:
inmm/memcontrol.c:163, functionobj_cgroup_release():
WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));in
mm/page_counter.c:60, functionpage_counter_cancel():
if (WARN_ONCE(new < 0, "page_counterunderflow: %ld nr_pages=%lu\n",
new,nr_pages))
zswap_stored_pagesalso becomes inconsistent in the same way.As suggested by Kanchana, increment
zswap_stored_pagesand charge zswap entries withinzswap_store_page()when it succeeds. This way,zswap_entry_free()will decrement the counter and uncharge the entries when it failed to zswap the entire folio.While this could potentially be optimized by batching objcg charging and incrementing the counter, let's focus on fixing the bug this time and leave the optimization for later after some evaluation.
Quoted 7 lines
After resolving the inconsistency, the warnings disappear.
Fixes b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()")Co-developed-by Kanchana P Sridhar <[email protected]>Signed-off-by Kanchana P Sridhar <[email protected]>Signed-off-by Hyeonggon Yoo <[email protected]>
I have a few nits, but generally:
Quoted 16 lines
---
v2 -> v3:
- Adjusted Kanchana's feedback:
- Fixed inconsistency inzswap_stored_pages
- Now objcg charging and incrementingzswap_store_pagesis done withinzswap_stored_pages, one by onemm/zswap.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)mm/zswap.c +0
-0
mm/zswap.c static ssize_t zswap_store_page(struct page *page, · line 1508
Quoted 5 lines
mm/zswap.c +2
-0@@ -1504,11 +1504,14 @@ static ssize_t zswap_store_page(struct page *page, entry->pool = pool; entry->swpentry = page_swpentry; entry->objcg = objcg; +if (objcg) +obj_cgroup_charge_zswap (objcg, entry->length);
nit: This can be moved to the existing if (objcg) check where we call obj_cgroup_get(). At that point there shouldn't be a possibility of failure. If you want to keep it here to make it obvious that we only charge when we set entry->objcg that's fine, but we can probably move obj_cgroup_get() here as well in this case.
mm/zswap.c static ssize_t zswap_store_page(struct page *page, · line 1514
Quoted 6 lines
mm/zswap.c +1
-0entry->referenced = true;if (entry->length) {INIT_LIST_HEAD (&entry->lru);zswap_lru_add (&zswap_list_lru, entry); } +atomic_long_inc (&zswap_stored_pages);
nit: If you keep the charging after setting entry->objcg because that's when the freeing path will uncharge, then perhaps you want to move this after the tree store is successful, because at that point the freeing path will decrement the counter.
Quoted 11 lines
mm/zswap.c +0
-1return entry->length;@@ -1526,7 +1529,6 @@ bool zswap_store(struct folio *folio)struct obj_cgroup *objcg =NULL ;struct mem_cgroup *memcg =NULL ;struct zswap_pool *pool; -size_t compressed_bytes =0 ;bool ret = false;long index;
mm/zswap.c bool zswap_store(struct folio *folio) · line 1573
Quoted 3 lines
mm/zswap.c +0
-0@@ -1569,15 +1571,11 @@ bool zswap_store(struct folio *folio) bytes =zswap_store_page (page, objcg, pool);if (bytes <0 ) goto put_pool;
Do we need 'bytes' anymore? I think we don't even need zswap_store_page() to return the compressed size anymore. Seems like a boolean will suffice.
Quoted 17 lines
mm/zswap.c +1
-5- compressed_bytes += bytes; } -if (objcg) { -obj_cgroup_charge_zswap (objcg, compressed_bytes); +if (objcg)count_objcg_events (objcg,ZSWPOUT , nr_pages); - } -atomic_long_add (nr_pages, &zswap_stored_pages);count_vm_events (ZSWPOUT , nr_pages); ret = true;Signature · 1 line
2.47.1
PATCH 1/1 mm/zswap: refactor zswap_store_page()
In reply to Hyeonggon YooChange zswap_store_page() to return a boolean value, where true indicates the page is successfully stored in zswap, and false otherwise.
Since zswap_store_page() no longer returns the size of the entry, remove 'bytes' variable from zswap_store(), and jump to the put_pool label when it fails.
Move the objcg charging and the incrementing of zswap_store_pages just after the tree store is successful.
At that point, there's no chance of failure, and the freeing path will certainly uncharge zswap memory and decrement the counter.
--- mm/zswap.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)
mm/zswap.c +15 -16
@@ -1445,9 +1445,9 @@ static void shrink_worker(struct work_struct *w)* main API **********************************/ -static ssize_t zswap_store_page (struct page *page, -struct obj_cgroup *objcg, -struct zswap_pool *pool) +static bool zswap_store_page (struct page *page, +struct obj_cgroup *objcg, +struct zswap_pool *pool) {swp_entry_t page_swpentry =page_swap_entry (page);struct zswap_entry *entry, *old;
@@ -1456,7 +1456,7 @@ static ssize_t zswap_store_page(struct page *page, entry =zswap_entry_cache_alloc (GFP_KERNEL ,page_to_nid (page));if (!entry) { zswap_reject_kmemcache_fail++; -return -EINVAL ; +return false; }if (!zswap_compress (page, entry, pool))
@@ -1483,13 +1483,17 @@ static ssize_t zswap_store_page(struct page *page,/* * The entry is successfully compressed and stored in the tree, there is -* no further possibility of failure. Grab refs to the pool and objcg. -* These refs will be dropped by zswap_entry_free() when the entry is -* removed from the tree. +* no further possibility of failure. Grab refs to the pool and objcg, +* charge zswap memory, and increment zswap_stored_pages. +* The opposite actions will be performed by zswap_entry_free() +* when the entry is removed from the tree. */ zswap_pool_get (pool); -if (objcg) +if (objcg) {obj_cgroup_get (objcg); +obj_cgroup_charge_zswap (objcg, entry->length); + } +atomic_long_inc (&zswap_stored_pages);/* * We finish initializing the entry while it's already in xarray.
@@ -1504,22 +1508,19 @@ static ssize_t zswap_store_page(struct page *page, entry->pool = pool; entry->swpentry = page_swpentry; entry->objcg = objcg; -if (objcg) -obj_cgroup_charge_zswap (objcg, entry->length); entry->referenced = true;if (entry->length) {INIT_LIST_HEAD (&entry->lru);zswap_lru_add (&zswap_list_lru, entry); } -atomic_long_inc (&zswap_stored_pages); -return entry->length; +return true; store_failed:zpool_free (pool->zpool, entry->handle); compress_failed:zswap_entry_cache_free (entry); -return -EINVAL ; +return false; }bool zswap_store (struct folio *folio)
@@ -1566,10 +1567,8 @@ bool zswap_store(struct folio *folio)for (index =0 ; index < nr_pages; ++index) {struct page *page =folio_page (folio, index); -ssize_t bytes; - bytes =zswap_store_page (page, objcg, pool); -if (bytes <0 ) +if (!zswap_store_page (page, objcg, pool)) goto put_pool; }
Signature · 1 line
2.47.1
Re: [PATCH mm-hotfixes] mm/zswap: refactor zswap_store_page()
Quoted 8 lines · Hyeonggon Yoo
Change
zswap_store_page()to return a boolean value, where true indicates the page is successfully stored in zswap, and false otherwise.Since
zswap_store_page()no longer returns the size of the entry, remove 'bytes' variable fromzswap_store(), and jump to theput_poollabel when it fails.
Quoted 8 lines
Move the objcg charging and the incrementing of
zswap_store_pagesjust after the tree store is successful.
At that point, there's no chance of failure, and the freeing path will certainly uncharge zswap memory and decrement the counter.Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Hyeonggon Yoo <[email protected]>---
Uh, I was trying to send it in-reply-to this:
https://lore.kernel.org/linux-mm/[email protected]
Andrew, could you please fold it into the patch?
No major functional difference, just adjusted Yosry's comments.
Quoted 91 lines
mm/zswap.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)mm/zswap.c +15
-17@@ -1445,9 +1445,9 @@ static void shrink_worker(struct work_struct *w)* main API **********************************/ -static ssize_t zswap_store_page (struct page *page, -struct obj_cgroup *objcg, -struct zswap_pool *pool) +static bool zswap_store_page (struct page *page, +struct obj_cgroup *objcg, +struct zswap_pool *pool) {swp_entry_t page_swpentry =page_swap_entry (page);struct zswap_entry *entry, *old;@@ -1456,7 +1456,7 @@ static ssize_t zswap_store_page(struct page *page, entry =zswap_entry_cache_alloc (GFP_KERNEL ,page_to_nid (page));if (!entry) { zswap_reject_kmemcache_fail++; -return -EINVAL ; +return false; }if (!zswap_compress (page, entry, pool))@@ -1483,13 +1483,17 @@ static ssize_t zswap_store_page(struct page *page,/* * The entry is successfully compressed and stored in the tree, there is -* no further possibility of failure. Grab refs to the pool and objcg. -* These refs will be dropped by zswap_entry_free() when the entry is -* removed from the tree. +* no further possibility of failure. Grab refs to the pool and objcg, +* charge zswap memory, and increment zswap_stored_pages. +* The opposite actions will be performed by zswap_entry_free() +* when the entry is removed from the tree. */ zswap_pool_get (pool); -if (objcg) +if (objcg) {obj_cgroup_get (objcg); +obj_cgroup_charge_zswap (objcg, entry->length); + } +atomic_long_inc (&zswap_stored_pages);/* * We finish initializing the entry while it's already in xarray. @@ -1504,22 +1508,19 @@ static ssize_t zswap_store_page(struct page *page, entry->pool = pool; entry->swpentry = page_swpentry; entry->objcg = objcg; -if (objcg) -obj_cgroup_charge_zswap (objcg, entry->length); entry->referenced = true;if (entry->length) {INIT_LIST_HEAD (&entry->lru);zswap_lru_add (&zswap_list_lru, entry); } -atomic_long_inc (&zswap_stored_pages); -return entry->length; +return true; store_failed:zpool_free (pool->zpool, entry->handle); compress_failed:zswap_entry_cache_free (entry); -return -EINVAL ; +return false; }bool zswap_store (struct folio *folio)@@ -1566,10 +1567,8 @@ bool zswap_store(struct folio *folio)for (index =0 ; index < nr_pages; ++index) {struct page *page =folio_page (folio, index); -ssize_t bytes; - bytes =zswap_store_page (page, objcg, pool); -if (bytes <0 ) +if (!zswap_store_page (page, objcg, pool)) goto put_pool; } --2.47.1
Quoted 20 lines · Hyeonggon Yoo
Commit
b7c0ccdfbafd("mm: zswap: support large folios inzswap_store()") skips charging any zswap entries when it failed to zswap the entire folio.However, when some base pages are zswapped but it failed to zswap the entire folio, the zswap operation is rolled back.
When freeing zswap entries for those pages,zswap_entry_free()uncharges the zswap entries that were not previously charged, causing zswap charging to become inconsistent.This inconsistency triggers two warnings with following steps:
# On a machine with 64GiB of RAM and 36GiB of zswap$ stress-ng --bigheap 2 # wait until the OOM-killer kills stress-ng $ sudo rebootThe two warnings are:
inmm/memcontrol.c:163, functionobj_cgroup_release():
WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));
Quoted 5 lines
in
mm/page_counter.c:60, functionpage_counter_cancel():
if (WARN_ONCE(new < 0, "page_counterunderflow: %ld nr_pages=%lu\n",
new,nr_pages))
zswap_stored_pagesalso becomes inconsistent in the same way.
Nice catch haha.
Quoted 10 lines
As suggested by Kanchana, increment
zswap_stored_pagesand charge zswap entries withinzswap_store_page()when it succeeds. This way,zswap_entry_free()will decrement the counter and uncharge the entries when it failed to zswap the entire folio.While this could potentially be optimized by batching objcg charging and incrementing the counter, let's focus on fixing the bug this time and leave the optimization for later after some evaluation.
Quoted 7 lines
After resolving the inconsistency, the warnings disappear.
Fixes b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()")Co-developed-by Kanchana P Sridhar <[email protected]>Signed-off-by Kanchana P Sridhar <[email protected]>Signed-off-by Hyeonggon Yoo <[email protected]>
With your fixlet applied: