Skip to contentSkip to conversation

mm/zswap: fix inconsistency when zswap_store_page() fails

PATCH Applied Hyeonggon Yoo 3 participants 5 messages Jan 29, 2025 – Jan 31, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 mm/zswap: fix inconsistency when zswap_store_page() fails

Commit b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()") skips charging any zswap entries when it failed to zswap the entire folio.

However, when some base pages are zswapped but it failed to zswap the entire folio, the zswap operation is rolled back.
When freeing zswap entries for those pages, zswap_entry_free() uncharges the zswap entries that were not previously charged, causing zswap charging to become inconsistent.

This inconsistency triggers two warnings with following steps:
# On a machine with 64GiB of RAM and 36GiB of zswap

  $ stress-ng --bigheap 2 # wait until the OOM-killer kills stress-ng
  $ sudo reboot

The two warnings are:
in mm/memcontrol.c:163, function obj_cgroup_release():
WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));

in mm/page_counter.c:60, function page_counter_cancel():
if (WARN_ONCE(new < 0, "page_counter underflow: %ld nr_pages=%lu\n",
new, nr_pages))

zswap_stored_pages also becomes inconsistent in the same way.

As suggested by Kanchana, increment zswap_stored_pages and charge zswap entries within zswap_store_page() when it succeeds. This way, zswap_entry_free() will decrement the counter and uncharge the entries when it failed to zswap the entire folio.

While this could potentially be optimized by batching objcg charging and incrementing the counter, let's focus on fixing the bug this time and leave the optimization for later after some evaluation.

After resolving the inconsistency, the warnings disappear.

Fixes b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()")

---

v2 -> v3:
- Adjusted Kanchana's feedback:
- Fixed inconsistency in zswap_stored_pages
- Now objcg charging and incrementing zswap_store_pages is done within zswap_stored_pages, one by one

 mm/zswap.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
mm/zswap.c +4 -6
index 6504174fbc6a..f0bd962bffd5 100644
@@ -1504,11 +1504,14 @@ static ssize_t zswap_store_page(struct page *page,
 	entry->pool = pool;
 	entry->swpentry = page_swpentry;
 	entry->objcg = objcg;
+	if (objcg)
+		obj_cgroup_charge_zswap(objcg, entry->length);
 	entry->referenced = true;
 	if (entry->length) {
 		INIT_LIST_HEAD(&entry->lru);
 		zswap_lru_add(&zswap_list_lru, entry);
 	}
+	atomic_long_inc(&zswap_stored_pages);
 
 	return entry->length;
 
@@ -1526,7 +1529,6 @@ bool zswap_store(struct folio *folio)
 	struct obj_cgroup *objcg = NULL;
 	struct mem_cgroup *memcg = NULL;
 	struct zswap_pool *pool;
-	size_t compressed_bytes = 0;
 	bool ret = false;
 	long index;
 
@@ -1569,15 +1571,11 @@ bool zswap_store(struct folio *folio)
 		bytes = zswap_store_page(page, objcg, pool);
 		if (bytes < 0)
 			goto put_pool;
-		compressed_bytes += bytes;
 	}
 
-	if (objcg) {
-		obj_cgroup_charge_zswap(objcg, compressed_bytes);
+	if (objcg)
 		count_objcg_events(objcg, ZSWPOUT, nr_pages);
-	}
 
-	atomic_long_add(nr_pages, &zswap_stored_pages);
 	count_vm_events(ZSWPOUT, nr_pages);
 
 	ret = true;
Signature · 1 line
2.47.1
Acked-by on v3 1/1
#1 lore raw
Quoted 34 lines · Hyeonggon Yoo

Commit b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()") skips charging any zswap entries when it failed to zswap the entire folio.

However, when some base pages are zswapped but it failed to zswap the entire folio, the zswap operation is rolled back.
When freeing zswap entries for those pages, zswap_entry_free() uncharges the zswap entries that were not previously charged, causing zswap charging to become inconsistent.

This inconsistency triggers two warnings with following steps:
# On a machine with 64GiB of RAM and 36GiB of zswap

  $ stress-ng --bigheap 2 # wait until the OOM-killer kills stress-ng
  $ sudo reboot

The two warnings are:
in mm/memcontrol.c:163, function obj_cgroup_release():
WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));

in mm/page_counter.c:60, function page_counter_cancel():
if (WARN_ONCE(new < 0, "page_counter underflow: %ld nr_pages=%lu\n",
new, nr_pages))

zswap_stored_pages also becomes inconsistent in the same way.

As suggested by Kanchana, increment zswap_stored_pages and charge zswap entries within zswap_store_page() when it succeeds. This way, zswap_entry_free() will decrement the counter and uncharge the entries when it failed to zswap the entire folio.

While this could potentially be optimized by batching objcg charging and incrementing the counter, let's focus on fixing the bug this time and leave the optimization for later after some evaluation.

Quoted 7 lines

After resolving the inconsistency, the warnings disappear.

Fixes b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()")

I have a few nits, but generally:

Quoted 16 lines

---

v2 -> v3:
- Adjusted Kanchana's feedback:
- Fixed inconsistency in zswap_stored_pages
- Now objcg charging and incrementing zswap_store_pages is done within zswap_stored_pages, one by one

 mm/zswap.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
mm/zswap.c +0 -0
index 6504174fbc6a..f0bd962bffd5 100644

mm/zswap.c static ssize_t zswap_store_page(struct page *page, · line 1508

Quoted 5 lines
mm/zswap.c +2 -0
@@ -1504,11 +1504,14 @@ static ssize_t zswap_store_page(struct page *page,
 	entry->pool = pool;
 	entry->swpentry = page_swpentry;
 	entry->objcg = objcg;
+	if (objcg)
+		obj_cgroup_charge_zswap(objcg, entry->length);

nit: This can be moved to the existing if (objcg) check where we call obj_cgroup_get(). At that point there shouldn't be a possibility of failure. If you want to keep it here to make it obvious that we only charge when we set entry->objcg that's fine, but we can probably move obj_cgroup_get() here as well in this case.

mm/zswap.c static ssize_t zswap_store_page(struct page *page, · line 1514

Quoted 6 lines
mm/zswap.c +1 -0
 	entry->referenced = true;
 	if (entry->length) {
 		INIT_LIST_HEAD(&entry->lru);
 		zswap_lru_add(&zswap_list_lru, entry);
 	}
+	atomic_long_inc(&zswap_stored_pages);

nit: If you keep the charging after setting entry->objcg because that's when the freeing path will uncharge, then perhaps you want to move this after the tree store is successful, because at that point the freeing path will decrement the counter.

Quoted 11 lines
mm/zswap.c +0 -1
 	return entry->length;
 
@@ -1526,7 +1529,6 @@ bool zswap_store(struct folio *folio)
 	struct obj_cgroup *objcg = NULL;
 	struct mem_cgroup *memcg = NULL;
 	struct zswap_pool *pool;
-	size_t compressed_bytes = 0;
 	bool ret = false;
 	long index;
 

mm/zswap.c bool zswap_store(struct folio *folio) · line 1573

Quoted 3 lines
mm/zswap.c +0 -0
@@ -1569,15 +1571,11 @@ bool zswap_store(struct folio *folio)
 		bytes = zswap_store_page(page, objcg, pool);
 		if (bytes < 0)
 			goto put_pool;

Do we need 'bytes' anymore? I think we don't even need zswap_store_page() to return the compressed size anymore. Seems like a boolean will suffice.

Quoted 17 lines
mm/zswap.c +1 -5
-		compressed_bytes += bytes;
 	}
 
-	if (objcg) {
-		obj_cgroup_charge_zswap(objcg, compressed_bytes);
+	if (objcg)
 		count_objcg_events(objcg, ZSWPOUT, nr_pages);
-	}
 
-	atomic_long_add(nr_pages, &zswap_stored_pages);
 	count_vm_events(ZSWPOUT, nr_pages);
 
 	ret = true;
Signature · 1 line
2.47.1
#2 lore raw

PATCH 1/1 mm/zswap: refactor zswap_store_page()

In reply to Hyeonggon Yoo

Change zswap_store_page() to return a boolean value, where true indicates the page is successfully stored in zswap, and false otherwise.

Since zswap_store_page() no longer returns the size of the entry, remove 'bytes' variable from zswap_store(), and jump to the put_pool label when it fails.

Move the objcg charging and the incrementing of zswap_store_pages just after the tree store is successful.
At that point, there's no chance of failure, and the freeing path will certainly uncharge zswap memory and decrement the counter.

---
 mm/zswap.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
mm/zswap.c +15 -16
index f0bd962bffd5..ac9d299e7d0c 100644
@@ -1445,9 +1445,9 @@ static void shrink_worker(struct work_struct *w)
 * main API
 **********************************/
 
-static ssize_t zswap_store_page(struct page *page,
-				struct obj_cgroup *objcg,
-				struct zswap_pool *pool)
+static bool zswap_store_page(struct page *page,
+			     struct obj_cgroup *objcg,
+			     struct zswap_pool *pool)
 {
 	swp_entry_t page_swpentry = page_swap_entry(page);
 	struct zswap_entry *entry, *old;
@@ -1456,7 +1456,7 @@ static ssize_t zswap_store_page(struct page *page,
 	entry = zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page));
 	if (!entry) {
 		zswap_reject_kmemcache_fail++;
-		return -EINVAL;
+		return false;
 	}
 
 	if (!zswap_compress(page, entry, pool))
@@ -1483,13 +1483,17 @@ static ssize_t zswap_store_page(struct page *page,
 
 	/*
 	 * The entry is successfully compressed and stored in the tree, there is
-	 * no further possibility of failure. Grab refs to the pool and objcg.
-	 * These refs will be dropped by zswap_entry_free() when the entry is
-	 * removed from the tree.
+	 * no further possibility of failure. Grab refs to the pool and objcg,
+	 * charge zswap memory, and increment zswap_stored_pages.
+	 * The opposite actions will be performed by zswap_entry_free()
+	 * when the entry is removed from the tree.
 	 */
 	zswap_pool_get(pool);
-	if (objcg)
+	if (objcg) {
 		obj_cgroup_get(objcg);
+		obj_cgroup_charge_zswap(objcg, entry->length);
+	}
+	atomic_long_inc(&zswap_stored_pages);
 
 	/*
 	 * We finish initializing the entry while it's already in xarray.
@@ -1504,22 +1508,19 @@ static ssize_t zswap_store_page(struct page *page,
 	entry->pool = pool;
 	entry->swpentry = page_swpentry;
 	entry->objcg = objcg;
-	if (objcg)
-		obj_cgroup_charge_zswap(objcg, entry->length);
 	entry->referenced = true;
 	if (entry->length) {
 		INIT_LIST_HEAD(&entry->lru);
 		zswap_lru_add(&zswap_list_lru, entry);
 	}
-	atomic_long_inc(&zswap_stored_pages);
 
-	return entry->length;
+	return true;
 
 store_failed:
 	zpool_free(pool->zpool, entry->handle);
 compress_failed:
 	zswap_entry_cache_free(entry);
-	return -EINVAL;
+	return false;
 }
 
 bool zswap_store(struct folio *folio)
@@ -1566,10 +1567,8 @@ bool zswap_store(struct folio *folio)
 
 	for (index = 0; index < nr_pages; ++index) {
 		struct page *page = folio_page(folio, index);
-		ssize_t bytes;
 
-		bytes = zswap_store_page(page, objcg, pool);
-		if (bytes < 0)
+		if (!zswap_store_page(page, objcg, pool))
 			goto put_pool;
 	}
 
Signature · 1 line
2.47.1
#3 lore raw

Re: [PATCH mm-hotfixes] mm/zswap: refactor zswap_store_page()

Quoted 8 lines · Hyeonggon Yoo

Change zswap_store_page() to return a boolean value, where true indicates the page is successfully stored in zswap, and false otherwise.

Since zswap_store_page() no longer returns the size of the entry, remove 'bytes' variable from zswap_store(), and jump to the put_pool label when it fails.

Quoted 8 lines

Move the objcg charging and the incrementing of zswap_store_pages just after the tree store is successful.
At that point, there's no chance of failure, and the freeing path will certainly uncharge zswap memory and decrement the counter.

---

Uh, I was trying to send it in-reply-to this:
https://lore.kernel.org/linux-mm/[email protected]

Andrew, could you please fold it into the patch?
No major functional difference, just adjusted Yosry's comments.

Quoted 91 lines
 mm/zswap.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
mm/zswap.c +15 -17
index f0bd962bffd5..ac9d299e7d0c 100644
@@ -1445,9 +1445,9 @@ static void shrink_worker(struct work_struct *w)
 * main API
 **********************************/
 
-static ssize_t zswap_store_page(struct page *page,
-                               struct obj_cgroup *objcg,
-                               struct zswap_pool *pool)
+static bool zswap_store_page(struct page *page,
+                            struct obj_cgroup *objcg,
+                            struct zswap_pool *pool)
 {
        swp_entry_t page_swpentry = page_swap_entry(page);
        struct zswap_entry *entry, *old;
@@ -1456,7 +1456,7 @@ static ssize_t zswap_store_page(struct page *page,
        entry = zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page));
        if (!entry) {
                zswap_reject_kmemcache_fail++;
-               return -EINVAL;
+               return false;
        }
 
        if (!zswap_compress(page, entry, pool))
@@ -1483,13 +1483,17 @@ static ssize_t zswap_store_page(struct page *page,
 
        /*
         * The entry is successfully compressed and stored in the tree, there is
-        * no further possibility of failure. Grab refs to the pool and objcg.
-        * These refs will be dropped by zswap_entry_free() when the entry is
-        * removed from the tree.
+        * no further possibility of failure. Grab refs to the pool and objcg,
+        * charge zswap memory, and increment zswap_stored_pages.
+        * The opposite actions will be performed by zswap_entry_free()
+        * when the entry is removed from the tree.
         */
        zswap_pool_get(pool);
-       if (objcg)
+       if (objcg) {
                obj_cgroup_get(objcg);
+               obj_cgroup_charge_zswap(objcg, entry->length);
+       }
+       atomic_long_inc(&zswap_stored_pages);
 
        /*
         * We finish initializing the entry while it's already in xarray.
@@ -1504,22 +1508,19 @@ static ssize_t zswap_store_page(struct page *page,
        entry->pool = pool;
        entry->swpentry = page_swpentry;
        entry->objcg = objcg;
-       if (objcg)
-               obj_cgroup_charge_zswap(objcg, entry->length);
        entry->referenced = true;
        if (entry->length) {
                INIT_LIST_HEAD(&entry->lru);
                zswap_lru_add(&zswap_list_lru, entry);
        }
-       atomic_long_inc(&zswap_stored_pages);
 
-       return entry->length;
+       return true;
 
 store_failed:
        zpool_free(pool->zpool, entry->handle);
 compress_failed:
        zswap_entry_cache_free(entry);
-       return -EINVAL;
+       return false;
 }
 
 bool zswap_store(struct folio *folio)
@@ -1566,10 +1567,8 @@ bool zswap_store(struct folio *folio)
 
        for (index = 0; index < nr_pages; ++index) {
                struct page *page = folio_page(folio, index);
-               ssize_t bytes;
 
-               bytes = zswap_store_page(page, objcg, pool);
-               if (bytes < 0)
+               if (!zswap_store_page(page, objcg, pool))
                        goto put_pool;
        }
 
--

2.47.1

maintainer Acked-by on v3 1/1
#4 lore raw
In reply to Hyeonggon Yoo
Quoted 20 lines · Hyeonggon Yoo

Commit b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()") skips charging any zswap entries when it failed to zswap the entire folio.

However, when some base pages are zswapped but it failed to zswap the entire folio, the zswap operation is rolled back.
When freeing zswap entries for those pages, zswap_entry_free() uncharges the zswap entries that were not previously charged, causing zswap charging to become inconsistent.

This inconsistency triggers two warnings with following steps:
# On a machine with 64GiB of RAM and 36GiB of zswap

  $ stress-ng --bigheap 2 # wait until the OOM-killer kills stress-ng
  $ sudo reboot

The two warnings are:
in mm/memcontrol.c:163, function obj_cgroup_release():
WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));

Quoted 5 lines

in mm/page_counter.c:60, function page_counter_cancel():
if (WARN_ONCE(new < 0, "page_counter underflow: %ld nr_pages=%lu\n",
new, nr_pages))

zswap_stored_pages also becomes inconsistent in the same way.

Nice catch haha.

Quoted 10 lines

As suggested by Kanchana, increment zswap_stored_pages and charge zswap entries within zswap_store_page() when it succeeds. This way, zswap_entry_free() will decrement the counter and uncharge the entries when it failed to zswap the entire folio.

While this could potentially be optimized by batching objcg charging and incrementing the counter, let's focus on fixing the bug this time and leave the optimization for later after some evaluation.

Quoted 7 lines

After resolving the inconsistency, the warnings disappear.

Fixes b7c0ccdfbafd ("mm: zswap: support large folios in zswap_store()")

With your fixlet applied: