mm: zswap: Store large folios without splitting
Subsystems (2)
RFC PATCH 1/1 mm: zswap: Store large folios without splitting
Previously zswap would refuse to store any folio bigger than order-0, and therefore all of those folios would be sent directly to the swap file. This is a minor inconvenience since swap can currently only support order-0 and PMD-sized THP, but with the pending introduction of "small-sized THP", and corresponding changes to swapfile to support any order of folio, these large folios will become more prevalent and without this zswap change, zswap will become unusable. Independently of the "small-sized THP" feature, this change makes it possible to store existing PMD-sized THPs in zswap.
Modify zswap_store() to allow storing large folios. The function is split into 2 parts; zswap_store() does all the per-folio operations (i.e. checking there is enough space, etc). Then it calls a new helper, zswap_store_page(), for each page in the folio, which are stored as their own entries in the zswap pool. (These entries continue to be loaded back individually as single pages). If a store fails for any single page, then all previously successfully stored folio pages are invalidated.
---
I've tested this on arm64 (m2) with zswap enabled, and running vm-scalability's usemem across multiple cores from within a memory-constrained memcg to force high volumes of swap. I've also run mm selftests and observe no regressions (although there is nothing [z]swap specific there) - does zswap have any specific tests I should run?
This is based on mm-stable, since mm-unstable contains a zswap patch known to be buggy [1]. I thought it would be best to get comments on the shape, then do the rebase after that patch has been fixed.
For context, small-sized THP is being discussed here [2], and I'm working on changes to swapfile to support non-PMD-sized large folios here [3].
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
mm/zswap.c | 155 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 98 insertions(+), 57 deletions(-)
mm/zswap.c +98 -57
@@ -1188,18 +1188,17 @@ static void zswap_fill_page(void *ptr, unsigned long value)memset_l (page, value,PAGE_SIZE /sizeof (unsigned long )); } -bool zswap_store (struct folio *folio) +static bool zswap_store_page (struct folio *folio,long index, +struct obj_cgroup *objcg,struct zswap_pool *pool) {swp_entry_t swp = folio->swap;int type =swp_type (swp); -pgoff_t offset =swp_offset (swp); -struct page *page = &folio->page; +pgoff_t offset =swp_offset (swp) + index; +struct page *page =folio_page (folio, index);struct zswap_tree *tree = zswap_trees[type];struct zswap_entry *entry, *dupentry;struct scatterlist input, output;struct crypto_acomp_ctx *acomp_ctx; -struct obj_cgroup *objcg =NULL ; -struct zswap_pool *pool;struct zpool *zpool;unsigned int dlen =PAGE_SIZE ;unsigned long handle, value;
@@ -1208,51 +1207,11 @@ bool zswap_store(struct folio *folio)gfp_t gfp;int ret; -VM_WARN_ON_ONCE (!folio_test_locked (folio)); -VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); - -/* Large folios aren't supported */ -if (folio_test_large (folio)) -return false; - -if (!zswap_enabled || !tree) +/* entry keeps the references if successfully stored. */ +if (!zswap_pool_get (pool))return false; - -/* -* If this is a duplicate, it must be removed before attempting to store -* it, otherwise, if the store fails the old page won't be removed from -* the tree, and it might be written back overriding the new data. -*/ -spin_lock (&tree->lock); - dupentry =zswap_rb_search (&tree->rbroot, offset); -if (dupentry) { - zswap_duplicate_entry++; -zswap_invalidate_entry (tree, dupentry); - } -spin_unlock (&tree->lock); - -/* -* XXX: zswap reclaim does not work with cgroups yet. Without a -* cgroup-aware entry LRU, we will push out entries system-wide based on -* local cgroup limits. -*/ - objcg =get_obj_cgroup_from_folio (folio); -if (objcg && !obj_cgroup_may_zswap (objcg)) - goto reject; - -/* reclaim space if needed */ -if (zswap_is_full ()) { - zswap_pool_limit_hit++; - zswap_pool_reached_full = true; - goto shrink; - } - -if (zswap_pool_reached_full) { -if (!zswap_can_accept ()) - goto shrink; -else - zswap_pool_reached_full = false; - } +if (objcg) +obj_cgroup_get (objcg);/* allocate entry */ entry =zswap_entry_cache_alloc (GFP_KERNEL );
@@ -1260,6 +1219,8 @@ bool zswap_store(struct folio *folio) zswap_reject_kmemcache_fail++; goto reject; } + entry->objcg = objcg; + entry->pool = pool;if (zswap_same_filled_pages_enabled) { src =kmap_atomic (page);
@@ -1277,11 +1238,6 @@ bool zswap_store(struct folio *folio)if (!zswap_non_same_filled_pages_enabled) goto freepage; -/* if entry is successfully added, it keeps the reference */ - entry->pool =zswap_pool_current_get (); -if (!entry->pool) - goto freepage; -/* compress */ acomp_ctx =raw_cpu_ptr (entry->pool->acomp_ctx);
@@ -1337,7 +1293,6 @@ bool zswap_store(struct folio *folio) entry->length = dlen; insert_entry: - entry->objcg = objcg;if (objcg) {obj_cgroup_charge_zswap (objcg, entry->length);/* Account before objcg ref is moved to tree */
@@ -1373,19 +1328,105 @@ bool zswap_store(struct folio *folio) put_dstmem:mutex_unlock (acomp_ctx->mutex); -zswap_pool_put (entry->pool); freepage:zswap_entry_cache_free (entry); reject:if (objcg)obj_cgroup_put (objcg); +zswap_pool_put (pool);return false; +} +bool zswap_store (struct folio *folio) +{ +long nr_pages =folio_nr_pages (folio); +swp_entry_t swp = folio->swap; +int type =swp_type (swp); +pgoff_t offset =swp_offset (swp); +struct zswap_tree *tree = zswap_trees[type]; +struct zswap_entry *entry; +struct obj_cgroup *objcg =NULL ; +struct zswap_pool *pool; +bool ret = false; +long i; + +VM_WARN_ON_ONCE (!folio_test_locked (folio)); +VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); + +if (!zswap_enabled || !tree) +return false; + +/* +* If this is a duplicate, it must be removed before attempting to store +* it, otherwise, if the store fails the old page won't be removed from +* the tree, and it might be written back overriding the new data. +*/ +spin_lock (&tree->lock); +for (i =0 ; i < nr_pages; i++) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) { + zswap_duplicate_entry++; +zswap_invalidate_entry (tree, entry); + } + } +spin_unlock (&tree->lock); + +/* +* XXX: zswap reclaim does not work with cgroups yet. Without a +* cgroup-aware entry LRU, we will push out entries system-wide based on +* local cgroup limits. +*/ + objcg =get_obj_cgroup_from_folio (folio); +if (objcg && !obj_cgroup_may_zswap (objcg)) + goto put_objcg; + +/* reclaim space if needed */ +if (zswap_is_full ()) { + zswap_pool_limit_hit++; + zswap_pool_reached_full = true; + goto shrink; + } + +if (zswap_pool_reached_full) { +if (!zswap_can_accept ()) + goto shrink; +else + zswap_pool_reached_full = false; + } + + pool =zswap_pool_current_get (); +if (!pool) + goto put_objcg; + +/* +* Store each page of the folio as a separate entry. If we fail to store +* a page, unwind by removing all the previous pages we stored. +*/ +for (i =0 ; i < nr_pages; i++) { +if (!zswap_store_page (folio, i, objcg, pool)) { +spin_lock (&tree->lock); +for (i--; i >=0 ; i--) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) +zswap_invalidate_entry (tree, entry); + } +spin_unlock (&tree->lock); + goto put_pool; + } + } + + ret = true; +put_pool: +zswap_pool_put (pool); +put_objcg: +if (objcg) +obj_cgroup_put (objcg); +return ret; shrink: pool =zswap_pool_last_get ();if (pool && !queue_work (shrink_wq, &pool->shrink_work))zswap_pool_put (pool); - goto reject; + goto put_objcg; }bool zswap_load (struct folio *folio)
base-commit: 158978945f3173b8c1a88f8c5684a629736a57ac --
2.25.1
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Ryan,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on 158978945f3173b8c1a88f8c5684a629736a57ac]
url: https://github.com/intel-lab-lkp/linux/commits/Ryan-Roberts/mm-zswap-Store-large-folios-without-splitting/20231019-190740 base: 158978945f3173b8c1a88f8c5684a629736a57ac patch link: https://lore.kernel.org/r/20231019110543.3284654-1-ryan.roberts%40arm.com patch subject: [RFC PATCH v1] mm: zswap: Store large folios without splitting config: arc-randconfig-002-20231022 (https://download.01.org/0day-ci/archive/20231022/[email protected]/config) compiler: arceb-elf-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231022/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
mm/zswap.c: In function 'zswap_store_page':
>> mm/zswap.c:1214:17: error: implicit declaration of function 'obj_cgroup_get'; did you mean 'obj_cgroup_put'? [-Werror=implicit-function-declaration]
1214 | obj_cgroup_get(objcg);
| ^~~~~~~~~~~~~~
| obj_cgroup_putcc1: some warnings being treated as errors
vim +1214 mm/zswap.c
1190
1191 static bool zswap_store_page(struct folio *folio, long index,
1192 struct obj_cgroup *objcg, struct zswap_pool *pool)
1193 {
1194 swp_entry_t swp = folio->swap;
1195 int type = swp_type(swp);
1196 pgoff_t offset = swp_offset(swp) + index;
1197 struct page *page = folio_page(folio, index);
1198 struct zswap_tree *tree = zswap_trees[type];
1199 struct zswap_entry *entry, *dupentry;
1200 struct scatterlist input, output;
1201 struct crypto_acomp_ctx *acomp_ctx;
1202 struct zpool *zpool;
1203 unsigned int dlen = PAGE_SIZE;
1204 unsigned long handle, value;
1205 char *buf;
1206 u8 *src, *dst;
1207 gfp_t gfp;
1208 int ret;
1209
1210 /* entry keeps the references if successfully stored. */
1211 if (!zswap_pool_get(pool))
1212 return false;
1213 if (objcg)
> 1214 obj_cgroup_get(objcg);
1215
1216 /* allocate entry */
1217 entry = zswap_entry_cache_alloc(GFP_KERNEL);
1218 if (!entry) {
1219 zswap_reject_kmemcache_fail++;
1220 goto reject;
1221 }
1222 entry->objcg = objcg;
1223 entry->pool = pool;
1224
1225 if (zswap_same_filled_pages_enabled) {
1226 src = kmap_atomic(page);
1227 if (zswap_is_page_same_filled(src, &value)) {
1228 kunmap_atomic(src);
1229 entry->swpentry = swp_entry(type, offset);
1230 entry->length = 0;
1231 entry->value = value;
1232 atomic_inc(&zswap_same_filled_pages);
1233 goto insert_entry;
1234 }
1235 kunmap_atomic(src);
1236 }
1237
1238 if (!zswap_non_same_filled_pages_enabled)
1239 goto freepage;
1240
1241 /* compress */
1242 acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx);
1243
1244 mutex_lock(acomp_ctx->mutex);
1245
1246 dst = acomp_ctx->dstmem;
1247 sg_init_table(&input, 1);
1248 sg_set_page(&input, page, PAGE_SIZE, 0);
1249
1250 /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */
1251 sg_init_one(&output, dst, PAGE_SIZE * 2);
1252 acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen);
1253 /*
1254 * it maybe looks a little bit silly that we send an asynchronous request,
1255 * then wait for its completion synchronously. This makes the process look
1256 * synchronous in fact.
1257 * Theoretically, acomp supports users send multiple acomp requests in one
1258 * acomp instance, then get those requests done simultaneously. but in this
1259 * case, zswap actually does store and load page by page, there is no
1260 * existing method to send the second page before the first page is done
1261 * in one thread doing zwap.
1262 * but in different threads running on different cpu, we have different
1263 * acomp instance, so multiple threads can do (de)compression in parallel.
1264 */
1265 ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait);
1266 dlen = acomp_ctx->req->dlen;
1267
1268 if (ret)
1269 goto put_dstmem;
1270
1271 /* store */
1272 zpool = zswap_find_zpool(entry);
1273 gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM;
1274 if (zpool_malloc_support_movable(zpool))
1275 gfp |= __GFP_HIGHMEM | __GFP_MOVABLE;
1276 ret = zpool_malloc(zpool, dlen, gfp, &handle);
1277 if (ret == -ENOSPC) {
1278 zswap_reject_compress_poor++;
1279 goto put_dstmem;
1280 }
1281 if (ret) {
1282 zswap_reject_alloc_fail++;
1283 goto put_dstmem;
1284 }
1285 buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO);
1286 memcpy(buf, dst, dlen);
1287 zpool_unmap_handle(zpool, handle);
1288 mutex_unlock(acomp_ctx->mutex);
1289
1290 /* populate entry */
1291 entry->swpentry = swp_entry(type, offset);
1292 entry->handle = handle;
1293 entry->length = dlen;
1294
1295 insert_entry:
1296 if (objcg) {
1297 obj_cgroup_charge_zswap(objcg, entry->length);
1298 /* Account before objcg ref is moved to tree */
1299 count_objcg_event(objcg, ZSWPOUT);
1300 }
1301
1302 /* map */
1303 spin_lock(&tree->lock);
1304 /*
1305 * A duplicate entry should have been removed at the beginning of this
1306 * function. Since the swap entry should be pinned, if a duplicate is
1307 * found again here it means that something went wrong in the swap
1308 * cache.
1309 */
1310 while (zswap_rb_insert(&tree->rbroot, entry, &dupentry) == -EEXIST) {
1311 WARN_ON(1);
1312 zswap_duplicate_entry++;
1313 zswap_invalidate_entry(tree, dupentry);
1314 }
1315 if (entry->length) {
1316 spin_lock(&entry->pool->lru_lock);
1317 list_add(&entry->lru, &entry->pool->lru);
1318 spin_unlock(&entry->pool->lru_lock);
1319 }
1320 spin_unlock(&tree->lock);
1321
1322 /* update stats */
1323 atomic_inc(&zswap_stored_pages);
1324 zswap_update_total_size();
1325 count_vm_event(ZSWPOUT);
1326
1327 return true;
1328
1329 put_dstmem:
1330 mutex_unlock(acomp_ctx->mutex);
1331 freepage:
1332 zswap_entry_cache_free(entry);
1333 reject:
1334 if (objcg)
1335 obj_cgroup_put(objcg);
1336 zswap_pool_put(pool);
1337 return false;
1338 }
1339
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Nhat Pham
maintainerQuoted 20 lines · Ryan Roberts
Previously zswap would refuse to store any folio bigger than order-0, and therefore all of those folios would be sent directly to the swap file. This is a minor inconvenience since swap can currently only support order-0 and PMD-sized THP, but with the pending introduction of "small-sized THP", and corresponding changes to swapfile to support any order of folio, these large folios will become more prevalent and without this zswap change, zswap will become unusable. Independently of the "small-sized THP" feature, this change makes it possible to store existing PMD-sized THPs in zswap.
Modify
zswap_store()to allow storing large folios. The function is split into 2 parts;zswap_store()does all the per-folio operations (i.e. checking there is enough space, etc). Then it calls a new helper,zswap_store_page(), for each page in the folio, which are stored as their own entries in the zswap pool. (These entries continue to be loaded back individually as single pages). If a store fails for any single page, then all previously successfully stored folio pages are invalidated.
Quoted 7 lines
Signed-off-by Ryan Roberts <[email protected]>---
I've tested this on arm64 (m2) with zswap enabled, and running vm-scalability'susememacross multiple cores from within a memory-constrained memcg to force high volumes of swap. I've also run mm selftests and observe no regressions (although there is nothing [z]swap specific there) - does zswap have any specific tests I should run?
There is a zswap kselftest in the cgroup suite:
https://lore.kernel.org/all/[email protected]/
Regardless, I feel like this kind of change is probably better to be tested via stress tests anyway - allocating a bunch of anon memory with a certain pattern, making sure they're not corrupted after being zswapped out etc.
Quoted 178 lines
This is based on mm-stable, since mm-unstable contains a zswap patch known to be buggy [1]. I thought it would be best to get comments on the shape, then do the rebase after that patch has been fixed.
For context, small-sized THP is being discussed here [2], and I'm working on changes to swapfile to support non-PMD-sized large folios here [3].
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
Ryanmm/zswap.c | 155 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 98 insertions(+), 57 deletions(-)mm/zswap.c +44
-56@@ -1188,18 +1188,17 @@ static void zswap_fill_page(void *ptr, unsigned long value)memset_l (page, value,PAGE_SIZE /sizeof (unsigned long )); } -bool zswap_store (struct folio *folio) +static bool zswap_store_page (struct folio *folio,long index, +struct obj_cgroup *objcg,struct zswap_pool *pool) {swp_entry_t swp = folio->swap;int type =swp_type (swp); -pgoff_t offset =swp_offset (swp); -struct page *page = &folio->page; +pgoff_t offset =swp_offset (swp) + index; +struct page *page =folio_page (folio, index);struct zswap_tree *tree = zswap_trees[type];struct zswap_entry *entry, *dupentry;struct scatterlist input, output;struct crypto_acomp_ctx *acomp_ctx; -struct obj_cgroup *objcg =NULL ; -struct zswap_pool *pool;struct zpool *zpool;unsigned int dlen =PAGE_SIZE ;unsigned long handle, value;@@ -1208,51 +1207,11 @@ bool zswap_store(struct folio *folio)gfp_t gfp;int ret; -VM_WARN_ON_ONCE (!folio_test_locked (folio)); -VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); - -/* Large folios aren't supported */ -if (folio_test_large (folio)) -return false; - -if (!zswap_enabled || !tree) +/* entry keeps the references if successfully stored. */ +if (!zswap_pool_get (pool))return false; - -/* -* If this is a duplicate, it must be removed before attempting to store -* it, otherwise, if the store fails the old page won't be removed from -* the tree, and it might be written back overriding the new data. -*/ -spin_lock (&tree->lock); - dupentry =zswap_rb_search (&tree->rbroot, offset); -if (dupentry) { - zswap_duplicate_entry++; -zswap_invalidate_entry (tree, dupentry); - } -spin_unlock (&tree->lock); - -/* -* XXX: zswap reclaim does not work with cgroups yet. Without a -* cgroup-aware entry LRU, we will push out entries system-wide based on -* local cgroup limits. -*/ - objcg =get_obj_cgroup_from_folio (folio); -if (objcg && !obj_cgroup_may_zswap (objcg)) - goto reject; - -/* reclaim space if needed */ -if (zswap_is_full ()) { - zswap_pool_limit_hit++; - zswap_pool_reached_full = true; - goto shrink; - } - -if (zswap_pool_reached_full) { -if (!zswap_can_accept ()) - goto shrink; -else - zswap_pool_reached_full = false; - } +if (objcg) +obj_cgroup_get (objcg);/* allocate entry */ entry =zswap_entry_cache_alloc (GFP_KERNEL );@@ -1260,6 +1219,8 @@ bool zswap_store(struct folio *folio) zswap_reject_kmemcache_fail++; goto reject; } + entry->objcg = objcg; + entry->pool = pool;if (zswap_same_filled_pages_enabled) { src =kmap_atomic (page);@@ -1277,11 +1238,6 @@ bool zswap_store(struct folio *folio)if (!zswap_non_same_filled_pages_enabled) goto freepage; -/* if entry is successfully added, it keeps the reference */ - entry->pool =zswap_pool_current_get (); -if (!entry->pool) - goto freepage; -/* compress */ acomp_ctx =raw_cpu_ptr (entry->pool->acomp_ctx);@@ -1337,7 +1293,6 @@ bool zswap_store(struct folio *folio) entry->length = dlen; insert_entry: - entry->objcg = objcg;if (objcg) {obj_cgroup_charge_zswap (objcg, entry->length);/* Account before objcg ref is moved to tree */ @@ -1373,19 +1328,105 @@ bool zswap_store(struct folio *folio) put_dstmem:mutex_unlock (acomp_ctx->mutex); -zswap_pool_put (entry->pool); freepage:zswap_entry_cache_free (entry); reject:if (objcg)obj_cgroup_put (objcg); +zswap_pool_put (pool);return false; +} +bool zswap_store (struct folio *folio) +{ +long nr_pages =folio_nr_pages (folio); +swp_entry_t swp = folio->swap; +int type =swp_type (swp); +pgoff_t offset =swp_offset (swp); +struct zswap_tree *tree = zswap_trees[type]; +struct zswap_entry *entry; +struct obj_cgroup *objcg =NULL ; +struct zswap_pool *pool; +bool ret = false; +long i; + +VM_WARN_ON_ONCE (!folio_test_locked (folio)); +VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); + +if (!zswap_enabled || !tree) +return false; + +/* +* If this is a duplicate, it must be removed before attempting to store +* it, otherwise, if the store fails the old page won't be removed from +* the tree, and it might be written back overriding the new data. +*/ +spin_lock (&tree->lock); +for (i =0 ; i < nr_pages; i++) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) { + zswap_duplicate_entry++; +zswap_invalidate_entry (tree, entry); + } + }
mm/zswap.c bool zswap_store(struct folio *folio) · line 1381
Quoted 10 lines
mm/zswap.c +10
-0+spin_unlock (&tree->lock); + +/* +* XXX: zswap reclaim does not work with cgroups yet. Without a +* cgroup-aware entry LRU, we will push out entries system-wide based on +* local cgroup limits. +*/ + objcg =get_obj_cgroup_from_folio (folio); +if (objcg && !obj_cgroup_may_zswap (objcg)) + goto put_objcg;
Hmm would this make more sense to check these limits at a higher order page level or at a backing page (4 KB) level?
What if the cgroup still has some space before the new folio comes in, but the new folio would drive the pool size beyond the limit? Ditto for global zswap pool limit.
Previously, the object size is capped by the size of a page (since we don't accept bigger size pages into zswap). If zswap limit is exceded, it will not be exceeded by more than 4 KB. No big deal. But I'm not sure this will be OK as we move to bigger and bigger sizes for the pages...
If we do decide to check the cgroup's zswap limit for each backing page, I'm not sure how the reclaim logic (which will be introduced in the cgroup-aware zswap LRU patch) will interact with this though.
Quoted 53 lines
mm/zswap.c +44
-1+ +/* reclaim space if needed */ +if (zswap_is_full ()) { + zswap_pool_limit_hit++; + zswap_pool_reached_full = true; + goto shrink; + } + +if (zswap_pool_reached_full) { +if (!zswap_can_accept ()) + goto shrink; +else + zswap_pool_reached_full = false; + } + + pool =zswap_pool_current_get (); +if (!pool) + goto put_objcg; + +/* +* Store each page of the folio as a separate entry. If we fail to store +* a page, unwind by removing all the previous pages we stored. +*/ +for (i =0 ; i < nr_pages; i++) { +if (!zswap_store_page (folio, i, objcg, pool)) { +spin_lock (&tree->lock); +for (i--; i >=0 ; i--) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) +zswap_invalidate_entry (tree, entry); + } +spin_unlock (&tree->lock); + goto put_pool; + } + } + + ret = true; +put_pool: +zswap_pool_put (pool); +put_objcg: +if (objcg) +obj_cgroup_put (objcg); +return ret; shrink: pool =zswap_pool_last_get ();if (pool && !queue_work (shrink_wq, &pool->shrink_work))zswap_pool_put (pool); - goto reject; + goto put_objcg; }bool zswap_load (struct folio *folio)
Quoted 5 lines
base-commit:
158978945f3173b8c1a88f8c5684a629736a57ac--
2.25.1
I don't see anything else that is obviously wrong with this. Seems straightforward to me.
But I'm not too super familiar with THP swapping logic either :) So maybe someone with exposure to both should take a look too.
And would it make sense to introduce a gate that guard this so that users can opt-in/opt-out of this new feature, at least as we experiment more with it to get more data?
Hi Nhat - thanks for the review!
Quoted 29 lines · Nhat Pham
Quoted 20 lines · Ryan Roberts
Previously zswap would refuse to store any folio bigger than order-0, and therefore all of those folios would be sent directly to the swap file. This is a minor inconvenience since swap can currently only support order-0 and PMD-sized THP, but with the pending introduction of "small-sized THP", and corresponding changes to swapfile to support any order of folio, these large folios will become more prevalent and without this zswap change, zswap will become unusable. Independently of the "small-sized THP" feature, this change makes it possible to store existing PMD-sized THPs in zswap.
Modify
zswap_store()to allow storing large folios. The function is split into 2 parts;zswap_store()does all the per-folio operations (i.e. checking there is enough space, etc). Then it calls a new helper,zswap_store_page(), for each page in the folio, which are stored as their own entries in the zswap pool. (These entries continue to be loaded back individually as single pages). If a store fails for any single page, then all previously successfully stored folio pages are invalidated.Quoted 7 lines
Signed-off-by Ryan Roberts <[email protected]>---
I've tested this on arm64 (m2) with zswap enabled, and running vm-scalability'susememacross multiple cores from within a memory-constrained memcg to force high volumes of swap. I've also run mm selftests and observe no regressions (although there is nothing [z]swap specific there) - does zswap have any specific tests I should run?
Quoted 2 lines
There is a zswap kselftest in the cgroup suite:
https://lore.kernel.org/all/[email protected]/
ahh great - I'll run that against future versions.
Quoted 4 lines
Regardless, I feel like this kind of change is probably better to be tested via stress tests anyway - allocating a bunch of anon memory with a certain pattern, making sure they're not corrupted after being zswapped out etc.
Yes - that's exactly what the usemem test I described above is doing (and its not reporting any corruption).
Quoted 195 lines
Quoted 180 lines
This is based on mm-stable, since mm-unstable contains a zswap patch known to be buggy [1]. I thought it would be best to get comments on the shape, then do the rebase after that patch has been fixed.
For context, small-sized THP is being discussed here [2], and I'm working on changes to swapfile to support non-PMD-sized large folios here [3].
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
Ryanmm/zswap.c | 155 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 98 insertions(+), 57 deletions(-)mm/zswap.c +44
-56@@ -1188,18 +1188,17 @@ static void zswap_fill_page(void *ptr, unsigned long value)memset_l (page, value,PAGE_SIZE /sizeof (unsigned long )); } -bool zswap_store (struct folio *folio) +static bool zswap_store_page (struct folio *folio,long index, +struct obj_cgroup *objcg,struct zswap_pool *pool) {swp_entry_t swp = folio->swap;int type =swp_type (swp); -pgoff_t offset =swp_offset (swp); -struct page *page = &folio->page; +pgoff_t offset =swp_offset (swp) + index; +struct page *page =folio_page (folio, index);struct zswap_tree *tree = zswap_trees[type];struct zswap_entry *entry, *dupentry;struct scatterlist input, output;struct crypto_acomp_ctx *acomp_ctx; -struct obj_cgroup *objcg =NULL ; -struct zswap_pool *pool;struct zpool *zpool;unsigned int dlen =PAGE_SIZE ;unsigned long handle, value;@@ -1208,51 +1207,11 @@ bool zswap_store(struct folio *folio)gfp_t gfp;int ret; -VM_WARN_ON_ONCE (!folio_test_locked (folio)); -VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); - -/* Large folios aren't supported */ -if (folio_test_large (folio)) -return false; - -if (!zswap_enabled || !tree) +/* entry keeps the references if successfully stored. */ +if (!zswap_pool_get (pool))return false; - -/* -* If this is a duplicate, it must be removed before attempting to store -* it, otherwise, if the store fails the old page won't be removed from -* the tree, and it might be written back overriding the new data. -*/ -spin_lock (&tree->lock); - dupentry =zswap_rb_search (&tree->rbroot, offset); -if (dupentry) { - zswap_duplicate_entry++; -zswap_invalidate_entry (tree, dupentry); - } -spin_unlock (&tree->lock); - -/* -* XXX: zswap reclaim does not work with cgroups yet. Without a -* cgroup-aware entry LRU, we will push out entries system-wide based on -* local cgroup limits. -*/ - objcg =get_obj_cgroup_from_folio (folio); -if (objcg && !obj_cgroup_may_zswap (objcg)) - goto reject; - -/* reclaim space if needed */ -if (zswap_is_full ()) { - zswap_pool_limit_hit++; - zswap_pool_reached_full = true; - goto shrink; - } - -if (zswap_pool_reached_full) { -if (!zswap_can_accept ()) - goto shrink; -else - zswap_pool_reached_full = false; - } +if (objcg) +obj_cgroup_get (objcg);/* allocate entry */ entry =zswap_entry_cache_alloc (GFP_KERNEL );@@ -1260,6 +1219,8 @@ bool zswap_store(struct folio *folio) zswap_reject_kmemcache_fail++; goto reject; } + entry->objcg = objcg; + entry->pool = pool;if (zswap_same_filled_pages_enabled) { src =kmap_atomic (page);@@ -1277,11 +1238,6 @@ bool zswap_store(struct folio *folio)if (!zswap_non_same_filled_pages_enabled) goto freepage; -/* if entry is successfully added, it keeps the reference */ - entry->pool =zswap_pool_current_get (); -if (!entry->pool) - goto freepage; -/* compress */ acomp_ctx =raw_cpu_ptr (entry->pool->acomp_ctx);@@ -1337,7 +1293,6 @@ bool zswap_store(struct folio *folio) entry->length = dlen; insert_entry: - entry->objcg = objcg;if (objcg) {obj_cgroup_charge_zswap (objcg, entry->length);/* Account before objcg ref is moved to tree */ @@ -1373,19 +1328,105 @@ bool zswap_store(struct folio *folio) put_dstmem:mutex_unlock (acomp_ctx->mutex); -zswap_pool_put (entry->pool); freepage:zswap_entry_cache_free (entry); reject:if (objcg)obj_cgroup_put (objcg); +zswap_pool_put (pool);return false; +} +bool zswap_store (struct folio *folio) +{ +long nr_pages =folio_nr_pages (folio); +swp_entry_t swp = folio->swap; +int type =swp_type (swp); +pgoff_t offset =swp_offset (swp); +struct zswap_tree *tree = zswap_trees[type]; +struct zswap_entry *entry; +struct obj_cgroup *objcg =NULL ; +struct zswap_pool *pool; +bool ret = false; +long i; + +VM_WARN_ON_ONCE (!folio_test_locked (folio)); +VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); + +if (!zswap_enabled || !tree) +return false; + +/* +* If this is a duplicate, it must be removed before attempting to store +* it, otherwise, if the store fails the old page won't be removed from +* the tree, and it might be written back overriding the new data. +*/ +spin_lock (&tree->lock); +for (i =0 ; i < nr_pages; i++) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) { + zswap_duplicate_entry++; +zswap_invalidate_entry (tree, entry); + } + }mm/zswap.c bool zswap_store(struct folio *folio) · line 1381
Quoted 10 lines
mm/zswap.c +10
-0+spin_unlock (&tree->lock); + +/* +* XXX: zswap reclaim does not work with cgroups yet. Without a +* cgroup-aware entry LRU, we will push out entries system-wide based on +* local cgroup limits. +*/ + objcg =get_obj_cgroup_from_folio (folio); +if (objcg && !obj_cgroup_may_zswap (objcg)) + goto put_objcg;Hmm would this make more sense to check these limits at a higher order page level or at a backing page (4 KB) level?
Quoted 3 lines
What if the cgroup still has some space before the new folio comes in, but the new folio would drive the pool size beyond the limit? Ditto for global zswap pool limit.
I did consider this, but I thought I would keep it simple for the RFC and accept that we may go over the limits by (HPAGE_PMD_NR - 1) pages. It sounds like you don't think that will be acceptable.
I see 2 options:
- move this checking logic to be per-page in zswap_store_page()
- pass a size (or nr_pages or order) to obj_cgroup_may_zswap(), zswap_is_full() and zswap_can_accept() so they know how much we are proposing to add and can make a correct decision.
Personally I prefer the second option for efficiency.
Quoted 8 lines
Previously, the object size is capped by the size of a page (since we don't accept bigger size pages into zswap). If zswap limit is exceded, it will not be exceeded by more than 4 KB. No big deal. But I'm not sure this will be OK as we move to bigger and bigger sizes for the pages...
Quoted 4 lines
If we do decide to check the cgroup's zswap limit for each backing page, I'm not sure how the reclaim logic (which will be introduced in the cgroup-aware zswap LRU patch) will interact with this though.
Ok so this points us in the direction of my option 2 above as well?
Quoted 63 lines
Quoted 54 lines
mm/zswap.c +44
-1+ +/* reclaim space if needed */ +if (zswap_is_full ()) { + zswap_pool_limit_hit++; + zswap_pool_reached_full = true; + goto shrink; + } + +if (zswap_pool_reached_full) { +if (!zswap_can_accept ()) + goto shrink; +else + zswap_pool_reached_full = false; + } + + pool =zswap_pool_current_get (); +if (!pool) + goto put_objcg; + +/* +* Store each page of the folio as a separate entry. If we fail to store +* a page, unwind by removing all the previous pages we stored. +*/ +for (i =0 ; i < nr_pages; i++) { +if (!zswap_store_page (folio, i, objcg, pool)) { +spin_lock (&tree->lock); +for (i--; i >=0 ; i--) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) +zswap_invalidate_entry (tree, entry); + } +spin_unlock (&tree->lock); + goto put_pool; + } + } + + ret = true; +put_pool: +zswap_pool_put (pool); +put_objcg: +if (objcg) +obj_cgroup_put (objcg); +return ret; shrink: pool =zswap_pool_last_get ();if (pool && !queue_work (shrink_wq, &pool->shrink_work))zswap_pool_put (pool); - goto reject; + goto put_objcg; }bool zswap_load (struct folio *folio)Quoted 5 lines
base-commit:
158978945f3173b8c1a88f8c5684a629736a57ac--
2.25.1I don't see anything else that is obviously wrong with this. Seems straightforward to me.
Quoted 8 lines
But I'm not too super familiar with THP swapping logic either :) So maybe someone with exposure to both should take a look too.
And would it make sense to introduce a gate that guard this so that users can opt-in/opt-out of this new feature, at least as we experiment more with it to get more data?
There is already a gate at a higher level; CONFIG_THP_SWAP. If that is not enabled, then all large folios will be split to single pages before a swap entry is even allocated.
I considered adding a separate gate for this, but given the above control, I didn't think the zswap-specific control was really neccessary. What do you think?
Thanks,
Ryan
Nhat Pham
maintainerQuoted 40 lines · Ryan Roberts
Hi Nhat - thanks for the review!
Quoted 29 lines · Nhat Pham
Quoted 20 lines · Ryan Roberts
Previously zswap would refuse to store any folio bigger than order-0, and therefore all of those folios would be sent directly to the swap file. This is a minor inconvenience since swap can currently only support order-0 and PMD-sized THP, but with the pending introduction of "small-sized THP", and corresponding changes to swapfile to support any order of folio, these large folios will become more prevalent and without this zswap change, zswap will become unusable. Independently of the "small-sized THP" feature, this change makes it possible to store existing PMD-sized THPs in zswap.
Modify
zswap_store()to allow storing large folios. The function is split into 2 parts;zswap_store()does all the per-folio operations (i.e. checking there is enough space, etc). Then it calls a new helper,zswap_store_page(), for each page in the folio, which are stored as their own entries in the zswap pool. (These entries continue to be loaded back individually as single pages). If a store fails for any single page, then all previously successfully stored folio pages are invalidated.Quoted 7 lines
Signed-off-by Ryan Roberts <[email protected]>---
I've tested this on arm64 (m2) with zswap enabled, and running vm-scalability'susememacross multiple cores from within a memory-constrained memcg to force high volumes of swap. I've also run mm selftests and observe no regressions (although there is nothing [z]swap specific there) - does zswap have any specific tests I should run?Quoted 2 lines
There is a zswap kselftest in the cgroup suite:
https://lore.kernel.org/all/[email protected]/ahh great - I'll run that against future versions.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Regardless, I feel like this kind of change is probably better to be tested via stress tests anyway - allocating a bunch of anon memory with a certain pattern, making sure they're not corrupted after being zswapped out etc.
Yes - that's exactly what the
usememtest I described above is doing (and its not reporting any corruption).
Nice, and I assume no regression (in runtime for e.g) in the usemem test as well?
Quoted 196 lines
Quoted 196 lines
Quoted 181 lines
This is based on mm-stable, since mm-unstable contains a zswap patch known to be buggy [1]. I thought it would be best to get comments on the shape, then do the rebase after that patch has been fixed.
For context, small-sized THP is being discussed here [2], and I'm working on changes to swapfile to support non-PMD-sized large folios here [3].
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
Ryanmm/zswap.c | 155 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 98 insertions(+), 57 deletions(-)mm/zswap.c +44
-56@@ -1188,18 +1188,17 @@ static void zswap_fill_page(void *ptr, unsigned long value)memset_l (page, value,PAGE_SIZE /sizeof (unsigned long )); } -bool zswap_store (struct folio *folio) +static bool zswap_store_page (struct folio *folio,long index, +struct obj_cgroup *objcg,struct zswap_pool *pool) {swp_entry_t swp = folio->swap;int type =swp_type (swp); -pgoff_t offset =swp_offset (swp); -struct page *page = &folio->page; +pgoff_t offset =swp_offset (swp) + index; +struct page *page =folio_page (folio, index);struct zswap_tree *tree = zswap_trees[type];struct zswap_entry *entry, *dupentry;struct scatterlist input, output;struct crypto_acomp_ctx *acomp_ctx; -struct obj_cgroup *objcg =NULL ; -struct zswap_pool *pool;struct zpool *zpool;unsigned int dlen =PAGE_SIZE ;unsigned long handle, value;@@ -1208,51 +1207,11 @@ bool zswap_store(struct folio *folio)gfp_t gfp;int ret; -VM_WARN_ON_ONCE (!folio_test_locked (folio)); -VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); - -/* Large folios aren't supported */ -if (folio_test_large (folio)) -return false; - -if (!zswap_enabled || !tree) +/* entry keeps the references if successfully stored. */ +if (!zswap_pool_get (pool))return false; - -/* -* If this is a duplicate, it must be removed before attempting to store -* it, otherwise, if the store fails the old page won't be removed from -* the tree, and it might be written back overriding the new data. -*/ -spin_lock (&tree->lock); - dupentry =zswap_rb_search (&tree->rbroot, offset); -if (dupentry) { - zswap_duplicate_entry++; -zswap_invalidate_entry (tree, dupentry); - } -spin_unlock (&tree->lock); - -/* -* XXX: zswap reclaim does not work with cgroups yet. Without a -* cgroup-aware entry LRU, we will push out entries system-wide based on -* local cgroup limits. -*/ - objcg =get_obj_cgroup_from_folio (folio); -if (objcg && !obj_cgroup_may_zswap (objcg)) - goto reject; - -/* reclaim space if needed */ -if (zswap_is_full ()) { - zswap_pool_limit_hit++; - zswap_pool_reached_full = true; - goto shrink; - } - -if (zswap_pool_reached_full) { -if (!zswap_can_accept ()) - goto shrink; -else - zswap_pool_reached_full = false; - } +if (objcg) +obj_cgroup_get (objcg);/* allocate entry */ entry =zswap_entry_cache_alloc (GFP_KERNEL );@@ -1260,6 +1219,8 @@ bool zswap_store(struct folio *folio) zswap_reject_kmemcache_fail++; goto reject; } + entry->objcg = objcg; + entry->pool = pool;if (zswap_same_filled_pages_enabled) { src =kmap_atomic (page);@@ -1277,11 +1238,6 @@ bool zswap_store(struct folio *folio)if (!zswap_non_same_filled_pages_enabled) goto freepage; -/* if entry is successfully added, it keeps the reference */ - entry->pool =zswap_pool_current_get (); -if (!entry->pool) - goto freepage; -/* compress */ acomp_ctx =raw_cpu_ptr (entry->pool->acomp_ctx);@@ -1337,7 +1293,6 @@ bool zswap_store(struct folio *folio) entry->length = dlen; insert_entry: - entry->objcg = objcg;if (objcg) {obj_cgroup_charge_zswap (objcg, entry->length);/* Account before objcg ref is moved to tree */ @@ -1373,19 +1328,105 @@ bool zswap_store(struct folio *folio) put_dstmem:mutex_unlock (acomp_ctx->mutex); -zswap_pool_put (entry->pool); freepage:zswap_entry_cache_free (entry); reject:if (objcg)obj_cgroup_put (objcg); +zswap_pool_put (pool);return false; +} +bool zswap_store (struct folio *folio) +{ +long nr_pages =folio_nr_pages (folio); +swp_entry_t swp = folio->swap; +int type =swp_type (swp); +pgoff_t offset =swp_offset (swp); +struct zswap_tree *tree = zswap_trees[type]; +struct zswap_entry *entry; +struct obj_cgroup *objcg =NULL ; +struct zswap_pool *pool; +bool ret = false; +long i; + +VM_WARN_ON_ONCE (!folio_test_locked (folio)); +VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); + +if (!zswap_enabled || !tree) +return false; + +/* +* If this is a duplicate, it must be removed before attempting to store +* it, otherwise, if the store fails the old page won't be removed from +* the tree, and it might be written back overriding the new data. +*/ +spin_lock (&tree->lock); +for (i =0 ; i < nr_pages; i++) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) { + zswap_duplicate_entry++; +zswap_invalidate_entry (tree, entry); + } + }mm/zswap.c bool zswap_store(struct folio *folio) · line 1381
Quoted 10 lines
mm/zswap.c +10
-0+spin_unlock (&tree->lock); + +/* +* XXX: zswap reclaim does not work with cgroups yet. Without a +* cgroup-aware entry LRU, we will push out entries system-wide based on +* local cgroup limits. +*/ + objcg =get_obj_cgroup_from_folio (folio); +if (objcg && !obj_cgroup_may_zswap (objcg)) + goto put_objcg;Hmm would this make more sense to check these limits at a higher order page level or at a backing page (4 KB) level?
Quoted 7 lines
Quoted 3 lines
What if the cgroup still has some space before the new folio comes in, but the new folio would drive the pool size beyond the limit? Ditto for global zswap pool limit.
I did consider this, but I thought I would keep it simple for the RFC and accept that we may go over the limits by (
HPAGE_PMD_NR- 1) pages. It sounds like you don't think that will be acceptable.
Not necessarily unacceptable - I just wanted to point out a potential change in our assumption with zswap that might go unnoticed.
I think there should be at least some sort of documentation or comments spelling this out.
Quoted 9 lines
I see 2 options:
- move this checking logic to be per-page in
zswap_store_page()
- pass a size (ornr_pagesor order) toobj_cgroup_may_zswap(),zswap_is_full()andzswap_can_accept()so they know how much we are proposing to add and can make a correct decision.Personally I prefer the second option for efficiency.
Hmm if we go with the second option, how should this decision be made? It's impossible to predict the post-compression size of the huge page, even if we know its order.
I don't have a better suggestion than the first option, so if anyone else has a better idea please chime in :)
Personally, I would not flat out NACK this patch because of this limitation though, as long as it is at least spelled out somewhere.
Quoted 83 lines
Quoted 9 lines
Previously, the object size is capped by the size of a page (since we don't accept bigger size pages into zswap). If zswap limit is exceded, it will not be exceeded by more than 4 KB. No big deal. But I'm not sure this will be OK as we move to bigger and bigger sizes for the pages...
Quoted 4 lines
If we do decide to check the cgroup's zswap limit for each backing page, I'm not sure how the reclaim logic (which will be introduced in the cgroup-aware zswap LRU patch) will interact with this though.
Ok so this points us in the direction of my option 2 above as well?
Quoted 67 lines
Quoted 54 lines
mm/zswap.c +44
-1+ +/* reclaim space if needed */ +if (zswap_is_full ()) { + zswap_pool_limit_hit++; + zswap_pool_reached_full = true; + goto shrink; + } + +if (zswap_pool_reached_full) { +if (!zswap_can_accept ()) + goto shrink; +else + zswap_pool_reached_full = false; + } + + pool =zswap_pool_current_get (); +if (!pool) + goto put_objcg; + +/* +* Store each page of the folio as a separate entry. If we fail to store +* a page, unwind by removing all the previous pages we stored. +*/ +for (i =0 ; i < nr_pages; i++) { +if (!zswap_store_page (folio, i, objcg, pool)) { +spin_lock (&tree->lock); +for (i--; i >=0 ; i--) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) +zswap_invalidate_entry (tree, entry); + } +spin_unlock (&tree->lock); + goto put_pool; + } + } + + ret = true; +put_pool: +zswap_pool_put (pool); +put_objcg: +if (objcg) +obj_cgroup_put (objcg); +return ret; shrink: pool =zswap_pool_last_get ();if (pool && !queue_work (shrink_wq, &pool->shrink_work))zswap_pool_put (pool); - goto reject; + goto put_objcg; }bool zswap_load (struct folio *folio)Quoted 5 lines
base-commit:
158978945f3173b8c1a88f8c5684a629736a57ac--
2.25.1I don't see anything else that is obviously wrong with this. Seems straightforward to me.
But I'm not too super familiar with THP swapping logic either :) So maybe someone with exposure to both should take a look too.
Quoted 8 lines
Quoted 4 lines
And would it make sense to introduce a gate that guard this so that users can opt-in/opt-out of this new feature, at least as we experiment more with it to get more data?
There is already a gate at a higher level;
CONFIG_THP_SWAP. If that is not enabled, then all large folios will be split to single pages before a swap entry is even allocated.
Hmm is there a runtime option as well, or does it not make sense to have it? But yeah, in principle I'm not against having a single knob controlling this behavior for both swap and zswap (as opposed to a zswap-specific knob).
I'll defer any further objections to this aspect to other zswap contributors and maintainers :)
Quoted 8 lines
I considered adding a separate gate for this, but given the above control, I didn't think the zswap-specific control was really neccessary. What do you think?
Thanks,
Ryan
Quoted 45 lines · Nhat Pham
Quoted 44 lines · Ryan Roberts
Hi Nhat - thanks for the review!
Quoted 29 lines · Nhat Pham
Quoted 20 lines · Ryan Roberts
Previously zswap would refuse to store any folio bigger than order-0, and therefore all of those folios would be sent directly to the swap file. This is a minor inconvenience since swap can currently only support order-0 and PMD-sized THP, but with the pending introduction of "small-sized THP", and corresponding changes to swapfile to support any order of folio, these large folios will become more prevalent and without this zswap change, zswap will become unusable. Independently of the "small-sized THP" feature, this change makes it possible to store existing PMD-sized THPs in zswap.
Modify
zswap_store()to allow storing large folios. The function is split into 2 parts;zswap_store()does all the per-folio operations (i.e. checking there is enough space, etc). Then it calls a new helper,zswap_store_page(), for each page in the folio, which are stored as their own entries in the zswap pool. (These entries continue to be loaded back individually as single pages). If a store fails for any single page, then all previously successfully stored folio pages are invalidated.Quoted 7 lines
Signed-off-by Ryan Roberts <[email protected]>---
I've tested this on arm64 (m2) with zswap enabled, and running vm-scalability'susememacross multiple cores from within a memory-constrained memcg to force high volumes of swap. I've also run mm selftests and observe no regressions (although there is nothing [z]swap specific there) - does zswap have any specific tests I should run?Quoted 2 lines
There is a zswap kselftest in the cgroup suite:
https://lore.kernel.org/all/[email protected]/ahh great - I'll run that against future versions.
Quoted 4 lines
Regardless, I feel like this kind of change is probably better to be tested via stress tests anyway - allocating a bunch of anon memory with a certain pattern, making sure they're not corrupted after being zswapped out etc.
Quoted 5 lines
Quoted 2 lines
Yes - that's exactly what the
usememtest I described above is doing (and its not reporting any corruption).Nice, and I assume no regression (in runtime for e.g) in the usemem test as well?
I was mainly focussing on correctness. I don't recall seeing any perf regression, but don't have the data to check right now. I'll post numbers with the next version (although I'm likely not going to have bandwidth until early December to pick this up again).
Quoted 219 lines
Quoted 197 lines
Quoted 197 lines
Quoted 182 lines
This is based on mm-stable, since mm-unstable contains a zswap patch known to be buggy [1]. I thought it would be best to get comments on the shape, then do the rebase after that patch has been fixed.
For context, small-sized THP is being discussed here [2], and I'm working on changes to swapfile to support non-PMD-sized large folios here [3].
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
Ryanmm/zswap.c | 155 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 98 insertions(+), 57 deletions(-)mm/zswap.c +44
-56@@ -1188,18 +1188,17 @@ static void zswap_fill_page(void *ptr, unsigned long value)memset_l (page, value,PAGE_SIZE /sizeof (unsigned long )); } -bool zswap_store (struct folio *folio) +static bool zswap_store_page (struct folio *folio,long index, +struct obj_cgroup *objcg,struct zswap_pool *pool) {swp_entry_t swp = folio->swap;int type =swp_type (swp); -pgoff_t offset =swp_offset (swp); -struct page *page = &folio->page; +pgoff_t offset =swp_offset (swp) + index; +struct page *page =folio_page (folio, index);struct zswap_tree *tree = zswap_trees[type];struct zswap_entry *entry, *dupentry;struct scatterlist input, output;struct crypto_acomp_ctx *acomp_ctx; -struct obj_cgroup *objcg =NULL ; -struct zswap_pool *pool;struct zpool *zpool;unsigned int dlen =PAGE_SIZE ;unsigned long handle, value;@@ -1208,51 +1207,11 @@ bool zswap_store(struct folio *folio)gfp_t gfp;int ret; -VM_WARN_ON_ONCE (!folio_test_locked (folio)); -VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); - -/* Large folios aren't supported */ -if (folio_test_large (folio)) -return false; - -if (!zswap_enabled || !tree) +/* entry keeps the references if successfully stored. */ +if (!zswap_pool_get (pool))return false; - -/* -* If this is a duplicate, it must be removed before attempting to store -* it, otherwise, if the store fails the old page won't be removed from -* the tree, and it might be written back overriding the new data. -*/ -spin_lock (&tree->lock); - dupentry =zswap_rb_search (&tree->rbroot, offset); -if (dupentry) { - zswap_duplicate_entry++; -zswap_invalidate_entry (tree, dupentry); - } -spin_unlock (&tree->lock); - -/* -* XXX: zswap reclaim does not work with cgroups yet. Without a -* cgroup-aware entry LRU, we will push out entries system-wide based on -* local cgroup limits. -*/ - objcg =get_obj_cgroup_from_folio (folio); -if (objcg && !obj_cgroup_may_zswap (objcg)) - goto reject; - -/* reclaim space if needed */ -if (zswap_is_full ()) { - zswap_pool_limit_hit++; - zswap_pool_reached_full = true; - goto shrink; - } - -if (zswap_pool_reached_full) { -if (!zswap_can_accept ()) - goto shrink; -else - zswap_pool_reached_full = false; - } +if (objcg) +obj_cgroup_get (objcg);/* allocate entry */ entry =zswap_entry_cache_alloc (GFP_KERNEL );@@ -1260,6 +1219,8 @@ bool zswap_store(struct folio *folio) zswap_reject_kmemcache_fail++; goto reject; } + entry->objcg = objcg; + entry->pool = pool;if (zswap_same_filled_pages_enabled) { src =kmap_atomic (page);@@ -1277,11 +1238,6 @@ bool zswap_store(struct folio *folio)if (!zswap_non_same_filled_pages_enabled) goto freepage; -/* if entry is successfully added, it keeps the reference */ - entry->pool =zswap_pool_current_get (); -if (!entry->pool) - goto freepage; -/* compress */ acomp_ctx =raw_cpu_ptr (entry->pool->acomp_ctx);@@ -1337,7 +1293,6 @@ bool zswap_store(struct folio *folio) entry->length = dlen; insert_entry: - entry->objcg = objcg;if (objcg) {obj_cgroup_charge_zswap (objcg, entry->length);/* Account before objcg ref is moved to tree */ @@ -1373,19 +1328,105 @@ bool zswap_store(struct folio *folio) put_dstmem:mutex_unlock (acomp_ctx->mutex); -zswap_pool_put (entry->pool); freepage:zswap_entry_cache_free (entry); reject:if (objcg)obj_cgroup_put (objcg); +zswap_pool_put (pool);return false; +} +bool zswap_store (struct folio *folio) +{ +long nr_pages =folio_nr_pages (folio); +swp_entry_t swp = folio->swap; +int type =swp_type (swp); +pgoff_t offset =swp_offset (swp); +struct zswap_tree *tree = zswap_trees[type]; +struct zswap_entry *entry; +struct obj_cgroup *objcg =NULL ; +struct zswap_pool *pool; +bool ret = false; +long i; + +VM_WARN_ON_ONCE (!folio_test_locked (folio)); +VM_WARN_ON_ONCE (!folio_test_swapcache (folio)); + +if (!zswap_enabled || !tree) +return false; + +/* +* If this is a duplicate, it must be removed before attempting to store +* it, otherwise, if the store fails the old page won't be removed from +* the tree, and it might be written back overriding the new data. +*/ +spin_lock (&tree->lock); +for (i =0 ; i < nr_pages; i++) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) { + zswap_duplicate_entry++; +zswap_invalidate_entry (tree, entry); + } + }mm/zswap.c bool zswap_store(struct folio *folio) · line 1381
Quoted 10 lines
mm/zswap.c +10
-0+spin_unlock (&tree->lock); + +/* +* XXX: zswap reclaim does not work with cgroups yet. Without a +* cgroup-aware entry LRU, we will push out entries system-wide based on +* local cgroup limits. +*/ + objcg =get_obj_cgroup_from_folio (folio); +if (objcg && !obj_cgroup_may_zswap (objcg)) + goto put_objcg;Hmm would this make more sense to check these limits at a higher order page level or at a backing page (4 KB) level?
Quoted 7 lines
Quoted 3 lines
What if the cgroup still has some space before the new folio comes in, but the new folio would drive the pool size beyond the limit? Ditto for global zswap pool limit.
I did consider this, but I thought I would keep it simple for the RFC and accept that we may go over the limits by (
HPAGE_PMD_NR- 1) pages. It sounds like you don't think that will be acceptable.Not necessarily unacceptable - I just wanted to point out a potential change in our assumption with zswap that might go unnoticed.
I think there should be at least some sort of documentation or comments spelling this out.
Quoted 8 lines
I see 2 options:
- move this checking logic to be per-page in
zswap_store_page()
- pass a size (ornr_pagesor order) toobj_cgroup_may_zswap(),zswap_is_full()andzswap_can_accept()so they know how much we are proposing to add and can make a correct decision.
Quoted 5 lines
Quoted 1 line
Personally I prefer the second option for efficiency.
Hmm if we go with the second option, how should this decision be made? It's impossible to predict the post-compression size of the huge page, even if we know its order.
But we have a worst case size (the uncompressed size). So we could use that as a conservative estimate? It means we might under-utilize the quota a bit, but I guess that's better than going over?
Quoted 6 lines
I don't have a better suggestion than the first option, so if anyone else has a better idea please chime in :)
Personally, I would not flat out NACK this patch because of this limitation though, as long as it is at least spelled out somewhere.
OK, that sounds like the easiest approach :). I'm happy to just add a comment for v2.
Quoted 89 lines
Quoted 89 lines
Quoted 10 lines
Previously, the object size is capped by the size of a page (since we don't accept bigger size pages into zswap). If zswap limit is exceded, it will not be exceeded by more than 4 KB. No big deal. But I'm not sure this will be OK as we move to bigger and bigger sizes for the pages...
Quoted 4 lines
If we do decide to check the cgroup's zswap limit for each backing page, I'm not sure how the reclaim logic (which will be introduced in the cgroup-aware zswap LRU patch) will interact with this though.
Ok so this points us in the direction of my option 2 above as well?
Quoted 71 lines
Quoted 54 lines
mm/zswap.c +44
-1+ +/* reclaim space if needed */ +if (zswap_is_full ()) { + zswap_pool_limit_hit++; + zswap_pool_reached_full = true; + goto shrink; + } + +if (zswap_pool_reached_full) { +if (!zswap_can_accept ()) + goto shrink; +else + zswap_pool_reached_full = false; + } + + pool =zswap_pool_current_get (); +if (!pool) + goto put_objcg; + +/* +* Store each page of the folio as a separate entry. If we fail to store +* a page, unwind by removing all the previous pages we stored. +*/ +for (i =0 ; i < nr_pages; i++) { +if (!zswap_store_page (folio, i, objcg, pool)) { +spin_lock (&tree->lock); +for (i--; i >=0 ; i--) { + entry =zswap_rb_search (&tree->rbroot, offset + i); +if (entry) +zswap_invalidate_entry (tree, entry); + } +spin_unlock (&tree->lock); + goto put_pool; + } + } + + ret = true; +put_pool: +zswap_pool_put (pool); +put_objcg: +if (objcg) +obj_cgroup_put (objcg); +return ret; shrink: pool =zswap_pool_last_get ();if (pool && !queue_work (shrink_wq, &pool->shrink_work))zswap_pool_put (pool); - goto reject; + goto put_objcg; }bool zswap_load (struct folio *folio)Quoted 5 lines
base-commit:
158978945f3173b8c1a88f8c5684a629736a57ac--
2.25.1I don't see anything else that is obviously wrong with this. Seems straightforward to me.
But I'm not too super familiar with THP swapping logic either :) So maybe someone with exposure to both should take a look too.
And would it make sense to introduce a gate that guard this so that users can opt-in/opt-out of this new feature, at least as we experiment more with it to get more data?
Quoted 8 lines
Quoted 3 lines
There is already a gate at a higher level;
CONFIG_THP_SWAP. If that is not enabled, then all large folios will be split to single pages before a swap entry is even allocated.Hmm is there a runtime option as well, or does it not make sense to have it? But yeah, in principle I'm not against having a single knob controlling this behavior for both swap and zswap (as opposed to a zswap-specific knob).
There is no runtime knob equivalent to CONFIG_THP_SWAP at the moment, so I guess that proves that it is not needed at the swap level. And I don't really see the benefit for explicitly disabling at the zswap level. So I would prefer to keep it simple and not add any new runtime knob.
The closest existing knob is the runtime THP sysfs interface - if THP is disabled (from boot) then there are no THPs to swap. (of course if disabled after boot, there may still be some THPs that were created earlier).
Quoted 11 lines
I'll defer any further objections to this aspect to other zswap contributors and maintainers :)
Quoted 8 lines
I considered adding a separate gate for this, but given the above control, I didn't think the zswap-specific control was really neccessary. What do you think?
Thanks,
Ryan