mm: add in-memory compression of reclaimed pages
Subsystems (19)
cover mm: add in-memory compression of reclaimed pages
This is a proof of concept, posted to ask whether the approach is worth pursuing. It is not meant for merging. The series is based on Linux 6.12.23, not on mm-unstable, and has not been ported to the swap-table and zpool changes that have landed since. If there is interest, the next version will be rebased onto mm-unstable.
memcomp stores reclaimed anonymous and shmem pages in compressed form in memory, using the kernel's compressors and zsmalloc unchanged. Reclaim replaces the PTE, or the shmem page-cache entry, with a swap-like token that names a 16-byte slot. A fault on the token decompresses the page and maps it again. No block device is needed. If swap is configured, cold entries can be written back to it. The sysfs interface is under /sys/kernel/mm/memcomp.
The implementation uses 16-byte slots, a separate owner domain for zero pages, per-key storage and bounded metadata aging. Memory-cgroup accounting, cold-page writeback, asynchronous compression, precise residency reporting and mincore support are separate patches.
Refaults follow stock workingset accounting. The first reader claims and clears the shadow under the slot lock. Feedback runs after charging, outside the PTL and slot locks, before the page is read and installed. Consumed feedback is not restored if the read or the installation fails. An mTHP uses the shadow of its first base page and clears the shadows of all its pages. Swapoff returns payloads to the pool without building user pages.
Base and branches
=================
The 16 patches sit on Linux 6.12.23 plus eight prerequisite commits; base-commit below names the top of that layer. The prerequisites make the series work on 6.12 and are not part of the proposal. Those already upstream drop out on a newer base. An android16-6.12 port carries 18 patches, the 16 here plus two ANDROID patches for vendor hooks and gki_defconfig, on a nine-commit prerequisite layer.
mainline: https://github.com/nakixii/linux/tree/memcomp/series-v1
ACK: https://github.com/nakixii/linux/tree/memcomp/android16-6.12
whitepaper, benchmarks and results: https://github.com/nakixii/memcomp
Testing
=======
KUnit: 492 tests pass on x86, both with the classic LRU in a debug configuration and with MGLRU. On arm64 ACK with MTE (HW_TAGS), 487 pass and six are skipped as architecture-specific.
All 17 cases of an end-to-end test in KVM guests passed their checks. The nine cases with swap ran every check; in the eight cases without swap the checker stops early, so those results are partial.
These are retained results, not reruns. KUnit ran on the final test candidate, before the sysfs directory was renamed and the matching comments and documentation were updated. The end-to-end test ran on an earlier implementation checkpoint, before later test additions and the rename.
checkpatch --strict reports no errors and no checks, and 17 warnings:
six FILE_PATH_CHANGES notes for new files; one allocation that would use kzalloc_obj(), which 6.12 does not have; one ENOSYS in a selftest that compares a userspace errno; and nine BUG_ON warnings, eight on assertions carried over from stock page_io.c and workingset.c code and one on a shmem token reference. sparse was run on x86 with LLVM and GCC; arm64 sparse and W=1 were not run.
Not covered by tests
====================
Hibernation, memory hot-remove, DAMON and khugepaged have had static review only. The paths without an exact test include swapon unwind and retry after a bitmap allocation failure, a genuine shmem charge failure as distinct from an earlier allocation failure, zero-page refault windows attributable to one operation, two concurrent readers of one token, a pool return followed by its first consumption, and ordinary reuse of the same swap offset. The error path for a failed pin of a still-online source memory cgroup is defensive and was never reached at runtime.
Questions for review
====================
- The source and patch layout: the implementation is one compilation unit with internal headers split by function.
- The multivalue sysfs ABI and the debug counters.
- Whether direct reclaim should skip folios with a high mapcount. A failed store can cost an unmap plus a reverse-map walk.
- The encoding uses one swap type and supports only 64-bit configurations, and memcomp cannot run together with zram or zswap.
- KUnit probes are kept in production code.
Results
=======
The comparison ran 965 fresh guest boots on real application pages, under nested KVM on WSL2, with a paired rule fixed before collection. Selected results:
Without real swap, single-page refault p99 was 10.7 us against 20.7 us for zram with lz4, and 22.0 us against 27.3 us with zstd. Under 512 MiB of anonymous pressure, refault p99 was 37% lower than zram with lz4 and 36% lower than zswap.
Completed-switch p99 with zstd was 33-36% lower than zram in 10/10 rounds and 20-22% lower than zswap in 9/10; these differences were smaller than the reference's variation between rounds. After writeback, readback p50 over an 832 MiB target was 11.9 us against 105.5 us for zswap and 114.0 us for zram writeback, with 70-76% fewer bytes written before readback; p99 was about the same.
Less favorable:
With real swap, single-page refault p99 was 4.8 times zram with lz4 and about 1.5 times zram and zswap with zstd.
Sequential refault was 30-31% slower than zswap, since there is no swap readahead. Single-page pageout p99 was 35-38% slower than zswap. In the pageout and refault workload, 64 MiB of real pages per application, memcomp used 5-8% more memory than zram; this leaves out zram's disksize-sized table, which was larger than the difference.
The measured kernels differ from this series only by the sysfs directory rename and by seven removed header-comment lines. All decisions, including undecided rows, are in the whitepaper. No phone results are
reported: the variables on a phone could not be controlled, so all numbers come from KVM guests, and they do not establish device latency, energy use or compatibility.
Tool use
========
Most of the code, the tests, the changelogs and this letter were written with LLM coding assistants, working from my design decisions and under my review over many iterations. I reviewed every patch and take responsibility for it. The assistants also ran the builds, KUnit, the KVM end-to-end test and the 965-boot comparison described above; the test programs and raw results are in the GitHub repository.
nakixii (16):
mm: userfaultfd: check the exclusivity of the page being moved
mm: add a swap entry type for in-memory compressed pages
mm: memcomp: compress reclaimed anonymous pages in memory
mm: userfaultfd: let UFFDIO_MOVE move memcomp entries
mm: memcomp: charge compressed pages to the source memcg
mm: memcomp: store shmem and tmpfs pages
mm: memcomp: write cold pages back to swap
mm: memcomp: compress foreground batches asynchronously
mm: memcomp: batch unmap reserved large folios
mm: memcomp: queue restored large folios for splitting
mm: memcomp: account user eviction and refault
mm: memcomp: report precise token residency
mm: memcomp: add KUnit tests
selftests/mm: add a memcomp pageout and refault test
mm: memcomp: add the design doc
mm: memcomp: add the admin guide
.../ABI/testing/sysfs-kernel-mm-memcomp | 363 + .../admin-guide/kernel-parameters.txt | 10 + Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/memcomp.rst | 149 + Documentation/mm/index.rst | 1 + Documentation/mm/memcomp.rst | 215 + MAINTAINERS | 12 + fs/proc/task_mmu.c | 20 + include/linux/memcomp.h | 481 ++ include/linux/memcontrol.h | 45 + include/linux/swap.h | 52 +- include/linux/swapops.h | 22 + mm/Kconfig | 75 + mm/Makefile | 1 + mm/hmm.c | 3 + mm/huge_memory.c | 44 + mm/internal.h | 84 + mm/madvise.c | 10 +- mm/memcomp.c | 6181 +++++++++++++++++ mm/memcomp/access.h | 132 + mm/memcomp/anon_folio.h | 84 + mm/memcomp/anon_store.h | 268 + mm/memcomp/api.h | 205 + mm/memcomp/cache_folio.h | 103 + mm/memcomp/cache_store.h | 97 + mm/memcomp/consumers.h | 129 + mm/memcomp/defrag.h | 169 + mm/memcomp/drain.h | 83 + mm/memcomp/fault.h | 320 + mm/memcomp/fault_alloc.h | 23 + mm/memcomp/fifo.h | 136 + mm/memcomp/folio.h | 434 ++ mm/memcomp/format.h | 308 + mm/memcomp/lifetime.h | 87 + mm/memcomp/load.h | 96 + mm/memcomp/migrate.h | 155 + mm/memcomp/mincore.h | 83 + mm/memcomp/owner.h | 157 + mm/memcomp/payload.h | 151 + mm/memcomp/query.h | 187 + mm/memcomp/refs.h | 83 + mm/memcomp/resident.h | 106 + mm/memcomp/scan.h | 115 + mm/memcomp/span.h | 89 + mm/memcomp/storage.h | 1131 +++ mm/memcomp/swapoff.h | 190 + mm/memcomp/swapoff_walk.h | 122 + mm/memcomp/tests/.kunitconfig | 16 + mm/memcomp/tests/access_test.h | 197 + mm/memcomp/tests/aging_test.h | 771 ++ mm/memcomp/tests/anon_store_test.h | 704 ++ mm/memcomp/tests/async_test.h | 703 ++ mm/memcomp/tests/budget_packed_test.h | 435 ++ mm/memcomp/tests/budget_swapped_test.h | 292 + mm/memcomp/tests/budget_test.h | 542 ++ mm/memcomp/tests/cache_folio_test.h | 627 ++ mm/memcomp/tests/cache_reserve_test.h | 447 ++ mm/memcomp/tests/cache_store_test.h | 216 + mm/memcomp/tests/cache_xarray_race_test.h | 193 + mm/memcomp/tests/codec_scope_test.h | 94 + mm/memcomp/tests/consumers_test.h | 478 ++ mm/memcomp/tests/core_test.h | 119 + mm/memcomp/tests/defrag_test.h | 601 ++ mm/memcomp/tests/drain_test.h | 581 ++ mm/memcomp/tests/enospc_test.h | 236 + mm/memcomp/tests/fault_alloc_test.h | 119 + mm/memcomp/tests/fault_test.h | 851 +++ mm/memcomp/tests/folio_test.h | 769 ++ mm/memcomp/tests/format_test.h | 520 ++ mm/memcomp/tests/late_async_test.h | 331 + mm/memcomp/tests/load_test.h | 133 + mm/memcomp/tests/migrate_test.h | 795 +++ mm/memcomp/tests/mincore_test.h | 462 ++ mm/memcomp/tests/owner_test.h | 400 ++ mm/memcomp/tests/payload_test.h | 220 + mm/memcomp/tests/public_api_test.h | 422 ++ mm/memcomp/tests/public_driver_test.h | 358 + mm/memcomp/tests/query_test.h | 480 ++ mm/memcomp/tests/refs_test.h | 224 + mm/memcomp/tests/replacement_test.h | 360 + mm/memcomp/tests/resident_test.h | 796 +++ mm/memcomp/tests/rollback_test.h | 163 + mm/memcomp/tests/scan_test.h | 422 ++ mm/memcomp/tests/shared_test.h | 3477 ++++++++++ mm/memcomp/tests/shmem_retry_test.h | 340 + mm/memcomp/tests/shmem_zero_fault_test.h | 121 + mm/memcomp/tests/span_test.h | 128 + mm/memcomp/tests/storage_fixture.h | 84 + mm/memcomp/tests/storage_test.h | 468 ++ mm/memcomp/tests/store_transaction_test.h | 587 ++ mm/memcomp/tests/suite.h | 693 ++ mm/memcomp/tests/swapoff_test.h | 558 ++ mm/memcomp/tests/swapoff_walk_test.h | 562 ++ mm/memcomp/tests/swapoff_yield_test.h | 532 ++ mm/memcomp/tests/test_api.h | 41 + mm/memcomp/tests/transport_reclaim_test.h | 298 + mm/memcomp/tests/transport_test.h | 114 + mm/memcomp/tests/workingset_load_test.h | 1095 +++ mm/memcomp/tests/workingset_mthp_error_test.h | 164 + mm/memcomp/tests/workingset_storage_test.h | 442 ++ mm/memcomp/tests/workingset_store_test.h | 572 ++ mm/memcomp/tests/writeback_batch_test.h | 688 ++ mm/memcomp/tests/writeback_test.h | 540 ++ mm/memcomp/tests/zero_callers_test.h | 717 ++ mm/memcomp/tests/zero_carrier_test.h | 405 ++ mm/memcomp/tests/zero_convert_test.h | 694 ++ mm/memcomp/tests/zero_hint_test.h | 860 +++ mm/memcomp/tests/zero_load_test.h | 645 ++ mm/memcomp/tests/zero_test.h | 313 + mm/memcomp/workingset_load.h | 97 + mm/memcomp/workingset_store.h | 62 + mm/memcomp/writeback.h | 133 + mm/memcomp/writeback_batch.h | 398 ++ mm/memcomp/zero.h | 205 + mm/memcomp/zero_convert.h | 78 + mm/memcomp/zero_load.h | 91 + mm/memcomp_rmap.h | 139 + mm/memcontrol.c | 99 + mm/memory.c | 42 +- mm/migrate.c | 11 +- mm/mincore.c | 125 +- mm/page_io.c | 42 + mm/rmap.c | 41 +- mm/shmem.c | 251 + mm/swap.h | 4 + mm/swap_state.c | 5 + mm/swapfile.c | 49 + mm/userfaultfd.c | 94 +- mm/vmscan.c | 651 +- mm/workingset.c | 32 + tools/testing/selftests/mm/Makefile | 9 +- tools/testing/selftests/mm/config | 11 + tools/testing/selftests/mm/hmm-tests.c | 112 + tools/testing/selftests/mm/memcomp.c | 831 +++ 134 files changed, 46825 insertions(+), 14 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-memcomp create mode 100644 Documentation/admin-guide/mm/memcomp.rst create mode 100644 Documentation/mm/memcomp.rst create mode 100644 include/linux/memcomp.h create mode 100644 mm/memcomp.c create mode 100644 mm/memcomp/access.h create mode 100644 mm/memcomp/anon_folio.h create mode 100644 mm/memcomp/anon_store.h create mode 100644 mm/memcomp/api.h create mode 100644 mm/memcomp/cache_folio.h create mode 100644 mm/memcomp/cache_store.h create mode 100644 mm/memcomp/consumers.h create mode 100644 mm/memcomp/defrag.h create mode 100644 mm/memcomp/drain.h create mode 100644 mm/memcomp/fault.h create mode 100644 mm/memcomp/fault_alloc.h create mode 100644 mm/memcomp/fifo.h create mode 100644 mm/memcomp/folio.h create mode 100644 mm/memcomp/format.h create mode 100644 mm/memcomp/lifetime.h create mode 100644 mm/memcomp/load.h create mode 100644 mm/memcomp/migrate.h create mode 100644 mm/memcomp/mincore.h create mode 100644 mm/memcomp/owner.h create mode 100644 mm/memcomp/payload.h create mode 100644 mm/memcomp/query.h create mode 100644 mm/memcomp/refs.h create mode 100644 mm/memcomp/resident.h create mode 100644 mm/memcomp/scan.h create mode 100644 mm/memcomp/span.h create mode 100644 mm/memcomp/storage.h create mode 100644 mm/memcomp/swapoff.h create mode 100644 mm/memcomp/swapoff_walk.h create mode 100644 mm/memcomp/tests/.kunitconfig create mode 100644 mm/memcomp/tests/access_test.h create mode 100644 mm/memcomp/tests/aging_test.h create mode 100644 mm/memcomp/tests/anon_store_test.h create mode 100644 mm/memcomp/tests/async_test.h create mode 100644 mm/memcomp/tests/budget_packed_test.h create mode 100644 mm/memcomp/tests/budget_swapped_test.h create mode 100644 mm/memcomp/tests/budget_test.h create mode 100644 mm/memcomp/tests/cache_folio_test.h create mode 100644 mm/memcomp/tests/cache_reserve_test.h create mode 100644 mm/memcomp/tests/cache_store_test.h create mode 100644 mm/memcomp/tests/cache_xarray_race_test.h create mode 100644 mm/memcomp/tests/codec_scope_test.h create mode 100644 mm/memcomp/tests/consumers_test.h create mode 100644 mm/memcomp/tests/core_test.h create mode 100644 mm/memcomp/tests/defrag_test.h create mode 100644 mm/memcomp/tests/drain_test.h create mode 100644 mm/memcomp/tests/enospc_test.h create mode 100644 mm/memcomp/tests/fault_alloc_test.h create mode 100644 mm/memcomp/tests/fault_test.h create mode 100644 mm/memcomp/tests/folio_test.h create mode 100644 mm/memcomp/tests/format_test.h create mode 100644 mm/memcomp/tests/late_async_test.h create mode 100644 mm/memcomp/tests/load_test.h create mode 100644 mm/memcomp/tests/migrate_test.h create mode 100644 mm/memcomp/tests/mincore_test.h create mode 100644 mm/memcomp/tests/owner_test.h create mode 100644 mm/memcomp/tests/payload_test.h create mode 100644 mm/memcomp/tests/public_api_test.h create mode 100644 mm/memcomp/tests/public_driver_test.h create mode 100644 mm/memcomp/tests/query_test.h create mode 100644 mm/memcomp/tests/refs_test.h create mode 100644 mm/memcomp/tests/replacement_test.h create mode 100644 mm/memcomp/tests/resident_test.h create mode 100644 mm/memcomp/tests/rollback_test.h create mode 100644 mm/memcomp/tests/scan_test.h create mode 100644 mm/memcomp/tests/shared_test.h create mode 100644 mm/memcomp/tests/shmem_retry_test.h create mode 100644 mm/memcomp/tests/shmem_zero_fault_test.h create mode 100644 mm/memcomp/tests/span_test.h create mode 100644 mm/memcomp/tests/storage_fixture.h create mode 100644 mm/memcomp/tests/storage_test.h create mode 100644 mm/memcomp/tests/store_transaction_test.h create mode 100644 mm/memcomp/tests/suite.h create mode 100644 mm/memcomp/tests/swapoff_test.h create mode 100644 mm/memcomp/tests/swapoff_walk_test.h create mode 100644 mm/memcomp/tests/swapoff_yield_test.h create mode 100644 mm/memcomp/tests/test_api.h create mode 100644 mm/memcomp/tests/transport_reclaim_test.h create mode 100644 mm/memcomp/tests/transport_test.h create mode 100644 mm/memcomp/tests/workingset_load_test.h create mode 100644 mm/memcomp/tests/workingset_mthp_error_test.h create mode 100644 mm/memcomp/tests/workingset_storage_test.h create mode 100644 mm/memcomp/tests/workingset_store_test.h create mode 100644 mm/memcomp/tests/writeback_batch_test.h create mode 100644 mm/memcomp/tests/writeback_test.h create mode 100644 mm/memcomp/tests/zero_callers_test.h create mode 100644 mm/memcomp/tests/zero_carrier_test.h create mode 100644 mm/memcomp/tests/zero_convert_test.h create mode 100644 mm/memcomp/tests/zero_hint_test.h create mode 100644 mm/memcomp/tests/zero_load_test.h create mode 100644 mm/memcomp/tests/zero_test.h create mode 100644 mm/memcomp/workingset_load.h create mode 100644 mm/memcomp/workingset_store.h create mode 100644 mm/memcomp/writeback.h create mode 100644 mm/memcomp/writeback_batch.h create mode 100644 mm/memcomp/zero.h create mode 100644 mm/memcomp/zero_convert.h create mode 100644 mm/memcomp/zero_load.h create mode 100644 mm/memcomp_rmap.h create mode 100644 tools/testing/selftests/mm/memcomp.c
base-commit: e728da875eb1d50add12357d01f44a3edf9a6f8c
RFC PATCH 1/16 mm: userfaultfd: check the exclusivity of the page being moved
UFFDIO_MOVE checks PageAnonExclusive on the first page of the source folio even when the source PTE maps another page of that folio. This can reject a move whose mapped page is still exclusive.
try_to_unmap_one() can replace earlier pages with swap entries before folio_try_share_anon_rmap_pte() refuses a later page, for example because of a concurrent GUP-fast pin. The first page can then lose its exclusive flag while a later mapped page remains exclusive. Checking the first page incorrectly returns -EBUSY for a move of that later page.
Look up the page mapped by the source PTE, check that page's exclusive flag and derive the folio from it. Preserve the existing folio locking, reference counting and split handling. This applies to ordinary swap unmapping without requiring an additional page-storage backend.
--- mm/userfaultfd.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
mm/userfaultfd.c +15 -2
@@ -1225,6 +1225,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,*/ if (!src_folio) {struct folio *folio; +struct page *page;bool locked;/*
@@ -1238,8 +1239,20 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - folio =vm_normal_folio (src_vma, src_addr, orig_src_pte); -if (!folio || !PageAnonExclusive (&folio->page)) { +/* +* Check the page to be moved, not the first of its +* folio. try_to_unmap() unshares a large folio page +* by page into swap entries, so while +* reclaim holds it, or after reclaim gave up partway, +* the first page can have lost its exclusive flag +* when this one has not. While reclaim still holds +* the folio, the trylock below sends a large one back +* with -EAGAIN; once the folio is split they are the +* same. +*/ + page =vm_normal_page (src_vma, src_addr, orig_src_pte); + folio = page ?page_folio (page) :NULL ; +if (!folio || !PageAnonExclusive (page)) {spin_unlock (src_ptl); err = -EBUSY ; goto out;
RFC PATCH 2/16 mm: add a swap entry type for in-memory compressed pages
In reply to nakixiiAn in-memory compressed page has no swap device or swap offset. Encoding its reference as an ordinary swap entry would direct faults to the swap cache and allow swap-device lookup to interpret an unrelated token.
Reserve a non-swap entry type for memcomp and provide helpers to construct and recognize it. Account for the reserved type when locating PTE markers and computing the maximum number of swap areas, keeping these encodings disjoint. A build without MEMCOMP reserves no additional entry type.
Provide the disabled-backend interface needed by callers without sending these entries through the ordinary swap-device paths. The compression backend and its reclaim hooks are introduced separately.
--- MAINTAINERS | 1 + include/linux/memcomp.h | 13 +++++++++++++ include/linux/swap.h | 13 +++++++++++-- include/linux/swapops.h | 22 ++++++++++++++++++++++ 4 files changed, 47 insertions(+), 2 deletions(-) create mode 100644 include/linux/memcomp.h
MAINTAINERS +1 -0
@@ -14852,6 +14852,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm T: quilt git://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new F: include/linux/gfp.h F: include/linux/gfp_types.h +F: include/linux/memcomp.h F: include/linux/memfd.h F: include/linux/memory.h F: include/linux/memory_hotplug.h
include/linux/memcomp.h new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMCOMP_H +#define _LINUX_MEMCOMP_H + +#include <linux/types.h> + +/* The encoding reservation alone does not provide a compression backend. */ +static inline bool memcomp_available (void ) +{ +return false; +} + +#endif /* _LINUX_MEMCOMP_H */
include/linux/swap.h +11 -2
@@ -62,7 +62,8 @@ static inline int current_is_kswapd(void)*/ #define SWP_PTE_MARKER_NUM 1#define SWP_PTE_MARKER (MAX_SWAPFILES + SWP_HWPOISON_NUM + \ - SWP_MIGRATION_NUM + SWP_DEVICE_NUM) + SWP_MIGRATION_NUM + SWP_DEVICE_NUM + \ + SWP_MEMCOMP_NUM)/* * Unaddressable device memory support. See include/linux/hmm.h and
@@ -86,6 +87,14 @@ static inline int current_is_kswapd(void)#define SWP_DEVICE_NUM 0#endif +#ifdef CONFIG_MEMCOMP +#define SWP_MEMCOMP_NUM 1 +#define SWP_MEMCOMP (MAX_SWAPFILES + SWP_HWPOISON_NUM + \ + SWP_MIGRATION_NUM + SWP_DEVICE_NUM) +#else +#define SWP_MEMCOMP_NUM 0 +#endif +/* * Page migration support. *
@@ -116,7 +125,7 @@ static inline int current_is_kswapd(void)#define MAX_SWAPFILES \ ((1 << MAX_SWAPFILES_SHIFT) - SWP_DEVICE_NUM - \ - SWP_MIGRATION_NUM - SWP_HWPOISON_NUM - \ + SWP_MIGRATION_NUM - SWP_HWPOISON_NUM - SWP_MEMCOMP_NUM - \ SWP_PTE_MARKER_NUM)/*
include/linux/swapops.h +22 -0
@@ -109,6 +109,28 @@ static inline pgoff_t swp_offset(swp_entry_t entry)return entry.val &SWP_OFFSET_MASK ; } +#ifdef CONFIG_MEMCOMP +static inline bool is_memcomp_entry (swp_entry_t entry) +{ +return swp_type (entry) ==SWP_MEMCOMP ; +} + +static inline swp_entry_t make_memcomp_entry (pgoff_t offset) +{ +return swp_entry (SWP_MEMCOMP , offset); +} +#else +static inline bool is_memcomp_entry (swp_entry_t entry) +{ +return false; +} + +static inline swp_entry_t make_memcomp_entry (pgoff_t offset) +{ +return swp_entry (0 ,0 ); +} +#endif +/* * This should only be called upon a pfn swap entry to get the PFN stored * in the swap entry. Please refers to is_pfn_swap_entry() for definition
RFC PATCH 3/16 mm: memcomp: compress reclaimed anonymous pages in memory
In reply to nakixiiAnonymous reclaim normally needs a swap slot before it can unmap a page. Compressing contents in RAM can release the source folio without first issuing block I/O, but each non-present PTE still needs stable backing and ownership across reclaim, fork and faults.
Reserve consecutive tokens before unmapping and publish compressed payloads through the crypto compression API and zsmalloc. Add fault, fork, zap, migration, smaps, HMM and MADV_FREE handling for memcomp entries. Refuse KSM folios rather than applying anonymous restoration rules to merged pages.
Keep stable backing in 16-byte slots containing a payload word, reference count, length and flags. Allocate slot segments on demand within per-key clusters. Keys identify the source object cgroup, NUMA node, page kind and zero hint. Retain a separate record while a resident folio or pending transaction needs producer, member and anonymous-rmap state; the slot size is not a claim about total metadata bytes per stored page.
Route all-zero contents through a separate zero domain without allocating a slot or compressed payload. Account committed anonymous zero PTE owners separately from page-cache zero owners. Same-filled nonzero pages retain their value in a slot without allocating a zsmalloc payload. Keep zero ownership outside slot reference counting and writeback eligibility.
Retire unused metadata with RCU protection. Age retained empty cluster headers using a persistent cursor and batches of 32: each acquisition of the storage lock examines at most 32 xarray entries, including entries that cannot be retired. Release the lock before queuing detached headers for deferred freeing, and schedule another batch when work remains. This bounds the scan count per lock acquisition, not its elapsed time.
Pin anon_vma before replacing present PTEs so rollback can still walk the original anonymous mappings. Restore only matching token PTEs under the page-table lock, with a single-PTE rmap update for the mapped subpage. Use bounded re-walks when pending references remain, yielding between passes. Call deferred_split_folio_remapped() after restoring a large folio so deferred-split accounting follows its remapped state.
Keep the producer reference and pending-record ownership distinct from PTE and lookup references. If bounded rollback leaves token PTEs behind, retain their resident backing until faults or removal consume the last references. Detach the producer's reservation and drop the anon_vma hold only through the corresponding handoff paths.
Record genuine incompressible results using a content hash and marker in folio->swap while the folio is outside the swap cache. Recompute the hash on reclaim to skip unchanged contents, and clear the marker when contents change. Compute the rollback hash before restored mappings can be written. Other store failures do not mark a folio incompressible.
Acquire bounded compression resources before unmapping and release the context before finishing or restoring the folio. Limit repeated foreground anonymous-admission failures within a reclaim pass so file reclaim can progress. Avoid reclaiming allocations while token or storage locks are held.
On fault, handle zero tokens separately and retain slot lookup references through allocation and decompression. Restore resident pending backing through its dedicated path. Revalidate the original PTE before installing a restored folio, preserve PTE attributes, and consume only the ownership replaced by a successful installation. Fall back to a single-page fault when a multi-page mapping cannot be restored together. Return vm_fault_t results and release temporary references on retry and error paths.
Observe dirty PTEs and PMDs before unmapping a swap-cache folio. Release a dirty folio's swap slot so its current contents can be compressed; drop a clean swap-cache folio through the normal path.
Expose stat and compressor attributes under the existing memcomp sysfs directory and document memcomp.compressor= selection. Add direct maintenance coverage for the core and its implementation headers. Ordinary swap remains the fallback before token ownership; compressed writeback is not part of this change. UFFDIO_MOVE returns -EFAULT for memcomp entries until the following change adds their movement.
--- .../admin-guide/kernel-parameters.txt | 10 + MAINTAINERS | 9 +- fs/proc/task_mmu.c | 20 + include/linux/memcomp.h | 297 +- mm/Kconfig | 55 + mm/Makefile | 1 + mm/hmm.c | 3 + mm/huge_memory.c | 44 + mm/internal.h | 84 + mm/madvise.c | 10 +- mm/memcomp.c | 3561 +++++++++++++++++ mm/memcomp/access.h | 132 + mm/memcomp/anon_folio.h | 84 + mm/memcomp/anon_store.h | 245 ++ mm/memcomp/api.h | 149 + mm/memcomp/consumers.h | 64 + mm/memcomp/fault.h | 254 ++ mm/memcomp/fault_alloc.h | 23 + mm/memcomp/fifo.h | 136 + mm/memcomp/folio.h | 386 ++ mm/memcomp/format.h | 308 ++ mm/memcomp/lifetime.h | 87 + mm/memcomp/load.h | 59 + mm/memcomp/migrate.h | 152 + mm/memcomp/owner.h | 111 + mm/memcomp/payload.h | 151 + mm/memcomp/query.h | 187 + mm/memcomp/refs.h | 83 + mm/memcomp/resident.h | 103 + mm/memcomp/scan.h | 115 + mm/memcomp/span.h | 89 + mm/memcomp/storage.h | 963 +++++ mm/memcomp/zero.h | 205 + mm/memcomp/zero_convert.h | 78 + mm/memcomp/zero_load.h | 71 + mm/memory.c | 42 +- mm/migrate.c | 11 +- mm/rmap.c | 29 +- mm/swapfile.c | 11 + mm/vmscan.c | 313 +- 40 files changed, 8723 insertions(+), 12 deletions(-) create mode 100644 mm/memcomp.c create mode 100644 mm/memcomp/access.h create mode 100644 mm/memcomp/anon_folio.h create mode 100644 mm/memcomp/anon_store.h create mode 100644 mm/memcomp/api.h create mode 100644 mm/memcomp/consumers.h create mode 100644 mm/memcomp/fault.h create mode 100644 mm/memcomp/fault_alloc.h create mode 100644 mm/memcomp/fifo.h create mode 100644 mm/memcomp/folio.h create mode 100644 mm/memcomp/format.h create mode 100644 mm/memcomp/lifetime.h create mode 100644 mm/memcomp/load.h create mode 100644 mm/memcomp/migrate.h create mode 100644 mm/memcomp/owner.h create mode 100644 mm/memcomp/payload.h create mode 100644 mm/memcomp/query.h create mode 100644 mm/memcomp/refs.h create mode 100644 mm/memcomp/resident.h create mode 100644 mm/memcomp/scan.h create mode 100644 mm/memcomp/span.h create mode 100644 mm/memcomp/storage.h create mode 100644 mm/memcomp/zero.h create mode 100644 mm/memcomp/zero_convert.h create mode 100644 mm/memcomp/zero_load.h
Documentation/admin-guide/kernel-parameters.txt +10 -0
@@ -3339,6 +3339,16 @@ [KNL,SH] Allow user to override the default size for per-device physically contiguous DMA buffers. + memcomp.compressor= + [KNL] Set the crypto compression algorithm used by + CONFIG_MEMCOMP. The default is + CONFIG_MEMCOMP_DEFAULT_COMPRESSOR. memcomp + waits up to ten seconds for an algorithm that is not + registered by late_initcall; if it does not appear, + or memcomp cannot be set up on it, memcomp + falls back to "zstd", and is disabled if that + fails too. + memhp_default_state=online/offline/online_kernel/online_movable [KNL] Set the initial state for the memory hotplug onlining policy. If not specified, the default value is
MAINTAINERS +8 -1
@@ -14802,6 +14802,14 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMCOMP IN-MEMORY COMPRESSION +M: nakixii <[email protected]> +L: [email protected] +S: Experimental +F: include/linux/memcomp.h +F: mm/memcomp.c +F: mm/memcomp/ + MEMORY ALLOCATION PROFILING M: Suren Baghdasaryan <[email protected]> M: Kent Overstreet <[email protected]>
@@ -14852,7 +14860,6 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm T: quilt git://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new F: include/linux/gfp.h F: include/linux/gfp_types.h -F: include/linux/memcomp.h F: include/linux/memfd.h F: include/linux/memory.h F: include/linux/memory_hotplug.h
fs/proc/task_mmu.c +20 -0
@@ -15,6 +15,7 @@#include <linux/swap.h> #include <linux/sched/mm.h> #include <linux/swapops.h> +#include <linux/memcomp.h> #include <linux/mmu_notifier.h> #include <linux/page_idle.h> #include <linux/shmem_fs.h>
@@ -807,7 +808,26 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, }else if (is_swap_pte (ptent)) {swp_entry_t swpent =pte_to_swp_entry (ptent); +#ifdef CONFIG_MEMCOMP +if (is_memcomp_entry (swpent)) { +int mapcount; + +if (memcomp_entry_stat (swp_offset (swpent), +NULL , &mapcount)) { + mss->swap +=PAGE_SIZE ; +if (mapcount >=2 ) { +u64 pss_delta = (u64 )PAGE_SIZE <<PSS_SHIFT ; + +do_div (pss_delta, mapcount); + mss->swap_pss += pss_delta; + }else { + mss->swap_pss += (u64 )PAGE_SIZE <<PSS_SHIFT ; + } + } + }else if (!non_swap_entry (swpent)) { +#else if (!non_swap_entry (swpent)) { +#endif int mapcount; mss->swap +=PAGE_SIZE ;
include/linux/memcomp.h +295 -2
307 lines not on this page · raw message
mm/Kconfig +55 -0
61 lines not on this page · raw message
mm/Makefile +1 -0
@@ -76,6 +76,7 @@ endif obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o swap_slots.o obj-$(CONFIG_ZSWAP) += zswap.o +obj-$(CONFIG_MEMCOMP) += memcomp.o obj-$(CONFIG_HAS_DMA) += dmapool.o obj-$(CONFIG_HUGETLBFS) += hugetlb.o obj-$(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) += hugetlb_vmemmap.o
mm/hmm.c +3 -0
@@ -273,6 +273,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,if (is_device_exclusive_entry (entry)) goto fault; +if (is_memcomp_entry (entry)) + goto fault; +if (is_migration_entry (entry)) {pte_unmap (ptep); hmm_vma_walk->last = addr;
mm/huge_memory.c +44 -0
62 lines not on this page · raw message
mm/internal.h +84 -0
102 lines not on this page · raw message
mm/madvise.c +9 -1
@@ -7,6 +7,7 @@*/ #include <linux/mman.h> +#include <linux/memcomp.h> #include <linux/pagemap.h> #include <linux/syscalls.h> #include <linux/mempolicy.h>
@@ -680,7 +681,14 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,swp_entry_t entry; entry =pte_to_swp_entry (ptent); -if (!non_swap_entry (entry)) { +if (is_memcomp_entry (entry)) { + max_nr = (end - addr) /PAGE_SIZE ; + nr =memcomp_pte_batch (pte, max_nr); + nr_swap -= nr; +memcomp_invalidate_nr (swp_offset (entry), nr); +clear_not_present_full_ptes (mm, addr, pte, nr, + tlb->fullmm); + }else if (!non_swap_entry (entry)) { max_nr = (end - addr) /PAGE_SIZE ; nr =swap_pte_batch (pte, max_nr, ptent); nr_swap -= nr;
mm/memcomp.c new file +3561 -0
2,360 lines not on this page · raw message
mm/memcomp/access.h new file +132 -0
132 lines not on this page · raw message
mm/memcomp/anon_folio.h new file +84 -0
84 lines not on this page · raw message
mm/memcomp/anon_store.h new file +245 -0
245 lines not on this page · raw message
mm/memcomp/api.h new file +149 -0
149 lines not on this page · raw message
mm/memcomp/consumers.h new file +64 -0
64 lines not on this page · raw message
mm/memcomp/fault.h new file +254 -0
254 lines not on this page · raw message
mm/memcomp/fifo.h new file +136 -0
136 lines not on this page · raw message
mm/memcomp/folio.h new file +386 -0
386 lines not on this page · raw message
mm/memcomp/format.h new file +308 -0
308 lines not on this page · raw message
mm/memcomp/lifetime.h new file +87 -0
87 lines not on this page · raw message
mm/memcomp/load.h new file +59 -0
59 lines not on this page · raw message
mm/memcomp/migrate.h new file +152 -0
152 lines not on this page · raw message
mm/memcomp/owner.h new file +111 -0
111 lines not on this page · raw message
mm/memcomp/payload.h new file +151 -0
151 lines not on this page · raw message
mm/memcomp/query.h new file +187 -0
187 lines not on this page · raw message
mm/memcomp/refs.h new file +83 -0
83 lines not on this page · raw message
mm/memcomp/resident.h new file +103 -0
103 lines not on this page · raw message
mm/memcomp/scan.h new file +115 -0
115 lines not on this page · raw message
mm/memcomp/span.h new file +89 -0
89 lines not on this page · raw message
mm/memcomp/storage.h new file +963 -0
963 lines not on this page · raw message
mm/memcomp/zero.h new file +205 -0
205 lines not on this page · raw message
mm/memcomp/zero_convert.h new file +78 -0
78 lines not on this page · raw message
mm/memcomp/zero_load.h new file +71 -0
71 lines not on this page · raw message
mm/memory.c +41 -1
42 lines not on this page · raw message
mm/vmscan.c +311 -2
313 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
RFC PATCH 4/16 mm: userfaultfd: let UFFDIO_MOVE move memcomp entries
In reply to nakixiiUFFDIO_MOVE rejects memcomp entries as unsupported non-swap entries. Support moving them without requiring the memory-cgroup, shmem, writeback or asynchronous-compression paths.
Move nonresident tokens like swap entries while keeping a lookup reference across PTE revalidation. The reference prevents token reuse from changing the backing whose resident state was checked. Return -EAGAIN for a busy entry or a stale token rather than moving it without a stable handle. Keep zero-token movement neutral to zero ownership.
A resident pending token still names a folio whose anonymous rmap points to the source address. Restore that folio there, then retry through the present-PTE move path instead of moving the resident token directly. Drop both RCU PTE mappings before the token-lock lookup and preserve the existing PTE and exclusivity checks.
Rollback of memcomp entries makes partially unmapped large folios more common. The per-page exclusivity check also covers that state without treating the first page's flag as the flag of every mapped subpage.
--- mm/userfaultfd.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+)
mm/userfaultfd.c +77 -0
@@ -15,6 +15,7 @@#include <linux/mmu_notifier.h> #include <linux/hugetlb.h> #include <linux/shmem_fs.h> +#include <linux/memcomp.h> #include <asm/tlbflush.h> #include <asm/tlb.h> #include "internal.h"
@@ -1150,6 +1151,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,struct anon_vma *src_anon_vma =NULL ;struct mmu_notifier_range range;int err =0 ; +swp_entry_t memcomp_checked_entry = { }; +bool memcomp_token_checked = false; +struct memcomp_ref memcomp_checked_ref = { };flush_cache_range (src_vma, src_addr, src_addr +PAGE_SIZE ); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm,
@@ -1338,6 +1342,77 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd,struct folio *folio =NULL ; entry =pte_to_swp_entry (orig_src_pte); +if (is_memcomp_entry (entry)) { +int resident; + +if (!pte_swp_exclusive (orig_src_pte)) { + err = -EBUSY ; + goto out; + } +/* +* A resident token must be faulted back where its +* folio's rmap expects it; the present-pte path then +* moves the folio. Any other token carries no +* address and moves like a swap entry. +* +* pte_offset_map_nolock() leaves us in an RCU read-side +* section. Drop both mappings before querying the token: +* memcomp_entry_resident_get() may take the token mutex when +* it wins its trylock, and lockdep quite correctly rejects +* even that conditional mutex acquisition in this context. +* The PTEs are revalidated by retry before either move path. +*/ +/* Zero is a pure value: MOVE keeps its logical owner count. */ +if (!swp_offset (entry)) { + err =move_swap_pte (mm, dst_vma, dst_addr, src_addr, + dst_pte, src_pte, orig_dst_pte, + orig_src_pte, dst_ptl, src_ptl,NULL ); + goto out; + } +if (!memcomp_token_checked || + memcomp_checked_entry.val != entry.val) { +if (memcomp_checked_ref.slot) +memcomp_entry_resident_put (&memcomp_checked_ref); +pte_unmap (src_pte); +pte_unmap (dst_pte); + dst_pte =NULL ; + src_pte =NULL ; + resident =memcomp_entry_resident_get (swp_offset (entry), + &memcomp_checked_ref); +if (resident <0 ) { + err = resident; + goto out; + } +if (resident) { +struct vm_fault vmf = { + .vma = src_vma, + .address = src_addr, + .pgoff =linear_page_index (src_vma, src_addr), + .pmd = src_pmd, + .orig_pte = orig_src_pte, + }; + + err =memcomp_restore_resident (&vmf, + &memcomp_checked_ref); + memcomp_token_checked = false; +if (!err) + goto retry; + goto out; + } +if (!memcomp_checked_ref.slot) { +/* Do not move a stale token without a stable entry. */ + err = -EAGAIN ; + goto out; + } + memcomp_checked_entry = entry; + memcomp_token_checked = true; + goto retry; + } + err =move_swap_pte (mm, dst_vma, dst_addr, src_addr, + dst_pte, src_pte, orig_dst_pte, + orig_src_pte, dst_ptl, src_ptl,NULL ); + goto out; + }if (non_swap_entry (entry)) {if (is_migration_entry (entry)) {pte_unmap (src_pte);
@@ -1400,6 +1475,8 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, } out: +if (memcomp_checked_ref.slot) +memcomp_entry_resident_put (&memcomp_checked_ref);if (src_anon_vma) {anon_vma_unlock_write (src_anon_vma);put_anon_vma (src_anon_vma);
RFC PATCH 5/16 mm: memcomp: charge compressed pages to the source memcg
In reply to nakixiiReclaim can execute in a different memory cgroup from the folio that owns the contents. Charging compressed storage to the reclaiming task can move memory usage away from the cgroup that supplied those pages.
Capture and retain the source folio's object cgroup for resident records and per-key storage. Charge allocated metadata and payload bytes to that owner, including allocator-sized metadata where applicable. Nonzero same-filled slots have metadata but no allocated payload; the zero domain has neither a slot nor a payload charge.
Reserve payload charges outside the token lock with non-reclaiming allocation flags. Transfer a charge only when its payload is published, and refund it when allocation or revalidation prevents publication. Refresh releases the superseded payload charge while preserving the ownership needed by its replacement.
Release payload charges with their backing, and retain metadata charges and owner references until their corresponding records, segments, clusters and keys are freed. RCU readers must not outlive the metadata or its accounting owner. Leave restored-folio charging to the ordinary memory-management paths rather than adding a separate cgroup quota.
--- mm/memcomp.c | 22 ++++++++++++++++++---- mm/memcomp/storage.h | 22 ++++++++++++++++++++-- 2 files changed, 38 insertions(+), 6 deletions(-)
mm/memcomp.c +18 -4
@@ -663,29 +663,43 @@ static void memcomp_zs_free(u8 pool_idx, unsigned long handle)static struct obj_cgroup *memcomp_objcg_get (struct folio *folio) { -/* Compressed objects do not have an accounting owner yet. */ -return NULL ; +return get_obj_cgroup_from_folio (folio); }/* Hold an additional metadata owner without bypassing the test model. */ static __maybe_unused void memcomp_objcg_get_ref (struct obj_cgroup *objcg) { -/* No accounting owner was acquired. */ +#ifdef CONFIG_MEMCG +if (objcg) +obj_cgroup_get (objcg); +#endif }static void memcomp_objcg_put (struct obj_cgroup *objcg) { -/* No accounting owner was acquired. */ +obj_cgroup_put (objcg); }static int memcomp_objcg_charge (struct obj_cgroup *objcg,gfp_t gfp,size_t size) { +#if defined(CONFIG_MEMCG ) +if (!objcg || !size) +return 0 ; + +return obj_cgroup_charge (objcg, gfp, size); +#else return 0 ; +#endif }static void memcomp_objcg_uncharge (struct obj_cgroup *objcg,size_t size) { +#if defined(CONFIG_MEMCG ) +if (!objcg || !size) +return ; +obj_cgroup_uncharge (objcg, size); +#endif }#include "memcomp/format.h"
mm/memcomp/storage.h +20 -2
@@ -223,8 +223,26 @@ static inline void mc16_retained_empty_dec_locked(void)static inline void *mc16_kmalloc_owner (size_t bytes,gfp_t gfp,struct obj_cgroup *objcg) { -/* Metadata remains unaccounted until a source owner is introduced. */ -return kmalloc (bytes, gfp & ~__GFP_ACCOUNT); +struct mem_cgroup *memcg =NULL , *old; +void *ptr; + +#ifdef CONFIG_MEMCG +if (objcg) + memcg =get_mem_cgroup_from_objcg (objcg); +#endif + old =set_active_memcg (memcg); +/* An unaccounted/root source must not charge the submitting task. */ +if (objcg) + gfp |= __GFP_ACCOUNT; +else + gfp &= ~__GFP_ACCOUNT; + ptr =kmalloc (bytes, gfp); +set_active_memcg (old); +#ifdef CONFIG_MEMCG +if (memcg) +css_put (&memcg->css); +#endif +return ptr; } static inline void *mc16_owner_alloc(size_t bytes, gfp_t gfp,
RFC PATCH 6/16 mm: memcomp: store shmem and tmpfs pages
In reply to nakixiiAnonymous token PTEs do not cover shmem contents held by a page-cache mapping. Shmem needs an exceptional entry that survives eviction and can be replaced atomically when the contents are faulted back.
Store eligible single-page shmem folios in memcomp and place their tokens in the page-cache xarray. Reserve per-key slot storage with cache ownership distinct from an anonymous producer's resident-record ownership. Prepare payload and accounting before activating the slot references that the published shadow will own.
Keep a lookup reference while allocating, charging and decompressing a refault folio. Confirm that the expected shadow remains installed before committing the page-cache replacement. Only the winning replacement consumes the cache owner; an aborted load retains it. Balance inode accounting, memory charges and temporary references on every exit.
Publish a zero shadow's owner count only when the expected xarray replacement succeeds. Keep that cache-owner ledger separate from anonymous zero PTEs, and do not allocate slot metadata for a zero shadow.
Retain ordinary shmem swap as a fallback. A successful in-memory store needs no swap device, and truncate or swapin races can retry without publishing contents from a stale shadow.
--- include/linux/memcomp.h | 80 +++++++++++++ mm/Kconfig | 3 +- mm/memcomp.c | 46 ++++++++ mm/memcomp/api.h | 34 ++++++ mm/memcomp/cache_folio.h | 70 ++++++++++++ mm/memcomp/cache_store.h | 70 ++++++++++++ mm/memcomp/consumers.h | 55 +++++++++ mm/shmem.c | 238 +++++++++++++++++++++++++++++++++++++++ 8 files changed, 595 insertions(+), 1 deletion(-) create mode 100644 mm/memcomp/cache_folio.h create mode 100644 mm/memcomp/cache_store.h
include/linux/memcomp.h +80 -0
@@ -103,6 +103,9 @@ void memcomp_entry_resident_put(struct memcomp_ref *ref);/* Consumes and clears the held guard on every return. */ int memcomp_restore_resident (struct vm_fault *vmf,struct memcomp_ref *ref);void memcomp_invalidate (unsigned long token); +void memcomp_invalidate_cache (unsigned long token); +/* Called only by the winning zero-cache XA replacement while locked. */ +void memcomp_publish_cache_zero (void );void memcomp_invalidate_nr (unsigned long token,unsigned int nr_pages);void memcomp_invalidate_folio (struct folio *folio);bool memcomp_release_folio (struct folio *folio);
@@ -125,6 +128,30 @@ void memcomp_entry_clear_refresh_pending(unsigned long token);bool memcomp_entry_get (unsigned long token);bool memcomp_entry_present (unsigned long token); +/* +* Page-cache variants for shmem/tmpfs pages. A zero store result prepares +* a value only: its winning XA replacement must publish the cache owner +* before unlocking. An aborted zero preparation creates no owner to drop. +*/ +bool memcomp_store_cache (struct folio *folio,unsigned long *token); +/* +* Pin rejects token0 with an empty guard. Explicit zero load consumes no +* owner; the winning shadow replacement drops its cache owner separately. +* Pin must be resolved with exactly one abort or commit call. Commit also +* drops the shadow owner and is valid only after replacing the expected +* shadow in the page cache. Failed pins clear the output; abort and commit +* consume and clear the guard. Load keeps the guard on every return. +*/ +bool memcomp_load_folio_pin (unsigned long token,struct memcomp_ref *ref); +int memcomp_load_folio (struct memcomp_ref *ref,struct folio *folio); +int memcomp_load_folio_zero (struct folio *folio); +void memcomp_load_folio_abort (struct memcomp_ref *ref); +void memcomp_load_folio_commit (struct memcomp_ref *ref); + +/* Check if a page-cache xa_value is a memcomp entry. */ +bool is_memcomp_shadow (void *shadow); +unsigned long memcomp_shadow_token (void *shadow); +#else static inline bool memcomp_available (void )
@@ -225,6 +252,14 @@ static inline void memcomp_invalidate(unsigned long token) { } +static inline void memcomp_invalidate_cache (unsigned long token) +{ +} + +static inline void memcomp_publish_cache_zero (void ) +{ +} +static inline void memcomp_invalidate_nr (unsigned long token,unsigned int nr_pages) {
@@ -301,6 +336,51 @@ static inline bool memcomp_entry_present(unsigned long token)return false; } +static inline bool memcomp_store_cache (struct folio *folio, +unsigned long *token) +{ +return false; +} + +static inline bool memcomp_load_folio_pin (unsigned long token, +struct memcomp_ref *ref) +{ + *ref = (struct memcomp_ref ) { }; +return false; +} + +static inline int memcomp_load_folio (struct memcomp_ref *ref, +struct folio *folio) +{ + *ref = (struct memcomp_ref ) { }; +return -ENOENT ; +} + +static inline int memcomp_load_folio_zero (struct folio *folio) +{ +return -ENOENT ; +} + +static inline void memcomp_load_folio_abort (struct memcomp_ref *ref) +{ + *ref = (struct memcomp_ref ) { }; +} + +static inline void memcomp_load_folio_commit (struct memcomp_ref *ref) +{ + *ref = (struct memcomp_ref ) { }; +} + +static inline bool is_memcomp_shadow (void *shadow) +{ +return false; +} + +static inline unsigned long memcomp_shadow_token (void *shadow) +{ +return 0 ; +} +#endif #endif
mm/Kconfig +2 -1
@@ -195,7 +195,8 @@ config MEMCOMP select XXHASH default n help - Enable direct in-memory compression for reclaimed anonymous pages. + Enable direct in-memory compression for reclaimed anonymous and + shmem/tmpfs pages. Pages are compressed into RAM and mapped through a special non-swap PTE entry type. This does not require swap block I/O as
mm/memcomp.c +46 -0
70 lines not on this page · raw message
mm/memcomp/api.h +34 -0
@@ -29,6 +29,16 @@ void memcomp_invalidate_nr(unsigned long token, unsigned int nr)mc16_pte_put_nr (token, nr); } +void memcomp_publish_cache_zero (void ) +{ +mc16_zero_cache_get (1 ); +} + +void memcomp_invalidate_cache (unsigned long token) +{ +mc16_cache_drop (token); +} +bool memcomp_entry_present (unsigned long token) {return mc16_entry_present (token);
@@ -146,4 +156,28 @@ int memcomp_restore_resident(struct vm_fault *vmf, struct memcomp_ref *ref) return (ret & VM_FAULT_SIGBUS) ? -EFAULT : 0; } +bool memcomp_store_cache (struct folio *folio,unsigned long *token) +{ +return mc16_store_cache_folio (folio, token); +} + +bool memcomp_load_folio_pin (unsigned long token,struct memcomp_ref *ref) +{ +return mc16_cache_pin (token, ref); +} + +int memcomp_load_folio (struct memcomp_ref *ref,struct folio *folio) +{ +return mc16_cache_load (ref, folio); +} + +void memcomp_load_folio_abort (struct memcomp_ref *ref) +{ +mc16_cache_abort (ref); +} + +void memcomp_load_folio_commit (struct memcomp_ref *ref) +{ +mc16_cache_commit (ref); +}#endif /* _MM_MEMCOMP_API_H */
mm/memcomp/cache_folio.h new file +70 -0
70 lines not on this page · raw message
mm/memcomp/cache_store.h new file +70 -0
70 lines not on this page · raw message
mm/memcomp/consumers.h +55 -0
74 lines not on this page · raw message
mm/shmem.c +238 -0
324 lines not on this page · raw message
RFC PATCH 7/16 mm: memcomp: write cold pages back to swap
In reply to nakixiiCompressed storage can consume its RAM budget even when real swap is available. Give cold allocated payloads a disk-backed state so reclaim can release their zsmalloc pages while preserving the memcomp tokens.
Select candidates through per-key cluster FIFO queues and enter writeback through reclaim scopes. Derive the logical-admission watermark from total RAM and use a 15 percent physical-pool setpoint for feedback. Bound each pass, including visits to zero-reference entries, so stale candidates cannot consume an unbounded reclaim scan. The zero domain has no allocated payload and does not enter these queues.
Pack compressed payloads into shared swap slots and retain raw slots when sharing is not suitable. Use sibling read caching, slot prefetch and conservative sole-survivor defragmentation without changing the swap allocator. Preserve the token's lookup and backing ownership through I/O and revalidate its state before publishing the result.
Keep state transitions under token locking while allocation and I/O run outside non-reclaimable locks. Restore the in-memory state when writeback cannot complete, and release backing only after the last owner leaves. Refund source-object-cgroup payload charges when backing leaves RAM.
Drain the affected swap type before the ordinary swapoff scan. Walk cluster and slot storage with bounded progress, restore the payload charge before converting backing to RAM, and refund losing races. Preserve nofs allocation and the reference lifetime across waits, so swapoff does not reclaim recursively while holding a token lock.
--- include/linux/memcomp.h | 23 + mm/Kconfig | 8 +- mm/memcomp.c | 1742 +++++++++++++++++++++++++++++++++- mm/memcomp/defrag.h | 163 ++++ mm/memcomp/drain.h | 83 ++ mm/memcomp/load.h | 29 + mm/memcomp/swapoff.h | 186 ++++ mm/memcomp/swapoff_walk.h | 122 +++ mm/memcomp/writeback.h | 132 +++ mm/memcomp/writeback_batch.h | 386 ++++++++ mm/swapfile.c | 4 + mm/vmscan.c | 84 ++ 12 files changed, 2942 insertions(+), 20 deletions(-) create mode 100644 mm/memcomp/defrag.h create mode 100644 mm/memcomp/drain.h create mode 100644 mm/memcomp/swapoff.h create mode 100644 mm/memcomp/swapoff_walk.h create mode 100644 mm/memcomp/writeback.h create mode 100644 mm/memcomp/writeback_batch.h
include/linux/memcomp.h +23 -0
@@ -52,6 +52,8 @@ static inline void memcomp_count_normal_swap_fallback(void) { }#endif +void memcomp_reclaim_writeback_begin (int nid,int reclaim_idx); +void memcomp_reclaim_writeback_end (int nid);#ifdef CONFIG_MEMCOMP_DEBUG_STATS void memcomp_count_anon_fail (enum memcomp_anon_fail_event event,unsigned int nr_pages);
@@ -127,6 +129,14 @@ bool memcomp_entry_set_refresh_pending(unsigned long token);void memcomp_entry_clear_refresh_pending (unsigned long token);bool memcomp_entry_get (unsigned long token);bool memcomp_entry_present (unsigned long token); +#ifdef CONFIG_SWAP +int memcomp_swapoff (unsigned int type); +#else +static inline int memcomp_swapoff (unsigned int type) +{ +return 0 ; +} +#endif /* * Page-cache variants for shmem/tmpfs pages. A zero store result prepares
@@ -167,6 +177,14 @@ static inline void memcomp_count_normal_swap_fallback(void) { } +static inline void memcomp_reclaim_writeback_begin (int nid,int reclaim_idx) +{ +} + +static inline void memcomp_reclaim_writeback_end (int nid) +{ +} +static inline void memcomp_count_anon_fail (enum memcomp_anon_fail_event event,unsigned int nr_pages)
@@ -336,6 +354,11 @@ static inline bool memcomp_entry_present(unsigned long token)return false; } +static inline int memcomp_swapoff (unsigned int type) +{ +return 0 ; +} +static inline bool memcomp_store_cache (struct folio *folio,unsigned long *token) {
mm/Kconfig +5 -3
@@ -173,8 +173,9 @@ config ZSWAP_ZPOOL_DEFAULT config MEMCOMP bool "Compress reclaimed pages in memory (memcomp)" # - # SWAP: memcomp installs swap PTEs of its own type and uses the - # generic swap-entry encoding and page-table helpers. + # SWAP: memcomp installs swap PTEs of its own type, hooks swapoff, + # and writes cold payloads back through real swap. Without it the + # file does not even compile. # # ZRAM=n: !ZRAM also permits ZRAM=m for this bool, so require it off. #
@@ -200,7 +201,8 @@ config MEMCOMP Pages are compressed into RAM and mapped through a special non-swap PTE entry type. This does not require swap block I/O as - the first storage layer. + the first storage layer. When real swap is enabled, memcomp + may lazily write cold compressed entries back under memory pressure. If unsure, say N.
mm/memcomp.c +1725 -17
1,863 lines not on this page · raw message
mm/memcomp/defrag.h new file +163 -0
163 lines not on this page · raw message
mm/memcomp/drain.h new file +83 -0
83 lines not on this page · raw message
mm/memcomp/load.h +29 -0
@@ -40,6 +40,35 @@ static inline int mc16_load_page(struct memcomp_ref *ref, void *dst)memset_l (dst, word,PAGE_SIZE /sizeof (unsigned long ));return 0 ; } +#ifdef CONFIG_SWAP +if (state ==MC16_SWAPPED ) { +struct mc16_swap_fields fields; +swp_entry_t swap; + +if (!mc16_swap_decode (word, *view.length, &fields)) { + ret = -EIO ; + goto out; + } + swap =swp_entry (fields.type, fields.swap_offset); +if (!swap.val) { + ret = -EIO ; + goto out; + } +/* Swapoff may consume the entry's distinct owner after T drops. */ + ret =memcomp_slot_dup (swap); +mutex_unlock (memcomp_lock_of (ref->token)); +if (ret) +return ret == -ENOMEM ? -ENOMEM : -EIO ; + ret =memcomp_read_swapped_page (swap, fields.packed, + fields.byte_offset, fields.length, + dst,GFP_HIGHUSER_MOVABLE | + __GFP_NOWARN); +memcomp_slot_put (swap); +if (!ret) +memcomp_debug_inc (&memcomp_swapped_read_io); +return ret; + } +#endif if ((state !=MC16_STORED && state !=MC16_WRITEBACK ) || !word) { ret = -EIO ; goto out;
mm/memcomp/swapoff.h new file +186 -0
186 lines not on this page · raw message
mm/memcomp/swapoff_walk.h new file +122 -0
122 lines not on this page · raw message
mm/memcomp/writeback.h new file +132 -0
132 lines not on this page · raw message
mm/memcomp/writeback_batch.h new file +386 -0
349 lines not on this page · raw message
mm/vmscan.c +84 -0
84 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
RFC PATCH 8/16 mm: memcomp: compress foreground batches asynchronously
In reply to nakixiiForeground reclaim can have several eligible folios ready for compression. Processing each request synchronously prevents a matching asynchronous acomp provider from overlapping those requests.
Prepare bounded batches and submit through the standard asynchronous compression API after unmapping and flushing source translations. Keep requests, completions and payload buffers alive through completion and backlog handling. Drain outstanding requests before releasing a batch that cannot continue.
Consume completed payloads through the store and refresh transactions, revalidating each folio before publication. Acquire store leases after asynchronous waits so a suspended batch does not retain the synchronous context lease. Fall back to synchronous compression when no batch is available, and keep busy-folio handling within the current reclaim pass.
Carry finish-failure handling into deferred processing. Preserve resident rollback, slot and producer ownership, object-cgroup refunds and the ordinary swap fallback when compression finishes but reclaim does not.
Keep task-side batching without a dedicated compression worker. Order compressor selection and late registration so a matching asynchronous form cannot change the chosen synchronous codec. Leave codec implementations unchanged, and provide a KUnit-only asynchronous provider for controlled completion and failure coverage.
--- include/linux/memcomp.h | 59 ++++++ mm/Kconfig | 4 + mm/memcomp.c | 426 ++++++++++++++++++++++++++++++++++++++-- mm/memcomp/anon_store.h | 3 +- mm/vmscan.c | 248 ++++++++++++++++++++++- 5 files changed, 727 insertions(+), 13 deletions(-)
include/linux/memcomp.h +59 -0
@@ -10,6 +10,7 @@struct folio ;struct mc16_slot ;struct mc16_cluster ; +struct memcomp_async ;struct memcomp_reclaim_ctx ;struct vm_fault ;
@@ -24,6 +25,9 @@ struct memcomp_ref {u32 token; }; +/* The most pages one asynchronous compression batch takes. */ +#define MEMCOMP_ASYNC_PAGES 32 +enum memcomp_reclaim_source {MEMCOMP_RECLAIM_DIRECT ,MEMCOMP_RECLAIM_KSWAPD ,
@@ -88,6 +92,23 @@ bool memcomp_reserve(struct folio *folio);/* A non-NULL *@ctxp lease is consumed and cleared on every return. */ bool memcomp_store_after_unmap (struct folio *folio,struct memcomp_reclaim_ctx **ctxp); +/* +* A batch to compress a run of folios asynchronously with, or NULL when the +* compressor cannot or none is free. Submit each folio once it is unmapped, +* wait once, store each with memcomp_store_after_unmap_async(), and put +* the batch back. All batches serve foreground reclaim; a spare batch is +* available only while another batch would remain free. +*/ +struct memcomp_async *memcomp_async_get (void ); +struct memcomp_async *memcomp_async_get_spare (void ); +bool memcomp_async_submit (struct memcomp_async *async, +struct folio *folio); +void memcomp_async_wait (struct memcomp_async *async); +bool memcomp_async_complete (struct memcomp_async *async); +bool memcomp_store_after_unmap_async (struct folio *folio, +struct memcomp_reclaim_ctx **ctxp, +struct memcomp_async *async); +void memcomp_async_put (struct memcomp_async *async);void memcomp_rollback_folio (struct folio *folio);bool memcomp_folio_dirty (struct folio *folio);bool memcomp_folio_incompressible (struct folio *folio);
@@ -233,6 +254,44 @@ memcomp_store_after_unmap(struct folio *folio,return false; } +static inline struct memcomp_async *memcomp_async_get (void ) +{ +return NULL ; +} + +static inline struct memcomp_async * +memcomp_async_get_spare (void ) +{ +return NULL ; +} + +static inline bool memcomp_async_submit (struct memcomp_async *async, +struct folio *folio) +{ +return false; +} + +static inline void memcomp_async_wait (struct memcomp_async *async) +{ +} + +static inline bool memcomp_async_complete (struct memcomp_async *async) +{ +return true; +} + +static inline bool +memcomp_store_after_unmap_async (struct folio *folio, +struct memcomp_reclaim_ctx **ctxp, +struct memcomp_async *async) +{ +return memcomp_store_after_unmap (folio, ctxp); +} + +static inline void memcomp_async_put (struct memcomp_async *async) +{ +} +static inline void memcomp_rollback_folio (struct folio *folio) { }
mm/Kconfig +4 -0
@@ -217,6 +217,10 @@ config MEMCOMP_DEFAULT_COMPRESSOR for registration before trying zstd. Setup failures also fall back to zstd. Memcomp stays disabled if neither can be set up. + The synchronous algorithm choice is final. A matching asynchronous + implementation registered later can enable batched compression + without changing that choice. + config MEMCOMP_DEBUG_STATS bool "Debug statistics for memcomp" depends on MEMCOMP
mm/memcomp.c +415 -11
574 lines not on this page · raw message
mm/memcomp/anon_store.h +2 -1
@@ -172,7 +172,8 @@ mc16_anon_refresh_full(struct memcomp_reclaim_ctx **ctxp, const void *src, ret =memcomp_prepare_payload_locked (ctxp, src + i *PAGE_SIZE ,PAGE_SIZE , &lengths[i], &handle, &pool, nr_pages >1 , - nr_pages ==1 ? store :NULL ); + nr_pages ==1 ? store :NULL , +memcomp_store_precomp (store, i));if (ret) goto payload_fail; payloads[i].word = handle;
mm/vmscan.c +247 -1
287 lines not on this page · raw message
RFC PATCH 9/16 mm: memcomp: batch unmap reserved large folios
In reply to nakixiiBatch uniform PTE runs only for reserved memcomp folios. Retain the single-PTE path for mixed protection, soft-dirty or exclusivity state. Acquire all token owners before clearing mappings and unwind the whole prefix when sharing or architecture checks fail.
Keep TLB ordering, notifier scope and the native rmap completion context.
Do not import the general lazyfree batching policy.
--- mm/memcomp_rmap.h | 139 ++++++++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 12 ++++ 2 files changed, 151 insertions(+) create mode 100644 mm/memcomp_rmap.h
mm/memcomp_rmap.h new file +139 -0
@@ -0,0 +1,139 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _MM_MEMCOMP_RMAP_H +#define _MM_MEMCOMP_RMAP_H + +#ifdef CONFIG_MEMCOMP +/* +* Unlike the newer generic lazyfree batching path, this backport only batches +* a reserved memcomp span. Mixed protection or exclusive bits stay on the +* existing single-PTE path. The caller owns folio lock, PTL and notifier scope. +*/ +static bool memcomp_unmap_exclusive_uniform (struct folio *folio) +{ +bool exclusive =PageAnonExclusive (folio_page (folio,0 )); +unsigned int i; + +for (i =1 ; i <folio_nr_pages (folio); i++) { +if (PageAnonExclusive (folio_page (folio, i)) != exclusive) +return false; + } +return true; +} + +static int memcomp_share_anon_ptes (struct folio *folio,struct page *page, +unsigned int nr) +{ +unsigned int i; + +if (IS_ENABLED (CONFIG_HAVE_GUP_FAST )) +/* Paired with the memory barrier in try_grab_folio(). */ +smp_mb (); +if (unlikely (folio_maybe_dma_pinned (folio))) +return -EBUSY ; +for (i =0 ; i < nr; i++) +ClearPageAnonExclusive (page + i); +if (IS_ENABLED (CONFIG_HAVE_GUP_FAST )) +/* Paired with the smp_rmb() in gup_must_unshare(). */ +smp_mb__after_atomic (); +return 0 ; +} + +/* 1: use the original actor; >1: complete batch; <0: restored, abort walk. */ +static int memcomp_unmap_ptes (struct page_vma_mapped_walk *pvmw, +struct folio *folio,struct vm_area_struct *vma, +enum ttu_flags flags) +{ +struct mm_struct *mm = vma->vm_mm; +unsigned long address = pvmw->address, end, token; +unsigned int i, nr, got =0 ; +struct page *page; +swp_entry_t entry; +pte_t first, old, swap; +bool exclusive; + +if (!folio_memcomp_reserved (folio) || !folio_test_large (folio) || + !folio_test_anon (folio) || !folio_test_swapbacked (folio) || +folio_test_hugetlb (folio) || (flags &TTU_HWPOISON ) || + !memcomp_unmap_exclusive_uniform (folio)) +return 1 ; + first =ptep_get (pvmw->pte); +if (!pte_present (first) ||pte_unused (first)) +return 1 ; + page =pfn_to_page (pte_pfn (first)); + nr =folio_nr_pages (folio) -folio_page_idx (folio, page); + end =min (pmd_addr_end (address, vma->vm_end), vma->vm_end); + nr = min_t(unsigned int, nr, (end - address) >> PAGE_SHIFT); +if (nr <2 ) +return 1 ; +for (i =1 ; i < nr; i++) { +if (!pte_same (pte_advance_pfn (first, i),ptep_get (pvmw->pte + i))) +break ; + } + nr = i; +if (nr <2 ) +return 1 ; + entry =page_swap_entry (page); + token =swp_offset (entry); +if (!is_memcomp_entry (entry) || !token || token >U32_MAX - (nr -1 )) +return 1 ; +/* Each future PTE gets a real owner; failure unwinds the whole prefix. */ +for (; got < nr; got++) { +if (!memcomp_entry_get (token + got)) + goto refs_fail; + } + exclusive =PageAnonExclusive (page); +flush_cache_range (vma, address, address + nr *PAGE_SIZE ); + old =get_and_clear_full_ptes (mm, address, pvmw->pte, nr,0 ); +if (should_defer_flush (mm, flags)) { +for (i =0 ; i < nr; i++) +set_tlb_ubc_flush_pending (mm, old, address + i *PAGE_SIZE ); + }else { +flush_tlb_range (vma, address, address + nr *PAGE_SIZE ); + } +if (pte_dirty (old)) +folio_mark_dirty (folio); +update_hiwater_rss (mm); +for (i =0 ; i < nr; i++) { +if (arch_unmap_one (mm, vma, address + i *PAGE_SIZE , +pte_advance_pfn (old, i)) <0 ) + goto restore; + } +if (exclusive &&memcomp_share_anon_ptes (folio, page, nr)) + goto restore; +if (list_empty (&mm->mmlist)) { +spin_lock (&mmlist_lock); +if (list_empty (&mm->mmlist)) +list_add (&mm->mmlist, &init_mm.mmlist); +spin_unlock (&mmlist_lock); + } +add_mm_counter (mm,MM_ANONPAGES , -(int )nr); +add_mm_counter (mm,MM_SWAPENTS , nr); + swap =swp_entry_to_pte (entry); +if (exclusive) + swap =pte_swp_mkexclusive (swap); +if (pte_soft_dirty (old)) + swap =pte_swp_mksoft_dirty (swap); +if (pte_uffd_wp (old)) + swap =pte_swp_mkuffd_wp (swap); +for (i =0 ; i < nr; i++) +set_pte_at (mm, address + i *PAGE_SIZE , pvmw->pte + i, +pte_move_swp_offset (swap, i)); +folio_remove_rmap_ptes (folio, page, nr, vma); +if (vma->vm_flags &VM_LOCKED ) +mlock_drain_local (); +folio_put_refs (folio, nr); + pvmw->pte += nr -1 ; + pvmw->address += (nr -1 ) *PAGE_SIZE ; +return nr; +restore: +/* Complete invalidation before set_ptes() may form a contpte range. */ +flush_tlb_range (vma, address, address + nr *PAGE_SIZE ); +/* Dirty/young aggregation is conservative; protection bits were uniform. */ +set_ptes (mm, address, pvmw->pte, old, nr); +refs_fail: +while (got) +memcomp_invalidate (token + --got); +return -EBUSY ; +} +#endif /* CONFIG_MEMCOMP */ +#endif /* _MM_MEMCOMP_RMAP_H */
mm/rmap.c +12 -0
@@ -1656,6 +1656,8 @@ void folio_remove_rmap_pmd(struct folio *folio, struct page *page,#endif } +#include "memcomp_rmap.h" +/* * @arg: enum ttu_flags will be passed to this argument */
@@ -1737,6 +1739,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO (!pvmw.pte, folio); +#ifdef CONFIG_MEMCOMP + { +int nr =memcomp_unmap_ptes (&pvmw, folio, vma, flags); + +if (nr <0 ) + goto walk_abort; +if (nr >1 ) +continue ; + } +#endif pfn =pte_pfn (ptep_get (pvmw.pte)); subpage =folio_page (folio, pfn -folio_pfn (folio));
RFC PATCH 10/16 mm: memcomp: queue restored large folios for splitting
In reply to nakixiiRegister a successfully restored multi-page anonymous folio with the deferred split queue before installing its PTEs. Keep the existing single-page, failed-load and rollback paths unchanged.
--- mm/memcomp/fault.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
mm/memcomp/fault.h +4 -2
@@ -218,10 +218,12 @@ static inline vm_fault_t mc16_load(struct vm_fault *vmf, bool *locked)folio_add_new_anon_rmap (folio, vma, start_addr,RMAP_EXCLUSIVE );folio_add_lru_vma (folio, vma);folio_ref_add (folio, nr_pages -1 ); -if (nr_pages >1 ) +if (nr_pages >1 ) { +deferred_split_folio (folio, false);set_ptes (vma->vm_mm, start_addr, install_pte, pte, nr_pages); -else + }else {set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); + }arch_do_swap_page_nr (vma->vm_mm, vma, start_addr, pte, pte, nr_pages);update_mmu_cache_range (vmf, vma, start_addr, install_pte, nr_pages);folio_unlock (folio);
RFC PATCH 11/16 mm: memcomp: account user eviction and refault
In reply to nakixiiPin source memory-cgroup IDs independently of object-charge reparenting. Keep per-base-page stock shadows behind the publication gate, publish after source exit and consume them after charging a new user folio.
Account native eviction, refault, major faults, PSI and delay only for user materialization. Mark real swap transport reservations with a per-device bitmap and clear that marker through the native free path. Keep publication retries and locking fixes with this protocol.
The shmem cache-store caller keeps the initial unpublished cache reference until XArray publication. It keeps the slot, segment and cluster alive while pin performs only lookup and a nonzero reference increment. A failed pin cannot be safely reported by this void helper without invalidating its caller's completed cache transaction, so retain the ownership precondition check. Keep the temporary pin through publication because truncate can then consume the cache reference.
--- include/linux/memcomp.h | 6 ++ include/linux/memcontrol.h | 45 ++++++++++ include/linux/swap.h | 39 ++++++++ mm/memcomp.c | 13 ++- mm/memcomp/anon_store.h | 10 ++- mm/memcomp/api.h | 17 ++++ mm/memcomp/cache_folio.h | 26 +++++- mm/memcomp/cache_store.h | 33 ++++++- mm/memcomp/consumers.h | 6 ++ mm/memcomp/fault.h | 15 +++- mm/memcomp/folio.h | 130 ++++++++++++++++++--------- mm/memcomp/load.h | 4 + mm/memcomp/owner.h | 52 ++++++++++- mm/memcomp/storage.h | 161 +++++++++++++++++++++++++++++----- mm/memcomp/swapoff.h | 4 + mm/memcomp/workingset_load.h | 97 ++++++++++++++++++++ mm/memcomp/workingset_store.h | 52 +++++++++++ mm/memcomp/writeback.h | 3 +- mm/memcontrol.c | 99 +++++++++++++++++++++ mm/page_io.c | 42 +++++++++ mm/shmem.c | 101 +++++++++++---------- mm/swap.h | 4 + mm/swapfile.c | 34 +++++++ mm/vmscan.c | 14 ++- mm/workingset.c | 32 +++++++ 25 files changed, 915 insertions(+), 124 deletions(-) create mode 100644 mm/memcomp/workingset_load.h create mode 100644 mm/memcomp/workingset_store.h
include/linux/memcomp.h +6 -0
@@ -8,6 +8,7 @@#include <linux/types.h> struct folio ; +struct mem_cgroup ;struct mc16_slot ;struct mc16_cluster ;struct memcomp_async ;
@@ -44,6 +45,11 @@ enum memcomp_anon_fail_event {#ifdef CONFIG_MEMCOMP bool memcomp_available (void ); +bool memcomp_release_folio_reclaim (struct folio *folio, +struct mem_cgroup *target_memcg); +void memcomp_publish_cache_eviction (struct folio *folio,struct memcomp_ref *ref); +int memcomp_charge_folio (struct memcomp_ref *ref,struct folio *folio, +struct mm_struct *mm,gfp_t gfp);#ifdef CONFIG_MEMCOMP_DEBUG_STATS void memcomp_count_initial_compress (void );void memcomp_count_normal_swap_fallback (void );
include/linux/memcontrol.h +45 -0
57 lines not on this page · raw message
include/linux/swap.h +39 -0
@@ -306,6 +306,10 @@ struct swap_info_struct {unsigned int max;/* extent of the swap_map */ unsigned char *swap_map;/* vmalloc'ed array of usage counts */ unsigned long *zeromap;/* kvmalloc'ed bitmap to track zero pages */ +#ifdef CONFIG_MEMCOMP +/* One bit per swap offset: 32 KiB for 1 GiB at 4 KiB/page. */ +unsigned long *memcomp_transport; +#endif struct swap_cluster_info *cluster_info;/* cluster info. Only for SSD */ struct list_head free_clusters;/* free clusters list */ struct list_head full_clusters;/* full clusters list */
@@ -358,6 +362,37 @@ struct swap_info_struct {*/ }; +#ifdef CONFIG_MEMCOMP +/* The caller keeps the device alive and owns the allocated swap slot. */ +static inline void swap_memcomp_transport_set (struct swap_info_struct *si, +unsigned long offset) +{ +set_bit (offset, si->memcomp_transport); +} + +static inline void swap_memcomp_transport_clear (struct swap_info_struct *si, +unsigned long offset, +unsigned int nr) +{ +unsigned int i; + +/* Adjacent slots can be freed concurrently under different cluster locks. */ +for (i =0 ; i < nr; i++) +clear_bit (offset + i, si->memcomp_transport); +} + +/* A locked swapcache folio holds SWAP_HAS_CACHE, so the bit is stable. */ +static inline bool swap_memcomp_transport_test (struct swap_info_struct *si, +unsigned long offset) +{ +return test_bit (offset, si->memcomp_transport); +} + +/* Only an inactive device may allocate or release its transport bitmap. */ +int swap_memcomp_transport_alloc (struct swap_info_struct *si); +void swap_memcomp_transport_free (struct swap_info_struct *si); +#endif +static inline swp_entry_t page_swap_entry (struct page *page) {struct folio *folio =page_folio (page);
@@ -372,6 +407,10 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, bool flush);void workingset_age_nonresident (struct lruvec *lruvec,unsigned long nr_pages);void *workingset_eviction (struct folio *folio,struct mem_cgroup *target_memcg); +#ifdef CONFIG_MEMCOMP +void *workingset_memcomp_eviction (struct folio *folio, +struct mem_cgroup *target_memcg); +#endif void workingset_refault (struct folio *folio,void *shadow);void workingset_activation (struct folio *folio);
mm/memcomp.c +11 -2
@@ -3140,6 +3140,7 @@ memcomp_arch_prepare_to_swap(struct folio *folio)/* Allocate @nr_pages consecutive tokens, skipping fragmented free ranges. */ +#include "memcomp/workingset_store.h" #include "memcomp/cache_store.h" #include "memcomp/cache_folio.h"
@@ -3278,7 +3279,7 @@ static void memcomp_writeback_slots_return(void)*/ static int memcomp_writeback_slot_take (swp_entry_t *swapp) { -unsigned int want; +unsigned int want, i;if (!memcomp_writeback_slots_free) { want = min_t(unsigned int, MEMCOMP_WRITEBACK_BATCH,
@@ -3290,6 +3291,13 @@ static int memcomp_writeback_slot_take(swp_entry_t *swapp)get_swap_pages (want, memcomp_writeback_slot_pool,0 );if (!memcomp_writeback_slots_free)return get_nr_swap_pages () <=0 ? -ENOSPC : -ENOMEM ; +/* Mark the whole reservation, including slots later returned unused. */ +for (i =0 ; i < memcomp_writeback_slots_free; i++) { +swp_entry_t entry = memcomp_writeback_slot_pool[i]; + +swap_memcomp_transport_set (swp_swap_info (entry), +swp_offset (entry)); + } } *swapp = memcomp_writeback_slot_pool[--memcomp_writeback_slots_free];
@@ -3473,7 +3481,7 @@ static int memcomp_read_swapped_page(swp_entry_t swap, bool packed,if (!folio)return -ENOMEM ;if (allocated) -swap_read_folio (folio,NULL ); +swap_read_folio_memcomp (folio,NULL );folio_lock (folio);if (!folio_test_uptodate (folio)) {
@@ -3895,6 +3903,7 @@ static bool memcomp_writeback_defrag_pass(void)* Core load - decompress one page from @entry into @dst. Sleeps on the * token lock and possibly swap IO; never called under the PTE lock. */ +#include "memcomp/workingset_load.h" #include "memcomp/load.h" #include "memcomp/consumers.h"
mm/memcomp/anon_store.h +8 -2
@@ -199,8 +199,14 @@ mc16_anon_refresh_full(struct memcomp_reclaim_ctx **ctxp, const void *src,if (ret) goto state_fail;mutex_lock (memcomp_lock_of (refs[i].token)); - ret =mc16_payload_commit_locked (&refs[i], folio, nr_pages, i, - &payloads[i], &tail); +/* Close the gate before STORED/FIFO becomes visible under this T. */ + ret =mc16_store_begin_locked (&refs[i], folio, nr_pages, i); +if (!ret) { + ret =mc16_payload_commit_locked (&refs[i], folio, nr_pages, i, + &payloads[i], &tail); +if (ret) + *mc16_shadow_ptr (&refs[i]) =0 ; + }if (!ret)mc16_zero_carrier_prepare_locked (store, &refs[i], nr_pages);mutex_unlock (memcomp_lock_of (refs[i].token));
mm/memcomp/api.h +17 -0
@@ -121,6 +121,12 @@ bool memcomp_release_folio(struct folio *folio)return mc16_release_folio (folio); } +bool memcomp_release_folio_reclaim (struct folio *folio, +struct mem_cgroup *target_memcg) +{ +return mc16_release_folio_reclaim (folio, target_memcg); +} +void memcomp_rollback_folio (struct folio *folio) {mc16_rollback_folio (folio);
@@ -161,6 +167,17 @@ bool memcomp_store_cache(struct folio *folio, unsigned long *token)return mc16_store_cache_folio (folio, token); } +void memcomp_publish_cache_eviction (struct folio *folio,struct memcomp_ref *ref) +{ +mc16_publish_cache_eviction (folio, ref); +} + +int memcomp_charge_folio (struct memcomp_ref *ref,struct folio *folio, +struct mm_struct *mm,gfp_t gfp) +{ +return mc16_user_charge (ref, folio, mm, gfp); +} +bool memcomp_load_folio_pin (unsigned long token,struct memcomp_ref *ref) {return mc16_cache_pin (token, ref);
mm/memcomp/cache_folio.h +25 -1
@@ -38,7 +38,7 @@ static inline bool mc16_store_cache_folio(struct folio *folio, goto success; } src =kmap_local_folio (folio,0 ); - ret =mc16_cache_store (src, folio, &initial); + ret =mc16_cache_store_prepared (src, folio, &initial);kunmap_local (src);if (ret) goto fail_payload;
@@ -67,4 +67,28 @@ static inline bool mc16_store_cache_folio(struct folio *folio,memcomp_debug_add (nr_pages, &memcomp_shmem_store_fail_pages); return false; } + +/* +* The successful XA replacer borrows a guard acquired before taking XA. +* Cache preparation set PUBLISHING before the initial reference became +* selectable. That marker is never reopened by a cache reader/writer. +* The locked source pins its original folio_memcg until eviction completes; +* truncate may remove the cache owner, but cannot retire our guarded slot. +*/ +static inline void +mc16_publish_cache_eviction (struct folio *folio,struct memcomp_ref *ref) +{ +void *shadow; +bool prepared; + +VM_WARN_ON_ONCE_FOLIO (!folio_test_locked (folio), folio); +mutex_lock (memcomp_lock_of (ref->token)); + prepared = *mc16_shadow_ptr (ref) ==MC16_SHADOW_PUBLISHING ; +mutex_unlock (memcomp_lock_of (ref->token)); +/* Idempotent: an already consumed shadow does not reopen its epoch. */ +if (!prepared) +return ; + shadow =mc16_store_eviction (folio,NULL ); +mc16_store_publish (ref,1 , shadow); +}#endif /* _MM_MEMCOMP_CACHE_FOLIO_H */
mm/memcomp/cache_store.h +30 -3
@@ -17,13 +17,16 @@ static inline void mc16_cache_publish(struct memcomp_ref *ref, unsigned int bytemc16_cache_activate (ref); } -static inline int mc16_cache_store (const void *src,struct folio *account, -struct memcomp_ref *out) +static inline int mc16_cache_store_internal (const void *src, +struct folio *account, +struct memcomp_ref *out, +bool prepared) {struct mc16_initial_payload payload = { };struct obj_cgroup *objcg;unsigned long handle =0 ;unsigned int bytes =0 ; +unsigned short owner_id;u8 pool =0 ;int ret;
@@ -34,6 +37,9 @@ static inline int mc16_cache_store(const void *src, struct folio *account,return -EINVAL ;if (!memcomp_is_enabled ())return -EOPNOTSUPP ; + ret =mc16_source_id_get (account, &owner_id); +if (ret) +return ret;/* Independent source owner, never a borrowed registry key across X1/X2. */ objcg =memcomp_objcg_get (account); ret = memcomp_prepare_payload(src, PAGE_SIZE, &bytes, &handle, &pool,
@@ -50,11 +56,18 @@ static inline int mc16_cache_store(const void *src, struct folio *account,if (ret) goto out_payload; payload = (struct mc16_initial_payload ) { .word = handle, .bytes = bytes }; - ret =mc16_reserve_cache (objcg,folio_nid (account), &payload,1 , out); + ret =mc16_reserve_cache_id (objcg, owner_id,folio_nid (account), + &payload,1 , out);if (ret) {memcomp_objcg_uncharge (objcg, bytes); goto out_payload; } +/* The private refs-zero reservation is not selectable until activation. */ +if (prepared) { +mutex_lock (memcomp_lock_of (out->token)); + *mc16_shadow_ptr (out) =MC16_SHADOW_PUBLISHING ; +mutex_unlock (memcomp_lock_of (out->token)); + }/* No pending/resident/OWNER transition or record/member RMW occurs. */ mc16_cache_publish (out, bytes); ret =0 ;
@@ -65,6 +78,20 @@ static inline int mc16_cache_store(const void *src, struct folio *account, out_owner:if (objcg)memcomp_objcg_put (objcg); +mem_cgroup_memcomp_id_put (owner_id); return ret; } + +static inline int mc16_cache_store (const void *src,struct folio *account, +struct memcomp_ref *out) +{ +return mc16_cache_store_internal (src, account, out, false); +} + +static inline int mc16_cache_store_prepared (const void *src, +struct folio *account, +struct memcomp_ref *out) +{ +return mc16_cache_store_internal (src, account, out, true); +}#endif /* _MM_MEMCOMP_CACHE_STORE_H */
mm/memcomp/consumers.h +6 -0
@@ -70,6 +70,7 @@ static inline bool mc16_cache_pin(unsigned long token, struct memcomp_ref *ref)*/ static inline int mc16_cache_load (struct memcomp_ref *ref,struct folio *folio) { +struct mc16_user_read scope;void *dst;int ret;
@@ -77,9 +78,14 @@ static inline int mc16_cache_load(struct memcomp_ref *ref, struct folio *folio)return -ENOENT ;if (!folio ||folio_test_large (folio) || !folio_test_locked (folio))return -EINVAL ; + ret =mc16_user_refault (ref,1 , folio); +if (ret) +return ret; +mc16_user_read_start (&scope, folio); dst =kmap_local_folio (folio,0 ); ret =mc16_load_page (ref, dst);kunmap_local (dst); +mc16_user_read_end (&scope);if (!ret) {__folio_mark_uptodate (folio);flush_dcache_folio (folio);
mm/memcomp/fault.h +13 -2
@@ -22,6 +22,7 @@ static inline vm_fault_t mc16_load(struct vm_fault *vmf, bool *locked)unsigned int nr_pages =1 , fault_index =0 , i;unsigned long start_addr = vmf->address, id_start = id;struct folio *folio =NULL ; +struct mc16_user_read scope;pte_t *install_pte;vm_fault_t ret =0 ;gfp_t alloc_gfp;
@@ -112,7 +113,7 @@ static inline vm_fault_t mc16_load(struct vm_fault *vmf, bool *locked) ret =VM_FAULT_OOM ; goto out_put; } - err =mem_cgroup_charge (folio, vma->vm_mm, alloc_gfp); + err =mc16_user_charge (&refs[0 ], folio, vma->vm_mm, alloc_gfp);if (err) {if (nr_pages >1 ) {folio_put (folio);
@@ -162,6 +163,12 @@ static inline vm_fault_t mc16_load(struct vm_fault *vmf, bool *locked) goto fallback_single; } } + err =mc16_user_refault (refs, nr_pages, folio); +if (err) { + ret = err == -EAGAIN ?0 :VM_FAULT_SIGBUS ; + goto out_put; + } +mc16_user_read_start (&scope, folio);/* A local highmem mapping covers one page, even for a large folio. */ for (i =0 , err =0 ; i < nr_pages; i++) { dst =kmap_local_folio (folio, i *PAGE_SIZE );
@@ -170,6 +177,7 @@ static inline vm_fault_t mc16_load(struct vm_fault *vmf, bool *locked)if (err)break ; } +mc16_user_read_end (&scope);if (err) {if (err == -EAGAIN ) ret =0 ;
@@ -179,6 +187,9 @@ static inline vm_fault_t mc16_load(struct vm_fault *vmf, bool *locked) ret =memcomp_fault_sigbus_or_retry (vmf); goto out_put; } +/* A new user folio was materialized; a later PTE race does not undo it. */ +mc16_user_major (vma->vm_mm); + ret =VM_FAULT_MAJOR ;memcomp_arch_restore (id_start, folio);__folio_mark_uptodate (folio);/* No T survives mc16_load_page; PTL is acquired only after it returns. */
@@ -234,7 +245,7 @@ static inline vm_fault_t mc16_load(struct vm_fault *vmf, bool *locked)/* Each committed PTE owns a folio ref and relinquishes its token owner. */ for (i =0 ; i < nr_pages; i++)mc16_ref_put_count (&refs[i],2 ); -return VM_FAULT_NOPAGE ; +return VM_FAULT_NOPAGE | ret; out_unlock:pte_unmap_unlock (vmf->pte, vmf->ptl);
mm/memcomp/folio.h +89 -41
155 lines not on this page · raw message
mm/memcomp/load.h +4 -0
@@ -19,6 +19,10 @@ static inline int mc16_load_page(struct memcomp_ref *ref, void *dst) ret = -EIO ; goto out; } +if (*mc16_shadow_ptr (ref) ==MC16_SHADOW_PUBLISHING ) { + ret = -EAGAIN ; + goto out; + }if (*view.flags &MC16_F_REFRESH_PENDING ) { ret = -EAGAIN ; goto out;
mm/memcomp/owner.h +49 -3
70 lines not on this page · raw message
mm/memcomp/storage.h +138 -23
374 lines not on this page · raw message
mm/memcomp/swapoff.h +4 -0
@@ -101,6 +101,10 @@ mc16_swapoff_entry(struct memcomp_ref *ref, unsigned int type, ret = -EIO ; goto out_unlock; } +if (*mc16_shadow_ptr (ref) ==MC16_SHADOW_PUBLISHING ) { + ret =0 ; + goto out_unlock; + }if (mc16_flags_state (*view.flags) !=MC16_SWAPPED ) { ret =0 ; goto out_unlock;
mm/memcomp/workingset_load.h new file +97 -0
97 lines not on this page · raw message
mm/memcomp/workingset_store.h new file +52 -0
52 lines not on this page · raw message
mm/memcomp/writeback.h +2 -1
@@ -28,7 +28,8 @@ mc16_writeback_claim_locked(struct memcomp_ref *ref,struct mc16_view view;u32 bytes; - if (!mc16_view_locked(ref, &view) || + if (*mc16_shadow_ptr(ref) == MC16_SHADOW_PUBLISHING || + !mc16_view_locked(ref, &view) || !mc16_flags_valid(*view.flags) || mc16_flags_state(*view.flags) != MC16_STORED || (*view.flags & MC16_F_RESIDENT) || !*view.word ||
mm/memcontrol.c +99 -0
111 lines not on this page · raw message
mm/page_io.c +42 -0
60 lines not on this page · raw message
mm/shmem.c +57 -44
148 lines not on this page · raw message
mm/swap.h +4 -0
@@ -12,6 +12,10 @@ struct mempolicy;int sio_pool_init (void );struct swap_iocb ;void swap_read_folio (struct folio *folio,struct swap_iocb **plug); +#ifdef CONFIG_MEMCOMP +/* Transport I/O: the user-folio caller owns PSI and delay accounting. */ +void swap_read_folio_memcomp (struct folio *folio,struct swap_iocb **plug); +#endif void __swap_read_unplug (struct swap_iocb *plug);static inline void swap_read_unplug (struct swap_iocb *plug) {
mm/swapfile.c +34 -0
@@ -957,6 +957,22 @@ static void add_to_avail_list(struct swap_info_struct *si)spin_unlock (&swap_avail_lock); } +#ifdef CONFIG_MEMCOMP +int swap_memcomp_transport_alloc (struct swap_info_struct *si) +{ +/* Use virtual fallback for large swap areas, just like zeromap. */ + si->memcomp_transport =kvcalloc (BITS_TO_LONGS ((unsigned long )si->max), +sizeof (unsigned long ),GFP_KERNEL ); +return si->memcomp_transport ?0 : -ENOMEM ; +} + +void swap_memcomp_transport_free (struct swap_info_struct *si) +{ +kvfree (si->memcomp_transport); + si->memcomp_transport =NULL ; +} +#endif +static void swap_range_free (struct swap_info_struct *si,unsigned long offset,unsigned int nr_entries) {
@@ -992,6 +1008,9 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,swap_slot_free_notify (si->bdev, offset); offset++; } +#ifdef CONFIG_MEMCOMP +swap_memcomp_transport_clear (si, begin, nr_entries); +#endif clear_shadow_from_swap_cache (si->type, begin, end);/*
@@ -2900,6 +2919,10 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) p->cluster_next_cpu =NULL ;vfree (swap_map);kvfree (zeromap); +#ifdef CONFIG_MEMCOMP +/* Device users and scans have drained; SWP_USED still prevents reuse. */ +swap_memcomp_transport_free (p); +#endif kvfree (cluster_info);/* Destroy swap account information */ swap_cgroup_swapoff (p->type);
@@ -3500,6 +3523,13 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) goto bad_swap_unlock_inode; } +#ifdef CONFIG_MEMCOMP +/* si->max is final; allocate before enable_swap_info publishes the device. */ + error =swap_memcomp_transport_alloc (si); +if (error) + goto bad_swap_unlock_inode; +#endif +if (si->bdev &&bdev_stable_writes (si->bdev)) si->flags |=SWP_STABLE_WRITES ;
@@ -3605,6 +3635,10 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) inode =NULL ;destroy_swap_extents (si);swap_cgroup_swapoff (si->type); +#ifdef CONFIG_MEMCOMP +/* Clear the pointer before flags allow alloc_swap_info to reuse this si. */ +swap_memcomp_transport_free (si); +#endif spin_lock (&swap_lock); si->swap_file =NULL ; si->flags =0 ;
mm/vmscan.c +12 -2
@@ -843,7 +843,13 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,if (folio_test_swapcache (folio)) {swp_entry_t swap = folio->swap; +#ifdef CONFIG_MEMCOMP +if (reclaimed && !mapping_exiting (mapping) && + !swap_memcomp_transport_test (swp_swap_info (swap), +swp_offset (swap))) +#else if (reclaimed && !mapping_exiting(mapping)) +#endif shadow =workingset_eviction (folio, target_memcg);__delete_from_swap_cache (folio, swap, shadow);mem_cgroup_swapout (folio, swap);
@@ -1150,6 +1156,7 @@ struct memcomp_deferred {struct memcomp_deferred_batch fill;struct memcomp_deferred_batch busy;enum memcomp_reclaim_source source; +struct mem_cgroup *target_memcg;/* Where shrink_folio_list() puts what it frees and what it keeps. */ struct folio_batch *free_folios;struct list_head *ret_folios;
@@ -1201,7 +1208,7 @@ static void memcomp_store_deferred(struct memcomp_deferred *d,memcomp_finish_fail (folio, nr_pages); goto activate; } -if (!memcomp_release_folio (folio)) { +if (!memcomp_release_folio_reclaim (folio, d->target_memcg)) {memcomp_finish_fail (folio, nr_pages); goto keep; }
@@ -1360,6 +1367,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, .busy.folios = LIST_HEAD_INIT(mc_deferred.busy.folios), .source =current_is_kswapd () ?MEMCOMP_RECLAIM_KSWAPD : MEMCOMP_RECLAIM_DIRECT, + .target_memcg = sc->target_mem_cgroup, .free_folios = &free_folios, .ret_folios = &ret_folios, .stat = stat,
@@ -1975,7 +1983,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, }else if (!mapping || !__remove_mapping (mapping, folio, true, #ifdef CONFIG_MEMCOMP sc->target_mem_cgroup)) { -if (mapping || !memcomp_release_folio (folio)) +if (mapping || + !memcomp_release_folio_reclaim (folio, + sc->target_mem_cgroup)) #else sc->target_mem_cgroup)) #endif
mm/workingset.c +32 -0
@@ -379,17 +379,24 @@ void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages)* Return: a shadow entry to be stored in @folio->mapping->i_pages in place * of the evicted @folio so that a later refault can be detected. */ +#ifndef CONFIG_MEMCOMP void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) +#else +static void *workingset_eviction_common (struct folio *folio, +struct mem_cgroup *target_memcg) +#endif {struct pglist_data *pgdat =folio_pgdat (folio);unsigned long eviction;struct lruvec *lruvec;int memcgid; +#ifndef CONFIG_MEMCOMP /* Folio is fully exclusive and pins folio's memory cgroup pointer */ VM_BUG_ON_FOLIO (folio_test_lru (folio), folio);VM_BUG_ON_FOLIO (folio_ref_count (folio), folio);VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio); +#endif if (lru_gen_enabled ())return lru_gen_eviction (folio);
@@ -404,6 +411,31 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) folio_test_workingset(folio)); } +#ifdef CONFIG_MEMCOMP +void *workingset_eviction (struct folio *folio,struct mem_cgroup *target_memcg) +{ +/* Keep the stock exclusivity contract, including its zero-ref check. */ +VM_BUG_ON_FOLIO (folio_test_lru (folio), folio); +VM_BUG_ON_FOLIO (folio_ref_count (folio), folio); +VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio); +return workingset_eviction_common (folio, target_memcg); +} + +/* +* memcomp has irrevocably committed source exit. Anonymous sources are +* frozen; shmem still holds page-cache references while publishing its +* successful XA replacement. The locked, isolated folio pins folio_memcg(). +* Caller holds no T, PTL or XA lock, and has excluded all token consumers. +*/ +void *workingset_memcomp_eviction (struct folio *folio, +struct mem_cgroup *target_memcg) +{ +VM_BUG_ON_FOLIO (folio_test_lru (folio), folio); +VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio); +return workingset_eviction_common (folio, target_memcg); +} +#endif +/** * workingset_test_recent - tests if the shadow entry is for a folio that was * recently evicted. Also fills in @workingset with the value unpacked from
RFC PATCH 12/16 mm: memcomp: report precise token residency
In reply to nakixiiReport whether a token can satisfy a read without I/O for anonymous and shmem mappings, including each base page of a large folio. Query backing swap-cache uptodate state without reading, decompressing or consuming a shadow. Drop PTL and XArray locks around the query and revalidate.
Preserve each base kernel's native syscall and page-size handling. Add test-only query seams with the KUnit patch, not this production step.
--- include/linux/memcomp.h | 7 +++ mm/memcomp.c | 1 + mm/memcomp/api.h | 5 ++ mm/memcomp/mincore.h | 77 +++++++++++++++++++++++++++++ mm/mincore.c | 104 +++++++++++++++++++++++++++++++++++++++- mm/swap_state.c | 5 ++ 6 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 mm/memcomp/mincore.h
include/linux/memcomp.h +7 -0
@@ -45,6 +45,8 @@ enum memcomp_anon_fail_event {#ifdef CONFIG_MEMCOMP bool memcomp_available (void ); +/* Process context only, with no PTL or XArray lock held. */ +bool memcomp_mincore (unsigned long token);bool memcomp_release_folio_reclaim (struct folio *folio,struct mem_cgroup *target_memcg);void memcomp_publish_cache_eviction (struct folio *folio,struct memcomp_ref *ref);
@@ -191,6 +193,11 @@ unsigned long memcomp_shadow_token(void *shadow);#else +static inline bool memcomp_mincore (unsigned long token) +{ +return false; +} +static inline bool memcomp_available (void ) {return false;
mm/memcomp.c +1 -0
@@ -3903,6 +3903,7 @@ static bool memcomp_writeback_defrag_pass(void)* Core load - decompress one page from @entry into @dst. Sleeps on the * token lock and possibly swap IO; never called under the PTE lock. */ +#include "memcomp/mincore.h" #include "memcomp/workingset_load.h" #include "memcomp/load.h" #include "memcomp/consumers.h"
mm/memcomp/api.h +5 -0
@@ -39,6 +39,11 @@ void memcomp_invalidate_cache(unsigned long token)mc16_cache_drop (token); } +bool memcomp_mincore (unsigned long token) +{ +return mc16_mincore (token); +} +bool memcomp_entry_present (unsigned long token) {return mc16_entry_present (token);
mm/memcomp/mincore.h new file +77 -0
77 lines not on this page · raw message
mm/mincore.c +103 -1
143 lines not on this page · raw message
mm/swap_state.c +5 -0
@@ -416,6 +416,11 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping,return ERR_PTR (-ENOENT ); swp =radix_to_swp_entry (folio); +#ifdef CONFIG_MEMCOMP +/* A compressed value is not a folio; mincore queries it outside PTL. */ +if (is_memcomp_entry (swp)) +return ERR_PTR (-EEXIST ); +#endif /* There might be swapin error entries in shmem mapping. */ if (non_swap_entry (swp))return ERR_PTR (-ENOENT );
RFC PATCH 13/16 mm: memcomp: add KUnit tests
In reply to nakixiiThe tokens used by memcomp can outlive individual reclaim attempts and name resident, compressed, same-filled or swap-backed contents. Exercise their bounds, reference ownership and state transitions without relying solely on a userspace workload observing a compressed PTE.
Add 425 named cases covering slot layouts, per-key storage, allocation cancellation, lookup guards, resident records, rollback and fault restoration. Check single-page and multi-page geometry, migration, page- cache replacement, source-object-cgroup accounting, writeback and swapoff lifetime rules.
Cover zero PTE and cache ownership separately from slot references, including successful publication and losing transactions. Exercise bounded metadata aging and writeback scans, asynchronous completion and finish failures, same-filled values and incompressible content.
Retain assertions that only genuine -E2BIG results set the incompressible marker, that rollback installs the folio hash and that unchanged contents skip recompression. Reject KSM folios with a direct eligibility check. Guard advice-dependent cases so configurations without those syscalls still link and report their configuration- specific skips.
Build fixtures and production-side probes only with MEMCOMP_KUNIT_TEST. Include the split test headers from the production compilation unit and provide a dedicated KUnit configuration. Controlled allocation, object- cgroup and I/O models expose otherwise difficult lifetime paths; they do not substitute for end-to-end runtime validation.
--- mm/Kconfig | 13 + mm/memcomp.c | 438 +++ mm/memcomp/anon_store.h | 16 + mm/memcomp/cache_folio.h | 9 + mm/memcomp/consumers.h | 4 + mm/memcomp/defrag.h | 6 + mm/memcomp/fault.h | 53 + mm/memcomp/load.h | 4 + mm/memcomp/migrate.h | 3 + mm/memcomp/mincore.h | 6 + mm/memcomp/resident.h | 3 + mm/memcomp/storage.h | 35 + mm/memcomp/tests/.kunitconfig | 16 + mm/memcomp/tests/access_test.h | 197 + mm/memcomp/tests/aging_test.h | 771 ++++ mm/memcomp/tests/anon_store_test.h | 704 ++++ mm/memcomp/tests/async_test.h | 703 ++++ mm/memcomp/tests/budget_packed_test.h | 435 +++ mm/memcomp/tests/budget_swapped_test.h | 292 ++ mm/memcomp/tests/budget_test.h | 542 +++ mm/memcomp/tests/cache_folio_test.h | 627 +++ mm/memcomp/tests/cache_reserve_test.h | 447 +++ mm/memcomp/tests/cache_store_test.h | 216 + mm/memcomp/tests/cache_xarray_race_test.h | 193 + mm/memcomp/tests/codec_scope_test.h | 94 + mm/memcomp/tests/consumers_test.h | 478 +++ mm/memcomp/tests/core_test.h | 119 + mm/memcomp/tests/defrag_test.h | 601 +++ mm/memcomp/tests/drain_test.h | 581 +++ mm/memcomp/tests/enospc_test.h | 236 ++ mm/memcomp/tests/fault_alloc_test.h | 119 + mm/memcomp/tests/fault_test.h | 851 ++++ mm/memcomp/tests/folio_test.h | 769 ++++ mm/memcomp/tests/format_test.h | 520 +++ mm/memcomp/tests/late_async_test.h | 331 ++ mm/memcomp/tests/load_test.h | 133 + mm/memcomp/tests/migrate_test.h | 795 ++++ mm/memcomp/tests/mincore_test.h | 462 +++ mm/memcomp/tests/owner_test.h | 400 ++ mm/memcomp/tests/payload_test.h | 220 ++ mm/memcomp/tests/public_api_test.h | 422 ++ mm/memcomp/tests/public_driver_test.h | 358 ++ mm/memcomp/tests/query_test.h | 480 +++ mm/memcomp/tests/refs_test.h | 224 ++ mm/memcomp/tests/replacement_test.h | 360 ++ mm/memcomp/tests/resident_test.h | 796 ++++ mm/memcomp/tests/rollback_test.h | 163 + mm/memcomp/tests/scan_test.h | 422 ++ mm/memcomp/tests/shared_test.h | 3477 +++++++++++++++++ mm/memcomp/tests/shmem_retry_test.h | 340 ++ mm/memcomp/tests/shmem_zero_fault_test.h | 121 + mm/memcomp/tests/span_test.h | 128 + mm/memcomp/tests/storage_fixture.h | 84 + mm/memcomp/tests/storage_test.h | 468 +++ mm/memcomp/tests/store_transaction_test.h | 587 +++ mm/memcomp/tests/suite.h | 693 ++++ mm/memcomp/tests/swapoff_test.h | 558 +++ mm/memcomp/tests/swapoff_walk_test.h | 562 +++ mm/memcomp/tests/swapoff_yield_test.h | 532 +++ mm/memcomp/tests/test_api.h | 41 + mm/memcomp/tests/transport_reclaim_test.h | 298 ++ mm/memcomp/tests/transport_test.h | 114 + mm/memcomp/tests/workingset_load_test.h | 1095 ++++++ mm/memcomp/tests/workingset_mthp_error_test.h | 164 + mm/memcomp/tests/workingset_storage_test.h | 442 +++ mm/memcomp/tests/workingset_store_test.h | 572 +++ mm/memcomp/tests/writeback_batch_test.h | 688 ++++ mm/memcomp/tests/writeback_test.h | 540 +++ mm/memcomp/tests/zero_callers_test.h | 717 ++++ mm/memcomp/tests/zero_carrier_test.h | 405 ++ mm/memcomp/tests/zero_convert_test.h | 694 ++++ mm/memcomp/tests/zero_hint_test.h | 860 ++++ mm/memcomp/tests/zero_load_test.h | 645 +++ mm/memcomp/tests/zero_test.h | 313 ++ mm/memcomp/workingset_store.h | 10 + mm/memcomp/writeback_batch.h | 12 + mm/memcomp/zero_load.h | 20 + mm/mincore.c | 21 + 78 files changed, 30868 insertions(+) create mode 100644 mm/memcomp/tests/.kunitconfig create mode 100644 mm/memcomp/tests/access_test.h create mode 100644 mm/memcomp/tests/aging_test.h create mode 100644 mm/memcomp/tests/anon_store_test.h create mode 100644 mm/memcomp/tests/async_test.h create mode 100644 mm/memcomp/tests/budget_packed_test.h create mode 100644 mm/memcomp/tests/budget_swapped_test.h create mode 100644 mm/memcomp/tests/budget_test.h create mode 100644 mm/memcomp/tests/cache_folio_test.h create mode 100644 mm/memcomp/tests/cache_reserve_test.h create mode 100644 mm/memcomp/tests/cache_store_test.h create mode 100644 mm/memcomp/tests/cache_xarray_race_test.h create mode 100644 mm/memcomp/tests/codec_scope_test.h create mode 100644 mm/memcomp/tests/consumers_test.h create mode 100644 mm/memcomp/tests/core_test.h create mode 100644 mm/memcomp/tests/defrag_test.h create mode 100644 mm/memcomp/tests/drain_test.h create mode 100644 mm/memcomp/tests/enospc_test.h create mode 100644 mm/memcomp/tests/fault_alloc_test.h create mode 100644 mm/memcomp/tests/fault_test.h create mode 100644 mm/memcomp/tests/folio_test.h create mode 100644 mm/memcomp/tests/format_test.h create mode 100644 mm/memcomp/tests/late_async_test.h create mode 100644 mm/memcomp/tests/load_test.h create mode 100644 mm/memcomp/tests/migrate_test.h create mode 100644 mm/memcomp/tests/mincore_test.h create mode 100644 mm/memcomp/tests/owner_test.h create mode 100644 mm/memcomp/tests/payload_test.h create mode 100644 mm/memcomp/tests/public_api_test.h create mode 100644 mm/memcomp/tests/public_driver_test.h create mode 100644 mm/memcomp/tests/query_test.h create mode 100644 mm/memcomp/tests/refs_test.h create mode 100644 mm/memcomp/tests/replacement_test.h create mode 100644 mm/memcomp/tests/resident_test.h create mode 100644 mm/memcomp/tests/rollback_test.h create mode 100644 mm/memcomp/tests/scan_test.h create mode 100644 mm/memcomp/tests/shared_test.h create mode 100644 mm/memcomp/tests/shmem_retry_test.h create mode 100644 mm/memcomp/tests/shmem_zero_fault_test.h create mode 100644 mm/memcomp/tests/span_test.h create mode 100644 mm/memcomp/tests/storage_fixture.h create mode 100644 mm/memcomp/tests/storage_test.h create mode 100644 mm/memcomp/tests/store_transaction_test.h create mode 100644 mm/memcomp/tests/suite.h create mode 100644 mm/memcomp/tests/swapoff_test.h create mode 100644 mm/memcomp/tests/swapoff_walk_test.h create mode 100644 mm/memcomp/tests/swapoff_yield_test.h create mode 100644 mm/memcomp/tests/test_api.h create mode 100644 mm/memcomp/tests/transport_reclaim_test.h create mode 100644 mm/memcomp/tests/transport_test.h create mode 100644 mm/memcomp/tests/workingset_load_test.h create mode 100644 mm/memcomp/tests/workingset_mthp_error_test.h create mode 100644 mm/memcomp/tests/workingset_storage_test.h create mode 100644 mm/memcomp/tests/workingset_store_test.h create mode 100644 mm/memcomp/tests/writeback_batch_test.h create mode 100644 mm/memcomp/tests/writeback_test.h create mode 100644 mm/memcomp/tests/zero_callers_test.h create mode 100644 mm/memcomp/tests/zero_carrier_test.h create mode 100644 mm/memcomp/tests/zero_convert_test.h create mode 100644 mm/memcomp/tests/zero_hint_test.h create mode 100644 mm/memcomp/tests/zero_load_test.h create mode 100644 mm/memcomp/tests/zero_test.h
mm/Kconfig +13 -0
@@ -232,6 +232,19 @@ config MEMCOMP_DEBUG_STATS These counters add hot-path accounting and are intended for diagnosis and tuning. If unsure, say N. +config MEMCOMP_KUNIT_TEST + bool "KUnit tests for memcomp" if !KUNIT_ALL_TESTS + depends on MEMCOMP && KUNIT=y + select MEMCOMP_DEBUG_STATS + default KUNIT_ALL_TESTS + help + Build unit tests for memcomp token bounds, reference lifetime, + compression execution admission, payload states, allocation flags and + failure attribution, same-filled pages and swap writeback selection. + The tests run at boot and do not require a swap file. + + If unsure, say N. + config ZBUD tristate "2:1 compression allocator (zbud)" depends on ZSWAP
mm/memcomp.c +438 -0
628 lines not on this page · raw message
mm/memcomp/anon_store.h +16 -0
@@ -118,7 +118,23 @@ mc16_refresh_validate(struct memcomp_ref *ref, struct folio **folio,static inline int mc16_refresh_commit_error (struct memcomp_store_ctx *store) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +int ret; + +if (store && store->kunit_refresh_commit_error) { + ret = store->kunit_refresh_commit_error; + store->kunit_refresh_commit_error =0 ; +return ret; + } + ret =memcomp_kunit_should_fail (MEMCOMP_KUNIT_FAIL_REFRESH_COMMIT ); +if (ret && store && store->kunit_async_probe) { + store->kunit_async_probe->commit_fail_hits++; + store->kunit_async_probe->commit_error = ret; + } +return ret; +#else return 0 ; +#endif }/*
mm/memcomp/cache_folio.h +9 -0
@@ -5,7 +5,16 @@#include <linux/highmem.h> /* Include after cache_store.h and memcomp_arch_prepare_to_swap(). */ +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +#include <kunit/static_stub.h> +static int mc16_cache_folio_arch_prepare (struct folio *folio) +{ +KUNIT_STATIC_STUB_REDIRECT (mc16_cache_folio_arch_prepare, folio); +return memcomp_arch_prepare_to_swap (folio); +} +#else #define mc16_cache_folio_arch_prepare memcomp_arch_prepare_to_swap +#endif /* * Caller holds the order-0 source folio lock throughout preparation and XA
mm/memcomp/consumers.h +4 -0
@@ -74,6 +74,10 @@ static inline int mc16_cache_load(struct memcomp_ref *ref, struct folio *folio)void *dst;int ret; +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_cache_load, ref, folio); +#endif +if (!ref || !ref->slot)return -ENOENT ;if (!folio ||folio_test_large (folio) || !folio_test_locked (folio))
mm/memcomp/defrag.h +6 -0
@@ -6,6 +6,9 @@/* Include after memcomp_defrag_ops, its scan/move limits, swapoff.h and scan.h. */ static inline void *mc16_defrag_buffer_alloc (void ) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_defrag_buffer_alloc); +#endif return kmalloc (PAGE_SIZE ,GFP_NOWAIT | __GFP_NOWARN); }
@@ -17,6 +20,9 @@ static inline int mc16_defrag_copy_payload(const void *src, unsigned int bytes,unsigned int copied =0 ;int ret; +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_defrag_copy_payload, src, bytes, payload); +#endif ret =memcomp_alloc_raw_payload (0 , src, bytes, &copied, &handle, false,NULL );if (!ret)
mm/memcomp/fault.h +53 -0
85 lines not on this page · raw message
mm/memcomp/load.h +4 -0
@@ -12,6 +12,10 @@ static inline int mc16_load_page(struct memcomp_ref *ref, void *dst)void *src;int ret; +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_load_page, ref, dst); +#endif +if (!ref->slot || !dst)return -EINVAL ;mutex_lock (memcomp_lock_of (ref->token));
mm/memcomp/migrate.h +3 -0
@@ -25,6 +25,9 @@ static inline int mc16_migrate_move_error(struct memcomp_ref *ref, struct folio *from, struct folio *to, unsigned int index) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_migrate_move_error, ref, from, to, index); +#endif return 0 ; }
mm/memcomp/mincore.h +6 -0
@@ -4,6 +4,9 @@#include <linux/pagemap.h> #include <linux/swap.h> +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +#include <kunit/static_stub.h> +#endif /* Include after storage/access/refs and the memcomp_slot_* helpers. */ #ifdef CONFIG_SWAP
@@ -12,6 +15,9 @@ static struct folio *mc16_mincore_swap_folio(swp_entry_t swap)struct swap_info_struct *si;struct folio *folio; +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_mincore_swap_folio, swap); +#endif si =get_swap_device (swap);if (!si)return ERR_PTR (-ENOENT );
mm/memcomp/resident.h +3 -0
@@ -6,6 +6,9 @@static inline vm_fault_t mc16_resident_lock_or_retry (struct folio *folio,struct vm_fault *vmf) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_resident_lock_or_retry, folio, vmf); +#endif return folio_lock_or_retry (folio, vmf); }
mm/memcomp/storage.h +35 -0
@@ -35,6 +35,9 @@#include <linux/xarray.h> #include "format.h" +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +#include <kunit/static_stub.h> +#endif #define MC16_CLUSTER_SHIFT 9#define MC16_CLUSTER_SLOTS BIT(MC16_CLUSTER_SHIFT)
@@ -176,11 +179,17 @@ static struct {static unsigned long mc16_aging_now (void ) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_aging_now); +#endif return jiffies; }static void mc16_queue_aging (unsigned long delay) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_queue_aging, delay); +#endif mod_delayed_work (system_unbound_wq, &mc16_retained_aging, delay); }
@@ -188,6 +197,9 @@ static void mc16_schedule_aging(void) {unsigned long irqflags, now, delay; +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_schedule_aging); +#endif xa_lock_irqsave (&mc16_clusters, irqflags);if (mc16_aging.requested) {xa_unlock_irqrestore (&mc16_clusters, irqflags);
@@ -257,12 +269,20 @@ static inline void *mc16_kmalloc_owner(size_t bytes, gfp_t gfp,static inline void *mc16_owner_alloc (size_t bytes,gfp_t gfp,struct obj_cgroup *objcg) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +return mc16_kunit_storage_alloc (bytes, gfp, objcg); +#else return mc16_kmalloc_owner (bytes, gfp, objcg); +#endif }static inline void mc16_queue_rcu (struct rcu_head *head,rcu_callback_t func) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +mc16_kunit_queue_rcu (head, func); +#else call_rcu (head, func); +#endif } static inline void *mc16_alloc_zero(enum mc16_storage_class type,
@@ -284,16 +304,25 @@ static inline void *mc16_alloc_zero(enum mc16_storage_class type,*/ static inline int mc16_source_id_get (struct folio *folio,unsigned short *id) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_source_id_get, folio, id); +#endif return mem_cgroup_memcomp_id_get_checked (folio, id); }static inline bool mc16_owner_id_get_ref (unsigned short id) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_owner_id_get_ref, id); +#endif return mem_cgroup_memcomp_id_get_ref (id); }static inline void mc16_owner_id_put (unsigned short id) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_owner_id_put, id); +#endif mem_cgroup_memcomp_id_put (id); }
@@ -331,6 +360,9 @@ mc16_key_find_id(struct obj_cgroup *objcg, unsigned int nid, u8 kind,static inline int mc16_private_id_alloc (u32 *id) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_private_id_alloc, id); +#endif return __xa_alloc (&mc16_clusters, id,NULL ,XA_LIMIT (1 ,MC16_MAX_CLUSTER_ID ),GFP_NOWAIT | __GFP_NOWARN);
@@ -338,6 +370,9 @@ static inline int mc16_private_id_alloc(u32 *id)static inline struct mc16_key *mc16_key_zalloc (void ) { +#ifdef CONFIG_MEMCOMP_KUNIT_TEST +KUNIT_STATIC_STUB_REDIRECT (mc16_key_zalloc); +#endif return kzalloc (sizeof (struct mc16_key ),GFP_NOIO | __GFP_NOWARN); }
mm/memcomp/tests/.kunitconfig new file +16 -0
@@ -0,0 +1,16 @@ +CONFIG_KUNIT=y +CONFIG_CRYPTO=y +CONFIG_CRYPTO_ZSTD=y +CONFIG_SWAP=y +CONFIG_MMU=y +CONFIG_64BIT=y +CONFIG_ZSMALLOC=y +CONFIG_ZRAM=n +CONFIG_ZSWAP=n +CONFIG_MEMCOMP=y +CONFIG_MEMCOMP_KUNIT_TEST=y +CONFIG_CRYPTO_LZ4=y +# Compile the batched folio-release path with memory accounting enabled. +CONFIG_CGROUPS=y +CONFIG_MEMCG=y +CONFIG_KSM=y
mm/memcomp/tests/access_test.h new file +197 -0
197 lines not on this page · raw message
mm/memcomp/tests/aging_test.h new file +771 -0
771 lines not on this page · raw message
mm/memcomp/tests/anon_store_test.h new file +704 -0
704 lines not on this page · raw message
mm/memcomp/tests/async_test.h new file +703 -0
369 lines not on this page · raw message
mm/memcomp/tests/budget_packed_test.h new file +435 -0
435 lines not on this page · raw message
mm/memcomp/tests/budget_swapped_test.h new file +292 -0
292 lines not on this page · raw message
mm/memcomp/tests/budget_test.h new file +542 -0
542 lines not on this page · raw message
mm/memcomp/tests/cache_folio_test.h new file +627 -0
627 lines not on this page · raw message
mm/memcomp/tests/cache_reserve_test.h new file +447 -0
447 lines not on this page · raw message
mm/memcomp/tests/cache_store_test.h new file +216 -0
216 lines not on this page · raw message
mm/memcomp/tests/cache_xarray_race_test.h new file +193 -0
193 lines not on this page · raw message
mm/memcomp/tests/codec_scope_test.h new file +94 -0
94 lines not on this page · raw message
mm/memcomp/tests/consumers_test.h new file +478 -0
478 lines not on this page · raw message
mm/memcomp/tests/core_test.h new file +119 -0
119 lines not on this page · raw message
mm/memcomp/tests/defrag_test.h new file +601 -0
601 lines not on this page · raw message
mm/memcomp/tests/drain_test.h new file +581 -0
581 lines not on this page · raw message
mm/memcomp/tests/enospc_test.h new file +236 -0
236 lines not on this page · raw message
mm/memcomp/tests/fault_alloc_test.h new file +119 -0
119 lines not on this page · raw message
mm/memcomp/tests/fault_test.h new file +851 -0
851 lines not on this page · raw message
mm/memcomp/tests/folio_test.h new file +769 -0
769 lines not on this page · raw message
mm/memcomp/tests/format_test.h new file +520 -0
520 lines not on this page · raw message
mm/memcomp/tests/late_async_test.h new file +331 -0
331 lines not on this page · raw message
mm/memcomp/tests/load_test.h new file +133 -0
133 lines not on this page · raw message
mm/memcomp/tests/migrate_test.h new file +795 -0
795 lines not on this page · raw message
mm/memcomp/tests/mincore_test.h new file +462 -0
462 lines not on this page · raw message
mm/memcomp/tests/owner_test.h new file +400 -0
400 lines not on this page · raw message
mm/memcomp/tests/payload_test.h new file +220 -0
220 lines not on this page · raw message
mm/memcomp/tests/public_api_test.h new file +422 -0
422 lines not on this page · raw message
mm/memcomp/tests/public_driver_test.h new file +358 -0
358 lines not on this page · raw message
mm/memcomp/tests/query_test.h new file +480 -0
480 lines not on this page · raw message
mm/memcomp/tests/refs_test.h new file +224 -0
224 lines not on this page · raw message
mm/memcomp/tests/replacement_test.h new file +360 -0
360 lines not on this page · raw message
mm/memcomp/tests/resident_test.h new file +796 -0
796 lines not on this page · raw message
mm/memcomp/tests/rollback_test.h new file +163 -0
163 lines not on this page · raw message
mm/memcomp/tests/scan_test.h new file +422 -0
422 lines not on this page · raw message
mm/memcomp/tests/shared_test.h new file +3477 -0
3,477 lines not on this page · raw message
mm/memcomp/tests/shmem_retry_test.h new file +340 -0
340 lines not on this page · raw message
mm/memcomp/tests/shmem_zero_fault_test.h new file +121 -0
121 lines not on this page · raw message
mm/memcomp/tests/span_test.h new file +128 -0
128 lines not on this page · raw message
mm/memcomp/tests/storage_fixture.h new file +84 -0
84 lines not on this page · raw message
mm/memcomp/tests/storage_test.h new file +468 -0
468 lines not on this page · raw message
mm/memcomp/tests/store_transaction_test.h new file +587 -0
587 lines not on this page · raw message
mm/memcomp/tests/suite.h new file +693 -0
693 lines not on this page · raw message
mm/memcomp/tests/swapoff_test.h new file +558 -0
558 lines not on this page · raw message
mm/memcomp/tests/swapoff_walk_test.h new file +562 -0
562 lines not on this page · raw message
mm/memcomp/tests/swapoff_yield_test.h new file +532 -0
532 lines not on this page · raw message
mm/memcomp/tests/test_api.h new file +41 -0
41 lines not on this page · raw message
mm/memcomp/tests/transport_reclaim_test.h new file +298 -0
298 lines not on this page · raw message
mm/memcomp/tests/transport_test.h new file +114 -0
114 lines not on this page · raw message
mm/memcomp/tests/workingset_load_test.h new file +1095 -0
1,095 lines not on this page · raw message
mm/memcomp/tests/workingset_mthp_error_test.h new file +164 -0
164 lines not on this page · raw message
mm/memcomp/tests/workingset_storage_test.h new file +442 -0
442 lines not on this page · raw message
mm/memcomp/tests/workingset_store_test.h new file +572 -0
572 lines not on this page · raw message
mm/memcomp/tests/writeback_batch_test.h new file +688 -0
688 lines not on this page · raw message
mm/memcomp/tests/writeback_test.h new file +540 -0
540 lines not on this page · raw message
mm/memcomp/tests/zero_callers_test.h new file +717 -0
717 lines not on this page · raw message
mm/memcomp/tests/zero_carrier_test.h new file +405 -0
405 lines not on this page · raw message
mm/memcomp/tests/zero_convert_test.h new file +694 -0
694 lines not on this page · raw message
mm/memcomp/tests/zero_hint_test.h new file +860 -0
860 lines not on this page · raw message
mm/memcomp/tests/zero_load_test.h new file +645 -0
645 lines not on this page · raw message
mm/memcomp/tests/zero_test.h new file +313 -0
313 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
RFC PATCH 14/16 selftests/mm: add a memcomp pageout and refault test
In reply to nakixiiExercise anonymous PAGEOUT and refault through kselftest. Require an observed memcomp pagemap entry before reporting memcomp coverage, and check every byte after refault. Report a skip when admission or pagemap visibility prevents observing a compressed entry rather than treating that run as coverage.
Cover clean swap-cache eviction and recompression after a dirty PTE rewrite. Keep allocation, PAGEOUT and refault on the initial CPU so PAGEOUT drains the same per-CPU LRU batch. Report affinity setup errors without changing a case's result or silently skipping its coverage.
Keep swapoff opt-in for a disposable QEMU guest with an explicit boot argument, selected swap area and original swapon flags. Arm a separate recovery process before disabling swap so parent exit requests restoration of the selected area and its original priority. Normal invocation changes no swap policy and never calls swapoff.
Add HMM fault coverage for memcomp entries alongside the existing HMM tests. Register memcomp_selftest with the mm selftests and include its kernel configuration requirements. These tests complement the internal ownership checks by exercising the userspace interfaces and checking restored contents.
--- MAINTAINERS | 1 + tools/testing/selftests/mm/Makefile | 9 +- tools/testing/selftests/mm/config | 11 + tools/testing/selftests/mm/hmm-tests.c | 112 ++++ tools/testing/selftests/mm/memcomp.c | 831 +++++++++++++++++++++++++ 5 files changed, 963 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/mm/memcomp.c
MAINTAINERS +1 -0
@@ -14809,6 +14809,7 @@ S: Experimental F: include/linux/memcomp.h F: mm/memcomp.c F: mm/memcomp/ +F: tools/testing/selftests/mm/memcomp.c MEMORY ALLOCATION PROFILING M: Suren Baghdasaryan <[email protected]>
tools/testing/selftests/mm/Makefile +8 -1
@@ -43,6 +43,9 @@ LDLIBS = -lrt -lpthread -lm # warnings. CFLAGS += -U_FORTIFY_SOURCE +# Run memcomp directly via kselftest; run_vmtests.sh need not invoke it. +TEST_GEN_PROGS += memcomp_selftest + TEST_GEN_FILES = cow TEST_GEN_FILES += compaction_test TEST_GEN_FILES += gup_longterm
@@ -139,9 +142,13 @@ TEST_FILES += write_hugetlb_memory.sh include ../lib.mk -$(TEST_GEN_PROGS): vm_util.c thp_settings.c +$(filter-out $(OUTPUT)/memcomp_selftest,$(TEST_GEN_PROGS)): vm_util.c thp_settings.c $(TEST_GEN_FILES): vm_util.c thp_settings.c +$(OUTPUT)/memcomp_selftest: memcomp.c vm_util.h ../kselftest.h + @mkdir -p $(@D) + $(CC) $(filter-out -no-pie,$(CFLAGS)) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) $< $(LDLIBS) -o $@ + $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c
tools/testing/selftests/mm/config +11 -0
@@ -7,3 +7,14 @@ CONFIG_TEST_HMM=m CONFIG_GUP_TEST=y CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_MEM_SOFT_DIRTY=y +CONFIG_SYSFS=y +CONFIG_PROC_PAGE_MONITOR=y +CONFIG_SWAP=y +CONFIG_CRYPTO=y +CONFIG_CRYPTO_ZSTD=y +CONFIG_ZSMALLOC=y +CONFIG_64BIT=y +CONFIG_MMU=y +CONFIG_MEMCOMP=y +CONFIG_ZRAM=n +CONFIG_ZSWAP=n
tools/testing/selftests/mm/hmm-tests.c +112 -0
124 lines not on this page · raw message
tools/testing/selftests/mm/memcomp.c new file +831 -0
831 lines not on this page · raw message
RFC PATCH 15/16 mm: memcomp: add the design doc
In reply to nakixiiThe memcomp backend spans page-table ownership, compressed storage and real swap. Document the design so readers can follow which object keeps contents alive when reclaim, faults, fork and backing conversion overlap.
Describe token encoding, 16-byte slots, per-key clusters and temporary resident records. Explain the separate zero-owner domain, nonzero same- filled values and source-object-cgroup accounting without treating the slot layout as the entire per-page metadata cost.
Document anonymous unmap and rollback, single-PTE rmap restoration, fault revalidation, shmem shadows, writeback and swapoff. Explain the lookup, producer and mapping references and the locking boundaries that keep allocation or I/O out of non-reclaimable critical sections.
Describe bounded compression contexts, foreground asynchronous batches and metadata aging in batches of at most 32 examined entries per storage lock acquisition. Document the stock crypto and zsmalloc dependencies, supported contexts and restrictions, and link the design from the memory-management documentation index.
--- Documentation/mm/index.rst | 1 + Documentation/mm/memcomp.rst | 215 +++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 217 insertions(+) create mode 100644 Documentation/mm/memcomp.rst
Documentation/mm/index.rst +1 -0
@@ -46,6 +46,7 @@ documentation, or deleted if it has served its purpose. hwpoison hugetlbfs_reserv ksm + memcomp memory-model mmu_notifier multigen_lru
Documentation/mm/memcomp.rst new file +215 -0
215 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -14806,6 +14806,7 @@ MEMCOMP IN-MEMORY COMPRESSION M: nakixii <[email protected]> L: [email protected] S: Experimental +F: Documentation/mm/memcomp.rst F: include/linux/memcomp.h F: mm/memcomp.c F: mm/memcomp/
RFC PATCH 16/16 mm: memcomp: add the admin guide
In reply to nakixiiThe memcomp backend has boot-time policy, compressor selection and observable statistics that need an operational description separate from the internal ownership design.
Explain the experimental configuration and default-off policy, including built-in compressor requirements and the memcomp.compressor= override. Describe initialization and compressor availability without implying a runtime enable or disable interface.
Document stat and compressor under the established memcomp sysfs directory, with optional diagnostic fields gated by MEMCOMP_DEBUG_STATS. Describe writeback policy, accounting limits and supported operation. Add the sysfs ABI descriptions and link the guide from the memory- management index.
Keep implementation, documentation and selftest maintenance coverage aligned with the series. State operational restrictions and untested areas explicitly rather than promising behavior beyond the supported configuration or available validation.
--- .../ABI/testing/sysfs-kernel-mm-memcomp | 363 ++++++++++++++++++ Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/memcomp.rst | 149 +++++++ MAINTAINERS | 2 + 4 files changed, 515 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-memcomp create mode 100644 Documentation/admin-guide/mm/memcomp.rst
Documentation/ABI/testing/sysfs-kernel-mm-memcomp new file +363 -0
@@ -0,0 +1,363 @@ +What: /sys/kernel/mm/memcomp/stat +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Read-only memcomp capacity counters. The three decimal + fields are logical stored bytes, compressed payload bytes, and + zsmalloc pool bytes, in that order. A pending logical page is + still resident and must not be treated as committed capacity. + + Slotless zero tokens contribute one base page per logical owner. + Anonymous PTEs and shmem page-cache shadows have independent + ownership counts. Fork adds a zero PTE owner; fault, zap and + MADV_FREE remove it. Moving a zero PTE does not change the count. + A present shmem mapping adds no owner beyond its cache shadow. + Positive slots retain the per-slot counting convention, so fork + can increase the logical zero capacity without allocating any + entry metadata. Counter fields remain observational snapshots. + +What: /sys/kernel/mm/memcomp/compressor +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Read-only name of the compression algorithm selected at boot. + While memcomp is still waiting for a compressor that has + not registered yet, this is the name it waits for; it changes + to "zstd" if memcomp falls back to that. + +What: /sys/kernel/mm/memcomp/events +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Present only when CONFIG_MEMCOMP_DEBUG_STATS=y. + + Read-only text with one key, one space, a decimal value and a + newline per line. Reading does not reset statistics. Values are + collected independently, so a read is not a coherent snapshot + against concurrent work. + + The ``folio_owner_pages``, ``samefill_pages``, ``raw_pages`` and + ``shmem_pages`` fields are current entry-state gauges. They count + entries with a retained folio owner, same-filled payloads, page-sized + raw payloads and shmem ownership, respectively. Owner detachment + and payload replacement update the corresponding gauge; + entry removal subtracts its remaining state before payload release. + These are sums of signed per-CPU values, not an object index or a + transactional snapshot. Cross-CPU reads may race concurrent updates; + negative aggregate samples are clamped to zero. Compare quiescent + samples when checking balanced ownership. + + Event counters accumulate; they are not instantaneous state. The + other current gauges are: ``enabled``, ``comp_ctx_count``, + ``comp_ctx_reserved``, ``comp_ctx_borrow_limit``, ``comp_ctx_free``, + ``comp_ctx_waiters``, ``comp_ctx_borrowed_active``, ``pending_pages``, + ``swapped_entries`` and ``zs_huge_class_size``. ``enabled`` is 1 + when memcomp is enabled and 0 otherwise. ``zs_huge_class_size`` is + the request boundary, in bytes, below which zsmalloc can share + backing pages; zero means memcomp is not enabled or no allocator + pool is initialized. + + ``comp_ctx_borrowed_peak`` is the largest recorded count of + simultaneously borrowed compression contexts, not current + occupancy. ``comp_ctx_count`` and ``comp_ctx_reserved`` report + the initialized compression-context pool sizes. + + ``swapped_read_io`` counts successful read/decode calls for + swap-backed pages, including calls satisfied from swapcache; it is + not a physical I/O count or proof of a later PTE mapping. ``swapped_read_error`` counts read-page or decode + failures. ``swapoff_entries`` counts entries successfully converted + back to RAM backing. ``swapoff_read_io`` counts successful read-page + operations in that conversion path. These four fields are + accumulated events, not current gauges. + + Compression staging and reservation rollback/split diagnostics are + reported by ``execution_events``, not by this file. + +What: /sys/kernel/mm/memcomp/placement_events +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Present only when CONFIG_MEMCOMP_DEBUG_STATS=y. + + Read-only text with one key, one space, a decimal value and a + newline per line. Reading does not reset statistics. Values are + collected separately and do not form a coherent whole-file snapshot. + + Accumulated routing-attempt counts: ``compress`` counts attempts to + route anonymous or shmem folios to memcomp; ``normal_swap_fallback`` + counts fallback attempts to ordinary swap after a memcomp attempt. + These are routing events, not successful stores, reclaimed pages or + byte counts. An event can concern a multi-page folio; retries can + cause further events. + +What: /sys/kernel/mm/memcomp/packed_swap_events +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Present only when CONFIG_MEMCOMP_DEBUG_STATS=y. + + Read-only text with one key, one space, a decimal value and a + newline per line. Reading does not reset statistics. Values are + collected separately and do not form a coherent whole-file snapshot. + + Compressed-pool writeback diagnostics. The following fields are + current gauges: + + * ``writeback_pool_pages``: allocator backing footprint in base + pages. + * ``writeback_pressure_scopes``: active reclaim writeback scopes, + summed across memory nodes, including overlapping scopes. + * ``writeback_gate_open``: last evaluated writeback-gate state, 0 + for closed or 1 for open. + + The remaining fields are accumulated counts: + + * ``writeback_pressure_events``: node scope transitions from no + active scope to an active scope. + * ``writeback_pressure_scans``: eligible pressure-driven writeback + scan invocations. + * ``writeback_compression_throttled``: budget-grant episodes with + effective compression feedback; this is not a count of compressor + calls delayed. Feedback is effective after sufficient logical + stored-byte growth when backing-byte growth is at most half that + growth. + * ``writeback_limited_batches``: scanned batches which exhaust their + node's writeback budget. + * ``writeback_budget_granted``: sum of payload-entry allowances + granted, not bytes or completed writes. + * ``writeback_payloads``: payload entries successfully converted + from in-pool storage to swap-backed storage. + * ``writeback_zs_reclaimed``: sum of observed decreases in + allocator backing base pages during pressure scans, including + compaction. It is not a logical-page count or exclusive + attribution against concurrent pool changes. + * ``writeback_zs_no_progress``: pressure scans which converted + payloads but observed no decrease in backing pages after + compaction. + * ``writeback_decompress_error``: failures to decompress a payload + when preparing an unpacked swap page. + * ``writeback_write_error``: submitted swap-page slots observed + dirty after waiting for writeback, indicating write failure. This + counts slots, not the payload entries within them. + * ``writeback_gate_opens``: recorded transitions from a closed gate + to an open gate. + * ``writeback_packed_slots``: packed backing-page slots submitted + for writeback, not successfully completed I/O. + * ``writeback_packed_payloads``: payload entries successfully + converted to packed swap storage. Multiple payloads can share a + slot. + * ``writeback_defrag_moved``: packed swap payload entries + successfully moved back into the compressed pool by + defragmentation. + + Submission, conversion and page-footprint statistics have different + units and accounting points; they must not be equated. + +What: /sys/kernel/mm/memcomp/admission_events +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Present only when CONFIG_MEMCOMP_DEBUG_STATS=y. + + Read-only text with one key, one space, a decimal value and a + newline per line. Reading does not reset statistics. Values are + collected separately and do not form a coherent whole-file snapshot. + + Accumulated direct-reclaim admission diagnostics: + + ``reclaim_ctx_early_reject`` counts direct-reclaim context prechecks + which rejected an attempt; ``reclaim_ctx_early_reject_pages`` sums + the associated base pages. This is a precheck result, not proof of + an eventual failed store. + +What: /sys/kernel/mm/memcomp/context_events +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Present only when CONFIG_MEMCOMP_DEBUG_STATS=y. + + Read-only text with one key, one space, a decimal value and a + newline per line. Reading does not reset statistics. Values are + collected separately and do not form a coherent whole-file snapshot. + + Compression-context timing diagnostics. ``sample_interval`` is the + lease sampling period, currently 256 successful reclaim-context + acquisitions per context and reclaim source, with staggered initial + phases. Sampled values are raw measurements, not extrapolated totals + for all work. + + ``wait_events`` counts entries into the context wait path, including + retries, independently of lease sampling. ``wait_ns`` is accumulated + elapsed wait-path time in nanoseconds and ``wait_max_ns`` is its + largest recorded individual duration; a wait-path entry need not + actually sleep. + + The prefixes ``direct`` and ``bg`` select direct reclaim and all + non-direct reclaim sources. Each prefix has the following suffixes, + joined with an underscore: + + * ``samples`` counts accounted sampled leases; ``pages`` sums their + associated base pages, not pages successfully reclaimed. + * ``store_samples`` counts sampled leases that reached a store + attempt; ``store_success`` counts successful store outcomes among + them. + * ``lease_ns`` is accumulated context-holding time; ``lease_max_ns`` + is its recorded maximum per sampled lease. A lease ends when the + context is released or its output is transferred to a staging + buffer; later store work may continue after it. + * ``pre_store_ns`` is accumulated time from lease acquisition to the + beginning of a store; ``pre_store_max_ns`` is its recorded + maximum. A lease that never starts a store contributes its full + lease duration here. + * ``compress_ns`` is accumulated elapsed compression time; + ``compress_max_ns`` is the maximum accumulated compression + duration for one sampled lease. + * ``compress_cpu_ns`` is accumulated task CPU time spent in measured + compression; ``compress_call_cpu_max_ns`` is the largest recorded + CPU duration of a timed compression operation. + * ``compress_store_success_ns`` and + ``compress_store_success_cpu_ns`` are elapsed and CPU + compression-time totals for sampled leases whose store succeeded. + ``compress_store_fail_ns`` and ``compress_store_fail_cpu_ns`` are + the corresponding totals for leases without a successful store, + including leases that never started one. + * ``zs_ns`` is accumulated measured allocator/payload-placement + time; ``zs_max_ns`` is its maximum per sampled lease. + * ``store_other_ns`` is accumulated store elapsed time after + subtracting measured compression and allocator time, floored at + zero; ``store_other_max_ns`` is its maximum per sampled lease. + + Each prefix additionally has ``compress_<outcome>_calls``, + ``compress_<outcome>_ns`` and ``compress_<outcome>_cpu_ns`` for + ``compressible``, ``raw_huge``, ``raw_ge_page`` and ``error`` + outcomes. These count timed compression operations and accumulate + their elapsed and task CPU nanoseconds. The outcomes mean a result + below both storage boundaries, a result reaching the allocator + huge-class boundary but below input size, a result at least input + size, and a compression-result error. They are distinct from the + later store-success classification. + + All durations are nanoseconds. Counts and duration totals + accumulate; maxima are recorded high-water durations, not gauges. + Lease and store phases must not be assumed to be a disjoint + decomposition of end-to-end request latency. + +What: /sys/kernel/mm/memcomp/context_sources +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Present only when CONFIG_MEMCOMP_DEBUG_STATS=y. + + Read-only text with one key, one space, a decimal value and a + newline per line. Reading does not reset statistics. Values are + collected separately and do not form a coherent whole-file snapshot. + + Source-attributed context timing. ``sample_interval`` reports the + lease sampling period, currently 256 successful reclaim-context + acquisitions per context and source, with staggered initial phases. + Lease measurements are sampled, not extrapolated; wait-path + measurements are not lease-sampled. + + Prefixes are ``direct`` for direct reclaim and ``kswapd`` for the + regular reclaim daemon. For each prefix, suffixes are joined with + an underscore: + + * ``samples`` and ``pages`` count sampled leases and their + associated base pages, not successful reclaim. + * ``store_samples`` and ``store_success`` count sampled store + attempts and successful store outcomes. + * ``wait_events`` counts context wait-path entries, including + retries; ``wait_ns`` sums their elapsed nanoseconds; + ``wait_max_ns`` is the largest individual duration. A wait-path + entry need not sleep. + * ``lease_ns``, ``pre_store_ns``, ``compress_ns``, ``zs_ns`` and + ``store_other_ns`` are accumulated elapsed nanoseconds; each has a + corresponding ``*_max_ns`` field for its recorded per-lease + maximum. + * ``compress_cpu_ns`` is accumulated compression task CPU time; + ``compress_call_cpu_max_ns`` is the maximum CPU duration of a + timed compression operation. + + Lease, pre-store, compression, allocator and residual store phases + have the same meanings as in + /sys/kernel/mm/memcomp/context_events. A context lease can end + before staged store work completes, and a lease without a store + contributes its full duration to pre-store time. Counts and times + accumulate; maxima are historical measurements, not current + occupancy. A read is not a consistent snapshot across source groups. + +What: /sys/kernel/mm/memcomp/execution_events +Date: July 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: + Present only when CONFIG_MEMCOMP_DEBUG_STATS=y. + + Read-only text with one key, one space, a decimal value and a + newline per line. Reading does not reset statistics. Values are + collected separately and do not form a coherent whole-file snapshot. + + Compression execution budget, staging-buffer and reservation + diagnostics. ``comp_budget_limit`` and + ``comp_budget_background_limit`` report the current background + execution-credit limit; ``comp_budget_active`` and + ``comp_budget_background_active`` report currently held background + credits. Their ``comp_budget_peak`` and + ``comp_budget_background_peak`` fields are recorded occupancy + high-water counts. These are credits, not pages or all contexts in + use. + + ``comp_budget_direct_limit``, ``comp_budget_direct_active`` and + ``comp_budget_direct_peak`` currently report zero: direct reclaim + does not consume these background execution credits. Zero does not + mean that direct compression is disabled. ``comp_budget_waiters`` is + the current number of registered context waiters, not only waiters + for budget exhaustion. Limits and occupancy are gauges; peaks can + restart if the associated context resources are reinitialized. + + ``reclaim_ctx_budget_pages`` accumulates base pages associated with + context misses attributed to execution-budget exhaustion. + ``direct_budget_reject`` counts direct-reclaim precheck or + acquisition failures attributed to that budget; + ``direct_budget_reject_pages`` sums their base pages. + ``kswapd_budget_wait`` counts reclaim-daemon context wait-loop + attempts attributed to budget exhaustion, not slept time or unique + callers. + + ``direct_staging_count`` is the initialized number of staging + buffers. ``direct_staging_free`` and ``direct_staging_active`` are + current free and occupied buffer counts. ``direct_staging_peak`` is + a recorded occupancy high-water count and can restart on resource + reinitialization. ``direct_staging_miss`` counts unsuccessful + attempts to reserve a staging buffer; compression can continue + without staging. ``direct_staging_detach`` counts transfers of + output and timing state to staging that release the compression + context before completing the store; it does not indicate store + success. + + Additional accumulated reservation/split diagnostics: + + * ``restore_spans`` counts reservation rollbacks requiring an + initial reverse-mapping restore walk; ``restore_rewalks`` counts + additional walks while reservation tokens remain referenced. + * ``restore_ptes`` counts page-table entries mapped back to resident + folio pages during those walks. + * ``restore_missed_pages`` sums base pages still referenced by + reservation-token PTEs after those walks when the reservation is + dropped. A subsequent fault restores the resident page instead. + * ``restore_ns`` sums elapsed reverse-mapping walk nanoseconds; + ``restore_max_ns`` is the largest recorded individual walk + duration. + * ``split_refused`` counts refused anonymous-folio splits due to a + reservation; ``split_refused_pages`` sums their base pages. + + Event counts and durations are accumulated statistics; gauge, limit, + initialized-capacity and peak fields are not interchangeable. + Separate locks and counter reads do not provide a coherent + whole-file snapshot. + + ``aging_work_calls`` counts retained-header aging callbacks; + ``aging_detached_headers`` counts the empty current headers they + detach. Both are accumulated per-CPU events. They do not reset on + read and are not atomic with ``retained_empty_headers`` or the + metadata allocation/free gauges. Detaching queues existing RCU + callbacks; it does not mean those callbacks have completed.
Documentation/admin-guide/mm/index.rst +1 -0
@@ -30,6 +30,7 @@ the Linux memory management. hugetlbpage idle_page_tracking ksm + memcomp memory-hotplug multigen_lru nommu-mmap
Documentation/admin-guide/mm/memcomp.rst new file +149 -0
149 lines not on this page · raw message
MAINTAINERS +2 -0
@@ -14806,6 +14806,8 @@ MEMCOMP IN-MEMORY COMPRESSION M: nakixii <[email protected]> L: [email protected] S: Experimental +F: Documentation/ABI/testing/sysfs-kernel-mm-memcomp +F: Documentation/admin-guide/mm/memcomp.rst F: Documentation/mm/memcomp.rst F: include/linux/memcomp.h F: mm/memcomp.c
Mike Rapoport
maintainerHi,
Quoted 71 lines · nakixii
This is a proof of concept, posted to ask whether the approach is worth pursuing. It is not meant for merging. The series is based on Linux 6.12.23, not on mm-unstable, and has not been ported to the swap-table and zpool changes that have landed since. If there is interest, the next version will be rebased onto mm-unstable.
memcomp stores reclaimed anonymous and shmem pages in compressed form in memory, using the kernel's compressors and zsmalloc unchanged. Reclaim replaces the PTE, or the shmem page-cache entry, with a swap-like token that names a 16-byte slot. A fault on the token decompresses the page and maps it again. No block device is needed. If swap is configured, cold entries can be written back to it. The sysfs interface is under
/sys/kernel/mm/memcomp.Results
=======The comparison ran 965 fresh guest boots on real application pages, under nested KVM on WSL2, with a paired rule fixed before collection. Selected results:
Without real swap, single-page refault p99 was 10.7 us against 20.7 us for zram with lz4, and 22.0 us against 27.3 us with zstd. Under 512 MiB of anonymous pressure, refault p99 was 37% lower than zram with lz4 and 36% lower than zswap.
Completed-switch p99 with zstd was 33-36% lower than zram in 10/10 rounds and 20-22% lower than zswap in 9/10; these differences were smaller than the reference's variation between rounds. After writeback, readback p50 over an 832 MiB target was 11.9 us against 105.5 us for zswap and 114.0 us for zram writeback, with 70-76% fewer bytes written before readback; p99 was about the same.Less favorable:
With real swap, single-page refault p99 was 4.8 times zram with lz4 and about 1.5 times zram and zswap with zstd.
Sequential refault was 30-31% slower than zswap, since there is no swap readahead. Single-page pageout p99 was 35-38% slower than zswap. In the pageout and refault workload, 64 MiB of real pages per application, memcomp used 5-8% more memory than zram; this leaves out zram's disksize-sized table, which was larger than the difference.nakixii (16):
mm: userfaultfd: check the exclusivity of the page being moved
mm: add a swap entry type for in-memory compressed pages
mm: memcomp: compress reclaimed anonymous pages in memory
mm: userfaultfd: letUFFDIO_MOVEmove memcomp entries
mm: memcomp: charge compressed pages to the source memcg
mm: memcomp: store shmem and tmpfs pages
mm: memcomp: write cold pages back to swap
mm: memcomp: compress foreground batches asynchronously
mm: memcomp: batch unmap reserved large folios
mm: memcomp: queue restored large folios for splitting
mm: memcomp: account user eviction and refault
mm: memcomp: report precise token residency
mm: memcomp: add KUnit tests
selftests/mm: add a memcomp pageout and refault test
mm: memcomp: add the design doc
mm: memcomp: add the admin guide.../ABI/testing/sysfs-kernel-mm-memcomp | 363 + .../admin-guide/kernel-parameters.txt | 10 + Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/memcomp.rst | 149 + Documentation/mm/index.rst | 1 + Documentation/mm/memcomp.rst | 215 + MAINTAINERS | 12 + fs/proc/task_mmu.c | 20 + include/linux/memcomp.h | 481 ++ include/linux/memcontrol.h | 45 + include/linux/swap.h | 52 +-
Quoted 8 lines
include/linux/swapops.h | 22 + mm/Kconfig | 75 + mm/Makefile | 1 + mm/hmm.c | 3 + mm/huge_memory.c | 44 + mm/internal.h | 84 + mm/madvise.c | 10 +- mm/memcomp.c | 6181 +++++++++++++++++
... (a lot of memcomp/ +++ changes)
Quoted 10 lines
mm/memcontrol.c | 99 + mm/memory.c | 42 +- mm/migrate.c | 11 +- mm/mincore.c | 125 +- mm/page_io.c | 42 + mm/rmap.c | 41 +- mm/shmem.c | 251 + mm/swap.h | 4 + mm/swap_state.c | 5 + mm/swapfile.c | 49 +
Quoted 8 lines
mm/userfaultfd.c | 94 +- mm/vmscan.c | 651 +- mm/workingset.c | 32 + tools/testing/selftests/mm/Makefile | 9 +- tools/testing/selftests/mm/config | 11 + tools/testing/selftests/mm/hmm-tests.c | 112 + tools/testing/selftests/mm/memcomp.c | 831 +++ 134 files changed, 46825 insertions(+), 14 deletions(-)
This seems to big of a change and the wall of text in your "results" section does not explain clearly why would we want to bother reading this.
Signature · 2 lines
Sincerely yours, Mike.
Quoted 2 lines · Mike Rapoport
This seems to big of a change and the wall of text in your "results" section does not explain clearly why would we want to bother reading this.
Fair enough. Looking at it again, most of what the series does is already covered, or about to be, by zswap and Nhat's virtual swap work, so it doesn't justify a new subsystem of this size.
What still looks interesting to me is a couple of backend mechanisms, e.g. packing several compressed objects into one swap slot on writeback. I'll evaluate those on top of zswap/virtual swap and come back with something much smaller.
Thanks,
nakixii
Nhat Pham
reviewerQuoted 20 lines · nakixii
This is a proof of concept, posted to ask whether the approach is worth pursuing. It is not meant for merging. The series is based on Linux 6.12.23, not on mm-unstable, and has not been ported to the swap-table and zpool changes that have landed since. If there is interest, the next version will be rebased onto mm-unstable.
memcomp stores reclaimed anonymous and shmem pages in compressed form in memory, using the kernel's compressors and zsmalloc unchanged. Reclaim replaces the PTE, or the shmem page-cache entry, with a swap-like token that names a 16-byte slot. A fault on the token decompresses the page and maps it again. No block device is needed. If swap is configured, cold entries can be written back to it. The sysfs interface is under
/sys/kernel/mm/memcomp.The implementation uses 16-byte slots, a separate owner domain for zero pages, per-key storage and bounded metadata aging. Memory-cgroup accounting, cold-page writeback, asynchronous compression, precise residency reporting and mincore support are separate patches.
Quoted 8 lines
Refaults follow stock workingset accounting. The first reader claims and clears the shadow under the slot lock. Feedback runs after charging, outside the PTL and slot locks, before the page is read and installed. Consumed feedback is not restored if the read or the installation fails. An mTHP uses the shadow of its first base page and clears the shadows of all its pages. Swapoff returns payloads to the pool without building user pages.
What's the motivation of this line of work?
Quoted 1 line · Nhat Pham
What's the motivation of this line of work?
It started as an attempt to get something like the Darwin memory compressor on Linux: keep reclaimed pages compressed in RAM, fault them back directly, and push cold data to disk only when needed.
Honestly, your virtual swap series already covers the core of that (zswap without a backing swapfile, no PTE walks when moving between backends), so I don't think a separate subsystem is justified.
The parts I still think are worth looking at are restoring directly from the pool without going through the swap cache, and packing several compressed objects into one swap slot on writeback. Do you think the latter would fit as another backend type in vswap, or should it stay inside zswap writeback?
The numbers in the cover letter are from 6.12 and don't compare against vswap, so they don't say much about this yet. I'll redo them on a current tree first.
Thanks,
nakixii
Nhat Pham
reviewerQuoted 7 lines · nakixii
Quoted 1 line · Nhat Pham
What's the motivation of this line of work?
It started as an attempt to get something like the Darwin memory compressor on Linux: keep reclaimed pages compressed in RAM, fault them back directly, and push cold data to disk only when needed.
Ohhh I see.
Quoted 10 lines
Honestly, your virtual swap series already covers the core of that (zswap without a backing swapfile, no PTE walks when moving between backends), so I don't think a separate subsystem is justified.
The parts I still think are worth looking at are restoring directly from the pool without going through the swap cache, and packing several compressed objects into one swap slot on writeback. Do you think the latter would fit as another backend type in vswap, or should it stay inside zswap writeback?
Ah interesting! In the future you should include the motivation in the cover letter - either new use case your work will enable, or performance improvements, or overhead reduction etc.
And yes, compressed writeback would be a good thing to explore! I was thinking about compressed writeback sometimes ago, and I came up with a slot-packing mechanism that is very similar to what you are proposing here (IIUC). It gives you both LRU-ish ordering and good storage density. Something along this line:
1. An in-memory staging area of "buffer(s)", where we pack slots to be written-back (by LRU order).
2. Once the buffer is full, we can writeback the entire buffer.
3. Refault of any written-back buffer brings it back to the staging area. We can use the same mechanism for defrag, compaction, etc.
(I'm not 100% sure how close this is to your idea - yours might be vastly smarter. I'd need to spend time reading yours...)
But the challenge is the synchronization mechanism. Right now we rely on the swap cache + folio lock as synchronization. We can't use that here, as the whole point is we want to avoid decompression. We'd have to add a backend lock *somewhere*.
Ideally I want this to be decoupled from zswap, i.e as a vswap backend) We'd have to re-introduce a bit lock (almost similar to SWAP_HAS_CACHE) - I don't know if it's ideal (Kairui, what do you think?). Or maybe we can add a backend-specific lock, which pagefault can block on at swap_read_folio() time...
The other option is we can re-use zsmalloc's internal synchronization. We already move slots around for internal and external compaction. But that would blow up zsmalloc - you'd have to implement the stuff above inside of zsmalloc. And we might have to go for the pool_lock, which is effectively a global lock...
Quoted 4 lines
The numbers in the cover letter are from 6.12 and don't compare against vswap, so they don't say much about this yet. I'll redo them on a current tree first.
Yeah let's focus on compressed writeback. It's a concrete problem to solve :)
Hi nakixii,
Thanks for sharing this, I do like new ideas on memory compression / eviction :-D. However, this approach is wrong in many ways.
Posting a mega-sized chunk of LLM generated unreviewable code is not helpful for anyone. And it's based on a rather old kernel, considering the parts you are trying to work on have changed significantly since then. It also lacks a convincing realistic workload, e.g. almost nothing is measured under sustained memory pressure.
I think you can start by describing the problem and goal clearly instead.
Quoted 5 lines
This is a proof of concept, posted to ask whether the approach is worth pursuing. It is not meant for merging. The series is based on Linux 6.12.23, not on mm-unstable, and has not been ported to the swap-table and zpool changes that have landed since. If there is interest, the next version will be rebased onto mm-unstable.
But anyway I took a look at the series (also with help of LLM :P), IIUC, the main goal is to cut down the intermediate layers for better swap performance.
Then basing on 6.12 is a bad start, those updates you ignored already achieved that, or are focusing on and getting close: Zpool layer is removed; Swap table & cluster allocator removed the Xarray layer and multiple synchronization layers and reduced contention and overhead; we are also flattening the zsmalloc handlers, and many more are being done step by step. You can check recent git log and upstream mails for the updates on these parts.
So in mainline, the win you measured is no longer there.
I asked an AI agent to try this out yesterday and it reported this today (256M anon, all paged out, 20k random single page faults, page-cluster=0, best of 3, canary filtered, same compressor on both sides):
pageout us/page p50 p99 6.12.23 memcomp lz4 1.11 0.96 us 1.63 us 6.12.23 zram lz4 1.31 1.13 us 7.75 us 7.4-rc1 zram lz4 1.13 1.04 us 2.42 us
6.12.23 memcomp zstd 4.15 2.42 us 3.21 us 7.4-rc1 zram zstd 4.92 2.63 us 4.08 us 7.4-rc1 zswap zstd 4.89 2.58 us 4.04 us
Against a 6.12 baseline your point reproduces (7.75 -> 1.63 us). But for mainline the p50 / avg are close to identical, and the p99 is only 26-33% better instead of the 2x you measured; considering the ongoing optimizations I'm pretty sure mainline will bypass that without bloating up the LOC, or while reducing it.
And I think there is a major misunderstanding here: you didn't bypass swap, you just re-invented it. And there is no "direct memorey compression", everything goes through a page fault, unless you have some fancy hardware support like CRAM.
So TLDR: the 46K LOC for an ambiguous performance gain is really not a good idea, it is basically a poorman's reimplementation of swap with many essential parts stripped to be fast (broken OOM handling, no cgroup control, many statistics and counters missing), and still not really fast. So don't reinvent wheels.
One thing to keep though: patch 1 (UFFDIO_MOVE exclusivity) seem valid? And it has a Fixes tag, so please double check and send it on its own and maybe we can review it :-)
Anyway, still thanks for sharing this - a good reminder that everyone is chasing every nanosecond of swap performance :).
Thanks.
Hi Kairui,
Quoted 2 lines · Kairui Song
Posting a mega-sized chunk of LLM generated unreviewable code is not helpful for anyone.
Yeah, that was a mistake. Sorry for the noise.
Quoted 1 line
So in mainline, the win you measured is no longer there.
Thanks for actually testing it and posting numbers, that helps a lot. I'll drop this series.
Quoted 1 line
you didn't bypass swap, you just re-invented it.
Fair enough, that's a better way to put it.
Quoted 2 lines
One thing to keep though: patch 1 (
UFFDIO_MOVEexclusivity) seem valid?
Yes, mm-unstable still has it. I reproduced it in a VM (fork, child exits, parent write-faults one page of a 64K folio, then moving that page gets -EBUSY), and the fix makes it work. Will send it separately.
The part I still want to try is packing several compressed objects into one swap slot on zswap writeback, so writeback doesn't need to decompress. If I do, it'll be a small series on mm-unstable, with numbers under sustained pressure. Any concerns with doing that on top of the swap table?
Thanks,
nakixii