dm-pcache – persistent-memory cache for block devices
Subsystems (4)
Dongsheng Yang
maintainerv5 cover dm-pcache – persistent-memory cache for block devices
Hi Mikulas,
This is V5 for dm-pcache, please consider merging.
As you requested, I've squashed these patches into a single patch, based
on v6.16. If you find anything that needs tweaking while merging, feel
free to modify it directly. I can also send a rebased version on top of the linux-dm branch if you prefer.
This patch has gone through quite a bit of testing on my side. I can share the highlights. Once you merge this patch into the linux-dm branch, I’ll keep running ongoing tests against dm-6.18.
1. dtg-tests for pcache[1]:
- Includes dmsetup management operations, including failure/exception scenarios.
- Failslab tests (covering the NOWAIT → NOIO fallback path you mentioned).
- fail_make_request fault-injection tests.
- Backing device delay tests.
- xfstests (group generic/rw) under different parameters.
2. Striped pmem as the cache_dev[2]: ran the same dtg-tests suite with a striped pmem configuration.
3. KASAN and memleak modes[3]: all of the above tests were also executed with KASAN and memory leak detection enabled.
4. Coverage with gcov: I used gcov to gather code coverage during the
test runs[4].
- The results show 100% function coverage and 90%+ line coverage. The remaining uncovered code is mostly BUG() lines and a few hard-to-simulate error paths.
- In the coverage report you can see the code path for mempool_alloc with GFP_NOIO is exercised and covered.
[1]: https://datatravelguide.github.io/dtg-blog/pcache/pcache_v5_result_virtio/results.html
[2]: https://datatravelguide.github.io/dtg-blog/pcache/pcache_v5_result_ram/results.html
[3]: https://datatravelguide.github.io/dtg-blog/pcache/pcache_v5_result_kasan/results.html
[4]: https://datatravelguide.github.io/dtg-blog/pcache/pcache_cov_v5/dm-pcache/index.html
Changelogs
V5 from V4:
- Fix get_n_vecs() bug when the buffer isn’t page-aligned; use
- bio_add_max_vecs() instead.
- Use kvcalloc() for allocations larger than 32,768 bytes.
- For mempool allocations, try GFP_NOWAIT first; if that fails, fall back to GFP_NOIO.
V4 from V3:
- go back to use mempool for allocation of cache_key and backing_dev_req.
- introduce backing_bvec_cache and backing_dev->bvec_pool to
allocate bvecs for backing_dev_req. Currently, the obj_size of bvec_cache is max size of bvecs. The bvecs are allocated only for write‑back backing_dev_req requests, and only a limited number of writeback requests are allocated at the same time, so the resulting memory overhead should not be too severe.
If in the futer we found that this implementation results in unacceptable memory waste, we can optimize it by switching to a multi‑level slab design.
- remove ret-value checking for bio_init_clone() as there should not be REQ_INTEGRITY in bio bi_opf, so bio_init_clone() never fail.
- remove ret-value checking for backing_dev_req_alloc() and cache_key_alloc().
V3 from V2:
- rebased against linux-dm dm-6.17
- add missing include file bitfiled.h (Mikulas)
- move kmem_cache from per-device to per-module (Mikulas)
- fix a memleak in error path found by failslab testing
- retry pcache_request in defer_req() when memory allocation fail.
V2 from V1:
- introduce req_alloc() and req_init() in backing_dev.c, then we can do req_alloc() before holding spinlock and do req_init() in subtree_walk().
- introduce pre_alloc_key and pre_alloc_req in walk_ctx, that means we can pre-allocate cache_key or backing_dev_request before subtree walking.
- use mempool_alloc() with NOIO for the allocation of cache_key and backing_dev_req.
- some coding style changes from comments of Jonathan.
V1 from RFC-V2:
- use crc32c to replace crc32
- only retry pcache_req when cache full, add pcache_req into defer_list, and wait cache invalidation happen.
- new format for pcache table, it is more easily extended with new parameters later.
- remove __packed.
- use spin_lock_irq in req_complete_fn to replace spin_lock_irqsave.
- fix bug in backing_dev_bio_end with spin_lock_irqsave.
- queue_work() inside spinlock.
- introduce inline_bvecs in backing_dev_req.
- use kmalloc_array for bvecs allocation.
- calculate ->off with dm_target_offset() before use it.
Dongsheng Yang (1):
dm-pcache: add persistent cache target in device-mapper
.../admin-guide/device-mapper/dm-pcache.rst | 201 ++++ MAINTAINERS | 8 + drivers/md/Kconfig | 2 + drivers/md/Makefile | 1 + drivers/md/dm-pcache/Kconfig | 17 + drivers/md/dm-pcache/Makefile | 3 + drivers/md/dm-pcache/backing_dev.c | 374 ++++++++ drivers/md/dm-pcache/backing_dev.h | 100 ++ drivers/md/dm-pcache/cache.c | 445 +++++++++ drivers/md/dm-pcache/cache.h | 632 +++++++++++++ drivers/md/dm-pcache/cache_dev.c | 299 ++++++ drivers/md/dm-pcache/cache_dev.h | 70 ++ drivers/md/dm-pcache/cache_gc.c | 170 ++++ drivers/md/dm-pcache/cache_key.c | 888 ++++++++++++++++++ drivers/md/dm-pcache/cache_req.c | 835 ++++++++++++++++ drivers/md/dm-pcache/cache_segment.c | 293 ++++++ drivers/md/dm-pcache/cache_writeback.c | 258 +++++ drivers/md/dm-pcache/dm_pcache.c | 497 ++++++++++ drivers/md/dm-pcache/dm_pcache.h | 67 ++ drivers/md/dm-pcache/pcache_internal.h | 117 +++ drivers/md/dm-pcache/segment.c | 61 ++ drivers/md/dm-pcache/segment.h | 74 ++ 22 files changed, 5412 insertions(+) create mode 100644 Documentation/admin-guide/device-mapper/dm-pcache.rst create mode 100644 drivers/md/dm-pcache/Kconfig create mode 100644 drivers/md/dm-pcache/Makefile create mode 100644 drivers/md/dm-pcache/backing_dev.c create mode 100644 drivers/md/dm-pcache/backing_dev.h create mode 100644 drivers/md/dm-pcache/cache.c create mode 100644 drivers/md/dm-pcache/cache.h create mode 100644 drivers/md/dm-pcache/cache_dev.c create mode 100644 drivers/md/dm-pcache/cache_dev.h create mode 100644 drivers/md/dm-pcache/cache_gc.c create mode 100644 drivers/md/dm-pcache/cache_key.c create mode 100644 drivers/md/dm-pcache/cache_req.c create mode 100644 drivers/md/dm-pcache/cache_segment.c create mode 100644 drivers/md/dm-pcache/cache_writeback.c create mode 100644 drivers/md/dm-pcache/dm_pcache.c create mode 100644 drivers/md/dm-pcache/dm_pcache.h create mode 100644 drivers/md/dm-pcache/pcache_internal.h create mode 100644 drivers/md/dm-pcache/segment.c create mode 100644 drivers/md/dm-pcache/segment.h
Signature · 1 line
2.43.0
Dongsheng Yang
maintainerPATCH v5 1/1 dm-pcache: add persistent cache target in device-mapper
This patch introduces dm-pcache, a new DM target that places a DAX- capable persistent-memory device in front of any slower block device and uses it as a high-throughput, low-latency cache.
Design highlights
-----------------
- DAX data path – data is copied directly between DRAM and the pmem mapping, bypassing the block layer’s overhead.
- Segmented, crash-consistent layout
- all layout metadata are dual-replicated CRC-protected.
- atomic kset flushes; key replay on mount guarantees cache integrity even after power loss.
- Striped multi-tree index
- Multi‑tree indexing for high parallelism.
- overlap-resolution logic ensures non-intersecting cached extents.
- Background services
- write-back worker flushes dirty keys in order, preserving backing-device crash consistency. This is important for checkpoint in cloud storage.
- garbage collector reclaims clean segments when utilisation exceeds a tunable threshold.
- Data integrity – optional CRC32 on cached payload; metadata always protected.
Comparison with existing block-level caches
---------------------------------------------------------------------------------------------------------------------------------
| Feature | pcache (this patch) | bcache | dm-writecache | |----------------------------------|---------------------------------|------------------------------|---------------------------| | pmem access method | DAX | bio (block I/O) | DAX | | Write latency (4 K rand-write) | ~5 µs | ~20 µs | ~5 µs | | Concurrency | multi subtree index | global index tree | single tree + wc_lock | | IOPS (4K randwrite, 32 numjobs) | 2.1 M | 352 K | 283 K | | Read-cache support | YES | YES | NO | | Deployment | no re-format of backend | backend devices must be | no re-format of backend | | | | reformatted | | | Write-back ordering | log-structured; | no ordering guarantee | no ordering guarantee | | | preserves app-IO-order | | | | Data integrity checks | metadata + data CRC(optional) | metadata CRC only | none |
---------------------------------------------------------------------------------------------------------------------------------
--- .../admin-guide/device-mapper/dm-pcache.rst | 201 ++++ MAINTAINERS | 8 + drivers/md/Kconfig | 2 + drivers/md/Makefile | 1 + drivers/md/dm-pcache/Kconfig | 17 + drivers/md/dm-pcache/Makefile | 3 + drivers/md/dm-pcache/backing_dev.c | 374 ++++++++ drivers/md/dm-pcache/backing_dev.h | 100 ++ drivers/md/dm-pcache/cache.c | 445 +++++++++ drivers/md/dm-pcache/cache.h | 632 +++++++++++++ drivers/md/dm-pcache/cache_dev.c | 299 ++++++ drivers/md/dm-pcache/cache_dev.h | 70 ++ drivers/md/dm-pcache/cache_gc.c | 170 ++++ drivers/md/dm-pcache/cache_key.c | 888 ++++++++++++++++++ drivers/md/dm-pcache/cache_req.c | 835 ++++++++++++++++ drivers/md/dm-pcache/cache_segment.c | 293 ++++++ drivers/md/dm-pcache/cache_writeback.c | 258 +++++ drivers/md/dm-pcache/dm_pcache.c | 497 ++++++++++ drivers/md/dm-pcache/dm_pcache.h | 67 ++ drivers/md/dm-pcache/pcache_internal.h | 117 +++ drivers/md/dm-pcache/segment.c | 61 ++ drivers/md/dm-pcache/segment.h | 74 ++ 22 files changed, 5412 insertions(+) create mode 100644 Documentation/admin-guide/device-mapper/dm-pcache.rst create mode 100644 drivers/md/dm-pcache/Kconfig create mode 100644 drivers/md/dm-pcache/Makefile create mode 100644 drivers/md/dm-pcache/backing_dev.c create mode 100644 drivers/md/dm-pcache/backing_dev.h create mode 100644 drivers/md/dm-pcache/cache.c create mode 100644 drivers/md/dm-pcache/cache.h create mode 100644 drivers/md/dm-pcache/cache_dev.c create mode 100644 drivers/md/dm-pcache/cache_dev.h create mode 100644 drivers/md/dm-pcache/cache_gc.c create mode 100644 drivers/md/dm-pcache/cache_key.c create mode 100644 drivers/md/dm-pcache/cache_req.c create mode 100644 drivers/md/dm-pcache/cache_segment.c create mode 100644 drivers/md/dm-pcache/cache_writeback.c create mode 100644 drivers/md/dm-pcache/dm_pcache.c create mode 100644 drivers/md/dm-pcache/dm_pcache.h create mode 100644 drivers/md/dm-pcache/pcache_internal.h create mode 100644 drivers/md/dm-pcache/segment.c create mode 100644 drivers/md/dm-pcache/segment.h
Documentation/admin-guide/device-mapper/dm-pcache.rst new file +201 -0
@@ -0,0 +1,201 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================= +dm-pcache — Persistent Cache +================================= + +*Author: Dongsheng Yang <[email protected]>* + +This document describes *dm-pcache*, a Device-Mapper target that lets a +byte-addressable *DAX* (persistent-memory, “pmem”) region act as a +high-performance, crash-persistent cache in front of a slower block +device. The code lives in `drivers/md/dm-pcache/`. + +Quick feature summary +===================== + +* *Write-back* caching (only mode currently supported). +* *16 MiB segments* allocated on the pmem device. +* *Data CRC32* verification (optional, per cache). +* Crash-safe: every metadata structure is duplicated (`PCACHE_META_INDEX_MAX + == 2`) and protected with CRC+sequence numbers. +* *Multi-tree indexing* (one radix tree per CPU backend) for high PMem + parallelism +* Pure *DAX path* I/O – no extra BIO round-trips +* *Log-structured write-back* that preserves backend crash-consistency + +------------------------------------------------------------------------------- +Constructor +=========== + +:: + + pcache <cache_dev> <backing_dev> [<number_of_optional_arguments> <cache_mode writeback> <data_crc true|false>] + +========================= ==================================================== +``cache_dev`` Any DAX-capable block device (``/dev/pmem0``…). + All metadata *and* cached blocks are stored here. + +``backing_dev`` The slow block device to be cached. + +``cache_mode`` Optional, Only ``writeback`` is accepted at the moment. + +``data_crc`` Optional, default to ``false`` + ``true`` – store CRC32 for every cached entry and + verify on reads + ``false`` – skip CRC (faster) +========================= ==================================================== + +Example +------- + +.. code-block:: shell + + dmsetup create pcache_sdb --table \ + "0 $(blockdev --getsz /dev/sdb) pcache /dev/pmem0 /dev/sdb 4 cache_mode writeback data_crc true" + +The first time a pmem device is used, dm-pcache formats it automatically +(super-block, cache_info, etc.). + +------------------------------------------------------------------------------- +Status line +=========== + +``dmsetup status <device>`` (``STATUSTYPE_INFO``) prints: + +:: + + <sb_flags> <seg_total> <cache_segs> <segs_used> \ + <gc_percent> <cache_flags> \ + <key_head_seg>:<key_head_off> \ + <dirty_tail_seg>:<dirty_tail_off> \ + <key_tail_seg>:<key_tail_off> + +Field meanings +-------------- + +=============================== ============================================= +``sb_flags`` Super-block flags (e.g. endian marker). + +``seg_total`` Number of physical *pmem* segments. + +``cache_segs`` Number of segments used for cache. + +``segs_used`` Segments currently allocated (bitmap weight). + +``gc_percent`` Current GC high-water mark (0-90). + +``cache_flags`` Bit 0 – DATA_CRC enabled + Bit 1 – INIT_DONE (cache initialised) + Bits 2-5 – cache mode (0 == WB). + +``key_head`` Where new key-sets are being written. + +``dirty_tail`` First dirty key-set that still needs + write-back to the backing device. + +``key_tail`` First key-set that may be reclaimed by GC. +=============================== ============================================= + +------------------------------------------------------------------------------- +Messages +======== + +*Change GC trigger* + +:: + + dmsetup message <dev> 0 gc_percent <0-90> + +------------------------------------------------------------------------------- +Theory of operation +=================== + +Sub-devices +----------- + +==================== ========================================================= +backing_dev Any block device (SSD/HDD/loop/LVM, etc.). +cache_dev DAX device; must expose direct-access memory. +==================== ========================================================= + +Segments and key-sets +--------------------- + +* The pmem space is divided into *16 MiB segments*. +* Each write allocates space from a per-CPU *data_head* inside a segment. +* A *cache-key* records a logical range on the origin and where it lives + inside pmem (segment + offset + generation). +* 128 keys form a *key-set* (kset); ksets are written sequentially in pmem + and are themselves crash-safe (CRC). +* The pair *(key_tail, dirty_tail)* delimit clean/dirty and live/dead ksets. + +Write-back +---------- + +Dirty keys are queued into a tree; a background worker copies data +back to the backing_dev and advances *dirty_tail*. A FLUSH/FUA bio from the +upper layers forces an immediate metadata commit. + +Garbage collection +------------------ + +GC starts when ``segs_used >= seg_total * gc_percent / 100``. It walks +from *key_tail*, frees segments whose every key has been invalidated, and +advances *key_tail*. + +CRC verification +---------------- + +If ``data_crc is enabled`` dm-pcache computes a CRC32 over every cached data +range when it is inserted and stores it in the on-media key. Reads +validate the CRC before copying to the caller. + +------------------------------------------------------------------------------- +Failure handling +================ + +* *pmem media errors* – all metadata copies are read with + ``copy_mc_to_kernel``; an uncorrectable error logs and aborts initialisation. +* *Cache full* – if no free segment can be found, writes return ``-EBUSY``; + dm-pcache retries internally (request deferral). +* *System crash* – on attach, the driver replays ksets from *key_tail* to + rebuild the in-core trees; every segment’s generation guards against + use-after-free keys. + +------------------------------------------------------------------------------- +Limitations & TODO +================== + +* Only *write-back* mode; other modes planned. +* Only FIFO cache invalidate; other (LRU, ARC...) planned. +* Table reload is not supported currently. +* Discard planned. + +------------------------------------------------------------------------------- +Example workflow +================ + +.. code-block:: shell + + # 1. Create devices + dmsetup create pcache_sdb --table \ + "0 $(blockdev --getsz /dev/sdb) pcache /dev/pmem0 /dev/sdb 4 cache_mode writeback data_crc true" + + # 2. Put a filesystem on top + mkfs.ext4 /dev/mapper/pcache_sdb + mount /dev/mapper/pcache_sdb /mnt + + # 3. Tune GC threshold to 80 % + dmsetup message pcache_sdb 0 gc_percent 80 + + # 4. Observe status + watch -n1 'dmsetup status pcache_sdb' + + # 5. Shutdown + umount /mnt + dmsetup remove pcache_sdb + +------------------------------------------------------------------------------- +``dm-pcache`` is under active development; feedback, bug reports and patches +are very welcome!
MAINTAINERS +8 -0
@@ -6950,6 +6950,14 @@ S: Maintained F: Documentation/admin-guide/device-mapper/vdo*.rst F: drivers/md/dm-vdo/ +DEVICE-MAPPER PCACHE TARGET +M: Dongsheng Yang <[email protected]> +M: Zheng Gu <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/admin-guide/device-mapper/dm-pcache.rst +F: drivers/md/dm-pcache/ + DEVLINK M: Jiri Pirko <[email protected]> L: [email protected]
drivers/md/Kconfig +2 -0
@@ -659,4 +659,6 @@ config DM_AUDIT source "drivers/md/dm-vdo/Kconfig" +source "drivers/md/dm-pcache/Kconfig" + endif # MD
drivers/md/Makefile +1 -0
@@ -71,6 +71,7 @@ obj-$(CONFIG_DM_RAID) += dm-raid.o obj-$(CONFIG_DM_THIN_PROVISIONING) += dm-thin-pool.o obj-$(CONFIG_DM_VERITY) += dm-verity.o obj-$(CONFIG_DM_VDO) += dm-vdo/ +obj-$(CONFIG_DM_PCACHE) += dm-pcache/ obj-$(CONFIG_DM_CACHE) += dm-cache.o obj-$(CONFIG_DM_CACHE_SMQ) += dm-cache-smq.o obj-$(CONFIG_DM_EBS) += dm-ebs.o
drivers/md/dm-pcache/Kconfig new file +17 -0
@@ -0,0 +1,17 @@ +config DM_PCACHE + tristate "Persistent cache for Block Device (Experimental)" + depends on BLK_DEV_DM + depends on DEV_DAX + help + PCACHE provides a mechanism to use persistent memory (e.g., CXL persistent memory, + DAX-enabled devices) as a high-performance cache layer in front of + traditional block devices such as SSDs or HDDs. + + PCACHE is implemented as a kernel module that integrates with the block + layer and supports direct access (DAX) to persistent memory for low-latency, + byte-addressable caching. + + Note: This feature is experimental and should be tested thoroughly + before use in production environments. + + If unsure, say 'N'.
drivers/md/dm-pcache/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +dm-pcache-y := dm_pcache.o cache_dev.o segment.o backing_dev.o cache.o cache_gc.o cache_writeback.o cache_segment.o cache_key.o cache_req.o + +obj-m += dm-pcache.o
drivers/md/dm-pcache/backing_dev.c new file +374 -0
374 lines not on this page · raw message
drivers/md/dm-pcache/backing_dev.h new file +100 -0
100 lines not on this page · raw message
drivers/md/dm-pcache/cache.c new file +445 -0
445 lines not on this page · raw message
drivers/md/dm-pcache/cache.h new file +632 -0
632 lines not on this page · raw message
drivers/md/dm-pcache/cache_dev.c new file +299 -0
299 lines not on this page · raw message
drivers/md/dm-pcache/cache_dev.h new file +70 -0
70 lines not on this page · raw message
drivers/md/dm-pcache/cache_gc.c new file +170 -0
170 lines not on this page · raw message
drivers/md/dm-pcache/cache_key.c new file +888 -0
662 lines not on this page · raw message
drivers/md/dm-pcache/cache_req.c new file +835 -0
835 lines not on this page · raw message
drivers/md/dm-pcache/cache_segment.c new file +293 -0
293 lines not on this page · raw message
drivers/md/dm-pcache/cache_writeback.c new file +258 -0
258 lines not on this page · raw message
drivers/md/dm-pcache/dm_pcache.c new file +497 -0
497 lines not on this page · raw message
drivers/md/dm-pcache/dm_pcache.h new file +67 -0
67 lines not on this page · raw message
drivers/md/dm-pcache/pcache_internal.h new file +117 -0
117 lines not on this page · raw message
drivers/md/dm-pcache/segment.c new file +61 -0
61 lines not on this page · raw message
drivers/md/dm-pcache/segment.h new file +74 -0
74 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.43.0
Part of the diff is not shown here. View the raw message.
Quoted 2 lines · Dongsheng Yang
Hi Mikulas,
This is V5 for dm-pcache, please consider merging.
Hi
I merged it to the device mapper git.
(there was a problem with the include file linux/pfn_t.h and the type pfn_t not being present on Linux 6.17-rc1, but I fixed it)
Mikulas
Dongsheng Yang
maintainerQuoted 11 lines · Mikulas Patocka
Quoted 2 lines · Dongsheng Yang
Hi Mikulas,
This is V5 for dm-pcache, please consider merging.Hi
I merged it to the device mapper git.
(there was a problem with the include file
linux/pfn_t.hand the typepfn_tnot being present on Linux 6.17-rc1, but I fixed it)
Thanx for the fix.
Dongsheng
Quoted 3 lines
Mikulas