Introduce CBD (CXL Block Device)
Subsystems (2)
v3 cover Introduce CBD (CXL Block Device)
Hi Jens,
Please help to take a look at this patchset. This is V3 for CBD (CXL Block Device).
CBD supports both single-host and multi-host scenarios, and allows the use of pmem devices as block device cache, providing low latency and high-concurrency performance.
(1) Better latency:
1 iodepth, 1 numjobs, randwrite bs=4K: cbd 7.72us vs bcache
25.30us, about 300% improvement
(2) Better iops:
1 iodepth, 32 numjobs, randwrite bs=4K: cbd 1,400K IOPS vs
bcache 210K IOPS, about 600% improvement
(3) Better stdev:
1 iodepth, 1 numjobs, randwrite bs=4K: cbd stdev=36.45 vs bcache
stdev=937.81, about 30 times improvement.
V3 of the code can be found at: https://github.com/DataTravelGuide/linux.git tag cbd-v3
Changelog from V2:
- Refactored the cbd_cache.c and cbd_internal.h files by splitting them into multiple files, making the structure clearer and increasing readability.
- Added CRC verification for all data and metadata. This means that if all CRC verification options are enabled in Kconfig, all information written to pmem, including data and metadata, will have CRC checks.
- Fixed some minor bugs discovered during long-term runs of xfstests.
- Added the cbd-utils (https://github.com/DataTravelGuide/cbd-utils) project to user-space tools, providing the cbdctrl command for cbd-related management operations.
You can create a cbd using the following commands:
# cbdctrl tp-reg --path /dev/pmem0 --host node0 --format --force
# cbdctrl backend-start --path /dev/sda --start-dev --cache-size 1G
/dev/cbd0
# cbdctrl backend-list
[
{
"backend_id": 0,
"host_id": 0,
"backend_path": "/dev/sda",
"alive": true,
"cache_segs": 64,
"cache_gc_percent": 70,
"cache_used_segs": 1,
"blkdevs": [
{
"blkdev_id": 0,
"host_id": 0,
"backend_id": 0,
"dev_name": "/dev/cbd0",
"alive": true
}
]
}
]Additional information about CBD cache:
(1) What is CBD Cache
cbd cache is a *lightweight* solution that uses persistent memory as block
device cache. It works similar with bcache, where bcache uses block devices as cache device, but cbd cache only supports persistent memory devices for caching. It accesses the cache device through DAX and is designed with features specifically for persistent memory scenarios, such as multi-cache tree structures and sync insertion of cached data.
+-----------------------------------------------------------------+ | single-host | +-----------------------------------------------------------------+ | | | | | | | | | | | +-----------+ +------------+ | | | /dev/cbd0 | | /dev/cbd1 | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | | | | | | | | | | /dev/pmem0 | cbd0 cache| | cbd1 cache | | | | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | |+---------+| |+----------+| | | ||/dev/sda || || /dev/sdb || | | |+---------+| |+----------+| | | +-----------+ +------------+ | +-----------------------------------------------------------------+
Note: cbd cache is not intended to replace your bcache. Instead, it offers an alternative solution specifically suited for scenarios where you want to use persistent memory devices as block device cache.
Another caching technique for accessing persistent memory using DAX is dm-writeback, but it is designed for scenarios based on device-mapper. On the other hand, cbd cache and bcache are caching solutions for block device scenarios. Therefore, I did not do a comparative analysis between cbd cache and dm-writeback.
(2) light software overhead cache write (low latency)
For cache write, handling a write request typically involves the following steps: (1) Allocating cache space -> (2) Writing data to the cache -> (3) Recording cache index metadata -> (4) Returning the result.
In cache modules using block devices as the cache (e.g., bcache), the steps of (2) writing data to the cache and (3) recording cache index metadata are asynchronous.
During step (2), submit_bio is issued to the cache block device, and after the bi_end_io callback completes, a new process continues with step (3). This incurs significant overhead for persistent memory cache.
However, cbd cache, which is designed for persistent memory, does not require asynchronous operations. It can directly proceed with steps (3) and (4) after completing the memcpy through DAX.
This makes a significant difference for small IO. In the case of 4K random writes, cbd cache achieves a latency of only 7.72us (compared to 25.30us for bcache in the same test, offering a 300% improvement).
Further comparative results for various scenarios are shown in the table below.
+------------+-------------------------+--------------------------+ | numjobs=1 | randwrite | randread | | iodepth=1 +------------+------------+-------------+------------+ | (latency) | cbd cache | bcache | cbd cache | bcache | +------------+------------+------------+-------------+------------+ | bs=512 | 6.10us | 23.08us | 4.82us | 5.57us | +------------+------------+------------+-------------+------------+ | bs=1K | 6.35us | 21.68us | 5.38us | 6.05us | +------------+------------+------------+-------------+------------+ | bs=4K | 7.72us | 25.30us | 6.06us | 6.00us | +------------+------------+------------+-------------+------------+ | bs=8K | 8.92us | 27.73us | 7.24us | 7.35us | +------------+------------+------------+-------------+------------+ | bs=16K | 12.25us | 34.04us | 9.06us | 9.10us | +------------+------------+------------+-------------+------------+ | bs=32K | 16.77us | 49.24us | 14.10us | 16.18us | +------------+------------+------------+-------------+------------+ | bs=64K | 30.52us | 63.72us | 30.69us | 30.38us | +------------+------------+------------+-------------+------------+ | bs=128K | 51.66us | 114.69us | 38.47us | 39.10us | +------------+------------+------------+-------------+------------+ | bs=256K | 110.16us | 204.41us | 79.64us | 99.98us | +------------+------------+------------+-------------+------------+ | bs=512K | 205.52us | 398.70us | 122.15us | 131.97us | +------------+------------+------------+-------------+------------+ | bs=1M | 493.57us | 623.31us | 233.48us | 246.56us | +------------+------------+------------+-------------+------------+
(3) multi-queue and multi cache tree (high iops)
For persistent memory, the hardware concurrency is very high. If an indexing tree is used to manage space indexing, the indexing will become a bottleneck for concurrency.
cbd cache independently manages its own indexing tree for each backend. Meanwhile, the indexing tree for the cache corresponding to each backend is divided into multiple RB trees based on the logical address space. All IO operations will find the corresponding indexing tree based on their offset. This design increases concurrency while ensuring that the depth of the indexing tree does not become too large.
From testing, in a scenario with 32 numjobs, cbd cache achieved nearly 1,400K IOPS for 4K random write (under the same test scenario, the IOPS of bcache was around 210K, meaning CBD Cache provided an improvement of over 600%).
More detailed comparison results are as follows:
+------------+-------------------------+--------------------------+ | bs=4K | randwrite | randread | | iodepth=1 +------------+------------+-------------+------------+ | (iops) | cbd cache | bcache | cbd cache | bcache | +------------+------------+------------+-------------+------------+ | numjobs=1 | 93652 | 38055 | 154101 | 142292 | +------------+------------+------------+-------------+------------+ | numjobs=2 | 205255 | 79322 | 317143 | 221957 | +------------+------------+------------+-------------+------------+ | numjobs=4 | 430588 | 124439 | 635760 | 513443 | +------------+------------+------------+-------------+------------+ | numjobs=8 | 852865 | 160980 | 1226714 | 505911 | +------------+------------+------------+-------------+------------+ | numjobs=16| 1140952 | 226094 | 2058178 | 996146 | +------------+------------+------------+-------------+------------+ | numjobs=32| 1418989 | 214447 | 2892710 | 1361308 | +------------+------------+------------+-------------+------------+
(4) better performance stablility (less stdev)
CBD Cache, through a streamlined design, simplifies and makes the IO process more controllable, which allows for stable performance output.
For example, in CBD Cache, the writeback does not need to walk through the indexing tree, meaning that the writeback process will not suffer from increased IO latency due to conflict in the indexing tree.
From testing, under random write, CBD Cache achieves an average latency of 6.80us, with a max latency of 2794us and a latency standard deviation of 36.45 (under the same test, Bcache has an average latency of 24.28us, but a max latency of 474,622us and a standard deviation as high as 937.81. This means that in terms of standard deviation, CBD Cache achieved approximately 30 times the improvement).
Bcache:
=================================================
write: IOPS=39.1k, BW=153MiB/s (160MB/s)(5120MiB/33479msec); 0 zone resets
slat (usec): min=4, max=157364, avg=12.47, stdev=138.93
clat (nsec): min=1168, max=474615k, avg=11808.80, stdev=927287.74
lat (usec): min=11, max=474622, avg=24.28, stdev=937.81
clat percentiles (nsec):
| 1.00th=[ 1256], 5.00th=[ 1304], 10.00th=[ 1320],
| 20.00th=[ 1400], 30.00th=[ 1448], 40.00th=[ 1672],
| 50.00th=[ 8640], 60.00th=[ 9152], 70.00th=[ 9664],
| 80.00th=[ 10048], 90.00th=[ 11328], 95.00th=[ 19072],
| 99.00th=[ 27776], 99.50th=[ 36608], 99.90th=[ 173056],
| 99.95th=[ 856064], 99.99th=[2039808]
bw ( KiB/s): min=28032, max=214664, per=99.69%, avg=156122.03, stdev=51649.87, samples=66
iops : min= 7008, max=53666, avg=39030.53, stdev=12912.50, samples=66
lat (usec) : 2=41.55%, 4=4.59%, 10=32.70%, 20=16.37%, 50=4.45%
lat (usec) : 100=0.10%, 250=0.17%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=11.93%, sys=38.61%, ctx=1311384, majf=0, minf=382
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1310718,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):
WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=5120MiB (5369MB), run=33479-33479msec
Disk stats (read/write):
bcache0: ios=0/1305444, sectors=0/10443552, merge=0/0,
ticks=0/21789, in_queue=21789, util=65.13%, aggrios=0/0, aggsectors=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
ram0: ios=0/0, sectors=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
pmem0: ios=0/0, sectors=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
CBD cache:
==============================================
write: IOPS=133k, BW=520MiB/s (545MB/s)(5120MiB/9848msec); 0 zone
resets
slat (usec): min=3, max=2786, avg= 5.84, stdev=36.41
clat (nsec): min=852, max=132404, avg=959.09, stdev=436.60
lat (usec): min=4, max=2794, avg= 6.80, stdev=36.45
clat percentiles (nsec):
| 1.00th=[ 884], 5.00th=[ 900], 10.00th=[ 908], 20.00th=[916],
| 30.00th=[ 924], 40.00th=[ 924], 50.00th=[ 932], 60.00th=[940],
| 70.00th=[ 948], 80.00th=[ 964], 90.00th=[ 1004], 95.00th=[1064],
| 99.00th=[ 1192], 99.50th=[ 1432], 99.90th=[ 6688], 99.95th=[7712],
| 99.99th=[12480]
bw ( KiB/s): min=487088, max=552928, per=99.96%, avg=532154.95, stdev=18228.92, samples=19
iops : min=121772, max=138232, avg=133038.84, stdev=4557.32, samples=19
lat (nsec) : 1000=89.09%
lat (usec) : 2=10.76%, 4=0.03%, 10=0.09%, 20=0.03%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=23.93%, sys=76.03%, ctx=61, majf=0, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):
WRITE: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s),
io=5120MiB (5369MB), run=9848-9848msec
Disk stats (read/write):
cbd0: ios=0/1280334, sectors=0/10242672, merge=0/0, ticks=0/0, in_queue=0, util=43.07%
(5) no need of formating for your existing disk
As a lightweight block storage caching technology, cbd cache does not require storing metadata on backend disk. This allows users to easily add caching to existing disks without the need for any formatting operations and data migration. They can also easily stop using the cbd cache without complications, The backend disk can be used independently as a raw disk.
(6) backend device is crash-consistency
The writeback mechanism of cbd cache strictly follows a log-structured approach when writeback data. Even if dirty cache data is overwritten by new data (e.g., the old data from 0-4K is A, and new data overwrites 0-4K with B), the old data A is writeback first, followed by writeback the new data B to overwrite on the backend disk. This ensures that the backend disk maintains crash consistency. In the event of a failure of the pmem device, the data on the backend disk remains usable, though crash consistency is maintained while losing the data in the cache. This feature is particularly useful in cloud storage for disaster recovery scenarios.
It is important to note that this approach may lead to cache space utilization issues if there are many overwrite operations. However, modern file systems, such as Btrfs and F2FS, take wear leveling of the disk into account, so they tend to avoid writing repeatedly to the same area. This means that there will not be a large number of overwrite writes for the disk. Additionally, modern databases, especially those using LSM engines, rarely perform overwrite operations.
Additionally, there is an entry on the TODO list to provide a parameter backend_consistency=false to allow users to achieve better cache space utilization. That depends on how urgent the requirment is.
(7) cache space for each disk is configurable
For each backend, when enabling caching, we can specify cache space size for this backend. This is different from bcache, where all backing devices can dynamically share the cache space within a single cache device. This improves cache utilization by achieving optimal utilization through time-sharing. However, this can lead to an issue where cache behavior becomes unpredictable. In enterprise applications, it's important to have a more precise understanding of the performance of each disk. When multiple disks dynamically share the cache, the exact amount of cache each disk receives becomes uncertain. cbd cache assigns a dedicated cache space for each disk, ensuring that the cache is exclusive and not affected by others, making the cache behavior more predictable.
(8) After all, all the performance test results mentioned
above were executed using the memmap=20G!4G option to simulate the /dev/pmem0 device.
Additionally, the cbd code runs the cbd-tests daily, including the xfstests suite, it passes xfstests test suite. (cbd-tests: https://github.com/DataTravelGuide/cbd-tests)
Thanx
Dongsheng Yang (8):
cbd: introduce cbd_transport
cbd: introduce cbd_host
cbd: introduce cbd_segment
cbd: introduce cbd_channel
cbd: introduce cbd_blkdev
cbd: introduce cbd_backend
cbd: introduce cbd_cache
block: Init for CBD(CXL Block Device)
MAINTAINERS | 7 + drivers/block/Kconfig | 2 + drivers/block/Makefile | 2 + drivers/block/cbd/Kconfig | 89 ++ drivers/block/cbd/Makefile | 14 + drivers/block/cbd/cbd_backend.c | 730 ++++++++++ drivers/block/cbd/cbd_backend.h | 137 ++ drivers/block/cbd/cbd_blkdev.c | 551 ++++++++ drivers/block/cbd/cbd_blkdev.h | 92 ++ drivers/block/cbd/cbd_cache/cbd_cache.c | 489 +++++++ drivers/block/cbd/cbd_cache/cbd_cache.h | 157 +++ drivers/block/cbd/cbd_cache/cbd_cache_gc.c | 167 +++ .../block/cbd/cbd_cache/cbd_cache_internal.h | 536 ++++++++ drivers/block/cbd/cbd_cache/cbd_cache_key.c | 881 ++++++++++++ drivers/block/cbd/cbd_cache/cbd_cache_req.c | 921 +++++++++++++ .../block/cbd/cbd_cache/cbd_cache_segment.c | 268 ++++ .../block/cbd/cbd_cache/cbd_cache_writeback.c | 197 +++ drivers/block/cbd/cbd_channel.c | 144 ++ drivers/block/cbd/cbd_channel.h | 429 ++++++ drivers/block/cbd/cbd_handler.c | 468 +++++++ drivers/block/cbd/cbd_handler.h | 66 + drivers/block/cbd/cbd_host.c | 227 ++++ drivers/block/cbd/cbd_host.h | 67 + drivers/block/cbd/cbd_internal.h | 482 +++++++ drivers/block/cbd/cbd_main.c | 230 ++++ drivers/block/cbd/cbd_queue.c | 516 +++++++ drivers/block/cbd/cbd_queue.h | 288 ++++ drivers/block/cbd/cbd_segment.c | 311 +++++ drivers/block/cbd/cbd_segment.h | 104 ++ drivers/block/cbd/cbd_transport.c | 1186 +++++++++++++++++ drivers/block/cbd/cbd_transport.h | 169 +++ 31 files changed, 9927 insertions(+) create mode 100644 drivers/block/cbd/Kconfig create mode 100644 drivers/block/cbd/Makefile create mode 100644 drivers/block/cbd/cbd_backend.c create mode 100644 drivers/block/cbd/cbd_backend.h create mode 100644 drivers/block/cbd/cbd_blkdev.c create mode 100644 drivers/block/cbd/cbd_blkdev.h create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache.h create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_gc.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_internal.h create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_key.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_req.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_segment.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_writeback.c create mode 100644 drivers/block/cbd/cbd_channel.c create mode 100644 drivers/block/cbd/cbd_channel.h create mode 100644 drivers/block/cbd/cbd_handler.c create mode 100644 drivers/block/cbd/cbd_handler.h create mode 100644 drivers/block/cbd/cbd_host.c create mode 100644 drivers/block/cbd/cbd_host.h create mode 100644 drivers/block/cbd/cbd_internal.h create mode 100644 drivers/block/cbd/cbd_main.c create mode 100644 drivers/block/cbd/cbd_queue.c create mode 100644 drivers/block/cbd/cbd_queue.h create mode 100644 drivers/block/cbd/cbd_segment.c create mode 100644 drivers/block/cbd/cbd_segment.h create mode 100644 drivers/block/cbd/cbd_transport.c create mode 100644 drivers/block/cbd/cbd_transport.h
Signature · 1 line
2.34.1
PATCH v3 1/8 cbd: introduce cbd_transport
cbd_transport represents the layout of the entire shared memory, as shown below.
+-------------------------------------------------------------------------------------------------------------------------------+ | cbd transport | +--------------------+-----------------------+-----------------------+----------------------+-----------------------------------+ | | hosts | backends | blkdevs | segments | | cbd transport info +----+----+----+--------+----+----+----+--------+----+----+----+-------+-------+-------+-------+-----------+ | | | | | ... | | | | ... | | | | ... | | | | ... | +--------------------+----+----+----+--------+----+----+----+--------+----+----+----+-------+---+---+---+---+-------+-----------+ | | | | | | | | +-------------------------------------------------------------------------------------+ | | | | | v | +-----------------------------------------------------------+ | | channel segment | | +--------------------+--------------------------------------+ | | channel meta | channel data | | +---------+----------+--------------------------------------+ | | | | | | | v | +----------------------------------------------------------+ | | channel meta | | +-----------+--------------+-------------------------------+ | | meta ctrl | comp ring | cmd ring | | +-----------+--------------+-------------------------------+ | | | | +--------------------------------------------------------------------------------------------+ | | | v +----------------------------------------------------------+ | cache segment | +-----------+----------------------------------------------+ | info | data | +-----------+----------------------------------------------+
The shared memory is divided into five regions:
a) Transport_info:
Information about the overall transport, including the layout
of the transport.
b) Hosts:
Each host wishing to utilize this transport needs to register
its own information within a host entry in this region.
c) Backends:
Starting a backend on a host requires filling in information
in a backend entry within this region.
d) Blkdevs:
Once a backend is established, it can be mapped to CBD device
on any associated host. The information about the blkdevs is then filled into the blkdevs region.
e) Segments:
This is the actual data communication area, where communication
between blkdev and backend occurs. Each queue of a block device uses a channel, and each backend has a corresponding handler interacting with this queue.
f) Channel segment:
Channel is one type of segment, is further divided into meta and
data regions.
The meta region includes subm rings and comp rings.
The blkdev converts upper-layer requests into cbd_se and fills them into the subm ring. The handler accepts the cbd_se from the subm ring and sends them to the local actual block device of the backend (e.g., sda). After completion, the results are formed into cbd_ce and filled into the comp ring. The blkdev then receives the cbd_ce and returns the results to the upper-layer IO sender.
g) Cache segment:
Cache segment is another type of segment, when cache enabled
for a backend, transport will allocate cache segments to this backend.
--- drivers/block/cbd/cbd_internal.h | 482 ++++++++++++ drivers/block/cbd/cbd_transport.c | 1186 +++++++++++++++++++++++++++++ drivers/block/cbd/cbd_transport.h | 169 ++++ 3 files changed, 1837 insertions(+) create mode 100644 drivers/block/cbd/cbd_internal.h create mode 100644 drivers/block/cbd/cbd_transport.c create mode 100644 drivers/block/cbd/cbd_transport.h
drivers/block/cbd/cbd_internal.h new file +482 -0
@@ -0,0 +1,482 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _CBD_INTERNAL_H +#define _CBD_INTERNAL_H + +#include <linux/delay.h> +#include <linux/crc32.h> + +/* +* CBD (CXL Block Device) provides two usage scenarios: single-host and multi-hosts. +* +* (1) Single-host scenario, CBD can use a pmem device as a cache for block devices, +* providing a caching mechanism specifically designed for persistent memory. +* +* +-----------------------------------------------------------------+ +* | single-host | +* +-----------------------------------------------------------------+ +* | | +* | | +* | | +* | | +* | | +* | +-----------+ +------------+ | +* | | /dev/cbd0 | | /dev/cbd1 | | +* | | | | | | +* | +---------------------|-----------|-----|------------|-------+ | +* | | | | | | | | +* | | /dev/pmem0 | cbd0 cache| | cbd1 cache | | | +* | | | | | | | | +* | +---------------------|-----------|-----|------------|-------+ | +* | |+---------+| |+----------+| | +* | ||/dev/sda || || /dev/sdb || | +* | |+---------+| |+----------+| | +* | +-----------+ +------------+ | +* +-----------------------------------------------------------------+ +* +* (2) Multi-hosts scenario, CBD also provides a cache while taking advantage of +* shared memory features, allowing users to access block devices on other nodes across +* different hosts. +* +* As shared memory is supported in CXL3.0 spec, we can transfer data via CXL shared memory. +* CBD use CXL shared memory to transfer data between node-1 and node-2. +* +* +--------------------------------------------------------------------------------------------------------+ +* | multi-hosts | +* +--------------------------------------------------------------------------------------------------------+ +* | | +* | | +* | +-------------------------------+ +------------------------------------+ | +* | | node-1 | | node-2 | | +* | +-------------------------------+ +------------------------------------+ | +* | | | | | | +* | | +-------+ +---------+ | | +* | | | cbd0 | | backend0+------------------+ | | +* | | +-------+ +---------+ | | | +* | | | pmem0 | | pmem0 | v | | +* | | +-------+-------+ +---------+----+ +---------------+ | +* | | | cxl driver | | cxl driver | | /dev/sda | | +* | +---------------+--------+------+ +-----+--------+-----+---------------+ | +* | | | | +* | | | | +* | | CXL CXL | | +* | +----------------+ +-----------+ | +* | | | | +* | | | | +* | | | | +* | +-------------------------+---------------+--------------------------+ | +* | | +---------------+ | | +* | | shared memory device | cbd0 cache | | | +* | | +---------------+ | | +* | +--------------------------------------------------------------------+ | +* | | +* +--------------------------------------------------------------------------------------------------------+ +*/ +#define cbd_err (fmt, ...) \ + pr_err("cbd: %s:%u " fmt, __func__, __LINE__, ##__VA_ARGS__) +#define cbd_info (fmt, ...) \ + pr_info("cbd: %s:%u " fmt, __func__, __LINE__, ##__VA_ARGS__) +#define cbd_debug (fmt, ...) \ + pr_debug("cbd: %s:%u " fmt, __func__, __LINE__, ##__VA_ARGS__) + +#define CBD_KB (1024)/* 1 Kilobyte in bytes */ +#define CBD_MB (CBD_KB * CBD_KB)/* 1 Megabyte in bytes */ + +#define CBD_TRANSPORT_MAX 1024/* Maximum number of transport instances */ +#define CBD_PATH_LEN 128 +#define CBD_NAME_LEN 32 + +#define CBD_QUEUES_MAX 128/* Maximum number of I/O queues */ +#define CBD_HANDLERS_MAX 128/* Maximum number of handlers */ + +#define CBD_PART_SHIFT 4/* Bit shift for partition identifier */ +#define CBD_DRV_NAME "cbd"/* Default driver name for CBD */ +#define CBD_DEV_NAME_LEN 32/* Maximum device name length */ + +#define CBD_HB_INTERVAL msecs_to_jiffies(5000)/* Heartbeat interval in jiffies (5 seconds) */ +#define CBD_HB_TIMEOUT (30 * 1000)/* Heartbeat timeout in milliseconds (30 seconds) */ + +/* +* CBD transport layout: +* +* +-------------------------------------------------------------------------------------------------------------------------------+ +* | cbd transport | +* +--------------------+-----------------------+-----------------------+----------------------+-----------------------------------+ +* | | hosts | backends | blkdevs | segments | +* | cbd transport info +----+----+----+--------+----+----+----+--------+----+----+----+-------+-------+-------+-------+-----------+ +* | | | | | ... | | | | ... | | | | ... | | | | ... | +* +--------------------+----+----+----+--------+----+----+----+--------+----+----+----+-------+---+---+---+---+-------+-----------+ +* | | +* | | +* | | +* | | +* +-------------------------------------------------------------------------------------+ | +* | | +* | | +* v | +* +-----------------------------------------------------------+ | +* | channel segment | | +* +--------------------+--------------------------------------+ | +* | channel meta | channel data | | +* +---------+----------+--------------------------------------+ | +* | | +* | | +* | | +* v | +* +----------------------------------------------------------+ | +* | channel meta | | +* +-----------+--------------+-------------------------------+ | +* | meta ctrl | comp ring | cmd ring | | +* +-----------+--------------+-------------------------------+ | +* | +* | +* | +* +--------------------------------------------------------------------------------------------+ +* | +* | +* | +* v +* +----------------------------------------------------------+ +* | cache segment | +* +-----------+----------------------------------------------+ +* | info | data | +* +-----------+----------------------------------------------+ +*/ + +/* cbd segment */ +#define CBDT_SEG_SIZE (16 * 1024 * 1024)/* Size of each CBD segment (16 MB) */ + +/* cbd transport */ +#define CBD_TRANSPORT_MAGIC 0x65B05EFA96C596EFULL/* Unique identifier for CBD transport layer */ +#define CBD_TRANSPORT_VERSION 1/* Version number for CBD transport layer */ + +/* Maximum number of metadata indices */ +#define CBDT_META_INDEX_MAX 2 + +/* +* CBD structure diagram: +* +* +--------------+ +* | cbd_transport| +----------+ +* +--------------+ | cbd_host | +* | | +----------+ +* | host +---------------------------------------------->| | +* +--------------------+ backends | | hostname | +* | | devices +------------------------------------------+ | | +* | | | | +----------+ +* | +--------------+ | +* | | +* | | +* | | +* | | +* | | +* v v +* +------------+ +-----------+ +------+ +-----------+ +-----------+ +------+ +* | cbd_backend+---->|cbd_backend+---->| NULL | | cbd_blkdev+----->| cbd_blkdev+---->| NULL | +* +------------+ +-----------+ +------+ +-----------+ +-----------+ +------+ +* +------------+ cbd_cache | | handlers | +------+ queues | | queues | +* | | | +-----------+ | | | +-----------+ +* | +------+ handlers | | | | +* | | +------------+ | | cbd_cache +-------------------------------------+ +* | | | +-----------+ | +* | | | | +* | | +-------------+ +-------------+ +------+ | +-----------+ +-----------+ +------+ | +* | +----->| cbd_handler +------>| cbd_handler +---------->| NULL | +----->| cbd_queue +----->| cbd_queue +---->| NULL | | +* | +-------------+ +-------------+ +------+ +-----------+ +-----------+ +------+ | +* | +------+ channel | | channel | +------+ channel | | channel | | +* | | +-------------+ +-------------+ | +-----------+ +-----------+ | +* | | | | +* | | | | +* | | | | +* | | v | +* | | +-----------------------+ | +* | +------------------------------------------------------->| cbd_channel | | +* | +-----------------------+ | +* | | channel_id | | +* | | cmdr (cmd ring) | | +* | | compr (complete ring) | | +* | | data (data area) | | +* | | | | +* | +-----------------------+ | +* | | +* | +-----------------------------+ | +* +------------------------------------------------>| cbd_cache |<------------------------------------------------------------+ +* +-----------------------------+ +* | cache_wq | +* | cache_tree | +* | segments[] | +* +-----------------------------+ +*/ + +#define CBD_DEVICE (OBJ ) \ +struct cbd_## OBJ ##_device { \ + struct device dev; \ + struct cbd_transport *cbdt; \ + u32 id; \ +}; \ + \ +struct cbd_## OBJ ##s_device { \ + struct device OBJ ##s_dev; \ + struct cbd_## OBJ ##_device OBJ ##_devs[]; \ +} + +/* cbd_worker_cfg - Structure to manage retry configurations for a worker */ +struct cbd_worker_cfg { +u32 busy_retry_cur; +u32 busy_retry_count; +u32 busy_retry_max; +u32 busy_retry_min; +u64 busy_retry_interval; +}; + +static inline void cbdwc_init (struct cbd_worker_cfg *cfg) +{ + cfg->busy_retry_cur =0 ; + cfg->busy_retry_count =100 ; + cfg->busy_retry_max = cfg->busy_retry_count *2 ; + cfg->busy_retry_min =0 ; + cfg->busy_retry_interval =1 ;/* 1 microsecond */ +} + +/** +* cbdwc_hit - Reset retry counter and increase busy_retry_count on success. +* @cfg: Pointer to the cbd_worker_cfg structure to update. +* +* Increases busy_retry_count by 1/16 of its current value, +* unless it's already at the maximum. +*/ +static inline void cbdwc_hit (struct cbd_worker_cfg *cfg) +{ +u32 delta; + + cfg->busy_retry_cur =0 ; + +if (cfg->busy_retry_count == cfg->busy_retry_max) +return ; + + delta = cfg->busy_retry_count >>4 ; +if (!delta) + delta = (cfg->busy_retry_max + cfg->busy_retry_min) >>1 ; + + cfg->busy_retry_count += delta; + +if (cfg->busy_retry_count > cfg->busy_retry_max) + cfg->busy_retry_count = cfg->busy_retry_max; +} + +/** +* cbdwc_miss - Reset retry counter and decrease busy_retry_count on failure. +* @cfg: Pointer to the cbd_worker_cfg structure to update. +* +* Decreases busy_retry_count by 1/16 of its current value, +* unless it's already at the minimum. +*/ +static inline void cbdwc_miss (struct cbd_worker_cfg *cfg) +{ +u32 delta; + + cfg->busy_retry_cur =0 ; + +if (cfg->busy_retry_count == cfg->busy_retry_min) +return ; + + delta = cfg->busy_retry_count >>4 ; +if (!delta) + delta = cfg->busy_retry_count; + + cfg->busy_retry_count -= delta; +} + +/** +* cbdwc_need_retry - Determine if another retry attempt should be made. +* @cfg: Pointer to the cbd_worker_cfg structure to check. +* +* Increments busy_retry_cur and compares it to busy_retry_count. +* If retry is needed, yields CPU and waits for busy_retry_interval. +* +* Return: true if retry is allowed, false if retry limit reached. +*/ +static inline bool cbdwc_need_retry (struct cbd_worker_cfg *cfg) +{ +if (++cfg->busy_retry_cur < cfg->busy_retry_count) { +cpu_relax (); +fsleep (cfg->busy_retry_interval); +return true; + } +return false; +} + +/* +* struct cbd_meta_header - CBD metadata header structure +* @crc: CRC checksum for validating metadata integrity. +* @seq: Sequence number to track metadata updates. +* @version: Metadata version. +* @res: Reserved space for future use. +*/ +struct cbd_meta_header { +u32 crc; +u8 seq; +u8 version; +u16 res; +}; + +/* +* cbd_meta_crc - Calculate CRC for the given metadata header. +* @header: Pointer to the metadata header. +* @meta_size: Size of the metadata structure. +* +* Returns the CRC checksum calculated by excluding the CRC field itself. +*/ +static inline u32 cbd_meta_crc (struct cbd_meta_header *header,u32 meta_size) +{ +return crc32 (0 , (void *)header +4 , meta_size -4 );/* CRC calculated starting after the crc field */ +} + +/* +* cbd_meta_seq_after - Check if a sequence number is more recent, accounting for overflow. +* @seq1: First sequence number. +* @seq2: Second sequence number. +* +* Determines if @seq1 is more recent than @seq2 by calculating the signed +* difference between them. This approach allows handling sequence number +* overflow correctly because the difference wraps naturally, and any value +* greater than zero indicates that @seq1 is "after" @seq2. This method +* assumes 8-bit unsigned sequence numbers, where the difference wraps +* around if seq1 overflows past seq2. +* +* Returns: +* - true if @seq1 is more recent than @seq2, indicating it comes "after" +* - false otherwise. +*/ +static inline bool cbd_meta_seq_after (u8 seq1,u8 seq2) +{ +return (s8)(seq1 - seq2) >0 ; +} + +/* +* cbd_meta_find_latest - Find the latest valid metadata. +* @header: Pointer to the metadata header. +* @meta_size: Size of each metadata block. +* +* Finds the latest valid metadata by checking sequence numbers. If a +* valid entry with the highest sequence number is found, its pointer +* is returned. Returns NULL if no valid metadata is found. +*/ +static inline void *cbd_meta_find_latest (struct cbd_meta_header *header, +u32 meta_size) +{ +struct cbd_meta_header *meta, *latest =NULL ; +u32 i; + +for (i =0 ; i <CBDT_META_INDEX_MAX ; i++) { + meta = (void *)header + (i * meta_size); + +/* Skip if CRC check fails */ +if (meta->crc !=cbd_meta_crc (meta, meta_size)) +continue ; + +/* Update latest if a more recent sequence is found */ +if (!latest ||cbd_meta_seq_after (meta->seq, latest->seq)) + latest = meta; + } + +return latest; +} + +/* +* cbd_meta_find_oldest - Find the oldest valid metadata. +* @header: Pointer to the metadata header. +* @meta_size: Size of each metadata block. +* +* Returns the oldest valid metadata by comparing sequence numbers. +* If an entry with the lowest sequence number is found, its pointer +* is returned. Returns NULL if no valid metadata is found. +*/ +static inline void *cbd_meta_find_oldest (struct cbd_meta_header *header, +u32 meta_size) +{ +struct cbd_meta_header *meta, *oldest =NULL ; +u32 i; + +for (i =0 ; i <CBDT_META_INDEX_MAX ; i++) { + meta = (void *)header + (meta_size * i); + +/* Mark as oldest if CRC check fails */ +if (meta->crc !=cbd_meta_crc (meta, meta_size)) { + oldest = meta; +break ; + } + +/* Update oldest if an older sequence is found */ +if (!oldest ||cbd_meta_seq_after (oldest->seq, meta->seq)) + oldest = meta; + } + +BUG_ON (!oldest); + +return oldest; +} + +/* +* cbd_meta_get_next_seq - Get the next sequence number for metadata. +* @header: Pointer to the metadata header. +* @meta_size: Size of each metadata block. +* +* Returns the next sequence number based on the latest metadata entry. +* If no latest metadata is found, returns 0. +*/ +static inline u32 cbd_meta_get_next_seq (struct cbd_meta_header *header, +u32 meta_size) +{ +struct cbd_meta_header *latest; + + latest =cbd_meta_find_latest (header, meta_size); +if (!latest) +return 0 ; + +return (latest->seq +1 ); +} + +#define CBD_OBJ_HEARTBEAT (OBJ ) \ +static void OBJ##_hb_workfn(struct work_struct *work) \ +{ \ + struct cbd_##OBJ *obj = container_of(work, struct cbd_##OBJ, hb_work.work); \ + \ + cbd_##OBJ##_hb(obj); \ + \ + queue_delayed_work(cbd_wq, &obj->hb_work, CBD_HB_INTERVAL); \ +} \ + \ +bool cbd_##OBJ##_info_is_alive(struct cbd_##OBJ##_info *info) \ +{ \ + ktime_t oldest, ts; \ + \ + ts = info->alive_ts; \ + oldest = ktime_sub_ms(ktime_get_real(), CBD_HB_TIMEOUT); \ + \ + if (ktime_after(ts, oldest)) \ + return true; \ + \ + return false; \ +} \ + \ +static ssize_t alive_show(struct device *dev, \ + struct device_attribute *attr, \ + char *buf) \ +{ \ + struct cbd_##OBJ##_device *_dev; \ + struct cbd_##OBJ##_info *info; \ + \ + _dev = container_of(dev, struct cbd_##OBJ##_device, dev); \ + info = cbdt_##OBJ##_info_read(_dev->cbdt, _dev->id); \ + if (!info) \ + goto out; \ + \ + if (cbd_##OBJ##_info_is_alive(info)) \ + return sprintf(buf, "true\n"); \ + \ +out: \ + return sprintf(buf, "false\n"); \ +} \ +static DEVICE_ATTR_ADMIN_RO(alive) \ + +#endif /* _CBD_INTERNAL_H */
drivers/block/cbd/cbd_transport.c new file +1186 -0
1,186 lines not on this page · raw message
drivers/block/cbd/cbd_transport.h new file +169 -0
169 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 2/8 cbd: introduce cbd_host
In reply to Dongsheng YangThe "cbd_host" represents a host node. Each node needs to be registered before it can use the "cbd_transport". After registration, the node's information, such as its hostname, will be recorded in the "hosts" area of this transport. Through this mechanism, we can know which nodes are currently using each transport.
If a host dies without unregistering, we allow the user to clear this host entry in the metadata.
--- drivers/block/cbd/cbd_host.c | 227 +++++++++++++++++++++++++++++++++++ drivers/block/cbd/cbd_host.h | 67 +++++++++++ 2 files changed, 294 insertions(+) create mode 100644 drivers/block/cbd/cbd_host.c create mode 100644 drivers/block/cbd/cbd_host.h
drivers/block/cbd/cbd_host.c new file +227 -0
@@ -0,0 +1,227 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include "cbd_host.h" +#include "cbd_blkdev.h" +#include "cbd_backend.h" + +static ssize_t hostname_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_host_device *host_dev; +struct cbd_host_info *host_info; + + host_dev = container_of(dev, struct cbd_host_device, dev); + host_info =cbdt_host_info_read (host_dev->cbdt, host_dev->id); +if (!host_info) +return 0 ; + +if (host_info->state ==CBD_HOST_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%s\n" , host_info->hostname); +} +static DEVICE_ATTR_ADMIN_RO (hostname); + +static void host_info_write (struct cbd_host *host); +static void cbd_host_hb (struct cbd_host *host) +{ +host_info_write (host); +} +CBD_OBJ_HEARTBEAT (host); + +static struct attribute *cbd_host_attrs[] = { + &dev_attr_hostname.attr, + &dev_attr_alive.attr, +NULL +}; + +static struct attribute_group cbd_host_attr_group = { + .attrs = cbd_host_attrs, +}; + +static const struct attribute_group *cbd_host_attr_groups[] = { + &cbd_host_attr_group, +NULL +}; + +static void cbd_host_release (struct device *dev) +{ +} + +const struct device_type cbd_host_type = { + .name ="cbd_host" , + .groups = cbd_host_attr_groups, + .release = cbd_host_release, +}; + +const struct device_type cbd_hosts_type = { + .name ="cbd_hosts" , + .release = cbd_host_release, +}; + +static void host_info_write (struct cbd_host *host) +{ +mutex_lock (&host->info_lock); + host->host_info.alive_ts =ktime_get_real (); +cbdt_host_info_write (host->cbdt, &host->host_info,sizeof (struct cbd_host_info ), + host->host_id); +mutex_unlock (&host->info_lock); +} + +static int host_register_validate (struct cbd_transport *cbdt,char *hostname,u32 *host_id) +{ +struct cbd_host_info *host_info; +u32 host_id_tmp; +int ret; +u32 i; + +if (cbdt->host) +return -EEXIST ; + +if (strlen (hostname) ==0 ) { +cbdt_err (cbdt,"hostname is empty\n" ); +return -EINVAL ; + } + +if (*host_id ==UINT_MAX ) { + ret =cbd_host_find_id_by_name (cbdt, hostname, host_id); +if (!ret) + goto host_id_found; + +/* In single-host case, set the host_id to 0 */ +if (cbdt_is_single_host (cbdt)) { + *host_id =0 ; + }else { + ret =cbdt_get_empty_host_id (cbdt, host_id); +if (ret) { +cbdt_err (cbdt,"no available host id found.\n" ); +return -EBUSY ; + } + } + } + +host_id_found: +if (*host_id >= cbdt->transport_info.host_num) { +cbdt_err (cbdt,"host_id: %u is too large, host_num: %u\n" , + *host_id, cbdt->transport_info.host_num); +return -EINVAL ; + } + +/* check for duplicated hostname */ + ret =cbd_host_find_id_by_name (cbdt, hostname, &host_id_tmp); +if (!ret && (host_id_tmp != *host_id)) { +cbdt_err (cbdt,"duplicated hostname: %s with host: %u\n" , hostname, i); +return -EINVAL ; + } + + host_info =cbdt_host_info_read (cbdt, *host_id); +if (host_info &&cbd_host_info_is_alive (host_info)) { +pr_err ("host id %u is still alive\n" , *host_id); +return -EBUSY ; + } + +return 0 ; +} + +int cbd_host_register (struct cbd_transport *cbdt,char *hostname,u32 host_id) +{ +struct cbd_host *host; +int ret; + + ret =host_register_validate (cbdt, hostname, &host_id); +if (ret) +return ret; + + host =kzalloc (sizeof (struct cbd_host ),GFP_KERNEL ); +if (!host) +return -ENOMEM ; + + host->cbdt = cbdt; + host->host_id = host_id; +mutex_init (&host->info_lock); +INIT_DELAYED_WORK (&host->hb_work, host_hb_workfn); + + host->host_info.state =CBD_HOST_STATE_RUNNING ; +memcpy (host->host_info.hostname, hostname,CBD_NAME_LEN ); + + cbdt->host = host; + +host_info_write (host); +queue_delayed_work (cbd_wq, &host->hb_work,0 ); + +return 0 ; +} + +static bool host_backends_stopped (struct cbd_transport *cbdt,u32 host_id) +{ +struct cbd_backend_info *backend_info; +u32 i; + +cbd_for_each_backend_info (cbdt, i, backend_info) { +if (!backend_info || backend_info->state !=CBD_BACKEND_STATE_RUNNING ) +continue ; + +if (backend_info->host_id == host_id) { +cbdt_err (cbdt,"backend %u is still on host %u\n" , i, host_id); +return false; + } + } + +return true; +} + +static bool host_blkdevs_stopped (struct cbd_transport *cbdt,u32 host_id) +{ +struct cbd_blkdev_info *blkdev_info; +int i; + +cbd_for_each_blkdev_info (cbdt, i, blkdev_info) { +if (!blkdev_info || blkdev_info->state !=CBD_BLKDEV_STATE_RUNNING ) +continue ; + +if (blkdev_info->host_id == host_id) { +cbdt_err (cbdt,"blkdev %u is still on host %u\n" , i, host_id); +return false; + } + } + +return true; +} + +void cbd_host_unregister (struct cbd_transport *cbdt) +{ +struct cbd_host *host = cbdt->host; + +if (!host) { +cbd_err ("This host is not registered." ); +return ; + } + +cancel_delayed_work_sync (&host->hb_work); +cbdt_host_info_clear (cbdt, host->host_id); + cbdt->host =NULL ; +kfree (host); +} + +int cbd_host_clear (struct cbd_transport *cbdt,u32 host_id) +{ +struct cbd_host_info *host_info; + + host_info =cbdt_get_host_info (cbdt, host_id); +if (cbd_host_info_is_alive (host_info)) { +cbdt_err (cbdt,"host %u is still alive\n" , host_id); +return -EBUSY ; + } + +if (host_info->state ==CBD_HOST_STATE_NONE ) +return 0 ; + +if (!host_blkdevs_stopped (cbdt, host_id) || + !host_backends_stopped (cbdt, host_id)) +return -EBUSY ; + +cbdt_host_info_clear (cbdt, host_id); + +return 0 ; +}
drivers/block/cbd/cbd_host.h new file +67 -0
67 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 3/8 cbd: introduce cbd_segment
In reply to Dongsheng YangThe cbd_segments is an abstraction of the data area in transport. The data area in transport is divided into segments.
The specific use of this area is determined by cbd_seg_type. For example, cbd_blkdev and cbd_backend data transfers need to access a segment of the type cbds_type_channel.
--- drivers/block/cbd/cbd_segment.c | 311 ++++++++++++++++++++++++++++++++ drivers/block/cbd/cbd_segment.h | 104 +++++++++++ 2 files changed, 415 insertions(+) create mode 100644 drivers/block/cbd/cbd_segment.c create mode 100644 drivers/block/cbd/cbd_segment.h
drivers/block/cbd/cbd_segment.c new file +311 -0
@@ -0,0 +1,311 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include <linux/dax.h> + +#include "cbd_internal.h" +#include "cbd_transport.h" +#include "cbd_segment.h" + +static ssize_t type_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_segment_device *segment_dev; +struct cbd_segment_info *segment_info; + + segment_dev = container_of(dev, struct cbd_segment_device, dev); + segment_info =cbdt_segment_info_read (segment_dev->cbdt, segment_dev->id); +if (!segment_info) +return 0 ; + +if (segment_info->state ==CBD_SEGMENT_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%s\n" ,cbds_type_str (segment_info->type)); +} + +static DEVICE_ATTR_ADMIN_RO (type); + +static struct attribute *cbd_segment_attrs[] = { + &dev_attr_type.attr, +NULL +}; + +static struct attribute_group cbd_segment_attr_group = { + .attrs = cbd_segment_attrs, +}; + +static const struct attribute_group *cbd_segment_attr_groups[] = { + &cbd_segment_attr_group, +NULL +}; + +static void cbd_segment_release (struct device *dev) +{ +} + +const struct device_type cbd_segment_type = { + .name ="cbd_segment" , + .groups = cbd_segment_attr_groups, + .release = cbd_segment_release, +}; + +const struct device_type cbd_segments_type = { + .name ="cbd_segments" , + .release = cbd_segment_release, +}; + +void cbd_segment_init (struct cbd_transport *cbdt,struct cbd_segment *segment, +struct cbds_init_options *options) +{ + segment->cbdt = cbdt; + segment->seg_id = options->seg_id; + segment->seg_ops = options->seg_ops; + segment->data_size =CBDT_SEG_SIZE - options->data_off; + + segment->data =cbd_segment_addr (segment) + options->data_off; +} + +void cbd_segment_clear (struct cbd_transport *cbdt,u32 seg_id) +{ +struct cbd_segment_info *segment_info; + + segment_info =cbdt_get_segment_info (cbdt, seg_id); +cbdt_zero_range (cbdt, segment_info,CBDT_SEG_SIZE ); +} + +void cbd_segment_info_clear (struct cbd_segment *segment) +{ +cbdt_segment_info_clear (segment->cbdt, segment->seg_id); +} + +void cbds_copy_data (struct cbd_seg_pos *dst_pos, +struct cbd_seg_pos *src_pos,u32 len) +{ +u32 copied =0 ; +u32 to_copy; + +while (copied < len) { +if (dst_pos->off >= dst_pos->segment->data_size) + dst_pos->segment->seg_ops->sanitize_pos (dst_pos); +if (src_pos->off >= src_pos->segment->data_size) + src_pos->segment->seg_ops->sanitize_pos (src_pos); + + to_copy = len - copied; + +if (to_copy > dst_pos->segment->data_size - dst_pos->off) + to_copy = dst_pos->segment->data_size - dst_pos->off; +if (to_copy > src_pos->segment->data_size - src_pos->off) + to_copy = src_pos->segment->data_size - src_pos->off; + +memcpy_flushcache (dst_pos->segment->data + dst_pos->off, src_pos->segment->data + src_pos->off, to_copy); + + copied += to_copy; +cbds_pos_advance (dst_pos, to_copy); +cbds_pos_advance (src_pos, to_copy); + } +} + +int cbds_copy_to_bio (struct cbd_segment *segment, +u32 data_off,u32 data_len,struct bio *bio,u32 bio_off) +{ +struct bio_vec bv; +struct bvec_iter iter; +void *dst; +u32 to_copy, page_off =0 ; +struct cbd_seg_pos pos = { .segment = segment, + .off = data_off }; +next: +bio_for_each_segment (bv, bio, iter) { +if (bio_off > bv.bv_len) { + bio_off -= bv.bv_len; +continue ; + } + page_off = bv.bv_offset; + page_off += bio_off; + bio_off =0 ; + + dst =kmap_local_page (bv.bv_page); +again: +if (pos.off >= pos.segment->data_size) + segment->seg_ops->sanitize_pos (&pos); + segment = pos.segment; + + to_copy =min (bv.bv_offset + bv.bv_len - page_off, + segment->data_size - pos.off); +if (to_copy > data_len) + to_copy = data_len; + +flush_dcache_page (bv.bv_page); +memcpy (dst + page_off, segment->data + pos.off, to_copy); + +/* advance */ + pos.off += to_copy; + page_off += to_copy; + data_len -= to_copy; +if (!data_len) { +kunmap_local (dst); +return 0 ; + } + +/* more data in this bv page */ +if (page_off < bv.bv_offset + bv.bv_len) + goto again; +kunmap_local (dst); + } + +if (bio->bi_next) { + bio = bio->bi_next; + goto next; + } + +return 0 ; +} + +void cbds_copy_from_bio (struct cbd_segment *segment, +u32 data_off,u32 data_len,struct bio *bio,u32 bio_off) +{ +struct bio_vec bv; +struct bvec_iter iter; +void *src; +u32 to_copy, page_off =0 ; +struct cbd_seg_pos pos = { .segment = segment, + .off = data_off }; +next: +bio_for_each_segment (bv, bio, iter) { +if (bio_off > bv.bv_len) { + bio_off -= bv.bv_len; +continue ; + } + page_off = bv.bv_offset; + page_off += bio_off; + bio_off =0 ; + + src =kmap_local_page (bv.bv_page); +again: +if (pos.off >= pos.segment->data_size) + segment->seg_ops->sanitize_pos (&pos); + segment = pos.segment; + + to_copy =min (bv.bv_offset + bv.bv_len - page_off, + segment->data_size - pos.off); +if (to_copy > data_len) + to_copy = data_len; + +memcpy_flushcache (segment->data + pos.off, src + page_off, to_copy); +flush_dcache_page (bv.bv_page); + +/* advance */ + pos.off += to_copy; + page_off += to_copy; + data_len -= to_copy; +if (!data_len) { +kunmap_local (src); +return ; + } + +/* more data in this bv page */ +if (page_off < bv.bv_offset + bv.bv_len) + goto again; +kunmap_local (src); + } + +if (bio->bi_next) { + bio = bio->bi_next; + goto next; + } +} + +u32 cbd_seg_crc (struct cbd_segment *segment,u32 data_off,u32 data_len) +{ +u32 crc =0 ; +u32 crc_size; +struct cbd_seg_pos pos = { .segment = segment, + .off = data_off }; + +while (data_len) { +if (pos.off >= pos.segment->data_size) + segment->seg_ops->sanitize_pos (&pos); + segment = pos.segment; + + crc_size =min (segment->data_size - pos.off, data_len); + + crc =crc32 (crc, segment->data + pos.off, crc_size); + + data_len -= crc_size; + pos.off += crc_size; + } + +return crc; +} + +int cbds_map_pages (struct cbd_segment *segment, +struct bio *bio, +u32 off,u32 size) +{ +struct cbd_transport *cbdt = segment->cbdt; +u32 done =0 ; +struct page *page; +u32 page_off; +int ret =0 ; +int id; + + id =dax_read_lock (); +while (size) { +unsigned int len =min_t (size_t,PAGE_SIZE , size); +struct cbd_seg_pos pos = { .segment = segment, + .off = off + done }; + +if (pos.off >= pos.segment->data_size) + segment->seg_ops->sanitize_pos (&pos); + segment = pos.segment; + +u64 transport_off = segment->data - + (void *)cbdt->transport_info_addr + pos.off; + + page =cbdt_page (cbdt, transport_off, &page_off); + + ret =bio_add_page (bio, page, len,0 ); +if (unlikely (ret != len)) { +cbd_segment_err (segment,"failed to add page\n" ); + goto out; + } + + done += len; + size -= len; + } + + ret =0 ; +out: +dax_read_unlock (id); +return ret; +} + +int cbds_pos_advance (struct cbd_seg_pos *seg_pos,u32 len) +{ +u32 to_advance; + +while (len) { + to_advance = len; + +if (seg_pos->off >= seg_pos->segment->data_size) + seg_pos->segment->seg_ops->sanitize_pos (seg_pos); + +if (to_advance > seg_pos->segment->data_size - seg_pos->off) + to_advance = seg_pos->segment->data_size - seg_pos->off; + + seg_pos->off += to_advance; + + len -= to_advance; + } + +return 0 ; +} + +void *cbd_segment_addr (struct cbd_segment *segment) +{ +struct cbd_segment_info *seg_info; + + seg_info =cbdt_get_segment_info (segment->cbdt, segment->seg_id); + +return (void *)seg_info; +}
drivers/block/cbd/cbd_segment.h new file +104 -0
104 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 4/8 cbd: introduce cbd_channel
In reply to Dongsheng YangThe "cbd_channel" is the component responsible for the interaction between the blkdev and the backend. It mainly provides the functions "cbdc_copy_to_bio", "cbdc_copy_from_bio" and "cbd_channel_crc"
If the blkdev or backend is alive, that means there is active user for this channel, then channel is alive.
--- drivers/block/cbd/cbd_channel.c | 144 +++++++++++ drivers/block/cbd/cbd_channel.h | 429 ++++++++++++++++++++++++++++++++ 2 files changed, 573 insertions(+) create mode 100644 drivers/block/cbd/cbd_channel.c create mode 100644 drivers/block/cbd/cbd_channel.h
drivers/block/cbd/cbd_channel.c new file +144 -0
@@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include "cbd_transport.h" +#include "cbd_channel.h" + +int cbdc_copy_to_bio (struct cbd_channel *channel, +u32 data_off,u32 data_len,struct bio *bio,u32 bio_off) +{ +return cbds_copy_to_bio (&channel->segment, data_off, data_len, bio, bio_off); +} + +void cbdc_copy_from_bio (struct cbd_channel *channel, +u32 data_off,u32 data_len,struct bio *bio,u32 bio_off) +{ +cbds_copy_from_bio (&channel->segment, data_off, data_len, bio, bio_off); +} + +u32 cbd_channel_crc (struct cbd_channel *channel,u32 data_off,u32 data_len) +{ +return cbd_seg_crc (&channel->segment, data_off, data_len); +} + +int cbdc_map_pages (struct cbd_channel *channel,struct bio *bio,u32 off,u32 size) +{ +return cbds_map_pages (&channel->segment, bio, off, size); +} + +void cbd_channel_reset (struct cbd_channel *channel) +{ +/* Reset channel data head and tail pointers */ + channel->data_head = channel->data_tail =0 ; + +/* Reset submr and compr control pointers */ + channel->ctrl->submr_tail = channel->ctrl->submr_head =0 ; + channel->ctrl->compr_tail = channel->ctrl->compr_head =0 ; + +cbdt_zero_range (channel->cbdt, channel->submr,CBDC_SUBMR_SIZE ); +cbdt_zero_range (channel->cbdt, channel->compr,CBDC_COMPR_SIZE ); +} + +/* +* cbd_channel_seg_sanitize_pos - Sanitize position within a channel segment ring +* @pos: Position structure within the segment to sanitize +* +* This function ensures that the offset in the segment position wraps around +* correctly when the channel is using a single segment in a ring structure. If +* the offset exceeds the data size of the segment, it wraps back to the start of +* the segment by reducing it by the segment's data size. This allows the channel +* to reuse the segment space efficiently in a circular manner, preventing overflows. +*/ +static void cbd_channel_seg_sanitize_pos (struct cbd_seg_pos *pos) +{ +struct cbd_segment *segment = pos->segment; + +/* Channel only uses one segment as a ring */ +while (pos->off >= segment->data_size) + pos->off -= segment->data_size; +} + +static struct cbd_seg_ops cbd_channel_seg_ops = { + .sanitize_pos = cbd_channel_seg_sanitize_pos +}; + +static int channel_info_load (struct cbd_channel *channel) +{ +struct cbd_channel_seg_info *channel_info; +int ret; + +mutex_lock (&channel->info_lock); + channel_info = (struct cbd_channel_seg_info *)cbdt_segment_info_read (channel->cbdt, + channel->seg_id); +if (!channel_info) { +cbd_channel_err (channel,"can't read info from segment id: %u\n" , + channel->seg_id); + ret = -EINVAL ; + goto out; + } +memcpy (&channel->channel_info, channel_info,sizeof (struct cbd_channel_seg_info )); + ret =0 ; +out: +mutex_unlock (&channel->info_lock); +return ret; +} + +static void channel_info_write (struct cbd_channel *channel) +{ +mutex_lock (&channel->info_lock); +cbdt_segment_info_write (channel->cbdt, &channel->channel_info,sizeof (struct cbd_channel_seg_info ), + channel->seg_id); +mutex_unlock (&channel->info_lock); +} + +int cbd_channel_init (struct cbd_channel *channel,struct cbd_channel_init_options *init_opts) +{ +struct cbds_init_options seg_options = {0 }; +void *seg_addr; +int ret; + + seg_options.seg_id = init_opts->seg_id; + seg_options.data_off =CBDC_DATA_OFF ; + seg_options.seg_ops = &cbd_channel_seg_ops; + +cbd_segment_init (init_opts->cbdt, &channel->segment, &seg_options); + + channel->cbdt = init_opts->cbdt; + channel->seg_id = init_opts->seg_id; + channel->submr_size =rounddown (CBDC_SUBMR_SIZE ,sizeof (struct cbd_se )); + channel->compr_size =rounddown (CBDC_COMPR_SIZE ,sizeof (struct cbd_ce )); + channel->data_size =CBDC_DATA_SIZE ; + + seg_addr =cbd_segment_addr (&channel->segment); + channel->ctrl = seg_addr +CBDC_CTRL_OFF ; + channel->submr = seg_addr +CBDC_SUBMR_OFF ; + channel->compr = seg_addr +CBDC_COMPR_OFF ; + +spin_lock_init (&channel->submr_lock); +spin_lock_init (&channel->compr_lock); +mutex_init (&channel->info_lock); + +if (init_opts->new_channel) { +/* Initialize new channel state */ + channel->channel_info.seg_info.type =CBDS_TYPE_CHANNEL ; + channel->channel_info.seg_info.state =CBD_SEGMENT_STATE_RUNNING ; + channel->channel_info.seg_info.flags =0 ; + channel->channel_info.seg_info.backend_id = init_opts->backend_id; + +/* Persist new channel information */ +channel_info_write (channel); + }else { +/* Load existing channel information for reattachment or blkdev side */ + ret =channel_info_load (channel); +if (ret) + goto out; + } + ret =0 ; + +out: +return ret; +} + +void cbd_channel_destroy (struct cbd_channel *channel) +{ +cbdt_segment_info_clear (channel->cbdt, channel->seg_id); +}
drivers/block/cbd/cbd_channel.h new file +429 -0
429 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 5/8 cbd: introduce cbd_blkdev
In reply to Dongsheng YangThe "cbd_blkdev" represents a virtual block device named "/dev/cbdX". It corresponds to a backend. The "blkdev" interacts with upper-layer users and accepts IO requests from them. A "blkdev" includes multiple "cbd_queues", each of which requires a "cbd_channel" to interact with the backend's handler. The "cbd_queue" forwards IO requests from the upper layer to the backend's handler through the channel.
--- drivers/block/cbd/cbd_blkdev.c | 551 +++++++++++++++++++++++++++++++++ drivers/block/cbd/cbd_blkdev.h | 92 ++++++ drivers/block/cbd/cbd_queue.c | 516 ++++++++++++++++++++++++++++++ drivers/block/cbd/cbd_queue.h | 288 +++++++++++++++++ 4 files changed, 1447 insertions(+) create mode 100644 drivers/block/cbd/cbd_blkdev.c create mode 100644 drivers/block/cbd/cbd_blkdev.h create mode 100644 drivers/block/cbd/cbd_queue.c create mode 100644 drivers/block/cbd/cbd_queue.h
drivers/block/cbd/cbd_blkdev.c new file +551 -0
@@ -0,0 +1,551 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include "cbd_internal.h" +#include "cbd_blkdev.h" + +static ssize_t backend_id_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_blkdev_device *blkdev_dev; +struct cbd_blkdev_info *blkdev_info; + + blkdev_dev = container_of(dev, struct cbd_blkdev_device, dev); + blkdev_info =cbdt_blkdev_info_read (blkdev_dev->cbdt, blkdev_dev->id); +if (!blkdev_info) +return 0 ; + +if (blkdev_info->state ==CBD_BLKDEV_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%u\n" , blkdev_info->backend_id); +} +static DEVICE_ATTR_ADMIN_RO (backend_id); + +static ssize_t host_id_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_blkdev_device *blkdev_dev; +struct cbd_blkdev_info *blkdev_info; + + blkdev_dev = container_of(dev, struct cbd_blkdev_device, dev); + blkdev_info =cbdt_blkdev_info_read (blkdev_dev->cbdt, blkdev_dev->id); +if (!blkdev_info) +return 0 ; + +if (blkdev_info->state ==CBD_BLKDEV_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%u\n" , blkdev_info->host_id); +} +static DEVICE_ATTR_ADMIN_RO (host_id); + +static ssize_t mapped_id_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_blkdev_device *blkdev_dev; +struct cbd_blkdev_info *blkdev_info; + + blkdev_dev = container_of(dev, struct cbd_blkdev_device, dev); + blkdev_info =cbdt_blkdev_info_read (blkdev_dev->cbdt, blkdev_dev->id); +if (!blkdev_info) +return 0 ; + +if (blkdev_info->state ==CBD_BLKDEV_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%u\n" , blkdev_info->mapped_id); +} +static DEVICE_ATTR_ADMIN_RO (mapped_id); + +static void blkdev_info_write (struct cbd_blkdev *blkdev) +{ +mutex_lock (&blkdev->info_lock); + blkdev->blkdev_info.alive_ts =ktime_get_real (); +cbdt_blkdev_info_write (blkdev->cbdt, &blkdev->blkdev_info, +sizeof (struct cbd_blkdev_info ), + blkdev->blkdev_id); +mutex_unlock (&blkdev->info_lock); +} + +static void cbd_blkdev_hb (struct cbd_blkdev *blkdev) +{ +blkdev_info_write (blkdev); +} +CBD_OBJ_HEARTBEAT (blkdev); + +static struct attribute *cbd_blkdev_attrs[] = { + &dev_attr_mapped_id.attr, + &dev_attr_host_id.attr, + &dev_attr_backend_id.attr, + &dev_attr_alive.attr, +NULL +}; + +static struct attribute_group cbd_blkdev_attr_group = { + .attrs = cbd_blkdev_attrs, +}; + +static const struct attribute_group *cbd_blkdev_attr_groups[] = { + &cbd_blkdev_attr_group, +NULL +}; + +static void cbd_blkdev_release (struct device *dev) +{ +} + +const struct device_type cbd_blkdev_type = { + .name ="cbd_blkdev" , + .groups = cbd_blkdev_attr_groups, + .release = cbd_blkdev_release, +}; + +const struct device_type cbd_blkdevs_type = { + .name ="cbd_blkdevs" , + .release = cbd_blkdev_release, +}; + + +static int cbd_major; +static DEFINE_IDA (cbd_mapped_id_ida); + +static int minor_to_cbd_mapped_id (int minor) +{ +return minor >>CBD_PART_SHIFT ; +} + + +static int cbd_open (struct gendisk *disk,blk_mode_t mode) +{ +struct cbd_blkdev *cbd_blkdev = disk->private_data; + +mutex_lock (&cbd_blkdev->lock); + cbd_blkdev->open_count++; +mutex_unlock (&cbd_blkdev->lock); + +return 0 ; +} + +static void cbd_release (struct gendisk *disk) +{ +struct cbd_blkdev *cbd_blkdev = disk->private_data; + +mutex_lock (&cbd_blkdev->lock); + cbd_blkdev->open_count--; +mutex_unlock (&cbd_blkdev->lock); +} + +static const struct block_device_operations cbd_bd_ops = { + .owner =THIS_MODULE , + .open = cbd_open, + .release = cbd_release, +}; + +/** +* cbd_blkdev_destroy_queues - Stop and free the queues associated with the block device +* @cbd_blkdev: Pointer to the block device structure +* +* Note: The cbd_queue_stop function checks the state of each queue before attempting +* to stop it. If a queue's state is not running, it will return immediately, +* ensuring that only running queues are affected by this operation. +*/ +static void cbd_blkdev_destroy_queues (struct cbd_blkdev *cbd_blkdev) +{ +int i; + +/* Stop each queue associated with the block device */ +for (i =0 ; i < cbd_blkdev->num_queues; i++) +cbd_queue_stop (&cbd_blkdev->queues[i]); + +/* Free the memory allocated for the queues */ +kfree (cbd_blkdev->queues); +} + +/** +* cbd_blkdev_create_queues - Create and initialize queues for the block device +* @cbd_blkdev: Pointer to the block device structure +* @channels: Array of channel identifiers for each queue +* +* Note: The cbd_blkdev_destroy_queues function checks the state of each queue. +* Only queues that have been started will be stopped in the error path. +* Therefore, any queues that were not started will not be affected. +*/ +static int cbd_blkdev_create_queues (struct cbd_blkdev *cbd_blkdev,u32 *channels) +{ +int i; +int ret; +struct cbd_queue *cbdq; + + cbd_blkdev->queues =kcalloc (cbd_blkdev->num_queues,sizeof (struct cbd_queue ),GFP_KERNEL ); +if (!cbd_blkdev->queues) +return -ENOMEM ; + +for (i =0 ; i < cbd_blkdev->num_queues; i++) { + cbdq = &cbd_blkdev->queues[i]; + cbdq->cbd_blkdev = cbd_blkdev; + cbdq->index = i; + + ret =cbd_queue_start (cbdq, channels[i]); +if (ret) + goto err; + } + +return 0 ; + +err: +cbd_blkdev_destroy_queues (cbd_blkdev); +return ret; +} + +static int disk_start (struct cbd_blkdev *cbd_blkdev) +{ +struct gendisk *disk; +struct queue_limits lim = { + .max_hw_sectors =BIO_MAX_VECS *PAGE_SECTORS , + .io_min =4096 , + .io_opt =4096 , + .max_segments =USHRT_MAX , + .max_segment_size =UINT_MAX , + .discard_granularity =0 , + .max_hw_discard_sectors =0 , + .max_write_zeroes_sectors =0 + }; +int ret; + +memset (&cbd_blkdev->tag_set,0 ,sizeof (cbd_blkdev->tag_set)); + cbd_blkdev->tag_set.ops = &cbd_mq_ops; + cbd_blkdev->tag_set.queue_depth =128 ; + cbd_blkdev->tag_set.numa_node =NUMA_NO_NODE ; + cbd_blkdev->tag_set.flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_NO_SCHED ; + cbd_blkdev->tag_set.nr_hw_queues = cbd_blkdev->num_queues; + cbd_blkdev->tag_set.cmd_size =sizeof (struct cbd_request ); + cbd_blkdev->tag_set.timeout =0 ; + cbd_blkdev->tag_set.driver_data = cbd_blkdev; + + ret =blk_mq_alloc_tag_set (&cbd_blkdev->tag_set); +if (ret) { +cbd_blk_err (cbd_blkdev,"failed to alloc tag set %d" , ret); + goto err; + } + + disk =blk_mq_alloc_disk (&cbd_blkdev->tag_set, &lim, cbd_blkdev); +if (IS_ERR (disk)) { + ret =PTR_ERR (disk); +cbd_blk_err (cbd_blkdev,"failed to alloc disk" ); + goto out_tag_set; + } + +snprintf (disk->disk_name,sizeof (disk->disk_name),"cbd%d" , + cbd_blkdev->mapped_id); + + disk->major = cbd_major; + disk->first_minor = cbd_blkdev->mapped_id <<CBD_PART_SHIFT ; + disk->minors = (1 <<CBD_PART_SHIFT ); + disk->fops = &cbd_bd_ops; + disk->private_data = cbd_blkdev; + + cbd_blkdev->disk = disk; +cbdt_add_blkdev (cbd_blkdev->cbdt, cbd_blkdev); + cbd_blkdev->blkdev_info.mapped_id = cbd_blkdev->blkdev_id; + +set_capacity (cbd_blkdev->disk, cbd_blkdev->dev_size); +set_disk_ro (cbd_blkdev->disk, false); + +/* Register the disk with the system */ + ret =add_disk (cbd_blkdev->disk); +if (ret) + goto put_disk; + +/* Create a symlink to the block device */ + ret =sysfs_create_link (&disk_to_dev (cbd_blkdev->disk)->kobj, + &cbd_blkdev->blkdev_dev->dev.kobj,"cbd_blkdev" ); +if (ret) + goto del_disk; + +return 0 ; + +del_disk: +del_gendisk (cbd_blkdev->disk); +put_disk: +put_disk (cbd_blkdev->disk); +out_tag_set: +blk_mq_free_tag_set (&cbd_blkdev->tag_set); +err: +return ret; +} + +static void disk_stop (struct cbd_blkdev *cbd_blkdev) +{ +sysfs_remove_link (&disk_to_dev (cbd_blkdev->disk)->kobj,"cbd_blkdev" ); +del_gendisk (cbd_blkdev->disk); +put_disk (cbd_blkdev->disk); +blk_mq_free_tag_set (&cbd_blkdev->tag_set); +} + +/** +* If *queues is 0, it defaults to backend_info->n_handlers, matching the backend's +* handler capacity. +*/ +static int blkdev_start_validate (struct cbd_transport *cbdt,struct cbd_backend_info *backend_info, +u32 backend_id,u32 *queues) +{ +struct cbd_blkdev_info *blkdev_info; +u32 backend_blkdevs =0 ; +u32 i; + +if (!backend_info || !cbd_backend_info_is_alive (backend_info)) { +cbdt_err (cbdt,"backend %u is not alive\n" , backend_id); +return -EINVAL ; + } + +cbd_for_each_blkdev_info (cbdt, i, blkdev_info) { +if (!blkdev_info || blkdev_info->state !=CBD_BLKDEV_STATE_RUNNING ) +continue ; + +if (blkdev_info->backend_id == backend_id) + backend_blkdevs++; + } + +if (backend_blkdevs >=CBDB_BLKDEV_COUNT_MAX ) { +cbdt_err (cbdt,"too many(%u) blkdevs connected to backend %u.\n" , backend_blkdevs, backend_id); +return -EBUSY ; + } + +if (*queues ==0 ) + *queues = backend_info->n_handlers; + +if (*queues > backend_info->n_handlers) { +cbdt_err (cbdt,"invalid queues: %u, larger than backend handlers: %u\n" , + *queues, backend_info->n_handlers); +return -EINVAL ; + } + +return 0 ; +} + +static struct cbd_blkdev *blkdev_alloc (struct cbd_transport *cbdt) +{ +struct cbd_blkdev *cbd_blkdev; +int ret; + + cbd_blkdev =kzalloc (sizeof (struct cbd_blkdev ),GFP_KERNEL ); +if (!cbd_blkdev) +return NULL ; + + cbd_blkdev->cbdt = cbdt; +mutex_init (&cbd_blkdev->lock); +mutex_init (&cbd_blkdev->info_lock); +INIT_LIST_HEAD (&cbd_blkdev->node); +INIT_DELAYED_WORK (&cbd_blkdev->hb_work, blkdev_hb_workfn); + + ret =cbdt_get_empty_blkdev_id (cbdt, &cbd_blkdev->blkdev_id); +if (ret <0 ) + goto blkdev_free; + + cbd_blkdev->mapped_id =ida_simple_get (&cbd_mapped_id_ida,0 , +minor_to_cbd_mapped_id (1 <<MINORBITS ), +GFP_KERNEL ); +if (cbd_blkdev->mapped_id <0 ) { + ret = -ENOENT ; + goto blkdev_free; + } + + cbd_blkdev->task_wq =alloc_workqueue ("cbdt%d-d%u" ,WQ_UNBOUND |WQ_MEM_RECLAIM , +0 , cbdt->id, cbd_blkdev->mapped_id); +if (!cbd_blkdev->task_wq) { + ret = -ENOMEM ; + goto ida_remove; + } + +return cbd_blkdev; + +ida_remove: +ida_simple_remove (&cbd_mapped_id_ida, cbd_blkdev->mapped_id); +blkdev_free: +kfree (cbd_blkdev); + +return NULL ; +} + +static void blkdev_free (struct cbd_blkdev *cbd_blkdev) +{ +drain_workqueue (cbd_blkdev->task_wq); +destroy_workqueue (cbd_blkdev->task_wq); +ida_simple_remove (&cbd_mapped_id_ida, cbd_blkdev->mapped_id); +kfree (cbd_blkdev); +} + +static int blkdev_cache_init (struct cbd_blkdev *cbd_blkdev) +{ +struct cbd_transport *cbdt = cbd_blkdev->cbdt; +struct cbd_cache_opts cache_opts = {0 }; + + cache_opts.cache_info = &cbd_blkdev->cache_info; + cache_opts.cache_id = cbd_blkdev->backend_id; + cache_opts.owner =NULL ; + cache_opts.new_cache = false; + cache_opts.start_writeback = false; + cache_opts.start_gc = true; + cache_opts.init_req_keys = true; + cache_opts.dev_size = cbd_blkdev->dev_size; + cache_opts.n_paral = cbd_blkdev->num_queues; + + cbd_blkdev->cbd_cache =cbd_cache_alloc (cbdt, &cache_opts); +if (!cbd_blkdev->cbd_cache) +return -ENOMEM ; + +return 0 ; +} + +static void blkdev_cache_destroy (struct cbd_blkdev *cbd_blkdev) +{ +if (cbd_blkdev->cbd_cache) +cbd_cache_destroy (cbd_blkdev->cbd_cache); +} + +static int blkdev_init (struct cbd_blkdev *cbd_blkdev,struct cbd_backend_info *backend_info, +u32 backend_id,u32 queues) +{ +struct cbd_transport *cbdt = cbd_blkdev->cbdt; +int ret; + + cbd_blkdev->backend_id = backend_id; + cbd_blkdev->num_queues = queues; + cbd_blkdev->dev_size = backend_info->dev_size; + cbd_blkdev->blkdev_dev = &cbdt->cbd_blkdevs_dev->blkdev_devs[cbd_blkdev->blkdev_id]; + +/* Get the backend if it is hosted on the same machine */ +if (backend_info->host_id == cbdt->host->host_id) + cbd_blkdev->backend =cbdt_get_backend (cbdt, backend_id); + + cbd_blkdev->blkdev_info.backend_id = backend_id; + cbd_blkdev->blkdev_info.host_id = cbdt->host->host_id; + cbd_blkdev->blkdev_info.state =CBD_BLKDEV_STATE_RUNNING ; + + ret =cbd_blkdev_create_queues (cbd_blkdev, backend_info->handler_channels); +if (ret <0 ) + goto err; + +if (cbd_backend_cache_on (backend_info)) { + ret =blkdev_cache_init (cbd_blkdev); +if (ret) + goto destroy_queues; + } + +return 0 ; +destroy_queues: +cbd_blkdev_destroy_queues (cbd_blkdev); +err: +return ret; +} + +static void blkdev_destroy (struct cbd_blkdev *cbd_blkdev) +{ +cancel_delayed_work_sync (&cbd_blkdev->hb_work); +blkdev_cache_destroy (cbd_blkdev); +cbd_blkdev_destroy_queues (cbd_blkdev); +} + +int cbd_blkdev_start (struct cbd_transport *cbdt,u32 backend_id,u32 queues) +{ +struct cbd_blkdev *cbd_blkdev; +struct cbd_backend_info *backend_info; +int ret; + + backend_info =cbdt_backend_info_read (cbdt, backend_id); +if (!backend_info) { +cbdt_err (cbdt,"cant read backend info for backend%u.\n" , backend_id); +return -ENOENT ; + } + + ret =blkdev_start_validate (cbdt, backend_info, backend_id, &queues); +if (ret) +return ret; + + cbd_blkdev =blkdev_alloc (cbdt); +if (!cbd_blkdev) +return -ENOMEM ; + + ret =blkdev_init (cbd_blkdev, backend_info, backend_id, queues); +if (ret) + goto blkdev_free; + + ret =disk_start (cbd_blkdev); +if (ret <0 ) + goto blkdev_destroy; + +blkdev_info_write (cbd_blkdev); +queue_delayed_work (cbd_wq, &cbd_blkdev->hb_work,0 ); + +return 0 ; + +blkdev_destroy: +blkdev_destroy (cbd_blkdev); +blkdev_free: +blkdev_free (cbd_blkdev); +return ret; +} + +int cbd_blkdev_stop (struct cbd_transport *cbdt,u32 devid) +{ +struct cbd_blkdev *cbd_blkdev; + + cbd_blkdev =cbdt_get_blkdev (cbdt, devid); +if (!cbd_blkdev) +return -EINVAL ; + +mutex_lock (&cbd_blkdev->lock); +if (cbd_blkdev->open_count >0 ) { +mutex_unlock (&cbd_blkdev->lock); +return -EBUSY ; + } + +cbdt_del_blkdev (cbdt, cbd_blkdev); +mutex_unlock (&cbd_blkdev->lock); + +disk_stop (cbd_blkdev); +blkdev_destroy (cbd_blkdev); +blkdev_free (cbd_blkdev); +cbdt_blkdev_info_clear (cbdt, devid); + +return 0 ; +} + +int cbd_blkdev_clear (struct cbd_transport *cbdt,u32 devid) +{ +struct cbd_blkdev_info *blkdev_info; + + blkdev_info =cbdt_blkdev_info_read (cbdt, devid); +if (!blkdev_info) { +cbdt_err (cbdt,"all blkdev_info in blkdev_id: %u are corrupted.\n" , devid); +return -EINVAL ; + } + +if (cbd_blkdev_info_is_alive (blkdev_info)) { +cbdt_err (cbdt,"blkdev %u is still alive\n" , devid); +return -EBUSY ; + } + +if (blkdev_info->state ==CBD_BLKDEV_STATE_NONE ) +return 0 ; + +cbdt_blkdev_info_clear (cbdt, devid); + +return 0 ; +} + +int cbd_blkdev_init (void ) +{ + cbd_major =register_blkdev (0 ,"cbd" ); +if (cbd_major <0 ) +return cbd_major; + +return 0 ; +} + +void cbd_blkdev_exit (void ) +{ +unregister_blkdev (cbd_major,"cbd" ); +}
drivers/block/cbd/cbd_blkdev.h new file +92 -0
92 lines not on this page · raw message
drivers/block/cbd/cbd_queue.c new file +516 -0
516 lines not on this page · raw message
drivers/block/cbd/cbd_queue.h new file +288 -0
288 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 6/8 cbd: introduce cbd_backend
In reply to Dongsheng YangThe "cbd_backend" is responsible for exposing a local block device (such as "/dev/sda") through the "cbd_transport" to other hosts.
Any host that registers this transport can map this backend to a local "cbd device"(such as "/dev/cbd0"). All reads and writes to "cbd0" are transmitted through the channel inside the transport to the backend. The handler inside the backend is responsible for processing these read and write requests, converting them into read and write requests corresponding to "sda".
--- drivers/block/cbd/cbd_backend.c | 730 ++++++++++++++++++++++++++++++++ drivers/block/cbd/cbd_backend.h | 137 ++++++ drivers/block/cbd/cbd_handler.c | 468 ++++++++++++++++++++ drivers/block/cbd/cbd_handler.h | 66 +++ 4 files changed, 1401 insertions(+) create mode 100644 drivers/block/cbd/cbd_backend.c create mode 100644 drivers/block/cbd/cbd_backend.h create mode 100644 drivers/block/cbd/cbd_handler.c create mode 100644 drivers/block/cbd/cbd_handler.h
drivers/block/cbd/cbd_backend.c new file +730 -0
@@ -0,0 +1,730 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include "cbd_internal.h" +#include "cbd_transport.h" +#include "cbd_host.h" +#include "cbd_segment.h" +#include "cbd_channel.h" +#include "cbd_cache/cbd_cache.h" +#include "cbd_handler.h" +#include "cbd_backend.h" + +static ssize_t host_id_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_backend_device *backend; +struct cbd_backend_info *latest_info; + + backend = container_of(dev, struct cbd_backend_device, dev); + latest_info =cbdt_backend_info_read (backend->cbdt, backend->id); +if (!latest_info || latest_info->state ==CBD_BACKEND_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%u\n" , latest_info->host_id); +} +static DEVICE_ATTR_ADMIN_RO (host_id); + +static ssize_t path_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_backend_device *backend; +struct cbd_backend_info *latest_info; + + backend = container_of(dev, struct cbd_backend_device, dev); + latest_info =cbdt_backend_info_read (backend->cbdt, backend->id); +if (!latest_info || latest_info->state ==CBD_BACKEND_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%s\n" , latest_info->path); +} +static DEVICE_ATTR_ADMIN_RO (path); + +/* sysfs for cache */ +static ssize_t cache_segs_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_backend_device *backend; +struct cbd_backend_info *latest_info; + + backend = container_of(dev, struct cbd_backend_device, dev); + latest_info =cbdt_backend_info_read (backend->cbdt, backend->id); +if (!latest_info || latest_info->state ==CBD_BACKEND_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%u\n" , latest_info->cache_info.n_segs); +} +static DEVICE_ATTR_ADMIN_RO (cache_segs); + +static ssize_t cache_used_segs_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_backend_device *backend; +struct cbd_backend_info *latest_info; +u32 used_segs =0 ; + + backend = container_of(dev, struct cbd_backend_device, dev); + latest_info =cbdt_backend_info_read (backend->cbdt, backend->id); +if (!latest_info || latest_info->state ==CBD_BACKEND_STATE_NONE ) +return 0 ; + +if (!latest_info->cache_info.n_segs) + goto out; + + used_segs =cache_info_used_segs (backend->cbdt, &latest_info->cache_info); +out: +return sprintf (buf,"%u\n" , used_segs); +} +static DEVICE_ATTR_ADMIN_RO (cache_used_segs); + +static ssize_t cache_gc_percent_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cbd_backend_device *backend; +struct cbd_backend_info *latest_info; + + backend = container_of(dev, struct cbd_backend_device, dev); + latest_info =cbdt_backend_info_read (backend->cbdt, backend->id); +if (!latest_info || latest_info->state ==CBD_BACKEND_STATE_NONE ) +return 0 ; + +return sprintf (buf,"%u\n" , latest_info->cache_info.gc_percent); +} + +static void __backend_info_write (struct cbd_backend *cbdb); +static ssize_t cache_gc_percent_store (struct device *dev, +struct device_attribute *attr, +const char *buf, +size_t size) +{ +struct cbd_backend_device *backend; +struct cbd_backend *cbdb; +unsigned long val; +int ret; + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; + + backend = container_of(dev, struct cbd_backend_device, dev); + ret =kstrtoul (buf,10 , &val); +if (ret) +return ret; + +if (val <CBD_CACHE_GC_PERCENT_MIN || + val >CBD_CACHE_GC_PERCENT_MAX ) +return -EINVAL ; + + cbdb =cbdt_get_backend (backend->cbdt, backend->id); +if (!cbdb) { +cbdt_err (backend->cbdt,"gc_percent is only allowed to set in backend node.\n" ); +return -EINVAL ; + } + +mutex_lock (&cbdb->info_lock); +if (cbdb->backend_info.cache_info.n_segs ==0 ) { +mutex_unlock (&cbdb->info_lock); +return -EINVAL ; + } + + cbdb->backend_info.cache_info.gc_percent = val; +__backend_info_write (cbdb); +mutex_unlock (&cbdb->info_lock); + +return size; +} +static DEVICE_ATTR_ADMIN_RW (cache_gc_percent); + +static void cbd_backend_hb (struct cbd_backend *cbdb) +{ +cbd_backend_info_write (cbdb); +} +CBD_OBJ_HEARTBEAT (backend); + +static struct attribute *cbd_backend_attrs[] = { + &dev_attr_path.attr, + &dev_attr_host_id.attr, + &dev_attr_alive.attr, + &dev_attr_cache_segs.attr, + &dev_attr_cache_gc_percent.attr, + &dev_attr_cache_used_segs.attr, +NULL +}; + +static struct attribute_group cbd_backend_attr_group = { + .attrs = cbd_backend_attrs, +}; + +static const struct attribute_group *cbd_backend_attr_groups[] = { + &cbd_backend_attr_group, +NULL +}; + +static void cbd_backend_release (struct device *dev) +{ +} + +const struct device_type cbd_backend_type = { + .name ="cbd_backend" , + .groups = cbd_backend_attr_groups, + .release = cbd_backend_release, +}; + +const struct device_type cbd_backends_type = { + .name ="cbd_backends" , + .release = cbd_backend_release, +}; + +int cbdb_add_handler (struct cbd_backend *cbdb,struct cbd_handler *handler) +{ +int ret =0 ; + +spin_lock (&cbdb->lock); +if (cbdb->backend_info.state ==CBD_BACKEND_STATE_STOPPING ) { + ret = -EFAULT ; + goto out; + } +hash_add (cbdb->handlers_hash, &handler->hash_node, handler->channel.seg_id); +out: +spin_unlock (&cbdb->lock); +return ret; +} + +void cbdb_del_handler (struct cbd_backend *cbdb,struct cbd_handler *handler) +{ +if (hlist_unhashed (&handler->hash_node)) +return ; + +spin_lock (&cbdb->lock); +hash_del (&handler->hash_node); +spin_unlock (&cbdb->lock); +} + +static struct cbd_handler *cbdb_get_handler (struct cbd_backend *cbdb,u32 seg_id) +{ +struct cbd_handler *handler; +bool found = false; + +spin_lock (&cbdb->lock); +hash_for_each_possible (cbdb->handlers_hash, handler, + hash_node, seg_id) { +if (handler->channel.seg_id == seg_id) { + found = true; +break ; + } + } +spin_unlock (&cbdb->lock); + +if (found) +return handler; + +return NULL ; +} + +static void destroy_handlers (struct cbd_backend *cbdb) +{ +struct cbd_handler *handler; +struct hlist_node *tmp; +int i; + +hash_for_each_safe (cbdb->handlers_hash, i, tmp, handler, hash_node) { +hash_del (&handler->hash_node); +cbd_handler_destroy (handler); + } +} + +static int create_handlers (struct cbd_backend *cbdb,bool new_backend) +{ +struct cbd_backend_info *backend_info; +u32 channel_id; +int ret; +int i; + + backend_info = &cbdb->backend_info; + +for (i =0 ; i < backend_info->n_handlers; i++) { +if (new_backend) { + ret =cbdt_get_empty_segment_id (cbdb->cbdt, &channel_id); +if (ret <0 ) { +cbdb_err (cbdb,"failed find available channel_id.\n" ); + goto destroy_handlers; + } +/* clear all channel segment before using it */ +cbd_segment_clear (cbdb->cbdt, channel_id); + backend_info->handler_channels[i] = channel_id; + }else { + channel_id = backend_info->handler_channels[i]; + } + + ret =cbd_handler_create (cbdb, channel_id, new_backend); +if (ret) { +cbdb_err (cbdb,"failed to create handler: %d\n" , ret); + goto destroy_handlers; + } + } + +return 0 ; + +destroy_handlers: +destroy_handlers (cbdb); + +return ret; +} + +static int backend_open_bdev (struct cbd_backend *cbdb,bool new_backend) +{ +int ret; + + cbdb->bdev_file =bdev_file_open_by_path (cbdb->backend_info.path, +BLK_OPEN_READ |BLK_OPEN_WRITE , cbdb,NULL ); +if (IS_ERR (cbdb->bdev_file)) { +cbdb_err (cbdb,"failed to open bdev: %d" , (int )PTR_ERR (cbdb->bdev_file)); + ret =PTR_ERR (cbdb->bdev_file); + goto err; + } + + cbdb->bdev =file_bdev (cbdb->bdev_file); + +if (new_backend) { + cbdb->backend_info.dev_size =bdev_nr_sectors (cbdb->bdev); + }else { +if (cbdb->backend_info.dev_size !=bdev_nr_sectors (cbdb->bdev)) { +cbdb_err (cbdb,"Unexpected backend size: %llu, expected: %llu\n" , +bdev_nr_sectors (cbdb->bdev), cbdb->backend_info.dev_size); + ret = -EINVAL ; + goto close_file; + } + } + +return 0 ; + +close_file: +fput (cbdb->bdev_file); +err: +return ret; +} + +static void backend_close_bdev (struct cbd_backend *cbdb) +{ +fput (cbdb->bdev_file); +} + +static int backend_cache_init (struct cbd_backend *cbdb,u32 cache_segs,bool new_backend) +{ +struct cbd_cache_opts cache_opts = {0 }; +int ret; + + cache_opts.cache_info = &cbdb->backend_info.cache_info; + cache_opts.cache_id = cbdb->backend_id; + cache_opts.owner = cbdb; + cache_opts.n_segs = cache_segs; + cache_opts.n_paral = cbdb->backend_info.n_handlers; + cache_opts.new_cache = new_backend; + cache_opts.start_writeback = true; + cache_opts.start_gc = false; + cache_opts.init_req_keys = false; + cache_opts.bdev_file = cbdb->bdev_file; + cache_opts.dev_size = cbdb->backend_info.dev_size; + +/* Allocate the cache with specified options. */ + cbdb->cbd_cache =cbd_cache_alloc (cbdb->cbdt, &cache_opts); +if (!cbdb->cbd_cache) { + ret = -ENOMEM ; + goto err; + } + +return 0 ; + +err: +return ret; +} + +static void backend_cache_destroy (struct cbd_backend *cbdb) +{ +if (cbdb->cbd_cache) +cbd_cache_destroy (cbdb->cbd_cache); +} + +static int cbd_backend_info_init (struct cbd_backend *cbdb,char *path, +u32 handlers,u32 cache_segs) +{ +struct cbd_transport *cbdt = cbdb->cbdt; +u32 backend_id; +int ret; + + ret =cbdt_get_empty_backend_id (cbdt, &backend_id); +if (ret) + goto err; + + cbdb->backend_id = backend_id; + cbdb->backend_info.meta_header.version =0 ; + cbdb->backend_info.host_id = cbdb->host_id; + cbdb->backend_info.n_handlers = handlers; + +strscpy (cbdb->backend_info.path, path,CBD_PATH_LEN ); + +cbd_cache_info_init (&cbdb->backend_info.cache_info, cache_segs); + +return 0 ; +err: +return ret; +} + +static int cbd_backend_info_load (struct cbd_backend *cbdb,u32 backend_id); +static int cbd_backend_init (struct cbd_backend *cbdb,char *path,u32 backend_id, +u32 handlers,u32 cache_segs) +{ +struct cbd_transport *cbdt = cbdb->cbdt; +bool new_backend = false; +int ret; + +if (backend_id ==U32_MAX ) + new_backend = true; + +if (new_backend) { +/* new backend */ + ret =cbd_backend_info_init (cbdb, path, handlers, cache_segs); +if (ret) + goto err; + }else { +/* attach backend, this could happen after an unexpected power off */ +cbdt_info (cbdt,"attach backend to backend_id: %u\n" , backend_id); + cbdb->backend_id = backend_id; + ret =cbd_backend_info_load (cbdb, cbdb->backend_id); +if (ret) + goto err; + } + + cbdb->backend_device = &cbdt->cbd_backends_dev->backend_devs[cbdb->backend_id]; + + ret =backend_open_bdev (cbdb, new_backend); +if (ret) + goto err; + + ret =create_handlers (cbdb, new_backend); +if (ret) + goto close_bdev; + +if (cbdb->backend_info.cache_info.n_segs) { + ret =backend_cache_init (cbdb, cbdb->backend_info.cache_info.n_segs, new_backend); +if (ret) + goto destroy_handlers; + } + + cbdb->backend_info.state =CBD_BACKEND_STATE_RUNNING ; +cbdt_add_backend (cbdt, cbdb); + +return 0 ; + +destroy_handlers: +destroy_handlers (cbdb); +close_bdev: +backend_close_bdev (cbdb); +err: +return ret; +} + +static void cbd_backend_destroy (struct cbd_backend *cbdb) +{ +struct cbd_transport *cbdt = cbdb->cbdt; + +cbdt_del_backend (cbdt, cbdb); +backend_cache_destroy (cbdb); +destroy_handlers (cbdb); +backend_close_bdev (cbdb); +} + +static void __backend_info_write (struct cbd_backend *cbdb) +{ + cbdb->backend_info.alive_ts =ktime_get_real (); +cbdt_backend_info_write (cbdb->cbdt, &cbdb->backend_info,sizeof (struct cbd_backend_info ), + cbdb->backend_id); +} + +void cbd_backend_info_write (struct cbd_backend *cbdb) +{ +mutex_lock (&cbdb->info_lock); +__backend_info_write (cbdb); +mutex_unlock (&cbdb->info_lock); +} + +static int cbd_backend_info_load (struct cbd_backend *cbdb,u32 backend_id) +{ +struct cbd_backend_info *backend_info; +int ret =0 ; + +mutex_lock (&cbdb->info_lock); + backend_info =cbdt_backend_info_read (cbdb->cbdt, backend_id); +if (!backend_info) { +cbdt_err (cbdb->cbdt,"can't read info from backend id %u.\n" , + cbdb->backend_id); + ret = -EINVAL ; + goto out; + } + +if (cbd_backend_info_is_alive (backend_info)) { +cbdt_err (cbdb->cbdt,"backend %u is alive\n" , backend_id); + ret = -EBUSY ; + goto out; + } + +if (backend_info->host_id != cbdb->host_id) { +cbdt_err (cbdb->cbdt,"backend_id: %u is on host %u but not on host %u\n" , + cbdb->backend_id, backend_info->host_id, cbdb->host_id); + ret = -EINVAL ; + goto out; + } + +memcpy (&cbdb->backend_info, backend_info,sizeof (struct cbd_backend_info )); +out: +mutex_unlock (&cbdb->info_lock); +return ret; +} + +static struct cbd_backend *cbd_backend_alloc (struct cbd_transport *cbdt) +{ +struct cbd_backend *cbdb; + + cbdb =kzalloc (sizeof (*cbdb),GFP_KERNEL ); +if (!cbdb) +return NULL ; + + cbdb->backend_io_cache =KMEM_CACHE (cbd_backend_io,0 ); +if (!cbdb->backend_io_cache) + goto free_cbdb; + + cbdb->task_wq =alloc_workqueue ("cbdt%d-b%u" ,WQ_UNBOUND |WQ_MEM_RECLAIM , +0 , cbdt->id, cbdb->backend_id); +if (!cbdb->task_wq) + goto destroy_io_cache; + + cbdb->cbdt = cbdt; + cbdb->host_id = cbdt->host->host_id; + +mutex_init (&cbdb->info_lock); +INIT_LIST_HEAD (&cbdb->node); +INIT_DELAYED_WORK (&cbdb->hb_work, backend_hb_workfn); +hash_init (cbdb->handlers_hash); +spin_lock_init (&cbdb->lock); + +return cbdb; + +destroy_io_cache: +kmem_cache_destroy (cbdb->backend_io_cache); +free_cbdb: +kfree (cbdb); +return NULL ; +} + +static void cbd_backend_free (struct cbd_backend *cbdb) +{ +drain_workqueue (cbdb->task_wq); +destroy_workqueue (cbdb->task_wq); +kmem_cache_destroy (cbdb->backend_io_cache); +kfree (cbdb); +} + +static int backend_validate (struct cbd_transport *cbdt,char *path, +u32 *backend_id,u32 handlers,u32 cache_segs) +{ +u32 host_id = cbdt->host->host_id; +int ret; + +/* Check if path starts with "/dev/" */ +if (strncmp (path,"/dev/" ,5 ) !=0 ) +return -EINVAL ; + +/* Validate backend_id */ +if (*backend_id ==U32_MAX ) { + ret =cbd_backend_find_id_by_path (cbdt, host_id, path, backend_id); +if (!ret) +cbdt_info (cbdt,"found backend_id: %u for host_id: %u, path: %s\n" , + *backend_id, host_id, path); + }else { +u32 backend_id_tmp; + +if (*backend_id !=U32_MAX && *backend_id >= cbdt->transport_info.backend_num) +return -EINVAL ; + + ret =cbd_backend_find_id_by_path (cbdt, host_id, path, &backend_id_tmp); +if (!ret && (*backend_id != backend_id_tmp)) { +cbdt_err (cbdt,"duplicated backend path: %s with backend_id: %u\n" , + path, backend_id_tmp); +return -EINVAL ; + } + } + +/* Ensure handlers count is within valid range */ +if (handlers ==0 || handlers >=CBD_HANDLERS_MAX ) +return -EINVAL ; + +/* All checks passed */ +return 0 ; +} + +int cbd_backend_start (struct cbd_transport *cbdt,char *path,u32 backend_id, +u32 handlers,u32 cache_segs) +{ +struct cbd_backend *cbdb; +int ret; + + ret =backend_validate (cbdt, path, &backend_id, handlers, cache_segs); +if (ret) +return ret; + + cbdb =cbd_backend_alloc (cbdt); +if (!cbdb) +return -ENOMEM ; + + ret =cbd_backend_init (cbdb, path, backend_id, handlers, cache_segs); +if (ret) + goto destroy_cbdb; + +cbd_backend_info_write (cbdb); +queue_delayed_work (cbd_wq, &cbdb->hb_work,CBD_HB_INTERVAL ); + +return 0 ; + +destroy_cbdb: +cbd_backend_free (cbdb); + +return ret; +} + +static bool backend_blkdevs_stopped (struct cbd_transport *cbdt,u32 backend_id) +{ +struct cbd_blkdev_info *blkdev_info; +int i; + +cbd_for_each_blkdev_info (cbdt, i, blkdev_info) { +if (!blkdev_info) +continue ; + +if (blkdev_info->state !=CBD_BLKDEV_STATE_RUNNING ) +continue ; + +if (blkdev_info->backend_id == backend_id) { +cbdt_err (cbdt,"blkdev %u is connected to backend %u\n" , + i, backend_id); +return false; + } + } + +return true; +} + +int cbd_backend_stop (struct cbd_transport *cbdt,u32 backend_id) +{ +struct cbd_backend *cbdb; + + cbdb =cbdt_get_backend (cbdt, backend_id); +if (!cbdb) +return -ENOENT ; + +if (!backend_blkdevs_stopped (cbdt, backend_id)) +return -EBUSY ; + +spin_lock (&cbdb->lock); +if (cbdb->backend_info.state ==CBD_BACKEND_STATE_STOPPING ) { +spin_unlock (&cbdb->lock); +return -EBUSY ; + } + + cbdb->backend_info.state =CBD_BACKEND_STATE_STOPPING ; +spin_unlock (&cbdb->lock); + +cancel_delayed_work_sync (&cbdb->hb_work); +cbd_backend_destroy (cbdb); +cbd_backend_free (cbdb); + +cbdt_backend_info_clear (cbdt, backend_id); + +return 0 ; +} + +static void backend_segs_clear (struct cbd_transport *cbdt,u32 backend_id) +{ +struct cbd_segment_info *seg_info; +u32 i; + +cbd_for_each_segment_info (cbdt, i, seg_info) { +if (!seg_info) +continue ; + +if (seg_info->backend_id == backend_id) +cbdt_segment_info_clear (cbdt, i); + } +} + +int cbd_backend_clear (struct cbd_transport *cbdt,u32 backend_id) +{ +struct cbd_backend_info *backend_info; + + backend_info =cbdt_backend_info_read (cbdt, backend_id); +if (!backend_info) { +cbdt_err (cbdt,"all backend_info in backend_id: %u are corrupted.\n" , backend_id); +return -EINVAL ; + } + +if (cbd_backend_info_is_alive (backend_info)) { +cbdt_err (cbdt,"backend %u is still alive\n" , backend_id); +return -EBUSY ; + } + +if (backend_info->state ==CBD_BACKEND_STATE_NONE ) +return 0 ; + +if (!backend_blkdevs_stopped (cbdt, backend_id)) +return -EBUSY ; + +backend_segs_clear (cbdt, backend_id); +cbdt_backend_info_clear (cbdt, backend_id); + +return 0 ; +} + +bool cbd_backend_cache_on (struct cbd_backend_info *backend_info) +{ +return (backend_info->cache_info.n_segs !=0 ); +} + +/** +* cbd_backend_notify - Notify the backend to handle an I/O request. +* @cbdb: Pointer to the cbd_backend structure. +* @seg_id: Segment ID associated with the request. +* +* This function is called in a single-host scenario after a block device +* sends an I/O request. It retrieves the corresponding handler for the +* given segment ID and, if the handler is ready, notifies it to proceed +* with handling the request. If the handler is not ready, the function +* returns immediately, allowing the handler to queue the handle_work +* while being created. +*/ +void cbd_backend_notify (struct cbd_backend *cbdb,u32 seg_id) +{ +struct cbd_handler *handler; + + handler =cbdb_get_handler (cbdb, seg_id); +/* +* If the handler is not ready, return directly and +* wait for the handler to queue the handle_work during creation. +*/ +if (!handler) +return ; + +cbd_handler_notify (handler); +} + +void cbd_backend_mgmt_notify (struct cbd_backend *cbdb,u32 seg_id) +{ +struct cbd_handler *handler; + + handler =cbdb_get_handler (cbdb, seg_id); +if (!handler) +return ; + +cbd_handler_mgmt_notify (handler); +}
drivers/block/cbd/cbd_backend.h new file +137 -0
137 lines not on this page · raw message
drivers/block/cbd/cbd_handler.c new file +468 -0
468 lines not on this page · raw message
drivers/block/cbd/cbd_handler.h new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v3 7/8 cbd: introduce cbd_cache
In reply to Dongsheng Yangcbd cache is a lightweight solution that uses persistent memory as block device cache. It works similar with bcache, where bcache uses block devices as cache drives, but cbd cache only supports persistent memory devices for caching. It s designed specifically for PMEM scenarios, with a simple design and implementation, aiming to provide a low-latency, high-concurrency, and performance-stable caching solution.
Note: cbd cache is not intended to replace your bcache. Instead, it offers an alternative solution specifically suited for scenarios where you want to use persistent memory devices as block device cache.
--- drivers/block/cbd/cbd_cache/cbd_cache.c | 489 ++++++++++ drivers/block/cbd/cbd_cache/cbd_cache.h | 157 +++ drivers/block/cbd/cbd_cache/cbd_cache_gc.c | 167 ++++ .../block/cbd/cbd_cache/cbd_cache_internal.h | 536 ++++++++++ drivers/block/cbd/cbd_cache/cbd_cache_key.c | 881 +++++++++++++++++ drivers/block/cbd/cbd_cache/cbd_cache_req.c | 921 ++++++++++++++++++ .../block/cbd/cbd_cache/cbd_cache_segment.c | 268 +++++ .../block/cbd/cbd_cache/cbd_cache_writeback.c | 197 ++++ 8 files changed, 3616 insertions(+) create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache.h create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_gc.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_internal.h create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_key.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_req.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_segment.c create mode 100644 drivers/block/cbd/cbd_cache/cbd_cache_writeback.c
drivers/block/cbd/cbd_cache/cbd_cache.c new file +489 -0
@@ -0,0 +1,489 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include <linux/blk_types.h> + +#include "../cbd_backend.h" +#include "cbd_cache_internal.h" + +void cbd_cache_info_init (struct cbd_cache_info *cache_info,u32 cache_segs) +{ + cache_info->n_segs = cache_segs; + cache_info->gc_percent =CBD_CACHE_GC_PERCENT_DEFAULT ; +} + +static void cache_segs_destroy (struct cbd_cache *cache) +{ +u32 i; + +if (!cache->owner) +return ; + +for (i =0 ; i < cache->n_segs; i++) +cache_seg_destroy (&cache->segments[i]); +} + +static void cache_info_set_seg_id (struct cbd_cache *cache,u32 seg_id) +{ + cache->cache_info->seg_id = seg_id; +cache_info_write (cache); +} + +/* +* get_seg_id - Retrieve the segment ID for cache initialization +* @cache: Pointer to the cache structure +* @prev_cache_seg: Pointer to the previous cache segment in the sequence +* @new_cache: Flag indicating if this is a new cache initialization +* @seg_id: Pointer to store the retrieved or allocated segment ID +* +* For a new cache, this function allocates a new segment ID, +* and links it with the previous segment in the chain. +* +* If reloading an existing cache, it retrieves the segment ID based on the +* segment chain, using the previous segment information to maintain continuity. +*/ +static int get_seg_id (struct cbd_cache *cache, +struct cbd_cache_segment *prev_cache_seg, +bool new_cache,u32 *seg_id) +{ +struct cbd_transport *cbdt = cache->cbdt; +int ret; + +if (new_cache) { + ret =cbdt_get_empty_segment_id (cbdt, seg_id); +if (ret) { +cbd_cache_err (cache,"no available segment\n" ); + goto err; + } + +if (prev_cache_seg) +cache_seg_set_next_seg (prev_cache_seg, *seg_id); +else +cache_info_set_seg_id (cache, *seg_id); + }else { +if (prev_cache_seg) { +struct cbd_segment_info *prev_seg_info; + + prev_seg_info = &prev_cache_seg->cache_seg_info.segment_info; +if (!cbd_segment_info_has_next (prev_seg_info)) { + ret = -EFAULT ; + goto err; + } + *seg_id = prev_cache_seg->cache_seg_info.segment_info.next_seg; + }else { + *seg_id = cache->cache_info->seg_id; + } + } +return 0 ; +err: +return ret; +} + +static int cache_segs_init (struct cbd_cache *cache,bool new_cache) +{ +struct cbd_cache_segment *prev_cache_seg =NULL ; +struct cbd_cache_info *cache_info = cache->cache_info; +u32 seg_id; +int ret; +u32 i; + +for (i =0 ; i < cache_info->n_segs; i++) { + ret =get_seg_id (cache, prev_cache_seg, new_cache, &seg_id); +if (ret) + goto segments_destroy; + + ret =cache_seg_init (cache, seg_id, i, new_cache); +if (ret) + goto segments_destroy; + + prev_cache_seg = &cache->segments[i]; + } +return 0 ; + +segments_destroy: +cache_segs_destroy (cache); + +return ret; +} + +static void used_segs_update_work_fn (struct work_struct *work) +{ + struct cbd_cache *cache = container_of(work, struct cbd_cache, used_segs_update_work); +struct cbd_cache_used_segs *used_segs; + + used_segs =cbd_meta_find_oldest (&cache->cache_ctrl->used_segs->header,sizeof (struct cbd_cache_used_segs )); + + used_segs->header.seq =cbd_meta_get_next_seq (&used_segs->header,sizeof (struct cbd_cache_used_segs )); + used_segs->used_segs =bitmap_weight (cache->seg_map, cache->n_segs); + used_segs->header.crc =cbd_meta_crc (&used_segs->header,sizeof (struct cbd_cache_used_segs )); + +cbdt_flush (cache->cbdt, used_segs,sizeof (struct cbd_cache_used_segs )); +} + +static struct cbd_cache *cache_alloc (struct cbd_transport *cbdt,struct cbd_cache_info *cache_info) +{ +struct cbd_cache *cache; + + cache =kvzalloc (struct_size (cache, segments, cache_info->n_segs),GFP_KERNEL ); +if (!cache) + goto err; + + cache->seg_map =bitmap_zalloc (cache_info->n_segs,GFP_KERNEL ); +if (!cache->seg_map) + goto free_cache; + + cache->req_cache =KMEM_CACHE (cbd_request,0 ); +if (!cache->req_cache) + goto free_bitmap; + + cache->cache_wq =alloc_workqueue ("cbdt%d-c%u" ,WQ_UNBOUND |WQ_MEM_RECLAIM , +0 , cbdt->id, cache->cache_id); +if (!cache->cache_wq) + goto free_req_cache; + + cache->cbdt = cbdt; + cache->cache_info = cache_info; + cache->n_segs = cache_info->n_segs; +spin_lock_init (&cache->seg_map_lock); +spin_lock_init (&cache->key_head_lock); +spin_lock_init (&cache->miss_read_reqs_lock); +INIT_LIST_HEAD (&cache->miss_read_reqs); + +mutex_init (&cache->key_tail_lock); +mutex_init (&cache->dirty_tail_lock); + +INIT_DELAYED_WORK (&cache->writeback_work, cache_writeback_fn); +INIT_DELAYED_WORK (&cache->gc_work, cbd_cache_gc_fn); +INIT_WORK (&cache->clean_work, clean_fn); +INIT_WORK (&cache->miss_read_end_work, miss_read_end_work_fn); +INIT_WORK (&cache->used_segs_update_work, used_segs_update_work_fn); + +return cache; + +free_req_cache: +kmem_cache_destroy (cache->req_cache); +free_bitmap: +bitmap_free (cache->seg_map); +free_cache: +kvfree (cache); +err: +return NULL ; +} + +static void cache_free (struct cbd_cache *cache) +{ +drain_workqueue (cache->cache_wq); +destroy_workqueue (cache->cache_wq); +kmem_cache_destroy (cache->req_cache); +bitmap_free (cache->seg_map); +kvfree (cache); +} + +static int cache_init_req_keys (struct cbd_cache *cache,u32 n_paral) +{ +u32 n_subtrees; +int ret; +u32 i; + +/* Calculate number of cache trees based on the device size */ + n_subtrees =DIV_ROUND_UP (cache->dev_size <<SECTOR_SHIFT ,CBD_CACHE_SUBTREE_SIZE ); + ret =cache_tree_init (cache, &cache->req_key_tree, n_subtrees); +if (ret) + goto err; + +/* Set the number of ksets based on n_paral, often corresponding to blkdev multiqueue count */ + cache->n_ksets = n_paral; + cache->ksets =kcalloc (cache->n_ksets,CBD_KSET_SIZE ,GFP_KERNEL ); +if (!cache->ksets) { + ret = -ENOMEM ; + goto req_tree_exit; + } + +/* +* Initialize each kset with a spinlock and delayed work for flushing. +* Each kset is associated with one queue to ensure independent handling +* of cache keys across multiple queues, maximizing multiqueue concurrency. +*/ +for (i =0 ; i < cache->n_ksets; i++) { +struct cbd_cache_kset *kset =get_kset (cache, i); + + kset->cache = cache; +spin_lock_init (&kset->kset_lock); +INIT_DELAYED_WORK (&kset->flush_work, kset_flush_fn); + } + + cache->n_heads = n_paral; + cache->data_heads =kcalloc (cache->n_heads,sizeof (struct cbd_cache_data_head ),GFP_KERNEL ); +if (!cache->data_heads) { + ret = -ENOMEM ; + goto free_kset; + } + +for (i =0 ; i < cache->n_heads; i++) { +struct cbd_cache_data_head *data_head = &cache->data_heads[i]; + +spin_lock_init (&data_head->data_head_lock); + } + +/* +* Replay persisted cache keys using cache_replay. +* This function loads and replays cache keys from previously stored +* ksets, allowing the cache to restore its state after a restart. +*/ + ret =cache_replay (cache); +if (ret) { +cbd_cache_err (cache,"failed to replay keys\n" ); + goto free_heads; + } + +return 0 ; + +free_heads: +kfree (cache->data_heads); +free_kset: +kfree (cache->ksets); +req_tree_exit: +cache_tree_exit (&cache->req_key_tree); +err: +return ret; +} + +static void cache_destroy_req_keys (struct cbd_cache *cache) +{ +u32 i; + +for (i =0 ; i < cache->n_ksets; i++) { +struct cbd_cache_kset *kset =get_kset (cache, i); + +cancel_delayed_work_sync (&kset->flush_work); + } + +kfree (cache->data_heads); +kfree (cache->ksets); +cache_tree_exit (&cache->req_key_tree); +} + +static int __cache_info_load (struct cbd_transport *cbdt, +struct cbd_cache_info *cache_info, +u32 cache_id); +static int cache_validate (struct cbd_transport *cbdt, +struct cbd_cache_opts *opts) +{ +struct cbd_cache_info *cache_info; +int ret = -EINVAL ; + +if (opts->n_paral >CBD_CACHE_PARAL_MAX ) { +cbdt_err (cbdt,"n_paral too large (max %u).\n" , +CBD_CACHE_PARAL_MAX ); + goto err; + } + +/* +* For a new cache, ensure an owner backend is specified +* and initialize cache information with the specified number of segments. +*/ +if (opts->new_cache) { +if (!opts->owner) { +cbdt_err (cbdt,"owner is needed for new cache.\n" ); + goto err; + } + +cbd_cache_info_init (opts->cache_info, opts->n_segs); + }else { +/* Load cache information from storage for existing cache */ + ret =__cache_info_load (cbdt, opts->cache_info, opts->cache_id); +if (ret) + goto err; + } + + cache_info = opts->cache_info; + +/* +* Check if the number of segments required for the specified n_paral +* exceeds the available segments in the cache. If so, report an error. +*/ +if (opts->n_paral *CBD_CACHE_SEGS_EACH_PARAL > cache_info->n_segs) { +cbdt_err (cbdt,"n_paral %u requires cache size (%llu), more than current (%llu)." , + opts->n_paral, opts->n_paral *CBD_CACHE_SEGS_EACH_PARAL * (u64 )CBDT_SEG_SIZE , + cache_info->n_segs * (u64 )CBDT_SEG_SIZE ); + goto err; + } + +if (cache_info->n_segs > cbdt->transport_info.segment_num) { +cbdt_err (cbdt,"too large cache_segs: %u, segment_num: %u\n" , + cache_info->n_segs, cbdt->transport_info.segment_num); + goto err; + } + +if (cache_info->n_segs >CBD_CACHE_SEGS_MAX ) { +cbdt_err (cbdt,"cache_segs: %u larger than CBD_CACHE_SEGS_MAX: %u\n" , + cache_info->n_segs,CBD_CACHE_SEGS_MAX ); + goto err; + } + +return 0 ; + +err: +return ret; +} + +static int cache_tail_init (struct cbd_cache *cache,bool new_cache) +{ +int ret; + +if (new_cache) { +set_bit (0 , cache->seg_map); + + cache->key_head.cache_seg = &cache->segments[0 ]; + cache->key_head.seg_off =0 ; +cache_pos_copy (&cache->key_tail, &cache->key_head); +cache_pos_copy (&cache->dirty_tail, &cache->key_head); + +cache_encode_dirty_tail (cache); +cache_encode_key_tail (cache); + }else { +if (cache_decode_key_tail (cache) ||cache_decode_dirty_tail (cache)) { +cbd_cache_err (cache,"Corrupted key tail or dirty tail.\n" ); + ret = -EIO ; + goto err; + } + } +return 0 ; +err: +return ret; +} + +struct cbd_cache *cbd_cache_alloc (struct cbd_transport *cbdt, +struct cbd_cache_opts *opts) +{ +struct cbd_cache *cache; +int ret; + + ret =cache_validate (cbdt, opts); +if (ret) +return NULL ; + + cache =cache_alloc (cbdt, opts->cache_info); +if (!cache) +return NULL ; + + cache->bdev_file = opts->bdev_file; + cache->dev_size = opts->dev_size; + cache->cache_id = opts->cache_id; + cache->owner = opts->owner; + cache->state =CBD_CACHE_STATE_RUNNING ; + + ret =cache_segs_init (cache, opts->new_cache); +if (ret) + goto free_cache; + + ret =cache_tail_init (cache, opts->new_cache); +if (ret) + goto segs_destroy; + +if (opts->init_req_keys) { + ret =cache_init_req_keys (cache, opts->n_paral); +if (ret) + goto segs_destroy; + } + +if (opts->start_writeback) { + cache->start_writeback =1 ; + ret =cache_writeback_init (cache); +if (ret) + goto destroy_keys; + } + +if (opts->start_gc) { + cache->start_gc =1 ; +queue_delayed_work (cache->cache_wq, &cache->gc_work,0 ); + } + +return cache; + +destroy_keys: +cache_destroy_req_keys (cache); +segs_destroy: +cache_segs_destroy (cache); +free_cache: +cache_free (cache); + +return NULL ; +} + +void cbd_cache_destroy (struct cbd_cache *cache) +{ + cache->state =CBD_CACHE_STATE_STOPPING ; + +flush_work (&cache->miss_read_end_work); +cache_flush (cache); + +if (cache->start_gc) { +cancel_delayed_work_sync (&cache->gc_work); +flush_work (&cache->clean_work); + } + +if (cache->start_writeback) +cache_writeback_exit (cache); + +if (cache->req_key_tree.n_subtrees) +cache_destroy_req_keys (cache); + +flush_work (&cache->used_segs_update_work); + +cache_segs_destroy (cache); +cache_free (cache); +} + +/* +* cache_info_write - Write cache information to backend +* @cache: Pointer to the cache structure +* +* This function writes the cache's metadata to the backend. Only the owner +* backend of the cache is permitted to perform this operation. It asserts +* that the cache has an owner backend. +*/ +void cache_info_write (struct cbd_cache *cache) +{ +struct cbd_backend *backend = cache->owner; + +/* Ensure only owner backend is allowed to write */ +BUG_ON (!backend); + +cbd_backend_info_write (backend); +} + +static int __cache_info_load (struct cbd_transport *cbdt, +struct cbd_cache_info *cache_info, +u32 cache_id) +{ +struct cbd_backend_info *backend_info; + + backend_info =cbdt_backend_info_read (cbdt, cache_id); +if (!backend_info) +return -ENOENT ; + +memcpy (cache_info, &backend_info->cache_info,sizeof (struct cbd_cache_info )); + +return 0 ; +} + +int cache_info_load (struct cbd_cache *cache) +{ +return __cache_info_load (cache->cbdt, cache->cache_info, cache->cache_id); +} + +u32 cache_info_used_segs (struct cbd_transport *cbdt,struct cbd_cache_info *cache_info) +{ +struct cbd_cache_used_segs *latest_used_segs; +struct cbd_cache_ctrl *cache_ctrl; +void *seg_info; + + seg_info =cbdt_get_segment_info (cbdt, cache_info->seg_id); + cache_ctrl = (seg_info +CBDT_CACHE_SEG_CTRL_OFF ); + + latest_used_segs =cbd_meta_find_latest (&cache_ctrl->used_segs->header, +sizeof (struct cbd_cache_used_segs )); +if (!latest_used_segs) +return 0 ; + +return latest_used_segs->used_segs; +}
drivers/block/cbd/cbd_cache/cbd_cache.h new file +157 -0
157 lines not on this page · raw message
drivers/block/cbd/cbd_cache/cbd_cache_gc.c new file +167 -0
167 lines not on this page · raw message
drivers/block/cbd/cbd_cache/cbd_cache_internal.h new file +536 -0
536 lines not on this page · raw message
drivers/block/cbd/cbd_cache/cbd_cache_key.c new file +881 -0
881 lines not on this page · raw message
drivers/block/cbd/cbd_cache/cbd_cache_req.c new file +921 -0
770 lines not on this page · raw message
drivers/block/cbd/cbd_cache/cbd_cache_segment.c new file +268 -0
268 lines not on this page · raw message
drivers/block/cbd/cbd_cache/cbd_cache_writeback.c new file +197 -0
197 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1
Part of the diff is not shown here. View the raw message.
PATCH v3 8/8 block: Init for CBD(CXL Block Device)
In reply to Dongsheng YangCBD (CXL Block Device) provides two usage scenarios: single-host and multi-hosts.
(1) Single-host scenario, CBD can use a pmem device as a cache for block devices, providing a caching mechanism specifically designed for persistent memory.
+-----------------------------------------------------------------+ | single-host | +-----------------------------------------------------------------+ | | | | | | | | | | | +-----------+ +------------+ | | | /dev/cbd0 | | /dev/cbd1 | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | | | | | | | | | | /dev/pmem0 | cbd0 cache| | cbd1 cache | | | | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | |+---------+| |+----------+| | | ||/dev/sda || || /dev/sdb || | | |+---------+| |+----------+| | | +-----------+ +------------+ | +-----------------------------------------------------------------+
(2) Multi-hosts scenario, CBD also provides a cache while taking advantage of shared memory features, allowing users to access block devices on other nodes across different hosts.
As shared memory is supported in CXL3.0 spec, we can transfer data via CXL shared memory. CBD use CXL shared memory to transfer data between node-1 and node-2.
This scenario require your shared memory device support Hardware-consistency as CXL 3.0 described, and CONFIG_CBD_MULTIHOST to be enabled.
--- MAINTAINERS | 7 ++ drivers/block/Kconfig | 2 + drivers/block/Makefile | 2 + drivers/block/cbd/Kconfig | 89 ++++++++++++++ drivers/block/cbd/Makefile | 14 +++ drivers/block/cbd/cbd_main.c | 230 +++++++++++++++++++++++++++++++++++ 6 files changed, 344 insertions(+) create mode 100644 drivers/block/cbd/Kconfig create mode 100644 drivers/block/cbd/Makefile create mode 100644 drivers/block/cbd/cbd_main.c
MAINTAINERS +7 -0
@@ -5198,6 +5198,13 @@ S: Odd Fixes F: Documentation/devicetree/bindings/arm/cavium-thunder2.txt F: arch/arm64/boot/dts/cavium/thunder2-99xx* +CBD (CXL Block Device) +M: Dongsheng Yang <[email protected]> +R: Gu Zheng <[email protected]> +L: [email protected] +S: Maintained +F: drivers/block/cbd/ + CBS/ETF/TAPRIO QDISCS M: Vinicius Costa Gomes <[email protected]> L: [email protected]
drivers/block/Kconfig +2 -0
@@ -219,6 +219,8 @@ config BLK_DEV_NBD If unsure, say N. +source "drivers/block/cbd/Kconfig" + config BLK_DEV_RAM tristate "RAM block device support" help
drivers/block/Makefile +2 -0
@@ -42,4 +42,6 @@ obj-$(CONFIG_BLK_DEV_NULL_BLK) += null_blk/ obj-$(CONFIG_BLK_DEV_UBLK) += ublk_drv.o +obj-$(CONFIG_BLK_DEV_CBD) += cbd/ + swim_mod-y := swim.o swim_asm.o
drivers/block/cbd/Kconfig new file +89 -0
89 lines not on this page · raw message
drivers/block/cbd/Makefile new file +14 -0
@@ -0,0 +1,14 @@ +CBD_CACHE_DIR := cbd_cache/ + +cbd-y := cbd_main.o cbd_transport.o cbd_channel.o cbd_host.o \ + cbd_backend.o cbd_handler.o cbd_blkdev.o cbd_queue.o \ + cbd_segment.o \ + $(CBD_CACHE_DIR)cbd_cache.o \ + $(CBD_CACHE_DIR)cbd_cache_key.o \ + $(CBD_CACHE_DIR)cbd_cache_segment.o \ + $(CBD_CACHE_DIR)cbd_cache_req.o \ + $(CBD_CACHE_DIR)cbd_cache_gc.o \ + $(CBD_CACHE_DIR)cbd_cache_writeback.o \ + +obj-$(CONFIG_BLK_DEV_CBD) += cbd.o +
drivers/block/cbd/cbd_main.c new file +230 -0
230 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 101 lines · Dongsheng Yang
Hi Jens,
Please help to take a look at this patchset. This is V3 for CBD (CXL Block Device).
CBD supports both single-host and multi-host scenarios, and allows the use of pmem devices as block device cache, providing low latency and high-concurrency performance.
(1) Better latency:
1 iodepth, 1 numjobs, randwrite bs=4K: cbd 7.72us vs bcache
25.30us, about 300% improvement
(2) Better iops:
1 iodepth, 32 numjobs, randwrite bs=4K: cbd 1,400K IOPS vs
bcache 210K IOPS, about 600% improvement
(3) Better stdev:
1 iodepth, 1 numjobs, randwrite bs=4K: cbd stdev=36.45 vs bcache
stdev=937.81, about 30 times improvement.V3 of the code can be found at: https://github.com/DataTravelGuide/linux.git tag cbd-v3
Changelog from V2:
- Refactored thecbd_cache.candcbd_internal.hfiles by splitting them into multiple files, making the structure clearer and increasing readability.
- Added CRC verification for all data and metadata. This means that if all CRC verification options are enabled inKconfig, all information written to pmem, including data and metadata, will have CRC checks.
- Fixed some minor bugs discovered during long-term runs of xfstests.
- Added the cbd-utils (https://github.com/DataTravelGuide/cbd-utils) project to user-space tools, providing the cbdctrl command for cbd-related management operations.You can create a cbd using the following commands:
# cbdctrl tp-reg --path /dev/pmem0 --host node0 --format --force # cbdctrl backend-start --path /dev/sda --start-dev --cache-size 1G /dev/cbd0 # cbdctrl backend-list [ { "backend_id": 0, "host_id": 0, "backend_path": "/dev/sda", "alive": true, "cache_segs": 64, "cache_gc_percent": 70, "cache_used_segs": 1, "blkdevs": [ { "blkdev_id": 0, "host_id": 0, "backend_id": 0, "dev_name": "/dev/cbd0", "alive": true } ] } ]Additional information about CBD cache:
(1) What is CBD Cache
cbd cache is a *lightweight* solution that uses persistent memory as block
device cache. It works similar with bcache, where bcache uses block devices as cache device, but cbd cache only supports persistent memory devices for caching. It accesses the cache device through DAX and is designed with features specifically for persistent memory scenarios, such as multi-cache tree structures and sync insertion of cached data.+-----------------------------------------------------------------+ | single-host | +-----------------------------------------------------------------+ | | | | | | | | | | | +-----------+ +------------+ | | | /dev/cbd0 | | /dev/cbd1 | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | | | | | | | | | | /dev/pmem0 | cbd0 cache| | cbd1 cache | | | | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | |+---------+| |+----------+| | | ||/dev/sda || || /dev/sdb || | | |+---------+| |+----------+| | | +-----------+ +------------+ | +-----------------------------------------------------------------+Note: cbd cache is not intended to replace your bcache. Instead, it offers an alternative solution specifically suited for scenarios where you want to use persistent memory devices as block device cache.
Another caching technique for accessing persistent memory using DAX is dm-writeback, but it is designed for scenarios based on device-mapper. On the other hand, cbd cache and bcache are caching solutions for block device scenarios. Therefore, I did not do a comparative analysis between cbd cache and dm-writeback.
(2) light software overhead cache write (low latency)
For cache write, handling a write request typically involves the following steps: (1) Allocating cache space -> (2) Writing data to the cache -> (3) Recording cache index metadata -> (4) Returning the result.
In cache modules using block devices as the cache (e.g., bcache), the steps of (2) writing data to the cache and (3) recording cache index metadata are asynchronous.
Quoted 7 lines
During step (2),
submit_biois issued to the cache block device, and after thebi_end_iocallback completes, a new process continues with step (3). This incurs significant overhead for persistent memory cache.However, cbd cache, which is designed for persistent memory, does not require asynchronous operations. It can directly proceed with steps (3) and (4) after completing the memcpy through DAX.
So, is this the main source of performance improvement? And I'm curious if supporting nvm as a cache layer in bcache could achieve the same effect. What's the different between cbd and bcache over nvm? Some works about bcache over nvm have been done in [1][2], they only store part of data on nvm. So can we achieve the same purpose if we make pmem as the cache layer for bcache? That means the layer is: backend block device --> pmem device (cache layer) ---> FS/application.
Thanks,
Hongbo
[1] https://lore.kernel.org/linux-block/[email protected]/T/
[2] https://patchwork.kernel.org/project/linux-block/patch/[email protected]/
Quoted 285 lines
This makes a significant difference for small IO. In the case of 4K random writes, cbd cache achieves a latency of only 7.72us (compared to 25.30us for bcache in the same test, offering a 300% improvement).
Further comparative results for various scenarios are shown in the table below.
+------------+-------------------------+--------------------------+ | numjobs=1 | randwrite | randread | | iodepth=1 +------------+------------+-------------+------------+ | (latency) | cbd cache | bcache | cbd cache | bcache | +------------+------------+------------+-------------+------------+ | bs=512 | 6.10us | 23.08us | 4.82us | 5.57us | +------------+------------+------------+-------------+------------+ | bs=1K | 6.35us | 21.68us | 5.38us | 6.05us | +------------+------------+------------+-------------+------------+ | bs=4K | 7.72us | 25.30us | 6.06us | 6.00us | +------------+------------+------------+-------------+------------+ | bs=8K | 8.92us | 27.73us | 7.24us | 7.35us | +------------+------------+------------+-------------+------------+ | bs=16K | 12.25us | 34.04us | 9.06us | 9.10us | +------------+------------+------------+-------------+------------+ | bs=32K | 16.77us | 49.24us | 14.10us | 16.18us | +------------+------------+------------+-------------+------------+ | bs=64K | 30.52us | 63.72us | 30.69us | 30.38us | +------------+------------+------------+-------------+------------+ | bs=128K | 51.66us | 114.69us | 38.47us | 39.10us | +------------+------------+------------+-------------+------------+ | bs=256K | 110.16us | 204.41us | 79.64us | 99.98us | +------------+------------+------------+-------------+------------+ | bs=512K | 205.52us | 398.70us | 122.15us | 131.97us | +------------+------------+------------+-------------+------------+ | bs=1M | 493.57us | 623.31us | 233.48us | 246.56us | +------------+------------+------------+-------------+------------+(3) multi-queue and multi cache tree (high iops)
For persistent memory, the hardware concurrency is very high. If an indexing tree is used to manage space indexing, the indexing will become a bottleneck for concurrency.
cbd cache independently manages its own indexing tree for each backend. Meanwhile, the indexing tree for the cache corresponding to each backend is divided into multiple RB trees based on the logical address space. All IO operations will find the corresponding indexing tree based on their offset. This design increases concurrency while ensuring that the depth of the indexing tree does not become too large.
Quoted 1 line
From testing, in a scenario with 32 numjobs, cbd cache achieved nearly
1,400K IOPS for 4K random write (under the same test scenario, the IOPS of bcache was around 210K, meaning CBD Cache provided an improvement of over 600%).
More detailed comparison results are as follows:
+------------+-------------------------+--------------------------+ | bs=4K | randwrite | randread | | iodepth=1 +------------+------------+-------------+------------+ | (iops) | cbd cache | bcache | cbd cache | bcache | +------------+------------+------------+-------------+------------+ | numjobs=1 | 93652 | 38055 | 154101 | 142292 | +------------+------------+------------+-------------+------------+ | numjobs=2 | 205255 | 79322 | 317143 | 221957 | +------------+------------+------------+-------------+------------+ | numjobs=4 | 430588 | 124439 | 635760 | 513443 | +------------+------------+------------+-------------+------------+ | numjobs=8 | 852865 | 160980 | 1226714 | 505911 | +------------+------------+------------+-------------+------------+ | numjobs=16| 1140952 | 226094 | 2058178 | 996146 | +------------+------------+------------+-------------+------------+ | numjobs=32| 1418989 | 214447 | 2892710 | 1361308 | +------------+------------+------------+-------------+------------+(4) better performance stablility (less stdev)
CBD Cache, through a streamlined design, simplifies and makes the IO process more controllable, which allows for stable performance output.
For example, in CBD Cache, the writeback does not need to walk through the indexing tree, meaning that the writeback process will not suffer from increased IO latency due to conflict in the indexing tree.
Quoted 1 line
From testing, under random write, CBD Cache achieves an average latency
of 6.80us, with a max latency of 2794us and a latency standard deviation of 36.45 (under the same test, Bcache has an average latency of 24.28us, but a max latency of 474,622us and a standard deviation as high as 937.81. This means that in terms of standard deviation, CBD Cache achieved approximately 30 times the improvement).
Bcache:
=================================================
write: IOPS=39.1k, BW=153MiB/s (160MB/s)(5120MiB/33479msec); 0 zone resetsslat (usec): min=4, max=157364, avg=12.47, stdev=138.93 clat (nsec): min=1168, max=474615k, avg=11808.80, stdev=927287.74 lat (usec): min=11, max=474622, avg=24.28, stdev=937.81 clat percentiles (nsec): | 1.00th=[ 1256], 5.00th=[ 1304], 10.00th=[ 1320], | 20.00th=[ 1400], 30.00th=[ 1448], 40.00th=[ 1672], | 50.00th=[ 8640], 60.00th=[ 9152], 70.00th=[ 9664], | 80.00th=[ 10048], 90.00th=[ 11328], 95.00th=[ 19072], | 99.00th=[ 27776], 99.50th=[ 36608], 99.90th=[ 173056], | 99.95th=[ 856064], 99.99th=[2039808] bw ( KiB/s): min=28032, max=214664, per=99.69%, avg=156122.03, stdev=51649.87, samples=66 iops : min= 7008, max=53666, avg=39030.53, stdev=12912.50, samples=66 lat (usec) : 2=41.55%, 4=4.59%, 10=32.70%, 20=16.37%, 50=4.45% lat (usec) : 100=0.10%, 250=0.17%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=11.93%, sys=38.61%, ctx=1311384, majf=0, minf=382 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310718,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):
WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=5120MiB (5369MB), run=33479-33479msecDisk stats (read/write):
bcache0: ios=0/1305444, sectors=0/10443552, merge=0/0,ticks=0/21789, in_queue=21789, util=65.13%, aggrios=0/0, aggsectors=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
ram0: ios=0/0, sectors=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
pmem0: ios=0/0, sectors=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%CBD cache:
==============================================
write: IOPS=133k, BW=520MiB/s (545MB/s)(5120MiB/9848msec); 0 zone
resetsslat (usec): min=3, max=2786, avg= 5.84, stdev=36.41 clat (nsec): min=852, max=132404, avg=959.09, stdev=436.60 lat (usec): min=4, max=2794, avg= 6.80, stdev=36.45 clat percentiles (nsec): | 1.00th=[ 884], 5.00th=[ 900], 10.00th=[ 908], 20.00th=[916], | 30.00th=[ 924], 40.00th=[ 924], 50.00th=[ 932], 60.00th=[940], | 70.00th=[ 948], 80.00th=[ 964], 90.00th=[ 1004], 95.00th=[1064], | 99.00th=[ 1192], 99.50th=[ 1432], 99.90th=[ 6688], 99.95th=[7712], | 99.99th=[12480] bw ( KiB/s): min=487088, max=552928, per=99.96%, avg=532154.95, stdev=18228.92, samples=19 iops : min=121772, max=138232, avg=133038.84, stdev=4557.32, samples=19 lat (nsec) : 1000=89.09% lat (usec) : 2=10.76%, 4=0.03%, 10=0.09%, 20=0.03%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01% cpu : usr=23.93%, sys=76.03%, ctx=61, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):
WRITE: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s),io=5120MiB (5369MB), run=9848-9848msec
Disk stats (read/write):
cbd0: ios=0/1280334, sectors=0/10242672, merge=0/0, ticks=0/0, in_queue=0, util=43.07%(5) no need of formating for your existing disk
As a lightweight block storage caching technology, cbd cache does not require storing metadata on backend disk. This allows users to easily add caching to existing disks without the need for any formatting operations and data migration. They can also easily stop using the cbd cache without complications, The backend disk can be used independently as a raw disk.
(6) backend device is crash-consistency
The writeback mechanism of cbd cache strictly follows a log-structured approach when writeback data. Even if dirty cache data is overwritten by new data (e.g., the old data from 0-4K is A, and new data overwrites 0-4K with B), the old data A is writeback first, followed by writeback the new data B to overwrite on the backend disk. This ensures that the backend disk maintains crash consistency. In the event of a failure of the pmem device, the data on the backend disk remains usable, though crash consistency is maintained while losing the data in the cache. This feature is particularly useful in cloud storage for disaster recovery scenarios.
It is important to note that this approach may lead to cache space utilization issues if there are many overwrite operations. However, modern file systems, such as Btrfs and F2FS, take wear leveling of the disk into account, so they tend to avoid writing repeatedly to the same area. This means that there will not be a large number of overwrite writes for the disk. Additionally, modern databases, especially those using LSM engines, rarely perform overwrite operations.
Additionally, there is an entry on the TODO list to provide a parameter backend_consistency=false to allow users to achieve better cache space utilization. That depends on how urgent the requirment is.
(7) cache space for each disk is configurable
For each backend, when enabling caching, we can specify cache space size for this backend. This is different from bcache, where all backing devices can dynamically share the cache space within a single cache device. This improves cache utilization by achieving optimal utilization through time-sharing. However, this can lead to an issue where cache behavior becomes unpredictable. In enterprise applications, it's important to have a more precise understanding of the performance of each disk. When multiple disks dynamically share the cache, the exact amount of cache each disk receives becomes uncertain. cbd cache assigns a dedicated cache space for each disk, ensuring that the cache is exclusive and not affected by others, making the cache behavior more predictable.
(8) After all, all the performance test results mentioned
above were executed using thememmap=20G!4Goption to simulate the/dev/pmem0device.Additionally, the cbd code runs the cbd-tests daily, including the xfstests suite, it passes xfstests test suite. (cbd-tests: https://github.com/DataTravelGuide/cbd-tests)
Thanx
Dongsheng Yang (8):
cbd: introducecbd_transport
cbd: introducecbd_host
cbd: introducecbd_segment
cbd: introducecbd_channel
cbd: introducecbd_blkdev
cbd: introducecbd_backend
cbd: introducecbd_cache
block: Init for CBD(CXL Block Device)MAINTAINERS | 7 + drivers/block/Kconfig | 2 + drivers/block/Makefile | 2 + drivers/block/cbd/Kconfig | 89 ++ drivers/block/cbd/Makefile | 14 + drivers/block/cbd/cbd_backend.c | 730 ++++++++++ drivers/block/cbd/cbd_backend.h | 137 ++ drivers/block/cbd/cbd_blkdev.c | 551 ++++++++ drivers/block/cbd/cbd_blkdev.h | 92 ++ drivers/block/cbd/cbd_cache/cbd_cache.c | 489 +++++++ drivers/block/cbd/cbd_cache/cbd_cache.h | 157 +++ drivers/block/cbd/cbd_cache/cbd_cache_gc.c | 167 +++ .../block/cbd/cbd_cache/cbd_cache_internal.h | 536 ++++++++ drivers/block/cbd/cbd_cache/cbd_cache_key.c | 881 ++++++++++++ drivers/block/cbd/cbd_cache/cbd_cache_req.c | 921 +++++++++++++ .../block/cbd/cbd_cache/cbd_cache_segment.c | 268 ++++ .../block/cbd/cbd_cache/cbd_cache_writeback.c | 197 +++ drivers/block/cbd/cbd_channel.c | 144 ++ drivers/block/cbd/cbd_channel.h | 429 ++++++ drivers/block/cbd/cbd_handler.c | 468 +++++++ drivers/block/cbd/cbd_handler.h | 66 + drivers/block/cbd/cbd_host.c | 227 ++++ drivers/block/cbd/cbd_host.h | 67 + drivers/block/cbd/cbd_internal.h | 482 +++++++ drivers/block/cbd/cbd_main.c | 230 ++++ drivers/block/cbd/cbd_queue.c | 516 +++++++ drivers/block/cbd/cbd_queue.h | 288 ++++ drivers/block/cbd/cbd_segment.c | 311 +++++ drivers/block/cbd/cbd_segment.h | 104 ++ drivers/block/cbd/cbd_transport.c | 1186 +++++++++++++++++ drivers/block/cbd/cbd_transport.h | 169 +++ 31 files changed, 9927 insertions(+)create mode 100644
drivers/block/cbd/Kconfig
create mode 100644drivers/block/cbd/Makefile
create mode 100644drivers/block/cbd/cbd_backend.c
create mode 100644drivers/block/cbd/cbd_backend.h
create mode 100644drivers/block/cbd/cbd_blkdev.c
create mode 100644drivers/block/cbd/cbd_blkdev.h
create mode 100644drivers/block/cbd/cbd_cache/cbd_cache.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache.hcreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_gc.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_internal.hcreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_key.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_req.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_segment.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_writeback.ccreate mode 100644drivers/block/cbd/cbd_channel.c
create mode 100644drivers/block/cbd/cbd_channel.h
create mode 100644drivers/block/cbd/cbd_handler.c
create mode 100644drivers/block/cbd/cbd_handler.h
create mode 100644drivers/block/cbd/cbd_host.c
create mode 100644drivers/block/cbd/cbd_host.h
create mode 100644drivers/block/cbd/cbd_internal.hcreate mode 100644drivers/block/cbd/cbd_main.c
create mode 100644drivers/block/cbd/cbd_queue.c
create mode 100644drivers/block/cbd/cbd_queue.h
create mode 100644drivers/block/cbd/cbd_segment.c
create mode 100644drivers/block/cbd/cbd_segment.h
create mode 100644drivers/block/cbd/cbd_transport.ccreate mode 100644drivers/block/cbd/cbd_transport.h
Quoted 108 lines · Hongbo Li
Quoted 105 lines · Dongsheng Yang
Hi Jens,
Please help to take a look at this patchset. This is V3 for CBD (CXL Block Device). CBD supports both single-host and multi-host scenarios, and allows the use of pmem devices as block device cache, providing low latency and high-concurrency performance.
(1) Better latency:
1 iodepth, 1 numjobs, randwrite bs=4K: cbd 7.72us vs bcache 25.30us, about 300% improvement
(2) Better iops:
1 iodepth, 32 numjobs, randwrite bs=4K: cbd 1,400K IOPS vs
bcache 210K IOPS, about 600% improvement
(3) Better stdev:
1 iodepth, 1 numjobs, randwrite bs=4K: cbd stdev=36.45 vs bcache stdev=937.81, about 30 times improvement.V3 of the code can be found at: https://github.com/DataTravelGuide/linux.git tag cbd-v3
Changelog from V2:
- Refactored thecbd_cache.candcbd_internal.hfiles by splitting them into multiple files, making the structure clearer and increasing readability.
- Added CRC verification for all data and metadata. This means that if all CRC verification options are enabled inKconfig, all information written to pmem, including data and metadata, will have CRC checks.
- Fixed some minor bugs discovered during long-term runs of xfstests.
- Added the cbd-utils (https://github.com/DataTravelGuide/cbd-utils) project to user-space tools, providing the cbdctrl command for cbd-related management operations.You can create a cbd using the following commands:
# cbdctrl tp-reg --path /dev/pmem0 --host node0 --format --force # cbdctrl backend-start --path /dev/sda --start-dev --cache-size 1G /dev/cbd0 # cbdctrl backend-list [ { "backend_id": 0, "host_id": 0, "backend_path": "/dev/sda", "alive": true, "cache_segs": 64, "cache_gc_percent": 70, "cache_used_segs": 1, "blkdevs": [ { "blkdev_id": 0, "host_id": 0, "backend_id": 0, "dev_name": "/dev/cbd0", "alive": true } ] } ]Additional information about CBD cache:
(1) What is CBD Cache
cbd cache is a *lightweight* solution that uses persistent memory as block device cache. It works similar with bcache, where bcache uses block devices as cache device, but cbd cache only supports persistent memory devices for caching. It accesses the cache device through DAX and is designed with features specifically for persistent memory scenarios, such as multi-cache tree structures and sync insertion of cached data.+-----------------------------------------------------------------+ | single-host | +-----------------------------------------------------------------+ | | | | | | | | | | | +-----------+ +------------+ | | | /dev/cbd0 | | /dev/cbd1 | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | | | | | | | | | | /dev/pmem0 | cbd0 cache| | cbd1 cache | | | | | | | | | | | | +---------------------|-----------|-----|------------|-------+ | | |+---------+| |+----------+| | | ||/dev/sda || || /dev/sdb || | | |+---------+| |+----------+| | | +-----------+ +------------+ | +-----------------------------------------------------------------+Note: cbd cache is not intended to replace your bcache. Instead, it offers an alternative solution specifically suited for scenarios where you want to use persistent memory devices as block device cache.
Another caching technique for accessing persistent memory using DAX is dm-writeback, but it is designed for scenarios based on device-mapper. On the other hand, cbd cache and bcache are caching solutions for block device scenarios. Therefore, I did not do a comparative analysis between cbd cache and dm-writeback.
(2) light software overhead cache write (low latency)
For cache write, handling a write request typically involves the following steps: (1) Allocating cache space -> (2) Writing data to the cache -> (3) Recording cache index metadata -> (4) Returning the result.
In cache modules using block devices as the cache (e.g., bcache), the steps of (2) writing data to the cache and (3) recording cache index metadata are asynchronous.
During step (2),
submit_biois issued to the cache block device, and after thebi_end_iocallback completes, a new process continues with step (3). This incurs significant overhead for persistent memory cache.
Quoted 5 lines
Quoted 3 lines
However, cbd cache, which is designed for persistent memory, does not require asynchronous operations. It can directly proceed with steps (3) and (4) after completing the memcpy through DAX.
So, is this the main source of performance improvement? And I'm curious if supporting nvm as a cache layer in bcache could achieve the same effect. What's the different between cbd and bcache over nvm? Some works about bcache over nvm have been done in [1][2], they only store part of data on nvm. So can we achieve the same purpose if we make pmem as the cache layer for bcache?
Hi Hongbo,
Short answer: No
In the email, I presented three aspects of performance advantages, including latency, IOPS, and standard deviation (stdev):
1. Latency: A significant portion of the latency improvement comes from minimizing NVM software overhead. As far as I know, the NVM support patch for bcache only implements the journal part. Therefore, even if that code is accepted, it cannot achieve the same level of latency. Additionally, theoretically, if bcache fully supports NVM (with data, keys, and journal all stored in NVM), it might be able to achieve similar latency levels. However, further optimization of the I/O path would be required, including converting many current asynchronous operations to synchronous ones. I dont think that's easy to be done. 2. IOPS: The IOPS advantage is due to the multi-tree design in cbd_cache, achieving 1400K IOPS compared to bcache’s 210K IOPS. This cannot be addressed merely by enabling NVM support in bcache.
3. Stdev: In terms of standard deviation, cbd_cache was designed to avoid latency long-tail issues as much as possible. This involves mechanisms for space allocation and GC, which also cannot be resolved by adding NVM support to bcache.
In summary, cbd_cache is not simply “bcache with NVM support.” It is a new solution specifically designed for using PMEM as block storage cache.
Thanx
Quoted 296 lines
That means the layer is: backend block device --> pmem device (cache layer) ---> FS/application.
Thanks,
Hongbo[1] https://lore.kernel.org/linux-block/[email protected]/T/
[2] https://patchwork.kernel.org/project/linux-block/patch/[email protected]/
Quoted 287 lines
This makes a significant difference for small IO. In the case of 4K random writes, cbd cache achieves a latency of only 7.72us (compared to 25.30us for bcache in the same test, offering a 300% improvement).
Further comparative results for various scenarios are shown in the table below.
+------------+-------------------------+--------------------------+ | numjobs=1 | randwrite | randread | | iodepth=1 +------------+------------+-------------+------------+ | (latency) | cbd cache | bcache | cbd cache | bcache | +------------+------------+------------+-------------+------------+ | bs=512 | 6.10us | 23.08us | 4.82us | 5.57us | +------------+------------+------------+-------------+------------+ | bs=1K | 6.35us | 21.68us | 5.38us | 6.05us | +------------+------------+------------+-------------+------------+ | bs=4K | 7.72us | 25.30us | 6.06us | 6.00us | +------------+------------+------------+-------------+------------+ | bs=8K | 8.92us | 27.73us | 7.24us | 7.35us | +------------+------------+------------+-------------+------------+ | bs=16K | 12.25us | 34.04us | 9.06us | 9.10us | +------------+------------+------------+-------------+------------+ | bs=32K | 16.77us | 49.24us | 14.10us | 16.18us | +------------+------------+------------+-------------+------------+ | bs=64K | 30.52us | 63.72us | 30.69us | 30.38us | +------------+------------+------------+-------------+------------+ | bs=128K | 51.66us | 114.69us | 38.47us | 39.10us | +------------+------------+------------+-------------+------------+ | bs=256K | 110.16us | 204.41us | 79.64us | 99.98us | +------------+------------+------------+-------------+------------+ | bs=512K | 205.52us | 398.70us | 122.15us | 131.97us | +------------+------------+------------+-------------+------------+ | bs=1M | 493.57us | 623.31us | 233.48us | 246.56us | +------------+------------+------------+-------------+------------+(3) multi-queue and multi cache tree (high iops)
For persistent memory, the hardware concurrency is very high. If an indexing tree is used to manage space indexing, the indexing will become a bottleneck for concurrency.
cbd cache independently manages its own indexing tree for each backend. Meanwhile, the indexing tree for the cache corresponding to each backend is divided into multiple RB trees based on the logical address space. All IO operations will find the corresponding indexing tree based on their offset. This design increases concurrency while ensuring that the depth of the indexing tree does not become too large.
Quoted 1 line
From testing, in a scenario with 32 numjobs, cbd cache achieved nearly
1,400K IOPS for 4K random write (under the same test scenario, the IOPS of bcache was around 210K, meaning CBD Cache provided an improvement of over 600%).
More detailed comparison results are as follows:
+------------+-------------------------+--------------------------+ | bs=4K | randwrite | randread | | iodepth=1 +------------+------------+-------------+------------+ | (iops) | cbd cache | bcache | cbd cache | bcache | +------------+------------+------------+-------------+------------+ | numjobs=1 | 93652 | 38055 | 154101 | 142292 | +------------+------------+------------+-------------+------------+ | numjobs=2 | 205255 | 79322 | 317143 | 221957 | +------------+------------+------------+-------------+------------+ | numjobs=4 | 430588 | 124439 | 635760 | 513443 | +------------+------------+------------+-------------+------------+ | numjobs=8 | 852865 | 160980 | 1226714 | 505911 | +------------+------------+------------+-------------+------------+ | numjobs=16| 1140952 | 226094 | 2058178 | 996146 | +------------+------------+------------+-------------+------------+ | numjobs=32| 1418989 | 214447 | 2892710 | 1361308 | +------------+------------+------------+-------------+------------+(4) better performance stablility (less stdev)
CBD Cache, through a streamlined design, simplifies and makes the IO process more controllable, which allows for stable performance output.
For example, in CBD Cache, the writeback does not need to walk through the indexing tree, meaning that the writeback process will not suffer from increased IO latency due to conflict in the indexing tree.
Quoted 1 line
From testing, under random write, CBD Cache achieves an average latency
of 6.80us, with a max latency of 2794us and a latency standard deviation of 36.45 (under the same test, Bcache has an average latency of 24.28us, but a max latency of 474,622us and a standard deviation as high as 937.81. This means that in terms of standard deviation, CBD Cache achieved approximately 30 times the improvement).
Bcache:
=================================================
write: IOPS=39.1k, BW=153MiB/s (160MB/s)(5120MiB/33479msec); 0 zone resets
slat (usec): min=4, max=157364, avg=12.47, stdev=138.93 clat (nsec): min=1168, max=474615k, avg=11808.80, stdev=927287.74 lat (usec): min=11, max=474622, avg=24.28, stdev=937.81 clat percentiles (nsec):| 1.00th=[ 1256], 5.00th=[ 1304], 10.00th=[ 1320], | 20.00th=[ 1400], 30.00th=[ 1448], 40.00th=[ 1672], | 50.00th=[ 8640], 60.00th=[ 9152], 70.00th=[ 9664], | 80.00th=[ 10048], 90.00th=[ 11328], 95.00th=[ 19072], | 99.00th=[ 27776], 99.50th=[ 36608], 99.90th=[ 173056], | 99.95th=[ 856064], 99.99th=[2039808]bw ( KiB/s): min=28032, max=214664, per=99.69%, avg=156122.03, stdev=51649.87, samples=66 iops : min= 7008, max=53666, avg=39030.53, stdev=12912.50, samples=66 lat (usec) : 2=41.55%, 4=4.59%, 10=32.70%, 20=16.37%, 50=4.45%
lat (usec) : 100=0.10%, 250=0.17%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=11.93%, sys=38.61%, ctx=1311384, majf=0, minf=382
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,Quoted 1 line
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310718,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=5120MiB (5369MB), run=33479-33479msecDisk stats (read/write):
bcache0: ios=0/1305444, sectors=0/10443552, merge=0/0, ticks=0/21789, in_queue=21789, util=65.13%, aggrios=0/0, aggsectors=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
ram0: ios=0/0, sectors=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
pmem0: ios=0/0, sectors=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%CBD cache:
==============================================
write: IOPS=133k, BW=520MiB/s (545MB/s)(5120MiB/9848msec); 0 zone resets
slat (usec): min=3, max=2786, avg= 5.84, stdev=36.41 clat (nsec): min=852, max=132404, avg=959.09, stdev=436.60 lat (usec): min=4, max=2794, avg= 6.80, stdev=36.45 clat percentiles (nsec):| 1.00th=[ 884], 5.00th=[ 900], 10.00th=[ 908], 20.00th=[916], | 30.00th=[ 924], 40.00th=[ 924], 50.00th=[ 932], 60.00th=[940], | 70.00th=[ 948], 80.00th=[ 964], 90.00th=[ 1004], 95.00th=[1064], | 99.00th=[ 1192], 99.50th=[ 1432], 99.90th=[ 6688], 99.95th=[7712], | 99.99th=[12480]bw ( KiB/s): min=487088, max=552928, per=99.96%, avg=532154.95, stdev=18228.92, samples=19 iops : min=121772, max=138232, avg=133038.84, stdev=4557.32, samples=19 lat (nsec) : 1000=89.09%
lat (usec) : 2=10.76%, 4=0.03%, 10=0.09%, 20=0.03%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=23.93%, sys=76.03%, ctx=61, majf=0, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,Quoted 1 line
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s), io=5120MiB (5369MB), run=9848-9848msecDisk stats (read/write):
cbd0: ios=0/1280334, sectors=0/10242672, merge=0/0, ticks=0/0, in_queue=0, util=43.07%(5) no need of formating for your existing disk
As a lightweight block storage caching technology, cbd cache does not require storing metadata on backend disk. This allows users to easily add caching to existing disks without the need for any formatting operations and data migration. They can also easily stop using the cbd cache without complications, The backend disk can be used independently as a raw disk.
(6) backend device is crash-consistency
The writeback mechanism of cbd cache strictly follows a log-structured approach when writeback data. Even if dirty cache data is overwritten by new data (e.g., the old data from 0-4K is A, and new data overwrites 0-4K with B), the old data A is writeback first, followed by writeback the new data B to overwrite on the backend disk. This ensures that the backend disk maintains crash consistency. In the event of a failure of the pmem device, the data on the backend disk remains usable, though crash consistency is maintained while losing the data in the cache. This feature is particularly useful in cloud storage for disaster recovery scenarios.
It is important to note that this approach may lead to cache space utilization issues if there are many overwrite operations. However, modern file systems, such as Btrfs and F2FS, take wear leveling of the disk into account, so they tend to avoid writing repeatedly to the same area. This means that there will not be a large number of overwrite writes for the disk. Additionally, modern databases, especially those using LSM engines, rarely perform overwrite operations.
Additionally, there is an entry on the TODO list to provide a parameter backend_consistency=false to allow users to achieve better cache space utilization. That depends on how urgent the requirment is.
(7) cache space for each disk is configurable
For each backend, when enabling caching, we can specify cache space size for this backend. This is different from bcache, where all backing devices can dynamically share the cache space within a single cache device. This improves cache utilization by achieving optimal utilization through time-sharing. However, this can lead to an issue where cache behavior becomes unpredictable. In enterprise applications, it's important to have a more precise understanding of the performance of each disk. When multiple disks dynamically share the cache, the exact amount of cache each disk receives becomes uncertain. cbd cache assigns a dedicated cache space for each disk, ensuring that the cache is exclusive and not affected by others, making the cache behavior more predictable.
(8) After all, all the performance test results mentioned
above were executed using thememmap=20G!4Goption to simulate the/dev/pmem0device.Additionally, the cbd code runs the cbd-tests daily, including the xfstests suite, it passes xfstests test suite. (cbd-tests: https://github.com/DataTravelGuide/cbd-tests)
Thanx
Dongsheng Yang (8):
cbd: introducecbd_transport
cbd: introducecbd_host
cbd: introducecbd_segment
cbd: introducecbd_channel
cbd: introducecbd_blkdev
cbd: introducecbd_backend
cbd: introducecbd_cache
block: Init for CBD(CXL Block Device)MAINTAINERS | 7 + drivers/block/Kconfig | 2 + drivers/block/Makefile | 2 + drivers/block/cbd/Kconfig | 89 ++ drivers/block/cbd/Makefile | 14 + drivers/block/cbd/cbd_backend.c | 730 ++++++++++ drivers/block/cbd/cbd_backend.h | 137 ++ drivers/block/cbd/cbd_blkdev.c | 551 ++++++++ drivers/block/cbd/cbd_blkdev.h | 92 ++ drivers/block/cbd/cbd_cache/cbd_cache.c | 489 +++++++ drivers/block/cbd/cbd_cache/cbd_cache.h | 157 +++ drivers/block/cbd/cbd_cache/cbd_cache_gc.c | 167 +++ .../block/cbd/cbd_cache/cbd_cache_internal.h | 536 ++++++++
drivers/block/cbd/cbd_cache/cbd_cache_key.c | 881 ++++++++++++ drivers/block/cbd/cbd_cache/cbd_cache_req.c | 921 +++++++++++++ .../block/cbd/cbd_cache/cbd_cache_segment.c | 268 ++++ .../block/cbd/cbd_cache/cbd_cache_writeback.c | 197 +++ drivers/block/cbd/cbd_channel.c | 144 ++ drivers/block/cbd/cbd_channel.h | 429 ++++++ drivers/block/cbd/cbd_handler.c | 468 +++++++ drivers/block/cbd/cbd_handler.h | 66 +drivers/block/cbd/cbd_host.c | 227 ++++ drivers/block/cbd/cbd_host.h | 67 + drivers/block/cbd/cbd_internal.h | 482 +++++++ drivers/block/cbd/cbd_main.c | 230 ++++ drivers/block/cbd/cbd_queue.c | 516 +++++++ drivers/block/cbd/cbd_queue.h | 288 ++++ drivers/block/cbd/cbd_segment.c | 311 +++++ drivers/block/cbd/cbd_segment.h | 104 ++ drivers/block/cbd/cbd_transport.c | 1186 +++++++++++++++++ drivers/block/cbd/cbd_transport.h | 169 +++
31 files changed, 9927 insertions(+)create mode 100644
drivers/block/cbd/Kconfig
create mode 100644drivers/block/cbd/Makefile
create mode 100644drivers/block/cbd/cbd_backend.c
create mode 100644drivers/block/cbd/cbd_backend.h
create mode 100644drivers/block/cbd/cbd_blkdev.c
create mode 100644drivers/block/cbd/cbd_blkdev.h
create mode 100644drivers/block/cbd/cbd_cache/cbd_cache.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache.hcreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_gc.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_internal.hcreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_key.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_req.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_segment.ccreate mode 100644drivers/block/cbd/cbd_cache/cbd_cache_writeback.ccreate mode 100644drivers/block/cbd/cbd_channel.c
create mode 100644drivers/block/cbd/cbd_channel.h
create mode 100644drivers/block/cbd/cbd_handler.c
create mode 100644drivers/block/cbd/cbd_handler.h
create mode 100644drivers/block/cbd/cbd_host.c
create mode 100644drivers/block/cbd/cbd_host.h
create mode 100644drivers/block/cbd/cbd_internal.hcreate mode 100644drivers/block/cbd/cbd_main.c
create mode 100644drivers/block/cbd/cbd_queue.c
create mode 100644drivers/block/cbd/cbd_queue.h
create mode 100644drivers/block/cbd/cbd_segment.c
create mode 100644drivers/block/cbd/cbd_segment.h
create mode 100644drivers/block/cbd/cbd_transport.ccreate mode 100644drivers/block/cbd/cbd_transport.h