dm: Introduce dm-qcow2 driver to attach QCOW2 files as block device
Subsystems (3)
cover dm: Introduce dm-qcow2 driver to attach QCOW2 files as block device
This patchset adds a new driver allowing to attach QCOW2 files as block devices. Its idea is to implement in kernel only that features, which affect runtime IO performance (IO requests processing functionality). The maintenance operations are synchronously processed in userspace, while device is suspended.
Userspace is allowed to do only that operations, which never modifies virtual disk's data. It is only allowed to modify QCOW2 file metadata providing that disk's data. The examples of allowed operations is snapshot creation and resize.
Userspace part is handled by already existing utils (qemu-img).
For instance, snapshot creation on attached dm-qcow2 device looks like:
# dmsetup suspend $device # qemu-img snapshot -c <snapshot_name> $device.qcow2 # dmsetup resume $device
1)Suspend flushes all pending IO and related metadata to file,
leaving the file in consistent QCOW2 format.
Driver .postsuspend throws out all images's cached metadata.
2)qemu-img creates snapshot: changes/moves metadata inside QCOW2 file. 3)Driver .preresume reads new version of metadata
from file (1 page is required), and the device is ready to continue handling of IO requests.
This example shows the way of device-mapper infrastructure allows to implement drivers following the idea of kernel/userspace components demarcation. Thus, the driver uses advantages of device-mapper instead of implementing its own suspend/resume engine.
The below fio test was used to measure performance:
# fio --name=test --ioengine=libaio --direct=1 --bs=$bs --filename=$dev
--readwrite=$rw --runtime=60 --numjobs=2 --iodepth=8The collected results consists of the both: fio measurement and system load taken from /proc/loadavg. Since loadavg min period is 60 seconds, fio's runtime is 60 too.
Here is average results of 5 runs (IO/loadavg is also average of IO/loadavg of 5 runs):
-------------------------------------------+---------------------------------+--------------------------+
qemu-nbd (native aio) | dm-qcow2 | diff, % |
----+--------------------------------------+---------------------------------+--------------------------+
bs | RW | IO,MiB/s loadavg IO/loadavg| IO,MiB/s loadavg IO/loadavg|IO loadavg IO/loadavg| ------------|------------------------------+---------------------------------+--------------------------+ 4K | READ | 279 1.986 147 | 512 2.088 248 |+83.7 +5.1 +68.4 | 4K | WRITE | 242 2.31 105 | 770 2.172 357 |+217.9 -5.9 +239.7 | ----+-------|------------------------------+---------------------------------+--------------------------+ 64K | READ | 1199 1.794 691 | 1218 1.118 1217 |+1.6 -37.7 +76 | 64K | WRITE | 946 1.084 877 | 1003 0.466 2144 |+6.1 -57 +144.5 | ------------|------------------------------+---------------------------------+--------------------------+ 512K| READ | 1741 1.142 1526 | 2196 0.546 4197 |+26.1 -52.2 +175.1 | 512K| WRITE | 1016 1.084 941 | 993 0.306 3267 |-2.2 -71.7 +246.9 | ----|-------|------------------------------+---------------------------------+--------------------------+ 1M | READ | 1793 1.174 1542 | 2373 0.566 4384 |+32.4 -51.8 +184.2 | 1M | WRITE | 1037 0.894 1165 | 1068 0.892 1196 |+2.9 -0.2 +2.7 | ----|-------+------------------------------+---------------------------------+--------------------------+ 2M | READ | 1784 1.084 1654 | 2431 0.788 3090 |+36.3 -27.3 +86.8 | 2M | WRITE | 1027 0.878 1172 | 1063 0.878 1212 |+3.6 0 +3.4 |
----+-------+------------------------------+---------------------------------+--------------------------+
(NBD attaching string: qemu-nbd -c $dev --aio=native --nocache file.qcow2)
As in diff column, dm-qcow2 driver has the best throughput (the only exception is 512K WRITE), and the smallest loadavg (the only exception is 4K READ). The density of dm-qcow2 is significantly better.
(Note, that tests are made on preallocated images, when
all L2 table is allocated, since QEMU has lazy L2 allocation feature, which is not implemented in dm-qcow2 yet).
So, one of the reasons of implementing the driver is providing better performance and density than it's done in qemu-nbd. The second reason is a possibility to unify virtual disks format for VMs and containers, so a disk image can be used to start both of them.
This patchset consists of 4 patches. Patches [1-2] make small changes in dm code: [1] exports a function, while [2] makes .io_hints be called for drivers not having .iterate_devices. Patch [3] adds dm-qcow2, while patch [4] adds a userspace wrapper for attaching such the devices.
---
Kirill Tkhai (4):
dm: Export dm_complete_request()
dm: Process .io_hints for drivers not having underlying devices
dm-qcow2: Introduce driver to create block devices over QCOW2 files
dm-qcow2: Add helper for working with dm-qcow2 devices
drivers/md/Kconfig | 17 + drivers/md/Makefile | 2 + drivers/md/dm-qcow2-cmd.c | 383 +++ drivers/md/dm-qcow2-map.c | 4256 ++++++++++++++++++++++++++++++++++ drivers/md/dm-qcow2-target.c | 1026 ++++++++ drivers/md/dm-qcow2.h | 368 +++ drivers/md/dm-rq.c | 3 +- drivers/md/dm-rq.h | 2 + drivers/md/dm-table.c | 5 +- scripts/qcow2-dm.sh | 249 ++ 10 files changed, 6309 insertions(+), 2 deletions(-) create mode 100644 drivers/md/dm-qcow2-cmd.c create mode 100644 drivers/md/dm-qcow2-map.c create mode 100644 drivers/md/dm-qcow2-target.c create mode 100644 drivers/md/dm-qcow2.h create mode 100755 scripts/qcow2-dm.sh
--
PATCH 1/4 dm: Export dm_complete_request()
This function is required for dm-qcow2 driver going in next patches. The driver transforms block requests into file operations on underlining file in QCOW2 format (like loop but over QCOW2 instead of RAW file). We need to have a possibility to complete a request after corresponding file operations are finished.
--- drivers/md/dm-rq.c | 3 ++- drivers/md/dm-rq.h | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/md/dm-rq.c +2 -1
@@ -275,7 +275,7 @@ static void dm_softirq_done(struct request *rq)* Complete the clone and the original request with the error status * through softirq context. */ -static void dm_complete_request (struct request *rq,blk_status_t error) +void dm_complete_request (struct request *rq,blk_status_t error) {struct dm_rq_target_io *tio =tio_from_request (rq);
@@ -283,6 +283,7 @@ static void dm_complete_request(struct request *rq, blk_status_t error)if (likely (!blk_should_fake_timeout (rq->q)))blk_mq_complete_request (rq); } +EXPORT_SYMBOL (dm_complete_request);/* * Complete the not-mapped clone and the original request with the error status
drivers/md/dm-rq.h +2 -0
@@ -44,4 +44,6 @@ ssize_t dm_attr_rq_based_seq_io_merge_deadline_show(struct mapped_device *md, chssize_t dm_attr_rq_based_seq_io_merge_deadline_store (struct mapped_device *md,const char *buf,size_t count); +void dm_complete_request (struct request *rq,blk_status_t error); +#endif
PATCH 2/4 dm: Process .io_hints for drivers not having underlying devices
In reply to Kirill TkhaiThis makes dm_calculate_queue_limits() sets the hints also for drivers, which do not have underlying devices.
This is useful for dm-qcow2 driver, that wants to have incomming requests are likely splitted by QCOW2 cluster size alignment.
--- drivers/md/dm-table.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/md/dm-table.c +4 -1
@@ -1660,8 +1660,11 @@ int dm_calculate_queue_limits(struct dm_table *table, ti =dm_table_get_target (table, i); -if (!ti->type->iterate_devices) +if (!ti->type->iterate_devices) { +if (ti->type->io_hints) + ti->type->io_hints (ti, &ti_limits); goto combine_limits; + }/* * Combine queue limits of all the devices this target uses.
PATCH 3/4 dm-qcow2: Introduce driver to create block devices over QCOW2 files
In reply to Kirill TkhaiDriver for attaching QCOW2 files as block devices. It cares about performance-critical actions like actual IO and snapshot COW and merge, while complex and fast metadata-related service actions (e.g., snapshot creation and resize) are delegated to userspace.
Suspend/resume is a barrier between the driver and userspace. On suspend the driver brings QCOW2 images in consistent state, while on resume it re-reads metadata from QCOW2 images. Userspace proceeds metadata-related service actions while device is suspended, and it never does this on running device. This demarcation allows driver code to be pretty small.
Implemented full support for QCOW2 format: compressed clusters, internal snapshots, backing files and extended L2 table. Current limitations: the lowest backing file must be QCOW2 (RAW backing files are not supported), cluster and subcluster sizes for all backing files in chain must be equal (I'm going to change this in the future).
Also, implemented backward merge (from top image to its backing file).
The driver is request based, since this allows to use blk-mq merging of request. Driver splits requests itself, and every request (i.e., qio) after splitting fits a single cluster. (In some cases it is worth to create bigger splits, and this is a subject of further optimizations).
--- drivers/md/Kconfig | 17 drivers/md/Makefile | 2 drivers/md/dm-qcow2-cmd.c | 383 ++++ drivers/md/dm-qcow2-map.c | 4256 ++++++++++++++++++++++++++++++++++++++++++ drivers/md/dm-qcow2-target.c | 1026 ++++++++++ drivers/md/dm-qcow2.h | 368 ++++ 6 files changed, 6052 insertions(+) create mode 100644 drivers/md/dm-qcow2-cmd.c create mode 100644 drivers/md/dm-qcow2-map.c create mode 100644 drivers/md/dm-qcow2-target.c create mode 100644 drivers/md/dm-qcow2.h
drivers/md/Kconfig +17 -0
@@ -652,4 +652,21 @@ config DM_AUDIT Enables audit logging of several security relevant events in the particular device-mapper targets, especially the integrity target. +config DM_QCOW2 + tristate "QCOW2 target support" + depends on BLK_DEV_DM + depends on ZLIB_INFLATE + help + Driver for attaching QCOW2 files as block devices. It cares + about performance-critical actions like actual IO and + snapshot COW and merge, while complex and fast metadata-related + service actions (e.g., snapshot creation and resize) + are delegated to userspace. + + Suspend/resume is a barrier between the driver and userspace. + On suspend the driver brings QCOW2 images in consistent state, + while on resume it re-reads metadata from QCOW2 images. Thus, + userspace proceeds metadata-related service actions, while + device is suspended. + endif # MD
drivers/md/Makefile +2 -0
@@ -26,6 +26,7 @@ dm-era-y += dm-era-target.o dm-clone-y += dm-clone-target.o dm-clone-metadata.o dm-verity-y += dm-verity-target.o dm-zoned-y += dm-zoned-target.o dm-zoned-metadata.o dm-zoned-reclaim.o +dm-qcow2-y += dm-qcow2-target.o dm-qcow2-map.o dm-qcow2-cmd.o md-mod-y += md.o md-bitmap.o raid456-y += raid5.o raid5-cache.o raid5-ppl.o
@@ -83,6 +84,7 @@ obj-$(CONFIG_DM_LOG_WRITES) += dm-log-writes.o obj-$(CONFIG_DM_INTEGRITY) += dm-integrity.o obj-$(CONFIG_DM_ZONED) += dm-zoned.o obj-$(CONFIG_DM_WRITECACHE) += dm-writecache.o +obj-$(CONFIG_DM_QCOW2) += dm-qcow2.o ifeq ($(CONFIG_DM_INIT),y) dm-mod-objs += dm-init.o
drivers/md/dm-qcow2-cmd.c new file +383 -0
383 lines not on this page · raw message
drivers/md/dm-qcow2-map.c new file +4256 -0
2,582 lines not on this page · raw message
drivers/md/dm-qcow2-target.c new file +1026 -0
1,026 lines not on this page · raw message
drivers/md/dm-qcow2.h new file +368 -0
368 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
kernel test robot
build botRe: [PATCH 3/4] dm-qcow2: Introduce driver to create block devices over QCOW2 files
Thank you for the patch! Yet something to improve:
Hi Kirill,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on device-mapper-dm/for-next]
[also build test ERROR on song-md/md-next linus/master v5.17 next-20220328] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Kirill-Tkhai/dm-Introduce-dm-qcow2-driver-to-attach-QCOW2-files-as-block-device/20220328-192031 base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next config: mips-allyesconfig (https://download.01.org/0day-ci/archive/20220329/[email protected]/config) compiler: mips-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/c41a15c11909af3588885a88f6622c36dcc9ca35
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Kirill-Tkhai/dm-Introduce-dm-qcow2-driver-to-attach-QCOW2-files-as-block-device/20220328-192031 git checkout c41a15c11909af3588885a88f6622c36dcc9ca35
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=mips SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
>> drivers/md/dm-qcow2-target.c:428:6: warning: no previous prototype for 'ploop_enospc_timer' [-Wmissing-prototypes]
428 | void ploop_enospc_timer(struct timer_list *timer)
| ^~~~~~~~~~~~~~~~~~
--
drivers/md/dm-qcow2-map.c: In function 'process_cow_indexes_write':
>> drivers/md/dm-qcow2-map.c:3861:28: warning: variable 'qvec' set but not used [-Wunused-but-set-variable]
3861 | struct qcow2_bvec *qvec;
| ^~~~
drivers/md/dm-qcow2-map.c: At top level:
>> drivers/md/dm-qcow2-map.c:4136:6: warning: no previous prototype for 'submit_embedded_qio' [-Wmissing-prototypes] 4136 | void submit_embedded_qio(struct qcow2_target *tgt, struct qio *qio)
| ^~~~~~~~~~~~~~~~~~~
--
arch/mips/kernel/head.o: in function `kernel_entry':
(.ref.text+0xac): relocation truncated to fit: R_MIPS_26 against start_kernel' init/main.o: in function set_reset_devices':
main.c:(.init.text+0x20): relocation truncated to fit: R_MIPS_26 against _mcount' main.c:(.init.text+0x30): relocation truncated to fit: R_MIPS_26 against __sanitizer_cov_trace_pc' init/main.o: in function `debug_kernel':
main.c:(.init.text+0xa4): relocation truncated to fit: R_MIPS_26 against _mcount' main.c:(.init.text+0xb4): relocation truncated to fit: R_MIPS_26 against __sanitizer_cov_trace_pc' init/main.o: in function `quiet_kernel':
main.c:(.init.text+0x128): relocation truncated to fit: R_MIPS_26 against _mcount' main.c:(.init.text+0x138): relocation truncated to fit: R_MIPS_26 against __sanitizer_cov_trace_pc' init/main.o: in function `warn_bootconfig':
main.c:(.init.text+0x1ac): relocation truncated to fit: R_MIPS_26 against _mcount' main.c:(.init.text+0x1bc): relocation truncated to fit: R_MIPS_26 against __sanitizer_cov_trace_pc' init/main.o: in function `init_setup':
main.c:(.init.text+0x238): relocation truncated to fit: R_MIPS_26 against `_mcount' main.c:(.init.text+0x258): additional relocation overflows omitted from the output
mips-linux-ld: drivers/md/dm-qcow2-target.o: in function `calc_cached_parameters':
>> dm-qcow2-target.c:(.text.calc_cached_parameters+0x194): undefined reference to `__divdi3' >> mips-linux-ld: dm-qcow2-target.c:(.text.calc_cached_parameters+0x264): undefined reference to `__divdi3' mips-linux-ld: drivers/md/dm-qcow2-map.o: in function `qio_subclu_indexes': >> dm-qcow2-map.c:(.text.qio_subclu_indexes+0x21c): undefined reference to `__udivdi3' >> mips-linux-ld: dm-qcow2-map.c:(.text.qio_subclu_indexes+0x298): undefined reference to `__udivdi3' mips-linux-ld: drivers/md/dm-qcow2-map.o: in function `calc_refcounters_map': >> dm-qcow2-map.c:(.text.calc_refcounters_map+0x118): undefined reference to `__divdi3' >> mips-linux-ld: dm-qcow2-map.c:(.text.calc_refcounters_map+0x184): undefined reference to `__moddi3' >> mips-linux-ld: dm-qcow2-map.c:(.text.calc_refcounters_map+0x23c): undefined reference to `__divdi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_refcounters_map+0x46c): undefined reference to `__moddi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_refcounters_map+0x4a8): undefined reference to `__divdi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_refcounters_map+0x4f0): undefined reference to `__moddi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_refcounters_map+0x574): undefined reference to `__divdi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_refcounters_map+0x598): undefined reference to `__moddi3' mips-linux-ld: drivers/md/dm-qcow2-map.o: in function `calc_cluster_map': >> dm-qcow2-map.c:(.text.calc_cluster_map+0x178): undefined reference to `__divdi3' >> mips-linux-ld: dm-qcow2-map.c:(.text.calc_cluster_map+0x198): undefined reference to `__divdi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_cluster_map+0x3d8): undefined reference to `__divdi3' >> mips-linux-ld: dm-qcow2-map.c:(.text.calc_cluster_map+0x5a8): undefined reference to `__moddi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_cluster_map+0x5d0): undefined reference to `__moddi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_cluster_map+0x650): undefined reference to `__moddi3' mips-linux-ld: dm-qcow2-map.c:(.text.calc_cluster_map+0x6bc): undefined reference to `__moddi3' mips-linux-ld: drivers/md/dm-qcow2-map.o: in function `split_qio_to_list': >> dm-qcow2-map.c:(.text.split_qio_to_list+0x228): undefined reference to `__divdi3' >> mips-linux-ld: dm-qcow2-map.c:(.text.split_qio_to_list+0x244): undefined reference to `__divdi3' mips-linux-ld: drivers/md/dm-qcow2-map.o: in function `relocate_refcount_table': >> dm-qcow2-map.c:(.text.relocate_refcount_table+0x5c8): undefined reference to `__udivdi3'
Signature · 7 lines
0-DAY CI Kernel Test Service https://01.org/lkp -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
kernel test robot
build botRe: [PATCH 3/4] dm-qcow2: Introduce driver to create block devices over QCOW2 files
In reply to Kirill TkhaiThank you for the patch! Yet something to improve:
Hi Kirill,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on device-mapper-dm/for-next]
[also build test ERROR on song-md/md-next linus/master v5.17 next-20220328] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Kirill-Tkhai/dm-Introduce-dm-qcow2-driver-to-attach-QCOW2-files-as-block-device/20220328-192031 base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next config: m68k-allyesconfig (https://download.01.org/0day-ci/archive/20220329/[email protected]/config) compiler: m68k-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/c41a15c11909af3588885a88f6622c36dcc9ca35
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Kirill-Tkhai/dm-Introduce-dm-qcow2-driver-to-attach-QCOW2-files-as-block-device/20220328-192031 git checkout c41a15c11909af3588885a88f6622c36dcc9ca35
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=m68k SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
m68k-linux-ld: drivers/md/dm-qcow2-target.o: in function `calc_cached_parameters':
dm-qcow2-target.c:(.text+0x10ee): undefined reference to `__divdi3'
m68k-linux-ld: drivers/md/dm-qcow2-map.o: in function `qio_subclu_indexes':
dm-qcow2-map.c:(.text+0x15e): undefined reference to `__udivdi3'
m68k-linux-ld: drivers/md/dm-qcow2-map.o: in function `calc_refcounters_map':
dm-qcow2-map.c:(.text+0x2be): undefined reference to `__divdi3'
>> m68k-linux-ld: dm-qcow2-map.c:(.text+0x2e8): undefined reference to `__moddi3' m68k-linux-ld: drivers/md/dm-qcow2-map.o: in function `process_embedded_qios': dm-qcow2-map.c:(.text+0x38fa): undefined reference to `__divdi3' m68k-linux-ld: drivers/md/dm-qcow2-map.o: in function `parse_metadata': dm-qcow2-map.c:(.text+0x4d7a): undefined reference to `__divdi3' >> m68k-linux-ld: dm-qcow2-map.c:(.text+0x4d98): undefined reference to `__divdi3'
m68k-linux-ld: dm-qcow2-map.c:(.text+0x4e00): undefined reference to `__moddi3'
m68k-linux-ld: dm-qcow2-map.c:(.text+0x4e26): undefined reference to `__divdi3'
m68k-linux-ld: drivers/md/dm-qcow2-map.o: in function `relocate_refcount_table':
dm-qcow2-map.c:(.text+0x52ce): undefined reference to `__udivdi3'
.exit.text' referenced in section .data' of sound/soc/codecs/tlv320adc3xxx.o: defined in discarded section `.exit.text' of sound/soc/codecs/tlv320adc3xxx.o
Signature · 2 lines
0-DAY CI Kernel Test Service https://01.org/lkp
PATCH 3/4 dm-qcow2: Introduce driver to create block devices over QCOW2 files
In reply to Kirill TkhaiDriver for attaching QCOW2 files as block devices. It cares about performance-critical actions like actual IO and snapshot COW and merge, while complex and fast metadata-related service actions (e.g., snapshot creation and resize) are delegated to userspace.
Suspend/resume is a barrier between the driver and userspace. On suspend the driver brings QCOW2 images in consistent state, while on resume it re-reads metadata from QCOW2 images. Userspace proceeds metadata-related service actions while device is suspended, and it never does this on running device. This demarcation allows driver code to be pretty small.
Implemented full support for QCOW2 format: compressed clusters, internal snapshots, backing files and extended L2 table. Current limitations: the lowest backing file must be QCOW2 (RAW backing files are not supported), cluster and subcluster sizes for all backing files in chain must be equal (I'm going to change this in the future).
Also, implemented backward merge (from top image to its backing file).
The driver is request based, since this allows to use blk-mq merging of request. Driver splits requests itself, and every request (i.e., qio) after splitting fits a single cluster. (In some cases it is worth to create bigger splits, and this is a subject of further optimizations).
v1.5: Fix 32-bit platform compilation problems (64-bit division,
reported by kernel test robot <[email protected]>). Replace to_bytes() with "<< SECTOR_SHIFT" since to_bytes() is unsigned long and it is not enough for 32-bit platforms. Select ZLIB_INFLATE instead of depends on ZLIB_INFLATE.
--- drivers/md/Kconfig | 17 drivers/md/Makefile | 2 drivers/md/dm-qcow2-cmd.c | 383 ++++ drivers/md/dm-qcow2-map.c | 4252 ++++++++++++++++++++++++++++++++++++++++++ drivers/md/dm-qcow2-target.c | 1026 ++++++++++ drivers/md/dm-qcow2.h | 368 ++++ 6 files changed, 6048 insertions(+) create mode 100644 drivers/md/dm-qcow2-cmd.c create mode 100644 drivers/md/dm-qcow2-map.c create mode 100644 drivers/md/dm-qcow2-target.c create mode 100644 drivers/md/dm-qcow2.h
drivers/md/Kconfig +17 -0
@@ -652,4 +652,21 @@ config DM_AUDIT Enables audit logging of several security relevant events in the particular device-mapper targets, especially the integrity target. +config DM_QCOW2 + tristate "QCOW2 target support" + depends on BLK_DEV_DM + select ZLIB_INFLATE + help + Driver for attaching QCOW2 files as block devices. It cares + about performance-critical actions like actual IO and + snapshot COW and merge, while complex and fast metadata-related + service actions (e.g., snapshot creation and resize) + are delegated to userspace. + + Suspend/resume is a barrier between the driver and userspace. + On suspend the driver brings QCOW2 images in consistent state, + while on resume it re-reads metadata from QCOW2 images. Thus, + userspace proceeds metadata-related service actions, while + device is suspended. + endif # MD
drivers/md/Makefile +2 -0
@@ -26,6 +26,7 @@ dm-era-y += dm-era-target.o dm-clone-y += dm-clone-target.o dm-clone-metadata.o dm-verity-y += dm-verity-target.o dm-zoned-y += dm-zoned-target.o dm-zoned-metadata.o dm-zoned-reclaim.o +dm-qcow2-y += dm-qcow2-target.o dm-qcow2-map.o dm-qcow2-cmd.o md-mod-y += md.o md-bitmap.o raid456-y += raid5.o raid5-cache.o raid5-ppl.o
@@ -83,6 +84,7 @@ obj-$(CONFIG_DM_LOG_WRITES) += dm-log-writes.o obj-$(CONFIG_DM_INTEGRITY) += dm-integrity.o obj-$(CONFIG_DM_ZONED) += dm-zoned.o obj-$(CONFIG_DM_WRITECACHE) += dm-writecache.o +obj-$(CONFIG_DM_QCOW2) += dm-qcow2.o ifeq ($(CONFIG_DM_INIT),y) dm-mod-objs += dm-init.o
drivers/md/dm-qcow2-cmd.c new file +383 -0
383 lines not on this page · raw message
drivers/md/dm-qcow2-map.c new file +4252 -0
2,582 lines not on this page · raw message
drivers/md/dm-qcow2-target.c new file +1026 -0
1,026 lines not on this page · raw message
drivers/md/dm-qcow2.h new file +368 -0
368 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Re: [dm-devel] [PATCH 3/4] dm-qcow2: Introduce driver to create block devices over QCOW2 files
In reply to Kirill TkhaiQuoted 5 lines · Kirill Tkhai
The driver is request based, since this allows to use blk-mq merging of request. Driver splits requests itself, and every request (i.e., qio) after splitting fits a single cluster. (In some cases it is worth to create bigger splits, and this is a subject of further optimizations).
Nak, please don't do that. If you want finer grained processing use a bio based driver, not a request based one. This is just getting us into tons of problems.
Re: [dm-devel] [PATCH 3/4] dm-qcow2: Introduce driver to create block devices over QCOW2 files
Quoted 10 lines · Christoph Hellwig
Quoted 5 lines · Kirill Tkhai
The driver is request based, since this allows to use blk-mq merging of request. Driver splits requests itself, and every request (i.e., qio) after splitting fits a single cluster. (In some cases it is worth to create bigger splits, and this is a subject of further optimizations).
Nak, please don't do that. If you want finer grained processing use a bio based driver, not a request based one. This is just getting us into tons of problems.
Could you explain what you mean? Why shouldn't I use generic bio merging code, but implementing my own merging? Which problems you point?
Generic blk-mq is well tested and lots of people work on its performance. It's not obvious reason I should better implement own realization.
Re: [dm-devel] [PATCH 3/4] dm-qcow2: Introduce driver to create block devices over QCOW2 files
Quoted 12 lines · Kirill Tkhai
Quoted 10 lines · Christoph Hellwig
Quoted 5 lines · Kirill Tkhai
The driver is request based, since this allows to use blk-mq merging of request. Driver splits requests itself, and every request (i.e., qio) after splitting fits a single cluster. (In some cases it is worth to create bigger splits, and this is a subject of further optimizations).
Nak, please don't do that. If you want finer grained processing use a bio based driver, not a request based one. This is just getting us into tons of problems.
Quoted 5 lines
Could you explain what you mean? Why shouldn't I use generic bio merging code, but implementing my own merging? Which problems you point?
Generic blk-mq is well tested and lots of people work on its performance. It's not obvious reason I should better implement own realization.
In addition to this message. Possible, I confused you with patch description. Let I explain on example.
There are a lot cases, when block device receives sequential bios, which consist of one page. The driver wants to have these bios are merged together because of it's not a good idea to call call_write_iter() for each of such 1-page bios.
The same time, in QCOW2 format two sequential clusters may be placed inconsequently. For instance, let we have 1Mb cluster size and mapping:
virtual device range range inside QCOW2 file [0Mb, 1Mb] -> [20Mb, 21Mb] [1Mb, 2Mb] -> [30Mb, 31Mb].
This may happen because of cluster #1 were allocated later than cluster #0. In case of a request going to [512K, 1.5Mb], it intersects cluster boundary and we want to split it into two. Otherwise it impossible to call call_write_iter().
Solving the problem with 1-page bios, an implementing own elevator for only driver does not look a good idea. Making an elevator with performance comparable with generic block layer's is not a trivial task. At least it will require the same amount of code that block layer has. This is not a solution for a single driver.
From the driver side, it tells block layer about optimal IO size, so the number of splits should gravitate to zero.
Looking at another drivers like loop and nbd, they both are request based. Necessity to split a request to fit a filesystem extent is not a reason to stop merging loop requests, isn't it?! Also, nbd merges bios despite they are splitted by qemu-nbd in userspace level to fit 1 cluster boundary just like my driver does. Because, this is the only way. I've already tried another way, when I started writing this driver as bio-based. But then found that this is not a good solution, and converted it in request-based.
Kirill
PATCH 4/4 dm-qcow2: Add helper for working with dm-qcow2 devices
In reply to Kirill TkhaiThis scripts allows to create and remove dm-qcow2 device, create internal snapshot, resize virtual device size, or check image consistency on live device.
--- scripts/qcow2-dm.sh | 249 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 249 insertions(+) create mode 100755 scripts/qcow2-dm.sh
scripts/qcow2-dm.sh new file +249 -0
@@ -0,0 +1,249 @@ +#!/bin/bash +# +# This requires parameter dm_qcow2.kernel_sets_dirty_bit=y + +usage () { + cat <<EOF +Usage: + $prog_name create <file.qcow2> <dev_name> + $prog_name remove <file.qcow2> + $prog_name check [qemu-img check optional args] <file.qcow2> + $prog_name snapshot [qemu-img snapshot optional args] <file.qcow2> + $prog_name resize [qemu-img resize optional args] <file.qcow2> [+ | -]size +EOF +} + +get_img_size () { + sz=`qemu-img info -f qcow2 $1 | grep "virtual size" | sed 's/.*(\(.*\) bytes)/\1/'` + echo $sz +} + +get_nr_imgs () { + nr=`echo $1 | sed "s/.*qcow2 \(\w\) .*$/\1/"` + echo $nr +} + +get_dev_of_image () { + abs_path=$1 + + while read line; do + dev=`echo $line | sed "s/:.*//"` + nr_imgs=$(get_nr_imgs "$line") + top_img_id=$((nr_imgs - 1)) + + top_img_path=`dmsetup message $dev 0 get_img_name $top_img_id` + if [ -z "$top_img_path" ]; then + echo >&2 "Error during search of device"; exit 1; + return 1 + fi + + if [ "$abs_path" != "$top_img_path" ]; then + continue + fi + + echo $dev + return 0 + + done < <(LANG=C dmsetup table --target=qcow2 | grep -v "No devices found") + + return 0 +} + +create () { + if [ "$#" -ne 2 ]; then + echo >&2 "Wrong number of arguments."; usage; exit 1; + fi + + file=$1 + dev=$2 + files=() + fds="" + + qemu-img check $file || exit 1 + + disk_sz=$(get_img_size "$file") + echo disk_sz=$disk_sz + if [ -z "$disk_sz" ]; then + echo "Can't get disk size."; exit 1; + fi + + while :; do + if [ ! -f "$file" ]; then + echo "$file does not exist."; exit 1; + fi + + files+=("$file") + + exec {fd}<>$file || exit 1 + flock -x $fd || exit 1 + fds="$fd $fds" + + file=`qemu-img info $file | grep "backing file:" | sed "s/backing file: //"` + if [ -z "$file" ]; then + break + fi + done + + echo "Create device [$dev] of size $disk_sz from [${files[*]}]." + dmsetup create $dev --table "0 $((disk_sz / 512)) qcow2 ${fds}" +} + +reload_device () { + disk_sz=$(get_img_size "$abs_path") + if [ -z "$disk_sz" ]; then + echo "Can't get disk size."; return; + fi + + line=`dmsetup table $dev` + nr_imgs=$(get_nr_imgs "$line") + top_img_id=$((nr_imgs - 1)) + fds="" + + for id in `seq 0 $top_img_id`; + do + file=`dmsetup message $dev 0 get_img_name $id` + if [ -z "$file" ]; then + echo "Can't get image file."; return; + fi + exec {fd}<>$file || exit 1 + + fds="$fds $fd" + done + + echo "Reloading $dev." + dmsetup reload $dev --table "0 $((disk_sz / 512)) qcow2 ${fds}" +} + +remove () { + if [ "$#" -ne 1 ]; then + echo >&2 "Wrong number of arguments."; usage; exit 1; + fi + user_path=$1 + abs_path=`realpath $user_path` + + dev=$(get_dev_of_image "$abs_path") + if [ -z "$dev" ]; then + echo >&2 "Can't find device with [$user_path] top image."; exit 1 + fi + + echo "Removing device [$dev]." + dmsetup remove $dev + ret=$? + + if [ $ret -eq 0 ]; then + #Sanity check + echo "Checking [$abs_path]." + qemu-img check $abs_path + fi + exit $ret +} + +qemu_img () { + wants_reload=0 + if [ $1 == "--wants_reload" ]; then + wants_reload=1 + shift + fi + + if [ "$#" -lt 3 ]; then + echo >&2 "Wrong number of arguments."; usage; exit 1; + fi + + user_path=$1 + cmd=$2 + abs_path=`realpath $user_path` + qemu_img_args=${@: 2} + + dev=$(get_dev_of_image "$abs_path") + if [ -z "$dev" ]; then + echo >&2 "Can't find device by [$user_path]."; return 1 + fi + + echo "Suspending $dev." + dmsetup suspend $dev || exit 1 + + if [ "$cmd" != "check" ]; then + echo "Checking $abs_path" + qemu-img check $abs_path + ret=$? + if [ $ret -ne 0 ]; then + echo "Resuming $dev." + dmsetup resume $dev + exit 1 + fi + fi + + echo "===== Call: qemu-img $qemu_img_args. =====" + qemu-img $qemu_img_args + ret=$? + if [ $ret -ne 0 ]; then + echo >&2 "Failed during qemu-img call." + fi + echo "===== End of qemu-img $qemu_img_args. =====" + + if [ $wants_reload -ne 0 ]; then + reload_device $dev $abs_path + fi + + echo "Resuming $dev." + dmsetup resume $dev || exit 1 + if [ $? -ne 0 ]; then + ret=$? + fi + + return $ret +} + +check () { + user_path=${@: -1} + qemu_img_args=$@ + + qemu_img $user_path check $qemu_img_args + return $? +} + +snapshot () { + user_path=${@: -1} + qemu_img_args=$@ + + qemu_img $user_path snapshot $qemu_img_args + return $? +} + +resize () { + user_path=${@:(-2):1} + qemu_img_args=$@ + + qemu_img --wants_reload $user_path resize $qemu_img_args + return $? +} + +prog_name=$(basename $0) + +case $1 in + "create") + shift + create "$@" + exit 0 + ;; + "remove") + shift + remove "$@" + ;; + "check") + shift + check "$@" + ;; + "snapshot") + shift + snapshot "$@" + ;; + "resize") + shift + resize "$@" + ;; + *) + usage + exit 1 + ;; +esac
Quoted 4 lines · Kirill Tkhai
This patchset adds a new driver allowing to attach QCOW2 files as block devices. Its idea is to implement in kernel only that features, which affect runtime IO performance (IO requests processing functionality).
From a quick looks it seems to be like this should be a block driver just like the loop driver and not use device mapper. Why would you use device mapper to basically reimplement a fancy loop driver to start with?
Quoted 7 lines
The maintenance operations are
synchronously processed in userspace, while device is suspended.Userspace is allowed to do only that operations, which never modifies virtual disk's data. It is only allowed to modify QCOW2 file metadata providing that disk's data. The examples of allowed operations is snapshot creation and resize.
And this sounds like a pretty fragile design. It basically requires both userspace and the kernel driver to access metadata on disk, which sounds rather dangerous.
Quoted 5 lines
This example shows the way of device-mapper infrastructure allows to implement drivers following the idea of kernel/userspace components demarcation. Thus, the driver uses advantages of device-mapper instead of implementing its own suspend/resume engine.
What do you need more than a queue freeze?
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Kirill Tkhai
This patchset adds a new driver allowing to attach QCOW2 files as block devices. Its idea is to implement in kernel only that features, which affect runtime IO performance (IO requests processing functionality).
From a quick looks it seems to be like this should be a block driver just like the loop driver and not use device mapper. Why would you use device mapper to basically reimplement a fancy loop driver to start with?
This is a driver for containers and virtual machines. One of basic features for them is migration and backups. There are several drives for that, which are already implemented in device-mapper. For example, dm-era and dm-snap. Instead of implementing such the functionality in QCOW2 driver once again, the sane behavior is to use already implemented drivers. The module-approach is better for support and errors eliminating just because of less code.
1)A device-mapper based driver does not require migration and backup devices
are built in a stack for the whole device lifetime:
a)Normal work, almost 100% of time: there is only /dev/mapper/qcow2_dev. b)Migration: /dev/mapper/qcow2_dev is reloaded with a migration device,
which points to new qcow2_dev.real:
/dev/mapper/qcow2_dev [migration driver]
/dev/mapper/qcow2_dev.real [dm-qcow2 driver]
After migration is completed, we reload /dev/mapper/qcow2_dev back to use dm-qcow2 driver. So, there is no excess dm layers during normal work.
2)In case of the driver is not a device-mapper based, it's necessary to have
the stack built for the whole device lifetime, since it's impossible to reload bare block driver with dm-based driver on demand:
/dev/mapper/qcow2_dev [migration driver]
/dev/qcow2_dev.real [bare qcow2 driver]
So, we would have excess dm layer during whole device lifetime.
Our performance tests show, that a single dm layer may cause up to 10% performance decrease on NVME, so the reason is to eliminate such the fall. Also, the general reasoning say that excess layer is a wrong way.
Other reason is previous experience of implementing file-backed block drivers. We had ploop driver before. Ploop format is much simpler that QCOW2 format, but there are about 17K strings, while dm-qcow2 driver took about 6K strings. Device mapper allows to avoid writing a lot of code, the only thing you need is to implement proper .ctr and .dtr functions, while the rest of configuration actions are done by simple device-mapper reload.
Quoted 11 lines
Quoted 7 lines
The maintenance operations are
synchronously processed in userspace, while device is suspended.Userspace is allowed to do only that operations, which never modifies virtual disk's data. It is only allowed to modify QCOW2 file metadata providing that disk's data. The examples of allowed operations is snapshot creation and resize.
And this sounds like a pretty fragile design. It basically requires both userspace and the kernel driver to access metadata on disk, which sounds rather dangerous.
I don't think so. Device-mapper already allows to replace a device driver with another driver. Nobody blames dm-linear, that it may be reloaded to point wrong partition, while it can. Nobody blames loop, that someone in userspace may corrupt its blocks, and filesystem on that device will become broken.
The thing is kernel and userspace never access the file at the same time. It case of maintenance actions may be called in userspace, they must be, since this reduces the kernel code.
Quoted 7 lines
Quoted 5 lines
This example shows the way of device-mapper infrastructure allows to implement drivers following the idea of kernel/userspace components demarcation. Thus, the driver uses advantages of device-mapper instead of implementing its own suspend/resume engine.
What do you need more than a queue freeze?
Theoretically, I can, in case of this flushes all pending requests. But this will increase driver code significantly, since there won't be possible to use reload mechanism, and there are other problems like problems with performance like I wrote above. So, this approach look significantly worse.
Kirill