dm-unstripe
Subsystems (3)
v2 cover dm-unstripe
V1->v2 Changes:
1) Fixed up some spelling errors in documentation.
2) Cleaned up some variable names to something more appropriate.
PATCH v2 1/2 dm-unstripe: unstripe of IO across RAID 0
This device mapper module remaps and unstripes IO so it lands solely on a single drive in a RAID 0. In a 4 drive RAID 0 the mapper exposes 1/4th of the LBA range as a virtual drive. Each IO to that virtual drive will land on only one of the 4 drives, selected by the user.
As an example:
Intel NVMe drives contain two cores on the physical device. Each core of the drive has segregated access to its LBA range. The current LBA model has a RAID 0 128k stripe across the two cores:
Core 0: Core 1:
__________ __________ | LBA 511| | LBA 768| | LBA 0 | | LBA 256|
The purpose of this unstriping is to provide better QoS in noisy neighbor environments. When two partitions are created on the aggregate drive without this unstriping, reads on one partition can affect writes on another partition. With the striping concurrent reads and writes and I/O on opposite cores have lower completion times, and better tail latencies.
--- drivers/md/Kconfig | 10 +++ drivers/md/Makefile | 1 + drivers/md/dm-unstripe.c | 197 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 208 insertions(+) create mode 100644 drivers/md/dm-unstripe.c
drivers/md/Kconfig +10 -0
@@ -269,6 +269,16 @@ config DM_BIO_PRISON source "drivers/md/persistent-data/Kconfig" +config DM_UN_STRIPE + tristate "Transpose IO to individual drives on a raid device" + depends on BLK_DEV_DM + ---help--- + Enable this feature if you with to unstripe I/O on a RAID 0 + device to the respective drive. If your hardware has physical + RAID 0 this module can unstripe the I/O to respective sides. + + If unsure say N. + config DM_CRYPT tristate "Crypt target support" depends on BLK_DEV_DM
drivers/md/Makefile +1 -0
@@ -43,6 +43,7 @@ obj-$(CONFIG_BCACHE) += bcache/ obj-$(CONFIG_BLK_DEV_MD) += md-mod.o obj-$(CONFIG_BLK_DEV_DM) += dm-mod.o obj-$(CONFIG_BLK_DEV_DM_BUILTIN) += dm-builtin.o +obj-$(CONFIG_DM_UN_STRIPE) += dm-unstripe.o obj-$(CONFIG_DM_BUFIO) += dm-bufio.o obj-$(CONFIG_DM_BIO_PRISON) += dm-bio-prison.o obj-$(CONFIG_DM_CRYPT) += dm-crypt.o
drivers/md/dm-unstripe.c new file +197 -0
197 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [PATCH v2 1/2] dm-unstripe: unstripe of IO across RAID 0
Review
Quoted 2 lines · Scott Bauer
(quoted diff) +2
-0+ + dm_set_target_max_io_len(ti, target->max_hw_sectors);
The return for this function has "__must_check", so it's currently throwing an a compiler warning.
Otherwise, this looks like it's doing what you want, and tests successfully on my synthetic workloads.
PATCH v2 2/2 dm unstripe: Add documentation for unstripe target
In reply to Scott Bauer--- Documentation/device-mapper/dm-unstripe.txt | 82 +++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 Documentation/device-mapper/dm-unstripe.txt
Documentation/device-mapper/dm-unstripe.txt new file +82 -0
@@ -0,0 +1,82 @@ +Device-Mapper Unstripe +===================== + +The device-mapper Unstripe (dm-unstripe) target provides a transparent +mechanism to unstripe a RAID 0 striping to access segregated disks. + +This module should be used by users who understand what the underlying +disks look like behind the software/hardware RAID. + +Parameters: +<drive (ex: /dev/nvme0n1)> <drive #> <# of drives> <stripe sectors> + + +<drive> + The block device you wish to unstripe. + +<drive #> + The physical drive you wish to expose via this "virtual" device + mapper target. This must be 0 indexed. + +<# of drives> + The number of drives in the RAID 0. + +<stripe sectors> + The amount of 512B sectors in the raid striping, or zero, if you + wish you use max_hw_sector_size. + + +Why use this module? +===================== + +As a use case: + + + As an example: + + Intel NVMe drives contain two cores on the physical device. + Each core of the drive has segregated access to its LBA range. + The current LBA model has a RAID 0 128k stripe across the two cores: + + Core 0: Core 1: + __________ __________ + | LBA 511| | LBA 768| + | LBA 0 | | LBA 256| + ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻ ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻ + + The purpose of this unstriping is to provide better QoS in noisy + neighbor environments. When two partitions are created on the + aggregate drive without this unstriping, reads on one partition + can affect writes on another partition. With the striping concurrent + reads and writes and I/O on opposite cores have lower completion times, + and better tail latencies. + + With the module we were able to segregate a fio script that has read and + write jobs that are independent of each other. Compared to when we run + the test on a combined drive with partitions, we were able to get a 92% + reduction in five-9ths read latency using this device mapper target. + + + One could use the module to Logical de-pop a HDD if you have sufficient + geometry information regarding the drive. + + +Example scripts: +==================== + +dmsetup create nvmset1 --table '0 1 dm-unstripe /dev/nvme0n1 1 2 0' +dmsetup create nvmset0 --table '0 1 dm-unstripe /dev/nvme0n1 0 2 0' + +There will now be two mappers: +/dev/mapper/nvmset1 +/dev/mapper/nvmset0 + +that will expose core 0 and core 1. + + +In a Raid 0 with 4 drives of stripe size 128K: +dmsetup create raid_disk0 --table '0 1 dm-unstripe /dev/nvme0n1 0 4 256' +dmsetup create raid_disk1 --table '0 1 dm-unstripe /dev/nvme0n1 1 4 256' +dmsetup create raid_disk2 --table '0 1 dm-unstripe /dev/nvme0n1 2 4 256' +dmsetup create raid_disk3 --table '0 1 dm-unstripe /dev/nvme0n1 3 4 256' +
Signature · 1 line
2.11.0
Re: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target
Quoted 8 lines · Scott Bauer
Documentation/device-mapper/dm-unstripe.txt new file +8
-0+Example scripts: +==================== + +dmsetup create nvmset1 --table '0 1 dm-unstripe /dev/nvme0n1 1 2 0' +dmsetup create nvmset0 --table '0 1 dm-unstripe /dev/nvme0n1 0 2 0' + +There will now be two mappers: +/dev/mapper/nvmset1
Documentation/device-mapper/dm-unstripe.txt line 81
Quoted 10 lines
Documentation/device-mapper/dm-unstripe.txt new file +10
-0+/dev/mapper/nvmset0 + +that will expose core 0 and core 1. + + +In a Raid 0 with 4 drives of stripe size 128K: +dmsetup create raid_disk0 --table '0 1 dm-unstripe /dev/nvme0n1 0 4 256' +dmsetup create raid_disk1 --table '0 1 dm-unstripe /dev/nvme0n1 1 4 256' +dmsetup create raid_disk2 --table '0 1 dm-unstripe /dev/nvme0n1 2 4 256' +dmsetup create raid_disk3 --table '0 1 dm-unstripe /dev/nvme0n1 3 4 256'
While this device mapper is intended for H/W RAID where the member disks are hidden, we can test it using DM software striping so we don't need any particular hardware.
Here's a little test script I wrote for that. It sets up a striped device backed by files, unstripes it into different sets, then compares each to its original backing file after writing random data to it, cleaning up the test artifacts before exiting. The parameters at the top can be modified to test different striping scenarios.
---
#!/bin/bash
MEMBER_SIZE=$((128 * 1024 * 1024))
NUM=4
SEQ_END=$((${NUM}-1))
CHUNK=256
BS=4096
RAID_SIZE=$((${MEMBER_SIZE}*${NUM}/512))
DM_PARMS="0 ${RAID_SIZE} striped ${NUM} ${CHUNK}"
COUNT=$((${MEMBER_SIZE} / ${BS}))
for i in $(seq 0 ${SEQ_END}); do
dd if=/dev/zero of=member-${i} bs=${MEMBER_SIZE} count=1 oflag=direct losetup /dev/loop${i} member-${i}
DM_PARMS+=" /dev/loop${i} 0"
done
echo $DM_PARMS | dmsetup create raid0
for i in $(seq 0 ${SEQ_END}); do
echo "0 1 dm-unstripe /dev/mapper/raid0 ${i} ${NUM} ${CHUNK}" | dmsetup create set-${i}
done;
for i in $(seq 0 ${SEQ_END}); do
dd if=/dev/urandom of=/dev/mapper/set-${i} bs=${BS} count=${COUNT} oflag=direct diff /dev/mapper/set-${i} member-${i}
done;
for i in $(seq 0 ${SEQ_END}); do
dmsetup remove set-${i}
done
dmsetup remove raid0
for i in $(seq 0 ${SEQ_END}); do
losetup -d /dev/loop${i}
rm -f member-${i}
done
--
Re: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target
In reply to Scott BauerQuoted 47 lines · Scott Bauer
Signed-off-by Scott Bauer <[email protected]>--- Documentation/device-mapper/dm-unstripe.txt | 82 +++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 Documentation/device-mapper/dm-unstripe.txtDocumentation/device-mapper/dm-unstripe.txt new file +35
-0@@ -0,0 +1,82 @@ +Device-Mapper Unstripe +===================== + +The device-mapper Unstripe (dm-unstripe) target provides a transparent +mechanism to unstripe a RAID 0 striping to access segregated disks. + +This module should be used by users who understand what the underlying +disks look like behind the software/hardware RAID. + +Parameters: +<drive (ex: /dev/nvme0n1)> <drive #> <# of drives> <stripe sectors> + + +<drive> + The block device you wish to unstripe. + +<drive #> + The physical drive you wish to expose via this "virtual" device + mapper target. This must be 0 indexed. + +<# of drives> + The number of drives in the RAID 0. + +<stripe sectors> + The amount of 512B sectors in the raid striping, or zero, if you + wish you use max_hw_sector_size. + + +Why use this module? +===================== + +As a use case: + + + As an example:
Documentation/device-mapper/dm-unstripe.txt line 45
Quoted 10 lines
Documentation/device-mapper/dm-unstripe.txt new file +10
-0+ + Intel NVMe drives contain two cores on the physical device. + Each core of the drive has segregated access to its LBA range. + The current LBA model has a RAID 0 128k stripe across the two cores: + + Core 0: Core 1: + __________ __________ + | LBA 511| | LBA 768| + | LBA 0 | | LBA 256| + ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻ ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻
If it's 128k stripe shouldn't it be LBAs 0/256 on core0 and LBAs 128/511 on core1?
Quoted 38 lines
Documentation/device-mapper/dm-unstripe.txt new file +37
-0+ + The purpose of this unstriping is to provide better QoS in noisy + neighbor environments. When two partitions are created on the + aggregate drive without this unstriping, reads on one partition + can affect writes on another partition. With the striping concurrent + reads and writes and I/O on opposite cores have lower completion times, + and better tail latencies. + + With the module we were able to segregate a fio script that has read and + write jobs that are independent of each other. Compared to when we run + the test on a combined drive with partitions, we were able to get a 92% + reduction in five-9ths read latency using this device mapper target. + + + One could use the module to Logical de-pop a HDD if you have sufficient + geometry information regarding the drive. + + +Example scripts: +==================== + +dmsetup create nvmset1 --table '0 1 dm-unstripe /dev/nvme0n1 1 2 0' +dmsetup create nvmset0 --table '0 1 dm-unstripe /dev/nvme0n1 0 2 0' + +There will now be two mappers: +/dev/mapper/nvmset1 +/dev/mapper/nvmset0 + +that will expose core 0 and core 1. + + +In a Raid 0 with 4 drives of stripe size 128K: +dmsetup create raid_disk0 --table '0 1 dm-unstripe /dev/nvme0n1 0 4 256' +dmsetup create raid_disk1 --table '0 1 dm-unstripe /dev/nvme0n1 1 4 256' +dmsetup create raid_disk2 --table '0 1 dm-unstripe /dev/nvme0n1 2 4 256' +dmsetup create raid_disk3 --table '0 1 dm-unstripe /dev/nvme0n1 3 4 256' +
Re: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target
Quoted 15 lines · Nikolay Borisov
Documentation/device-mapper/dm-unstripe.txt line 45
Quoted 11 lines · Scott Bauer
Documentation/device-mapper/dm-unstripe.txt new file +11
-0+ As an example: + + Intel NVMe drives contain two cores on the physical device. + Each core of the drive has segregated access to its LBA range. + The current LBA model has a RAID 0 128k stripe across the two cores: + + Core 0: Core 1: + __________ __________ + | LBA 511| | LBA 768| + | LBA 0 | | LBA 256| + ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻ ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻If it's 128k stripe shouldn't it be LBAs 0/256 on core0 and LBAs 128/511 on core1?
Ah, this device's makers call the "stripe" size what should be called "chunk". This device has a 128k chunk per core with two cores, so the full stripe is 256k. The above should have core 0 owning LBA 512 rather than 511 (assuming 512b LBA format).
Alasdair G Kergon
maintainerRe: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target
Quoted 2 lines · Keith Busch
Ah, this device's makers call the "stripe" size what should be called "chunk".
If this target is to go anywhere, let's try to define it as 'undoing' the existing dm-stripe target using primary terminology, field names etc. as close as possible to our existing target. We can still mention any alternative terminology encountered in the documentation. The first (simplest) documented example could be a dm-stripe target being 'unstriped'.
Alasdair
Re: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target
In reply to Scott BauerQuoted 63 lines · Scott Bauer
Signed-off-by Scott Bauer <[email protected]>--- Documentation/device-mapper/dm-unstripe.txt | 82 +++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 Documentation/device-mapper/dm-unstripe.txtDocumentation/device-mapper/dm-unstripe.txt new file +51
-0@@ -0,0 +1,82 @@ +Device-Mapper Unstripe +===================== + +The device-mapper Unstripe (dm-unstripe) target provides a transparent +mechanism to unstripe a RAID 0 striping to access segregated disks. + +This module should be used by users who understand what the underlying +disks look like behind the software/hardware RAID. + +Parameters: +<drive (ex: /dev/nvme0n1)> <drive #> <# of drives> <stripe sectors> + + +<drive> + The block device you wish to unstripe. + +<drive #> + The physical drive you wish to expose via this "virtual" device + mapper target. This must be 0 indexed. + +<# of drives> + The number of drives in the RAID 0. + +<stripe sectors> + The amount of 512B sectors in the raid striping, or zero, if you + wish you use max_hw_sector_size. + + +Why use this module? +===================== + +As a use case: + + + As an example: + + Intel NVMe drives contain two cores on the physical device. + Each core of the drive has segregated access to its LBA range. + The current LBA model has a RAID 0 128k stripe across the two cores: + + Core 0: Core 1: + __________ __________ + | LBA 511| | LBA 768| + | LBA 0 | | LBA 256| + ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻ ⎻⎻⎻⎻⎻⎻⎻⎻⎻⎻ + + The purpose of this unstriping is to provide better QoS in noisy + neighbor environments. When two partitions are created on the + aggregate drive without this unstriping, reads on one partition + can affect writes on another partition. With the striping concurrent + reads and writes and I/O on opposite cores have lower completion times,
Documentation/device-mapper/dm-unstripe.txt line 61
Quoted 10 lines
Documentation/device-mapper/dm-unstripe.txt new file +10
-0+ and better tail latencies. + + With the module we were able to segregate a fio script that has read and + write jobs that are independent of each other. Compared to when we run + the test on a combined drive with partitions, we were able to get a 92% + reduction in five-9ths read latency using this device mapper target. + + + One could use the module to Logical de-pop a HDD if you have sufficient + geometry information regarding the drive.
OK, but I'm left wondering: why doesn't the user avoid striping across the cores?
Do the Intel NVMe drives not provide the ability to present 1 device per NVMe core?
This DM target seems like a pretty nasty workaround for what should be fixed in the NVMe drive's firmware.
Mainly because there is no opportunity to use both striped and unstriped access to the same NVMe drive. So why impose striped on the user in the first place?
Mike
Re: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target
Quoted 9 lines · Mike Snitzer
On Mon, Dec 11 2017 at 11:00am -0500,
Scott Bauer <[email protected]> wrote:OK, but I'm left wondering: why doesn't the user avoid striping across the cores?
Do the Intel NVMe drives not provide the ability to present 1 device per NVMe core?
Quoted 8 lines
This DM target seems like a pretty nasty workaround for what should be fixed in the NVMe drive's firmware.
Mainly because there is no opportunity to use both striped and unstriped access to the same NVMe drive. So why impose striped on the user in the first place?
Mike
Unfortunately, the NVMe drives do not currently support exposing each core as seperate drives or namespaces. While it would be preferable if the controllers did expose such features, firmware development informed us there are sufficient reasons why it isn't possible for the existing generation of drives.
The NVMe working group just finalized a standard that presents isolated storage sets for users who wish to use them. As the standard was just recently finalized the use case was created well after the targeted drives were created. The Intel drives just so happen to have independent back-ends that align with the isolated storage sets use case. We would like to provide a way to exploit the independent back-ends to expose isolated storage in a way that is generic across all the Intel NVMe drive familes. The implementation is generic enough that it can be applied to any storage that has physical seperation at known block boundaries.