dm-unstripe
Subsystems (3)
v4 cover dm-unstripe
Change log:
v3->v4:
Addressed comments from Randy. Modified documentation to be clearer, and fixed small off by one in bounds checking in constructor.
v2->v3:
1) Renamed variables in-code to reflect correct terminology with respect to the dm-stripe target.
2) Fixed a __must_check missing check in the constructor.
3) Used correct types for working with sector remaping.
3) Fixed documentation to reflect the correct termonology, like the code. Added the test script Keith sent out in an email that makes a striped device and uses the un-stripe target to access the underlying loop devices.
PATCH v4 1/2 dm-unstripe: unstripe RAID 0/dm-striped device
This device mapper module remaps and unstripes IO so it lands solely on a single drive in a RAID 0/dm-stripe target. In a 4 drive RAID 0 the mapper exposes 1/4th of the LBA range as a virtual drive. Each IO to that virtual drive will land on only one of the 4 drives, selected by the user.
--- drivers/md/Kconfig | 10 +++ drivers/md/Makefile | 1 + drivers/md/dm-unstripe.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 215 insertions(+) create mode 100644 drivers/md/dm-unstripe.c
drivers/md/Kconfig +10 -0
@@ -269,6 +269,16 @@ config DM_BIO_PRISON source "drivers/md/persistent-data/Kconfig" +config DM_UN_STRIPE + tristate "Transpose IO to individual drives on a raid device" + depends on BLK_DEV_DM + ---help--- + Enable this feature if you with to unstripe I/O on a RAID 0 + device to the respective drive. If your hardware has physical + RAID 0 this module can unstripe the I/O to respective sides. + + If unsure say N. + config DM_CRYPT tristate "Crypt target support" depends on BLK_DEV_DM
drivers/md/Makefile +1 -0
@@ -43,6 +43,7 @@ obj-$(CONFIG_BCACHE) += bcache/ obj-$(CONFIG_BLK_DEV_MD) += md-mod.o obj-$(CONFIG_BLK_DEV_DM) += dm-mod.o obj-$(CONFIG_BLK_DEV_DM_BUILTIN) += dm-builtin.o +obj-$(CONFIG_DM_UN_STRIPE) += dm-unstripe.o obj-$(CONFIG_DM_BUFIO) += dm-bufio.o obj-$(CONFIG_DM_BIO_PRISON) += dm-bio-prison.o obj-$(CONFIG_DM_CRYPT) += dm-crypt.o
drivers/md/dm-unstripe.c new file +204 -0
204 lines not on this page · raw message
Signature · 1 line
2.11.0
Randy Dunlap
reviewerRe: [PATCH v4 1/2] dm-unstripe: unstripe RAID 0/dm-striped device
Quoted 20 lines · Scott Bauer
This device mapper module remaps and unstripes IO so it lands solely on a single drive in a RAID 0/dm-stripe target. In a 4 drive RAID 0 the mapper exposes 1/4th of the LBA range as a virtual drive. Each IO to that virtual drive will land on only one of the 4 drives, selected by the user.
Signed-off-by Scott Bauer <[email protected]>Acked-by Keith Busch <[email protected]>--- drivers/md/Kconfig | 10 +++ drivers/md/Makefile | 1 + drivers/md/dm-unstripe.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 215 insertions(+) create mode 100644 drivers/md/dm-unstripe.cdrivers/md/Kconfig +0
-0
drivers/md/Kconfig config DM_BIO_PRISON · line 278
Quoted 10 lines
drivers/md/Kconfig +7
-0@@ -269,6 +269,16 @@ config DM_BIO_PRISON source "drivers/md/persistent-data/Kconfig" +config DM_UN_STRIPE + tristate "Transpose IO to individual drives on a raid device" + depends on BLK_DEV_DM + ---help--- + Enable this feature if you with to unstripe I/O on a RAID 0 + device to the respective drive. If your hardware has physical + RAID 0 this module can unstripe the I/O to respective sides.
What does "sides" mean above?
drivers/md/Kconfig config DM_BIO_PRISON · line 281
Quoted 3 lines
drivers/md/Kconfig +3
-0+ + If unsure say N. +
thanks,
Signature · 1 line
~Randy
Re: [PATCH v4 1/2] dm-unstripe: unstripe RAID 0/dm-striped device
Quoted 10 lines
drivers/md/Kconfig config DM_BIO_PRISON · line 278
Quoted 8 lines
drivers/md/Kconfig +7
-0+config DM_UN_STRIPE + tristate "Transpose IO to individual drives on a raid device" + depends on BLK_DEV_DM + ---help--- + Enable this feature if you with to unstripe I/O on a RAID 0 + device to the respective drive. If your hardware has physical + RAID 0 this module can unstripe the I/O to respective sides.What does "sides" mean above?
Should say, "members". There's also an alternative patch set out by Heinz that I am currently testing/writing comments for. It looks like we'll use his version after I submit my comments (IE take the stuff I put in my v4 to fix your comments).
Re: [PATCH v4 1/2] dm-unstripe: unstripe RAID 0/dm-striped device
Quoted 11 lines · Scott Bauer
Quoted 10 lines
drivers/md/Kconfig config DM_BIO_PRISON · line 278
Quoted 8 lines
drivers/md/Kconfig +7
-0+config DM_UN_STRIPE + tristate "Transpose IO to individual drives on a raid device" + depends on BLK_DEV_DM + ---help--- + Enable this feature if you with to unstripe I/O on a RAID 0 + device to the respective drive. If your hardware has physical + RAID 0 this module can unstripe the I/O to respective sides.What does "sides" mean above?
Quoted 4 lines
Should say, "members". There's also an alternative patch set out by Heinz that I am currently testing/writing comments for. It looks like we'll use his version after I submit my comments (IE take the stuff I put in my v4 to fix your comments).
Can you be more specific? I've not seen anything from Heinz.
I've been reviewing your v4 and modified it to match dm-stripe.c more (like Alasdair asked for).
Mike
Re: [PATCH v4 1/2] dm-unstripe: unstripe RAID 0/dm-striped device
Quoted 19 lines · Mike Snitzer
Quoted 11 lines · Scott Bauer
Quoted 10 lines
drivers/md/Kconfig config DM_BIO_PRISON · line 278
Quoted 8 lines
drivers/md/Kconfig +7
-0+config DM_UN_STRIPE + tristate "Transpose IO to individual drives on a raid device" + depends on BLK_DEV_DM + ---help--- + Enable this feature if you with to unstripe I/O on a RAID 0 + device to the respective drive. If your hardware has physical + RAID 0 this module can unstripe the I/O to respective sides.What does "sides" mean above?
Quoted 4 lines
Should say, "members". There's also an alternative patch set out by Heinz that I am currently testing/writing comments for. It looks like we'll use his version after I submit my comments (IE take the stuff I put in my v4 to fix your comments).
Quoted 4 lines
Can you be more specific? I've not seen anything from Heinz.
I've been reviewing your v4 and modified it to match
dm-stripe.cmore (like Alasdair asked for).
When the right hand isn't aware the left is doing something ;)
I'll combine my effort with Heinz's.
Please don't send another revision until you see what I've come up with.
Thanks,
Mike
kbuild test robot
build botRe: [PATCH v4 1/2] dm-unstripe: unstripe RAID 0/dm-striped device
In reply to Scott BauerI love your patch! Yet something to improve:
Hi Scott,
I love your patch! Yet something to improve:
[auto build test ERROR on dm/for-next]
[also build test ERROR on v4.15-rc4]
[cannot apply to next-20171220]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Scott-Bauer/dm-unstripe/20171220-121845 base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next config: m68k-allyesconfig (attached as .config) compiler: m68k-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=m68k
All errors (new ones prefixed by >>):
drivers/md/dm-unstripe.o: In function `set_ctr':
>> dm-unstripe.c:(.text+0x36a): undefined reference to `__udivdi3' >> dm-unstripe.c:(.text+0x384): undefined reference to `__umoddi3'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
kbuild test robot
build botRe: [PATCH v4 1/2] dm-unstripe: unstripe RAID 0/dm-striped device
In reply to Scott BauerI love your patch! Yet something to improve:
Hi Scott,
I love your patch! Yet something to improve:
[auto build test ERROR on dm/for-next]
[also build test ERROR on v4.15-rc4]
[cannot apply to next-20171220]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Scott-Bauer/dm-unstripe/20171220-121845 base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next config: arm-allyesconfig (attached as .config) compiler: arm-linux-gnueabi-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm
All errors (new ones prefixed by >>):
drivers/md/dm-unstripe.o: In function `set_ctr':
>> dm-unstripe.c:(.text+0x438): undefined reference to `__aeabi_uldivmod'
dm-unstripe.c:(.text+0x450): undefined reference to `__aeabi_uldivmod'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH v4 1/2] dm-unstripe: unstripe RAID 0/dm-striped device
Quoted 21 lines · kbuild test robot
Hi Scott,
I love your patch! Yet something to improve:
[auto build test ERROR on dm/for-next]
[also build test ERROR on v4.15-rc4]
[cannot apply to next-20171220]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]url: https://github.com/0day-ci/linux/commits/Scott-Bauer/dm-unstripe/20171220-121845 base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next config: arm-allyesconfig (attached as .config) compiler: arm-linux-gnueabi-gcc (Debian 7.2.0-11) 7.2.0reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=armAll errors (new ones prefixed by >>):
Quoted 7 lines
drivers/md/dm-unstripe.o: In function `set_ctr':Quoted 1 line
Quoted 1 line
dm-unstripe.c:(.text+0x438): undefined reference to `__aeabi_uldivmod'
dm-unstripe.c:(.text+0x450): undefined reference to `__aeabi_uldivmod'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
These reports are no longer relevant given the code has changed since v4. In particular these 64-bit divide and mod in v4 no longer exist:
+ u64 tot_sec, mod;
...
+ tot_sec = i_size_read(bbdev->bd_inode) >> SECTOR_SHIFT;
+ mod = tot_sec % target->chunk_sectors;
+
+ if (ti->len == 1)
+ ti->len = (tot_sec / tot_drives) - mod;
PATCH v4 2/2 dm-unstripe: Add documentation for unstripe target
In reply to Scott Bauer--- Documentation/device-mapper/dm-unstripe.txt | 130 ++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 Documentation/device-mapper/dm-unstripe.txt
Documentation/device-mapper/dm-unstripe.txt new file +130 -0
@@ -0,0 +1,130 @@ +Device-Mapper Unstripe +===================== + +The device-mapper unstripe (dm-unstripe) target provides a transparent +mechanism to unstripe a device-mapper "striped" target to access the +underlying disks without having to touch the true backing block-device. +It can also be used to unstripe a hardware RAID-0 to access backing disks +as well. + + +Parameters: +<drive (ex: /dev/nvme0n1)> <drive #> <# of drives> <chunk sectors> + + +<drive> + The block device you wish to unstripe. + +<drive #> + The physical drive you wish to expose via this "virtual" device + mapper target. This must be 0 indexed. + +<# of drives> + The number of drives in the RAID 0. + +<chunk sectors> + The amount of 512B sectors in the chunk striping, or zero, if you + wish you use max_hw_sector_size. + + +Why use this module? +===================== + + An example of undoing an existing dm-stripe: + + This small bash script will setup 4 loop devices and use the existing + dm-stripe target to combine the 4 devices into one. It then will use + the unstripe target on the new combined stripe device to access the + individual backing loop devices. We write data to the newly exposed + unstriped devices and verify the data written matches the correct + underlying device on the striped array. + + #!/bin/bash + + MEMBER_SIZE=$((128 * 1024 * 1024)) + NUM=4 + SEQ_END=$((${NUM}-1)) + CHUNK=256 + BS=4096 + + RAID_SIZE=$((${MEMBER_SIZE}*${NUM}/512)) + DM_PARMS="0 ${RAID_SIZE} striped ${NUM} ${CHUNK}" + COUNT=$((${MEMBER_SIZE} / ${BS})) + + for i in $(seq 0 ${SEQ_END}); do + dd if=/dev/zero of=member-${i} bs=${MEMBER_SIZE} count=1 oflag=direct + losetup /dev/loop${i} member-${i} + DM_PARMS+=" /dev/loop${i} 0" + done + + echo $DM_PARMS | dmsetup create raid0 + for i in $(seq 0 ${SEQ_END}); do + echo "0 1 unstripe /dev/mapper/raid0 ${i} ${NUM} ${CHUNK}" | dmsetup create set-${i} + done; + + for i in $(seq 0 ${SEQ_END}); do + dd if=/dev/urandom of=/dev/mapper/set-${i} bs=${BS} count=${COUNT} oflag=direct + diff /dev/mapper/set-${i} member-${i} + done; + + for i in $(seq 0 ${SEQ_END}); do + dmsetup remove set-${i} + done + + dmsetup remove raid0 + + for i in $(seq 0 ${SEQ_END}); do + losetup -d /dev/loop${i} + rm -f member-${i} + done + +============== + + + Another example: + + Intel NVMe drives contain two cores on the physical device. + Each core of the drive has segregated access to its LBA range. + The current LBA model has a RAID 0 128k chunk on each core, resulting + in a 256k stripe across the two cores: + + Core 0: Core 1: + __________ __________ + | LBA 512| | LBA 768| + | LBA 0 | | LBA 256| + ---------- ---------- + + The purpose of this unstriping is to provide better QoS in noisy + neighbor environments. When two partitions are created on the + aggregate drive without this unstriping, reads on one partition + can affect writes on another partition. This is because the partitions + are striped across the two cores. When we unstripe this hardware RAID 0 + and make partitions on each new exposed device the two partitions are now + physically separated. + + With the module we were able to segregate a fio script that has read and + write jobs that are independent of each other. Compared to when we run + the test on a combined drive with partitions, we were able to get a 92% + reduction read latency using this device mapper target. + + +==================== +Example scripts: + + +dmsetup create nvmset1 --table '0 1 unstripe /dev/nvme0n1 1 2 0' +dmsetup create nvmset0 --table '0 1 unstripe /dev/nvme0n1 0 2 0' + +There will now be two mappers: +/dev/mapper/nvmset1 +/dev/mapper/nvmset0 + +that will expose core 0 and core 1. + + +# In a dm-stripe with 4 drives of chunk size 128K: +dmsetup create raid_disk0 --table '0 1 unstripe /dev/mapper/striped 0 4 256' +dmsetup create raid_disk1 --table '0 1 unstripe /dev/mapper/striped 1 4 256' +dmsetup create raid_disk2 --table '0 1 unstripe /dev/mapper/striped 2 4 256' +dmsetup create raid_disk3 --table '0 1 unstripe /dev/mapper/striped 3 4 256' +
Signature · 1 line
2.11.0