Regarding dm-ioband tests
Hi Ryo,
I decided to play a bit more with dm-ioband and started doing some testing. I am doing a simple two dd threads doing reads and don't seem to be gettting the fairness. So thought will ask you what's the issue here. Is there an issue with my testing procedure.
I got one 40G SATA drive (no hardware queuing). I have created two partitions on that disk /dev/sdd1 and /dev/sdd2 and created two ioband devices ioband1 and ioband2 on partitions sdd1 and sdd2 respectively. The weights of ioband1 and ioband2 devices are 200 and 100 respectively.
I am assuming that this setup will create two default groups and IO going to partition sdd1 should get double the BW of partition sdd2.
But it looks like I am not gettting that behavior. Following is the output of "dmsetup table" command. This snapshot has been taken every 2 seconds while IO was going on. Column 9 seems to be containing how many sectors of IO has been done on a particular io band device and group. Looking at the snapshot, it does not look like that ioband1 default group got double the BW of ioband2 default group.
Am I doing something wrong here?
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0
ioband2: 0 40355280 ioband 1 -1 96 0 11528 0 0 0
ioband1: 0 37768752 ioband 1 -1 82 0 9736 0 0 0
ioband2: 0 40355280 ioband 1 -1 748 2 93032 0 0 0
ioband1: 0 37768752 ioband 1 -1 896 0 112232 0 0 0
ioband2: 0 40355280 ioband 1 -1 1326 5 165816 0 0 0
ioband1: 0 37768752 ioband 1 -1 1816 0 228312 0 0 0
ioband2: 0 40355280 ioband 1 -1 1943 6 243712 0 0 0
ioband1: 0 37768752 ioband 1 -1 2692 0 338760 0 0 0
ioband2: 0 40355280 ioband 1 -1 2461 10 308576 0 0 0
ioband1: 0 37768752 ioband 1 -1 3618 0 455608 0 0 0
ioband2: 0 40355280 ioband 1 -1 3118 11 391352 0 0 0
ioband1: 0 37768752 ioband 1 -1 4406 0 555032 0 0 0
ioband2: 0 40355280 ioband 1 -1 3734 15 468760 0 0 0
ioband1: 0 37768752 ioband 1 -1 5273 0 664328 0 0 0
ioband2: 0 40355280 ioband 1 -1 4307 17 540784 0 0 0
ioband1: 0 37768752 ioband 1 -1 6181 0 778992 0 0 0
ioband2: 0 40355280 ioband 1 -1 4930 19 619208 0 0 0
ioband1: 0 37768752 ioband 1 -1 7028 0 885728 0 0 0
ioband2: 0 40355280 ioband 1 -1 5599 22 703280 0 0 0
ioband1: 0 37768752 ioband 1 -1 7815 0 985024 0 0 0
ioband2: 0 40355280 ioband 1 -1 6586 27 827456 0 0 0
ioband1: 0 37768752 ioband 1 -1 8327 0 1049624 0 0 0
Following are details of my test setup.
---------------------------------------
I took dm-ioband patch version 1.12.3 and applied on 2.6.31-rc6.
Created ioband devices using following command.
----------------------------------------------
echo "0 $(blockdev --getsize /dev/sdd1) ioband /dev/sdd1 1 0 0 none" "weight 0 :200" | dmsetup create ioband1
echo "0 $(blockdev --getsize /dev/sdd2) ioband /dev/sdd2 1 0 0 none" "weight 0 :100" | dmsetup create ioband2
mount /dev/mapper/ioband1 /mnt/sdd1
mount /dev/mapper/ioband2 /mnt/sdd2
Started two dd threads
======================
dd if=/mnt/sdd1/testzerofile1 of=/dev/null &
dd if=/mnt/sdd2/testzerofile1 of=/dev/null &
Output of dmsetup table command
================================
ioband2: 0 40355280 ioband 8:50 1 4 192 none weight 768 :100
ioband1: 0 37768752 ioband 8:49 1 4 192 none weight 768 :200
Thanks
Vivek
Quoted 15 lines · Vivek Goyal
Hi Ryo,
I decided to play a bit more with dm-ioband and started doing some testing. I am doing a simple two dd threads doing reads and don't seem to be gettting the fairness. So thought will ask you what's the issue here. Is there an issue with my testing procedure.
I got one 40G SATA drive (no hardware queuing). I have created two partitions on that disk
/dev/sdd1and/dev/sdd2and created two ioband devices ioband1 and ioband2 on partitions sdd1 and sdd2 respectively. The weights of ioband1 and ioband2 devices are 200 and 100 respectively.I am assuming that this setup will create two default groups and IO going to partition sdd1 should get double the BW of partition sdd2.
Quoted 9 lines
But it looks like I am not gettting that behavior. Following is the output of "dmsetup table" command. This snapshot has been taken every 2 seconds while IO was going on. Column 9 seems to be containing how many sectors of IO has been done on a particular io band device and group. Looking at the snapshot, it does not look like that ioband1 default group got double the BW of ioband2 default group.
Am I doing something wrong here?
I tried another variant of test. This time I also created two additional groups on ioband1 devices and linked these to cgroups test1 and test2 and launched two dd threads in two cgroups on device ioband1. There also I don't seem to be getting the right fairness numbers for cgroup test1 and test2.
Script to create ioband devices and additional groups
-----------------------------------------------------
echo "0 $(blockdev --getsize /dev/sdd1) ioband /dev/sdd1 1 0 0 none" "weight 0 :200" | dmsetup create ioband1
echo "0 $(blockdev --getsize /dev/sdd2) ioband /dev/sdd2 1 0 0 none" "weight 0 :100" | dmsetup create ioband2
# Some code to mount and create cgroups.
# Read group id
test1_id=cat /cgroup/ioband/test1/blkio.id
test2_id=cat /cgroup/ioband/test2/blkio.id
test1_weight=200
test2_weight=100
dmsetup message ioband1 0 type cgroup
dmsetup message ioband1 0 attach $test1_id dmsetup message ioband1 0 attach $test2_id dmsetup message ioband1 0 weight $test1_id:$test1_weight dmsetup message ioband1 0 weight $test2_id:$test2_weight
mount /dev/mapper/ioband1 /mnt/sdd1
mount /dev/mapper/ioband2 /mnt/sdd2
-----------------------------------------------------------------
Following are two dd jobs
-------------------------
dd if=/mnt/sdd1/testzerofile1 of=/dev/null &
echo $! > /cgroup/ioband/test1/tasks
dd if=/mnt/sdd1/testzerofile2 of=/dev/null &
echo $! > /cgroup/ioband/test2/tasks
Following are "dmsetup status" results every 2 seconds
======================================================
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 0 0 0 0 0 0 3 0 0 0 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 0 0 0 0 0 0 3 0 0 0 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 689 0 86336 0 0 0 3 650 3 81472 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 1725 0 217024 0 0 0 3 1270 11 158912 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 2690 0 338744 0 0 0 3 1978 15 247856 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 3762 0 474040 0 0 0 3 2583 21 323736 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 4745 0 598064 0 0 0 3 3275 27 410392 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 5737 0 723120 0 0 0 3 3985 31 499592 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 6815 0 859184 0 0 0 3 4594 37 575864 0 0 0
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 7823 0 986288 0 0 0 3 5276 43 661360 0 0 0
"dmsetup table" output
======================
ioband2: 0 40355280 ioband 8:50 1 4 192 none weight 768 :100
ioband1: 0 37768752 ioband 8:49 1 4 192 cgroup weight 768 :200 2:200 3:100
Because I am using "weight" policy, I thought that test1 cgroup with id "2" will issue double the number of requests of cgroup test2 with id "3". But that does not seem to be happening here. Is there an issue with my testing method.
Thanks
Vivek
Quoted 62 lines
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0ioband2: 0 40355280 ioband 1 -1 96 0 11528 0 0 0
ioband1: 0 37768752 ioband 1 -1 82 0 9736 0 0 0ioband2: 0 40355280 ioband 1 -1 748 2 93032 0 0 0
ioband1: 0 37768752 ioband 1 -1 896 0 112232 0 0 0ioband2: 0 40355280 ioband 1 -1 1326 5 165816 0 0 0
ioband1: 0 37768752 ioband 1 -1 1816 0 228312 0 0 0ioband2: 0 40355280 ioband 1 -1 1943 6 243712 0 0 0
ioband1: 0 37768752 ioband 1 -1 2692 0 338760 0 0 0ioband2: 0 40355280 ioband 1 -1 2461 10 308576 0 0 0
ioband1: 0 37768752 ioband 1 -1 3618 0 455608 0 0 0ioband2: 0 40355280 ioband 1 -1 3118 11 391352 0 0 0
ioband1: 0 37768752 ioband 1 -1 4406 0 555032 0 0 0ioband2: 0 40355280 ioband 1 -1 3734 15 468760 0 0 0
ioband1: 0 37768752 ioband 1 -1 5273 0 664328 0 0 0ioband2: 0 40355280 ioband 1 -1 4307 17 540784 0 0 0
ioband1: 0 37768752 ioband 1 -1 6181 0 778992 0 0 0ioband2: 0 40355280 ioband 1 -1 4930 19 619208 0 0 0
ioband1: 0 37768752 ioband 1 -1 7028 0 885728 0 0 0ioband2: 0 40355280 ioband 1 -1 5599 22 703280 0 0 0
ioband1: 0 37768752 ioband 1 -1 7815 0 985024 0 0 0ioband2: 0 40355280 ioband 1 -1 6586 27 827456 0 0 0
ioband1: 0 37768752 ioband 1 -1 8327 0 1049624 0 0 0Following are details of my test setup.
---------------------------------------
I took dm-ioband patch version 1.12.3 and applied on 2.6.31-rc6.Created ioband devices using following command.
----------------------------------------------
echo "0 $(blockdev --getsize/dev/sdd1) ioband/dev/sdd11 0 0 none" "weight 0 :200" | dmsetup create ioband1
echo "0 $(blockdev --getsize/dev/sdd2) ioband/dev/sdd21 0 0 none" "weight 0 :100" | dmsetup create ioband2mount
/dev/mapper/ioband1/mnt/sdd1
mount/dev/mapper/ioband2/mnt/sdd2Started two dd threads
======================
dd if=/mnt/sdd1/testzerofile1 of=/dev/null &
dd if=/mnt/sdd2/testzerofile1 of=/dev/null &Output of dmsetup table command
================================
ioband2: 0 40355280 ioband 8:50 1 4 192 none weight 768 :100
ioband1: 0 37768752 ioband 8:49 1 4 192 none weight 768 :200Thanks
Vivek
Quoted 23 lines · Vivek Goyal
Quoted 22 lines · Vivek Goyal
Hi Ryo,
I decided to play a bit more with dm-ioband and started doing some testing. I am doing a simple two dd threads doing reads and don't seem to be gettting the fairness. So thought will ask you what's the issue here. Is there an issue with my testing procedure.
I got one 40G SATA drive (no hardware queuing). I have created two partitions on that disk
/dev/sdd1and/dev/sdd2and created two ioband devices ioband1 and ioband2 on partitions sdd1 and sdd2 respectively. The weights of ioband1 and ioband2 devices are 200 and 100 respectively.I am assuming that this setup will create two default groups and IO going to partition sdd1 should get double the BW of partition sdd2.
But it looks like I am not gettting that behavior. Following is the output of "dmsetup table" command. This snapshot has been taken every 2 seconds while IO was going on. Column 9 seems to be containing how many sectors of IO has been done on a particular io band device and group. Looking at the snapshot, it does not look like that ioband1 default group got double the BW of ioband2 default group.
Quoted 3 lines
Quoted 2 lines
Am I doing something wrong here?
Hi Ryo,
Did you get a chance to look into it? Am I doing something wrong or it is an issue with dm-ioband.
Thanks
Vivek
Quoted 156 lines
I tried another variant of test. This time I also created two additional groups on ioband1 devices and linked these to cgroups test1 and test2 and launched two dd threads in two cgroups on device ioband1. There also I don't seem to be getting the right fairness numbers for cgroup test1 and test2.
Script to create ioband devices and additional groups
-----------------------------------------------------
echo "0 $(blockdev --getsize/dev/sdd1) ioband/dev/sdd11 0 0 none" "weight 0 :200" | dmsetup create ioband1
echo "0 $(blockdev --getsize/dev/sdd2) ioband/dev/sdd21 0 0 none" "weight 0 :100" | dmsetup create ioband2# Some code to mount and create cgroups.
# Read group id
test1_id=cat /cgroup/ioband/test1/blkio.id
test2_id=cat /cgroup/ioband/test2/blkio.idtest1_weight=200
test2_weight=100dmsetup message ioband1 0 type cgroup
dmsetup message ioband1 0 attach $test1_id dmsetup message ioband1 0 attach $test2_id dmsetup message ioband1 0 weight $test1_id:$test1_weight dmsetup message ioband1 0 weight $test2_id:$test2_weightmount
/dev/mapper/ioband1/mnt/sdd1
mount/dev/mapper/ioband2/mnt/sdd2
-----------------------------------------------------------------Following are two dd jobs
-------------------------
dd if=/mnt/sdd1/testzerofile1 of=/dev/null &
echo $! > /cgroup/ioband/test1/tasksdd if=/mnt/sdd1/testzerofile2 of=/dev/null &
echo $! > /cgroup/ioband/test2/tasksFollowing are "dmsetup status" results every 2 seconds
======================================================ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 0 0 0 0 0 0 3 0 0 0 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 0 0 0 0 0 0 3 0 0 0 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 689 0 86336 0 0 0 3 650 3 81472 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 1725 0 217024 0 0 0 3 1270 11 158912 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 2690 0 338744 0 0 0 3 1978 15 247856 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 3762 0 474040 0 0 0 3 2583 21 323736 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 4745 0 598064 0 0 0 3 3275 27 410392 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 5737 0 723120 0 0 0 3 3985 31 499592 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 6815 0 859184 0 0 0 3 4594 37 575864 0 0 0ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0 2 7823 0 986288 0 0 0 3 5276 43 661360 0 0 0"dmsetup table" output
======================
ioband2: 0 40355280 ioband 8:50 1 4 192 none weight 768 :100
ioband1: 0 37768752 ioband 8:49 1 4 192 cgroup weight 768 :200 2:200 3:100Because I am using "weight" policy, I thought that test1 cgroup with id "2" will issue double the number of requests of cgroup test2 with id "3". But that does not seem to be happening here. Is there an issue with my testing method.
Thanks
VivekQuoted 62 lines
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0ioband2: 0 40355280 ioband 1 -1 96 0 11528 0 0 0
ioband1: 0 37768752 ioband 1 -1 82 0 9736 0 0 0ioband2: 0 40355280 ioband 1 -1 748 2 93032 0 0 0
ioband1: 0 37768752 ioband 1 -1 896 0 112232 0 0 0ioband2: 0 40355280 ioband 1 -1 1326 5 165816 0 0 0
ioband1: 0 37768752 ioband 1 -1 1816 0 228312 0 0 0ioband2: 0 40355280 ioband 1 -1 1943 6 243712 0 0 0
ioband1: 0 37768752 ioband 1 -1 2692 0 338760 0 0 0ioband2: 0 40355280 ioband 1 -1 2461 10 308576 0 0 0
ioband1: 0 37768752 ioband 1 -1 3618 0 455608 0 0 0ioband2: 0 40355280 ioband 1 -1 3118 11 391352 0 0 0
ioband1: 0 37768752 ioband 1 -1 4406 0 555032 0 0 0ioband2: 0 40355280 ioband 1 -1 3734 15 468760 0 0 0
ioband1: 0 37768752 ioband 1 -1 5273 0 664328 0 0 0ioband2: 0 40355280 ioband 1 -1 4307 17 540784 0 0 0
ioband1: 0 37768752 ioband 1 -1 6181 0 778992 0 0 0ioband2: 0 40355280 ioband 1 -1 4930 19 619208 0 0 0
ioband1: 0 37768752 ioband 1 -1 7028 0 885728 0 0 0ioband2: 0 40355280 ioband 1 -1 5599 22 703280 0 0 0
ioband1: 0 37768752 ioband 1 -1 7815 0 985024 0 0 0ioband2: 0 40355280 ioband 1 -1 6586 27 827456 0 0 0
ioband1: 0 37768752 ioband 1 -1 8327 0 1049624 0 0 0Following are details of my test setup.
---------------------------------------
I took dm-ioband patch version 1.12.3 and applied on 2.6.31-rc6.Created ioband devices using following command.
----------------------------------------------
echo "0 $(blockdev --getsize/dev/sdd1) ioband/dev/sdd11 0 0 none" "weight 0 :200" | dmsetup create ioband1
echo "0 $(blockdev --getsize/dev/sdd2) ioband/dev/sdd21 0 0 none" "weight 0 :100" | dmsetup create ioband2mount
/dev/mapper/ioband1/mnt/sdd1
mount/dev/mapper/ioband2/mnt/sdd2Started two dd threads
======================
dd if=/mnt/sdd1/testzerofile1 of=/dev/null &
dd if=/mnt/sdd2/testzerofile1 of=/dev/null &Output of dmsetup table command
================================
ioband2: 0 40355280 ioband 8:50 1 4 192 none weight 768 :100
ioband1: 0 37768752 ioband 8:49 1 4 192 none weight 768 :200Thanks
Vivek
Hi Vivek,
Quoted 24 lines · Vivek Goyal
Quoted 23 lines · Vivek Goyal
Quoted 22 lines · Vivek Goyal
Hi Ryo,
I decided to play a bit more with dm-ioband and started doing some testing. I am doing a simple two dd threads doing reads and don't seem to be gettting the fairness. So thought will ask you what's the issue here. Is there an issue with my testing procedure.
I got one 40G SATA drive (no hardware queuing). I have created two partitions on that disk
/dev/sdd1and/dev/sdd2and created two ioband devices ioband1 and ioband2 on partitions sdd1 and sdd2 respectively. The weights of ioband1 and ioband2 devices are 200 and 100 respectively.I am assuming that this setup will create two default groups and IO going to partition sdd1 should get double the BW of partition sdd2.
But it looks like I am not gettting that behavior. Following is the output of "dmsetup table" command. This snapshot has been taken every 2 seconds while IO was going on. Column 9 seems to be containing how many sectors of IO has been done on a particular io band device and group. Looking at the snapshot, it does not look like that ioband1 default group got double the BW of ioband2 default group.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
Am I doing something wrong here?
Hi Ryo,
Did you get a chance to look into it? Am I doing something wrong or it is an issue with dm-ioband.
Sorry, I missed it. I'll look into it and report back to you.
Thanks,
Ryo Tsuruta
dm-ioband fairness in terms of sectors seems to be killing disk (Was: Re: Regarding dm-ioband tests)
Quoted 31 lines · Ryo Tsuruta
Hi Vivek,
Quoted 28 lines · Vivek Goyal
Quoted 26 lines · Vivek Goyal
Quoted 24 lines · Vivek Goyal
Hi Ryo,
I decided to play a bit more with dm-ioband and started doing some testing. I am doing a simple two dd threads doing reads and don't seem to be gettting the fairness. So thought will ask you what's the issue here. Is there an issue with my testing procedure.
I got one 40G SATA drive (no hardware queuing). I have created two partitions on that disk
/dev/sdd1and/dev/sdd2and created two ioband devices ioband1 and ioband2 on partitions sdd1 and sdd2 respectively. The weights of ioband1 and ioband2 devices are 200 and 100 respectively.I am assuming that this setup will create two default groups and IO going to partition sdd1 should get double the BW of partition sdd2.
But it looks like I am not gettting that behavior. Following is the output of "dmsetup table" command. This snapshot has been taken every 2 seconds while IO was going on. Column 9 seems to be containing how many sectors of IO has been done on a particular io band device and group. Looking at the snapshot, it does not look like that ioband1 default group got double the BW of ioband2 default group.
Am I doing something wrong here?
Quoted 6 lines
Quoted 4 lines
Hi Ryo,
Did you get a chance to look into it? Am I doing something wrong or it is an issue with dm-ioband.
Sorry, I missed it. I'll look into it and report back to you.
Hi Ryo,
I am running a sequential reader in one group and few random reader and writers in second group. Both groups are of same weight. I ran fio scripts for 60 seconds and then looked at the output. In this case looks like we just kill the throughput of sequential reader and disk (because random readers/writers take over).
I ran the test "with-dm-ioband", "without-dm-ioband" and "with ioscheduler based io controller".
First I am pasting the results and in the end I will paste my test scripts. I have cut fio output heavily so that we does not get lost in lots of output.
with-dm-ioband
==============
ioband1
-------
randread: (groupid=0, jobs=4): err= 0: pid=3610
read : io=18,432KiB, bw=314KiB/s, iops=76, runt= 60076msec
clat (usec): min=140, max=744K, avg=50866.75, stdev=61266.88
randwrite: (groupid=1, jobs=2): err= 0: pid=3614
write: io=920KiB, bw=15KiB/s, iops=3, runt= 60098msec
clat (usec): min=203, max=14,171K, avg=522937.86, stdev=960929.44
ioband2
-------
seqread0: (groupid=0, jobs=1): err= 0: pid=3609
read : io=37,904KiB, bw=636KiB/s, iops=155, runt= 61026msec
clat (usec): min=92, max=9,969K, avg=6437.89, stdev=168573.23
without dm-ioband (vanilla cfq, no grouping)
============================================
seqread0: (groupid=0, jobs=1): err= 0: pid=3969
read : io=321MiB, bw=5,598KiB/s, iops=1,366, runt= 60104msec
clat (usec): min=91, max=763K, avg=729.61, stdev=17402.63
randread: (groupid=0, jobs=4): err= 0: pid=3970
read : io=15,112KiB, bw=257KiB/s, iops=62, runt= 60039msec
clat (usec): min=124, max=1,066K, avg=63721.26, stdev=78215.17
randwrite: (groupid=1, jobs=2): err= 0: pid=3974
write: io=680KiB, bw=11KiB/s, iops=2, runt= 60073msec
clat (usec): min=199, max=24,646K, avg=706719.51, stdev=1774887.55
With ioscheduer based io controller patches
===========================================
cgroup 1 (weight 100)
---------------------
randread: (groupid=0, jobs=4): err= 0: pid=2995
read : io=9,484KiB, bw=161KiB/s, iops=39, runt= 60107msec
clat (msec): min=1, max=2,167, avg=95.47, stdev=131.60
randwrite: (groupid=1, jobs=2): err= 0: pid=2999
write: io=2,692KiB, bw=45KiB/s, iops=11, runt= 60131msec
clat (usec): min=199, max=30,043K, avg=178710.05, stdev=1281485.75
cgroup 2 (weight 100)
--------------------
seqread0: (groupid=0, jobs=1): err= 0: pid=2993
read : io=547mib, bw=9,556kib/s, iops=2,333, runt= 60043msec
clat (usec): min=92, max=224k, avg=426.74, stdev=5734.12
Note the BW of sequential reader in three cases
(636 KB/s, 5,598KiB/s, 9,556KiB/s). dm-ioband tries to provide fairness in terms of number of sectors and it completely kills the disk throughput.
with io scheduler based io controller, we see increased throughput for seqential reader as compared to CFQ, because now random readers are running in a separate group and hence reader gets isolation from random readers.
Here are my fio jobs
--------------------
First fio job file
-----------------
[global]
runtime=60
[randread]
rw=randread
size=2G
iodepth=20
directory=/mnt/sdd1/fio/
direct=1
numjobs=4group_reporting
[randwrite]
rw=randwrite
size=1G
iodepth=20
directory=/mnt/sdd1/fio/group_reporting
direct=1
numjobs=2
Second fio job file
-------------------
[global]
runtime=60
rw=read
size=4G
directory=/mnt/sdd2/fio/
direct=1
[seqread0]
numjobs=1group_reporting
Thanks
Vivek
Re: dm-ioband fairness in terms of sectors seems to be killing disk
Hi Vivek,
Quoted 7 lines · Vivek Goyal
Hi Ryo,
I am running a sequential reader in one group and few random reader and writers in second group. Both groups are of same weight. I ran fio scripts for 60 seconds and then looked at the output. In this case looks like we just kill the throughput of sequential reader and disk (because random readers/writers take over).
Thank you for testing dm-ioband.
I ran your script on my environment, and here are the results.
Throughput [KiB/s]
vanilla dm-ioband dm-ioband
(io-throttle = 4) (io-throttle = 50)
randread 312 392 368
randwrite 11 12 10
seqread 4341 651 1599
I ran the script on dm-ioband under two conditions, one is that the io-throttle options is set to 4, and the other is set to 50. When there are some in-flight IO requests in the group and those numbers exceed io-throttle, then dm-ioband gives priority to the group and the group can issue subsequent IO requests in preference to the other groups. 50 io-throttle means that it cancels this mechanism, so the seq-read got more bandwidth than 4 io-throttle.
I tried to test with 2.6.31-rc7 and io-controller v9, but unfortunately, a kernel panic happened. I'll try to test with your io-controller again later.
Quoted 4 lines
with io scheduler based io controller, we see increased throughput for seqential reader as compared to CFQ, because now random readers are running in a separate group and hence reader gets isolation from random readers.
I summarized your results in a tabular format.
Throughput [KiB/s]
vanilla io-controller dm-ioband
randread 257 161 314
randwrite 11 45 15
seqread 5598 9556 631
On the result of io-controller, the throughput of seqread was increased but randread was decreased against vanilla. Did it perform as you expected? Was disktime consumed equally on each group according to the weight settings? Could you tell me your opinion what an io-controller should do when this kind of workload is applied?
Thanks,
Ryo Tsuruta
Hi Vivek,
Quoted 6 lines · Vivek Goyal
Hi Ryo,
I decided to play a bit more with dm-ioband and started doing some testing. I am doing a simple two dd threads doing reads and don't seem to be gettting the fairness. So thought will ask you what's the issue here. Is there an issue with my testing procedure.
Thank you for testing dm-ioband. dm-ioband is designed to start throttling bandwidth when multiple IO requests are issued to devices simultaneously, IOW, to start throttling when IO load exceeds a certain level.
Here is my test script that runs multiple dd threads on each directory. Each directory stores 20 files of 2GB.
#!/bin/sh
tmout=60
for nr_threads in 1 4 8 12 16 20; do
sync; echo 3 > /proc/sys/vm/drop_caches
for i in $(seq $nr_threads); do
dd if=/mnt1/ioband1.${i}.0 of=/dev/null &
dd if=/mnt2/ioband2.${i}.0 of=/dev/null &
done
iostat -k 1 $tmout > ${nr_threads}.log
killall -ws TERM dd
done
exit 0Here is the result. The average throughputs of each device are according to the proportion of the weight settings when the number of thread is over four.
Average thoughput in 60 seconds [KB/s]
ioband1 ioband2
threads weight 200 weight 100 total
1 26642 (54.9%) 21925 (45.1%) 48568
4 33974 (67.7%) 16181 (32.3%) 50156
8 31952 (66.2%) 16297 (33.8%) 48249
12 32062 (67.8%) 15236 (32.2%) 47299
16 31780 (67.7%) 15165 (32.3%) 46946
20 29955 (66.3%) 15239 (33.7%) 45195Please try to run the above script on your envirionment and I would be glad if you let me know the result.
Thanks,
Ryo Tsuruta
Quoted 10 lines · Ryo Tsuruta
Hi Vivek,
Quoted 6 lines · Vivek Goyal
Hi Ryo,
I decided to play a bit more with dm-ioband and started doing some testing. I am doing a simple two dd threads doing reads and don't seem to be gettting the fairness. So thought will ask you what's the issue here. Is there an issue with my testing procedure.
[ CCing relevant folks on thread as what does fairness mean is becoming
interesting]
Quoted 6 lines
Thank you for testing dm-ioband. dm-ioband is designed to start throttling bandwidth when multiple IO requests are issued to devices simultaneously, IOW, to start throttling when IO load exceeds a certain level.
What is that certain level? Secondly what's the advantage of this?
I can see disadvantages though. So unless a group is really busy "up to that certain level" it will not get fairness? I breaks the isolation between groups.
Quoted 33 lines
Here is my test script that runs multiple dd threads on each directory. Each directory stores 20 files of 2GB.
#!/bin/sh tmout=60 for nr_threads in 1 4 8 12 16 20; do sync; echo 3 > /proc/sys/vm/drop_caches for i in $(seq $nr_threads); do dd if=/mnt1/ioband1.${i}.0 of=/dev/null & dd if=/mnt2/ioband2.${i}.0 of=/dev/null & done iostat -k 1 $tmout > ${nr_threads}.log killall -ws TERM dd done exit 0Here is the result. The average throughputs of each device are according to the proportion of the weight settings when the number of thread is over four.
Average thoughput in 60 seconds [KB/s]
ioband1 ioband2
threads weight 200 weight 100 total 1 26642 (54.9%) 21925 (45.1%) 48568 4 33974 (67.7%) 16181 (32.3%) 50156 8 31952 (66.2%) 16297 (33.8%) 48249 12 32062 (67.8%) 15236 (32.2%) 47299 16 31780 (67.7%) 15165 (32.3%) 46946 20 29955 (66.3%) 15239 (33.7%) 45195
Quoted 2 lines
Please try to run the above script on your envirionment and I would be glad if you let me know the result.
I ran my simple dd test again with two ioband deviecs of weight 200 (ioband1) and 100 (ioband2)respectively. I launched four sequential dd readers on ioband2 and and one sequential reader in ioband1.
Now if we are providing isolation between groups then ioband1 should get double the bandwidth of ioband1. But that does not happen. Following is the output of "dmsetup table" command.
Fri Sep 4 18:02:01 EDT 2009
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0
Fri Sep 4 18:02:08 EDT 2009
ioband2: 0 40355280 ioband 1 -1 2031 32 250280 0 0 0
ioband1: 0 37768752 ioband 1 -1 1484 0 186544 0 0 0
Fri Sep 4 18:02:12 EDT 2009
ioband2: 0 40355280 ioband 1 -1 3541 64 437192 0 0 0
ioband1: 0 37768752 ioband 1 -1 2802 0 352728 0 0 0
Fri Sep 4 18:02:16 EDT 2009
ioband2: 0 40355280 ioband 1 -1 5200 87 644144 0 0 0
ioband1: 0 37768752 ioband 1 -1 4003 0 504296 0 0 0
Fri Sep 4 18:02:20 EDT 2009
ioband2: 0 40355280 ioband 1 -1 7632 111 948232 0 0 0
ioband1: 0 37768752 ioband 1 -1 4494 0 566080 0 0 0
This seems to be breaking the isolation between two groups. Now if there is one bad group with lots of readers and writers of lower weight, it will overwhelm a group of higher weight with 1-2 readers running or some random readers running etc.
If there are lots of readers running in a group and then a small file reader comes in a different group of higher prio, it will not get any fairness and latency of file read will be very high. But one would expect that groups will provide isolation and latency of small file reader will not increase with number of readers in a low prio group.
I also ran your test of doing heavy IO in two groups. This time I am running 4 dd threads in both the ioband devices. Following is the snapshot of "dmsetup table" output.
Fri Sep 4 17:45:27 EDT 2009
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0
Fri Sep 4 17:45:29 EDT 2009
ioband2: 0 40355280 ioband 1 -1 41 0 4184 0 0 0
ioband1: 0 37768752 ioband 1 -1 173 0 20096 0 0 0
Fri Sep 4 17:45:37 EDT 2009
ioband2: 0 40355280 ioband 1 -1 1605 23 197976 0 0 0
ioband1: 0 37768752 ioband 1 -1 4640 1 583168 0 0 0
Fri Sep 4 17:45:45 EDT 2009
ioband2: 0 40355280 ioband 1 -1 3650 47 453488 0 0 0
ioband1: 0 37768752 ioband 1 -1 8572 1 1079144 0 0 0
Fri Sep 4 17:45:51 EDT 2009
ioband2: 0 40355280 ioband 1 -1 5111 68 635696 0 0 0
ioband1: 0 37768752 ioband 1 -1 11587 1 1459544 0 0 0
Fri Sep 4 17:45:53 EDT 2009
ioband2: 0 40355280 ioband 1 -1 5698 73 709272 0 0 0
ioband1: 0 37768752 ioband 1 -1 12503 1 1575112 0 0 0
Fri Sep 4 17:45:57 EDT 2009
ioband2: 0 40355280 ioband 1 -1 6790 87 845808 0 0 0
ioband1: 0 37768752 ioband 1 -1 14395 2 1813680 0 0 0
Note, it took me more than 20 seconds (since I started the threds) to reach close to desired fairness level. That's too long a duration. Again random readers or small file readers are compeltely out of picture for any kind of fairness or are not protected at all with dm-ioband controller.
I think there are serious issues with the notion of fairness and what kind of isolation dm-ioband provide between groups and it should be looked into.
Thanks
Vivek
Hi Vivek,
Quoted 11 lines · Vivek Goyal
Quoted 5 lines
Thank you for testing dm-ioband. dm-ioband is designed to start throttling bandwidth when multiple IO requests are issued to devices simultaneously, IOW, to start throttling when IO load exceeds a certain level.
What is that certain level? Secondly what's the advantage of this?
I can see disadvantages though. So unless a group is really busy "up to that certain level" it will not get fairness? I breaks the isolation between groups.
In your test case, at least more than one dd thread have to run simultaneously in the higher weight group. The reason is that if there is an IO group which does not issue a certain number of IO requests, dm-ioband assumes the IO group is inactive and assign its spare bandwidth to active IO groups. Then whole bandwidth of the device can be efficiently used. Please run two dd threads in the higher group, it will work as you expect.
However, if you want to get fairness in a case like this, a new bandwidth control policy which controls accurately according to assigned weights can be added to dm-ioband.
Quoted 28 lines
I also ran your test of doing heavy IO in two groups. This time I am running 4 dd threads in both the ioband devices. Following is the snapshot of "dmsetup table" output.
Fri Sep 4 17:45:27 EDT 2009
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0Fri Sep 4 17:45:29 EDT 2009
ioband2: 0 40355280 ioband 1 -1 41 0 4184 0 0 0
ioband1: 0 37768752 ioband 1 -1 173 0 20096 0 0 0Fri Sep 4 17:45:37 EDT 2009
ioband2: 0 40355280 ioband 1 -1 1605 23 197976 0 0 0
ioband1: 0 37768752 ioband 1 -1 4640 1 583168 0 0 0Fri Sep 4 17:45:45 EDT 2009
ioband2: 0 40355280 ioband 1 -1 3650 47 453488 0 0 0
ioband1: 0 37768752 ioband 1 -1 8572 1 1079144 0 0 0Fri Sep 4 17:45:51 EDT 2009
ioband2: 0 40355280 ioband 1 -1 5111 68 635696 0 0 0
ioband1: 0 37768752 ioband 1 -1 11587 1 1459544 0 0 0Fri Sep 4 17:45:53 EDT 2009
ioband2: 0 40355280 ioband 1 -1 5698 73 709272 0 0 0
ioband1: 0 37768752 ioband 1 -1 12503 1 1575112 0 0 0
Quoted 6 lines
Fri Sep 4 17:45:57 EDT 2009
ioband2: 0 40355280 ioband 1 -1 6790 87 845808 0 0 0
ioband1: 0 37768752 ioband 1 -1 14395 2 1813680 0 0 0Note, it took me more than 20 seconds (since I started the threds) to reach close to desired fairness level. That's too long a duration.
We regarded reducing throughput loss rather than reducing duration as the design of dm-ioband. Of course, it is possible to make a new policy which reduces duration.
Thanks,
Ryo Tsuruta
Quoted 9 lines
Again random readers or small file readers are compeltely out of picture for any kind of fairness or are not protected at all with dm-ioband controller.
I think there are serious issues with the notion of fairness and what kind of isolation dm-ioband provide between groups and it should be looked into.
Thanks
Vivek
Quoted 3 lines · Ryo Tsuruta
However, if you want to get fairness in a case like this, a new bandwidth control policy which controls accurately according to assigned weights can be added to dm-ioband.
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Quoted 3 lines
We regarded reducing throughput loss rather than reducing duration as the design of dm-ioband. Of course, it is possible to make a new policy which reduces duration.
... while also reducing overall system throughput by design?
Why are you even bothering to submit this to the linux-kernel mailing list, when there is a codebase available that has no throughput or fairness regressions? (Vivek's io scheduler based io controler)
Signature · 1 line
All rights reversed.
Hi Rik,
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Ryo Tsuruta
However, if you want to get fairness in a case like this, a new bandwidth control policy which controls accurately according to assigned weights can be added to dm-ioband.
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
Quoted 6 lines
Quoted 3 lines
We regarded reducing throughput loss rather than reducing duration as the design of dm-ioband. Of course, it is possible to make a new policy which reduces duration.
... while also reducing overall system throughput by design?
I think it reduces system throughput compared to the current implementation, because it causes more overhead to do fine grained control.
Quoted 4 lines
Why are you even bothering to submit this to the linux-kernel mailing list, when there is a codebase available that has no throughput or fairness regressions? (Vivek's io scheduler based io controler)
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
Thanks,
Ryo Tsuruta
Quoted 10 lines · Ryo Tsuruta
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
But don't you need page_cgroup for IO tracking?
Signature · 1 line
Balbir
Hi Balbir,
Quoted 13 lines · Balbir Singh
Quoted 10 lines · Ryo Tsuruta
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
Quoted 1 line
But don't you need
page_cgroupfor IO tracking?
It is not necessary when controlling bandwidth on a per partition basis or on a IO thread basis like Xen blkback kernel thread.
Here are configration examples.
http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples
Thanks,
Ryo Tsuruta
Quoted 16 lines · Ryo Tsuruta
Hi Balbir,
Quoted 13 lines · Balbir Singh
Quoted 10 lines · Ryo Tsuruta
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
Quoted 8 lines
Quoted 1 line
But don't you need
page_cgroupfor IO tracking?It is not necessary when controlling bandwidth on a per partition basis or on a IO thread basis like Xen blkback kernel thread.
Here are configration examples.
http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples
For partition based control, where a thread or group of threads is doing IO to a specific parition, why can't you simply create different cgroups for each partition and move threads in those partitions.
root / | \ sda1 sda2 sda3
Above are three groups and move threads doing IO into those groups and problem is solved. In fact that's what one will do for KVM virtual machines. Move all the qemu helper threds doing IO for a virtual machine instance into a specific group and control the IO.
Why do you have to come up with additional complicated grouping mechanism?
Thanks
Vivek
Hi Vivek,
Quoted 16 lines · Vivek Goyal
Quoted 6 lines
It is not necessary when controlling bandwidth on a per partition basis or on a IO thread basis like Xen blkback kernel thread.
Here are configration examples.
http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examplesFor partition based control, where a thread or group of threads is doing IO to a specific parition, why can't you simply create different cgroups for each partition and move threads in those partitions.
root / | \ sda1 sda2 sda3
Quoted 6 lines
Above are three groups and move threads doing IO into those groups and problem is solved. In fact that's what one will do for KVM virtual machines. Move all the qemu helper threds doing IO for a virtual machine instance into a specific group and control the IO.
Why do you have to come up with additional complicated grouping mechanism?
I don't get why you think it's complicated, your io-controller also provides the same grouping machanism which assigns bandwidth per device by io.policy file. What's the difference? The thread grouping machianism is also not special, it is the same concept as cgroup. These mechanisms are necessary to make use of dm-ioband on the systems which doesn't support cgroup such as RHEL 5.x. As you know, dm-ioband also supports cgroup, the configurations you mentioned above can apply to the system by dm-ioband. I think it's not bad to have several ways to setup.
Thanks,
Ryo Tsuruta
Quoted 24 lines · Ryo Tsuruta
Hi Vivek,
Quoted 21 lines · Vivek Goyal
Quoted 6 lines
It is not necessary when controlling bandwidth on a per partition basis or on a IO thread basis like Xen blkback kernel thread.
Here are configration examples.
http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examplesFor partition based control, where a thread or group of threads is doing IO to a specific parition, why can't you simply create different cgroups for each partition and move threads in those partitions.
root / | \ sda1 sda2 sda3Above are three groups and move threads doing IO into those groups and problem is solved. In fact that's what one will do for KVM virtual machines. Move all the qemu helper threds doing IO for a virtual machine instance into a specific group and control the IO.
Quoted 5 lines
Quoted 1 line
Why do you have to come up with additional complicated grouping mechanism?
I don't get why you think it's complicated, your io-controller also provides the same grouping machanism which assigns bandwidth per device by io.policy file. What's the difference?
I am using purely cgroup based interface. This makes life easier for user space tools and libraries like libcgroup and should also help libvirt. Now they can treat all the resource controllers in kernel in a uniform way (through cgroup interface).
With-in cgroups, there are controller specific files and libcgroup is aware of that. So libcgroup can be modified to take care of special syntax of io.policy file. But this is not too big a deviation in overall picture.
dm-ioband is coming up with a whole new way of configuring and managing groups and now these user space tools shall have to be modified to take care of this new way only for io controller.
The point is there is no need. You seem to be introducing this new interface because you want to use this module with RHEL5 which does not have cgroup support.
I think this is hard to sell argument that upstream should also introduce a new interface just because one wants to use the module with older releases of kernel which did not have cgroup support.
Taking this new interface solves your case but will make life harder for user space tools, libraries and applications making use of cgroups and various resource controllers.
Quoted 6 lines
The thread grouping
machianism is also not special, it is the same concept as cgroup. These mechanisms are necessary to make use of dm-ioband on the systems which doesn't support cgroup such as RHEL 5.x. As you know, dm-ioband also supports cgroup, the configurations you mentioned above can apply to the system by dm-ioband.
dm-ioband also supports cgroup but there is additional step required and that is passing all the cgroup ids to various ioband devices. This requires knowledge of all the ioband devices and how these have been created and usage of dm tools.
The only place where it helps a bit is that once the configuration is done, one can move the tasks in groups to arbitrarily group them instead of grouping these on pid, gid etc....
So it still does not solve the issue of dm-ioband being so different from rest of the controllers and introducing a new interface for creation and management of groups.
Quoted 2 lines
I think it's not bad to have several ways
to setup.
It is not bad if there is a proper justification for new interface and why existing standard mechanism does not meet that requirement.
In this case you are saying that in general cgroup mechanism is sufficient to take care of grouping of tasks but it is not available in older kernels hence let us introduce a new interface in upstream kernels. I think this does not work. This brings in unnecessary overhead of maintaining anohter interface for upstream and upstream does not benefit from this interface.
Thanks
Vivek
Quoted 9 lines · Ryo Tsuruta
Hi Rik,
Quoted 6 lines · Rik van Riel
Quoted 3 lines · Ryo Tsuruta
However, if you want to get fairness in a case like this, a new bandwidth control policy which controls accurately according to assigned weights can be added to dm-ioband.
Quoted 7 lines
Quoted 2 lines
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
This group is running a sequential reader. How can you call it an inactive group?
I think that whole problem is that like CFQ you have not taken care of idling into account.
Your solution seems to be designed only for processes doing bulk IO over a very long period of time. I think it limits the usefulness of solution severely.
Quoted 16 lines
Quoted 6 lines
Quoted 3 lines
We regarded reducing throughput loss rather than reducing duration as the design of dm-ioband. Of course, it is possible to make a new policy which reduces duration.
... while also reducing overall system throughput by design?
I think it reduces system throughput compared to the current implementation, because it causes more overhead to do fine grained control.
Quoted 4 lines
Why are you even bothering to submit this to the linux-kernel mailing list, when there is a codebase available that has no throughput or fairness regressions? (Vivek's io scheduler based io controler)
Quoted 6 lines
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
I think we can easily extent io scheduler based controller to also support max rate per group policy also. That should not be too hard. It is a matter of only keeping track of io rate per group and if a group is exceeding the rate, then schedule it out and move on to next group.
I can do that once proportional weight solution is stablized and gets merged.
So its not an advantage of dm-ioband.
Quoted 2 lines
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
We talked about this point in the past also. In io scheduler based controller, just move all the tasks to root group and you got a system not doing any io control.
By the way why would one like to do that?
So this is also not an advantage.
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
I don't understand what is that core requirement of yours which is not met by io scheduler based io controller. range policy control you have implemented recently. I don't think that removing dm-ioband module dynamically is core requirement. Also whatever you can do with additional grouping mechanism, you can do with cgroup also.
So if there is any of your core functionality which is not fulfilled by io scheduler based controller, please let me know. I will be happy to look into it and try to provide that feature. But looking at above list, I am not convinced that any of the above is a compelling argument for dm-ioband inclusion.
Thanks
Vivek
Quoted 18 lines · Vivek Goyal
Quoted 9 lines · Ryo Tsuruta
Hi Rik,
Quoted 6 lines · Rik van Riel
Quoted 3 lines · Ryo Tsuruta
However, if you want to get fairness in a case like this, a new bandwidth control policy which controls accurately according to assigned weights can be added to dm-ioband.
Quoted 7 lines
Quoted 2 lines
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
Quoted 5 lines
This group is running a sequential reader. How can you call it an inactive group?
I think that whole problem is that like CFQ you have not taken care of idling into account.
I think this is probably the key deal breaker. dm-ioband has no mechanism to anticipate or idle for a reader task. Without such a mechanism, a proportional division scheme cannot work for tasks doing reads. Most readers do not send down more than one IO at a time, and they do not send another until the previous one is complete. Anticipation helps in this case, as we would wait for the task to send down a new IO, before we expire its timeslice. Without anticipation, we would serve the one IO from reader and then go on to serve IOs from other tasks. When the reader would finally get around to sending next IO, it would have to wait behind other IOs that have sent down in the meanwhile.
IO schedulers in block layer have anticipation built into them, so a proportional scheduling scheduling at that layer does not have to repeat the logic or data structures for anticipation.
In fact, a rate limiting mechanism like dm-ioband can potentially break the anticipation logic at IO schedulers, by queuing up the IOs at an upper layer, while scheduler in block layer could have been anticipating for it.
Quoted 24 lines
Your solution seems to be designed only for processes doing bulk IO over a very long period of time. I think it limits the usefulness of solution severely.
Quoted 19 lines
Quoted 6 lines
Quoted 3 lines
We regarded reducing throughput loss rather than reducing duration as the design of dm-ioband. Of course, it is possible to make a new policy which reduces duration.
... while also reducing overall system throughput by design?
I think it reduces system throughput compared to the current implementation, because it causes more overhead to do fine grained control.
Quoted 4 lines
Why are you even bothering to submit this to the linux-kernel mailing list, when there is a codebase available that has no throughput or fairness regressions? (Vivek's io scheduler based io controler)
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
Quoted 8 lines
Quoted 3 lines
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
I think we can easily extent io scheduler based controller to also support max rate per group policy also. That should not be too hard. It is a matter of only keeping track of io rate per group and if a group is exceeding the rate, then schedule it out and move on to next group.
At Google, we have implemented a rate limiting mechanism on top of Vivek's patches, and have been testing it. But I feel like the patch set maintained by Vivek is pretty big already. Once we have those patches merged, we can introduce more functionality.
Quoted 40 lines
I can do that once proportional weight solution is stablized and gets merged.
So its not an advantage of dm-ioband.
Quoted 2 lines
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
We talked about this point in the past also. In io scheduler based controller, just move all the tasks to root group and you got a system not doing any io control.
By the way why would one like to do that?
So this is also not an advantage.
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
I don't understand what is that core requirement of yours which is not met by io scheduler based io controller. range policy control you have implemented recently. I don't think that removing dm-ioband module dynamically is core requirement. Also whatever you can do with additional grouping mechanism, you can do with cgroup also.
So if there is any of your core functionality which is not fulfilled by io scheduler based controller, please let me know. I will be happy to look into it and try to provide that feature. But looking at above list, I am not convinced that any of the above is a compelling argument for dm-ioband inclusion.
Thanks
Vivek
Quoted 4 lines · Nauman Rafique
I think this is probably the key deal breaker. dm-ioband has no mechanism to anticipate or idle for a reader task. Without such a mechanism, a proportional division scheme cannot work for tasks doing reads.
That is a really big issue, since most reads tend to be synchronous (the application is waiting for the read), while many writes are not (the application is doing something else while the data is written).
Having writes take precedence over reads will really screw over the readers, while not benefitting the writers all that much.
Signature · 1 line
All rights reversed.
Quoted 7 lines · Rik van Riel
Quoted 4 lines · Nauman Rafique
I think this is probably the key deal breaker. dm-ioband has no mechanism to anticipate or idle for a reader task. Without such a mechanism, a proportional division scheme cannot work for tasks doing reads.
Quoted 7 lines
That is a really big issue, since most reads tend to be synchronous (the application is waiting for the read), while many writes are not (the application is doing something else while the data is written).
Having writes take precedence over reads will really screw over the readers, while not benefitting the writers all that much.
I ran a test to show how readers can be starved in certain cases. I launched one reader and three writers. I ran this test twice. First without dm-ioband and then with dm-ioband.
Following are few lines from the script to launch readers and writers.
**************************************************************
sync
echo 3 > /proc/sys/vm/drop_caches
# Launch writers on sdd2
dd if=/dev/zero of=/mnt/sdd2/writezerofile1 bs=4K count=262144 &
# Launch writers on sdd1
dd if=/dev/zero of=/mnt/sdd1/writezerofile1 bs=4K count=262144 & dd if=/dev/zero of=/mnt/sdd1/writezerofile2 bs=4K count=262144 &
echo "sleeping for 5 seconds"
sleep 5
# launch reader on sdd1 time dd if=/mnt/sdd1/testzerofile1 of=/dev/zero & echo "launched reader $!" *********************************************************************
Without dm-ioband, reader finished in roughly 5 seconds.
289533952 bytes (290 MB) copied, 5.16765 s, 56.0 MB/s
real 0m5.300s
user 0m0.098s
sys 0m0.492s
With dm-ioband, reader took, more than 2 minutes to finish.
289533952 bytes (290 MB) copied, 122.386 s, 2.4 MB/s
real 2m2.569s
user 0m0.107s
sys 0m0.548s
I had created ioband1 on /dev/sdd1 and ioband2 on /dev/sdd2 with weights 200 and 100 respectively.
Thanks
Vivek
ioband: Writer starves reader even without competitors (Re: Regarding dm-ioband tests)
On Tue, Sep 08, 2009 at 01:54:00PM -0400, Vivek Goyal wrote:
[..]
Quoted 34 lines
I ran a test to show how readers can be starved in certain cases. I launched one reader and three writers. I ran this test twice. First without dm-ioband and then with dm-ioband.
Following are few lines from the script to launch readers and writers.
**************************************************************
sync
echo 3 >/proc/sys/vm/drop_caches# Launch writers on sdd2
dd if=/dev/zero of=/mnt/sdd2/writezerofile1 bs=4K count=262144 &# Launch writers on sdd1
dd if=/dev/zero of=/mnt/sdd1/writezerofile1 bs=4K count=262144 & dd if=/dev/zero of=/mnt/sdd1/writezerofile2 bs=4K count=262144 &echo "sleeping for 5 seconds"
sleep 5# launch reader on sdd1 time dd if=/mnt/sdd1/testzerofile1 of=/dev/zero & echo "launched reader $!" *********************************************************************Without dm-ioband, reader finished in roughly 5 seconds.
289533952 bytes (290 MB) copied, 5.16765 s, 56.0 MB/s
real 0m5.300s
user 0m0.098s
sys 0m0.492sWith dm-ioband, reader took, more than 2 minutes to finish.
Quoted 8 lines
289533952 bytes (290 MB) copied, 122.386 s, 2.4 MB/s
real 2m2.569s
user 0m0.107s
sys 0m0.548sI had created ioband1 on
/dev/sdd1and ioband2 on/dev/sdd2with weights 200 and 100 respectively.
Hi Ryo,
I notice that with-in a single ioband device, a single writer starves the reader even without any competitor groups being present.
I ran following two tests with and without dm-ioband devices
Test1
====
Try to use fio for a sequential reader job. First fio will lay out the file and do write operation. While those writes are going on, try to do ls on that partition and observe latency of ls operation.
with dm-ioband (ls test)
------------------------
# cd /mnt/sdd2 # time ls
real 0m9.483s user 0m0.000s sys 0m0.002s
without dm-ioband (ls test)
---------------------------
# cd /mnt/sdd2 # time ls
256M-file1 256M-file5 2G-file1 2G-file5 writefile1 writezerofile 256M-file2 256M-file6 2G-file2 files writefile2 256M-file3 256M-file7 2G-file3 fio writefile3 256M-file4 256M-file8 2G-file4 lost+found writefile4
real 0m0.067s user 0m0.000s sys 0m0.002s
Notice the time simle "ls" operation took in two cases.
Test2
=====
Same case where fio is laying out a file and then try to read some small files on that partition at the interval of 1 second.
small file read with dm-ioband
------------------------------
[root@chilli fairness-tests]# ./small-file-read.sh
file # 0, plain reading it took: 0.24 seconds file # 1, plain reading it took: 13.40 seconds file # 2, plain reading it took: 6.27 seconds file # 3, plain reading it took: 13.84 seconds file # 4, plain reading it took: 5.63 seconds
small file read with-out dm-ioband
==================================
[root@chilli fairness-tests]# ./small-file-read.sh
file # 0, plain reading it took: 0.04 seconds file # 1, plain reading it took: 0.03 seconds file # 2, plain reading it took: 0.04 seconds file # 3, plain reading it took: 0.03 seconds file # 4, plain reading it took: 0.03 seconds
Notice how small file read latencies have shot up.
Looks like a single writer is completely starving a reader even without any IO going in any of the other groups.
setup
=====
I created an two ioband device of weight 100 each on partition /dev/sdd1 and /dev/sdd2 respectively. I am doing IO only on partition /dev/sdd2 (ioband2).
Following is fio job script.
[seqread]
runtime=60
rw=read
size=2G
directory=/mnt/sdd2/fio/
numjobs=1group_reporting
Following is small file read script.
echo 3 > /proc/sys/vm/drop_caches
for ((i=0;i<5;i++)); do
printf "file #%4d, plain reading it took: " $i /usr/bin/time -f "%e seconds" cat /mnt/sdd2/files/$i >/dev/null sleep 1
done
Thanks
Vivek
Re: ioband: Writer starves reader even without competitors
Hi Vivek,
Quoted 6 lines · Vivek Goyal
Hi Ryo,
I notice that with-in a single ioband device, a single writer starves the reader even without any competitor groups being present.
I ran following two tests with and without dm-ioband devices
Thank you again for testing dm-ioband.
I got the similar results and am investigating it now. I'll let you know if I find something.
Thanks,
Ryo Tsuruta
Quoted 8 lines
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
I would tend to agree with this. With other resource management controllers using cgroups, having dm-ioband use something different will require a different set of userspace tools/libraries to be used. Something that will severly limit its usefulness froma programmer's perspective.
thanks,
Signature · 2 lines
regards, Dhaval
Hi,
Quoted 7 lines · Dhaval Giani
Quoted 7 lines
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
And I don't think that the grouping methods are not complicated, just stack a new device on the existing device and assign bandwidth to it, that is the same method as other device-mapper targets, if you would like to assign bandwidth per thread, then register the thread's ID to the device and assign bandwidth to it as well. I don't think it makes users confused.
Quoted 5 lines
I would tend to agree with this. With other resource management controllers using cgroups, having dm-ioband use something different will require a different set of userspace
tools/librariesto be used. Something that will severly limit its usefulness froma programmer's perspective.
Once we create a dm-ioband device, the device can be configured through the cgroup interface. I think it will not severly limit its usefulness.
Thanks,
Ryo Tsuruta
Quoted 11 lines · Ryo Tsuruta
Hi,
Quoted 7 lines · Dhaval Giani
Quoted 7 lines
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
Quoted 5 lines
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
Hi Ryo,
I am not sure that upstream should really be worrying about RHEL 5.x. cgroups is a relatively mature solution and is available in most (if not all) community distros today. We really should not be looking at another grouping solution if the sole reason is that then dm-ioband can be used on RHEL 5.x. The correct solution would be to maintain a separate patch for RHEL 5.x then and not to burden the upstream kernel.
Quoted 13 lines
And I don't think that the grouping methods are not complicated, just stack a new device on the existing device and assign bandwidth to it, that is the same method as other device-mapper targets, if you would like to assign bandwidth per thread, then register the thread's ID to the device and assign bandwidth to it as well. I don't think it makes users confused.
Quoted 5 lines
I would tend to agree with this. With other resource management controllers using cgroups, having dm-ioband use something different will require a different set of userspace
tools/librariesto be used. Something that will severly limit its usefulness froma programmer's perspective.
Quoted 4 lines
Once we create a dm-ioband device, the device can be configured through the cgroup interface. I think it will not severly limit its usefulness.
My objection is slightly different. My objection is that there are too many interfaces to do the same thing. Which one of these is the recommended one? WHich one is going to be supported? If we say that cgroups is not the preferred interface, do the application developers need to use yet another library for io control along with cpu/memory control?
thanks,
Signature · 2 lines
regards, Dhaval
Hi,
Quoted 6 lines · Dhaval Giani
Quoted 5 lines
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
Quoted 8 lines
Hi Ryo,
I am not sure that upstream should really be worrying about RHEL 5.x. cgroups is a relatively mature solution and is available in most (if not all) community distros today. We really should not be looking at another grouping solution if the sole reason is that then dm-ioband can be used on RHEL 5.x. The correct solution would be to maintain a separate patch for RHEL 5.x then and not to burden the upstream kernel.
RHEL 5.x is not the sole reason for that.
Quoted 13 lines
Quoted 13 lines
And I don't think that the grouping methods are not complicated, just stack a new device on the existing device and assign bandwidth to it, that is the same method as other device-mapper targets, if you would like to assign bandwidth per thread, then register the thread's ID to the device and assign bandwidth to it as well. I don't think it makes users confused.
Quoted 5 lines
I would tend to agree with this. With other resource management controllers using cgroups, having dm-ioband use something different will require a different set of userspace
tools/librariesto be used. Something that will severly limit its usefulness froma programmer's perspective.
Quoted 7 lines
Quoted 4 lines
Once we create a dm-ioband device, the device can be configured through the cgroup interface. I think it will not severly limit its usefulness.
My objection is slightly different. My objection is that there are too many interfaces to do the same thing.
Not too many, There are only two interfaces device-mapper and cgroup.
Quoted 1 line
Which one of these is the recommended one?
I think whichever users like, it's up to users.
Quoted 1 line
WHich one is going to be supported?
Both device-mapper and cgroup interfaces are supported continuously.
Quoted 3 lines
If we say that cgroups is not the preferred interface, do the application developers need to use yet another library for io control along with cpu/memory control?
I don't think that cgroup is not the preferred interface, especially not if dm-ioband uses with cpu/memory controller together. Once a dm-ioband device is created, all configurations for the device can be done through the cgroup interface like other cgroup subsystems. It does not require a different set of userspace tools/libraries to be used.
I think it is natural to make dm-ioband can be configured by device-mapper's interface since dm-ioband is a device-mapper driver, and it extends use case, rather it limits usefulness.
Thanks,
Ryo Tsuruta
Quoted 11 lines · Ryo Tsuruta
Hi,
Quoted 8 lines · Dhaval Giani
Quoted 5 lines
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
Hi Ryo,
Quoted 9 lines
Quoted 6 lines
I am not sure that upstream should really be worrying about RHEL 5.x. cgroups is a relatively mature solution and is available in most (if not all) community distros today. We really should not be looking at another grouping solution if the sole reason is that then dm-ioband can be used on RHEL 5.x. The correct solution would be to maintain a separate patch for RHEL 5.x then and not to burden the upstream kernel.
RHEL 5.x is not the sole reason for that.
Could you please enumerate the other reasons for pushing in another grouping mechanism then? (Why can we not resolve them via cgroups?)
Thanks,
Signature · 2 lines
regards, Dhaval
Hi Dhaval,
Quoted 16 lines · Dhaval Giani
Quoted 16 lines
Quoted 14 lines · Dhaval Giani
Quoted 5 lines
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
Hi Ryo,
I am not sure that upstream should really be worrying about RHEL 5.x. cgroups is a relatively mature solution and is available in most (if not all) community distros today. We really should not be looking at another grouping solution if the sole reason is that then dm-ioband can be used on RHEL 5.x. The correct solution would be to maintain a separate patch for RHEL 5.x then and not to burden the upstream kernel.
Quoted 5 lines
Quoted 2 lines
RHEL 5.x is not the sole reason for that.
Could you please enumerate the other reasons for pushing in another grouping mechanism then? (Why can we not resolve them via cgroups?)
I'm sorry for late reply.
I'm not only pushing in the grouping mechanism by using the dmsetup command. Please understand that dm-ioband also provides cgroup interface and can be configured in the same manner like other cgroup subsystems.
Why it is so bad to have multiple ways to configure? I think that it rather gains in flexibility of configurations.
Thanks,
Ryo Tsuruta
Quoted 25 lines · Ryo Tsuruta
Hi Dhaval,
Quoted 16 lines · Dhaval Giani
Quoted 16 lines
Quoted 14 lines · Dhaval Giani
Quoted 5 lines
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
Hi Ryo,
I am not sure that upstream should really be worrying about RHEL 5.x. cgroups is a relatively mature solution and is available in most (if not all) community distros today. We really should not be looking at another grouping solution if the sole reason is that then dm-ioband can be used on RHEL 5.x. The correct solution would be to maintain a separate patch for RHEL 5.x then and not to burden the upstream kernel.
Quoted 5 lines
Quoted 2 lines
RHEL 5.x is not the sole reason for that.
Could you please enumerate the other reasons for pushing in another grouping mechanism then? (Why can we not resolve them via cgroups?)
Quoted 9 lines
I'm sorry for late reply.
I'm not only pushing in the grouping mechanism by using the dmsetup command. Please understand that dm-ioband also provides cgroup interface and can be configured in the same manner like other cgroup subsystems.
Why it is so bad to have multiple ways to configure? I think that it rather gains in flexibility of configurations.
The main issue I see is user confusion and distro issues. If a distro compiles cgroups and dmsetup provides both methods, what method do we recommend to end users? Also should system management tool support two configuration mechanisms for the same functionality?
Signature · 1 line
Balbir
Quoted 10 lines · Balbir Singh
Quoted 3 lines · Ryo Tsuruta
Why it is so bad to have multiple ways to configure? I think that it rather gains in flexibility of configurations.
The main issue I see is user confusion and distro issues. If a distro compiles cgroups and dmsetup provides both methods, what method do we recommend to end users? Also should system management tool support two configuration mechanisms for the same functionality?
It gets worse.
If the distro sets up things via cgroups and the admin tries to use dmsetup - how does the configuration propagate between the two mechanisms?
The sysadmin would expect that any changes made via dmsetup will become visible via the config tools (that use cgroups), too.
This will quickly increase the code requirements to ridiculous proportions - or leave sysadmins confused and annoyed.
Neither is a good option, IMHO.
Signature · 1 line
All rights reversed.
Hi Balbir,
Quoted 37 lines · Balbir Singh
Quoted 25 lines · Ryo Tsuruta
Hi Dhaval,
Quoted 16 lines · Dhaval Giani
Quoted 16 lines
Quoted 14 lines · Dhaval Giani
Quoted 5 lines
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
Hi Ryo,
I am not sure that upstream should really be worrying about RHEL 5.x. cgroups is a relatively mature solution and is available in most (if not all) community distros today. We really should not be looking at another grouping solution if the sole reason is that then dm-ioband can be used on RHEL 5.x. The correct solution would be to maintain a separate patch for RHEL 5.x then and not to burden the upstream kernel.
Quoted 5 lines
Quoted 2 lines
RHEL 5.x is not the sole reason for that.
Could you please enumerate the other reasons for pushing in another grouping mechanism then? (Why can we not resolve them via cgroups?)
Quoted 9 lines
I'm sorry for late reply.
I'm not only pushing in the grouping mechanism by using the dmsetup command. Please understand that dm-ioband also provides cgroup interface and can be configured in the same manner like other cgroup subsystems.
Why it is so bad to have multiple ways to configure? I think that it rather gains in flexibility of configurations.
Quoted 4 lines
The main issue I see is user confusion and distro issues. If a distro compiles cgroups and dmsetup provides both methods, what method do we recommend to end users? Also should system management tool support two configuration mechanisms for the same functionality?
I think that it is up to users which mechanism they choose to use, and such kind of users who can use dmsetup or cgroup interface directly will not be confused in such a situation.
I also think that management tools are required for end users, and if a distro supports cgroups, I recommend the management tools configure dm-ioband by using cgroups, because dm-ioband is more usable when using with blkio-cgroup and memory cgroup.
Thanks,
Ryo Tsuruta
Quoted 11 lines · Ryo Tsuruta
Hi,
Quoted 7 lines · Dhaval Giani
Quoted 7 lines
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
Quoted 5 lines
I know that cgroup is a very well defined functionality, that is why dm-ioband also supports throttling per cgroup. But how are we supposed to do throttling on the system which doesn't support cgroup? As I wrote in another mail to Vivek, I would like to make use of dm-ioband on RHEL 5.x.
I think you need to maintain and support this module out of the kernel tree for older kernels. Does not make much sense to introduce new interfaces to support a functionality in older kernels.
Quoted 6 lines
And I don't think that the grouping methods are not complicated, just stack a new device on the existing device and assign bandwidth to it, that is the same method as other device-mapper targets, if you would like to assign bandwidth per thread, then register the thread's ID to the device and assign bandwidth to it as well. I don't think it makes users confused.
- First of all it is more about doing things a new way and not the standard way. Moreover upstream does not benefit from this new interface. It just stands to loose because of maintenance overhead and need of chaning user space tools to make use of this new interface.
- Secondly, personally I think it more twisted also. Following is the small code to setup two ioband devices ioband1 and ioband2 and two additional groups on ioband1 device using cgroup interface.
***********************************************************************
echo "0 $(blockdev --getsize /dev/sdd1) ioband /dev/sdd1 1 0 0 none" "weight 0 :200" | dmsetup create ioband1
echo "0 $(blockdev --getsize /dev/sdd2) ioband /dev/sdd2 1 0 0 none" "weight 0 :100" | dmsetup create ioband2
mount -t cgroup -o blkio hier1 /cgroup/ioband
mkdir /cgroup/ioband/test1 /cgroup/ioband/test2
test1_id=cat /cgroup/ioband/test1/blkio.id
test2_id=cat /cgroup/ioband/test2/blkio.id
test1_weight=200
test2_weight=100
dmsetup message ioband1 0 type cgroup
dmsetup message ioband1 0 attach $test1_id dmsetup message ioband1 0 attach $test2_id dmsetup message ioband1 0 weight $test1_id:$test1_weight dmsetup message ioband1 0 weight $test2_id:$test2_weight
mount /dev/mapper/ioband1 /mnt/sdd1
mount /dev/mapper/ioband2 /mnt/sdd2
*************************************************************************
For status of various settings one needs to use "dmsetup status" and "dmsetup table" commands. Look at the output of these commands with just two groups. Output for all the groups is on a single line. Think of the situation when there are 7-8 groups and how bad it will look.
#dmsetup status
ioband2: 0 40355280 ioband 1 -1 105 0 834 1 0 8
ioband1: 0 37768752 ioband 1 -1 105 0 834 1 0 8 2 0 0 0 0 0 0 3 0 0 0 0 0 0
#dmsetup table
ioband2: 0 40355280 ioband 8:50 1 4 192 none weight 768 :100
ioband1: 0 37768752 ioband 8:49 1 4 192 cgroup weight 768 :200 2:200 3:100
I find it so hard to interpre those numbers. Everything about a device is exported in a single line.
In cgroup based interface, things are deviced nicely among different files. Also one group shows statistics about that group only and not about all the groups present in the system. It is easier to parse and comprehend.
Quoted 10 lines
Quoted 6 lines
I would tend to agree with this. With other resource management controllers using cgroups, having dm-ioband use something different will require a different set of userspace
tools/librariesto be used. Something that will severly limit its usefulness froma programmer's perspective.Once we create a dm-ioband device, the device can be configured through the cgroup interface. I think it will not severly limit its usefulness.
To create the device once you need dm-tools and libcgroup needs to learn how to make various use of various dm commands. It also needs to learn how to parse outputs of "dmsetup table" and "dmsetup status" commands and consolidate that information.
This is despite the fact that it is using cgroup interface finally to group the task. But libcgroup still need to propagate cgroup id to individual ioband devices.
Thanks
Vivek
Hi Vivek,
Quoted 4 lines · Vivek Goyal
- Secondly, personally I think it more twisted also. Following is the small code to setup two ioband devices ioband1 and ioband2 and two additional groups on ioband1 device using cgroup interface.
The latest dm-ioband and blkio-cgroup, configurations can be done through the cgroup interface once a dm-device is created, blkio.setting file is appered under the cgroup directory. There is no need to run "dmsetup message" command and care the the blkio.id anymore.
The following is an example script based on yours.
***********************************************************************
echo "0 $(blockdev --getsize /dev/sdd1) ioband /dev/sdd1 1 0 0 cgroup"
"weight 0 :100" | dmsetup create ioband1
mount -t cgroup -o blkio hier1 /cgroup/ioband
mkdir /cgroup/ioband/test1 /cgroup/ioband/test2
echo ioband1 200 > /cgroup/ioband/test1/blkio.settings echo ioband1 100 > /cgroup/ioband/test2/blkio.settings
mount /dev/mapper/ioband1 /mnt/sdd1
Quoted 8 lines
For status of various settings one needs to use "dmsetup status" and "dmsetup table" commands. Look at the output of these commands with just two groups. Output for all the groups is on a single line. Think of the situation when there are 7-8 groups and how bad it will look.
#dmsetup status
ioband2: 0 40355280 ioband 1 -1 105 0 834 1 0 8
ioband1: 0 37768752 ioband 1 -1 105 0 834 1 0 8 2 0 0 0 0 0 0 3 0 0 0 0 0 0
I'll provide blkio.stat file to get statistics per cgroup in the next release.
Quoted 4 lines
Quoted 3 lines
Once we create a dm-ioband device, the device can be configured through the cgroup interface. I think it will not severly limit its usefulness.
Quoted 8 lines
To create the device once you need dm-tools and libcgroup needs to learn how to make various use of various dm commands. It also needs to learn how to parse outputs of "dmsetup table" and "dmsetup status" commands and consolidate that information.
This is despite the fact that it is using cgroup interface finally to group the task. But libcgroup still need to propagate cgroup id to individual ioband devices.
We still need to use dmsetup for the device creation, but It is not too much pain. I think it would be better If dm-ioband is intergrated into LVM, then we can handle dm-ioband devies in almost the same manner as other LV devices.
Thanks,
Ryo Tsuruta
Hi Vivek,
Quoted 12 lines · Vivek Goyal
Quoted 6 lines
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
I think we can easily extent io scheduler based controller to also support max rate per group policy also. That should not be too hard. It is a matter of only keeping track of io rate per group and if a group is exceeding the rate, then schedule it out and move on to next group.
Quoted 4 lines
I can do that once proportional weight solution is stablized and gets merged.
So its not an advantage of dm-ioband.
O.K.
Quoted 10 lines
Quoted 2 lines
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
We talked about this point in the past also. In io scheduler based controller, just move all the tasks to root group and you got a system not doing any io control.
By the way why would one like to do that?
So this is also not an advantage.
My point is that dm-ioband can be updated for improvements and bug-fixing without stopping the system.
Quoted 14 lines
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
I don't understand what is that core requirement of yours which is not met by io scheduler based io controller. range policy control you have implemented recently. I don't think that removing dm-ioband module dynamically is core requirement. Also whatever you can do with additional grouping mechanism, you can do with cgroup also.
Quoted 5 lines
So if there is any of your core functionality which is not fulfilled by io scheduler based controller, please let me know. I will be happy to look into it and try to provide that feature. But looking at above list, I am not convinced that any of the above is a compelling argument for dm-ioband inclusion.
As I wrote in another email, I would like to make use of dm-ioband on the system which doesn't support cgroup such as RHEL. In addition, there are devices which doesn't use standard IO schedulers, and dm-ioband can work on even such devices.
Thanks,
Ryo Tsuruta
Quoted 50 lines · Ryo Tsuruta
Hi Vivek,
Quoted 12 lines · Vivek Goyal
Quoted 6 lines
I think there are some advantages to dm-ioband. That's why I post dm-ioband to the mailing list.
- dm-ioband supports not only proportional weight policy but also rate limiting policy. Besides, new policies can be added to dm-ioband if a user wants to control bandwidth by his or her own policy.
I think we can easily extent io scheduler based controller to also support max rate per group policy also. That should not be too hard. It is a matter of only keeping track of io rate per group and if a group is exceeding the rate, then schedule it out and move on to next group.
Quoted 4 lines
I can do that once proportional weight solution is stablized and gets merged.
So its not an advantage of dm-ioband.
O.K.
Quoted 10 lines
Quoted 2 lines
- The dm-ioband driver can be replaced without stopping the system by using device-mapper's facility. It's easy to maintain.
We talked about this point in the past also. In io scheduler based controller, just move all the tasks to root group and you got a system not doing any io control.
By the way why would one like to do that?
So this is also not an advantage.
My point is that dm-ioband can be updated for improvements and bug-fixing without stopping the system.
Quoted 14 lines
Quoted 2 lines
- dm-ioband can use without cgroup. (I remember Vivek said it's not an advantage.)
I think this is more of a disadvantage than advantage. We have a very well defined functionality of cgroup in kernel to group the tasks. Now you are coming up with your own method of grouping the tasks which will make life even more confusing for users and application writers.
I don't understand what is that core requirement of yours which is not met by io scheduler based io controller. range policy control you have implemented recently. I don't think that removing dm-ioband module dynamically is core requirement. Also whatever you can do with additional grouping mechanism, you can do with cgroup also.
Quoted 8 lines
Quoted 5 lines
So if there is any of your core functionality which is not fulfilled by io scheduler based controller, please let me know. I will be happy to look into it and try to provide that feature. But looking at above list, I am not convinced that any of the above is a compelling argument for dm-ioband inclusion.
As I wrote in another email, I would like to make use of dm-ioband on the system which doesn't support cgroup such as RHEL.
For supporting io controller mechanism in older kernels which don't have cgroup interface support, I think one needs to maintain out of the tree module. Upstream does not benefit from it.
Quoted 3 lines
In addition,
there are devices which doesn't use standard IO schedulers, and dm-ioband can work on even such devices.
This is a interesting use case. Few thoughts.
- Can't io scheduling mechanism of these devices make use of elevator and elevator fair queuing interfaces to take advantage of io controlling mechanism. It should not be too difficult. Look at noop. It has just 131 lines of code and it now supports hierarchical io scheduling.
This will come with request queue and its merging and plug/unplug mechanism. Is that an issue?
- If not, then yes, for these corner cases, io scheduler based controller does not work as it is.
Thanks
Vivek
Hi Vivek,
Quoted 11 lines · Vivek Goyal
Quoted 3 lines
In addition,
there are devices which doesn't use standard IO schedulers, and dm-ioband can work on even such devices.This is a interesting use case. Few thoughts.
- Can't io scheduling mechanism of these devices make use of elevator and elevator fair queuing interfaces to take advantage of io controlling mechanism. It should not be too difficult. Look at noop. It has just 131 lines of code and it now supports hierarchical io scheduling.
Quoted 5 lines
This will come with request queue and its merging and plug/unplug mechanism. Is that an issue?
- If not, then yes, for these corner cases, io scheduler based controller does not work as it is.
I have a extreme fast SSD and its device driver provides its own make_request_fn(). So the device driver intercepts IO requests and the subsequent processes are done within it.
Thanks,
Ryo Tsuruta
Quoted 17 lines · Ryo Tsuruta
Hi Vivek,
Quoted 14 lines · Vivek Goyal
Quoted 3 lines
In addition,
there are devices which doesn't use standard IO schedulers, and dm-ioband can work on even such devices.This is a interesting use case. Few thoughts.
- Can't io scheduling mechanism of these devices make use of elevator and elevator fair queuing interfaces to take advantage of io controlling mechanism. It should not be too difficult. Look at noop. It has just 131 lines of code and it now supports hierarchical io scheduling.
This will come with request queue and its merging and plug/unplug mechanism. Is that an issue?
Quoted 6 lines
Quoted 2 lines
- If not, then yes, for these corner cases, io scheduler based controller does not work as it is.
I have a extreme fast SSD and its device driver provides its own
make_request_fn(). So the device driver intercepts IO requests and the subsequent processes are done within it.
IMHO, in those cases these SSD driver needs to hook into block layer's request queue mechanism if they need io controlling mechanism instead of we coming up a device mapper module.
Think of it that if somebody needs CFQ like tasks classes and prio suported on these devices, should we also come up with another device mapper module "dm-cfq"?
Jens, I am wondering if similiar concerns have popped in the past also for CFQ also? Somebody asking to support task prio and classes on devices which don't use standard IO scheduler?
Thanks
Vivek
Quoted 8 lines · Ryo Tsuruta
Quoted 2 lines · Rik van Riel
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
This sounds good, except that the lack of anticipation means that a group with just one task doing reads will be considered "inactive" in-between reads.
This means writes can always get in-between two reads, sometimes multiple writes at a time, really disadvantaging a group that is doing just disk reads.
This is a problem, because reads are generally more time sensitive than writes.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
We regarded reducing throughput loss rather than reducing duration as the design of dm-ioband. Of course, it is possible to make a new policy which reduces duration.
... while also reducing overall system throughput
by design?I think it reduces system throughput compared to the current implementation, because it causes more overhead to do fine grained control.
Except that the io scheduler based io controller seems to be able to enforce fairness while not reducing throughput.
Dm-ioband would have to address these issues to be a serious contender, IMHO.
Signature · 1 line
All rights reversed.
Hi,
Quoted 9 lines
From: Rik van Riel <[email protected]>
Date: Tue, Sep 08, 2009 03:24:08PM -0400Quoted 5 lines · Ryo Tsuruta
Quoted 2 lines · Rik van Riel
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Quoted 8 lines
Quoted 3 lines
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
This sounds good, except that the lack of anticipation means that a group with just one task doing reads will be considered "inactive" in-between reads.
anticipation helps in achieving fairness, but CFQ currently disables
idling for nonrot+NCQ media, to avoid the resulting throughput loss on some SSDs. Are we really sure that we want to introduce anticipation everywhere, not only to improve throughput on rotational media, but to achieve fairness too?
Quoted 20 lines · Fabio Checconi
Hi,
Quoted 9 lines
From: Rik van Riel <[email protected]>
Date: Tue, Sep 08, 2009 03:24:08PM -0400Quoted 5 lines · Ryo Tsuruta
Quoted 2 lines · Rik van Riel
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Quoted 8 lines
Quoted 3 lines
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
This sounds good, except that the lack of anticipation means that a group with just one task doing reads will be considered "inactive" in-between reads.
Quoted 5 lines
anticipation helps in achieving fairness, but CFQ currently disables
idling for nonrot+NCQ media, to avoid the resulting throughput loss on some SSDs. Are we really sure that we want to introduce anticipation everywhere, not only to improve throughput on rotational media, but to achieve fairness too?
That's a good point. Personally I think that fairness requirements for individual queues and groups are little different. CFQ in general seems to be focussing more on latency and throughput at the cost of fairness.
With groups, we probably need to put a greater amount of emphasis on group fairness. So group will be a relatively a slower entity (with anticiaption on and more idling), but it will also give you a greater amount of isolation. So in practice, one will create groups carefully and they will not proliferate like queues. This can mean overall reduced throughput on SSD.
Having said that, group idling is tunable and one can always reduce it to achieve a balance between fairness vs throughput depending on his need.
Thanks
Vivek
Quoted 34 lines
From: Vivek Goyal <[email protected]>
Date: Tue, Sep 08, 2009 10:06:20PM -0400Quoted 20 lines · Fabio Checconi
Hi,
Quoted 9 lines
From: Rik van Riel <[email protected]>
Date: Tue, Sep 08, 2009 03:24:08PM -0400Quoted 5 lines · Ryo Tsuruta
Quoted 2 lines · Rik van Riel
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Quoted 8 lines
Quoted 3 lines
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
This sounds good, except that the lack of anticipation means that a group with just one task doing reads will be considered "inactive" in-between reads.
Quoted 5 lines
anticipation helps in achieving fairness, but CFQ currently disables
idling for nonrot+NCQ media, to avoid the resulting throughput loss on some SSDs. Are we really sure that we want to introduce anticipation everywhere, not only to improve throughput on rotational media, but to achieve fairness too?That's a good point. Personally I think that fairness requirements for individual queues and groups are little different. CFQ in general seems to be focussing more on latency and throughput at the cost of fairness.
Quoted 7 lines
With groups, we probably need to put a greater amount of emphasis on group fairness. So group will be a relatively a slower entity (with anticiaption on and more idling), but it will also give you a greater amount of isolation. So in practice, one will create groups carefully and they will not proliferate like queues. This can mean overall reduced throughput on SSD.
Ok, I personally agree on that, but I think it's something to be documented.
Quoted 3 lines
Having said that, group idling is tunable and one can always reduce it to achieve a balance between fairness vs throughput depending on his need.
This is good, however tuning will not be an easy task (at least, in my experience with BFQ it has been a problem): while for throughput usually there are tradeoffs, as soon as a queue/group idles and then timeouts, from the fairness perspective the results soon become almost random (i.e., depending on the rate of successful anticipations, but in the common case they are unpredictable)...
Quoted 42 lines · Fabio Checconi
Quoted 34 lines
From: Vivek Goyal <[email protected]>
Date: Tue, Sep 08, 2009 10:06:20PM -0400Quoted 20 lines · Fabio Checconi
Hi,
Quoted 9 lines
From: Rik van Riel <[email protected]>
Date: Tue, Sep 08, 2009 03:24:08PM -0400Quoted 5 lines · Ryo Tsuruta
Quoted 2 lines · Rik van Riel
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Quoted 8 lines
Quoted 3 lines
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
This sounds good, except that the lack of anticipation means that a group with just one task doing reads will be considered "inactive" in-between reads.
Quoted 5 lines
anticipation helps in achieving fairness, but CFQ currently disables
idling for nonrot+NCQ media, to avoid the resulting throughput loss on some SSDs. Are we really sure that we want to introduce anticipation everywhere, not only to improve throughput on rotational media, but to achieve fairness too?That's a good point. Personally I think that fairness requirements for individual queues and groups are little different. CFQ in general seems to be focussing more on latency and throughput at the cost of fairness.
Quoted 7 lines
With groups, we probably need to put a greater amount of emphasis on group fairness. So group will be a relatively a slower entity (with anticiaption on and more idling), but it will also give you a greater amount of isolation. So in practice, one will create groups carefully and they will not proliferate like queues. This can mean overall reduced throughput on SSD.
Quoted 2 lines
Ok, I personally agree on that, but I think it's something to be documented.
Sure. I will document it in documentation file.
Quoted 11 lines
Quoted 4 lines
Having said that, group idling is tunable and one can always reduce it to achieve a balance between fairness vs throughput depending on his need.
This is good, however tuning will not be an easy task (at least, in my experience with BFQ it has been a problem): while for throughput usually there are tradeoffs, as soon as a queue/group idles and then timeouts, from the fairness perspective the results soon become almost random (i.e., depending on the rate of successful anticipations, but in the common case they are unpredictable)...
I am lost in last few lines. I guess you are suggesting that static tuning is hard and dynamically adjusting idling has limitations that it might not be accurate all the time?
I will explain how things are working in current set of io scheduler patches.
Currently on top of queue idling, I have implemented group idling also. Queue idling is dynamic and io scheduler like CFQ keeps track of traffic pattern on the queue and disables/enables idling dynamically. So in this case fairness depends on rate of successful anticipations by the io scheduler.
Group idling currently is static in nature and purely implemented in elevator fair queuing layer. Group idling kicks in only when a group is empty at the time of queue expiration and underlying ioscheduler has not chosen to enable idling on the queue. This provides us the gurantee that group will keep on getting its fair share of disk as long as a new request comes in the group with-in that idling period.
Implementing group idling ensures that it does not bog down the io scheduler and with-in group queue switching can still be very fast (no idling on many of the queues by cfq).
Now in case of SSD if group idling is really hurting somebody, I would expect him to set it to either 1 or 0. You might get better throughput but then expect fairness for the group only if the group is continuously backlogged. (Something what dm-ioband guys seem to be doing).
So do you think that adjusting this "group_idling" tunable is too complicated and there are better ways to handle it in case of SSD+NCQ?
Thanks
Vivek
Quoted 7 lines
From: Vivek Goyal <[email protected]>
Date: Wed, Sep 09, 2009 01:30:03PM -0400Quoted 3 lines · Fabio Checconi
Quoted 3 lines
From: Vivek Goyal <[email protected]>
Date: Tue, Sep 08, 2009 10:06:20PM -0400
...
Quoted 11 lines
Quoted 6 lines
This is good, however tuning will not be an easy task (at least, in my experience with BFQ it has been a problem): while for throughput usually there are tradeoffs, as soon as a queue/group idles and then timeouts, from the fairness perspective the results soon become almost random (i.e., depending on the rate of successful anticipations, but in the common case they are unpredictable)...
I am lost in last few lines. I guess you are suggesting that static tuning is hard and dynamically adjusting idling has limitations that it might not be accurate all the time?
Yes, this was the problem, at least for me. As soon as there were unsuccessful anticipations there was no graceful degradation of fairness, and bandwidth distribution became almost random. In this situation all the complexity of CFQ/BFQ/io-controller seems overkill; NCQ+SSD is or will be quite a common usage scenario triggering it.
Quoted 20 lines
I will explain how things are working in current set of io scheduler patches.
Currently on top of queue idling, I have implemented group idling also. Queue idling is dynamic and io scheduler like CFQ keeps track of traffic pattern on the queue and disables/enables idling dynamically. So in this case fairness depends on rate of successful anticipations by the io scheduler.
Group idling currently is static in nature and purely implemented in elevator fair queuing layer. Group idling kicks in only when a group is empty at the time of queue expiration and underlying ioscheduler has not chosen to enable idling on the queue. This provides us the gurantee that group will keep on getting its fair share of disk as long as a new request comes in the group with-in that idling period.
Implementing group idling ensures that it does not bog down the io scheduler and with-in group queue switching can still be very fast (no idling on many of the queues by cfq).
Quoted 8 lines
Now in case of SSD if group idling is really hurting somebody, I would expect him to set it to either 1 or 0. You might get better throughput but then expect fairness for the group only if the group is continuously backlogged. (Something what dm-ioband guys seem to be doing).
So do you think that adjusting this "
group_idling" tunable is too complicated and there are better ways to handle it in case of SSD+NCQ?
Unfortunately I am not aware of any reasonable and working method to properly handle this issue; anyway adjusting the tunable is something that needs a lot of care.
Hi,
Quoted 20 lines · Fabio Checconi
Hi,
Quoted 9 lines
From: Rik van Riel <[email protected]>
Date: Tue, Sep 08, 2009 03:24:08PM -0400Quoted 5 lines · Ryo Tsuruta
Quoted 2 lines · Rik van Riel
Are you saying that dm-ioband is purposely unfair, until a certain load level is reached?
Quoted 8 lines
Quoted 3 lines
Not unfair, dm-ioband(weight policy) is intentionally designed to use bandwidth efficiently, weight policy tries to give spare bandwidth of inactive groups to active groups.
This sounds good, except that the lack of anticipation means that a group with just one task doing reads will be considered "inactive" in-between reads.
Quoted 5 lines
anticipation helps in achieving fairness, but CFQ currently disables
idling for nonrot+NCQ media, to avoid the resulting throughput loss on some SSDs. Are we really sure that we want to introduce anticipation everywhere, not only to improve throughput on rotational media, but to achieve fairness too?
I'm also not sure if it's worth introducing anticipation everywhere. The storage devices are becoming faster and smarter every year. In practice, I did a benchmark with a SAN storage and the noop scheduler got the best result.
However, I'll consider how IO from one task should take care of.
Thanks,
Ryo Tsuruta
ioband: Limited fairness and weak isolation between groups (Was: Re: Regarding dm-ioband tests)
In reply to Ryo TsurutaQuoted 59 lines · Ryo Tsuruta
Hi Vivek,
Quoted 11 lines · Vivek Goyal
Quoted 5 lines
Thank you for testing dm-ioband. dm-ioband is designed to start throttling bandwidth when multiple IO requests are issued to devices simultaneously, IOW, to start throttling when IO load exceeds a certain level.
What is that certain level? Secondly what's the advantage of this?
I can see disadvantages though. So unless a group is really busy "up to that certain level" it will not get fairness? I breaks the isolation between groups.
In your test case, at least more than one dd thread have to run simultaneously in the higher weight group. The reason is that if there is an IO group which does not issue a certain number of IO requests, dm-ioband assumes the IO group is inactive and assign its spare bandwidth to active IO groups. Then whole bandwidth of the device can be efficiently used. Please run two dd threads in the higher group, it will work as you expect.
However, if you want to get fairness in a case like this, a new bandwidth control policy which controls accurately according to assigned weights can be added to dm-ioband.
Quoted 32 lines
I also ran your test of doing heavy IO in two groups. This time I am running 4 dd threads in both the ioband devices. Following is the snapshot of "dmsetup table" output.
Fri Sep 4 17:45:27 EDT 2009
ioband2: 0 40355280 ioband 1 -1 0 0 0 0 0 0
ioband1: 0 37768752 ioband 1 -1 0 0 0 0 0 0Fri Sep 4 17:45:29 EDT 2009
ioband2: 0 40355280 ioband 1 -1 41 0 4184 0 0 0
ioband1: 0 37768752 ioband 1 -1 173 0 20096 0 0 0Fri Sep 4 17:45:37 EDT 2009
ioband2: 0 40355280 ioband 1 -1 1605 23 197976 0 0 0
ioband1: 0 37768752 ioband 1 -1 4640 1 583168 0 0 0Fri Sep 4 17:45:45 EDT 2009
ioband2: 0 40355280 ioband 1 -1 3650 47 453488 0 0 0
ioband1: 0 37768752 ioband 1 -1 8572 1 1079144 0 0 0Fri Sep 4 17:45:51 EDT 2009
ioband2: 0 40355280 ioband 1 -1 5111 68 635696 0 0 0
ioband1: 0 37768752 ioband 1 -1 11587 1 1459544 0 0 0Fri Sep 4 17:45:53 EDT 2009
ioband2: 0 40355280 ioband 1 -1 5698 73 709272 0 0 0
ioband1: 0 37768752 ioband 1 -1 12503 1 1575112 0 0 0Fri Sep 4 17:45:57 EDT 2009
ioband2: 0 40355280 ioband 1 -1 6790 87 845808 0 0 0
ioband1: 0 37768752 ioband 1 -1 14395 2 1813680 0 0 0
Quoted 6 lines
Quoted 2 lines
Note, it took me more than 20 seconds (since I started the threds) to reach close to desired fairness level. That's too long a duration.
We regarded reducing throughput loss rather than reducing duration as the design of dm-ioband. Of course, it is possible to make a new policy which reduces duration.
Not anticipating on rotation media and letting other group do the dispatch is not only bad for fairness of random readers but it seems to be bad for overall throughput also. So letting other group dispatching thinking it will boost throughput is not necessarily right on rotational media.
I ran following test. Created two groups of weight 100 each and put a sequential dd reader in first group and put buffered writers in second group and let it run for 20 seconds and observed at the end of 20 seconds which group got how much work done. I ran this test multiple time, while increasing the number of writers by one each time. Did test this with dm-ioband and with io scheduler based io controller patches.
With dm-ioband
==============
launched reader 3176
launched 1 writers
waiting for 20 seconds
ioband2: 0 40355280 ioband 1 -1 159 0 1272 0 0 0
ioband1: 0 37768752 ioband 1 -1 13282 23 1673656 0 0 0 Total sectors transferred: 1674928
launched reader 3194
launched 2 writers
waiting for 20 seconds
ioband2: 0 40355280 ioband 1 -1 138 0 1104 54538 54081 436304
ioband1: 0 37768752 ioband 1 -1 4247 1 535056 0 0 0 Total sectors transferred: 972464
launched reader 3203
launched 3 writers
waiting for 20 seconds
ioband2: 0 40355280 ioband 1 -1 189 0 1512 44956 44572 359648
ioband1: 0 37768752 ioband 1 -1 3546 0 447128 0 0 0 Total sectors transferred: 808288
launched reader 3213
launched 4 writers
waiting for 20 seconds
ioband2: 0 40355280 ioband 1 -1 83 0 664 55937 55810 447496
ioband1: 0 37768752 ioband 1 -1 2243 0 282624 0 0 0 Total sectors transferred: 730784
launched reader 3224
launched 5 writers
waiting for 20 seconds
ioband2: 0 40355280 ioband 1 -1 179 0 1432 46544 46146 372352
ioband1: 0 37768752 ioband 1 -1 3348 0 422744 0 0 0 Total sectors transferred: 796528
launched reader 3236
launched 6 writers
waiting for 20 seconds
ioband2: 0 40355280 ioband 1 -1 176 0 1408 44499 44115 355992
ioband1: 0 37768752 ioband 1 -1 3998 0 504504 0 0 0 Total sectors transferred: 861904
launched reader 3250
launched 7 writers
waiting for 20 seconds
ioband2: 0 40355280 ioband 1 -1 451 0 3608 42267 42115 338136
ioband1: 0 37768752 ioband 1 -1 2682 0 337976 0 0 0 Total sectors transferred: 679720
With io scheduler based io controller
=====================================
launched reader 3026
launched 1 writers
waiting for 20 seconds
test1 statistics: time=8:48 8657 sectors=8:48 886112 dq=8:48 0
test2 statistics: time=8:48 7685 sectors=8:48 473384 dq=8:48 4
Total sectors transferred: 1359496
launched reader 3064
launched 2 writers
waiting for 20 seconds
test1 statistics: time=8:48 7429 sectors=8:48 856664 dq=8:48 0
test2 statistics: time=8:48 7431 sectors=8:48 376528 dq=8:48 0
Total sectors transferred: 1233192
launched reader 3094
launched 3 writers
waiting for 20 seconds
test1 statistics: time=8:48 7279 sectors=8:48 832840 dq=8:48 0
test2 statistics: time=8:48 7302 sectors=8:48 372120 dq=8:48 0
Total sectors transferred: 1204960
launched reader 3122
launched 4 writers
waiting for 20 seconds
test1 statistics: time=8:48 7291 sectors=8:48 846024 dq=8:48 0
test2 statistics: time=8:48 7314 sectors=8:48 361280 dq=8:48 0
Total sectors transferred: 1207304
launched reader 3151
launched 5 writers
waiting for 20 seconds
test1 statistics: time=8:48 7077 sectors=8:48 815184 dq=8:48 0
test2 statistics: time=8:48 7090 sectors=8:48 398472 dq=8:48 0
Total sectors transferred: 1213656
launched reader 3179
launched 6 writers
waiting for 20 seconds
test1 statistics: time=8:48 7494 sectors=8:48 873304 dq=8:48 1
test2 statistics: time=8:48 7034 sectors=8:48 316312 dq=8:48 2
Total sectors transferred: 1189616
launched reader 3209
launched 7 writers
waiting for 20 seconds
test1 statistics: time=8:48 6809 sectors=8:48 795528 dq=8:48 0
test2 statistics: time=8:48 6850 sectors=8:48 380008 dq=8:48 1
Total sectors transferred: 1175536
Few things stand out.
====================
- With dm-ioband, as number of writers increased, in group 2, it gave BW to those writes over reads running in group 1. It had two bad effects. First of all read throughput went down secondly overall disk throughput also went down.
So reader did not get fairness at the same time overall throughput went down. Hence probably it is not a very good idea to not anticipate and always let other groups dispatch on rotational media.
In contrast, io scheduler based controller seems to be steady and reader doest not suffer as number of writers increase in the second group and overall disk throughput also remains stable.
Follwoing is the sample script I used for above test.
*******************************************************************launch_writers() {
nr_writers=$1
for ((j=1;j<=$nr_writers;j++)); do
dd if=/dev/zero of=/mnt/sdd2/writefile$j bs=4K &
# echo "launched writer $!"
done
}
do_test () {
nr_writers=$1
sync
echo 3 > /proc/sys/vm/drop_caches
echo noop > /sys/block/sdd/queue/scheduler echo cfq > /sys/block/sdd/queue/scheduler
dmsetup message ioband1 0 reset
dmsetup message ioband2 0 reset
#launch a sequential reader in sdd1
dd if=/mnt/sdd1/4G-file of=/dev/null &
echo "launched reader $!"
launch_writers $nr_writers
echo "launched $nr_writers writers"
echo "waiting for 20 seconds"
sleep 20
dmsetup status
killall dd > /dev/null 2>&1
}
for ((i=1;i<8;i++)); do
do_test $i
echo
done
*********************************************************************
Re: ioband: Limited fairness and weak isolation between groups
Hi Vivek,
Quoted 6 lines · Vivek Goyal
I ran following test. Created two groups of weight 100 each and put a sequential dd reader in first group and put buffered writers in second group and let it run for 20 seconds and observed at the end of 20 seconds which group got how much work done. I ran this test multiple time, while increasing the number of writers by one each time. Did test this with dm-ioband and with io scheduler based io controller patches.
I did the same test on my environment (2.6.31 + dm-ioband v1.13.0) and here are the results.
The number of sectors transferred
writers read write total
1 800696 588600 1389296
2 747704 430736 1178440
3 757136 455808 1212944
4 704888 562912 1267800
5 788760 387672 1176432
6 730664 495832 1226496
7 765864 427384 1193248I got the different results to yours, the total throughput did not decreased according to increasing the number of writers. I've attached the outputs of the test script. Please note that the format of "dmsetup status" have been changed like /sys/block/dev/stat file.
launched reader 3567
launched 1 writers
waiting for 20 seconds
ioband2: 0 112455000 ioband share1 -1 85 0 680 0 100087 0 800696 0 384 0 0
ioband1: 0 112455000 ioband share1 -1 4673 0 588600 0 0 0 0 0 0 0 0
launched reader 3575
launched 2 writers
waiting for 20 seconds
ioband2: 0 112455000 ioband share1 -1 197 0 1576 0 93463 0 747704 0 384 0 0
ioband1: 0 112455000 ioband share1 -1 3420 0 430736 0 0 0 0 0 0 0 0
launched reader 3584
launched 3 writers
waiting for 20 seconds
ioband2: 0 112455000 ioband share1 -1 237 0 1896 0 94642 0 757136 0 384 0 0
ioband1: 0 112455000 ioband share1 -1 3614 0 455808 0 0 0 0 0 0 0 0
launched reader 3594
launched 4 writers
waiting for 20 seconds
ioband2: 0 112455000 ioband share1 -1 207 0 1656 0 88111 0 704888 0 159 0 0
ioband1: 0 112455000 ioband share1 -1 4462 0 562912 0 0 0 0 0 0 0 0
launched reader 3605
launched 5 writers
waiting for 20 seconds
ioband2: 0 112455000 ioband share1 -1 234 0 1872 0 98595 0 788760 0 384 0 0
ioband1: 0 112455000 ioband share1 -1 3077 0 387672 0 0 0 0 0 0 0 0
launched reader 3618
launched 6 writers
waiting for 20 seconds
ioband2: 0 112455000 ioband share1 -1 215 0 1720 0 91333 0 730664 0 384 0 0
ioband1: 0 112455000 ioband share1 -1 3937 0 495832 0 0 0 0 0 0 0 0
launched reader 3631
launched 7 writers
waiting for 20 seconds
ioband2: 0 112455000 ioband share1 -1 245 0 1960 0 95733 0 765864 0 384 0 0
ioband1: 0 112455000 ioband share1 -1 3391 0 427384 0 0 0 0 0 0 0 0
Thanks,
Ryo Tsuruta