device-mapper may have read performance issue

linux-lvm Ben Huang 4 participants 7 messages Oct 8, 2008 – Oct 14, 2008

Share & tools

Ben Huang

author

2008-10-08 07:51 UTC #0 lore raw

HI all,
I found a LVM+dm read performance issue on my storage server. On my system, I have a 12+0 RAID5 md, which was created by the following command mdadm -C /dev/md0 -l5 -n12 /dev/sd{a,b,l,d,e,f,g,h,i,j,k,o} --assume-clean --metadata=1

My first test was
dd if=/dev/md0 of=/dev/null bs=1M
And I got around 780MB/s sequence read performance

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0      0  31272 3369856 258728    0    0 794680     0 3223 4120  0 35 61  3
 1  0      0  24980 3371104 258664    0    0 788484     0 3271 3909  0 31 67  2
 1  0      0  35208 3363600 258760    0    0 792572     0 3288 4056  0 34 65  1
 0  1      0  22304 3380164 258560    0    0 753668     0 2875 3775  0 29 70  1
 1  0      0  41272 3361048 258660    0    0 770048     0 3124 3977  0 32 66  2
 1  0      0  32636 3366976 258536    0    0 759485     0 2920 3823  0 33 64  3
 1  0      0  21796 3376132 258632    0    0 776184     0 3053 3995  0 33 66  2
 1  0      0  28612 3373608 258872    0    0 758148     0 3050 3907  0 30 68  3
 3  0      0  35768 3366948 258504    0    0 778240     0 3154 3873  0 32 66  1
 1  0      0  25588 3377328 258864    0    0 737296     0 2913 3262  0 30 68  2
 1  0      0  31264 3370300 258828    0    0 792832     0 3339 2934  0 34 63  2
 0  1      0  21532 3380072 258708    0    0 755512     0 3067 2986  0 29 68  3

my second test is,
pvcreate /dev/md0
vgcreate DG5 /dev/md0
lvcreate -L 200G -n vd1 DG5
dd if=/dev/mapper/DG5-vd1 of=/dev/null bs=1M The sequence read performance dropped to 340MB/s

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  76440 3327728 258884    0    0 344612     0 3319 7564  0 19 65 16
 0  1      0  29920 3374372 258636    0    0 345088     0 3290 7620  0 21 67 12
 1  0      0  32608 3371800 258856    0    0 334464     0 3154 7513  0 19 67 14
 0  1      0  32340 3371800 258784    0    0 341176     0 3289 7538  0 17 69 14
 0  1      0  31812 3372424 258516    0    0 337664     0 3148 7443  0 19 68 13
 2  0      0  23964 3380052 258372    0    0 337792     0 3227 7656  0 19 67 14
 1  0      0  38496 3365564 258696    0    0 344564     0 3297 7690  0 20 66 13
 2  0      0  24120 3379592 258584    0    0 339468     0 3191 7448  0 19 68 13
 1  1      0  25072 3378380 258640    0    0 338360     0 3153 7471  0 17 73 11
 1  1      0  61488 3341904 258808    0    0 340992     0 3257 7721  0 21 64 15
 0  1      0  51504 3352048 258568    0    0 342528     0 3270 7648  0 20 65 15
 1  0      0  89064 3314560 258508    0    0 340224     0 3222 7764  0 20 68 13
 1  0      0  32008 3371572 258740    0    0 337716     0 3218 7302  0 21 63 17
 0  1      0  80608 3322828 258972    0    0 335364     0 3157 7661  0 18 62 21

root:~# uname -a
Linux ustor 2.6.26.2 #1 SMP PREEMPT Sun Sep 28 11:58:17 CST 2008 x86_64 x86_64 x86_64 GNU/Linux

root:~# lvm version
  LVM version:     2.02.39 (2008-06-27)
  Library version: 1.02.27 (2008-06-25)
  Driver version:  4.13.0
(The lvm snapshot merging patches has been applied)

I used the fowllowing scripts to monitor the /proc/diskstats,

root:~# while true; do clear; cat /proc/diskstats |grep -E "sd|md|dm" |awk '{printf "[%4s] %10s %10s %10s %10s\t%10s %10s %10s %10s\t%10s %10s %10s\n",$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14}'; sleep 1; done

I found that dm may hold too many IOs in its queue

Has anyone met with the same issues?
Sorry for my poor english

---------------------------------
锟脚伙拷锟斤拷锟戒，锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟戒！

[email protected]

2008-10-09 03:59 UTC #1 lore raw

Ben,

I have seen this same issue as well. I have created an md device capable of 425MB/sec using the hdparm -t command, yet an LVM volume fully comprising this md device only got about 150MB/sec. I am not sure what the issue is. I am running Ubuntu Hardy 804 server edition, 64-bit.

-Thomas

-----Original Message-----
From: Ben Huang <[email protected]>
To: [email protected]
Sent: Wed, 8 Oct 2008 12:51 am
Subject: [linux-lvm] device-mapper may have read performance issue

   My first test was
   dd if=/dev/md0 of=/dev/null bs=1M
   And I got around 780MB/s sequence read performance

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b   swpd   free   buff cache   si   so    bi    bo   in   cs us sy id wa 2 0      0 31272 3369856 258728    0    0
794680     0 3223 4120 0 35 61 3
1 0      0 24980 3371104 258664A
0   0    0 788484     0 3271 3909 0 31 67 2
1 0      0 35208 3363600 258760    0    0 792572     0 3288 4056 0 34 65 1 0 1      0 22304 3380164 258560    0    0 753668     0 2875 3775 0 29 70 1 1 0      0 41272 3361048 258660    0    0 770048     0 3124 3977 0 32 66 2 1 0      0 32636 3366976 258536    0    0 759485     0 2920 3823 0 33 64 3
1 0      0 21796 3376132 258632    0    0 776184     0 3053 3995 0 33 66 2 1 0      0 28612 3373608 258872    0    0 758148     0 3050 3907 0 30 68 3 3 0      0 35768 3366948 258504    0    0 778240     0 3154 3873 0 32 66 1 1 0      0 25588 3377328 258864    0    0 737296     0 2913 3262 0 30 68 2 1 0      0 31264 3370300 258828    0 =C
2 0 792832     0 3339 2934 0 34 63 2
0 1      0 21532 3380072
258708    0    0 755512     0 3067 2986 0 29 68 3

    my second test is,
    pvcreate /dev/md0
    vgcreate DG5 /dev/md0
    lvcreate -L 200G -n vd1 DG5
    dd if=/dev/mapper/DG5-vd1 of=/dev/null bs=1M The sequence read performance dropped to 340MB/s

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b   swpd   free   buff cache   si   so    bi    bo   in   cs us sy id wa
1 0      0 76440 3327728 258884    0    0 344612     0 3319 7564 0 19 65 16 0 1      0 29920 3374372 258636    0    0 345088     0 3290 7620 0 21 67 12 1 0      0 32608 3371800 258856    0    0 334464     0 3154 7513 0 19 67 14 0 1      0 32340 3371800 258784    0    0 341176     0 3289 7538 0 17 69 14 0 1      0 31812 3372424258516    0    0 337664     0 3148 7443 0 19 68 13 2 0      0 23964 3380052
258372    0    0 337792     0 3227 7656 0 19 67 14
1 0      0 38496 3365564 258696    0    0 344564     0 3297 7690 0 20 66 13 2 0      0 24120 3379592 258584    0    0 339468     0 3191 7448 0 19 68 13 1 1      0 25072 3378380 258640    0    0 338360     0 3153 7471 0 17 73 11 1 1      0 61488 3341904 258808    0    0 340992     0 3257 7721 0 21 64 15 0 1      0 51504 3352048 258568    0    0 342528     0 3270 7648 0 20 65 15
1 0      0 89064 3314560 258508    0    0 340224     0 3222 7764 0 20 68 13 1 0      0 32008 3371572 258740    0    0 337716     0 3218 7302 0 21 63 17 0 1      0 80608 3322828 258972    0    0 335364   A
0 0 3157 7661 0 18 62 21

root:~# uname -a
Linux ustor 2.6.26.2 #1 SMP PREEMPT Sun Sep 28 11:58:17 CST 2008 x86_64 x86_64 x86_64 GNU/Linux

root:~# lvm version
  LVM version:     2.02.39 (2008-06-27)
  Library version: 1.02.27 (2008-06-25)
  Driver version:  4.13.0
(The lvm snapshot merging patches has been applied)

I used the fowllowing scripts to monitor the
/proc/diskstats,

root:~# while true; do clear; cat /proc/diskstats |grep -E "sd|md|dm" |awk '{printf "[%4s] %10s %10s %10s %10s\t%10s %10s %10s %10s\t%10s %10s %10s\n",$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14}'; sleep 1; done

I found that dm may hold too many IOs in its queue

Has anyone met with the same issues?
Sorry for my poor english

------------------------------------------------------------
雅虎邮箱，您的终生邮箱！

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Peter Keller

2008-10-09 09:53 UTC #2 lore raw

Hi all,

Quoted 5 lines · On Wed, 8 Oct 2008, [email protected] wrote:

Ben,
I have seen this same issue as well. I have created an md device capable of 425MB/sec using the hdparm -t command, yet an LVM volume fully comprising this md device only got about 150MB/sec. I am not sure what the issue is. I am running Ubuntu Hardy 804 server edition, 64-bit.
-Thomas

I have fixed this kind of problem by tweaking the readahead of the LVM volume using 'blockdev --setra' and/or 'blockdev --setfra'.

I originally found this tip here: http://www.beowulf.org/archive/2007-May/018359.html

Good luck,
Peter.

Quoted 130 lines

-----Original Message-----
From: Ben Huang <[email protected]>
To: [email protected]
Sent: Wed, 8 Oct 2008 12:51 am
Subject: [linux-lvm] device-mapper may have read performance issue
HI all,
     I found a LVM+dm read performance issue on my storage server. On my system, I have a 12+0 RAID5 md, which was created by the following command mdadm -C /dev/md0 -l5 -n12 /dev/sd{a,b,l,d,e,f,g,h,i,j,k,o}
--assume-clean --metadata=1
   My first test was
   dd if=/dev/md0 of=/dev/null bs=1M
   And I got around 780MB/s sequence read performance
vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b   swpd   free   buff cache   si   so    bi    bo   in   cs us sy id wa 2 0      0 31272 3369856 258728    0    0
794680     0 3223 4120 0 35 61 3
1 0      0 24980 3371104 258664A
0   0    0 788484     0 3271 3909 0 31 67 2
1 0      0 35208 3363600 258760    0    0 792572     0 3288 4056 0 34 65 1 0 1      0 22304 3380164 258560    0    0 753668     0 2875 3775 0 29 70 1 1 0      0 41272 3361048 258660    0    0 770048     0 3124 3977 0 32 66 2 1 0      0 32636 3366976 258536    0    0 759485     0 2920 3823 0 33 64 3
1 0      0 21796 3376132 258632    0    0 776184     0 3053 3995 0 33 66 2 1 0      0 28612 3373608 258872    0    0 758148     0 3050 3907 0 30 68 3 3 0      0 35768 3366948 258504    0    0 778240     0 3154 3873 0 32 66 1 1 0      0 25588 3377328 258864    0    0 737296     0 2913 3262 0 30 68 2 1 0      0 31264 3370300 258828    0 =C
2 0 792832     0 3339 2934 0 34 63 2
0 1      0 21532 3380072
258708    0    0 755512     0 3067 2986 0 29 68 3
    my second test is,
    pvcreate /dev/md0
    vgcreate DG5 /dev/md0
    lvcreate -L 200G -n vd1 DG5
    dd if=/dev/mapper/DG5-vd1 of=/dev/null bs=1M The sequence read performance dropped to 340MB/s
vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b   swpd   free   buff cache   si   so    bi    bo   in   cs us sy id wa
1 0      0 76440 3327728 258884    0    0 344612     0 3319 7564 0 19 65 16 0 1      0 29920 3374372 258636    0    0 345088     0 3290 7620 0 21 67 12 1 0      0 32608 3371800 258856    0    0 334464     0 3154 7513 0 19 67 14 0 1      0 32340 3371800 258784    0    0 341176     0 3289 7538 0 17 69 14 0 1      0 31812 3372424258516    0    0 337664     0 3148 7443 0 19 68 13 2 0      0 23964 3380052
258372    0    0 337792     0 3227 7656 0 19 67 14
1 0      0 38496 3365564 258696    0    0 344564     0 3297 7690 0 20 66 13 2 0      0 24120 3379592 258584    0    0 339468     0 3191 7448 0 19 68 13 1 1      0 25072 3378380 258640    0    0 338360     0 3153 7471 0 17 73 11 1 1      0 61488 3341904 258808    0    0 340992     0 3257 7721 0 21 64 15 0 1      0 51504 3352048 258568    0    0 342528     0 3270 7648 0 20 65 15
1 0      0 89064 3314560 258508    0    0 340224     0 3222 7764 0 20 68 13 1 0      0 32008 3371572 258740    0    0 337716     0 3218 7302 0 21 63 17 0 1      0 80608 3322828 258972    0    0 335364   A
0 0 3157 7661 0 18 62 21
root:~# uname -a
Linux ustor 2.6.26.2 #1 SMP PREEMPT Sun Sep 28 11:58:17 CST 2008 x86_64 x86_64 x86_64 GNU/Linux
root:~# lvm version
  LVM version:     2.02.39 (2008-06-27)
  Library version: 1.02.27 (2008-06-25)
  Driver version:  4.13.0
(The lvm snapshot merging patches has been applied)
I used the fowllowing scripts to monitor the
/proc/diskstats,
root:~ # while true; do clear; cat /proc/diskstats |grep -E "sd|md|dm" |awk root:~ '{printf "[%4s] %10s %10s %10s %10s\t%10s %10s %10s %10s\t%10s %10s root:~ %10s\n",$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14}'; sleep 1; done
I found that dm may hold too many IOs in its queue
Has anyone met with the same issues?
Sorry for my poor english
------------------------------------------------------------
雅虎邮箱，您的终生邮箱！
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Quoted 6 lines

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

--
Peter Keller Tel.: +44 (0)1223 353033
Global Phasing Ltd., Fax.: +44 (0)1223 366889
Sheraton House,
Castle Park,
Cambridge CB3 0AX
United Kingdom

Peter Keller

2008-10-09 10:00 UTC #3 lore raw

Quoted 11 lines · Peter Keller

Hi all,
Quoted 7 lines · On Wed, 8 Oct 2008, [email protected] wrote:
Ben,
I have seen this same issue as well. I have created an md device capable of 425MB/sec using the hdparm -t command, yet an LVM volume fully comprising this md device only got about 150MB/sec. I am not sure what the issue is. I am running Ubuntu Hardy 804 server edition, 64-bit.

Quoted 8 lines

Quoted 1 line
-Thomas
I have fixed this kind of problem by tweaking the readahead of the LVM volume using 'blockdev --setra' and/or 'blockdev --setfra'.
I originally found this tip here: http://www.beowulf.org/archive/2007-May/018359.html
Good luck,
Peter.

I should also have said: don't forget to use 'blockdev --flushbufs' between tests.

Regards,
Peter.

Quoted 162 lines

Quoted 155 lines
-----Original Message-----
From: Ben Huang <[email protected]>
To: [email protected]
Sent: Wed, 8 Oct 2008 12:51 am
Subject: [linux-lvm] device-mapper may have read performance issue
HI all,
     I found a LVM+dm read performance issue on my storage server. On my system, I have a 12+0 RAID5 md, which was created by the following command mdadm -C /dev/md0 -l5 -n12 /dev/sd{a,b,l,d,e,f,g,h,i,j,k,o} --assume-clean --metadata=1
   My first test was
   dd if=/dev/md0 of=/dev/null bs=1M
   And I got around 780MB/s sequence read performance
vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b   swpd   free   buff cache   si   so    bi    bo   in   cs us sy id wa
2 0      0 31272 3369856 258728    0    0
794680     0 3223 4120 0 35 61 3
1 0      0 24980 3371104 258664A
0   0    0 788484     0 3271 3909 0 31 67 2
1 0      0 35208 3363600 258760    0    0 792572     0 3288 4056 0 34 65 1
0 1      0 22304 3380164 258560    0    0 753668     0 2875 3775 0 29 70 1
1 0      0 41272 3361048 258660    0    0 770048     0 3124 3977 0 32 66 2
1 0      0 32636 3366976 258536    0    0 759485     0 2920 3823 0 33 64
3
1 0      0 21796 3376132 258632    0    0 776184     0 3053 3995 0 33 66 2
1 0      0 28612 3373608 258872    0    0 758148     0 3050 3907 0 30 68 3
3 0      0 35768 3366948 258504    0    0 778240     0 3154 3873 0 32 66 1
1 0      0 25588 3377328 258864    0    0 737296     0 2913 3262 0 30 68 2
1 0      0 31264 3370300 258828    0 =C
2 0 792832     0 3339 2934 0 34 63 2
0 1      0 21532 3380072
258708    0    0 755512     0 3067 2986 0 29 68 3
    my second test is,
    pvcreate /dev/md0
    vgcreate DG5 /dev/md0
    lvcreate -L 200G -n vd1 DG5
    dd if=/dev/mapper/DG5-vd1 of=/dev/null bs=1M The sequence read performance dropped to 340MB/s
vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b   swpd   free   buff cache   si   so    bi    bo   in   cs us sy id wa
1 0      0 76440 3327728 258884    0    0 344612     0 3319 7564 0 19 65 16
0 1      0 29920 3374372 258636    0    0 345088     0 3290 7620 0 21 67 12
1 0      0 32608 3371800 258856    0    0 334464     0 3154 7513 0 19 67 14
0 1      0 32340 3371800 258784    0    0 341176     0 3289 7538 0 17 69 14
0 1      0 31812 3372424258516    0    0 337664     0 3148 7443 0 19 68 13
2 0      0 23964 3380052
258372    0    0 337792     0 3227 7656 0 19 67 14
1 0      0 38496 3365564 258696    0    0 344564     0 3297 7690 0 20 66 13
2 0      0 24120 3379592 258584    0    0 339468     0 3191 7448 0 19 68 13
1 1      0 25072 3378380 258640    0    0 338360     0 3153 7471 0 17 73 11
1 1      0 61488 3341904 258808    0    0 340992     0 3257 7721 0 21 64 15
0 1      0 51504 3352048 258568    0    0 342528     0 3270 7648 0 20 65 15
1 0      0 89064 3314560 258508    0    0 340224     0 3222 7764 0 20 68 13
1 0      0 32008 3371572 258740    0    0 337716     0 3218 7302 0 21 63 17
0 1      0 80608 3322828 258972    0    0 335364   A
0 0 3157 7661 0 18 62 21
 root:~# uname -a
 Linux ustor 2.6.26.2 #1 SMP PREEMPT Sun Sep 28 11:58:17 CST 2008 x86_64
 x86_64 x86_64 GNU/Linux
 root:~# lvm version
   LVM version:     2.02.39 (2008-06-27)
   Library version: 1.02.27 (2008-06-25)
   Driver version:  4.13.0
 (The lvm snapshot merging patches has been applied)
I used the fowllowing scripts to monitor the
/proc/diskstats,
root:~ # while true; do clear; cat /proc/diskstats |grep -E "sd|md|dm" root:~ |awk '{printf "[%4s] %10s %10s %10s %10s\t%10s %10s %10s %10s\t%10s root:~ %10s %10s\n",$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14}'; sleep 1; root:~ done
I found that dm may hold too many IOs in its queue
Has anyone met with the same issues?
Sorry for my poor english
------------------------------------------------------------
雅虎邮箱，您的终生邮箱！
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 6 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Quoted 8 lines

--
Peter Keller Tel.: +44 (0)1223 353033
Global Phasing Ltd., Fax.: +44 (0)1223 366889
Sheraton House,
Castle Park,
Cambridge CB3 0AX
United Kingdom

--
Peter Keller Tel.: +44 (0)1223 353033
Global Phasing Ltd., Fax.: +44 (0)1223 366889
Sheraton House,
Castle Park,
Cambridge CB3 0AX
United Kingdom

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Ben Huang

author

2008-10-10 08:30 UTC #4 lore raw

In reply to Peter Keller

Hi all,

--- Peter Keller <[email protected]>д��:

Quoted 13 lines

Hi all,
Quoted 3 lines · On Wed, 8 Oct 2008, [email protected] wrote:
Ben,
I have seen this same issue as well. I have
created an md device capable of
Quoted 1 line
425MB/sec using the hdparm -t command, yet an LVM
volume fully comprising
Quoted 1 line
this md device only got about 150MB/sec. I am not
sure what the issue is. I
Quoted 1 line
am running Ubuntu Hardy 804 server edition,

Quoted 8 lines

64-bit.
Quoted 2 lines
-Thomas
I have fixed this kind of problem by tweaking the
readahead of the LVM
volume using 'blockdev --setra' and/or 'blockdev
--setfra'.

It make effect

blockdev --setra 65536 /dev/md0 blockdev --setra 65536 /dev/mapper/DG5-lv1

dd if=/dev/md0 of=/dev/null bs=1M 353185562624 bytes (353 GB) copied, 420.033 s, 841 MB/s

dd if=/dev/mapper/DG5-lv1 of=/dev/null bs=1M 155551006720 bytes (156 GB) copied, 216.576 s, 718 MB/s

Warm regards,
-Ben

      ___________________________________________________________

�Ż��䣬��䣡
http://cn.mail.yahoo.com/

Eugene Vilensky

2008-10-11 04:18 UTC #5 lore raw

Hi all,
How important are these read ahead settings for random, database IO?

Quoted 54 lines · Ben Huang

Hi all,
--- Peter Keller <[email protected]>д��:
Quoted 13 lines
Hi all,
Quoted 3 lines · On Wed, 8 Oct 2008, [email protected] wrote:
Ben,
I have seen this same issue as well. I have
created an md device capable of
Quoted 1 line
425MB/sec using the hdparm -t command, yet an LVM
volume fully comprising
Quoted 1 line
this md device only got about 150MB/sec. I am not
sure what the issue is. I
Quoted 1 line
am running Ubuntu Hardy 804 server edition,
Quoted 8 lines
64-bit.
Quoted 2 lines
-Thomas
I have fixed this kind of problem by tweaking the
readahead of the LVM
volume using 'blockdev --setra' and/or 'blockdev
--setfra'.
It make effect
blockdev --setra 65536 /dev/md0 blockdev --setra 65536 /dev/mapper/DG5-lv1
dd if=/dev/md0 of=/dev/null bs=1M 353185562624 bytes (353 GB) copied, 420.033 s, 841 MB/s
dd if=/dev/mapper/DG5-lv1 of=/dev/null bs=1M 155551006720 bytes (156 GB) copied, 216.576 s, 718 MB/s
Warm regards,
-Ben
      ___________________________________________________________
�Ż��䣬��䣡
http://cn.mail.yahoo.com/
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Signature · 4 lines

Sent from my mobile device

Eugene Vilensky
[email protected]

Peter Keller

2008-10-14 15:50 UTC #6 lore raw

Hi,

Quoted 2 lines · Eugene Vilensky

Hi all,
How important are these read ahead settings for random, database IO?

Good question: by coincidence I've just had to exercise a system that has LVM2 volumes set up on a hardware RAID-6 array (SATA-II disks and an Adaptec ICP5085BL card). I took the opportunity to do some bonnie++ tests with readahead settings of 1024 and 8192. I was running 5 instances of bonnie++ at the same time (each on its own LVM volume, but locked together with the -p/-y options). I got numbers like these:

Readahead 1024:

                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
server1 (buildho 16  3946  99 +++++ +++  2957 100  4125  99 +++++ +++ 11400  88

Readahead 8192:

                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
server1 (buildho 16  3968  99 +++++ +++  5934 100  4113  99 +++++ +++  5037  99

So, on this system readahead makes no difference for create, and read is so quick that bonnie++ doesn't even want to report any numbers. However, sequential delete is about twice as fast with the larger readahead, and random delete is twice as fast with the smaller readahead.

It is hard to tune for general usage as opposed to specific tasks, and by making some things better you may make other things worse. I responded to the original question on the assumption that doing a sequential dd represented something like the expected usage pattern for the disks.

On the basis of this paticular (very limited) test on this particular system, I would answer Eugene's question by saying that the readahead setting is equally important for random and sequential I/O, but in opposite directions :-)

YMMV, especially on a typical desktop system where the disks are connected directly to the motherboard or via an eSATA port, and if RAID is being used at all it is likely to be software RAID-0 or RAID-1

Regards,
Peter.

Quoted 50 lines · Ben Huang

Quoted 48 lines
Hi all,
--- Peter Keller <[email protected]>写锟斤拷:
Quoted 13 lines
Hi all,
Quoted 3 lines · On Wed, 8 Oct 2008, [email protected] wrote:
Ben,
I have seen this same issue as well. I have
created an md device capable of
Quoted 1 line
425MB/sec using the hdparm -t command, yet an LVM
volume fully comprising
Quoted 1 line
this md device only got about 150MB/sec. I am not
sure what the issue is. I
Quoted 1 line
am running Ubuntu Hardy 804 server edition,
Quoted 8 lines
64-bit.
Quoted 2 lines
-Thomas
I have fixed this kind of problem by tweaking the
readahead of the LVM
volume using 'blockdev --setra' and/or 'blockdev
--setfra'.
It make effect
blockdev --setra 65536 /dev/md0 blockdev --setra 65536 /dev/mapper/DG5-lv1
dd if=/dev/md0 of=/dev/null bs=1M 353185562624 bytes (353 GB) copied, 420.033 s, 841 MB/s
dd if=/dev/mapper/DG5-lv1 of=/dev/null bs=1M 155551006720 bytes (156 GB) copied, 216.576 s, 718 MB/s
Warm regards,
-Ben
      ___________________________________________________________
锟脚伙拷锟斤拷锟戒，锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟戒！
http://cn.mail.yahoo.com/

Quoted 19 lines

Quoted 6 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Signature · 10 lines
Sent from my mobile device

Eugene Vilensky
[email protected]

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

--
Peter Keller Tel.: +44 (0)1223 353033
Global Phasing Ltd., Fax.: +44 (0)1223 366889
Sheraton House,
Castle Park,
Cambridge CB3 0AX
United Kingdom