Skip to contentSkip to conversation

How can I boost block I/O performance

DISCUSSION Dave Pitts 5 participants 10 messages May 6, 2006 – May 8, 2006

lore
Share & tools
lore

Dave Pitts

author
#0 lore raw

Hello all:

I've been trying some hacks to boost disk I/O performance mostly by changing values in the /proc/sys/vm filesystem. A vmstat display shows bursty block out counts with fairly consistent interrupt counts:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
4  0    720  80252   1820 7077456    0    0     9   852    5    11  1 14 84  0
1  0    720  80444   1820 7077456    0    0     0     0 15923 77712  4 20 76  0
2  0    720  80196   1820 7077524    0    0     0     0 14705 87207  4 18 79  0
2  0    720  79732   1828 7077856    0    0     8   116 16235 84459  4 20 76  0
4  0    720 104172   1812 7051964    0    0    24 62568 20447 73499  4 27 69  0
2  0    720 105172   1812 7051964    0    0     0 90740 16960 80149  1 21 78  0
2  0    720 104108   1812 7051964    0    0     0     0 14162 72632  3 13 85  0
4  0    720 103980   1812 7052032    0    0     0     0 13495 68133  4 16 80  0
1  0    720 103868   1820 7052704    0    0     0   128 15417 59969  4 17 79  0
0  0    720 104340   1828 7052696    0    0     0   280 19504 74281  0  8 92  0
0  0    720 104532   1828 7052696    0    0     0     0 14736 70017  0  5 95  0
1  0    720 104596   1828 7052696    0    0     0     0 16006 73173  0  6 94  0
2  0    720  92844   1828 7064256    0    0    12     0 16508 80601  0  9 91  0
2  0    720  91916   1836 7064248    0    0     4   104 20787 74676  0  7 92  0
0  0    720  92580   1844 7064240    0    0     0 14640 17789 71545  0 10 90  0
1  0    720  92900   1844 7064240    0    0     0     0 15460 74760  0  8 92  0
0  0    720  92668   1844 7065260    0    0     0     0 18585 77435  0  7 93  0
0  0    720  92604   1844 7065260    0    0     0     0 19187 86426  0  9 91  0
2  0    720  91964   1860 7065244    0    0     0   140 23659 87962  0  8 92  0
5  0    720  90364   1860 7067080    0    0    40 66956 17995 95384  0 17 82  0

This test is running several NFS clients to a RAID disk storage array. I also see the same behavior when running SFTP transfers. What I'd like is a more even block out behavior (even at the expense of other apps as this is a file server not an app server). The values that I've been hacking are the dirty_writeback_centisecs, dirty_background_ratio, etc. Am I barking up the wrong tree?

Thanks in advance.

Signature · 3 lines
Dave Pitts                   PULLMAN: Travel and sleep in safety and comfort.
[email protected]              My other RV IS a Pullman (Colorado Pine).
http://www.cozx.com/~dpitts
#1 lore raw

Does the adaptive readahead patch help in your case? Other people in similar situations are saying that it helps a lot.

Wu Fengguang
Subject [PATCH 00/23] Adaptive read-ahead V11
http://lkml.org/lkml/2006/3/18/235

I'm not sure of the status of this patch. I looked in the mm tree and didn't see it.

--
Jon Smirl
[email protected]

#3 lore raw
In reply to Dave Pitts
Quoted 10 lines · Dave Pitts

Hello all:

I've been trying some hacks to boost disk I/O performance mostly by changing values in the /proc/sys/vm filesystem. A vmstat display shows bursty block out counts with fairly consistent interrupt counts:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
4  0    720  80252   1820 7077456    0    0     9   852    5    11  1 14 84  0

[...]

Quoted 7 lines

5 0 720 90364 1860 7067080 0 0 40 66956 17995 95384 0 17 82 0

This test is running several NFS clients to a RAID disk storage array. I also see the same behavior when running SFTP transfers. What I'd like is a more even block out behavior (even at the expense of other apps as this is a file server not an app server). The values that I've been hacking are the dirty_writeback_centisecs, dirty_background_ratio, etc. Am I barking up the wrong tree?

No iowait time, plenty of idle time: looks like you are network bound. What time of network are you running?

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#2 lore raw
In reply to Jon Smirl
Quoted 7 lines · Jon Smirl

Does the adaptive readahead patch help in your case? Other people in similar situations are saying that it helps a lot.

Wu Fengguang
Subject [PATCH 00/23] Adaptive read-ahead V11
http://lkml.org/lkml/2006/3/18/235

Adaptive readahead is not going to help his write performance...

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
author
#4 lore raw
In reply to Avi Kivity
Quoted 17 lines · Avi Kivity
Quoted 10 lines · Dave Pitts

Hello all:

I've been trying some hacks to boost disk I/O performance mostly by changing values in the /proc/sys/vm filesystem. A vmstat display shows bursty block out counts with fairly consistent interrupt counts:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
4  0    720  80252   1820 7077456    0    0     9   852    5    11  1 14 84  0

[...]

Quoted 2 lines

5 0 720 90364 1860 7067080 0 0 40 66956 17995 95384 0 17 82 0

Quoted 8 lines
Quoted 5 lines

This test is running several NFS clients to a RAID disk storage array. I also see the same behavior when running SFTP transfers. What I'd like is a more even block out behavior (even at the expense of other apps as this is a file server not an app server). The values that I've been hacking are the dirty_writeback_centisecs, dirty_background_ratio, etc. Am I barking up the wrong tree?

No iowait time, plenty of idle time: looks like you are network bound. What time of network are you running?

Well, it's an 8 cpu system. Does the idle time reflect the idle time of all cpu's? The network is a Gigabit Ethernet.

Signature · 3 lines
Dave Pitts                   PULLMAN: Travel and sleep in safety and comfort.
[email protected]              My other RV IS a Pullman (Colorado Pine).
http://www.cozx.com/~dpitts
#8 lore raw
In reply to Dave Pitts
Quoted 3 lines · Dave Pitts

Hello all:

I've been trying some hacks to boost disk I/O performance

[snip]

Quoted 2 lines

This test is running several NFS clients to a RAID disk storage array. I

[snip]

For improving performance of NFS servers I've often had good success with increasing the 'rsize' and 'wsize' options.
The default values are 4096, I personally set them to 16384 which usually helps NFS performance quite a bit. At least that's my experience.
Simply add rsize=16384,wsize=16384 to the nfs mount options in /etc/fstab and see if that improves performance for you (values like 8192 and 32768 may also be worth testing, but personally I've found - at least with my setups - that 16384 seems to be the magic value). ('man 8 mount' and 'man 5 exports' also have more interresting options you may want to experiment with, but just rsize & wsize on their own should be a boost)

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#5 lore raw
In reply to Dave Pitts
Quoted 19 lines · Dave Pitts
Quoted 17 lines · Avi Kivity
Quoted 5 lines · Dave Pitts

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
4  0    720  80252   1820 7077456    0    0     9   852    5    11  1 14 84  0

[...]

Quoted 7 lines

5 0 720 90364 1860 7067080 0 0 40 66956 17995 95384 0 17 82 0

This test is running several NFS clients to a RAID disk storage array. I also see the same behavior when running SFTP transfers. What I'd like is a more even block out behavior (even at the expense of other apps as this is a file server not an app server). The values that I've been hacking are the dirty_writeback_centisecs, dirty_background_ratio, etc. Am I barking up the wrong tree?

Quoted 3 lines
Quoted 2 lines

No iowait time, plenty of idle time: looks like you are network bound. What time of network are you running?

Well, it's an 8 cpu system. Does the idle time reflect the idle time of all cpu's?

It's an average across all cpus. But the numbers are low even for a single cpu system.

Quoted 2 lines

The network is a Gigabit Ethernet.

I'd make sure the nics know that by running ethtool (on the clients as well as on the server).

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#9 lore raw
In reply to Jesper Juhl
Quoted 9 lines · Jesper Juhl
Quoted 3 lines · Dave Pitts

Hello all:

I've been trying some hacks to boost disk I/O performance

[snip]

Quoted 2 lines

This test is running several NFS clients to a RAID disk storage array. I

[snip]

Quoted 12 lines

For improving performance of NFS servers I've often had good success with increasing the 'rsize' and 'wsize' options.
The default values are 4096, I personally set them to 16384 which usually helps NFS performance quite a bit. At least that's my experience.
Simply add rsize=16384,wsize=16384 to the nfs mount options in /etc/fstab and see if that improves performance for you (values like 8192 and 32768 may also be worth testing, but personally I've found - at least with my setups - that 16384 seems to be the magic value). ('man 8 mount' and 'man 5 exports' also have more interresting options you may want to experiment with, but just rsize & wsize on their own should be a boost)

Another way is to increase the number of nfsd threads on the server. See rpc.nfsd(8). I usually increase it from the default 8 to 32 on busy machines.

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
author
#6 lore raw
In reply to Avi Kivity
Quoted 24 lines · Avi Kivity
Quoted 22 lines · Dave Pitts
Quoted 20 lines · Avi Kivity
Quoted 5 lines · Dave Pitts

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
4  0    720  80252   1820 7077456    0    0     9   852    5    11  1 14 84  0

[...]

Quoted 7 lines

5 0 720 90364 1860 7067080 0 0 40 66956 17995 95384 0 17 82 0

This test is running several NFS clients to a RAID disk storage array. I also see the same behavior when running SFTP transfers. What I'd like is a more even block out behavior (even at the expense of other apps as this is a file server not an app server). The values that I've been hacking are the dirty_writeback_centisecs, dirty_background_ratio, etc. Am I barking up the wrong tree?

Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

No iowait time, plenty of idle time: looks like you are network bound. What time of network are you running?

Well, it's an 8 cpu system. Does the idle time reflect the idle time of all cpu's?

It's an average across all cpus. But the numbers are low even for a single cpu system.

OK, I ran some tests, bypassing NFS, and got the following vmstat display:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
4  1    696  80692    844 7225424    0    0    92 263924 14380  6176  2 40 46 12
6  0    696 108668    784 7200936    0    0   456  6444 30515 20641  3 57 36  3
5  1    696 100916    768 7211220    0    0   508 97276 31761 19916  3 65 27  4
7  0    700  91188    764 7217340    0    0   428 121576 28704 21957  3 69 24 3
3  3    700 103356    780 7204472    0    0   408 121748 29513 22603  3 66 23 8
7  0    700  92836    780 7216168    0    0   360 43508 28784 21410  3 54 33 11
7  0    700  88364    768 7224272    0    0   296 158236 26530 17570  3 66 21 10
10  0    700  91068    776 7219096    0    0   444 141456 30306 16053  3 74 16 7
5  2    700 102212    752 7206676    0    0   340 170076 29249 14872  2 69 19 10
11  0    700  87884    768 7222096    0    0   392 143312 29743 19808  1 65 23 11
8  0    700 104692    744 7204644    0    0   240 159624 25814 18747  3 58 20 19
9  0    700 107196    736 7205196    0    0   344 148500 28191 18113  3 70 21 6
16  1    700  99300    768 7211364    0   12   464 164348 25671 18326  4 64 18 15
5  4    700 107412   1052 7204824    0    0   936 170904 28994 17062  3 73 11 14
8  1    700  98892   1284 7217512    0    0   596 182708 31520 18424  1 76 13 10

This ia with 6 concurrent data streams. In addition to the /proc/sys/vm/dirty* values I also adjusted values in mm/page-writeback.c eg. MAX_WRITEBACK_PAGES to 8192 . Our goal is to blast out as much as possible per pdflush invocation.

Quoted 8 lines
Quoted 3 lines

The network is a Gigabit Ethernet.

I'd make sure the nics know that by running ethtool (on the clients as well as on the server).

Signature · 3 lines
Dave Pitts                   PULLMAN: Travel and sleep in safety and comfort.
[email protected]              My other RV IS a Pullman (Colorado Pine).
http://www.cozx.com/~dpitts
#7 lore raw
Quoted 22 lines · Dave Pitts

OK, I ran some tests, bypassing NFS, and got the following vmstat display:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
4  1    696  80692    844 7225424    0    0    92 263924 14380  6176  2 40 46 12
6  0    696 108668    784 7200936    0    0   456  6444 30515 20641  3 57 36  3
5  1    696 100916    768 7211220    0    0   508 97276 31761 19916  3 65 27  4
7  0    700  91188    764 7217340    0    0   428 121576 28704 21957  3 69 24 3
3  3    700 103356    780 7204472    0    0   408 121748 29513 22603  3 66 23 8
7  0    700  92836    780 7216168    0    0   360 43508 28784 21410  3 54 33 11
7  0    700  88364    768 7224272    0    0   296 158236 26530 17570  3 66 21 10
10  0    700  91068    776 7219096    0    0   444 141456 30306 16053  3 74 16 7
5  2    700 102212    752 7206676    0    0   340 170076 29249 14872  2 69 19 10
11  0    700  87884    768 7222096    0    0   392 143312 29743 19808  1 65 23 11
8  0    700 104692    744 7204644    0    0   240 159624 25814 18747  3 58 20 19
9  0    700 107196    736 7205196    0    0   344 148500 28191 18113  3 70 21 6
16  1    700  99300    768 7211364    0   12   464 164348 25671 18326  4 64 18 15
5  4    700 107412   1052 7204824    0    0   936 170904 28994 17062  3 73 11 14
8  1    700  98892   1284 7217512    0    0   596 182708 31520 18424  1 76 13 10
Quoted 4 lines

This ia with 6 concurrent data streams. In addition to the /proc/sys/vm/dirty* values I also adjusted values in mm/page-writeback.c eg. MAX_WRITEBACK_PAGES to 8192 . Our goal is to blast out as much as possible per pdflush invocation.

These are healthier numbers. How many disks are on that machine? What raid level? Does it have battery backed up cache?

Anyway you need to check you network, since it looks like block throughput is not the issue.

Signature · 1 line
error compiling committee.c: too many arguments to function