Skip to contentSkip to conversation

disk performance strange/low

DISCUSSION Michael Monnerie 2 participants 6 messages Aug 17, 2009 – Aug 26, 2009 XFS filesystem

lore
Share & tools
lore

Michael Monnerie

author
#0 lore raw

I've got an Areca RAID Controller with 5x 1TB disks, and in the last time I often found it to be strange. Now I investigated, and found this.

Quoted 1 line

From another server I NFS mount a share, and make

dd if=/dev/sda of=/server/share/test.dd bs=1024k
so that should do big sequential writes. But the performance goes up and down, like you see in this output from "iostat -kx 5 555". You see write KB/s is 5824, 23207, 87283, 704, 40187. Up and down all the time. Could that be solved by the patch you discussed last time, or is there something else?

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
sdb 0,00 401,20 1,20 77,09 4,78 5824,00
148,91 152,25 2595,77 12,76 99,92

avg-cpu: %user %nice %system %iowait %steal %idle

           0,00    0,00   34,77   67,03    2,20    7,21

sdb 0,00 1094,40 2,20 245,80 8,80 23207,20
187,23 153,64 43,24 4,03 100,00

avg-cpu: %user %nice %system %iowait %steal %idle

           0,00    0,00   41,08   56,83    0,30   15,65

sdb 0,00 295,20 6,40 791,20 25,60 87283,40
218,93 119,19 259,89 1,21 96,56

avg-cpu: %user %nice %system %iowait %steal %idle

           0,00    0,00   61,45   55,92    0,50    4,62

sdb 0,00 3,60 0,00 4,60 0,00 704,80
306,43 140,01 2979,30 217,39 100,00

avg-cpu: %user %nice %system %iowait %steal %idle

           0,00    0,00   21,12   75,50    0,10   10,06

sdb 0,00 1252,80 0,20 270,00 0,80 40187,20
297,47 151,38 1160,49 3,70 99,84

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 9 lines · Michael Monnerie

I've got an Areca RAID Controller with 5x 1TB disks, and in the last time I often found it to be strange. Now I investigated, and found this.

Quoted 1 line

From another server I NFS mount a share, and make

dd if=/dev/sda of=/server/share/test.dd bs=1024k
so that should do big sequential writes. But the performance goes up and down, like you see in this output from "iostat -kx 5 555". You see write KB/s is 5824, 23207, 87283, 704, 40187. Up and down all the time. Could that be solved by the patch you discussed last time, or is there something else?

Yes, that sounds like the problem of the VM not feeding us enough pages to write. What kernel is this on? Latest 2.6.31-rc has a workaround for it.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#2 lore raw
Quoted 3 lines · On Montag 17 August 2009 Christoph Hellwig wrote:

Yes, that sounds like the problem of the VM not feeding us enough pages to write.  What kernel is this on?  Latest 2.6.31-rc has a workaround for it.

2.6.27.23-0.1-xen from openSUSE 11.1
I can't switch as I use it with the free Xen. Thanks for the info.

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 7 lines · Michael Monnerie
Quoted 3 lines · On Montag 17 August 2009 Christoph Hellwig wrote:

Yes, that sounds like the problem of the VM not feeding us enough pages to write. ?What kernel is this on? ?Latest 2.6.31-rc has a workaround for it.

2.6.27.23-0.1-xen from openSUSE 11.1
I can't switch as I use it with the free Xen. Thanks for the info.

Try backporting commit c8a4051c3731b6db224482218cfd535ab9393ff8 from mainline - it should apply just fine to a 2.6.27-ish kernel.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#4 lore raw
Quoted 11 lines · On Dienstag 18 August 2009 Christoph Hellwig wrote:
Quoted 7 lines · Michael Monnerie
Quoted 3 lines · On Montag 17 August 2009 Christoph Hellwig wrote:

Yes, that sounds like the problem of the VM not feeding us enough pages to write. ?What kernel is this on? ?Latest 2.6.31-rc has a workaround for it.

2.6.27.23-0.1-xen from openSUSE 11.1
I can't switch as I use it with the free Xen. Thanks for the info.

Try backporting commit c8a4051c3731b6db224482218cfd535ab9393ff8 from mainline - it should apply just fine to a 2.6.27-ish kernel.

Thank you. I'm not a patcher, could you tell me how to do that?

mfg zmi

Signature · 5 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by Felix Blyakher
#5 lore raw
Quoted 7 lines · Michael Monnerie
Quoted 6 lines · On Dienstag 18 August 2009 Christoph Hellwig wrote:
Quoted 5 lines · Michael Monnerie
Quoted 3 lines · On Montag 17 August 2009 Christoph Hellwig wrote:

Yes, that sounds like the problem of the VM not feeding us enough pages to write. ?What kernel is this on? ?Latest 2.6.31-rc has a workaround for it.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

2.6.27.23-0.1-xen from openSUSE 11.1
I can't switch as I use it with the free Xen. Thanks for the info.

Try backporting commit c8a4051c3731b6db224482218cfd535ab9393ff8 from mainline - it should apply just fine to a 2.6.27-ish kernel.

Thank you. I'm not a patcher, could you tell me how to do that?

Just take the patch I attached and apply it using

patch -p1 < xfs-nr-to-write.diff

if you want to make sure it applies cleanly do a

	patch -p1 --dry-run < xfs-nr-to-write.diff

beforee to check if it doesn't cause any reject.

After that rebuild your kernel.

commit c8a4051c3731b6db224482218cfd535ab9393ff8

Author: Eric Sandeen <[email protected]>
Date:   Fri Jul 31 00:02:17 2009 -0500

xfs: bump up nr_to_write in xfs_vm_writepage

VM calculation for nr_to_write seems off. Bump it way up, this gets simple streaming writes zippy again. To be reviewed again after Jens' writeback changes.

Signed-off-by Christoph Hellwig <[email protected]>
Signed-off-by Eric Sandeen <[email protected]>
Cc Chris Mason <[email protected]>
Reviewed-by Felix Blyakher <[email protected]>
Signed-off-by Felix Blyakher <[email protected]>
fs/xfs/linux-2.6/xfs_aops.c +8 -0
index 7ec89fc..aecf251 100644
@@ -1268,6 +1268,14 @@ xfs_vm_writepage(
 	if (!page_has_buffers(page))
 		create_empty_buffers(page, 1 << inode->i_blkbits, 0);
 
+
+	/*
+	 *  VM calculation for nr_to_write seems off.  Bump it way
+	 *  up, this gets simple streaming writes zippy again.
+	 *  To be reviewed again after Jens' writeback changes.
+	 */
+	wbc->nr_to_write *= 4;
+
 	/*
 	 * Convert delayed allocate, unwritten or unmapped space
 	 * to real space and flush out to disk.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs