sunit not working
Hi list,
I have a HW raid 5 array with chunck size 256KB and stripe size 2560KB. Basically a 10+1+1 array. 10 drives, one parity, one spare.
I am trying to create an xfs file system with the appropriate sunit and width.
# mkfs.xfs -d sunit=512,swidth=5120 -f /dev/mapper/mpath9
meta-data=/dev/mapper/mpath9 isize=256 agcount=32,
agsize=56652352 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=1812874752,
imaxpct=25
= sunit=64 swidth=640 blks,
unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0As you can see the sunit gets set to 64 upon creation and not 512 like I asked. Also if it try to give it some mount options it does the same thing.
sgi210a:~ # mount -o sunit=512,swidth=5120 /dev/mapper/mpath9 /mnt/ sgi210a:~ # xfs_info /mnt/
meta-data=/dev/mapper/mpath9 isize=256 agcount=32,
agsize=56652352 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=1812874752,
imaxpct=25
= sunit=64 swidth=640 blks,
unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0sgi210a:~ #
Any ideas?
Last I tried to subscribe to the list by sending email to [email protected] a couple of times but was unsuccessful should I send email elsewhere to subscribe?
thank you
Rene
Quoted 5 lines · Rene
As you can see the sunit gets set to 64 upon creation and not 512 like I asked. Also if it try to give it some mount options it does the same thing.
sgi210a:~ # mount -o sunit=512,swidth=5120
/dev/mapper/mpath9/mnt/
Its being reported in units of filesystem blocks, and its specified in 512 byte units. Pretty dopey, but thats why its different.
Quoted 12 lines
sgi210a:~ #
xfs_info/mnt/
meta-data=/dev/mapper/mpath9 isize=256 agcount=32,
agsize=56652352 blks= sectsz=512 attr=0 data = bsize=4096 blocks=1812874752, imaxpct=25 = sunit=64 swidth=640 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=65536 blocks=0, rtextents=0
Quoted 1 line
sgi210a:~ #
$ gdb -q (gdb) p 512 * 512 $1 = 262144 (gdb) p 64 * 4096 $2 = 262144 (gdb)
(thats 262144 bytes, of course)
Quoted 3 lines
Last I tried to subscribe to the list by sending email to [email protected] a couple of times but was unsuccessful should I send email elsewhere to subscribe?
Its a frikkin' lottery. :) Keep trying and keep whining is how I ended up getting back on (whining on IRC on #xfs helps too).
cheers.
Signature · 1 line
Nathan
Thanks that helps. Now that I know I have the right sunit and swidth I have a performace related question.
If I do a dd on the raw device or to the lun directy I get speeds of around 190-200 MBytes/sec.
As soon as I add xfs on top of the lun my speeds go to around 150 MBytes/sec. This is for a single stream write using various block sizes on a 2 Gbit/sec fiber channel card.
Is this overhead more or less what you would expect from xfs? Or is there some tunning I need to do?
Thanks
Rene
Rene
-----Original Message-----
From: Nathan Scott <[email protected]>
Date: Tue, 12 Jun 2007 10:34:04
To:"Salmon, Rene" <[email protected]>
Cc:[email protected]
Subject: Re: sunit not working
Quoted 5 lines · Rene
As you can see the sunit gets set to 64 upon creation and not 512 like I asked. Also if it try to give it some mount options it does the same thing.
sgi210a:~ # mount -o sunit=512,swidth=5120
/dev/mapper/mpath9/mnt/
Its being reported in units of filesystem blocks, and its specified in 512 byte units. Pretty dopey, but thats why its different.
Quoted 12 lines
sgi210a:~ #
xfs_info/mnt/
meta-data=/dev/mapper/mpath9 isize=256 agcount=32,
agsize=56652352 blks= sectsz=512 attr=0 data = bsize=4096 blocks=1812874752, imaxpct=25 = sunit=64 swidth=640 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=65536 blocks=0, rtextents=0
Quoted 1 line
sgi210a:~ #
$ gdb -q (gdb) p 512 * 512 $1 = 262144 (gdb) p 64 * 4096 $2 = 262144 (gdb)
(thats 262144 bytes, of course)
Quoted 3 lines
Last I tried to subscribe to the list by sending email to [email protected] a couple of times but was unsuccessful should I send email elsewhere to subscribe?
Its a frikkin' lottery. :) Keep trying and keep whining is how I ended up getting back on (whining on IRC on #xfs helps too).
cheers.
Signature · 1 line
Nathan
Quoted 8 lines · On Tue, 2007-06-12 at 13:12 +0000, [email protected] wrote:
Thanks that helps. Now that I know I have the right sunit and swidth I have a performace related question.
Quoted 7 lines
If I do a dd on the raw device or to the lun directy I get speeds of around 190-200 MBytes/sec.
As soon as I add xfs on top of the lun my speeds go to around 150 MBytes/sec. This is for a single stream write using various block sizes on a 2 Gbit/sec fiber channel card.
Reads or writes?
What are your I/O sizes?
Buffered or direct IO?
Including fsync time in there or not? etc, etc.
(Actual dd commands used and their output results would be best) xfs_io is pretty good for this kind of analysis, as it gives very fine grained control of operations performed, has integrated bmap command, etc - use the -F flag for the raw device comparisons).
Quoted 2 lines
Is this overhead more or less what you would expect from xfs? Or is there some tunning I need to do?
You should be able to get very close to raw device speeds esp. for a single stream reader/writer, with some tuning.
cheers.
Hi,
More details on this:
Using dd with various block sizes to measure write performance only for now.
This is using two options to dd. The direct I/O option for direct i/o and the fsync option for buffered i/o.
Using direct:/usr/bin/time -p dd of=/mnt/testfile if=/dev/zero oflag=direct
Using fsync:/usr/bin/time -p dd of=/mnt/testfile if=/dev/zero conv=fsync
Using a 2Gbit/sec fiber channel card my theoretical max is 256 MBytes/sec. If we allow a bit of overhead for the card driver and things the manufacturer claims the card should be able to max out at around 200 MBytes/sec.
The block sizes I used range from 128KBytes - 1024000Kbytes and all the writes generate a 1.0GB file.
Some of the results I got:
Buffered I/O(fsync):
--------------------
Linux seems to do a good job at buffering this. Regardless of the block size I choose I always get write speeds of around 150MBytes/sec
Direct I/O(direct):
-------------------
The speeds I get here of course are very dependent on the block size I choose and how well they align with the stripe size of the storage array underneath. For the appropriate block sizes I get really good performance about 200MBytes/sec.
Quoted 1 line
From your feedback is sounds like these are reasonable numbers.
Most of our user apps do not use direct I/O but rather buffered I/O. Is 150MBytes/sec as good as it gets for buffered I/O or is there something I can tune to get a bit more out of buffered I/O?
Thanks
Rene
Quoted 30 lines
Quoted 11 lines
Thanks that helps. Now that I know I have the right sunit and swidth I have a performace related question.
If I do a dd on the raw device or to the lun directy I get speeds of around 190-200 MBytes/sec.
As soon as I add xfs on top of the lun my speeds go to around 150 MBytes/sec. This is for a single stream write using various block sizes on a 2 Gbit/sec fiber channel card.
Reads or writes?
What are your I/O sizes?
Buffered or direct IO?
Including fsync time in there or not? etc, etc.(Actual dd commands used and their output results would be best)
xfs_iois pretty good for this kind of analysis, as it gives very fine grained control of operations performed, has integrated bmap command, etc - use the -F flag for the raw device comparisons).Quoted 2 lines
Is this overhead more or less what you would expect from xfs? Or is there some tunning I need to do?
You should be able to get very close to raw device speeds esp. for a single stream reader/writer, with some tuning.
cheers.
Is 1 GB a reasonable filesize in your environment? And also most user apps don't use fsync, but maybe I missed somehting. Not knowing your storage vendor the numbers look pretty good to me, but the way you tested this is close to a benchmarking environment.
Cheers
Sebastian
Quoted 2 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Salmon, Rene
Quoted 11 lines
Sent: Mittwoch, 13. Juni 2007 20:46
To: [email protected]; David Chinner
Cc: [email protected]; [email protected]
Subject: Re: sunit not workingHi,
More details on this:
Using dd with various block sizes to measure write performance only
for
Quoted 11 lines
now.
This is using two options to dd. The direct I/O option for direct i/o and the fsync option for buffered i/o.
Using direct:
/usr/bin/time-p dd of=/mnt/testfile if=/dev/zero oflag=directUsing fsync:
/usr/bin/time-p dd of=/mnt/testfile if=/dev/zero conv=fsync
Quoted 6 lines
Using a 2Gbit/sec fiber channel card my theoretical max is 256 MBytes/sec. If we allow a bit of overhead for the card driver and things the manufacturer claims the card should be able to max out at around 200 MBytes/sec.
The block sizes I used range from 128KBytes - 1024000Kbytes and all
the
Quoted 7 lines
writes generate a 1.0GB file.
Some of the results I got:
Buffered I/O(fsync):
--------------------
Linux seems to do a good job at buffering this. Regardless of the
block
Quoted 6 lines
size I choose I always get write speeds of around 150MBytes/sec
Direct I/O(direct):
-------------------
The speeds I get here of course are very dependent on the block size I choose and how well they align with the stripe size of the storage
array
Quoted 6 lines
underneath. For the appropriate block sizes I get really good performance about 200MBytes/sec.
Quoted 1 line
From your feedback is sounds like these are reasonable numbers.
Most of our user apps do not use direct I/O but rather buffered I/O.
Is
Quoted 1 line
150MBytes/sec as good as it gets for buffered I/O or is there
something
Quoted 10 lines
I can tune to get a bit more out of buffered I/O?
Thanks
ReneQuoted 2 lines
Quoted 2 lines
Thanks that helps. Now that I know I have the right sunit and
swidth
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I have a performace related question.
If I do a dd on the raw device or to the lun directy I get speeds
of
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
around 190-200 MBytes/sec.
As soon as I add xfs on top of the lun my speeds go to around 150 MBytes/sec. This is for a single stream write using various block sizes on a 2 Gbit/sec fiber channel card.
Reads or writes?
What are your I/O sizes?
Buffered or direct IO?
Including fsync time in there or not? etc, etc.
Quoted 6 lines
Quoted 6 lines
(Actual dd commands used and their output results would be best)
xfs_iois pretty good for this kind of analysis, as it gives very fine grained control of operations performed, has integrated bmap command, etc - use the -F flag for the raw device comparisons).Quoted 1 line
Is this overhead more or less what you would expect from xfs? Or
is
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
there some tunning I need to do?
You should be able to get very close to raw device speeds esp. for a single stream reader/writer, with some tuning.
cheers.
Quoted 14 lines · Rene
Hi,
More details on this:
Using dd with various block sizes to measure write performance only for now.
This is using two options to dd. The direct I/O option for direct i/o and the fsync option for buffered i/o.
Using direct:
/usr/bin/time-p dd of=/mnt/testfile if=/dev/zero oflag=direct
Quoted 7 lines
Using fsync:
/usr/bin/time-p dd of=/mnt/testfile if=/dev/zero conv=fsyncUsing a 2Gbit/sec fiber channel card my theoretical max is 256 MBytes/sec. If we allow a bit of overhead for the card driver and things the manufacturer claims the card should be able to max out at around 200 MBytes/sec.
Right.
Quoted 9 lines
The block sizes I used range from 128KBytes - 1024000Kbytes and all the writes generate a 1.0GB file.
Some of the results I got:
Buffered I/O(fsync):
--------------------
Linux seems to do a good job at buffering this. Regardless of the block size I choose I always get write speeds of around 150MBytes/sec
Because it does single threaded writeback via pdflush. It should always get the same throughput.
If you wind /proc/sys/vm/dirty_ratio down to 5, it might go a bit faster because writeback will start earlier in the write and so the fsync will have less to do and overall speed will appear faster.
What you should be looking at it iostat throughput in the steady state, not inferring the throughput from timing a write operation.....
Quoted 6 lines
Direct I/O(direct):
-------------------
The speeds I get here of course are very dependent on the block size I choose and how well they align with the stripe size of the storage array underneath. For the appropriate block sizes I get really good performance about 200MBytes/sec.
Also normal, because you're iop bound at small block sizes. At large block sizes, you saturate the fibre. Sounds like nothing is wrong here.
Quoted 4 lines
Quoted 1 line
From your feedback is sounds like these are reasonable numbers.
Most of our user apps do not use direct I/O but rather buffered I/O. Is 150MBytes/sec as good as it gets for buffered I/O or is there something I can tune to get a bit more out of buffered I/O?
That's about it, I think. With tweaking any tuning of the vm parameters, you might be able to get it higher, but it may be that writeback (when it occurs) is actually higher than this....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 10 lines · On Tue, Jun 12, 2007 at 01:12:15PM +0000, [email protected] wrote:
Thanks that helps. Now that I know I have the right sunit and swidth I have a performace related question.
If I do a dd on the raw device or to the lun directy I get speeds of around 190-200 MBytes/sec.
As soon as I add xfs on top of the lun my speeds go to around 150 MBytes/sec. This is for a single stream write using various block sizes on a 2 Gbit/sec fiber channel card.
That's for buffered I/O, right? That sounds about right - if you do two writes, it should increase a little further. Also, direct I/O should be able to get you to >90% of the raw device capability....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group