Poor read performance on high-end server
Hi,
We're interested in getting the highest possible read performance on a server. To that end, we have a high-end server with multiple solid state disks (SSDs). Since BtrFS outperformed other Linux filesystem, we choose that. Unfortunately, there seems to be an upper boundary in the performance of BtrFS of roughly 1 GiByte/s read speed. Compare the following results with either BTRFS on Ubuntu versus ZFS on FreeBSD:
ZFS BtrFS
1 SSD 256 MiByte/s 256 MiByte/s 2 SSDs 505 MiByte/s 504 MiByte/s 3 SSDs 736 MiByte/s 756 MiByte/s 4 SSDs 952 MiByte/s 916 MiByte/s 5 SSDs 1226 MiByte/s 986 MiByte/s 6 SSDs 1450 MiByte/s 978 MiByte/s 8 SSDs 1653 MiByte/s 932 MiByte/s 16 SSDs 2750 MiByte/s 919 MiByte/s
The results were originally measured on a Dell PowerEdge T610, but were repeated using a SuperMicro machine with 4 independent SAS+SATA controllers. We made sure that the PCI-e slots where not the bottleneck. The above results were for Ubuntu 10.04.1 server, with BtrFS v0.19, although earlier tests with Ubuntu 9.10 showed the same results.
Apparently, the limitation is not in the hardware (the same hardware with ZFS did scale near linear). We also tested both hardware RAID-0, software RAID-0 (md), and using the btrfs built-in software RAID-0, but the differences were small (<10%) (md-based software RAID was marginally slower on Linux; RAIDZ was marginally faster on FreeBSD). So we presume that the bottleneck is somewhere in the BtrFS (or kernel) software.
Are there suggestions how to tune the read performance? We like to scale this up to 32 solid state disks. The -o ssd option did not improve overall performance, although it did gave more stable results (less fluctuation in repeated tests).
Note that the write speeds did scale fine. In the scenario with 16 solid state disks, the write speed is 1596 MiByte/s (1.7 times as fast as the read speed! Suffice to say that for a single disk, write is much slower than read...).
Here are the exact settings:
~# mkfs.btrfs -d raid0 /dev/sdd /dev/sde /dev/sdf /dev/sdg \
/dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm \
/dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds
nodesize 4096 leafsize 4096 sectorsize 4096 size 2.33TB Btrfs Btrfs v0.19
~# mount -t btrfs -o ssd /dev/sdd /mnt/ssd6
~# iozone -s 32G -r 1024 -i 0 -i 1 -w -f /mnt/ssd6/iozone.tmp
KB reclen write rewrite read reread
33554432 1024 1628475 1640349 943416 951135
Regards,
Freek Dijkstra
SARA high performance- and computing
Quoted 8 lines · Freek Dijkstra
Hi,
We're interested in getting the highest possible read performance on a server. To that end, we have a high-end server with multiple solid state disks (SSDs). Since BtrFS outperformed other Linux filesystem, we choose that. Unfortunately, there seems to be an upper boundary in the performance of BtrFS of roughly 1 GiByte/s read speed. Compare the following results with either BTRFS on Ubuntu versus ZFS on FreeBSD:
Really cool, thanks for posting this.
Quoted 11 lines
ZFS BtrFS
1 SSD 256 MiByte/s 256 MiByte/s 2 SSDs 505 MiByte/s 504 MiByte/s 3 SSDs 736 MiByte/s 756 MiByte/s 4 SSDs 952 MiByte/s 916 MiByte/s 5 SSDs 1226 MiByte/s 986 MiByte/s 6 SSDs 1450 MiByte/s 978 MiByte/s 8 SSDs 1653 MiByte/s 932 MiByte/s 16 SSDs 2750 MiByte/s 919 MiByte/s
Quoted 5 lines
The results were originally measured on a Dell PowerEdge T610, but were repeated using a SuperMicro machine with 4 independent SAS+SATA controllers. We made sure that the PCI-e slots where not the bottleneck. The above results were for Ubuntu 10.04.1 server, with BtrFS v0.19, although earlier tests with Ubuntu 9.10 showed the same results.
Which kernels are those?
Basically we have two different things to tune. First the block layer and then btrfs.
Can I ask you to do a few experiments? First grab fio:
http://brick.kernel.dk/snaps/fio-git-latest.tar.gz
And then we need to setup a fio job file that hammers on all the ssds at once. I'd have it use adio/dio and talk directly to the drives. I'd do something like this for the fio job file, but Jens Axboe is cc'd and he might make another suggestion on the job file. I'd do something like this in a file named ssd.fio
[global]
size=32g
direct=1
iodepth=8
bs=20m
rw=read
[f1]
filename=/dev/sdd
[f2]
filename=/dev/sde
[f3]
filename=/dev/sdf
repeat for all the drives, then run fio ssd.fio
fio should be able to push these devices up to the line speed. If it doesn't I would suggest changing elevators (deadline, cfq, noop) and bumping the max request size to the max supported by the device.
When we have a config that does so, we can tune the btrfs side of things as well. The btrfs job file would look something like this:
[global]
size=32g
direct=1
iodepth=8
bs=20m
rw=read
[btrfs]
directory=/btrfs_mount_point
# experiment with numjobs numjobs=16
My first guess is just that your IOs are not large enough w/btrfs. The iozone command below is doing buffered reads, so our performance is going to be limited by the kernel readahead buffer size.
If you use a much larger IO size (the fio job above reads in 20M chunks) and aio/dio instead, you can have more control over how the IO goes down to the device.
-chris
Chris, Daniel and Mathieu,
Thanks for your constructive feedback!
Quoted 11 lines · Freek Dijkstra
Quoted 10 lines
ZFS BtrFS
1 SSD 256 MiByte/s 256 MiByte/s 2 SSDs 505 MiByte/s 504 MiByte/s 3 SSDs 736 MiByte/s 756 MiByte/s 4 SSDs 952 MiByte/s 916 MiByte/s 5 SSDs 1226 MiByte/s 986 MiByte/s 6 SSDs 1450 MiByte/s 978 MiByte/s 8 SSDs 1653 MiByte/s 932 MiByte/s 16 SSDs 2750 MiByte/s 919 MiByte/s
[...]
Quoted 3 lines
Quoted 1 line
The above results were for Ubuntu 10.04.1 server, with BtrFS v0.19,
=20
Which kernels are those?
=46or BtrFS: Linux 2.6.32-21-server #32-Ubuntu SMP x86_64 GNU/Linux =46or ZFS: FreeBSD 8.1-RELEASE (GENERIC)
(Note that we can currently not upgrade easily due to binary drivers fo= r
the SAS+SATA controllers :(. I'd be happy to push the vendor though, if you think it makes a difference.)
Quoted 1 line · Daniel J Blueman
Perhaps create a new filesystem and mount with 'nodatasum'
I get an improvement: 919 MiByte/s just became 1580 MiByte/s. Not as fast as it can, but most certainly an improvement.
Quoted 2 lines
existing extents which were previously created will be checked, so need to start fresh.
Indeed, also the other way around. I created two test files, while mounted with and without the -o nodatasum option:
write w/o nodatasum; read w/o nodatasum: 919 =B1 43 MiByte/s write w/o nodatasum; read w/ nodatasum: 922 =B1 72 MiByte/s write w/ nodatasum; read w/o nodatasum: 1082 =B1 46 MiByte/s write w/ nodatasum; read w/ nodatasum: 1586 =B1 126 MiByte/s
So even if I remount the disk in the normal way, and read a file create= d
without checksums, I still get a small improvement :)
(PS: the above tests were repeated 4 times, the last even 8 times. As you can see from the standard deviation, the results are not always ver= y
accurate. The cause is unknown; CPU load is low.)
Quoted 1 line · Chris Mason
Basically we have two different things to tune. First the block laye=
r
Quoted 4 lines
and then btrfs.
And then we need to setup a fio job file that hammers on all the ssds=
at
Quoted 15 lines
once. I'd have it use adio/dio and talk directly to the drives. =20
[global]
size=3D32g
direct=3D1
iodepth=3D8
bs=3D20m
rw=3Dread
=20
[f1]
filename=3D/dev/sdd
[f2]
filename=3D/dev/sde
[f3]
filename=3D/dev/sdf
[...]
Quoted 2 lines
[f16]
filename=3D/dev/sds
Thanks. First one disk:
Quoted 3 lines
f1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D6273
read : io=3D32780MB, bw=3D260964KB/s, iops=3D12, runt=3D128626msecclat (usec): min=3D74940, max=3D80721, avg=3D78449.61, stdev=3D92=
3.24
Quoted 1 line
bw (KB/s) : min=3D240469, max=3D269981, per=3D100.10%, avg=3D2612=
14.77, stdev=3D2765.91
Quoted 1 line
cpu : usr=3D0.01%, sys=3D2.69%, ctx=3D1747, majf=3D0, minf=
=3D5153
Quoted 1 line
IO depths : 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16=3D0.0%,=
32=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%=
, 64=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%=
, 64=3D0.0%, >=3D64=3D0.0%
Quoted 6 lines
issued r/w: total=3D1639/0, short=3D0/0
=20
lat (msec): 100=3D100.00%
=20
Run status group 0 (all jobs):
READ: io=3D32780MB, aggrb=3D260963KB/s, minb=3D267226KB/s, maxb=3D=267226KB/s, mint=3D128626msec, maxt=3D128626msec
Quoted 3 lines
=20
Disk stats (read/write):
sdd: ios=3D261901/0, merge=3D0/0, ticks=3D10135270/0, in_queue=3D10=
136460, util=3D99.30%
So 255 MiByte/s.
Out of curiousity, what is the distinction between the reported figures of 260964 kiB/s, 261214.77 kiB/s, 267226 kiB/s and 260963 kiB/s?
Now 16 disks (abbreviated):
Quoted 7 lines
~/fio# ./fio ssd.fio
Starting 16 processes
f1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D4756
read : io=3D32780MB, bw=3D212987KB/s, iops=3D10, runt=3D157600msecclat (msec): min=3D75, max=3D138, avg=3D96.15, stdev=3D 4.47 lat (msec): min=3D75, max=3D138, avg=3D96.15, stdev=3D 4.47 bw (KB/s) : min=3D153121, max=3D268968, per=3D6.31%, avg=3D213181=
=2E15, stdev=3D9052.26
Quoted 1 line
cpu : usr=3D0.00%, sys=3D1.71%, ctx=3D2737, majf=3D0, minf=
=3D5153
Quoted 1 line
IO depths : 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16=3D0.0%,=
32=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%=
, 64=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%=
, 64=3D0.0%, >=3D64=3D0.0%
Quoted 3 lines
issued r/w: total=3D1639/0, short=3D0/0
=20
lat (msec): 100=3D97.99%, 250=3D2.01%[..similar for f2 to f16..]
Quoted 1 line
f1: read : io=3D32780MB, bw=3D212987KB/s, iops=3D10, runt=3D1576=
00msec
Quoted 1 line
bw (KB/s) : min=3D153121, max=3D268968, per=3D6.31%, avg=3D213181=
=2E15, stdev=3D9052.26
Quoted 1 line
f2: read : io=3D32780MB, bw=3D213873KB/s, iops=3D10, runt=3D1569=
47msec
Quoted 1 line
bw (KB/s) : min=3D151143, max=3D251508, per=3D6.33%, avg=3D213987=
=2E34, stdev=3D8958.86
Quoted 1 line
f3: read : io=3D32780MB, bw=3D214613KB/s, iops=3D10, runt=3D1564=
06msec
Quoted 1 line
bw (KB/s) : min=3D149216, max=3D219037, per=3D6.35%, avg=3D214779=
=2E89, stdev=3D9332.99
Quoted 1 line
f4: read : io=3D32780MB, bw=3D214388KB/s, iops=3D10, runt=3D1565=
70msec
Quoted 1 line
bw (KB/s) : min=3D148675, max=3D226298, per=3D6.35%, avg=3D214576=
=2E51, stdev=3D8985.03
Quoted 1 line
f5: read : io=3D32780MB, bw=3D213848KB/s, iops=3D10, runt=3D1569=
65msec
Quoted 1 line
bw (KB/s) : min=3D144479, max=3D241414, per=3D6.33%, avg=3D213935=
=2E81, stdev=3D10023.68
Quoted 1 line
f6: read : io=3D32780MB, bw=3D213514KB/s, iops=3D10, runt=3D1572=
11msec
Quoted 1 line
bw (KB/s) : min=3D141730, max=3D264990, per=3D6.32%, avg=3D213656=
=2E75, stdev=3D10871.71
Quoted 1 line
f7: read : io=3D32780MB, bw=3D213431KB/s, iops=3D10, runt=3D1572=
72msec
Quoted 1 line
bw (KB/s) : min=3D148137, max=3D254635, per=3D6.32%, avg=3D213493=
=2E12, stdev=3D9319.08
Quoted 1 line
f8: read : io=3D32780MB, bw=3D213099KB/s, iops=3D10, runt=3D1575=
17msec
Quoted 1 line
bw (KB/s) : min=3D143467, max=3D267962, per=3D6.31%, avg=3D213267=
=2E60, stdev=3D11224.35
Quoted 1 line
f9: read : io=3D32780MB, bw=3D211254KB/s, iops=3D10, runt=3D1588=
93msec
Quoted 1 line
bw (KB/s) : min=3D149489, max=3D267962, per=3D6.25%, avg=3D211257=
=2E05, stdev=3D9370.64
Quoted 1 line
f10: read : io=3D32780MB, bw=3D212251KB/s, iops=3D10, runt=3D1581=
46msec
Quoted 1 line
bw (KB/s) : min=3D150865, max=3D225882, per=3D6.28%, avg=3D212300=
=2E50, stdev=3D8431.06
Quoted 1 line
f11: read : io=3D32780MB, bw=3D212988KB/s, iops=3D10, runt=3D1575=
99msec
Quoted 1 line
bw (KB/s) : min=3D149489, max=3D221007, per=3D6.31%, avg=3D213123=
=2E72, stdev=3D9569.27
Quoted 1 line
f12: read : io=3D32780MB, bw=3D212788KB/s, iops=3D10, runt=3D1577=
47msec
Quoted 1 line
bw (KB/s) : min=3D154274, max=3D218647, per=3D6.30%, avg=3D212957=
=2E41, stdev=3D8233.52
Quoted 1 line
f13: read : io=3D32780MB, bw=3D212315KB/s, iops=3D10, runt=3D1580=
99msec
Quoted 1 line
bw (KB/s) : min=3D153696, max=3D256000, per=3D6.29%, avg=3D212482=
=2E68, stdev=3D9203.34
Quoted 1 line
f14: read : io=3D32780MB, bw=3D212033KB/s, iops=3D10, runt=3D1583=
09msec
Quoted 1 line
bw (KB/s) : min=3D150588, max=3D267962, per=3D6.28%, avg=3D212198=
=2E76, stdev=3D9572.31
Quoted 1 line
f15: read : io=3D32780MB, bw=3D211720KB/s, iops=3D10, runt=3D1585=
43msec
Quoted 1 line
bw (KB/s) : min=3D146024, max=3D268968, per=3D6.27%, avg=3D211846=
=2E40, stdev=3D10341.58
Quoted 1 line
f16: read : io=3D32780MB, bw=3D211637KB/s, iops=3D10, runt=3D1586=
05msec
Quoted 1 line
bw (KB/s) : min=3D148945, max=3D261605, per=3D6.26%, avg=3D211618=
=2E40, stdev=3D9240.64
Quoted 3 lines
=20
Run status group 0 (all jobs):READ: io=3D524480MB, aggrb=3D3301MB/s, minb=3D216323KB/s, maxb=3D2=
19763KB/s, mint=3D156406msec, maxt=3D158893msec
Quoted 3 lines
=20
Disk stats (read/write):
sdd: ios=3D261902/0, merge=3D0/0, ticks=3D12531810/0, in_queue=3D12=
532910, util=3D99.46%
Quoted 1 line
sde: ios=3D262221/0, merge=3D0/0, ticks=3D12494200/0, in_queue=3D12=
495300, util=3D99.50%
Quoted 1 line
sdf: ios=3D261867/0, merge=3D0/0, ticks=3D12427000/0, in_queue=3D12=
430530, util=3D99.47%
Quoted 1 line
sdg: ios=3D261983/0, merge=3D0/0, ticks=3D12462320/0, in_queue=3D12=
466060, util=3D99.62%
Quoted 1 line
sdh: ios=3D262184/0, merge=3D0/0, ticks=3D12487350/0, in_queue=3D12=
489960, util=3D99.49%
Quoted 1 line
sdi: ios=3D262193/0, merge=3D0/0, ticks=3D12524400/0, in_queue=3D12=
526580, util=3D99.47%
Quoted 1 line
sdj: ios=3D262044/0, merge=3D0/0, ticks=3D12511850/0, in_queue=3D12=
513840, util=3D99.50%
Quoted 1 line
sdk: ios=3D262055/0, merge=3D0/0, ticks=3D12526560/0, in_queue=3D12=
527890, util=3D99.50%
Quoted 1 line
sdl: ios=3D261789/0, merge=3D0/0, ticks=3D12609230/0, in_queue=3D12=
610400, util=3D99.54%
Quoted 1 line
sdm: ios=3D261787/0, merge=3D0/0, ticks=3D12579000/0, in_queue=3D12=
581050, util=3D99.44%
Quoted 1 line
sdn: ios=3D261941/0, merge=3D0/0, ticks=3D12524530/0, in_queue=3D12=
525790, util=3D99.48%
Quoted 1 line
sdo: ios=3D262100/0, merge=3D0/0, ticks=3D12554650/0, in_queue=3D12=
555820, util=3D99.58%
Quoted 1 line
sdp: ios=3D261877/0, merge=3D0/0, ticks=3D12572220/0, in_queue=3D12=
574610, util=3D99.54%
Quoted 1 line
sdq: ios=3D261956/0, merge=3D0/0, ticks=3D12601480/0, in_queue=3D12=
603770, util=3D99.62%
Quoted 1 line
sdr: ios=3D261991/0, merge=3D0/0, ticks=3D12599680/0, in_queue=3D12=
602190, util=3D99.49%
Quoted 1 line
sds: ios=3D261852/0, merge=3D0/0, ticks=3D12624070/0, in_queue=3D12=
626580, util=3D99.58%
So, the maximum for these 16 disks is 3301 MiByte/s.
I also tried hardware RAID (2 sets of 8 disks), and got a similar resul= t:
Quoted 2 lines
Run status group 0 (all jobs):
READ: io=3D65560MB, aggrb=3D3024MB/s, minb=3D1548MB/s, maxb=3D1550=
MB/s, mint=3D21650msec, maxt=3D21681msec
Quoted 3 lines
fio should be able to push these devices up to the line speed. If it doesn't I would suggest changing elevators (deadline, cfq, noop) and bumping the max request size to the max supported by the device.
3301 MiByte/s seems like a reasonable number, given the theoretic maximum of 16 times the single disk performance of 16*256 MiByte/s =3D 4096 MiByte/s.
Based on this, I have not looked at tuning. Would you recommend that I = do?
Our minimal goal is 2500 MiByte/s; that seems achievable as ZFS was abl= e
to reach 2750 MiByte/s without tuning.
Quoted 1 line
When we have a config that does so, we can tune the btrfs side of thi=
ngs
Quoted 1 line
as well.
Some files are created in the root folder of the mount point, but I get errors instead of results:
Quoted 2 lines
~/fio# ./fio btrfs16.fio=20
btrfs: (g=3D0): rw=3Dread, bs=3D20M-20M/20M-20M, ioengine=3Dsync, iod=
epth=3D8
Quoted 3 lines
Starting 16 processes
btrfs: Laying out IO file(s) (1 file(s) / 32768MB)
btrfs: Laying out IO file(s) (1 file(s) / 32768MB)
[...]
Quoted 2 lines
btrfs: Laying out IO file(s) (1 file(s) / 32768MB)
fio: first direct IO errored. File system may not support direct IO, =
or iomem_align=3D is bad.
Quoted 1 line
fio: first direct IO errored. File system may not support direct IO, =
or iomem_align=3D is bad.
Quoted 1 line
fio: first direct IO errored. File system may not support direct IO, =
or iomem_align=3D is bad.
Quoted 1 line
fio: pid=3D5958, err=3D22/file:engines/sync.c:62, func=3Dxfer, error=3D=
Invalid argument
Quoted 1 line
fio: pid=3D5961, err=3D22/file:engines/sync.c:62, func=3Dxfer, error=3D=
Invalid argument
Quoted 1 line
fio: pid=3D5962, err=3D22/file:engines/sync.c:62, func=3Dxfer, error=3D=
Invalid argument
Quoted 1 line
fio: first direct IO errored. File system may not support direct IO, =
or iomem_align=3D is bad.
[...]
Quoted 2 lines
=20
btrfs: (groupid=3D0, jobs=3D1): err=3D22 (file:engines/sync.c:62, fun=
c=3Dxfer, error=3DInvalid argument): pid=3D5956
Quoted 1 line
cpu : usr=3D0.00%, sys=3D0.00%, ctx=3D1, majf=3D0, minf=3D=
52
Quoted 1 line
IO depths : 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16=3D0.0%,=
32=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%=
, 64=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
complete : 0=3D50.0%, 4=3D50.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%=
, 64=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
issued r/w: total=3D1/0, short=3D0/0
[no results]
What could be going on here?
(I get the same result from the github version of fio, fio 1.42, as wel= l
as the one that came with Ubuntu, fio 1.33.1).
Quoted 1 line
My first guess is just that your IOs are not large enough w/btrfs. T=
he
Quoted 4 lines
iozone command below is doing buffered reads, so our performance is going to be limited by the kernel readahead buffer size. =20
If you use a much larger IO size (the fio job above reads in 20M chun=
ks)
Quoted 1 line
and aio/dio instead, you can have more control over how the IO goes d=
own
Quoted 1 line
to the device.
I don't quite understand (I must warn you that I'm a novice here; I'm a networking expert by origin, not a storage expert).
I reran the first fio test with other "bs" settings:
Quoted 2 lines
1 disk, 1M buffer:
READ: io=3D32768MB, aggrb=3D247817KB/s, minb=3D253764KB/s, maxb=3D=
253764KB/s, mint=3D135400msec, maxt=3D135400msec
Quoted 3 lines
=20
1 disk, 20M buffer:READ: io=3D32780MB, aggrb=3D260963KB/s, minb=3D267226KB/s, maxb=3D=
267226KB/s, mint=3D128626msec, maxt=3D128626msec
Quoted 3 lines
=20
1 disk, 100M buffer:READ: io=3D32800MB, aggrb=3D263776KB/s, minb=3D270107KB/s, maxb=3D=
270107KB/s, mint=3D127332msec, maxt=3D127332msec
Quoted 3 lines
=20
16 disk, 1M buffer:READ: io=3D524288MB, aggrb=3D3265MB/s, minb=3D213983KB/s, maxb=3D2=
15761KB/s, mint=3D159249msec, maxt=3D160572msec
Quoted 3 lines
=20
16 disk, 20M buffer:READ: io=3D524480MB, aggrb=3D3301MB/s, minb=3D216323KB/s, maxb=3D2=
19763KB/s, mint=3D156406msec, maxt=3D158893msec
Quoted 3 lines
=20
16 disk, 100M buffer:READ: io=3D524800MB, aggrb=3D3272MB/s, minb=3D214443KB/s, maxb=3D2=
16446KB/s, mint=3D158900msec, maxt=3D160384msec
However, the buffer size does not seem to make that much of a difference. Or am I adjusting the wrong buffers here?
Quoted 2 lines · Mathieu Chouquet-Stringer
Don't you need to stripe metadata too (with -m raid0)? Or you may be limited by your metadata drive?
I presume that if this were the case, we would see good performance for hardware RAID and mdadm based software RAID, and poor performance for BtrFS. However, we saw poor performance for all three options.
Of course, seeing is believing.
Without metadata striping:
# mkfs.btrfs -d raid0 -m raid0 /dev/sdd ... /dev/sds
# mount -t btrfs -o ssd /dev/sdd /mnt/ssd6
# iozone -s 32G -r 1024 -i 0 -i 1 -w -f /mnt/ssd6/iozone.tmp
KB reclen write rewrite read reread
33554432 1024 1628475 1640349 943416 951135With metadata striping:
# mkfs.btrfs -d raid0 /dev/sdd ... /dev/sds
# mount -t btrfs -o ssd /dev/sdd /mnt/ssd6
# iozone -s 32G -r 1024 -i 0 -i 1 -w -f /mnt/ssd6/iozone.tmp
KB reclen write rewrite read reread
33554432 1024 1631833 1564137 950405 954434Unfortunately, no noticeable difference.
With kind regards,
=46reek Dijkstra
SARA High Performance Networking- and Computing
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Freek Dijkstra
Chris, Daniel and Mathieu,
Thanks for your constructive feedback!
Quoted 11 lines · Freek Dijkstra
Quoted 10 lines
=A0 =A0 =A0 =A0 =A0 =A0 =A0ZFS =A0 =A0 =A0 =A0 =A0 =A0 BtrFS 1 SSD =A0 =A0 =A0256 MiByte/s =A0 =A0 256 MiByte/s 2 SSDs =A0 =A0 505 MiByte/s =A0 =A0 504 MiByte/s
3 SSDs =A0 =A0 736 MiByte/s =A0 =A0 756 MiByte/s
4 SSDs =A0 =A0 952 MiByte/s =A0 =A0 916 MiByte/s
5 SSDs =A0 =A01226 MiByte/s =A0 =A0 986 MiByte/s
6 SSDs =A0 =A01450 MiByte/s =A0 =A0 978 MiByte/s
8 SSDs =A0 =A01653 MiByte/s =A0 =A0 932 MiByte/s 16 SSDs =A0 2750 MiByte/s =A0 =A0 919 MiByte/s[...]
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
The above results were for Ubuntu 10.04.1 server, with BtrFS v0.19,
Which kernels are those?
For BtrFS: Linux 2.6.32-21-server #32-Ubuntu SMP
x86_64GNU/Linux For ZFS: FreeBSD 8.1-RELEASE (GENERIC)(Note that we can currently not upgrade easily due to binary drivers =
for
Quoted 1 line
the SAS+SATA controllers :(. I'd be happy to push the vendor though, =
if
Quoted 13 lines
you think it makes a difference.)
Quoted 1 line · Daniel J Blueman
Perhaps create a new filesystem and mount with 'nodatasum'
I get an improvement: 919 MiByte/s just became 1580 MiByte/s. Not as fast as it can, but most certainly an improvement.
Quoted 2 lines
existing extents which were previously created will be checked, so need to start fresh.
Quoted 8 lines
Indeed, also the other way around. I created two test files, while mounted with and without the -o nodatasum option:
write w/o nodatasum; read w/o nodatasum: =A0919 =B1 43 MiByte/s write w/o nodatasum; read w/ =A0nodatasum: =A0922 =B1 72 MiByte/s write w/ =A0nodatasum; read w/o nodatasum: 1082 =B1 46 MiByte/s write w/ =A0nodatasum; read w/ =A0nodatasum: 1586 =B1 126 MiByte/sSo even if I remount the disk in the normal way, and read a file crea=
ted
Quoted 4 lines
without checksums, I still get a small improvement :)
(PS: the above tests were repeated 4 times, the last even 8 times. As you can see from the standard deviation, the results are not always v=
ery
Quoted 6 lines
accurate. The cause is unknown; CPU load is low.)
Quoted 1 line · Chris Mason
Basically we have two different things to tune. =A0First the block l=
ayer
Quoted 4 lines
Quoted 4 lines
and then btrfs.
And then we need to setup a fio job file that hammers on all the ssd=
s at
Quoted 15 lines
Quoted 9 lines
once. =A0I'd have it use adio/dio and talk directly to the drives.
[global]
size=3D32g
direct=3D1
iodepth=3D8
bs=3D20m
rw=3DreadQuoted 6 lines
[f1]
filename=3D/dev/sdd
[f2]
filename=3D/dev/sde
[f3]
filename=3D/dev/sdf
Quoted 8 lines
[...]
Quoted 2 lines
[f16]
filename=3D/dev/sdsThanks. First one disk:
Quoted 2 lines
f1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D6273
=A0 read : io=3D32780MB, bw=3D260964KB/s, iops=3D12, runt=3D128626ms=
ec
Quoted 1 line
Quoted 1 line
=A0 =A0 clat (usec): min=3D74940, max=3D80721, avg=3D78449.61, stdev=
=3D923.24
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D240469, max=3D269981, per=3D100.10%, avg=3D=
261214.77, stdev=3D2765.91
Quoted 1 line
Quoted 1 line
=A0 cpu =A0 =A0 =A0 =A0 =A0: usr=3D0.01%, sys=3D2.69%, ctx=3D1747, m=
ajf=3D0, minf=3D5153
Quoted 1 line
Quoted 1 line
=A0 IO depths =A0 =A0: 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16=3D=
0.0%, 32=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
Quoted 1 line
=A0 =A0 =A0submit =A0 =A0: 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%=
, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
Quoted 1 line
=A0 =A0 =A0complete =A0: 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, =
32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0%
Quoted 6 lines
Quoted 6 lines
=A0 =A0 =A0issued r/w: total=3D1639/0, short=3D0/0
=A0 =A0 =A0lat (msec): 100=3D100.00%
Run status group 0 (all jobs):
=A0 =A0READ: io=3D32780MB, aggrb=3D260963KB/s, minb=3D267226KB/s, ma=
xb=3D267226KB/s, mint=3D128626msec, maxt=3D128626msec
Quoted 3 lines
Quoted 3 lines
Disk stats (read/write):
=A0 sdd: ios=3D261901/0, merge=3D0/0, ticks=3D10135270/0, in_queue=3D=
10136460, util=3D99.30%
Quoted 3 lines
So 255 MiByte/s.
Out of curiousity, what is the distinction between the reported figur=
es
Quoted 9 lines
of 260964 kiB/s, 261214.77 kiB/s, 267226 kiB/s and 260963 kiB/s?
Now 16 disks (abbreviated):
Quoted 4 lines
~/fio# ./fio ssd.fio
Starting 16 processes
f1: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D4756
=A0 read : io=3D32780MB, bw=3D212987KB/s, iops=3D10, runt=3D157600ms=
ec
Quoted 2 lines
Quoted 2 lines
=A0 =A0 clat (msec): min=3D75, max=3D138, avg=3D96.15, stdev=3D 4.47 =A0 =A0 =A0lat (msec): min=3D75, max=3D138, avg=3D96.15, stdev=3D 4.=
47
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D153121, max=3D268968, per=3D6.31%, avg=3D2=
13181.15, stdev=3D9052.26
Quoted 1 line
Quoted 1 line
=A0 cpu =A0 =A0 =A0 =A0 =A0: usr=3D0.00%, sys=3D1.71%, ctx=3D2737, m=
ajf=3D0, minf=3D5153
Quoted 1 line
Quoted 1 line
=A0 IO depths =A0 =A0: 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16=3D=
0.0%, 32=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
Quoted 1 line
=A0 =A0 =A0submit =A0 =A0: 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%=
, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
Quoted 1 line
=A0 =A0 =A0complete =A0: 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, =
32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0%
Quoted 7 lines
Quoted 3 lines
=A0 =A0 =A0issued r/w: total=3D1639/0, short=3D0/0
=A0 =A0 =A0lat (msec): 100=3D97.99%, 250=3D2.01%
[..similar for f2 to f16..]
Quoted 1 line
f1: =A0 =A0 =A0read : io=3D32780MB, bw=3D212987KB/s, iops=3D10, runt=
=3D157600msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D153121, max=3D268968, per=3D6.31%, avg=3D2=
13181.15, stdev=3D9052.26
Quoted 1 line
Quoted 1 line
f2: =A0 =A0 =A0read : io=3D32780MB, bw=3D213873KB/s, iops=3D10, runt=
=3D156947msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D151143, max=3D251508, per=3D6.33%, avg=3D2=
13987.34, stdev=3D8958.86
Quoted 1 line
Quoted 1 line
f3: =A0 =A0 =A0read : io=3D32780MB, bw=3D214613KB/s, iops=3D10, runt=
=3D156406msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D149216, max=3D219037, per=3D6.35%, avg=3D2=
14779.89, stdev=3D9332.99
Quoted 1 line
Quoted 1 line
f4: =A0 =A0 =A0read : io=3D32780MB, bw=3D214388KB/s, iops=3D10, runt=
=3D156570msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D148675, max=3D226298, per=3D6.35%, avg=3D2=
14576.51, stdev=3D8985.03
Quoted 1 line
Quoted 1 line
f5: =A0 =A0 =A0read : io=3D32780MB, bw=3D213848KB/s, iops=3D10, runt=
=3D156965msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D144479, max=3D241414, per=3D6.33%, avg=3D2=
13935.81, stdev=3D10023.68
Quoted 1 line
Quoted 1 line
f6: =A0 =A0 =A0read : io=3D32780MB, bw=3D213514KB/s, iops=3D10, runt=
=3D157211msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D141730, max=3D264990, per=3D6.32%, avg=3D2=
13656.75, stdev=3D10871.71
Quoted 1 line
Quoted 1 line
f7: =A0 =A0 =A0read : io=3D32780MB, bw=3D213431KB/s, iops=3D10, runt=
=3D157272msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D148137, max=3D254635, per=3D6.32%, avg=3D2=
13493.12, stdev=3D9319.08
Quoted 1 line
Quoted 1 line
f8: =A0 =A0 =A0read : io=3D32780MB, bw=3D213099KB/s, iops=3D10, runt=
=3D157517msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D143467, max=3D267962, per=3D6.31%, avg=3D2=
13267.60, stdev=3D11224.35
Quoted 1 line
Quoted 1 line
f9: =A0 =A0 =A0read : io=3D32780MB, bw=3D211254KB/s, iops=3D10, runt=
=3D158893msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D149489, max=3D267962, per=3D6.25%, avg=3D2=
11257.05, stdev=3D9370.64
Quoted 1 line
Quoted 1 line
f10: =A0 =A0 read : io=3D32780MB, bw=3D212251KB/s, iops=3D10, runt=3D=
158146msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D150865, max=3D225882, per=3D6.28%, avg=3D2=
12300.50, stdev=3D8431.06
Quoted 1 line
Quoted 1 line
f11: =A0 =A0 read : io=3D32780MB, bw=3D212988KB/s, iops=3D10, runt=3D=
157599msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D149489, max=3D221007, per=3D6.31%, avg=3D2=
13123.72, stdev=3D9569.27
Quoted 1 line
Quoted 1 line
f12: =A0 =A0 read : io=3D32780MB, bw=3D212788KB/s, iops=3D10, runt=3D=
157747msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D154274, max=3D218647, per=3D6.30%, avg=3D2=
12957.41, stdev=3D8233.52
Quoted 1 line
Quoted 1 line
f13: =A0 =A0 read : io=3D32780MB, bw=3D212315KB/s, iops=3D10, runt=3D=
158099msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D153696, max=3D256000, per=3D6.29%, avg=3D2=
12482.68, stdev=3D9203.34
Quoted 1 line
Quoted 1 line
f14: =A0 =A0 read : io=3D32780MB, bw=3D212033KB/s, iops=3D10, runt=3D=
158309msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D150588, max=3D267962, per=3D6.28%, avg=3D2=
12198.76, stdev=3D9572.31
Quoted 1 line
Quoted 1 line
f15: =A0 =A0 read : io=3D32780MB, bw=3D211720KB/s, iops=3D10, runt=3D=
158543msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D146024, max=3D268968, per=3D6.27%, avg=3D2=
11846.40, stdev=3D10341.58
Quoted 1 line
Quoted 1 line
f16: =A0 =A0 read : io=3D32780MB, bw=3D211637KB/s, iops=3D10, runt=3D=
158605msec
Quoted 1 line
Quoted 1 line
=A0 =A0 bw (KB/s) : min=3D148945, max=3D261605, per=3D6.26%, avg=3D2=
11618.40, stdev=3D9240.64
Quoted 3 lines
Quoted 3 lines
Run status group 0 (all jobs):
=A0 =A0READ: io=3D524480MB, aggrb=3D3301MB/s, minb=3D216323KB/s, max=
b=3D219763KB/s, mint=3D156406msec, maxt=3D158893msec
Quoted 3 lines
Quoted 3 lines
Disk stats (read/write):
=A0 sdd: ios=3D261902/0, merge=3D0/0, ticks=3D12531810/0, in_queue=3D=
12532910, util=3D99.46%
Quoted 1 line
Quoted 1 line
=A0 sde: ios=3D262221/0, merge=3D0/0, ticks=3D12494200/0, in_queue=3D=
12495300, util=3D99.50%
Quoted 1 line
Quoted 1 line
=A0 sdf: ios=3D261867/0, merge=3D0/0, ticks=3D12427000/0, in_queue=3D=
12430530, util=3D99.47%
Quoted 1 line
Quoted 1 line
=A0 sdg: ios=3D261983/0, merge=3D0/0, ticks=3D12462320/0, in_queue=3D=
12466060, util=3D99.62%
Quoted 1 line
Quoted 1 line
=A0 sdh: ios=3D262184/0, merge=3D0/0, ticks=3D12487350/0, in_queue=3D=
12489960, util=3D99.49%
Quoted 1 line
Quoted 1 line
=A0 sdi: ios=3D262193/0, merge=3D0/0, ticks=3D12524400/0, in_queue=3D=
12526580, util=3D99.47%
Quoted 1 line
Quoted 1 line
=A0 sdj: ios=3D262044/0, merge=3D0/0, ticks=3D12511850/0, in_queue=3D=
12513840, util=3D99.50%
Quoted 1 line
Quoted 1 line
=A0 sdk: ios=3D262055/0, merge=3D0/0, ticks=3D12526560/0, in_queue=3D=
12527890, util=3D99.50%
Quoted 1 line
Quoted 1 line
=A0 sdl: ios=3D261789/0, merge=3D0/0, ticks=3D12609230/0, in_queue=3D=
12610400, util=3D99.54%
Quoted 1 line
Quoted 1 line
=A0 sdm: ios=3D261787/0, merge=3D0/0, ticks=3D12579000/0, in_queue=3D=
12581050, util=3D99.44%
Quoted 1 line
Quoted 1 line
=A0 sdn: ios=3D261941/0, merge=3D0/0, ticks=3D12524530/0, in_queue=3D=
12525790, util=3D99.48%
Quoted 1 line
Quoted 1 line
=A0 sdo: ios=3D262100/0, merge=3D0/0, ticks=3D12554650/0, in_queue=3D=
12555820, util=3D99.58%
Quoted 1 line
Quoted 1 line
=A0 sdp: ios=3D261877/0, merge=3D0/0, ticks=3D12572220/0, in_queue=3D=
12574610, util=3D99.54%
Quoted 1 line
Quoted 1 line
=A0 sdq: ios=3D261956/0, merge=3D0/0, ticks=3D12601480/0, in_queue=3D=
12603770, util=3D99.62%
Quoted 1 line
Quoted 1 line
=A0 sdr: ios=3D261991/0, merge=3D0/0, ticks=3D12599680/0, in_queue=3D=
12602190, util=3D99.49%
Quoted 1 line
Quoted 1 line
=A0 sds: ios=3D261852/0, merge=3D0/0, ticks=3D12624070/0, in_queue=3D=
12626580, util=3D99.58%
Quoted 4 lines
So, the maximum for these 16 disks is 3301 MiByte/s.
I also tried hardware RAID (2 sets of 8 disks), and got a similar res=
ult:
Quoted 3 lines
Quoted 3 lines
Run status group 0 (all jobs):
=A0 =A0READ: io=3D65560MB, aggrb=3D3024MB/s, minb=3D1548MB/s, maxb=3D=
1550MB/s, mint=3D21650msec, maxt=3D21681msec
Quoted 4 lines
Quoted 4 lines
fio should be able to push these devices up to the line speed. =A0If=
it
Quoted 8 lines
Quoted 2 lines
doesn't I would suggest changing elevators (deadline, cfq, noop) and bumping the max request size to the max supported by the device.
3301 MiByte/s seems like a reasonable number, given the theoretic maximum of 16 times the single disk performance of 16*256 MiByte/s =3D 4096 MiByte/s.
Based on this, I have not looked at tuning. Would you recommend that =
I do?
Quoted 2 lines
Our minimal goal is 2500 MiByte/s; that seems achievable as ZFS was a=
ble
Quoted 3 lines
to reach 2750 MiByte/s without tuning.
Quoted 1 line
When we have a config that does so, we can tune the btrfs side of th=
ings
Quoted 3 lines
Quoted 1 line
as well.
Some files are created in the root folder of the mount point, but I g=
et
Quoted 4 lines
errors instead of results:
Quoted 2 lines
~/fio# ./fio btrfs16.fio
btrfs: (g=3D0): rw=3Dread, bs=3D20M-20M/20M-20M, ioengine=3Dsync, io=
depth=3D8
Quoted 7 lines
Quoted 3 lines
Starting 16 processes
btrfs: Laying out IO file(s) (1 file(s) / 32768MB)
btrfs: Laying out IO file(s) (1 file(s) / 32768MB)[...]
Quoted 2 lines
btrfs: Laying out IO file(s) (1 file(s) / 32768MB)
fio: first direct IO errored. File system may not support direct IO,=
or iomem_align=3D is bad.
Quoted 1 line
Quoted 1 line
fio: first direct IO errored. File system may not support direct IO,=
or iomem_align=3D is bad.
Quoted 1 line
Quoted 1 line
fio: first direct IO errored. File system may not support direct IO,=
or iomem_align=3D is bad.
Quoted 1 line
Quoted 1 line
fio: pid=3D5958, err=3D22/file:engines/sync.c:62, func=3Dxfer, error=
=3DInvalid argument
Quoted 1 line
Quoted 1 line
fio: pid=3D5961, err=3D22/file:engines/sync.c:62, func=3Dxfer, error=
=3DInvalid argument
Quoted 1 line
Quoted 1 line
fio: pid=3D5962, err=3D22/file:engines/sync.c:62, func=3Dxfer, error=
=3DInvalid argument
Quoted 1 line
Quoted 1 line
fio: first direct IO errored. File system may not support direct IO,=
or iomem_align=3D is bad.
Quoted 3 lines
[...]
Quoted 2 lines
btrfs: (groupid=3D0, jobs=3D1): err=3D22 (file:engines/sync.c:62, fu=
nc=3Dxfer, error=3DInvalid argument): pid=3D5956
Quoted 1 line
Quoted 1 line
=A0 cpu =A0 =A0 =A0 =A0 =A0: usr=3D0.00%, sys=3D0.00%, ctx=3D1, majf=
=3D0, minf=3D52
Quoted 1 line
Quoted 1 line
=A0 IO depths =A0 =A0: 1=3D100.0%, 2=3D0.0%, 4=3D0.0%, 8=3D0.0%, 16=3D=
0.0%, 32=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
Quoted 1 line
=A0 =A0 =A0submit =A0 =A0: 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%=
, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0%
Quoted 1 line
Quoted 1 line
=A0 =A0 =A0complete =A0: 0=3D50.0%, 4=3D50.0%, 8=3D0.0%, 16=3D0.0%, =
32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0%
Quoted 5 lines
Quoted 1 line
=A0 =A0 =A0issued r/w: total=3D1/0, short=3D0/0
[no results]
What could be going on here?
(I get the same result from the github version of fio, fio 1.42, as w=
ell
Quoted 1 line
as the one that came with Ubuntu, fio 1.33.1).
If you are using 2.6.32 (as above), BTRFS on this release doesn't support direct I/O. It is supported on 2.6.35, so you could retry with (eg):
http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.35-maverick/linux-im=
age-2.6.35-020635-generic_2.6.35-020635_amd64.deb
--=20
Daniel J Blueman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 26 lines · Freek Dijkstra
Quoted 6 lines · Chris Mason
Basically we have two different things to tune. First the block layer and then btrfs.
And then we need to setup a fio job file that hammers on all the ssds at once. I'd have it use adio/dio and talk directly to the drives.
Thanks. First one disk:
Quoted 15 lines
f1: (groupid=0, jobs=1): err= 0: pid=6273
read : io=32780MB, bw=260964KB/s, iops=12, runt=128626msecclat (usec): min=74940, max=80721, avg=78449.61, stdev=923.24 bw (KB/s) : min=240469, max=269981, per=100.10%, avg=261214.77, stdev=2765.91 cpu : usr=0.01%, sys=2.69%, ctx=1747, majf=0, minf=5153 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=1639/0, short=0/0 lat (msec): 100=100.00%Run status group 0 (all jobs):
READ: io=32780MB, aggrb=260963KB/s, minb=267226KB/s, maxb=267226KB/s, mint=128626msec, maxt=128626msec
Quoted 6 lines
Quoted 2 lines
Disk stats (read/write):
sdd: ios=261901/0, merge=0/0, ticks=10135270/0, in_queue=10136460, util=99.30%So 255 MiByte/s.
Out of curiousity, what is the distinction between the reported figures of 260964 kiB/s, 261214.77 kiB/s, 267226 kiB/s and 260963 kiB/s?
When there is only one job, they should all be the same. aggr is the total seen across all the jobs, min is the lowest, max is the highest.
Quoted 17 lines
Now 16 disks (abbreviated):
Quoted 13 lines
~/fio# ./fio ssd.fio
Starting 16 processes
f1: (groupid=0, jobs=1): err= 0: pid=4756
read : io=32780MB, bw=212987KB/s, iops=10, runt=157600msecclat (msec): min=75, max=138, avg=96.15, stdev= 4.47 lat (msec): min=75, max=138, avg=96.15, stdev= 4.47 bw (KB/s) : min=153121, max=268968, per=6.31%, avg=213181.15, stdev=9052.26 cpu : usr=0.00%, sys=1.71%, ctx=2737, majf=0, minf=5153 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=1639/0, short=0/0
Quoted 10 lines
Quoted 3 lines
lat (msec): 100=97.99%, 250=2.01% Run status group 0 (all jobs): READ: io=524480MB, aggrb=3301MB/s, minb=216323KB/s, maxb=219763KB/s, mint=156406msec, maxt=158893msecSo, the maximum for these 16 disks is 3301 MiByte/s.
I also tried hardware RAID (2 sets of 8 disks), and got a similar result:
Quoted 2 lines
Run status group 0 (all jobs):
READ: io=65560MB, aggrb=3024MB/s, minb=1548MB/s, maxb=1550MB/s, mint=21650msec, maxt=21681msec
Great, so we know the drives are fast.
Quoted 13 lines
Quoted 6 lines
fio should be able to push these devices up to the line speed. If it doesn't I would suggest changing elevators (deadline, cfq, noop) and bumping the max request size to the max supported by the device.
3301 MiByte/s seems like a reasonable number, given the theoretic maximum of 16 times the single disk performance of 16*256 MiByte/s = 4096 MiByte/s.
Based on this, I have not looked at tuning. Would you recommend that I do?
Quoted 9 lines
Our minimal goal is 2500 MiByte/s; that seems achievable as ZFS was able to reach 2750 MiByte/s without tuning.
Quoted 2 lines
When we have a config that does so, we can tune the btrfs side of things as well.
Some files are created in the root folder of the mount point, but I get errors instead of results:
Someone else mentioned that btrfs only gained DIO reads in 2.6.35. I think you'll get the best results with that kernel if you can find an update.
If not, you can change the fio job file to remove direct=1 and increase the bs flag up to 20M.
I'd also suggest changing /sys/class/bdi/btrfs-1/read_ahead_kb to a bigger number. Try 20480
-chris
Quoted 10 lines · Chris Mason
And then we need to setup a fio job file that hammers on all the ssds at once. I'd have it use adio/dio and talk directly to the drives. I'd do something like this for the fio job file, but Jens Axboe is cc'd and he might make another suggestion on the job file. I'd do something like this in a file named ssd.fio
[global]
size=32g
direct=1
iodepth=8
iodepth=8 will have no effect if you don't also set a different IO engine, otherwise you would be using read(2) to fetch the data. So add ioengine=libaio to take advantage of a higher queue depth as well.
Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually. That helps a lot with higher transfer rates. You can check support for hw crc32c by checking for the 'sse4_2' flag in /proc/cpuinfo.
Signature · 1 line
Jens Axboe
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Chris Mason
And then we need to setup a fio job file that hammers on all the ssds at once. I'd have it use adio/dio and talk directly to the drives. I'd do something like this for the fio job file, but Jens Axboe is cc'd and he might make another suggestion on the job file. I'd do something like this in a file named ssd.fio
Quoted 8 lines
Quoted 4 lines
[global]
size=32g
direct=1
iodepth=8iodepth=8 will have no effect if you don't also set a different IO engine, otherwise you would be using read(2) to fetch the data. So add ioengine=libaio to take advantage of a higher queue depth as well.
Yeah, I just realized I messed up the suggested file, but it worked well enough on the block devices, so I think just having 16 procs hitting the array was enough. libaio will only help with O_DIRECT though, so this only applies to 2.6.35 as well.
Quoted 6 lines
Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually. That helps a lot with higher transfer rates. You can check support for hw crc32c by checking for the '
sse4_2' flag in/proc/cpuinfo.
Yeah, the HW assisted crc does make a huge difference.
-chris
Chris Mason wrote (ao):
Quoted 8 lines · Jens Axboe
Quoted 5 lines
Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually. That helps a lot with higher transfer rates. You can check support for hw crc32c by checking for the '
sse4_2' flag in/proc/cpuinfo.Yeah, the HW assisted crc does make a huge difference.
The above says "newer intel box". I did some googling and it seems to mean really Intel CPUs only, not AMD, correct?
Is there a way to get hardware support for crc32c on ARM based systems?
Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Quoted 8 lines · Sander
Chris Mason wrote (ao):
Quoted 7 lines · Jens Axboe
Quoted 5 lines
Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually. That helps a lot with higher transfer rates. You can check support for hw crc32c by checking for the '
sse4_2' flag in/proc/cpuinfo.
Quoted 6 lines
Quoted 1 line
Yeah, the HW assisted crc does make a huge difference.
The above says "newer intel box". I did some googling and it seems to mean really Intel CPUs only, not AMD, correct?
Is there a way to get hardware support for crc32c on ARM based systems?
So far I only know of the intel sse4.2 systems that support this.
-chris
Quoted 4 lines · Jens Axboe
Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually.
I have a patch for that, will post it later: autoloading of modules based on x86 cpuinfo.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 8 lines · Andi Kleen
Quoted 4 lines · Jens Axboe
Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually.
I have a patch for that, will post it later: autoloading of modules based on x86 cpuinfo.
Great, it is pretty annoying to have to do it manually. Sometimes you forget. And it's not possible to de-select CRC32C and have the intel variant loaded.
Signature · 1 line
Jens Axboe
Hi all,
Thanks a lot for the great feedback from before the weekend. Since one of my colleagues needed the machine, I could only do the tests today.
In short: just installing 2.6.35 did make some difference, but I was mostly impressed with the speedup gained by the hardware acceleration of the crc32c_intel module.
Here is some quick data.
Reference figures:
16* single disk (theoretical limit): 4092 MiByte/s
fio data layer tests (achievable limit): 3250 MiByte/s
ZFS performance: 2505 MiByte/s
BtrFS figures:
IOzone on 2.6.32: 919 MiByte/s fio btrfs tests on 2.6.35: 1460 MiByte/s IOzone on 2.6.35 with crc32c: 1250 MiByte/s
IOzone on 2.6.35 with crc32c_intel: 1629 MiByte/s
IOzone on 2.6.35, using -o nodatasum: 1955 MiByte/s
For those finding this message and want a howto: the easiest way to use crc32c_intel is to add the module name to /etc/modules:
# echo "crc32c_intel" >> /etc/modules # reboot
Now the next step for us is to tune the block sizes. We only did that preliminary, but now that we have a good knowledge of what software to use, we can start tuning that in more detail.
If there is interest on this list, I'll gladly post our results here.
Quoted 7 lines · Jens Axboe
Quoted 3 lines
Quoted 3 lines
Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually.
it is pretty annoying to have to do it manually. Sometimes you forget. And it's not possible to de-select CRC32C and have the intel variant loaded.
You can, but only if you first unmount the partition:
# unmount /mnt/mybtrfsdisk # rmmod btrfs # rmmod libcrc32c # rmmod crc32c # modprobe crc32c_intel # mount -t btrfs /dev/sda1 /mnt/mybtrfsdisk
We encountered a small bug: the btrfs partition with RAID0 that was made on 2.6.32 did not mount after a reboot or after unmounting. Running btrfsck fixes this, but after a next umount, we had to run btrfsck again. After recreating the btrfs partition on 2.6.35, all was well. btrfs partitions that don't use (software) RAID work fine.
~# mount -t btrfs -o ssd /dev/sdd /mnt/ssd3
mount: wrong fs type, bad option, bad superblock on /dev/sdd,
missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so
~# dmesg | tail
device fsid ec4d518ec61d4496-81e5aeda2d8ef7b5 devid 1 transid 69 /dev/sdd
btrfs: use ssd allocation scheme
btrfs: failed to read the system array on sdd
btrfs: open_ctree failed
~# btrfsck /dev/sdd
found 550511136768 bytes used err is 0
total csum bytes: 536870912
total tree bytes: 755322880
total fs tree bytes: 77824
btree space waste bytes: 169152328
file data blocks allocated: 549755813888
referenced 549755813888
Btrfs Btrfs v0.19
~# mount -t btrfs -o ssd /dev/sdd /mnt/ssd3
[and it mounts fine now]
Regards,
Freek Dijkstra
SARA High Performance Computing and Networking
Quoted 11 lines · Freek Dijkstra
Hi all,
Thanks a lot for the great feedback from before the weekend. Since one of my colleagues needed the machine, I could only do the tests today.
In short: just installing 2.6.35 did make some difference, but I was mostly impressed with the speedup gained by the hardware acceleration of the
crc32c_intelmodule.Here is some quick data.
Quoted 8 lines
Reference figures:
16* single disk (theoretical limit): 4092 MiByte/s
fio data layer tests (achievable limit): 3250 MiByte/s
ZFS performance: 2505 MiByte/sBtrFS figures:
IOzone on 2.6.32: 919 MiByte/s fio btrfs tests on 2.6.35: 1460 MiByte/s
Was this one with O_DIRECT?
Quoted 6 lines
IOzone on 2.6.35 with crc32c: 1250 MiByte/s IOzone on 2.6.35 with crc32c_intel: 1629 MiByte/s IOzone on 2.6.35, using -o nodatasum: 1955 MiByte/sFor those finding this message and want a howto: the easiest way to use
crc32c_intelis to add the module name to/etc/modules:
Quoted 8 lines
# echo "crc32c_intel" >> /etc/modules # rebootNow the next step for us is to tune the block sizes. We only did that preliminary, but now that we have a good knowledge of what software to use, we can start tuning that in more detail.
If there is interest on this list, I'll gladly post our results here.
Definitely, please do.
-chris
Quoted 3 lines · Freek Dijkstra
Hi,
We're interested in getting the highest possible read performance on =
a
Quoted 1 line
server. To that end, we have a high-end server with multiple solid st=
ate
Quoted 1 line
disks (SSDs). Since BtrFS outperformed other Linux filesystem, we cho=
ose
Quoted 14 lines
that. Unfortunately, there seems to be an upper boundary in the performance of BtrFS of roughly 1 GiByte/s read speed. Compare the following results with either BTRFS on Ubuntu versus ZFS on FreeBSD:
=A0 =A0 =A0 =A0 =A0 =A0 ZFS =A0 =A0 =A0 =A0 =A0 =A0 BtrFS 1 SSD =A0 =A0 =A0256 MiByte/s =A0 =A0 256 MiByte/s 2 SSDs =A0 =A0 505 MiByte/s =A0 =A0 504 MiByte/s 3 SSDs =A0 =A0 736 MiByte/s =A0 =A0 756 MiByte/s 4 SSDs =A0 =A0 952 MiByte/s =A0 =A0 916 MiByte/s 5 SSDs =A0 =A01226 MiByte/s =A0 =A0 986 MiByte/s 6 SSDs =A0 =A01450 MiByte/s =A0 =A0 978 MiByte/s 8 SSDs =A0 =A01653 MiByte/s =A0 =A0 932 MiByte/s 16 SSDs =A0 2750 MiByte/s =A0 =A0 919 MiByte/s
Quoted 1 line
The results were originally measured on a Dell PowerEdge T610, but we=
re
Quoted 2 lines
repeated using a SuperMicro machine with 4 independent SAS+SATA controllers. We made sure that the PCI-e slots where not the bottlene=
ck.
Quoted 6 lines
The above results were for Ubuntu 10.04.1 server, with BtrFS v0.19, although earlier tests with Ubuntu 9.10 showed the same results.
Apparently, the limitation is not in the hardware (the same hardware with ZFS did scale near linear). We also tested both hardware RAID-0, software RAID-0 (md), and using the btrfs built-in software RAID-0, b=
ut
Quoted 1 line
the differences were small (<10%) (md-based software RAID was margina=
lly
Quoted 1 line
slower on Linux; RAIDZ was marginally faster on FreeBSD). So we presu=
me
Quoted 3 lines
that the bottleneck is somewhere in the BtrFS (or kernel) software.
Are there suggestions how to tune the read performance? We like to sc=
ale
Quoted 5 lines
this up to 32 solid state disks. The -o ssd option did not improve overall performance, although it did gave more stable results (less fluctuation in repeated tests).
Note that the write speeds did scale fine. In the scenario with 16 so=
lid
Quoted 1 line
state disks, the write speed is 1596 MiByte/s (1.7 times as fast as t=
he
Quoted 1 line
read speed! Suffice to say that for a single disk, write is much slow=
er
Quoted 11 lines
than read...).
Here are the exact settings:
~# mkfs.btrfs -d raid0/dev/sdd/dev/sde/dev/sdf/dev/sdg\
=A0 =A0/dev/sdh/dev/sdi/dev/sdj/dev/sdk/dev/sdl/dev/sdm\
=A0 =A0/dev/sdn/dev/sdo/dev/sdp/dev/sdq/dev/sdr/dev/sdsnodesize 4096 leafsize 4096 sectorsize 4096 size 2.33TB Btrfs Btrfs v0.19
~# mount -t btrfs -o ssd/dev/sdd/mnt/ssd6
~# iozone -s 32G -r 1024 -i 0 -i 1 -w -f/mnt/ssd6/iozone.tmp=A0 =A0 =A0 =A0 =A0 =A0 =A0KB =A0reclen =A0 write rewrite =A0 =A0read=
=A0 =A0reread
Quoted 1 line
=A0 =A0 =A0 =A033554432 =A0 =A01024 1628475 1640349 =A0 943416 =A0 95=
1135
Perhaps create a new filesystem and mount with 'nodatasum' - existing extents which were previously created will be checked, so need to start fresh.
Daniel
--=20
Daniel J Blueman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello,
Quoted 8 lines · [email protected] (Freek Dijkstra) writes:
[...]
Here are the exact settings:
~# mkfs.btrfs -d raid0/dev/sdd/dev/sde/dev/sdf/dev/sdg\
/dev/sdh/dev/sdi/dev/sdj/dev/sdk/dev/sdl/dev/sdm\
/dev/sdn/dev/sdo/dev/sdp/dev/sdq/dev/sdr/dev/sds
nodesize 4096 leafsize 4096 sectorsize 4096 size 2.33TB Btrfs Btrfs v0.19
Don't you need to stripe metadata too (with -m raid0)? Or you may be limited by your metadata drive?
Signature · 3 lines
Mathieu Chouquet-Stringer [email protected] The sun itself sees not till heaven clears. -- William Shakespeare --