Poor write performance with raid5 on 4 drives
Hello,
I’m building my personal tertiary backup box running on HP Microserver G8 with 4x 6TB hard drives and I was planning to use raid5 profile for reasonable data ratio but I ran into really poor write performance on raid5 with 4 drives. On a filesystem with -d=raid5 -m=raid1c4 on 4 drives I only get about 40MB/s of write speed, but with -d=raid5 -m=raid1c3 on 3 drives or -d=raid6 -m raid1c4 on 4 drives I get 500MB/s of write speed.
I tried to investigate it somehow, I compared the properties of the filesystem and other variables in /sys/fs/btrfs between each other, but all I found was that raid5 on 4 drives reads from hard drives while writing data and that would be whole problem with poor performance imho. But that’s where my skills end.
Do you know if there is a solution to this or can you point me in which direction to look?
Thanks,
Tomas
[root@backupcube ~]# uname -a
Linux backupcube 5.19.11-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 23 15:07:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [root@backupcube ~]# btrfs --version
btrfs-progs v5.18
### -d=raid5 -m=raid1c4 on 4 drives ###
[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c4 /dev/sda1 /dev/sdb1 /dev/sdd1 /dev/sde1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.
WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Label: (null)
UUID: 279a913f-bf4a-44ba-a5cd-26c94c954365
Node size: 16384
Sector size: 4096
Filesystem size: 21.83TiB
Block group profiles:
Data: RAID5 3.00GiB
Metadata: RAID1C4 1.00GiB
System: RAID1C4 8.00MiB
SSD detected: no
Zoned device: no
Incompat features: extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features: free-space-tree
Checksum: crc32c
Number of devices: 4
Devices:
ID SIZE PATH
1 5.46TiB /dev/sda1
2 5.46TiB /dev/sdb1
3 5.46TiB /dev/sdd1
4 5.46TiB /dev/sde1[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4270850048 bytes (4.3 GB, 4.0 GiB) copied, 106 s, 40.3 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 106.619 s, 40.3 MB/s
[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-
read writ: read writ: read writ: read writ: read writ
1215k 13M:1215k 13M:1215k 13M:1215k 13M:4858k 52M 1216k 13M:1216k 13M:1216k 13M:1216k 13M:4864k 52M 1216k 13M:1216k 13M:1216k 13M:1216k 13M:4864k 52M 1280k 13M:1216k 13M:1216k 13M:1216k 13M:4928k 52M 1152k 13M:1216k 13M:1216k 13M:1216k 13M:4800k 52M 1216k 13M:1216k 13M:1216k 13M:1216k 13M:4864k 52M^C
[root@backupcube ~]#
### -d=raid6 -m=raid1c4 on 4 drives ###
[root@backupcube ~]# mkfs.btrfs -f -d raid6 -m raid1c4 /dev/sda1 /dev/sdb1 /dev/sdd1 /dev/sde1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.
WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Label: (null)
UUID: 9e4f6691-971d-4950-a032-d3c83f2fda26
Node size: 16384
Sector size: 4096
Filesystem size: 21.83TiB
Block group profiles:
Data: RAID6 2.00GiB
Metadata: RAID1C4 1.00GiB
System: RAID1C4 8.00MiB
SSD detected: no
Zoned device: no
Incompat features: extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features: free-space-tree
Checksum: crc32c
Number of devices: 4
Devices:
ID SIZE PATH
1 5.46TiB /dev/sda1
2 5.46TiB /dev/sdb1
3 5.46TiB /dev/sdd1
4 5.46TiB /dev/sde1[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4051697664 bytes (4.1 GB, 3.8 GiB) copied, 8 s, 506 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.58926 s, 500 MB/s
[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-
read writ: read writ: read writ: read writ: read writ 0 238M: 0 238M: 0 238M: 0 238M: 0 953M 0 250M: 0 250M: 0 250M: 0 250M: 0 1001M 0 242M: 0 242M: 0 242M: 0 242M: 0 968M 0 238M: 0 238M: 0 238M: 0 238M: 0 953M 0 242M: 0 242M: 0 243M: 0 242M: 0 970M 0 243M: 0 243M: 0 243M: 0 243M: 0 972M 0 192M: 0 192M: 0 192M: 0 192M: 0 769M
0 0 : 0 0 : 0 0 : 0 0 : 0 6144B^C
### -d=raid5 -m=raid1c3 on 3 drives ###
[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c3 /dev/sda1 /dev/sdb1 /dev/sdd1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.
WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Label: (null)
UUID: 7f06fbb0-9655-4f8d-af4a-2d06d8dc1d80
Node size: 16384
Sector size: 4096
Filesystem size: 16.37TiB
Block group profiles:
Data: RAID5 2.00GiB
Metadata: RAID1C3 1.00GiB
System: RAID1C3 8.00MiB
SSD detected: no
Zoned device: no
Incompat features: extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features: free-space-tree
Checksum: crc32c
Number of devices: 3
Devices:
ID SIZE PATH
1 5.46TiB /dev/sda1
2 5.46TiB /dev/sdb1
3 5.46TiB /dev/sdd1[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4034920448 bytes (4.0 GB, 3.8 GiB) copied, 8 s, 504 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.61927 s, 498 MB/s
[root@backupcube ~]#
[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-
read writ: read writ: read writ: read writ: read writ 0 231M: 0 231M: 0 231M: 0 0 : 0 694M 0 244M: 0 244M: 0 244M: 0 0 : 0 733M 0 250M: 0 250M: 0 250M: 0 0 : 0 751M 0 238M: 0 238M: 0 238M: 0 0 : 0 715M 0 239M: 0 239M: 0 239M: 0 0 : 0 718M 0 243M: 0 243M: 0 243M: 0 0 : 0 729M 0 243M: 0 243M: 0 243M: 0 0 : 0 728M 0 117M: 0 117M: 0 117M: 0 0 : 0 351M
0 0 : 0 0 : 0 0 : 0 0 : 0 0 ^C
Quoted 58 lines · Tomáš Holý
Hello,
I’m building my personal tertiary backup box running on HP Microserver G8 with 4x 6TB hard drives and I was planning to use raid5 profile for reasonable data ratio but I ran into really poor write performance on raid5 with 4 drives. On a filesystem with -d=raid5 -m=raid1c4 on 4 drives I only get about 40MB/s of write speed, but with -d=raid5 -m=raid1c3 on 3 drives or -d=raid6 -m raid1c4 on 4 drives I get 500MB/s of write speed.I tried to investigate it somehow, I compared the properties of the filesystem and other variables in
/sys/fs/btrfsbetween each other, but all I found was that raid5 on 4 drives reads from hard drives while writing data and that would be whole problem with poor performance imho. But that’s where my skills end.Do you know if there is a solution to this or can you point me in which direction to look?
Thanks,
Tomas[root@backupcube ~]# uname -a
Linux backupcube 5.19.11-200.fc36.x86_64 #1 SMPPREEMPT_DYNAMICFri Sep 23 15:07:44 UTC 2022x86_64x86_64x86_64GNU/Linux [root@backupcube ~]# btrfs --version
btrfs-progs v5.18### -d=raid5 -m=raid1c4 on 4 drives ###
[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c4
/dev/sda1/dev/sdb1/dev/sdd1/dev/sde1btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)Label: (null) UUID: 279a913f-bf4a-44ba-a5cd-26c94c954365 Node size: 16384 Sector size: 4096 Filesystem size: 21.83TiB Block group profiles: Data: RAID5 3.00GiB Metadata: RAID1C4 1.00GiB System: RAID1C4 8.00MiB SSD detected: no Zoned device: no Incompat features: extref, raid56, skinny-metadata, no-holes, raid1c34 Runtime features: free-space-tree Checksum: crc32c Number of devices: 4 Devices: ID SIZE PATH 1 5.46TiB /dev/sda1 2 5.46TiB /dev/sdb1 3 5.46TiB /dev/sdd1 4 5.46TiB /dev/sde1
Quoted 1 line
[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct
One thing I notice is, you're using blocksize which is not really aligned to a full stripe length.
For 4 disk RAID5, it's *3* data stripes + 1 parity.
Thus the full stripe length is 64Kx3.
1M block size contains 5 and 1/3 of a full stripe, thus I'm wondering if the 1/3 (substripe) write is causing problem.
Especially considering substripe will also have a wait time trying to merge more substripe write into a full one.
Considering you have already tested 3 disks RAID5, if you have enough disks, mind to test if 5 disks RAID5 would have a different result?
Thanks,
Qu
Quoted 138 lines
4270850048 bytes (4.3 GB, 4.0 GiB) copied, 106 s, 40.3 MB/s 4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 106.619 s, 40.3 MB/s[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-
read writ: read writ: read writ: read writ: read writ 1215k 13M:1215k 13M:1215k 13M:1215k 13M:4858k 52M 1216k 13M:1216k 13M:1216k 13M:1216k 13M:4864k 52M 1216k 13M:1216k 13M:1216k 13M:1216k 13M:4864k 52M 1280k 13M:1216k 13M:1216k 13M:1216k 13M:4928k 52M 1152k 13M:1216k 13M:1216k 13M:1216k 13M:4800k 52M 1216k 13M:1216k 13M:1216k 13M:1216k 13M:4864k 52M^C[root@backupcube ~]#
### -d=raid6 -m=raid1c4 on 4 drives ###
[root@backupcube ~]# mkfs.btrfs -f -d raid6 -m raid1c4
/dev/sda1/dev/sdb1/dev/sdd1/dev/sde1btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)Label: (null) UUID: 9e4f6691-971d-4950-a032-d3c83f2fda26 Node size: 16384 Sector size: 4096 Filesystem size: 21.83TiB Block group profiles: Data: RAID6 2.00GiB Metadata: RAID1C4 1.00GiB System: RAID1C4 8.00MiB SSD detected: no Zoned device: no Incompat features: extref, raid56, skinny-metadata, no-holes, raid1c34 Runtime features: free-space-tree Checksum: crc32c Number of devices: 4 Devices: ID SIZE PATH 1 5.46TiB /dev/sda1 2 5.46TiB /dev/sdb1 3 5.46TiB /dev/sdd1 4 5.46TiB /dev/sde1[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4051697664 bytes (4.1 GB, 3.8 GiB) copied, 8 s, 506 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.58926 s, 500 MB/s[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-
read writ: read writ: read writ: read writ: read writ 0 238M: 0 238M: 0 238M: 0 238M: 0 953M 0 250M: 0 250M: 0 250M: 0 250M: 0 1001M 0 242M: 0 242M: 0 242M: 0 242M: 0 968M 0 238M: 0 238M: 0 238M: 0 238M: 0 953M 0 242M: 0 242M: 0 243M: 0 242M: 0 970M 0 243M: 0 243M: 0 243M: 0 243M: 0 972M 0 192M: 0 192M: 0 192M: 0 192M: 0 769M 0 0 : 0 0 : 0 0 : 0 0 : 0 6144B^C### -d=raid5 -m=raid1c3 on 3 drives ###
[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c3
/dev/sda1/dev/sdb1/dev/sdd1btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)Label: (null) UUID: 7f06fbb0-9655-4f8d-af4a-2d06d8dc1d80 Node size: 16384 Sector size: 4096 Filesystem size: 16.37TiB Block group profiles: Data: RAID5 2.00GiB Metadata: RAID1C3 1.00GiB System: RAID1C3 8.00MiB SSD detected: no Zoned device: no Incompat features: extref, raid56, skinny-metadata, no-holes, raid1c34 Runtime features: free-space-tree Checksum: crc32c Number of devices: 3 Devices: ID SIZE PATH 1 5.46TiB /dev/sda1 2 5.46TiB /dev/sdb1 3 5.46TiB /dev/sdd1[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4034920448 bytes (4.0 GB, 3.8 GiB) copied, 8 s, 504 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.61927 s, 498 MB/s
[root@backupcube ~]#[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-
read writ: read writ: read writ: read writ: read writ 0 231M: 0 231M: 0 231M: 0 0 : 0 694M 0 244M: 0 244M: 0 244M: 0 0 : 0 733M 0 250M: 0 250M: 0 250M: 0 0 : 0 751M 0 238M: 0 238M: 0 238M: 0 0 : 0 715M 0 239M: 0 239M: 0 239M: 0 0 : 0 718M 0 243M: 0 243M: 0 243M: 0 0 : 0 729M 0 243M: 0 243M: 0 243M: 0 0 : 0 728M 0 117M: 0 117M: 0 117M: 0 0 : 0 351M 0 0 : 0 0 : 0 0 : 0 0 : 0 0 ^C