Skip to contentSkip to conversation

Poor write performance with raid5 on 4 drives

DISCUSSION Tomáš Holý 2 participants 2 messages Oct 1, 2022 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hello,
I’m building my personal tertiary backup box running on HP Microserver G8 with 4x 6TB hard drives and I was planning to use raid5 profile for reasonable data ratio but I ran into really poor write performance on raid5 with 4 drives. On a filesystem with -d=raid5 -m=raid1c4 on 4 drives I only get about 40MB/s of write speed, but with -d=raid5 -m=raid1c3 on 3 drives or -d=raid6 -m raid1c4 on 4 drives I get 500MB/s of write speed.

I tried to investigate it somehow, I compared the properties of the filesystem and other variables in /sys/fs/btrfs between each other, but all I found was that raid5 on 4 drives reads from hard drives while writing data and that would be whole problem with poor performance imho. But that’s where my skills end.

Do you know if there is a solution to this or can you point me in which direction to look?

Thanks,
Tomas

[root@backupcube ~]# uname -a
Linux backupcube 5.19.11-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 23 15:07:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [root@backupcube ~]# btrfs --version
btrfs-progs v5.18

### -d=raid5 -m=raid1c4 on 4 drives ###

[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c4 /dev/sda1 /dev/sdb1 /dev/sdd1 /dev/sde1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.

WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               279a913f-bf4a-44ba-a5cd-26c94c954365
Node size:          16384
Sector size:        4096
Filesystem size:    21.83TiB
Block group profiles:
  Data:             RAID5             3.00GiB
  Metadata:         RAID1C4           1.00GiB
  System:           RAID1C4           8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  4
Devices:
   ID        SIZE  PATH
    1     5.46TiB  /dev/sda1
    2     5.46TiB  /dev/sdb1
    3     5.46TiB  /dev/sdd1
    4     5.46TiB  /dev/sde1

[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4270850048 bytes (4.3 GB, 4.0 GiB) copied, 106 s, 40.3 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 106.619 s, 40.3 MB/s

[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-
read writ: read writ: read writ: read writ: read writ

1215k   13M:1215k   13M:1215k   13M:1215k   13M:4858k   52M
1216k   13M:1216k   13M:1216k   13M:1216k   13M:4864k   52M
1216k   13M:1216k   13M:1216k   13M:1216k   13M:4864k   52M
1280k   13M:1216k   13M:1216k   13M:1216k   13M:4928k   52M
1152k   13M:1216k   13M:1216k   13M:1216k   13M:4800k   52M
1216k   13M:1216k   13M:1216k   13M:1216k   13M:4864k   52M^C

[root@backupcube ~]#

### -d=raid6 -m=raid1c4 on 4 drives ###

[root@backupcube ~]# mkfs.btrfs -f -d raid6 -m raid1c4 /dev/sda1 /dev/sdb1 /dev/sdd1 /dev/sde1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.

WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               9e4f6691-971d-4950-a032-d3c83f2fda26
Node size:          16384
Sector size:        4096
Filesystem size:    21.83TiB
Block group profiles:
  Data:             RAID6             2.00GiB
  Metadata:         RAID1C4           1.00GiB
  System:           RAID1C4           8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  4
Devices:
   ID        SIZE  PATH
    1     5.46TiB  /dev/sda1
    2     5.46TiB  /dev/sdb1
    3     5.46TiB  /dev/sdd1
    4     5.46TiB  /dev/sde1

[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4051697664 bytes (4.1 GB, 3.8 GiB) copied, 8 s, 506 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.58926 s, 500 MB/s

[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-

 read  writ: read  writ: read  writ: read  writ: read  writ
   0   238M:   0   238M:   0   238M:   0   238M:   0   953M
   0   250M:   0   250M:   0   250M:   0   250M:   0  1001M
   0   242M:   0   242M:   0   242M:   0   242M:   0   968M
   0   238M:   0   238M:   0   238M:   0   238M:   0   953M
   0   242M:   0   242M:   0   243M:   0   242M:   0   970M
   0   243M:   0   243M:   0   243M:   0   243M:   0   972M
   0   192M:   0   192M:   0   192M:   0   192M:   0   769M

0 0 : 0 0 : 0 0 : 0 0 : 0 6144B^C

### -d=raid5 -m=raid1c3 on 3 drives ###

[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c3 /dev/sda1 /dev/sdb1 /dev/sdd1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.

WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               7f06fbb0-9655-4f8d-af4a-2d06d8dc1d80
Node size:          16384
Sector size:        4096
Filesystem size:    16.37TiB
Block group profiles:
  Data:             RAID5             2.00GiB
  Metadata:         RAID1C3           1.00GiB
  System:           RAID1C3           8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  3
Devices:
   ID        SIZE  PATH
    1     5.46TiB  /dev/sda1
    2     5.46TiB  /dev/sdb1
    3     5.46TiB  /dev/sdd1

[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4034920448 bytes (4.0 GB, 3.8 GiB) copied, 8 s, 504 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.61927 s, 498 MB/s
[root@backupcube ~]#

[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-

 read  writ: read  writ: read  writ: read  writ: read  writ
   0   231M:   0   231M:   0   231M:   0     0 :   0   694M
   0   244M:   0   244M:   0   244M:   0     0 :   0   733M
   0   250M:   0   250M:   0   250M:   0     0 :   0   751M
   0   238M:   0   238M:   0   238M:   0     0 :   0   715M
   0   239M:   0   239M:   0   239M:   0     0 :   0   718M
   0   243M:   0   243M:   0   243M:   0     0 :   0   729M
   0   243M:   0   243M:   0   243M:   0     0 :   0   728M
   0   117M:   0   117M:   0   117M:   0     0 :   0   351M

0 0 : 0 0 : 0 0 : 0 0 : 0 0 ^C

#1 lore raw
Quoted 58 lines · Tomáš Holý

Hello,
I’m building my personal tertiary backup box running on HP Microserver G8 with 4x 6TB hard drives and I was planning to use raid5 profile for reasonable data ratio but I ran into really poor write performance on raid5 with 4 drives. On a filesystem with -d=raid5 -m=raid1c4 on 4 drives I only get about 40MB/s of write speed, but with -d=raid5 -m=raid1c3 on 3 drives or -d=raid6 -m raid1c4 on 4 drives I get 500MB/s of write speed.

I tried to investigate it somehow, I compared the properties of the filesystem and other variables in /sys/fs/btrfs between each other, but all I found was that raid5 on 4 drives reads from hard drives while writing data and that would be whole problem with poor performance imho. But that’s where my skills end.

Do you know if there is a solution to this or can you point me in which direction to look?

Thanks,
Tomas

[root@backupcube ~]# uname -a
Linux backupcube 5.19.11-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 23 15:07:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [root@backupcube ~]# btrfs --version
btrfs-progs v5.18

### -d=raid5 -m=raid1c4 on 4 drives ###

[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c4 /dev/sda1 /dev/sdb1 /dev/sdd1 /dev/sde1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.

WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               279a913f-bf4a-44ba-a5cd-26c94c954365
Node size:          16384
Sector size:        4096
Filesystem size:    21.83TiB
Block group profiles:
   Data:             RAID5             3.00GiB
   Metadata:         RAID1C4           1.00GiB
   System:           RAID1C4           8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  4
Devices:
    ID        SIZE  PATH
     1     5.46TiB  /dev/sda1
     2     5.46TiB  /dev/sdb1
     3     5.46TiB  /dev/sdd1
     4     5.46TiB  /dev/sde1
Quoted 1 line

[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct

One thing I notice is, you're using blocksize which is not really aligned to a full stripe length.

For 4 disk RAID5, it's *3* data stripes + 1 parity.

Thus the full stripe length is 64Kx3.

1M block size contains 5 and 1/3 of a full stripe, thus I'm wondering if the 1/3 (substripe) write is causing problem.

Especially considering substripe will also have a wait time trying to merge more substripe write into a full one.

Considering you have already tested 3 disks RAID5, if you have enough disks, mind to test if 5 disks RAID5 would have a different result?

Thanks,
Qu

Quoted 138 lines

4270850048 bytes (4.3 GB, 4.0 GiB) copied, 106 s, 40.3 MB/s 4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 106.619 s, 40.3 MB/s

[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-

  read  writ: read  writ: read  writ: read  writ: read  writ
1215k   13M:1215k   13M:1215k   13M:1215k   13M:4858k   52M
1216k   13M:1216k   13M:1216k   13M:1216k   13M:4864k   52M
1216k   13M:1216k   13M:1216k   13M:1216k   13M:4864k   52M
1280k   13M:1216k   13M:1216k   13M:1216k   13M:4928k   52M
1152k   13M:1216k   13M:1216k   13M:1216k   13M:4800k   52M
1216k   13M:1216k   13M:1216k   13M:1216k   13M:4864k   52M^C

[root@backupcube ~]#

### -d=raid6 -m=raid1c4 on 4 drives ###

[root@backupcube ~]# mkfs.btrfs -f -d raid6 -m raid1c4 /dev/sda1 /dev/sdb1 /dev/sdd1 /dev/sde1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.

WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               9e4f6691-971d-4950-a032-d3c83f2fda26
Node size:          16384
Sector size:        4096
Filesystem size:    21.83TiB
Block group profiles:
   Data:             RAID6             2.00GiB
   Metadata:         RAID1C4           1.00GiB
   System:           RAID1C4           8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  4
Devices:
    ID        SIZE  PATH
     1     5.46TiB  /dev/sda1
     2     5.46TiB  /dev/sdb1
     3     5.46TiB  /dev/sdd1
     4     5.46TiB  /dev/sde1

[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4051697664 bytes (4.1 GB, 3.8 GiB) copied, 8 s, 506 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.58926 s, 500 MB/s

[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-

  read  writ: read  writ: read  writ: read  writ: read  writ
    0   238M:   0   238M:   0   238M:   0   238M:   0   953M
    0   250M:   0   250M:   0   250M:   0   250M:   0  1001M
    0   242M:   0   242M:   0   242M:   0   242M:   0   968M
    0   238M:   0   238M:   0   238M:   0   238M:   0   953M
    0   242M:   0   242M:   0   243M:   0   242M:   0   970M
    0   243M:   0   243M:   0   243M:   0   243M:   0   972M
    0   192M:   0   192M:   0   192M:   0   192M:   0   769M
    0     0 :   0     0 :   0     0 :   0     0 :   0  6144B^C

### -d=raid5 -m=raid1c3 on 3 drives ###

[root@backupcube ~]# mkfs.btrfs -f -d raid5 -m raid1c3 /dev/sda1 /dev/sdb1 /dev/sdd1 btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.

WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               7f06fbb0-9655-4f8d-af4a-2d06d8dc1d80
Node size:          16384
Sector size:        4096
Filesystem size:    16.37TiB
Block group profiles:
   Data:             RAID5             2.00GiB
   Metadata:         RAID1C3           1.00GiB
   System:           RAID1C3           8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, raid56, skinny-metadata, no-holes, raid1c34
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  3
Devices:
    ID        SIZE  PATH
     1     5.46TiB  /dev/sda1
     2     5.46TiB  /dev/sdb1
     3     5.46TiB  /dev/sdd1

[root@backupcube ~]# dd if=/dev/zero of=/mnt/sda1/tempfile bs=1M count=4096 conv=fdatasync,notrunc status=progress oflag=direct 4034920448 bytes (4.0 GB, 3.8 GiB) copied, 8 s, 504 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.61927 s, 498 MB/s
[root@backupcube ~]#

[root@backupcube ~]# dstat -d -D total,sda,sdb,sdd,sde --dsk/sda-----dsk/sdb-----dsk/sdd-----dsk/sde----dsk/total-

  read  writ: read  writ: read  writ: read  writ: read  writ
    0   231M:   0   231M:   0   231M:   0     0 :   0   694M
    0   244M:   0   244M:   0   244M:   0     0 :   0   733M
    0   250M:   0   250M:   0   250M:   0     0 :   0   751M
    0   238M:   0   238M:   0   238M:   0     0 :   0   715M
    0   239M:   0   239M:   0   239M:   0     0 :   0   718M
    0   243M:   0   243M:   0   243M:   0     0 :   0   729M
    0   243M:   0   243M:   0   243M:   0     0 :   0   728M
    0   117M:   0   117M:   0   117M:   0     0 :   0   351M
    0     0 :   0     0 :   0     0 :   0     0 :   0     0 ^C