How to format RAID1 correctly

So what is ultimately the correct way to format a RAID1?

The information provided in the FAQ and on several web sites is not really useful regarding RAID1.

According to the FAQ (entry 35):
The correct options to format a RAID1 (2 disks) with 64k chunk size is:
mkfs.xfs -d su=64k -d sw=1 /dev/mapper/data

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

But it also states that it would be automatically detected and used correctly, yet
mkfs.xfs /dev/mapper/data
yields a different result:

[root@atvie01s ~]# mkfs.xfs -f /dev/mapper/data

meta-data=/dev/mapper/data isize=256 agcount=4,
agsize=244173876 blks

         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=976695504, imaxpct=5
         =                       sunit=0      swidth=0 blks

Formel 1:

[root@atvie01s ~]# mkfs.xfs -f -d su=64k -d sw=1 /dev/mapper/data

meta-data=/dev/mapper/data isize=256 agcount=32,
agsize=30521728 blks

         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=976695296, imaxpct=5
         =                       sunit=16     swidth=16 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=476902, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Another inconsistency is that RAID1 doesn't use striping, so the chunk size should be irrelevant in the first place.

So what is ultimately the correct way to format a RAID1?

but! your raid doesn't have a 64k stripe, so that doesn't apply.

no geometry because md0 raid1 doesn't export any stripe geometry.

Quoted 1 line · Eric Sandeen

but! your raid doesn't have a 64k stripe, so that doesn't apply.

Yep, that's true, but see below.

Quoted 1 line

no geometry because md0 raid1 doesn't export any stripe geometry.

[root@atvie01s ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0] sdd1[1]

      3906784064 blocks super 1.2 [2/2] [UU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

So for some reason it shows a 64k chunk size even for RAID1.

That was what got me confused.

blkid # blockdev --getiomin --getioopt /dev/md0

(which here returns:
512
0
here)

will show you what your raid's queue is actually reporting, and what mkfs.xfs will pick up.

Quoted 1 line · Eric Sandeen

but! your raid doesn't have a 64k stripe, so that doesn't apply.

Yep, that's true, but see below.

Quoted 1 line

no geometry because md0 raid1 doesn't export any stripe geometry.

[root@atvie01s ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0] sdd1[1]

      3906784064 blocks super 1.2 [2/2] [UU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

So for some reason it shows a 64k chunk size even for RAID1.

That was what got me confused.

It confuses many people who are new to md RAID1. The above is the *bitmap* chunk size, not the array chunk size. There is no array chunk size for RAID1 as there is no striping. You must have striping to have chunks. With md RAID1 every 4KB page write is simply mirrored to each physical disk.

Quoted 5 lines · stan hoeppner

It confuses many people who are new to md RAID1. The above is the *bitmap* chunk size, not the array chunk size. There is no array chunk size for RAID1 as there is no striping. You must have striping to have chunks. With md RAID1 every 4KB page write is simply mirrored to each physical disk.

Thanks for the info. I'm used to big ass storage subsystems, but new to SW RAID. I seems I have some catching up to do.

If you create any striped arrays, especially parity arrays, with md make sure to manually specify chunk size and match it to your workload. The current default is 512KB. This is too large for a great many workloads, specifically those that are metadata heavy or manipulate many small files. 512KB wastes space and with parity arrays causes RMW, hammering throughput and increasing latency.

Quoted 6 lines · Stan Hoeppner

If you create any striped arrays, especially parity arrays, with md make sure to manually specify chunk size and match it to your workload. The current default is 512KB. This is too large for a great many workloads, specifically those that are metadata heavy or manipulate many small files. 512KB wastes space and with parity arrays causes RMW, hammering throughput and increasing latency.

Thanks again for the valueable information.

I used to work with databases on storage subsystems, so placing GBs of database containers for tableapaces on arrays with a larger stripe size was actually beneficial.
For log files and other data I usually used different cache settings and strip sizes.

So how does this work with SW RAID?

Does the XFS chunk size equal the amount of data touched by a single r/w operation?

Quoted 6 lines · Stan Hoeppner

If you create any striped arrays, especially parity arrays, with md make sure to manually specify chunk size and match it to your workload. The current default is 512KB. This is too large for a great many workloads, specifically those that are metadata heavy or manipulate many small files. 512KB wastes space and with parity arrays causes RMW, hammering throughput and increasing latency.

Thanks again for the valueable information.

I used to work with databases on storage subsystems, so placing GBs of database containers for tableapaces on arrays with a larger stripe size was actually beneficial.
For log files and other data I usually used different cache settings and strip sizes.

So how does this work with SW RAID?

Does the XFS chunk size equal the amount of data touched by a single r/w operation?

I'm asking because data is usually written in page/extent sizes for databases. Even if I have a container with 50GB, I might only have to read/write a 4k page.