Skip to content

raid1 balancing methods

Anand Jain linux-btrfs oe-kbuild oe-kbuild-all 6 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest10 patches 13 repliesIn review
  2. v4 posting 2/29 patches 6 repliesSuperseded
  3. v4 posting 1/20 of 10 patches 0 repliesSuperseded
Earlier versions · 3
VersionPostedPatchesRepliesStatus
v310 patches10 repliesSuperseded
v23 patches11 repliesSuperseded
v13 patches6 repliesSuperseded

Changes in v5 from the cover letter

Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/10btrfs: initialize fs_devices->fs_info earlierfs/btrfs/disk-io.c fs/btrfs/volumes.c+1 -20 replies
2/10btrfs: simplify output formatting in btrfs_read_policy_showfs/btrfs/sysfs.c+10 -80 replies
3/10btrfs: add btrfs_read_policy_to_enum helper and refactor read policy storefs/btrfs/sysfs.c+22 -120 replies
4/10btrfs: handle value associated with raid1 balancing parameterfs/btrfs/sysfs.c+14 -23 replies
5/10btrfs: add read count tracking for filesystem statsfs/btrfs/bio.c fs/btrfs/disk-io.c fs/btrfs/volumes.c +1 more+24 -23 replies
6/10btrfs: introduce RAID1 round-robin read balancingfs/btrfs/sysfs.c fs/btrfs/volumes.c fs/btrfs/volumes.h+130 -11 reply
7/10btrfs: add RAID1 preferred read devicefs/btrfs/sysfs.c fs/btrfs/volumes.c fs/btrfs/volumes.h+58 -10 replies
8/10btrfs: expose experimental mode in module informationfs/btrfs/super.c+3 -00 replies
9/10btrfs: enable RAID1 balancing configuration via modprobe parameterfs/btrfs/super.c fs/btrfs/sysfs.c fs/btrfs/sysfs.h +1 more+52 -22 replies
10/10btrfs: modload to print RAID1 balancing statusfs/btrfs/super.c+10 -00 replies

Reviews trailers per patch

1 of 10 patches has a Reviewed-by or Acked-by.

PatchNaohiro Aota
1/10R-b
2/10
3/10
4/10
5/10
6/10
7/10
8/10
9/10
10/10

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH v5 00/10] raid1 balancing methods

v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()

    and `strncpy()`.
  . Rewrite the round-robin method based on read counts.
  . Fix the smatch indentation warning.
 - Change the default minimum contiguous device read size for round-robin
   from 256K to 192K, as the latter performs slightly better.
 - Introduce a framework to track filesystem read counts. (New patch)
 - Reran defrag performance numbers
      $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  | gain  |
|         | Real  | devid1 | devid2 | w-PID |
|---------|-------|--------|--------|-------|
| pid     | 11.14s| 3808   | 0      |  -    |
| rotation|       |        |        |       |
|   196608|  6.54s| 2532   | 1276   | 41.29%|
|   262144|  8.42s| 1907   | 1901   | 24.41%|
| devid:1 | 10.95s| 3807   | 0      | 1.70% |

v4:
Fixes based on review comments:
3/10: Use == 0 to check strlen(str); drop dynamic alloc for %param.
4/10: Add __maybe_unused for %value_str in btrfs_read_policy_to_enum().
Return int instead of enum btrfs_read_policy.
5/10: Fix change-log (: is part of optional [ ]).
Wrap btrfs_read_policy_name[] with ifdef for new methods.
Use IS_ALIGNED() for sector-size alignment check.
Roundup misaligned %value.
Use named constants: BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ, BTRFS_RAID1_MAX_MIRRORS. Mark %s1 and %s2 in btrfs_cmp_devid() as const.
Add comments to btrfs_read_rr();
Use loop-local %i. Add space around /.
Use >> for sector-size division.
Prefix %min_contiguous_read with rr.
7/10: Move experimental to the top of the feature list.
Use experiment=on. Skip printing when off.

v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-set min_contiguous_read to 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.

The logic of individual methods remains unchanged, and performance metrics are consistent with v2.

-----
v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

     $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
        --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
        --time_based --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
  --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (10):
btrfs: initialize fs_devices->fs_info earlier
btrfs: simplify output formatting in btrfs_read_policy_show
btrfs: add btrfs_read_policy_to_enum helper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: add read count tracking for filesystem stats
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
btrfs: expose experimental mode in module information
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing status

 fs/btrfs/bio.c     |   8 +++
 fs/btrfs/disk-io.c |   4 ++
 fs/btrfs/super.c   |  18 +++++
 fs/btrfs/sysfs.c   | 168 ++++++++++++++++++++++++++++++++++++++++-----
 fs/btrfs/sysfs.h   |   5 ++
 fs/btrfs/volumes.c | 115 ++++++++++++++++++++++++++++++-
 fs/btrfs/volumes.h |  22 +++++-
 7 files changed, 317 insertions(+), 23 deletions(-)
Signature · 1 line
2.47.0