Skip to content

md: avoid counter operation conflicts

tada keisuke linux-raid lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest8 of 11 patches 7 repliesIn review
  2. v17 of 11 patches 5 repliesSuperseded

Changes in v2 from the cover letter

- Add message of performance in cover.
- Fix a problem of error code initialization in patch 6.
- Fix a problem of sleeping during rcu_read_lock() in patch 9.
- Change base-commit from md-6.9 to md-6.10

8 of 11 patches are in this archive. The cover letter announces 11.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11md: add infra for active_aligned_reads changesdrivers/md/raid5.c drivers/md/raid5.h+26 -60 replies
2/11md: add infra for nr_pending changesdrivers/md/md-bitmap.c drivers/md/md.c drivers/md/md.h +4 more+80 -500 replies
3/11md: workaround for inconsistency of config state in takeoverdrivers/md/raid5.c+31 -80 replies
4/11md: minimize execution of zero check for nr_pendingdrivers/md/md.h+6 -80 replies
5/11md: match the type of variables to percpu_refdrivers/md/md.h drivers/md/raid1.c drivers/md/raid10.c+6 -60 replies
6/11md: avoid conflicts in active_aligned_reads operationsdrivers/md/raid5.c drivers/md/raid5.h+23 -170 replies
7/11md: change the type of nr_pending from atomic_t to percpu_refdrivers/md/md.c drivers/md/md.h+21 -100 replies
8/11md: add atomic mode switching in RAID 1/10drivers/md/md.h drivers/md/raid1.c drivers/md/raid10.c+25 -06 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 00/11] md: avoid counter operation conflicts

Changes in v2:
- Add message of performance in cover.
- Fix a problem of error code initialization in patch 6.
- Fix a problem of sleeping during rcu_read_lock() in patch 9.
- Change base-commit from md-6.9 to md-6.10

Currently, active_aligned_reads and nr_pending used as counters are atomic types. Therefore, when inc/dec in a multi-core results in conflicts and READ I/O becomes slow. To improve performance, use "percpu_ref" counters that can avoid conflicts and maintain consistency.

Switch modes of percpu_ref to achieve both consistency and conflict avoidance. During normal operations such as inc/dec, it operates as percpu mode. When consistency is required, it operates as atomic mode.
The operations that require consistency are as follows:
- Zero check for the counter
- All operations in RAID 1/10

Patches 1, 3, 6 change active_aligned_reads, and patches 2, 4, 5, 7 to 11 change nr_pending. nr_pending temporarily switch from percpu mode to atomic mode in patch 7. This is to reduce the amount of changes from patches 8 to 10.
Finally, nr_pending switch from atomic mode to percpu mode in patch 11.

We applied the patch to base-commit and used fio to compare IOPS.
CPU: AMD EPYC 7313P (3.0GHz, 16cores)
DISK: ramdisk x 3 (modprobe brd rd_nr=3)
RAID: level 5
fio config: bs=4k, rw=randread, iodepth=128, numjobs=16

without patch: 3.64 MIOPS
with patch : 3.84 MIOPS

Keisuke TADA (11):
add infra for active_aligned_reads changes
add infra for nr_pending changes
workaround for inconsistency of config state in takeover minimize execution of zero check for nr_pending match the type of variables to percpu_ref
avoid conflicts in active_aligned_reads operations change the type of nr_pending from atomic_t to percpu_ref add atomic mode switching in RAID 1/10
add atomic mode switching when removing disk
add atomic mode switching when I/O completion avoid conflicts in nr_pending operations

 drivers/md/md-bitmap.c   |  2 +-
 drivers/md/md.c          | 48 ++++++++++++++++++---
 drivers/md/md.h          | 62 +++++++++++++++++++++++----
 drivers/md/raid1.c       | 37 +++++++++++------
 drivers/md/raid10.c      | 60 ++++++++++++++++-----------
 drivers/md/raid5-cache.c |  4 +-
 drivers/md/raid5.c       | 90 +++++++++++++++++++++++++++-------------
 drivers/md/raid5.h       | 17 +++++++-
 8 files changed, 238 insertions(+), 82 deletions(-)

base-commit: 9d1110f99c253ccef82e480bfe9f38a12eb797a7 --
2.34.1