Skip to content

RFC blk-cgroup: protect blkgs with blkcg_mutex

Yu Kuai cgroups linux-block linux-rt-devel lkml 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 RFC Latest6 patches 2 repliesIn review
  2. v2 RFC8 patches 12 repliesSuperseded
  3. v1 RFC17 patches 15 repliesSuperseded

Changes in v3 from the cover letter

- Base the series on the bio/blkcg preparatory series instead of moving blkg association into submit_bio(). Drop old patch 1 accordingly.
- Drop old patch 4 because the preparatory series replaces the per-blkcg radix tree with a request_queue rhashtable, so radix-tree preloading no longer exists.
- Drop old patch 7 because bios now retain their blkcg independently and BFQ can use the blkg already pinned by the bio.
- Add patch 1 to detach policy data under the existing spinlocks but call pd_free_fn() after dropping them. Keep this fix separate so the mutex conversion does not redesign generic policy teardown.
- In patch 2, shut down all per-group and top-level throttle timers before freeing throtl_data. Use timer_shutdown_sync() for final teardown so a concurrent callback cannot rearm a timer after shutdown.
- In patch 3, keep bio_blkg() on an RCU fast path when the target blkg already exists, taking blkcg_mutex only when hierarchy creation is needed.
- In patch 3, hold blkcg_mutex for the complete blkg_destroy_all() walk and remove the spinlock-era batch counter and cond_resched() restart.
- Adapt patches 4 and 5 to the rhashtable-based lookup and make blkg_lookup_create() return the closest existing blkg through an out parameter without acquiring a reference.
- Rewrite old patch 8 as patch 6. Keep allocation lazy in bio_blkg() instead of preparing every REQ_NOWAIT bio in submit_bio_noacct(). Use mutex_trylock() and GFP_ATOMIC when creation is possible; otherwise lookup and pin the closest existing blkg under RCU. Valid policy I/O therefore does not fail with BLK_STS_AGAIN or receive a NULL blkg.
- Do not add nowait-specific changes to blk-throttle, blk-iocost or blk-iolatency because bio_blkg() retains its non-NULL result for valid policy I/O.

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6blk-cgroup: call pd_free_fn() outside spinlocksblock/blk-cgroup.c+27 -100 replies
2/6blk-throttle: protect throttle state with td lockblock/blk-throttle.c+67 -200 replies
3/6blk-cgroup: protect blkgs with blkcg_mutexblock/bfq-cgroup.c block/blk-cgroup.c block/blk-cgroup.h +3 more+87 -1120 replies
4/6blk-cgroup: allocate blkgs in blkg_createblock/blk-cgroup.c+13 -352 replies
5/6blk-cgroup: share blkg creation between lookup and config prepblock/blk-cgroup.c+21 -560 replies
6/6blk-cgroup: make policy blkg creation nowait-safeblock/blk-cgroup.c+34 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [RFC PATCH v3 0/6] blk-cgroup: protect blkgs with blkcg_mutex

From: Yu Kuai <[email protected]>

This RFC moves queue-local blkg topology synchronization from q->queue_lock to q->blkcg_mutex. It is based on the preparatory series which stores a queue-independent blkcg in each bio and uses a request_queue-owned rhashtable for blkg lookup.

q->queue_lock also protects block core state and is taken from atomic contexts. Keeping blkg creation, destruction, configuration and policy lifecycle operations under that lock prevents those paths from using sleepable operations and couples blkcg policy state to unrelated queue locking.

Before switching locks, patch 1 moves pd_free_fn() calls outside spinlocks; iocost's free callback can wait for an hrtimer callback on PREEMPT_RT. Patch 2 gives throttle runtime state a private spinlock and shuts down its timers before freeing their state. Patch 3 then moves blkg topology and policy synchronization to blkcg_mutex while retaining an RCU fast path for existing-blkg I/O. Patches 4 and 5 move allocation into blkg_create() and share hierarchy creation with the configuration path. Patch 6 keeps blkg creation lazy for policy users and handles REQ_NOWAIT without sleeping, falling back to the closest existing blkg when creation cannot proceed.

A git branch is available at:
https://git.kernel.org/pub/scm/linux/kernel/git/yukuai/linux.git/log/?h=block-7.3-blkcg_mutex-v3

Changes since RFC v2:

- Base the series on the bio/blkcg preparatory series instead of moving blkg association into submit_bio(). Drop old patch 1 accordingly.
- Drop old patch 4 because the preparatory series replaces the per-blkcg radix tree with a request_queue rhashtable, so radix-tree preloading no longer exists.
- Drop old patch 7 because bios now retain their blkcg independently and BFQ can use the blkg already pinned by the bio.
- Add patch 1 to detach policy data under the existing spinlocks but call pd_free_fn() after dropping them. Keep this fix separate so the mutex conversion does not redesign generic policy teardown.
- In patch 2, shut down all per-group and top-level throttle timers before freeing throtl_data. Use timer_shutdown_sync() for final teardown so a concurrent callback cannot rearm a timer after shutdown.
- In patch 3, keep bio_blkg() on an RCU fast path when the target blkg already exists, taking blkcg_mutex only when hierarchy creation is needed.
- In patch 3, hold blkcg_mutex for the complete blkg_destroy_all() walk and remove the spinlock-era batch counter and cond_resched() restart.
- Adapt patches 4 and 5 to the rhashtable-based lookup and make blkg_lookup_create() return the closest existing blkg through an out parameter without acquiring a reference.
- Rewrite old patch 8 as patch 6. Keep allocation lazy in bio_blkg() instead of preparing every REQ_NOWAIT bio in submit_bio_noacct(). Use mutex_trylock() and GFP_ATOMIC when creation is possible; otherwise lookup and pin the closest existing blkg under RCU. Valid policy I/O therefore does not fail with BLK_STS_AGAIN or receive a NULL blkg.
- Do not add nowait-specific changes to blk-throttle, blk-iocost or blk-iolatency because bio_blkg() retains its non-NULL result for valid policy I/O.

Changes since RFC v1:

- Rework the series on top of "associate blkg in submit_bio instead of

    bio_set_dev".

- Drop the per-subsystem bio_set_dev() workarounds: NVMe multipath retarget (v1 patch 1), dm-thin (v1 patch 2), dm-snapshot (v1 patch 3), bcache (v1 patch 8), dm-bufio (v1 patch 9), dm-pcache (v1 patch 10) and DM NOWAIT remaps (v1 patch 12). They are no longer needed because bio_set_dev() no longer associates a blkg.
- Drop atomic bio allocation: bio_alloc_atomic() (v1 patch 5) and non-blocking bio allocation with a bdev (v1 patch 7). The nd_virtio/ocfs2 callers are handled separately.
- Drop the nowait-bio-allocation association helpers (v1 patches 6 and 11). Nowait is handled once at submission by failing the bio.
- Keep and adapt the blk-throttle private runtime lock (v1 patch 4), the blkcg_mutex conversion (v1 patch 14), radix preload removal (v1 patch 15), blkg_create() allocation (v1 patch 16), shared creation (v1 patch
17) and the BFQ locked-cgroup-update fix (v1 patch 13).

Previous versions:
RFC v2:

    https://lore.kernel.org/r/[email protected]
  RFC v1:
    https://lore.kernel.org/r/[email protected]

Yu Kuai (6):
blk-cgroup: call pd_free_fn() outside spinlocks
blk-throttle: protect throttle state with td lock
blk-cgroup: protect blkgs with blkcg_mutex
blk-cgroup: allocate blkgs in blkg_create
blk-cgroup: share blkg creation between lookup and config prep
blk-cgroup: make policy blkg creation nowait-safe

 block/bfq-cgroup.c    |  10 +-
 block/blk-cgroup.c    | 270 ++++++++++++++++++------------------------
 block/blk-cgroup.h    |  11 +-
 block/blk-iocost.c    |   8 +-
 block/blk-iolatency.c |   7 +-
 block/blk-throttle.c  |  93 +++++++++++----
 6 files changed, 207 insertions(+), 192 deletions(-)

base-commit: 81a5d989c04055860a66923e41151fd7faf4c295

Signature · 1 line
2.51.0