blk-cgroup: remove queue_lock nesting from blkcg paths
Subsystems (6)
cover blk-cgroup: remove queue_lock nesting from blkcg paths
From: Yu Kuai <[email protected]>
Hi,
This series is the follow-up blk-cgroup locking cleanup on top of the earlier blkg-list protection fixes, and prepares blk-cgroup to stop using q->queue_lock as the global blkg lifetime/iteration lock.
The current queue_lock based protection is hard to maintain because queue_lock is used from hardirq and softirq completion paths, while some blkcg cgroup file paths also need to iterate blkgs, print policy data, or create blkgs from RCU-protected contexts. This series first tightens the blkcg-side lifetime rules:
- blkcg_print_stat() iterates blkgs under blkcg->lock with IRQs disabled.
- policy data freeing is delayed past an RCU grace period.
- blkcg_print_blkgs(), blkg lookup/create, bio association, page-IO association, blkg destruction, and BFQ initialization stop nesting queue_lock under RCU or blkcg->lock.
Using blkcg->lock and RCU for blkcg-owned lists/data keeps the lock order local to blk-cgroup and avoids extending queue_lock into cgroup file iteration paths. It also makes the subsequent conversion to q->blkcg_mutex possible without carrying forward queue_lock's interrupt-context constraints.
Yu Kuai (8):
blk-cgroup: protect iterating blkgs with blkcg->lock in
blkcg_print_stat()
blk-cgroup: delay freeing policy data after rcu grace period
blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()
blk-cgroup: don't nest queue_lock under rcu in blkg_lookup_create()
blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()
blk-cgroup: don't nest queue_lock under blkcg->lock in
blkcg_destroy_blkgs()
mm/page_io: don't nest queue_lock under rcu in
bio_associate_blkg_from_page()
block, bfq: don't grab queue_lock to initialize bfq
block/bfq-cgroup.c | 17 ++++- block/bfq-iosched.c | 5 -- block/blk-cgroup-rwstat.c | 15 ++-- block/blk-cgroup.c | 151 ++++++++++++++++++++++---------------- block/blk-cgroup.h | 8 +- block/blk-iocost.c | 22 ++++-- block/blk-iolatency.c | 10 ++- block/blk-throttle.c | 13 +++- mm/page_io.c | 7 +- 9 files changed, 158 insertions(+), 90 deletions(-)
base-commit: b23df513de562739af61fa61ba80ef5e8059a636
Signature · 1 line
2.51.0
PATCH 1/8 blk-cgroup: protect iterating blkgs with blkcg->lock in blkcg_print_stat()
From: Yu Kuai <[email protected]>
blkcg_print_one_stat() will be called for each blkg:
- access blkg->iostat, which is freed from rcu callback blkg_free_workfn();
- access policy data from pd_stat_fn(), which is freed from pd_free_fn(), while pd_free_fn() can be called by removing blkcg or deactivating policy;
Take blkcg->lock while iterating so the blkgs stay online and both blkg->iostat and policy data for activated policies stay valid. Use irq-safe locking because blkcg->lock can be nested under q->queue_lock, which is used from IRQ completion paths.
Prepare to convert protecting blkgs from request_queue with mutex.
--- block/blk-cgroup.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
block/blk-cgroup.c +3 -6
@@ -1241,17 +1241,14 @@ static int blkcg_print_stat(struct seq_file *sf, void *v)if (!seq_css (sf)->parent)blkcg_fill_root_iostats ();else css_rstat_flush (&blkcg->css); -rcu_read_lock (); -hlist_for_each_entry_rcu (blkg, &blkcg->blkg_list, blkcg_node) { -spin_lock_irq (&blkg->q->queue_lock); +guard (spinlock_irq)(&blkcg->lock); +hlist_for_each_entry (blkg, &blkcg->blkg_list, blkcg_node)blkcg_print_one_stat (blkg, sf); -spin_unlock_irq (&blkg->q->queue_lock); - } -rcu_read_unlock (); +return 0 ; }static struct cftype blkcg_files[] = { {
Signature · 1 line
2.51.0
PATCH 2/8 blk-cgroup: delay freeing policy data after rcu grace period
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Currently blkcg_print_blkgs() must hold RCU to iterate blkgs from a blkcg, and prfill() must hold queue_lock to prevent policy data from being freed by policy deactivation. As a consequence, queue_lock has to be nested under RCU from blkcg_print_blkgs().
Delay freeing policy data until after an RCU grace period so prfill() can be protected by RCU alone.
--- block/bfq-cgroup.c | 9 ++++++++- block/blk-cgroup.h | 2 ++ block/blk-iocost.c | 14 ++++++++++++-- block/blk-iolatency.c | 10 +++++++++- block/blk-throttle.c | 13 +++++++++++-- 5 files changed, 42 insertions(+), 6 deletions(-)
block/bfq-cgroup.c +8 -1
@@ -548,17 +548,24 @@ static void bfq_pd_init(struct blkg_policy_data *pd) bfqg->active_entities =0 ; bfqg->num_queues_with_pending_reqs =0 ; bfqg->rq_pos_tree =RB_ROOT ; } -static void bfq_pd_free (struct blkg_policy_data *pd) +static void bfqg_release (struct rcu_head *rcu) { +struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head);struct bfq_group *bfqg =pd_to_bfqg (pd);bfqg_put (bfqg); } +static void bfq_pd_free (struct blkg_policy_data *pd) +{ +call_rcu (&pd->rcu_head, bfqg_release); +} +static void bfq_pd_reset_stats (struct blkg_policy_data *pd) {struct bfq_group *bfqg =pd_to_bfqg (pd);bfqg_stats_reset (&bfqg->stats);
block/blk-cgroup.h +2 -0
@@ -138,10 +138,12 @@ static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css)struct blkg_policy_data {/* the blkg and policy id this per-policy data belongs to */ struct blkcg_gq *blkg;int plid;bool online; + +struct rcu_head rcu_head; };/* * Policies that need to keep per-blkcg data which is independent from any * request_queue associated to it should implement cpd_alloc/free_fn()
block/blk-iocost.c +12 -2
@@ -3024,10 +3024,20 @@ static void ioc_pd_init(struct blkg_policy_data *pd)spin_lock_irqsave (&ioc->lock, flags);weight_updated (iocg, &now);spin_unlock_irqrestore (&ioc->lock, flags); } +static void iocg_release (struct rcu_head *rcu) +{ +struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head); +struct ioc_gq *iocg =pd_to_iocg (pd); + +free_percpu (iocg->pcpu_stat); +kfree (iocg); +} +static void ioc_pd_free (struct blkg_policy_data *pd) {struct ioc_gq *iocg =pd_to_iocg (pd);struct ioc *ioc = iocg->ioc;unsigned long flags;
@@ -3048,12 +3058,12 @@ static void ioc_pd_free(struct blkg_policy_data *pd)spin_unlock_irqrestore (&ioc->lock, flags);hrtimer_cancel (&iocg->waitq_timer); } -free_percpu (iocg->pcpu_stat); -kfree (iocg); + +call_rcu (&pd->rcu_head, iocg_release); }static void ioc_pd_stat (struct blkg_policy_data *pd,struct seq_file *s) {struct ioc_gq *iocg =pd_to_iocg (pd);
block/blk-iolatency.c +9 -1
@@ -1026,17 +1026,25 @@ static void iolatency_pd_offline(struct blkg_policy_data *pd)iolatency_set_min_lat_nsec (blkg,0 );iolatency_clear_scaling (blkg); } -static void iolatency_pd_free (struct blkg_policy_data *pd) +static void iolat_release (struct rcu_head *rcu) { +struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head);struct iolatency_grp *iolat =pd_to_lat (pd); +free_percpu (iolat->stats);kfree (iolat); } +static void iolatency_pd_free (struct blkg_policy_data *pd) +{ +call_rcu (&pd->rcu_head, iolat_release); +} +static struct cftype iolatency_files[] = { { .name ="latency" , .flags =CFTYPE_NOT_ON_ROOT , .seq_show = iolatency_print_limit,
block/blk-throttle.c +11 -2
@@ -351,20 +351,29 @@ static void throtl_pd_online(struct blkg_policy_data *pd)* Update has_rules[] after a new group is brought online. */ tg_update_has_rules (tg); } -static void throtl_pd_free (struct blkg_policy_data *pd) +static void tg_release (struct rcu_head *rcu) { +struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head);struct throtl_grp *tg =pd_to_tg (pd); -timer_delete_sync (&tg->service_queue.pending_timer);blkg_rwstat_exit (&tg->stat_bytes);blkg_rwstat_exit (&tg->stat_ios);kfree (tg); } +static void throtl_pd_free (struct blkg_policy_data *pd) +{ +struct throtl_grp *tg =pd_to_tg (pd); + +timer_delete_sync (&tg->service_queue.pending_timer); +call_rcu (&pd->rcu_head, tg_release); +} +static struct throtl_grp *throtl_rb_first (struct throtl_service_queue *parent_sq) {struct rb_node *n;
Signature · 1 line
2.51.0
PATCH 3/8 blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
With previous modification to delay freeing policy data after an RCU grace period, prfill() can run under RCU instead of taking queue_lock. However, policy teardown can still clear blkg->pd[plid] after blkcg_print_blkgs() observes the policy enabled bit.
Load policy data once with READ_ONCE() and skip the blkg if teardown already cleared it. Do the same in recursive stat walks for descendant blkgs. Remove the stale BFQ debug queue_lock assertion because blkcg_print_blkgs() no longer calls prfill() with queue_lock held. This also lets ioc_qos_prfill() and ioc_cost_model_prfill() use IRQ-safe ioc->lock locking without re-enabling IRQs while queue_lock is still held.
--- block/bfq-cgroup.c | 8 +++++--- block/blk-cgroup-rwstat.c | 15 +++++++++------ block/blk-cgroup.c | 22 +++++++++++++--------- block/blk-cgroup.h | 6 +++--- block/blk-iocost.c | 8 ++++---- 5 files changed, 34 insertions(+), 25 deletions(-)
block/bfq-cgroup.c +5 -3
@@ -1146,20 +1146,22 @@ static u64 bfqg_prfill_stat_recursive(struct seq_file *sf,struct blkcg_gq *blkg =pd_to_blkg (pd);struct blkcg_gq *pos_blkg;struct cgroup_subsys_state *pos_css;u64 sum =0 ; -lockdep_assert_held (&blkg->q->queue_lock); -rcu_read_lock ();blkg_for_each_descendant_pre (pos_blkg, pos_css, blkg) { +struct blkg_policy_data *pd;struct bfq_stat *stat;if (!pos_blkg->online)continue ; - stat = (void *)blkg_to_pd (pos_blkg, &blkcg_policy_bfq) + off; + pd =blkg_to_pd (pos_blkg, &blkcg_policy_bfq); +if (!pd) +continue ; + stat = (void *)pd + off; sum +=bfq_stat_read (stat) +atomic64_read (&stat->aux_cnt); }rcu_read_unlock ();return __blkg_prfill_u64 (sf, pd, sum);
block/blk-cgroup-rwstat.c +9 -6
@@ -99,26 +99,29 @@ void blkg_rwstat_recursive_sum(struct blkcg_gq *blkg, struct blkcg_policy *pol, {struct blkcg_gq *pos_blkg;struct cgroup_subsys_state *pos_css;unsigned int i; -lockdep_assert_held (&blkg->q->queue_lock); +WARN_ON_ONCE (!rcu_read_lock_held ());memset (sum,0 ,sizeof (*sum)); -rcu_read_lock ();blkg_for_each_descendant_pre (pos_blkg, pos_css, blkg) {struct blkg_rwstat *rwstat;if (!pos_blkg->online)continue ; -if (pol) - rwstat = (void *)blkg_to_pd (pos_blkg, pol) + off; -else +if (pol) { +struct blkg_policy_data *pd =blkg_to_pd (pos_blkg, pol); + +if (!pd) +continue ; + rwstat = (void *)pd + off; + }else { rwstat = (void *)pos_blkg + off; + }for (i =0 ; i <BLKG_RWSTAT_NR ; i++) sum->cnt[i] +=blkg_rwstat_read_counter (rwstat, i); } -rcu_read_unlock (); }EXPORT_SYMBOL_GPL (blkg_rwstat_recursive_sum);
block/blk-cgroup.c +13 -9
@@ -699,13 +699,13 @@ const char *blkg_dev_name(struct blkcg_gq *blkg)* @data: data to be passed to @prfill * @show_total: to print out sum of prfill return values or not * * This function invokes @prfill on each blkg of @blkcg if pd for the * policy specified by @pol exists. @prfill is invoked with @sf, the -* policy data and @data and the matching queue lock held. If @show_total -* is %true, the sum of the return values from @prfill is printed with -* "Total" label at the end. +* policy data and @data under RCU read lock. If @show_total is %true, the +* sum of the return values from @prfill is printed with "Total" label at the +* end. * * This is to be used to construct print functions for * cftype->read_seq_string method. */ void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg,
@@ -717,14 +717,18 @@ void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg,struct blkcg_gq *blkg;u64 total =0 ;rcu_read_lock ();hlist_for_each_entry_rcu (blkg, &blkcg->blkg_list, blkcg_node) { -spin_lock_irq (&blkg->q->queue_lock); -if (blkcg_policy_enabled (blkg->q, pol)) - total +=prfill (sf, blkg->pd[pol->plid], data); -spin_unlock_irq (&blkg->q->queue_lock); +struct blkg_policy_data *pd; + +if (!blkcg_policy_enabled (blkg->q, pol)) +continue ; + + pd =blkg_to_pd (blkg, pol); +if (pd) + total +=prfill (sf, pd, data); }rcu_read_unlock ();if (show_total)seq_printf (sf,"Total %llu\n" , (unsigned long long )total);
@@ -1591,11 +1595,11 @@ static void blkcg_policy_teardown_pds(struct request_queue *q,if (pd) {if (pd->online && pol->pd_offline_fn) pol->pd_offline_fn (pd); pd->online = false; pol->pd_free_fn (pd); - blkg->pd[pol->plid] =NULL ; +WRITE_ONCE (blkg->pd[pol->plid],NULL ); }spin_unlock (&blkcg->lock); } }
@@ -1683,11 +1687,11 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)spin_lock (&blkg->blkcg->lock); pd->blkg = blkg; pd->plid = pol->plid; - blkg->pd[pol->plid] = pd; +WRITE_ONCE (blkg->pd[pol->plid], pd);if (pol->pd_init_fn) pol->pd_init_fn (pd);if (pol->pd_online_fn)
block/blk-cgroup.h +3 -3
@@ -279,13 +279,13 @@ static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg,* @pol: policy of interest * * Return pointer to private data associated with the @blkg-@pol pair. */ static inline struct blkg_policy_data *blkg_to_pd (struct blkcg_gq *blkg, -struct blkcg_policy *pol) +const struct blkcg_policy *pol) { -return blkg ? blkg->pd[pol->plid] :NULL ; +return blkg ?READ_ONCE (blkg->pd[pol->plid]) :NULL ; }static inline struct blkcg_policy_data *blkcg_to_cpd (struct blkcg *blkcg,struct blkcg_policy *pol) {
@@ -488,11 +488,11 @@ static inline int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol) { return 0; }static inline void blkcg_deactivate_policy (struct gendisk *disk,const struct blkcg_policy *pol) { }static inline struct blkg_policy_data *blkg_to_pd (struct blkcg_gq *blkg, -struct blkcg_policy *pol) {return NULL ; } +const struct blkcg_policy *pol) {return NULL ; }static inline struct blkcg_gq *pd_to_blkg (struct blkg_policy_data *pd) {return NULL ; }static inline void blkg_get (struct blkcg_gq *blkg) { }static inline void blkg_put (struct blkcg_gq *blkg) { }static inline void blk_cgroup_bio_start (struct bio *bio) { }static inline bool blk_cgroup_mergeable (struct request *rq,struct bio *bio) {return true; }
block/blk-iocost.c +4 -4
@@ -3188,11 +3188,11 @@ static u64 ioc_qos_prfill(struct seq_file *sf, struct blkg_policy_data *pd,struct ioc *ioc =pd_to_iocg (pd)->ioc;if (!dname)return 0 ; -spin_lock (&ioc->lock); +spin_lock_irq (&ioc->lock); seq_printf(sf, "%s enable=%d ctrl=%s rpct=%u.%02u rlat=%u wpct=%u.%02u wlat=%u min=%u.%02u max=%u.%02u\n", dname, ioc->enabled, ioc->user_qos_params ? "user" : "auto", ioc->params.qos[QOS_RPPM] / 10000, ioc->params.qos[QOS_RPPM] % 10000 / 100, ioc->params.qos[QOS_RLAT],
@@ -3201,11 +3201,11 @@ static u64 ioc_qos_prfill(struct seq_file *sf, struct blkg_policy_data *pd, ioc->params.qos[QOS_WLAT ], ioc->params.qos[QOS_MIN ] /10000 , ioc->params.qos[QOS_MIN ] %10000 /100 , ioc->params.qos[QOS_MAX ] /10000 , ioc->params.qos[QOS_MAX] % 10000 / 100); -spin_unlock (&ioc->lock); +spin_unlock_irq (&ioc->lock);return 0 ; }static int ioc_qos_show (struct seq_file *sf,void *v) {
@@ -3386,18 +3386,18 @@ static u64 ioc_cost_model_prfill(struct seq_file *sf,u64 *u = ioc->params.i_lcoefs;if (!dname)return 0 ; -spin_lock (&ioc->lock); +spin_lock_irq (&ioc->lock);seq_printf (sf,"%s ctrl=%s model=linear " "rbps=%llu rseqiops=%llu rrandiops=%llu " "wbps=%llu wseqiops=%llu wrandiops=%llu\n" , dname, ioc->user_cost_model ?"user" :"auto" , u[I_LCOEF_RBPS ], u[I_LCOEF_RSEQIOPS ], u[I_LCOEF_RRANDIOPS ], u[I_LCOEF_WBPS ], u[I_LCOEF_WSEQIOPS ], u[I_LCOEF_WRANDIOPS ]); -spin_unlock (&ioc->lock); +spin_unlock_irq (&ioc->lock);return 0 ; }static int ioc_cost_model_show (struct seq_file *sf,void *v) {
Signature · 1 line
2.51.0
PATCH 4/8 blk-cgroup: don't nest queue_lock under rcu in blkg_lookup_create()
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Change this in two steps:
1) hold rcu lock and do blkg_lookup() from fast path;
2) hold queue_lock directly from slow path, and don't nest it under rcu lock;
Prepare to convert protecting blkcg with blkcg_mutex instead of queue_lock.
--- block/blk-cgroup.c | 57 +++++++++++++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 21 deletions(-)
block/blk-cgroup.c +36 -21
@@ -466,26 +466,21 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,static struct blkcg_gq *blkg_lookup_create (struct blkcg *blkcg,struct gendisk *disk) {struct request_queue *q = disk->queue;struct blkcg_gq *blkg; -unsigned long flags; - -WARN_ON_ONCE (!rcu_read_lock_held ()); - blkg =blkg_lookup (blkcg, q); -if (blkg) -return blkg; - -spin_lock_irqsave (&q->queue_lock, flags); +rcu_read_lock (); blkg =blkg_lookup (blkcg, q);if (blkg) {if (blkcg != &blkcg_root && blkg !=rcu_dereference (blkcg->blkg_hint))rcu_assign_pointer (blkcg->blkg_hint, blkg); - goto found; +rcu_read_unlock (); +return blkg; } +rcu_read_unlock ();/* * Create blkgs walking down from blkcg_root to @blkcg, so that all * non-root blkgs have access to their parents. Returns the closest * blkg to the intended blkg should blkg_create() fail.
@@ -513,12 +508,10 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg, }if (pos == blkcg)break ; } -found: -spin_unlock_irqrestore (&q->queue_lock, flags);return blkg; }static void blkg_destroy (struct blkcg_gq *blkg) {
@@ -2098,10 +2091,22 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta)return ;blkcg_scale_delay (blkg, now);atomic64_add (delta, &blkg->delay_nsec); } +static inline struct blkcg_gq *blkg_lookup_tryget (struct blkcg_gq *blkg) +{ +retry: +if (blkg_tryget (blkg)) +return blkg; + + blkg = blkg->parent; +if (blkg) + goto retry; + +return NULL ; +}/** * blkg_tryget_closest - try and get a blkg ref on the closet blkg * @bio: target bio * @css: target css *
@@ -2110,24 +2115,34 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta)* up taking a reference on or %NULL if no reference was taken. */ static inline struct blkcg_gq *blkg_tryget_closest (struct bio *bio,struct cgroup_subsys_state *css) { -struct blkcg_gq *blkg, *ret_blkg =NULL ; +struct request_queue *q = bio->bi_bdev->bd_queue; +struct blkcg *blkcg =css_to_blkcg (css); +struct blkcg_gq *blkg;rcu_read_lock (); - blkg =blkg_lookup_create (css_to_blkcg (css), bio->bi_bdev->bd_disk); -while (blkg) { -if (blkg_tryget (blkg)) { - ret_blkg = blkg; -break ; - } - blkg = blkg->parent; - } + blkg =blkg_lookup (blkcg, q); +if (likely (blkg)) + blkg =blkg_lookup_tryget (blkg);rcu_read_unlock (); -return ret_blkg; +if (blkg) +return blkg; + +/* +* Fast path failed, we're probably issuing IO in this cgroup the first +* time, hold lock to create new blkg. +*/ +spin_lock_irq (&q->queue_lock); + blkg =blkg_lookup_create (blkcg, bio->bi_bdev->bd_disk); +if (blkg) + blkg =blkg_lookup_tryget (blkg); +spin_unlock_irq (&q->queue_lock); + +return blkg; }/** * bio_associate_blkg_from_css - associate a bio with a specified css * @bio: target bio
Signature · 1 line
2.51.0
PATCH 5/8 blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
If a bio is already associated with a blkg, the blkcg is already pinned until the bio is done, so there is no need for RCU protection. Otherwise, protect blkcg_css() with RCU independently. Prepare to protect blkcg with blkcg_mutex instead of queue_lock.
--- block/blk-cgroup.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
block/blk-cgroup.c +11 -7
@@ -2186,20 +2186,24 @@ void bio_associate_blkg(struct bio *bio)struct cgroup_subsys_state *css;if (blk_op_is_passthrough (bio->bi_opf))return ; -rcu_read_lock (); - -if (bio->bi_blkg) +if (bio->bi_blkg) { css =bio_blkcg_css (bio); -else +bio_associate_blkg_from_css (bio, css); + }else { +rcu_read_lock (); css =blkcg_css (); +if (!css_tryget_online (css)) + css =NULL ; +rcu_read_unlock (); -bio_associate_blkg_from_css (bio, css); - -rcu_read_unlock (); +bio_associate_blkg_from_css (bio, css); +if (css) +css_put (css); + } }EXPORT_SYMBOL_GPL (bio_associate_blkg);/** * bio_clone_blkg_association - clone blkg association from src to dst bio
Signature · 1 line
2.51.0
PATCH 6/8 blk-cgroup: don't nest queue_lock under blkcg->lock in blkcg_destroy_blkgs()
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
The correct lock order is q->queue_lock before blkcg->lock, and in order to prevent deadlock from blkcg_destroy_blkgs(), trylock is used for q->queue_lock while blkcg->lock is already held, this is hacky.
Refactor blkcg_destroy_blkgs() to hold blkcg->lock only long enough to get the first blkg and then release it. Then take q->queue_lock and blkcg->lock in the correct order to destroy the blkg. This is a very cold path, so the extra lock/unlock cycles are acceptable.
Also prepare to convert protecting blkcg with blkcg_mutex instead of queue_lock.
--- block/blk-cgroup.c | 45 ++++++++++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 19 deletions(-)
block/blk-cgroup.c +26 -19
@@ -1289,10 +1289,25 @@ struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css)* * 3. Once the blkcg ref count goes to zero, blkcg_css_free() is called. * This finally frees the blkcg. */ +static struct blkcg_gq *blkcg_get_first_blkg (struct blkcg *blkcg) +{ +struct blkcg_gq *blkg =NULL ; + +spin_lock_irq (&blkcg->lock); +if (!hlist_empty (&blkcg->blkg_list)) { + blkg = hlist_entry(blkcg->blkg_list.first, struct blkcg_gq, + blkcg_node); +blkg_get (blkg); + } +spin_unlock_irq (&blkcg->lock); + +return blkg; +} +/** * blkcg_destroy_blkgs - responsible for shooting down blkgs * @blkcg: blkcg of interest * * blkgs should be removed while holding both q and blkcg locks. As blkcg lock
@@ -1302,36 +1317,28 @@ struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css)* * This is the blkcg counterpart of ioc_release_fn(). */ static void blkcg_destroy_blkgs (struct blkcg *blkcg) { -might_sleep (); +struct blkcg_gq *blkg; -spin_lock_irq (&blkcg->lock); +might_sleep (); -while (!hlist_empty (&blkcg->blkg_list)) { -struct blkcg_gq *blkg =hlist_entry (blkcg->blkg_list.first, - struct blkcg_gq, blkcg_node); +while ((blkg =blkcg_get_first_blkg (blkcg))) {struct request_queue *q = blkg->q; -if (need_resched () || !spin_trylock (&q->queue_lock)) { -/* -* Given that the system can accumulate a huge number -* of blkgs in pathological cases, check to see if we -* need to rescheduling to avoid softlockup. -*/ -spin_unlock_irq (&blkcg->lock); -cond_resched (); -spin_lock_irq (&blkcg->lock); -continue ; - } +spin_lock_irq (&q->queue_lock); +spin_lock (&blkcg->lock);blkg_destroy (blkg); -spin_unlock (&q->queue_lock); - } -spin_unlock_irq (&blkcg->lock); +spin_unlock (&blkcg->lock); +spin_unlock_irq (&q->queue_lock); + +blkg_put (blkg); +cond_resched (); + } }/** * blkcg_pin_online - pin online state * @blkcg_css: blkcg of interest
Signature · 1 line
2.51.0
PATCH 7/8 mm/page_io: don't nest queue_lock under rcu in bio_associate_blkg_from_page()
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
Take a css reference under RCU, drop RCU, and then associate the bio with the blkg. This avoids nesting queue_lock under RCU and prepares to protect blkcg with blkcg_mutex instead of queue_lock.
Use css_tryget() instead of css_tryget_online() so swap writeback for pages charged to a dying memcg still passes the dying css to bio_associate_blkg_from_css(). That preserves the existing closest-live ancestor fallback instead of charging those bios to the root blkg.
--- mm/page_io.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/page_io.c +6 -1
@@ -315,12 +315,17 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)return ;rcu_read_lock (); memcg =folio_memcg (folio); css =cgroup_e_css (memcg->css.cgroup, &io_cgrp_subsys); -bio_associate_blkg_from_css (bio, css); +if (!css || !css_tryget (css)) + css =NULL ;rcu_read_unlock (); + +bio_associate_blkg_from_css (bio, css); +if (css) + css_put(css); }#else #define bio_associate_blkg_from_page (bio, folio) do { } while (0)#endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */
Signature · 1 line
2.51.0
PATCH 8/8 block, bfq: don't grab queue_lock to initialize bfq
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
The request_queue is frozen and quiesced while the elevator init_sched() method runs, so queue_lock is not needed for BFQ cgroup initialization.
--- block/bfq-iosched.c | 5 ----- 1 file changed, 5 deletions(-)
block/bfq-iosched.c +0 -5
@@ -7207,14 +7207,11 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq) bfqd =kzalloc_node (sizeof (*bfqd),GFP_KERNEL , q->node);if (!bfqd)return -ENOMEM ; eq->elevator_data = bfqd; - -spin_lock_irq (&q->queue_lock); q->elevator = eq; -spin_unlock_irq (&q->queue_lock);/* * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues. * Grab a permanent reference to it, so that the normal code flow * will not attempt to free it.
@@ -7243,11 +7240,10 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq) bfqd->num_actuators =1 ;/* * If the disk supports multiple actuators, copy independent * access ranges from the request queue structure. */ -spin_lock_irq (&q->queue_lock);if (ia_ranges) {/* * Check if the disk ia_ranges size exceeds the current bfq * actuator limit. */
@@ -7269,11 +7265,10 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq)/* Otherwise use single-actuator dev info */ if (bfqd->num_actuators ==1 ) { bfqd->sector[0 ] =0 ; bfqd->nr_sectors[0 ] =get_capacity (q->disk); } -spin_unlock_irq (&q->queue_lock);INIT_LIST_HEAD (&bfqd->dispatch);hrtimer_setup (&bfqd->idle_slice_timer, bfq_idle_slice_timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL );
Signature · 1 line
2.51.0
yu kuai
maintainerFriendly ping ...
This set can still be applied cleanly for block-7.2 branch.
Quoted 52 lines · Yu Kuai
From: Yu Kuai <[email protected]>
Hi,
This series is the follow-up blk-cgroup locking cleanup on top of the earlier blkg-list protection fixes, and prepares blk-cgroup to stop using
q->queue_lockas the global blkg lifetime/iteration lock.The current
queue_lockbased protection is hard to maintain becausequeue_lockis used from hardirq and softirq completion paths, while some blkcg cgroup file paths also need to iterate blkgs, print policy data, or create blkgs from RCU-protected contexts. This series first tightens the blkcg-side lifetime rules:-
blkcg_print_stat()iterates blkgs underblkcg->lockwith IRQs disabled.
- policy data freeing is delayed past an RCU grace period.
-blkcg_print_blkgs(), blkg lookup/create, bio association, page-IO association, blkg destruction, and BFQ initialization stop nestingqueue_lockunder RCU orblkcg->lock.Using
blkcg->lockand RCU for blkcg-owned lists/data keeps the lock order local to blk-cgroup and avoids extendingqueue_lockinto cgroup file iteration paths. It also makes the subsequent conversion toq->blkcg_mutexpossible without carrying forward queue_lock's interrupt-context constraints.Yu Kuai (8):
blk-cgroup: protect iterating blkgs withblkcg->lockin
blkcg_print_stat()
blk-cgroup: delay freeing policy data after rcu grace period
blk-cgroup: don't nestqueue_lockunder rcu inblkcg_print_blkgs()
blk-cgroup: don't nestqueue_lockunder rcu inblkg_lookup_create()
blk-cgroup: don't nestqueue_lockunder rcu inbio_associate_blkg()
blk-cgroup: don't nestqueue_lockunderblkcg->lockin
blkcg_destroy_blkgs()
mm/page_io: don't nestqueue_lockunder rcu in
bio_associate_blkg_from_page()
block, bfq: don't grabqueue_lockto initialize bfqblock/bfq-cgroup.c | 17 ++++- block/bfq-iosched.c | 5 -- block/blk-cgroup-rwstat.c | 15 ++-- block/blk-cgroup.c | 151 ++++++++++++++++++++++---------------- block/blk-cgroup.h | 8 +- block/blk-iocost.c | 22 ++++-- block/blk-iolatency.c | 10 ++- block/blk-throttle.c | 13 +++- mm/page_io.c | 7 +- 9 files changed, 158 insertions(+), 90 deletions(-)base-commit:
b23df513de562739af61fa61ba80ef5e8059a636
Signature · 2 lines
Thanks, Kuai
Jens Axboe
maintainerQuoted 3 lines · yu kuai
Friendly ping ...
This set can still be applied cleanly for block-7.2 branch.
Not sure how you checked that, because patch 3 very much needs some manual attention to get applied. I have applied it now.
Signature · 1 line
Jens Axboe
yu kuai
maintainerHi,
Quoted 6 lines · Jens Axboe
Quoted 3 lines · yu kuai
Friendly ping ...
This set can still be applied cleanly for block-7.2 branch.
Not sure how you checked that, because patch 3 very much needs some manual attention to get applied. I have applied it now.
Thanks!
This was build on the top of my other set:
blk-cgroup: fix blkg list and policy data races
I'll rebase and resend this set :)
Quoted 1 line
Signature · 2 lines
Thanks, Kuai
Quoted 11 lines · Yu Kuai
This series is the follow-up blk-cgroup locking cleanup on top of the earlier blkg-list protection fixes, and prepares blk-cgroup to stop using
q->queue_lockas the global blkg lifetime/iteration lock.The current
queue_lockbased protection is hard to maintain becausequeue_lockis used from hardirq and softirq completion paths, while some blkcg cgroup file paths also need to iterate blkgs, print policy data, or create blkgs from RCU-protected contexts. This series first tightens the blkcg-side lifetime rules:[...]
Applied, thanks!
[1/8] blk-cgroup: protect iterating blkgs with blkcg->lock in blkcg_print_stat()
commit: 25656304dabd26198ec69460c594a19d086ef099
[2/8] blk-cgroup: delay freeing policy data after rcu grace period
commit: 0af3fedb8c8ed3c07b4f76927bd7fc88f6f82efb
[3/8] blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()
commit: 56cc24f59c145ce6938959f792df04b8a4f5a4d8
[4/8] blk-cgroup: don't nest queue_lock under rcu in blkg_lookup_create()
commit: 9327a865e395a53f67dffac4710beb1d4730495e
[5/8] blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()
commit: 457d3c4f0fdd6cf8a4bd8115bf470809984a9f02
[6/8] blk-cgroup: don't nest queue_lock under blkcg->lock in blkcg_destroy_blkgs()
commit: 4cfd7c1cff8f4c863b99d420cdbe0563802a9e80
[7/8] mm/page_io: don't nest queue_lock under rcu in bio_associate_blkg_from_page()
commit: f928145cbcb52544203808f159461d0a25543df7
[8/8] block, bfq: don't grab queue_lock to initialize bfq
commit: 3ca4f4e3ae811d414076a491cbf0dfcdae0dc01eBest regards,
Signature · 1 line
Jens Axboe