blk-cgroup: store blkcg in bio before blkcg_mutex conversion
Subsystems (16)
cover blk-cgroup: store blkcg in bio before blkcg_mutex conversion
From: Yu Kuai <[email protected]>
This is the first formal, non-RFC posting of the preparatory series for the blkcg_mutex conversion proposed in the related blkcg_mutex RFC v2 series [1]. That conversion moves queue-local blkg topology synchronization from q->queue_lock to q->blkcg_mutex, which is awkward while bios directly store queue-local blkg references.
RFC v1 made the stored bio association queue-independent by replacing bi_blkg with bi_blkcg, but a bio-owned blkg reference still had to be recovered by looking up the bio's blkcg and current request_queue. Tao Cui reported that cgroup removal deletes a dying blkg from the per-blkcg radix tree before a throttled bio drops its reference. A later lookup for that pinned blkg then fails, triggers the warning in bio_pinned_blkg(), and leaks the reference.
RFC v2 makes request_queue the authoritative lookup owner before converting the bio association. A queue-owned rhashtable, keyed by the blkcg CSS ID, keeps dying blkgs discoverable until their references drain while q->blkg_list remains available for ordered walks. The bio conversion then stores and pins the blkcg CSS, lazily creates a blkg only for users which need one, and uses lookup-only access for completion and accounting paths which already own a blkg reference. Async bio punt state is finally moved from blkg to blkcg so punting alone does not instantiate a queue-local blkg.
Changes since RFC v3:
- Drop the RFC prefix.
- Add Christoph Hellwig's Reviewed-by to patch 1.
- Add Tao Cui's Reviewed-by to patch 2.
- In patch 2, relax blkg_lookup_any()/blkg_lookup() to also allow
q->queue_lock as an alternative to the RCU read lock.
Changes since RFC v2:
- Drop the old patch 1 and send it separately as a bugfix for 7.3 and -stable, as suggested by Christoph Hellwig.
- Rebase onto the latest for-7.3/block branch.
- In the new patch 1, note that the remaining q->blkg_list walkers are cgroupfs/sysfs slow paths and can move to rhashtable iteration after the queue_lock-to-blkcg_mutex conversion.
- In the new patch 1, explain the list_empty case in blkg_release(), and add an RCU lockdep assertion plus documentation that blkg_lookup_any() does not acquire a reference.
Changes since RFC v1:
- Add patch 1 to wait for every old blkg to leave q->blkg_list before a shared request_queue is rebound, instead of treating root_blkg == NULL as completion of asynchronous blkg teardown.
- Add patch 2 to replace the per-blkcg radix tree and lookup hint with a request_queue rhashtable keyed by blkcg->css.id, as suggested by Christoph Hellwig. Keep dying pinned blkgs in the hash until blkg_release() so bio-owned references remain discoverable.
- Fold the v1 helper-only patch into patch 3, as suggested by Jan Kara and Christoph Hellwig, and make bio_blkcg() naturally return NULL for an unassociated bio.
- Rework patch 3 so blkg_lookup_create() acquires the bio-owned reference, falls back to a live parent when creation or tryget fails, and updates bi_blkcg when the returned blkg belongs to an ancestor.
- Make bio_blkg_lookup() lookup-only: it returns NULL unless BIO_BLKG_REF is already set. Use bio_blkg() in the BFQ and IOCOST merge paths which may need to create a blkg, while keeping blk_cgroup_bio_start() and completion paths lookup-only.
- Move CSS online-reference handling into bio_associate_blkcg_from_css(), including fallback to the root blkcg, so bio_associate_blkcg() does not take a redundant reference.
- Keep the v1 async bio punt conversion as patch 4 and document that async_bio_lock protects async_bios.
Previous versions:
RFC v3: https://lore.kernel.org/r/[email protected]
RFC v2: https://lore.kernel.org/r/[email protected]
RFC v1: https://lore.kernel.org/r/[email protected]
Related series:
[1] RFC v2 blk-cgroup: protect blkgs with blkcg_mutex
https://lore.kernel.org/r/[email protected]
Yu Kuai (3):
blk-cgroup: use a request_queue rhashtable for blkg lookup
blk-cgroup: store blkcg in bio instead of blkg
blk-cgroup: move async bio punt state to blkcg
Documentation/admin-guide/cgroup-v2.rst | 2 +- block/bfq-cgroup.c | 16 +- block/bfq-iosched.c | 19 +- block/bio.c | 22 +- block/blk-cgroup-fc-appid.c | 10 +- block/blk-cgroup.c | 339 ++++++++++++++---------- block/blk-cgroup.h | 92 +++++-- block/blk-core.c | 9 +- block/blk-crypto-fallback.c | 2 +- block/blk-iocost.c | 12 +- block/blk-iolatency.c | 11 +- block/blk-ioprio.c | 2 +- block/blk-throttle.c | 2 +- block/blk-throttle.h | 2 +- drivers/md/bcache/request.c | 2 +- drivers/md/dm.c | 2 +- drivers/md/md.c | 2 +- drivers/nvdimm/nd_virtio.c | 2 +- fs/gfs2/lops.c | 3 +- include/linux/bio.h | 26 +- include/linux/blk_types.h | 9 +- include/linux/blkdev.h | 2 + include/linux/writeback.h | 2 +- mm/page_io.c | 13 +- 24 files changed, 366 insertions(+), 237 deletions(-)
Signature · 1 line
2.51.0
PATCH 1/3 blk-cgroup: use a request_queue rhashtable for blkg lookup
From: Yu Kuai <[email protected]>
blkg lookup currently uses a per-blkcg radix tree keyed by request queue ID, plus a lookup hint for the common case. This spreads the queue-local blkcg association index across every blkcg and requires radix-tree preloading before creating a blkg while holding q->queue_lock.
Replace the radix tree and lookup hint with a request_queue-owned rhashtable keyed by the blkcg CSS ID. Cache the ID in each blkg; the blkg holds a CSS reference until after it leaves the hash, so the ID cannot be reused while it is hash-visible. The integer key also reduces hashing and comparison work relative to a pointer-sized key on 64-bit systems.
Keep entries until blkg_release() and provide blkg_lookup_any() for callers which need to find dying entries. blkg_lookup() filters offline entries so existing lookup semantics remain unchanged.
Keep q->blkg_list for ordered policy and scheduler walks. All current walkers are cgroupfs or sysfs slow paths, so they can move to rhashtable iteration once the q->queue_lock to q->blkcg_mutex conversion lands. Initialize and destroy the hash with request_queue, and remove the radix-tree preload paths which are no longer needed.
blkg_release() removes the hash entry only when the blkg was successfully inserted into q->blkg_list; the list_empty case covers allocation or creation failure before insertion.
--- block/blk-cgroup.c | 64 +++++++++++++++++++----------------------- block/blk-cgroup.h | 49 +++++++++++++++++++++++--------- block/blk-core.c | 9 ++++-- include/linux/blkdev.h | 2 ++ 4 files changed, 74 insertions(+), 50 deletions(-)
block/blk-cgroup.c +29 -35
@@ -64,10 +64,17 @@ bool blkcg_debug_stats = false;static DEFINE_RAW_SPINLOCK (blkg_stat_lock);#define BLKG_DESTROY_BATCH_SIZE 64 +const struct rhashtable_params blkg_hash_params = { + .key_len = sizeof_field(struct blkcg_gq, blkcg_id), + .key_offset = offsetof(struct blkcg_gq , blkcg_id), + .head_offset = offsetof(struct blkcg_gq , q_hash_node), + .automatic_shrinking = true, +}; +/* * Lockless lists for tracking IO stats update * * New IO stats are stored in the percpu iostat_cpu within blkcg_gq (blkg). * There are multiple blkg's (one for each block device) attached to each
@@ -194,10 +201,20 @@ static void blkg_release(struct percpu_ref *ref) { struct blkcg_gq *blkg = container_of(ref, struct blkcg_gq, refcnt);struct blkcg *blkcg = blkg->blkcg;int cpu; +/* +* A blkg that was never inserted into q->blkg_list has no hash +* entry. This happens when allocation or creation fails before +* rhashtable_insert_fast() succeeds. +*/ +if (!list_empty (&blkg->q_node)) +WARN_ON_ONCE (rhashtable_remove_fast (&blkg->q->blkg_hash, + &blkg->q_hash_node, + blkg_hash_params)); +/* * Flush all the non-empty percpu lockless lists before releasing * us, given these stat belongs to us. * * blkg_stat_lock is for serializing blkg stat update
@@ -327,10 +344,11 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct gendisk *disk, goto out_put_queue; blkg->q = disk->queue;INIT_LIST_HEAD (&blkg->q_node); blkg->blkcg = blkcg; + blkg->blkcg_id = blkcg->css.id; blkg->iostat.blkg = blkg;#ifdef CONFIG_BLK_CGROUP_PUNT_BIO spin_lock_init (&blkg->async_bio_lock);bio_list_init (&blkg->async_bios);INIT_WORK (&blkg->async_bio_work, blkg_async_bio_workfn);
@@ -423,11 +441,12 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk, pol->pd_init_fn (blkg->pd[i]); }/* insert */ spin_lock (&blkcg->lock); - ret =radix_tree_insert (&blkcg->blkg_tree, disk->queue->id, blkg); + ret =rhashtable_insert_fast (&disk->queue->blkg_hash, + &blkg->q_hash_node, blkg_hash_params);if (likely (!ret)) {hlist_add_head_rcu (&blkg->blkcg_node, &blkcg->blkg_list);list_add (&blkg->q_node, &disk->queue->blkg_list);for (i =0 ; i <BLKCG_MAX_POLS ; i++) {
@@ -476,13 +495,10 @@ static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,struct blkcg_gq *blkg;rcu_read_lock (); blkg =blkg_lookup (blkcg, q);if (blkg) { -if (blkcg != &blkcg_root && - blkg !=rcu_dereference (blkcg->blkg_hint)) -rcu_assign_pointer (blkcg->blkg_hint, blkg);rcu_read_unlock ();return blkg; }rcu_read_unlock ();
@@ -546,21 +562,12 @@ static void blkg_destroy(struct blkcg_gq *blkg) } } blkg->online = false; -radix_tree_delete (&blkcg->blkg_tree, blkg->q->id);hlist_del_init_rcu (&blkg->blkcg_node); -/* -* Both setting lookup hint to and clearing it from @blkg are done -* under queue_lock. If it's not pointing to @blkg now, it never -* will. Hint assignment itself can race safely. -*/ -if (rcu_access_pointer (blkcg->blkg_hint) == blkg) -rcu_assign_pointer (blkcg->blkg_hint,NULL ); -/* * Put the reference taken at the time of creation so that when all * queues are gone, group can be destroyed. */ percpu_ref_kill (&blkg->refcnt);
@@ -880,47 +887,37 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,if (unlikely (!new_blkg)) { ret = -ENOMEM ; goto fail_exit; } -if (radix_tree_preload (GFP_KERNEL )) { -blkg_free (new_blkg); - ret = -ENOMEM ; - goto fail_exit; - } -spin_lock_irq (&q->queue_lock);if (!blkcg_policy_enabled (q, pol)) {blkg_free (new_blkg); ret = -EOPNOTSUPP ; - goto fail_preloaded; + goto fail_unlock; } blkg =blkg_lookup (pos, q);if (blkg) {blkg_free (new_blkg); }else { blkg =blkg_create (pos, disk, new_blkg);if (IS_ERR (blkg)) { ret =PTR_ERR (blkg); - goto fail_preloaded; + goto fail_unlock; } } -radix_tree_preload_end (); -if (pos == blkcg) goto success; } success:mutex_unlock (&q->blkcg_mutex); ctx->blkg = blkg;return 0 ; -fail_preloaded: -radix_tree_preload_end (); fail_unlock:spin_unlock_irq (&q->queue_lock); fail_exit:mutex_unlock (&q->blkcg_mutex);/*
@@ -1418,11 +1415,10 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_css) cpd->plid = i; }spin_lock_init (&blkcg->lock);refcount_set (&blkcg->online_pin,1 ); -INIT_RADIX_TREE (&blkcg->blkg_tree,GFP_NOWAIT );INIT_HLIST_HEAD (&blkcg->blkg_list);#ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD (&blkcg->cgwb_list);#endif list_add_tail (&blkcg->all_blkcgs_node, &all_blkcgs);
@@ -1455,21 +1451,26 @@ static int blkcg_css_online(struct cgroup_subsys_state *css)if (parent)blkcg_pin_online (&parent->css);return 0 ; } -void blkg_init_queue (struct request_queue *q) +int blkg_init_queue (struct request_queue *q) {INIT_LIST_HEAD (&q->blkg_list);mutex_init (&q->blkcg_mutex); +return rhashtable_init (&q->blkg_hash, &blkg_hash_params); +} + +void blkg_exit_queue (struct request_queue *q) +{ +rhashtable_destroy (&q->blkg_hash); }int blkcg_init_disk (struct gendisk *disk) {struct request_queue *q = disk->queue;struct blkcg_gq *new_blkg, *blkg; -bool preloaded;/* * If the queue is shared across disk rebind (e.g., SCSI), the * previous disk's blkcg state is cleaned up asynchronously via * disk_release() -> blkcg_exit_disk(). Wait for all old blkgs to be
@@ -1479,30 +1480,23 @@ int blkcg_init_disk(struct gendisk *disk) new_blkg =blkg_alloc (&blkcg_root, disk,GFP_KERNEL );if (!new_blkg)return -ENOMEM ; - preloaded = !radix_tree_preload (GFP_KERNEL ); -/* Make sure the root blkg exists. */ /* spin_lock_irq can serve as RCU read-side critical section. */ spin_lock_irq (&q->queue_lock); blkg =blkg_create (&blkcg_root, disk, new_blkg);if (IS_ERR (blkg)) goto err_unlock; q->root_blkg = blkg;spin_unlock_irq (&q->queue_lock); -if (preloaded) -radix_tree_preload_end (); -return 0 ; err_unlock:spin_unlock_irq (&q->queue_lock); -if (preloaded) -radix_tree_preload_end ();return PTR_ERR (blkg); }void blkcg_exit_disk (struct gendisk *disk) {
block/blk-cgroup.h +36 -13
127 lines not on this page · raw message
block/blk-core.c +7 -2
@@ -301,10 +301,11 @@ static void blk_free_queue(struct request_queue *q) {blk_free_queue_stats (q->stats);if (queue_is_mq (q))blk_mq_release (q); +blkg_exit_queue (q);ida_free (&blk_queue_ida, q->id);lockdep_unregister_key (&q->io_lock_cls_key);lockdep_unregister_key (&q->q_lock_cls_key);call_rcu (&q->rcu_head, blk_free_queue_rcu); }
@@ -479,21 +480,23 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)spin_lock_init (&q->queue_lock);init_waitqueue_head (&q->mq_freeze_wq);mutex_init (&q->mq_freeze_lock); -blkg_init_queue (q); + error =blkg_init_queue (q); +if (error) + goto fail_stats;/* * Init percpu_ref in atomic mode so that it's faster to shutdown. * See blk_register_queue() for details. */ error =percpu_ref_init (&q->q_usage_counter, blk_queue_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL );if (error) - goto fail_stats; + goto fail_blkg;lockdep_register_key (&q->io_lock_cls_key);lockdep_register_key (&q->q_lock_cls_key);lockdep_init_map (&q->io_lockdep_map,"&q->q_usage_counter(io)" , &q->io_lock_cls_key,0 ); lockdep_init_map(&q->q_lockdep_map, "&q->q_usage_counter(queue)",
@@ -508,10 +511,12 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id) q->nr_requests =BLKDEV_DEFAULT_RQ ; q->async_depth =BLKDEV_DEFAULT_RQ ;return q; +fail_blkg: +blkg_exit_queue (q); fail_stats:blk_free_queue_stats (q->stats); fail_id:ida_free (&blk_queue_ida, q->id); fail_q:
include/linux/blkdev.h +2 -0
@@ -25,10 +25,11 @@#include <linux/sbitmap.h> #include <linux/uuid.h> #include <linux/xarray.h> #include <linux/file.h> #include <linux/lockdep.h> +#include <linux/rhashtable-types.h> struct module ;struct request_queue ;struct elevator_queue ;struct blk_trace ;
@@ -579,10 +580,11 @@ struct request_queue {struct list_head icq_list;#ifdef CONFIG_BLK_CGROUP DECLARE_BITMAP (blkcg_pols,BLKCG_MAX_POLS );struct blkcg_gq *root_blkg; +struct rhashtable blkg_hash;struct list_head blkg_list;struct mutex blkcg_mutex;#endif int node;
Signature · 1 line
2.51.0
Re: [PATCH 1/3] blk-cgroup: use a request_queue rhashtable for blkg lookup
Quoted 23 lines · Yu Kuai
From: Yu Kuai<[email protected]>
blkg lookup currently uses a per-blkcg radix tree keyed by request queue ID, plus a lookup hint for the common case. This spreads the queue-local blkcg association index across every blkcg and requires radix-tree preloading before creating a blkg while holding
q->queue_lock.Replace the radix tree and lookup hint with a request_queue-owned rhashtable keyed by the blkcg CSS ID. Cache the ID in each blkg; the blkg holds a CSS reference until after it leaves the hash, so the ID cannot be reused while it is hash-visible. The integer key also reduces hashing and comparison work relative to a pointer-sized key on 64-bit systems.
Keep entries until
blkg_release()and provideblkg_lookup_any()for callers which need to find dying entries.blkg_lookup()filters offline entries so existing lookup semantics remain unchanged.Keep
q->blkg_listfor ordered policy and scheduler walks. All current walkers are cgroupfs or sysfs slow paths, so they can move to rhashtable iteration once theq->queue_locktoq->blkcg_mutexconversion lands. Initialize and destroy the hash withrequest_queue, and remove the radix-tree preload paths which are no longer needed.
Quoted 6 lines
blkg_release()removes the hash entry only when the blkg was successfully inserted intoq->blkg_list; thelist_emptycase covers allocation or creation failure before insertion.Signed-off-by Yu Kuai<[email protected]>Reviewed-by Christoph Hellwig<[email protected]>
Looks good to me.
PATCH 2/3 blk-cgroup: store blkcg in bio instead of blkg
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
A bio currently stores and pins a queue-local blkg. This forces bio association and remap paths to look up or create a blkg even when no blkcg policy will use the bio, and ties the stored state to the current block device.
Store and reference the queue-independent blkcg in the bio instead. Add helpers that lazily look up or create the queue-local blkg when a policy needs it, and pin the result until the bio changes devices or releases its cgroup state.
If blkg creation fails while walking down the hierarchy, use the closest available ancestor and update the bio's blkcg association before recording the blkg reference. This keeps later CSS ID hash lookups matched with the pinned blkg.
Keep lookup-only users from creating missing blkgs. A pinned blkg remains in the queue hash until the bio drops its reference, so allow the bio to recover it from the hash after the blkg starts dying. Rename the bio association helpers to describe the blkcg state they now store.
--- Documentation/admin-guide/cgroup-v2.rst | 2 +- block/bfq-cgroup.c | 16 +- block/bfq-iosched.c | 19 +- block/bio.c | 22 +-- block/blk-cgroup-fc-appid.c | 10 +- block/blk-cgroup.c | 237 +++++++++++++++--------- block/blk-cgroup.h | 35 +++- block/blk-crypto-fallback.c | 2 +- block/blk-iocost.c | 12 +- block/blk-iolatency.c | 11 +- block/blk-ioprio.c | 2 +- block/blk-throttle.c | 2 +- block/blk-throttle.h | 2 +- drivers/md/bcache/request.c | 2 +- drivers/md/dm.c | 2 +- drivers/md/md.c | 2 +- drivers/nvdimm/nd_virtio.c | 2 +- fs/gfs2/lops.c | 3 +- include/linux/bio.h | 26 +-- include/linux/blk_types.h | 9 +- include/linux/writeback.h | 2 +- mm/page_io.c | 13 +- 22 files changed, 271 insertions(+), 162 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +1 -1
@@ -3233,11 +3233,11 @@ incompatible. wbc_init_bio() binds the specified bio to its cgroup. Depending on the configuration, the bio may be executed at a lower priority and if the writeback session is holding shared resources, e.g. a journal entry, may lead to priority inversion. There is no one easy solution for the problem. Filesystems can try to work around specific problem -cases by skipping wbc_init_bio() and using bio_associate_blkg() +cases by skipping wbc_init_bio() and using bio_associate_blkcg() directly. Deprecated v1 Core Features ===========================
block/bfq-cgroup.c +10 -6
@@ -361,15 +361,17 @@ void bfqg_and_blkg_put(struct bfq_group *bfqg)bfqg_put (bfqg); }void bfqg_stats_update_legacy_io (struct request_queue *q,struct request *rq) { -struct bfq_group *bfqg =blkg_to_bfqg (rq->bio->bi_blkg); +struct blkcg_gq *blkg =bio_blkg_lookup (rq->bio); +struct bfq_group *bfqg; -if (!bfqg) +if (!blkg)return ; + bfqg =blkg_to_bfqg (blkg);blkg_rwstat_add (&bfqg->stats.bytes, rq->cmd_flags,blk_rq_bytes (rq));blkg_rwstat_add (&bfqg->stats.ios, rq->cmd_flags,1 ); }/* @stats = 0 */
@@ -604,27 +606,29 @@ static void bfq_link_bfqg(struct bfq_data *bfqd, struct bfq_group *bfqg) } }struct bfq_group *bfq_bio_bfqg (struct bfq_data *bfqd,struct bio *bio) { -struct blkcg_gq *blkg = bio->bi_blkg; +struct blkcg_gq *blkg =bio_blkg_lookup (bio);struct bfq_group *bfqg;while (blkg) {if (!data_race (blkg->online)) { blkg = blkg->parent;continue ; } + bfqg =blkg_to_bfqg (blkg);if (bfqg->pd.online) { -bio_associate_blkg_from_css (bio, &blkg->blkcg->css); +bio_associate_blkcg_from_css (bio, &blkg->blkcg->css);return bfqg; } blkg = blkg->parent; } -bio_associate_blkg_from_css (bio, - &bfqg_to_blkg (bfqd->root_group)->blkcg->css); + + blkg =bfqg_to_blkg (bfqd->root_group); +bio_associate_blkcg_from_css (bio, &blkg->blkcg->css);return bfqd->root_group; }/** * bfq_bfqq_move - migrate @bfqq to @bfqg.
block/bfq-iosched.c +14 -5
@@ -126,10 +126,11 @@#include <trace/events/block.h> #include "elevator.h" #include "blk.h" +#include "blk-cgroup.h" #include "blk-mq.h" #include "blk-mq-sched.h" #include "bfq-iosched.h" #include "blk-wbt.h"
@@ -2450,19 +2451,20 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio,struct bfq_data *bfqd = q->elevator->elevator_data;struct bfq_io_cq *bic =bfq_bic_lookup (q);struct request *free =NULL ;bool ret; +#ifdef CONFIG_BFQ_GROUP_IOSCHED +/* blkg creation takes q->queue_lock, so do it before bfqd->lock. */ +if (bic) +bio_blkg (bio); +#endif +spin_lock_irq (&bfqd->lock);if (bic) { -/* -* Make sure cgroup info is uptodate for current process before -* considering the merge. -*/ bfq_bic_update_cgroup (bic, bio); - bfqd->bio_bfqq =bic_to_bfqq (bic,op_is_sync (bio->bi_opf),bfq_actuator_index (bfqd, bio)); }else { bfqd->bio_bfqq =NULL ; }
@@ -6243,10 +6245,17 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,bool idle_timer_disabled = false;blk_opf_t cmd_flags;LIST_HEAD (free);#ifdef CONFIG_BFQ_GROUP_IOSCHED +/* +* Pin the blkg used to look up bfqg. If this is the first IO for +* the blkcg on this queue, create the bfqg before holding bfqd->lock. +*/ +if (rq->bio && !bio_flagged (rq->bio,BIO_BLKG_REF )) +bio_blkg (rq->bio); +if (!cgroup_subsys_on_dfl (io_cgrp_subsys) && rq->bio)bfqg_stats_update_legacy_io (q, rq);#endif spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq);
block/bio.c +6 -16
@@ -177,16 +177,11 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp) __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN; }void bio_uninit (struct bio *bio) { -#ifdef CONFIG_BLK_CGROUP -if (bio->bi_blkg) { -blkg_put (bio->bi_blkg); - bio->bi_blkg =NULL ; - } -#endif +bio_clear_blkcg (bio);if (bio_integrity (bio))bio_integrity_free (bio);bio_crypt_free_ctx (bio); }
@@ -231,14 +226,14 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, bio->bi_iter.bi_idx =0 ; bio->bi_iter.bi_offset =0 ; bio->bi_end_io =NULL ; bio->bi_private =NULL ;#ifdef CONFIG_BLK_CGROUP - bio->bi_blkg =NULL ; + bio->bi_blkcg =NULL ; bio->issue_time_ns =0 ;if (bdev) -bio_associate_blkg (bio); +bio_associate_blkcg (bio);#ifdef CONFIG_BLK_CGROUP_IOCOST bio->bi_iocost_cost =0 ;#endif #endif #ifdef CONFIG_BLK_INLINE_ENCRYPTION
@@ -279,11 +274,11 @@ void bio_reset(struct bio *bio, struct block_device *bdev, blk_opf_t opf)memset (bio,0 ,BIO_RESET_BYTES );atomic_set (&bio->__bi_remaining,1 ); bio->bi_io_vec = bv; bio->bi_bdev = bdev;if (bio->bi_bdev) -bio_associate_blkg (bio); +bio_associate_blkcg (bio); bio->bi_opf = opf; }EXPORT_SYMBOL (bio_reset);/**
@@ -864,11 +859,11 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)if (bio->bi_bdev) {if (bio->bi_bdev == bio_src->bi_bdev &&bio_flagged (bio_src,BIO_REMAPPED ))bio_set_flag (bio,BIO_REMAPPED ); -bio_clone_blkg_association (bio, bio_src); +bio_clone_blkcg_association (bio, bio_src); }if (bio_crypt_clone (bio, bio_src, gfp) <0 )return -ENOMEM ; if (bio_integrity(bio_src) &&
@@ -1908,21 +1903,16 @@ void bio_endio(struct bio *bio)if (bio->bi_end_io == bio_chain_endio) { bio =__bio_chain_endio (bio); goto again; } -#ifdef CONFIG_BLK_CGROUP /* * Release cgroup info. We shouldn't have to do this here, but quite * a few callers of bio_init fail to call bio_uninit, so we cover up * for that here at least for now. */ -if (bio->bi_blkg) { -blkg_put (bio->bi_blkg); - bio->bi_blkg =NULL ; - } -#endif +bio_clear_blkcg (bio);if (bio_flagged (bio,BIO_COMPLETE_IN_TASK ) &&bio_in_atomic ())__bio_complete_in_task (bio);else if (bio->bi_end_io) bio->bi_end_io (bio);
block/blk-cgroup-fc-appid.c +8 -2
@@ -48,10 +48,16 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid);* * On success return the fc_app_id, on failure return NULL */ char *blkcg_get_fc_appid (struct bio *bio) { -if (!bio->bi_blkg || bio->bi_blkg->blkcg->fc_app_id[0 ] =='\0' ) +struct blkcg *blkcg =bio_blkcg (bio); + +if (!blkcg) +return NULL ; + +if (blkcg->fc_app_id[0 ] =='\0' )return NULL ; -return bio->bi_blkg->blkcg->fc_app_id; + +return blkcg->fc_app_id; }EXPORT_SYMBOL_GPL (blkcg_get_fc_appid);
block/blk-cgroup.c +150 -87
382 lines not on this page · raw message
block/blk-cgroup.h +30 -5
@@ -124,10 +124,15 @@ struct blkcg {static inline struct blkcg *css_to_blkcg (struct cgroup_subsys_state *css) { return css ? container_of(css, struct blkcg, css) : NULL; } +static inline struct blkcg *bio_blkcg (struct bio *bio) +{ +return bio->bi_blkcg; +} +/* * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a * request_queue (q). This is used by blkcg policies which need to track * information per blkcg - q pair. *
@@ -257,20 +262,21 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio)* @blkcg: blkcg of interest * @q: request_queue of interest * * Lookup a blkg for the @blkcg - @q pair, whether it is online or dying. * -* Must be called in a RCU critical section. +* Must be called with either RCU read lock or queue_lock held. * * This does not acquire a reference. The caller must already hold one, or * have the blkg pinned by I/O. */ static inline struct blkcg_gq *blkg_lookup_any (struct blkcg *blkcg,struct request_queue *q) { -RCU_LOCKDEP_WARN (!rcu_read_lock_held (), -"blkg_lookup_any() requires an RCU read lock" ); +RCU_LOCKDEP_WARN (!rcu_read_lock_held () && + !lockdep_is_held (&q->queue_lock), +"blkg_lookup_any() requires an RCU read lock or queue_lock" );return rhashtable_lookup (&q->blkg_hash, &blkcg->css.id, blkg_hash_params); }
@@ -279,11 +285,11 @@ static inline struct blkcg_gq *blkg_lookup_any(struct blkcg *blkcg,* @blkcg: blkcg of interest * @q: request_queue of interest * * Lookup an online blkg for the @blkcg - @q pair. * -* Must be called in a RCU critical section. +* Must be called with either RCU read lock or queue_lock held. */ static inline struct blkcg_gq *blkg_lookup (struct blkcg *blkcg,struct request_queue *q) {struct blkcg_gq *blkg;
@@ -295,10 +301,13 @@ static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg,if (blkg && !READ_ONCE (blkg->online)) blkg =NULL ;return blkg; } +struct blkcg_gq *bio_blkg_lookup (struct bio *bio); +struct blkcg_gq *bio_blkg (struct bio *bio); +/** * blkg_to_pd - get policy private data * @blkg: blkg of interest * @pol: policy of interest *
@@ -362,10 +371,22 @@ static inline bool blkg_tryget(struct blkcg_gq *blkg)static inline void blkg_put (struct blkcg_gq *blkg) {percpu_ref_put (&blkg->refcnt); } +static inline void bio_clear_blkcg (struct bio *bio) +{ +struct blkcg *blkcg =bio_blkcg (bio); + +bio_put_blkg_ref (bio); + +if (blkcg) { +css_put (&blkcg->css); + bio->bi_blkcg =NULL ; + } +} +/** * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants * @d_blkg: loop cursor pointing to the current descendant * @pos_css: used for iteration * @p_blkg: target blkg to walk descendants of
@@ -491,11 +512,11 @@ static inline void blkcg_clear_delay(struct blkcg_gq *blkg)* match. The latter is necessary as we don't want to throttle e.g. a metadata * update because it happens to be next to a regular IO. */ static inline bool blk_cgroup_mergeable (struct request *rq,struct bio *bio) { -return rq->bio->bi_blkg == bio->bi_blkg && +return bio_blkcg (rq->bio) ==bio_blkcg (bio) &&bio_issue_as_root_blkg (rq->bio) ==bio_issue_as_root_blkg (bio); }static inline bool blkcg_policy_enabled (struct request_queue *q,const struct blkcg_policy *pol)
@@ -517,10 +538,13 @@ struct blkcg_policy { };struct blkcg { }; +static inline struct blkcg *bio_blkcg (struct bio *bio) {return NULL ; } +static inline struct blkcg_gq *bio_blkg_lookup (struct bio *bio) {return NULL ; } +static inline struct blkcg_gq *bio_blkg (struct bio *bio) {return NULL ; }static inline struct blkcg_gq *blkg_lookup_any (struct blkcg *blkcg,void *key) {return NULL ; }static inline struct blkcg_gq *blkg_lookup (struct blkcg *blkcg,void *key) {return NULL ; }static inline int blkg_init_queue (struct request_queue *q) {return 0 ; }static inline void blkg_exit_queue (struct request_queue *q) { }static inline int blkcg_init_disk (struct gendisk *disk) {return 0 ; }
@@ -535,10 +559,11 @@ static inline void blkcg_deactivate_policy(struct gendisk *disk,static inline struct blkg_policy_data *blkg_to_pd (struct blkcg_gq *blkg,const struct blkcg_policy *pol) {return NULL ; }static inline struct blkcg_gq *pd_to_blkg (struct blkg_policy_data *pd) {return NULL ; }static inline void blkg_get (struct blkcg_gq *blkg) { }static inline void blkg_put (struct blkcg_gq *blkg) { } +static inline void bio_clear_blkcg (struct bio *bio) { }static inline void blk_cgroup_bio_start (struct bio *bio) { }static inline bool blk_cgroup_mergeable (struct request *rq,struct bio *bio) {return true; }#define blk_queue_for_each_rl (rl, q) \ for ((rl) = &(q)->root_rl; (rl); (rl) = NULL)
block/blk-crypto-fallback.c +1 -1
@@ -185,11 +185,11 @@ static struct bio *blk_crypto_alloc_enc_bio(struct bio *bio_src, bio->bi_end_io = blk_crypto_fallback_encrypt_endio; bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; -bio_clone_blkg_association (bio, bio_src); +bio_clone_blkcg_association (bio, bio_src);/* * Move page array up in the allocated memory for the bio vecs as far as * possible so that we can start filling biovecs from the beginning * without overwriting the temporary page array.
block/blk-iocost.c +8 -4
@@ -2684,11 +2684,11 @@ iocg_handle_over_budget(struct rq_qos *rqos, struct ioc_gq *iocg,return action_wait; }static void ioc_rqos_throttle (struct rq_qos *rqos,struct bio *bio) { -struct blkcg_gq *blkg = bio->bi_blkg; +struct blkcg_gq *blkg =bio_blkg (bio);struct ioc *ioc =rqos_to_ioc (rqos);struct ioc_gq *iocg =blkg_to_iocg (blkg);struct ioc_now now;struct iocg_wait wait;u64 abs_cost, cost, vtime;
@@ -2773,11 +2773,11 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio) }static void ioc_rqos_merge (struct rq_qos *rqos,struct request *rq,struct bio *bio) { -struct ioc_gq *iocg =blkg_to_iocg (bio->bi_blkg); +struct ioc_gq *iocg =blkg_to_iocg (bio_blkg (bio));struct ioc *ioc =rqos_to_ioc (rqos);sector_t bio_end =bio_end_sector (bio);struct ioc_now now;u64 vtime, abs_cost, cost;unsigned long flags;
@@ -2831,13 +2831,17 @@ static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,spin_unlock_irqrestore (&ioc->lock, flags); }static void ioc_rqos_done_bio (struct rq_qos *rqos,struct bio *bio) { -struct ioc_gq *iocg =blkg_to_iocg (bio->bi_blkg); +struct ioc_gq *iocg; + +if (!bio->bi_iocost_cost) +return ; -if (iocg && bio->bi_iocost_cost) + iocg =blkg_to_iocg (bio_blkg_lookup (bio)); +if (iocg)atomic64_add (bio->bi_iocost_cost, &iocg->done_vtime); }static void ioc_rqos_done (struct rq_qos *rqos,struct request *rq) {
block/blk-iolatency.c +7 -4
@@ -461,11 +461,11 @@ static void check_scale_change(struct iolatency_grp *iolat) }static void blkcg_iolatency_throttle (struct rq_qos *rqos,struct bio *bio) {struct blk_iolatency *blkiolat =BLKIOLATENCY (rqos); -struct blkcg_gq *blkg = bio->bi_blkg; +struct blkcg_gq *blkg =bio_blkg (bio);bool issue_as_root =bio_issue_as_root_blkg (bio);if (!blkiolat->enabled)return ;
@@ -588,15 +588,18 @@ static void blkcg_iolatency_done_bio(struct rq_qos *rqos, struct bio *bio)u64 window_start;u64 now;bool issue_as_root =bio_issue_as_root_blkg (bio);int inflight =0 ; - blkg = bio->bi_blkg; -if (!blkg || !bio_flagged (bio,BIO_QOS_THROTTLED )) +if (!bio_flagged (bio,BIO_QOS_THROTTLED ))return ; - iolat =blkg_to_lat (bio->bi_blkg); + blkg =bio_blkg_lookup (bio); +if (!blkg) +return ; + + iolat =blkg_to_lat (blkg);if (!iolat)return ;if (!iolat->blkiolat->enabled)return ;
block/blk-ioprio.c +1 -1
@@ -130,11 +130,11 @@ static struct blkcg_policy ioprio_policy = { .cpd_free_fn = ioprio_free_cpd, };void blkcg_set_ioprio (struct bio *bio) { -struct ioprio_blkcg *blkcg =blkcg_to_ioprio_blkcg (bio->bi_blkg->blkcg); +struct ioprio_blkcg *blkcg =blkcg_to_ioprio_blkcg (bio_blkcg (bio));u16 prio;if (!blkcg || blkcg->prio_policy ==POLICY_NO_CHANGE )return ;
block/blk-throttle.c +1 -1
@@ -1790,11 +1790,11 @@ static bool tg_within_limit(struct throtl_grp *tg, struct bio *bio, bool rw) }bool __blk_throtl_bio (struct bio *bio) {struct request_queue *q =bdev_get_queue (bio->bi_bdev); -struct blkcg_gq *blkg = bio->bi_blkg; +struct blkcg_gq *blkg =bio_blkg (bio);struct throtl_qnode *qn =NULL ;struct throtl_grp *tg =blkg_to_tg (blkg);struct throtl_service_queue *sq;bool rw =bio_data_dir (bio);bool throttled = false;
block/blk-throttle.h +1 -1
@@ -169,11 +169,11 @@ static inline bool blk_should_throtl(struct bio *bio)int rw =bio_data_dir (bio);if (!blk_throtl_activated (bio->bi_bdev->bd_queue))return false; - tg =blkg_to_tg (bio->bi_blkg); + tg =blkg_to_tg (bio_blkg (bio));if (!cgroup_subsys_on_dfl (io_cgrp_subsys)) {if (!bio_flagged (bio,BIO_CGROUP_ACCT )) {bio_set_flag (bio,BIO_CGROUP_ACCT );blkg_rwstat_add (&tg->stat_bytes, bio->bi_opf, bio->bi_iter.bi_size);
drivers/md/bcache/request.c +1 -1
@@ -846,11 +846,11 @@ static CLOSURE_CALLBACK(cached_dev_read_done)if (s->iop.bio) {bio_reset (s->iop.bio, s->cache_miss->bi_bdev,REQ_OP_READ ); s->iop.bio->bi_iter.bi_sector = s->cache_miss->bi_iter.bi_sector; s->iop.bio->bi_iter.bi_size = s->insert_bio_sectors <<9 ; -bio_clone_blkg_association (s->iop.bio, s->cache_miss); +bio_clone_blkcg_association (s->iop.bio, s->cache_miss);bch_bio_map (s->iop.bio,NULL );bio_copy_data (s->cache_miss, s->iop.bio);bio_put (s->cache_miss);
drivers/md/dm.c +1 -1
@@ -1371,11 +1371,11 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone)/* establish bio that will get submitted */ if (!tgt_clone) tgt_clone = clone; -bio_clone_blkg_association (tgt_clone, io->orig_bio); +bio_clone_blkcg_association (tgt_clone, io->orig_bio);/* * Account io->origin_bio to DM dev on behalf of target * that took ownership of IO with DM_MAPIO_SUBMITTED. */
drivers/md/md.c +1 -1
@@ -9384,11 +9384,11 @@ void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,__blkdev_issue_discard (rdev->bdev, start, size,GFP_NOIO , &discard_bio);if (!discard_bio)return ;bio_chain (discard_bio, bio); -bio_clone_blkg_association (discard_bio, bio); +bio_clone_blkcg_association (discard_bio, bio);mddev_trace_remap (mddev, discard_bio, bio->bi_iter.bi_sector);submit_bio_noacct (discard_bio); }EXPORT_SYMBOL_GPL (md_submit_discard_bio);
drivers/nvdimm/nd_virtio.c +1 -1
@@ -119,11 +119,11 @@ int async_pmem_flush(struct nd_region *nd_region, struct bio *bio)REQ_OP_WRITE |REQ_PREFLUSH , GFP_ATOMIC);if (!child)return -ENOMEM ; -bio_clone_blkg_association (child, bio); +bio_clone_blkcg_association (child, bio); child->bi_iter.bi_sector =-1 ;bio_chain (child, bio);submit_bio (child);return 0 ; }
fs/gfs2/lops.c +1 -2
@@ -482,11 +482,11 @@ static struct bio *gfs2_chain_bio(struct bio *prev, unsigned int nr_iovecs, sector_t sector, blk_opf_t opf) {struct bio *new; new =bio_alloc (prev->bi_bdev, nr_iovecs, opf,GFP_NOIO ); -bio_clone_blkg_association (new, prev); +bio_clone_blkcg_association (new, prev); new->bi_iter.bi_sector = sector;bio_chain (new, prev);submit_bio (prev);return new; }
@@ -1112,6 +1112,5 @@ const struct gfs2_log_operations *gfs2_log_ops[] = { &gfs2_databuf_lops, &gfs2_buf_lops, &gfs2_revoke_lops, NULL, }; -
include/linux/bio.h +15 -11
@@ -540,35 +540,39 @@ static inline void bio_release_pages(struct bio *bio, bool mark_dirty)#define bio_dev (bio) \ disk_devt((bio)->bi_bdev->bd_disk)#ifdef CONFIG_BLK_CGROUP -void bio_associate_blkg (struct bio *bio); -void bio_associate_blkg_from_css (struct bio *bio, -struct cgroup_subsys_state *css); -void bio_clone_blkg_association (struct bio *dst,struct bio *src); +void bio_associate_blkcg (struct bio *bio); +void bio_associate_blkcg_from_css (struct bio *bio, +struct cgroup_subsys_state *css); +void bio_clone_blkcg_association (struct bio *dst,struct bio *src); +void bio_put_blkg_ref (struct bio *bio);void blkcg_punt_bio_submit (struct bio *bio);#else /* CONFIG_BLK_CGROUP */ -static inline void bio_associate_blkg (struct bio *bio) { } -static inline void bio_associate_blkg_from_css (struct bio *bio, -struct cgroup_subsys_state *css) +static inline void bio_associate_blkcg (struct bio *bio) { } +static inline void bio_associate_blkcg_from_css (struct bio *bio, +struct cgroup_subsys_state *css) { } -static inline void bio_clone_blkg_association (struct bio *dst, -struct bio *src) { } +static inline void bio_clone_blkcg_association (struct bio *dst, +struct bio *src) { } +static inline void bio_put_blkg_ref (struct bio *bio) { }static inline void blkcg_punt_bio_submit (struct bio *bio) {submit_bio (bio); }#endif /* CONFIG_BLK_CGROUP */ static inline void bio_set_dev (struct bio *bio,struct block_device *bdev) {bio_clear_flag (bio,BIO_REMAPPED ); -if (bio->bi_bdev != bdev) +if (bio->bi_bdev != bdev) { +bio_put_blkg_ref (bio);bio_clear_flag (bio,BIO_BPS_THROTTLED ); + } bio->bi_bdev = bdev; -bio_associate_blkg (bio); +bio_associate_blkcg (bio); }/* * BIO list management for use by remapping drivers (e.g. DM or MD) and loop. *
include/linux/blk_types.h +4 -5
@@ -244,16 +244,14 @@ struct bio { };bio_end_io_t *bi_end_io;void *bi_private;#ifdef CONFIG_BLK_CGROUP /* -* Represents the association of the css and request_queue for the bio. -* If a bio goes direct to device, it will not have a blkg as it will -* not have a request_queue associated with it. The reference is put -* on release of the bio. +* Represents the blkcg css association for the bio. The reference is +* put on release of the bio. */ -struct blkcg_gq *bi_blkg; +struct blkcg *bi_blkcg;/* Time that this bio was issued. */ u64 issue_time_ns;#ifdef CONFIG_BLK_CGROUP_IOCOST u64 bi_iocost_cost;#endif
@@ -307,10 +305,11 @@ enum {BIO_BPS_THROTTLED ,/* This bio has already been subjected to * throttling rules. Don't do it again. */ BIO_TRACE_COMPLETION ,/* bio_endio() should trace the final completion * of this bio. */ BIO_CGROUP_ACCT ,/* has been accounted to a cgroup */ +BIO_BLKG_REF ,/* bio pins the associated blkg */ BIO_QOS_THROTTLED ,/* bio went through rq_qos throttle path */ /* * This bio has completed bps throttling at the single tg granularity, * which is different from BIO_BPS_THROTTLED. When the bio is enqueued * into the sq->queued of the upper tg, or is about to be dispatched,
include/linux/writeback.h +1 -1
@@ -260,11 +260,11 @@ static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio)* out. This is intentional as we don't want the function to block * behind a slow cgroup. Ultimately, we want pageout() to kick off * regular writeback instead of writing things out itself. */ if (wbc->wb) - bio_associate_blkg_from_css(bio, wbc->wb->blkcg_css); + bio_associate_blkcg_from_css(bio, wbc->wb->blkcg_css); }void inode_switch_wbs_work_fn (struct work_struct *work);#else /* CONFIG_CGROUP_WRITEBACK */
mm/page_io.c +8 -5
@@ -314,11 +314,11 @@ static inline void count_swpout_vm_event(struct folio *folio)count_memcg_folio_events (folio,PSWPOUT ,folio_nr_pages (folio));count_vm_events (PSWPOUT ,folio_nr_pages (folio)); }#if defined(CONFIG_MEMCG ) && defined(CONFIG_BLK_CGROUP ) -static void bio_associate_blkg_from_page (struct bio *bio,struct folio *folio) +static void bio_associate_blkcg_from_page (struct bio *bio,struct folio *folio) {struct cgroup_subsys_state *css;struct mem_cgroup *memcg;if (!folio_memcg_charged (folio))
@@ -329,16 +329,19 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio) css =cgroup_e_css (memcg->css.cgroup, &io_cgrp_subsys);if (!css || !css_tryget (css)) css =NULL ;rcu_read_unlock (); -bio_associate_blkg_from_css (bio, css); +bio_associate_blkcg_from_css (bio, css);if (css)css_put (css); }#else -#define bio_associate_blkg_from_page (bio, folio) do { } while (0) +static inline void bio_associate_blkcg_from_page (struct bio *bio, +struct folio *folio) +{ +}#endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */ struct swap_iocb {struct kiocb iocb;struct bio_vec bvecs[SWAP_CLUSTER_MAX ];
@@ -434,11 +437,11 @@ static void swap_writepage_bdev_sync(struct folio *folio,bio_init (&bio, sis->bdev, &bv,1 ,REQ_OP_WRITE |REQ_SWAP ); bio.bi_iter.bi_sector =swap_folio_sector (folio);bio_add_folio_nofail (&bio, folio,folio_size (folio),0 ); -bio_associate_blkg_from_page (&bio, folio); +bio_associate_blkcg_from_page (&bio, folio);count_swpout_vm_event (folio);folio_start_writeback (folio);folio_unlock (folio);
@@ -454,11 +457,11 @@ static void swap_writepage_bdev_async(struct folio *folio, bio =bio_alloc (sis->bdev,1 ,REQ_OP_WRITE |REQ_SWAP ,GFP_NOIO ); bio->bi_iter.bi_sector =swap_folio_sector (folio); bio->bi_end_io = end_swap_bio_write;bio_add_folio_nofail (bio, folio,folio_size (folio),0 ); -bio_associate_blkg_from_page (bio, folio); +bio_associate_blkcg_from_page (bio, folio);count_swpout_vm_event (folio);folio_start_writeback (folio);folio_unlock (folio);submit_bio (bio); }
Signature · 1 line
2.51.0
Re: [PATCH 2/3] blk-cgroup: store blkcg in bio instead of blkg
Looks good:
Re: [PATCH 2/3] blk-cgroup: store blkcg in bio instead of blkg
In reply to Yu KuaiQuoted 14 lines · Yu Kuai
block/blk-cgroup.h +1
-1@@ -257,20 +262,21 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio)* @blkcg: blkcg of interest * @q: request_queue of interest * * Lookup a blkg for the @blkcg - @q pair, whether it is online or dying. * -* Must be called in a RCU critical section. +* Must be called with either RCU read lock or queue_lock held. * * This does not acquire a reference. The caller must already hold one, or * have the blkg pinned by I/O. */ static inline struct blkcg_gq *blkg_lookup_any (struct blkcg *blkcg,struct request_queue *q)
block/blk-cgroup.h static inline bool bio_issue_as_root_blkg(struct bio *bio) · line 281
Quoted 10 lines
block/blk-cgroup.h +3
-2{ -RCU_LOCKDEP_WARN (!rcu_read_lock_held (), -"blkg_lookup_any() requires an RCU read lock" ); +RCU_LOCKDEP_WARN (!rcu_read_lock_held () && + !lockdep_is_held (&q->queue_lock), +"blkg_lookup_any() requires an RCU read lock or queue_lock" );return rhashtable_lookup (&q->blkg_hash, &blkcg->css.id, blkg_hash_params); }
The changes look good to me. However, rhashtable_lookup() is annotated with __must_hold_shared(RCU). Since we now officially support Clang context annotations in the block layer, compiling with the latest Clang/LLVM generates a number of context warnings, for example:
block/blk-cgroup.h:277:9: warning: calling function 'rhashtable_lookup' requires holding __ctx_lock_RCU 'RCU' [-Wthread-safety-analysis]
277 | return rhashtable_lookup(&q->blkg_hash, &blkcg->css.id,
| ^
The same warning is reported from several other block layer source files that include blk-cgroup.h.
This indicates that rhashtable_lookup() requires the RCU read lock from Clang's context-analysis perspective. Looking at the call paths of blkg_lookup_any(), it appears that not all callers hold rcu_read_lock() when invoking it.
I think we should fix those call paths to acquire rcu reader lock before calling blkg_lookup_any(), rather than weakening or bypassing the context annotation in blkg_lookup_any().
Thanks,
--Nilay
yu kuai
maintainerRe: [PATCH 2/3] blk-cgroup: store blkcg in bio instead of blkg
Hi,
Quoted 39 lines · Nilay Shroff
Quoted 19 lines · Yu Kuai
block/blk-cgroup.h +0
-0@@ -257,20 +262,21 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio) * @blkcg: blkcg of interest * @q: request_queue of interest * * Lookup a blkg for the @blkcg - @q pair, whether it is online ordying.
*
- * Must be called in a RCU critical section.
+ * Must be called with either RCU read lock orqueue_lockheld. *
* This does not acquire a reference. The caller must already hold one, or
* have the blkg pinned by I/O.
*/
static inline structblkcg_gq*blkg_lookup_any(struct blkcg *blkcg, structrequest_queue*q)
{
- RCU_LOCKDEP_WARN(!rcu_read_lock_held(),Quoted 8 lines
- "
blkg_lookup_any()requires an RCU read lock"); + RCU_LOCKDEP_WARN(!rcu_read_lock_held() &&
+ !lockdep_is_held(&q->queue_lock),
+ "blkg_lookup_any()requires an RCU read lock orqueue_lock");
returnrhashtable_lookup(&q->blkg_hash, &blkcg->css.id, blkg_hash_params);
}The changes look good to me. However,
rhashtable_lookup()is annotated with__must_hold_shared(RCU). Since we now officially support Clang context annotations in the block layer, compiling with the latest Clang/LLVM generates a number of context warnings, for example:
block/blk-cgroup.h:277:9: warning: calling function 'rhashtable_lookup' requires holding__ctx_lock_RCU'RCU' [-Wthread-safety-analysis] 277 | return rhashtable_lookup(&q->blkg_hash,&blkcg->css.id, | ^
Quoted 7 lines
The same warning is reported from several other block layer source files that include
blk-cgroup.h.This indicates that
rhashtable_lookup()requires the RCU read lock from Clang's context-analysis perspective. Looking at the call paths ofblkg_lookup_any(), it appears that not all callers holdrcu_read_lock()when invoking it.
I was not aware of the above clang warning, this should not be a real problem because queue_lock should have the same protection as rcu, and I was trying not to make this patch too complex.
I add the rcu_read_lock() in the following patch to covert queue_lock to blkcg_mutex:
[RFC PATCH v3 3/6] blk-cgroup: protect blkgs with blkcg_mutex - Yu Kuai <https://lore.kernel.org/all/[email protected]/>
Since this is related to compile warning, I'll move the rcu protection to this patch.
Quoted 7 lines
I think we should fix those call paths to acquire rcu reader lock before calling
blkg_lookup_any(), rather than weakening or bypassing the context annotation inblkg_lookup_any().Thanks,
--Nilay
Signature · 2 lines
Thanks, Kuai
Re: [PATCH 2/3] blk-cgroup: store blkcg in bio instead of blkg
Quoted 45 lines · yu kuai
Hi,
Quoted 42 lines · Nilay Shroff
Quoted 19 lines · Yu Kuai
block/blk-cgroup.h +0
-0@@ -257,20 +262,21 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio) * @blkcg: blkcg of interest * @q: request_queue of interest * * Lookup a blkg for the @blkcg - @q pair, whether it is online ordying.
*
- * Must be called in a RCU critical section.
+ * Must be called with either RCU read lock orqueue_lockheld. *
* This does not acquire a reference. The caller must already hold
one, or
* have the blkg pinned by I/O.
*/
static inline structblkcg_gq*blkg_lookup_any(struct blkcg *blkcg, structrequest_queue*q)
{
- RCU_LOCKDEP_WARN(!rcu_read_lock_held(),Quoted 8 lines
- "
blkg_lookup_any()requires an RCU read lock"); + RCU_LOCKDEP_WARN(!rcu_read_lock_held() &&
+ !lockdep_is_held(&q->queue_lock),
+ "blkg_lookup_any()requires an RCU read lock orqueue_lock");
returnrhashtable_lookup(&q->blkg_hash, &blkcg->css.id, blkg_hash_params);
}The changes look good to me. However,
rhashtable_lookup()is annotated with__must_hold_shared(RCU). Since we now officially support Clang context annotations in the block layer, compiling with the latest Clang/LLVM generates a number of context warnings, for example:
block/blk-cgroup.h:277:9: warning: calling function 'rhashtable_lookup' requires holding__ctx_lock_RCU'RCU' [-Wthread-safety-analysis]
277 | return rhashtable_lookup(&q->blkg_hash,&blkcg->css.id, | ^The same warning is reported from several other block layer source files that include
blk-cgroup.h.
Quoted 9 lines
Quoted 4 lines
This indicates that
rhashtable_lookup()requires the RCU read lock from Clang's context-analysis perspective. Looking at the call paths ofblkg_lookup_any(), it appears that not all callers holdrcu_read_lock()when invoking it.I was not aware of the above clang warning, this should not be a real problem because
queue_lockshould have the same protection as rcu, and I was trying not to make this patch too complex.
I agree that this is not a runtime correctness issue when the lookup is already protected by q->queue_lock.
Quoted 7 lines
I add the
rcu_read_lock()in the following patch to covertqueue_locktoblkcg_mutex:[RFC PATCH v3 3/6] blk-cgroup: protect blkgs with
blkcg_mutex- Yu Kuai <https://lore.kernel.org/all/[email protected]/>Since this is related to compile warning, I'll move the rcu protection to this patch.
Since rhashtable_lookup() is annotated with __must_hold_shared(RCU), Clang's context analysis still reports warnings for these call paths. While this doesn't cause a build failure (and only generates warning) when CONFIG_WERROR is not enabled, it could do so for configurations that enable CONFIG_WERROR=y.
So IMO, moving the rcu_read_lock() protection into this patch makes sense. This would avoid introducing the Clang warnings/errors in this patch, and then you may accordingly adjust your subsequent series where the q->queue_lock protection is removed and the blkg lookup paths are protected by RCU.
I also noticed that some paths already take q->queue_lock followed by rcu_read_lock(), so adding the RCU protection here should be consistent with those existing paths.
Thanks,
--Nilay
yu kuai
maintainerRe: [PATCH 2/3] blk-cgroup: store blkcg in bio instead of blkg
Hi,
Quoted 89 lines · Nilay Shroff
Quoted 51 lines · yu kuai
Hi,
Quoted 40 lines · Nilay Shroff
Quoted 19 lines · Yu Kuai
block/blk-cgroup.h +0
-0@@ -257,20 +262,21 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio) * @blkcg: blkcg of interest * @q: request_queue of interest * * Lookup a blkg for the @blkcg - @q pair, whether it is online ordying.
*
- * Must be called in a RCU critical section.
+ * Must be called with either RCU read lock orqueue_lockheld. *
* This does not acquire a reference. The caller must already hold one, or
* have the blkg pinned by I/O.
*/
static inline structblkcg_gq*blkg_lookup_any(struct blkcg *blkcg, structrequest_queue*q)
{
- RCU_LOCKDEP_WARN(!rcu_read_lock_held(),Quoted 8 lines
- "
blkg_lookup_any()requires an RCU read lock"); + RCU_LOCKDEP_WARN(!rcu_read_lock_held() &&
+ !lockdep_is_held(&q->queue_lock),
+ "blkg_lookup_any()requires an RCU read lock orqueue_lock");
returnrhashtable_lookup(&q->blkg_hash, &blkcg->css.id, blkg_hash_params);
}The changes look good to me. However,
rhashtable_lookup()is annotated with__must_hold_shared(RCU). Since we now officially support Clang context annotations in the block layer, compiling with the latest Clang/LLVM generates a number of context warnings, for example:
block/blk-cgroup.h:277:9: warning: calling function 'rhashtable_lookup' requires holding__ctx_lock_RCU'RCU' [-Wthread-safety-analysis] 277 | return rhashtable_lookup(&q->blkg_hash,&blkcg->css.id,
| ^Quoted 7 lines
The same warning is reported from several other block layer source files that include
blk-cgroup.h.This indicates that
rhashtable_lookup()requires the RCU read lock from Clang's context-analysis perspective. Looking at the call paths ofblkg_lookup_any(), it appears that not all callers holdrcu_read_lock()when invoking it.Quoted 6 lines
I was not aware of the above clang warning, this should not be a real problem
becausequeue_lockshould have the same protection as rcu, and I was trying
not to make this patch too complex.I agree that this is not a runtime correctness issue when the lookup is already
protected byq->queue_lock.Quoted 11 lines
I add the
rcu_read_lock()in the following patch to covertqueue_locktoblkcg_mutex:[RFC PATCH v3 3/6] blk-cgroup: protect blkgs with
blkcg_mutex- Yu Kuai
<https://lore.kernel.org/all/[email protected]/>Since this is related to compile warning, I'll move the rcu protection to this
patch.Since
rhashtable_lookup()is annotated with__must_hold_shared(RCU), Clang's
context analysis still reports warnings for these call paths. While this doesn't cause a build failure (and only generates warning) whenCONFIG_WERROR
is not enabled, it could do so for configurations that enableCONFIG_WERROR=y.So IMO, moving the
rcu_read_lock()protection into this patch makes sense.
This would avoid introducing the Clang warnings/errors in this patch, and then
you may accordingly adjust your subsequent series where theq->queue_lockprotection
is removed and the blkg lookup paths are protected by RCU.
Quoted 4 lines
I also noticed that some paths already take
q->queue_lockfollowed byrcu_read_lock(),
so adding the RCU protection here should be consistent with those existing paths.
Turns out Clang will still warn after adding rcu_read_lock(). I'm not familiar how this works, but AI told me this is related to inline helpers, And I'll have to add the same __must_hold_shared(RCU) in rhashtable_lookup() for following inline callers to kill the warning:
static inline struct blkcg_gq *blkg_lookup_any(...)
__must_hold_shared(RCU);
static inline struct blkcg_gq *blkg_lookup(...)
__must_hold_shared(RCU);Quoted 3 lines
Thanks,
--Nilay
Signature · 2 lines
Thanks, Kuai
Re: [PATCH 2/3] blk-cgroup: store blkcg in bio instead of blkg
Quoted 102 lines · yu kuai
Hi,
Quoted 89 lines · Nilay Shroff
Quoted 51 lines · yu kuai
Hi,
Quoted 40 lines · Nilay Shroff
Quoted 19 lines · Yu Kuai
block/blk-cgroup.h +0
-0@@ -257,20 +262,21 @@ static inline bool bio_issue_as_root_blkg(struct bio *bio) * @blkcg: blkcg of interest * @q: request_queue of interest * * Lookup a blkg for the @blkcg - @q pair, whether it is online ordying.
*
- * Must be called in a RCU critical section.
+ * Must be called with either RCU read lock orqueue_lockheld. *
* This does not acquire a reference. The caller must already hold
one, or
* have the blkg pinned by I/O.
*/
static inline structblkcg_gq*blkg_lookup_any(struct blkcg *blkcg, structrequest_queue*q)
{
- RCU_LOCKDEP_WARN(!rcu_read_lock_held(),Quoted 8 lines
- "
blkg_lookup_any()requires an RCU read lock"); + RCU_LOCKDEP_WARN(!rcu_read_lock_held() &&
+ !lockdep_is_held(&q->queue_lock),
+ "blkg_lookup_any()requires an RCU read lock orqueue_lock");
returnrhashtable_lookup(&q->blkg_hash, &blkcg->css.id, blkg_hash_params);
}The changes look good to me. However,
rhashtable_lookup()is annotated with__must_hold_shared(RCU). Since we now officially support Clang context annotations in the block layer, compiling with the latest Clang/LLVM generates a number of context warnings, for example:
block/blk-cgroup.h:277:9: warning: calling function 'rhashtable_lookup' requires holding__ctx_lock_RCU'RCU' [-Wthread-safety-analysis]
277 | return rhashtable_lookup(&q->blkg_hash,&blkcg->css.id,
| ^Quoted 7 lines
The same warning is reported from several other block layer source files that include
blk-cgroup.h.This indicates that
rhashtable_lookup()requires the RCU read lock from Clang's context-analysis perspective. Looking at the call paths ofblkg_lookup_any(), it appears that not all callers holdrcu_read_lock()when invoking it.Quoted 6 lines
I was not aware of the above clang warning, this should not be a real problem
becausequeue_lockshould have the same protection as rcu, and I was trying
not to make this patch too complex.I agree that this is not a runtime correctness issue when the lookup is already
protected byq->queue_lock.Quoted 11 lines
I add the
rcu_read_lock()in the following patch to covertqueue_locktoblkcg_mutex:[RFC PATCH v3 3/6] blk-cgroup: protect blkgs with
blkcg_mutex- Yu Kuai
<https://lore.kernel.org/all/[email protected]/>Since this is related to compile warning, I'll move the rcu protection to this
patch.Since
rhashtable_lookup()is annotated with__must_hold_shared(RCU), Clang's
context analysis still reports warnings for these call paths. While this doesn't cause a build failure (and only generates warning) whenCONFIG_WERROR
is not enabled, it could do so for configurations that enableCONFIG_WERROR=y.So IMO, moving the
rcu_read_lock()protection into this patch makes sense.
This would avoid introducing the Clang warnings/errors in this patch, and then
you may accordingly adjust your subsequent series where theq->queue_lockprotection
is removed and the blkg lookup paths are protected by RCU.Quoted 4 lines
I also noticed that some paths already take
q->queue_lockfollowed byrcu_read_lock(),
so adding the RCU protection here should be consistent with those existing paths.Turns out Clang will still warn after adding
rcu_read_lock(). I'm not familiar how this works, but AI told me this is related to inline helpers, And I'll have to add the same__must_hold_shared(RCU)inrhashtable_lookup()for following inline callers to kill the warning:
Quoted 6 lines
static inline struct
blkcg_gq*blkg_lookup_any(...)
__must_hold_shared(RCU);static inline struct
blkcg_gq*blkg_lookup(...)
__must_hold_shared(RCU);
Yes, that's correct. In this case the context requirement needs to be propagated through the inline helper call chain.
Thanks,
--Nilay
PATCH 3/3 blk-cgroup: move async bio punt state to blkcg
In reply to Yu KuaiFrom: Yu Kuai <[email protected]>
blkcg_punt_bio_submit() currently queues punted bios on blkg->async_bios, so it has to call bio_blkg() to find or create a queue-local blkg. Bios now carry and pin the blkcg css, so punted bio lifetime no longer needs to be anchored by a blkg.
Keeping the punt state in blkg can instantiate a blkg even when no blkcg policy is enabled, just to bounce submission from a shared kthread. Move async_bio_lock, async_bios and async_bio_work to struct blkcg, and queue punted bios on bio_blkcg() for non-root cgroups. Root or unassociated bios are submitted directly.
This preserves the priority-inversion avoidance while preventing blkcg_punt_bio_submit() from creating blkgs that are not needed by any policy.
--- block/blk-cgroup.c | 44 +++++++++++++++++++++----------------------- block/blk-cgroup.h | 14 ++++++-------- 2 files changed, 27 insertions(+), 31 deletions(-)
block/blk-cgroup.c +21 -23
@@ -180,14 +180,10 @@ static void blkg_free(struct blkcg_gq *blkg)static void __blkg_release (struct rcu_head *rcu) { struct blkcg_gq *blkg = container_of(rcu, struct blkcg_gq, rcu_head); -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO -WARN_ON (!bio_list_empty (&blkg->async_bios)); -#endif -blkg_free (blkg); }/* * A group is RCU protected, but having an rcu lock does not mean that one
@@ -226,23 +222,22 @@ static void blkg_release(struct percpu_ref *ref) }#ifdef CONFIG_BLK_CGROUP_PUNT_BIO static struct workqueue_struct *blkcg_punt_bio_wq; -static void blkg_async_bio_workfn (struct work_struct *work) +static void blkcg_async_bio_workfn (struct work_struct *work) { - struct blkcg_gq *blkg = container_of(work, struct blkcg_gq, - async_bio_work); + struct blkcg *blkcg = container_of(work, struct blkcg, async_bio_work);struct bio_list bios =BIO_EMPTY_LIST ;struct bio *bio;struct blk_plug plug;bool need_plug = false; -/* as long as there are pending bios, @blkg can't go away */ -spin_lock (&blkg->async_bio_lock); -bio_list_merge_init (&bios, &blkg->async_bios); -spin_unlock (&blkg->async_bio_lock); +/* as long as there are pending bios, @blkcg can't go away */ +spin_lock (&blkcg->async_bio_lock); +bio_list_merge_init (&bios, &blkcg->async_bios); +spin_unlock (&blkcg->async_bio_lock);/* start plug only when bio_list contains at least 2 bios */ if (bios.head && bios.head->bi_next) { need_plug = true;blk_start_plug (&plug);
@@ -259,19 +254,19 @@ static void blkg_async_bio_workfn(struct work_struct *work)* cgroup. Use this helper instead of submit_bio to punt the actual issuing to * a dedicated per-blkcg work item to avoid such priority inversions. */ void blkcg_punt_bio_submit (struct bio *bio) { -struct blkcg_gq *blkg =bio_blkg (bio); +struct blkcg *blkcg =bio_blkcg (bio); -if (blkg && blkg->parent) { -spin_lock (&blkg->async_bio_lock); -bio_list_add (&blkg->async_bios, bio); -spin_unlock (&blkg->async_bio_lock); -queue_work (blkcg_punt_bio_wq, &blkg->async_bio_work); +if (blkcg &&cgroup_parent (blkcg->css.cgroup)) { +spin_lock (&blkcg->async_bio_lock); +bio_list_add (&blkcg->async_bios, bio); +spin_unlock (&blkcg->async_bio_lock); +queue_work (blkcg_punt_bio_wq, &blkcg->async_bio_work); }else { -/* Never bounce if there is no non-root blkg to queue on. */ +/* Never bounce if there is no non-root blkcg to queue on. */ submit_bio (bio); } }EXPORT_SYMBOL_GPL (blkcg_punt_bio_submit);
@@ -350,15 +345,10 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct gendisk *disk, blkg->q = disk->queue;INIT_LIST_HEAD (&blkg->q_node); blkg->blkcg = blkcg; blkg->blkcg_id = blkcg->css.id; blkg->iostat.blkg = blkg; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO -spin_lock_init (&blkg->async_bio_lock); -bio_list_init (&blkg->async_bios); -INIT_WORK (&blkg->async_bio_work, blkg_async_bio_workfn); -#endif u64_stats_init (&blkg->iostat.sync);for_each_possible_cpu (cpu) {u64_stats_init (&per_cpu_ptr (blkg->iostat_cpu, cpu)->sync);per_cpu_ptr (blkg->iostat_cpu, cpu)->blkg = blkg;
@@ -1388,10 +1378,13 @@ static void blkcg_css_free(struct cgroup_subsys_state *css)if (blkcg->cpd[i]) blkcg_policy[i]->cpd_free_fn (blkcg->cpd[i]);mutex_unlock (&blkcg_pol_mutex); +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO +WARN_ON (!bio_list_empty (&blkcg->async_bios)); +#endif free_percpu (blkcg->lhead);kfree (blkcg); } static struct cgroup_subsys_state *
@@ -1436,10 +1429,15 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_css) }spin_lock_init (&blkcg->lock);refcount_set (&blkcg->online_pin,1 );INIT_HLIST_HEAD (&blkcg->blkg_list); +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO +spin_lock_init (&blkcg->async_bio_lock); +bio_list_init (&blkcg->async_bios); +INIT_WORK (&blkcg->async_bio_work, blkcg_async_bio_workfn); +#endif #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD (&blkcg->cgwb_list);#endif list_add_tail (&blkcg->all_blkcgs_node, &all_blkcgs);
block/blk-cgroup.h +6 -8
@@ -75,18 +75,11 @@ struct blkcg_gq {struct blkg_iostat_set __percpu *iostat_cpu;struct blkg_iostat_set iostat;struct blkg_policy_data *pd[BLKCG_MAX_POLS ]; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO -spinlock_t async_bio_lock; -struct bio_list async_bios; -#endif -union { -struct work_struct async_bio_work; -struct work_struct free_work; - }; +struct work_struct free_work;atomic_t use_delay;atomic64_t delay_nsec;atomic64_t delay_start;u64 last_delay;
@@ -111,10 +104,15 @@ struct blkcg {/* * List of updated percpu blkg_iostat_set's since the last flush. */ struct llist_head __percpu *lhead; +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO +spinlock_t async_bio_lock;/* protects async_bios */ +struct bio_list async_bios; +struct work_struct async_bio_work; +#endif #ifdef CONFIG_BLK_CGROUP_FC_APPID char fc_app_id[FC_APPID_LEN ];#endif #ifdef CONFIG_CGROUP_WRITEBACK struct list_head cgwb_list;
Signature · 1 line
2.51.0
Re: [PATCH 3/3] blk-cgroup: move async bio punt state to blkcg
Looks good:
Re: [PATCH 3/3] blk-cgroup: move async bio punt state to blkcg
In reply to Yu KuaiQuoted 6 lines · Yu Kuai
block/blk-cgroup.h +0
-0@@ -111,10 +104,15 @@ struct blkcg {/* * List of updated percpu blkg_iostat_set's since the last flush. */ struct llist_head __percpu *lhead;
block/blk-cgroup.h struct blkcg { · line 118
Quoted 10 lines
block/blk-cgroup.h +5
-0+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO +spinlock_t async_bio_lock;/* protects async_bios */ +struct bio_list async_bios; +struct work_struct async_bio_work; +#endif #ifdef CONFIG_BLK_CGROUP_FC_APPID char fc_app_id[FC_APPID_LEN ];#endif #ifdef CONFIG_CGROUP_WRITEBACK struct list_head cgwb_list;
As I mentioned, Clang context annotations are now officially supported in the block layer. Since you're touching this code, I suggest annotating async_bios with __guarded_by(&async_bio_lock) as well.
This would allow the Clang thread-safety analyzer to verify that every access to async_bios is properly protected by async_bio_lock.
Thanks,
--Nilay
yu kuai
maintainerRe: [PATCH 3/3] blk-cgroup: move async bio punt state to blkcg
Hi,
Quoted 21 lines · Nilay Shroff
Quoted 7 lines · Yu Kuai
block/blk-cgroup.h +0
-0@@ -111,10 +104,15 @@ struct blkcg { /* * List of updated percpu blkg_iostat_set's since the last flush. */ struct llist_head __percpu *lhead;+#ifdef
CONFIG_BLK_CGROUP_PUNT_BIO
+ spinlock_tasync_bio_lock; /* protectsasync_bios*/Quoted 8 lines
+ struct bio_list
async_bios;
+ struct work_structasync_bio_work; +#endif
#ifdefCONFIG_BLK_CGROUP_FC_APPID
char fc_app_id[FC_APPID_LEN]; #endif
#ifdefCONFIG_CGROUP_WRITEBACK
struct list_headcgwb_list;As I mentioned, Clang context annotations are now officially supported in the block layer. Since you're touching this code, I suggest annotating
async_bioswith__guarded_by(&async_bio_lock)as well.
Quoted 2 lines
This would allow the Clang thread-safety analyzer to verify that every access to
async_biosis properly protected byasync_bio_lock.
Sounds reasonable, will do this in the next version.
Quoted 3 lines
Thanks,
--Nilay
Signature · 2 lines
Thanks, Kuai