blk-mq: Implement runtime power management
Subsystems (2)
v7 cover blk-mq: Implement runtime power management
Hello Jens,
This patch series not only implements runtime power management for blk-mq but also fixes a starvation issue in the power management code for the legacy block layer. Please consider this patch series for the upstream kernel.
Thanks,
Bart.
Changes compared to v6:
- Left out the patches that split RQF_PREEMPT in three flags.
- Left out the patch that introduces the SCSI device state SDEV_SUSPENDED.
- Left out the patch that introduces blk_pm_runtime_exit().
- Restored the patch that changes the PREEMPT_ONLY flag into a counter.
Changes compared to v5:
- Introduced a new flag RQF_DV that replaces RQF_PREEMPT for SCSI domain validation.
- Introduced a new request queue state QUEUE_FLAG_DV_ONLY for SCSI domain validation.
- Instead of using SDEV_QUIESCE for both runtime suspend and SCSI domain validation, use that state for domain validation only and introduce a new state for runtime suspend, namely SDEV_QUIESCE.
- Reallow system suspend during SCSI domain validation.
- Moved the runtime resume call from the request allocation code into blk_queue_enter().
- Instead of relying on q_usage_counter, iterate over the tag set to determine whether or not any requests are in flight.
Changes compared to v4:
- Dropped the patches "Give RQF_PREEMPT back its original meaning" and "Serialize queue freezing and blk_pre_runtime_suspend()".
- Replaced "percpu_ref_read()" with "percpu_is_in_use()".
- Inserted pm_request_resume() calls in the block layer request allocation code such that the context that submits a request no longer has to call pm_runtime_get().
Changes compared to v3:
- Avoid adverse interactions between system-wide suspend/resume and runtime power management by changing the PREEMPT_ONLY flag into a counter.
- Give RQF_PREEMPT back its original meaning, namely that it is only set for ide_preempt requests.
- Remove the flag BLK_MQ_REQ_PREEMPT.
- Removed the pm_request_resume() call.
Changes compared to v2:
- Fixed the build for CONFIG_BLOCK=n.
- Added a patch that introduces percpu_ref_read() in the percpu-counter code.
- Added a patch that makes it easier to detect missing pm_runtime_get*() calls.
- Addressed Jianchao's feedback including the comment about runtime overhead of switching a per-cpu counter to atomic mode.
Changes compared to v1:
- Moved the runtime power management code into a separate file.
- Addressed Ming's feedback.
Bart Van Assche (6):
block: Move power management code into a new source file block, scsi: Change the preempt-only flag into a counter
block: Split blk_pm_add_request() and blk_pm_put_request()
block: Schedule runtime resume earlier
block: Make blk_get_request() block for non-PM requests while
suspended
blk-mq: Enable support for runtime power management
block/Kconfig | 3 + block/Makefile | 1 + block/blk-core.c | 271 +++++----------------------------------- block/blk-mq-debugfs.c | 10 +- block/blk-mq.c | 2 + block/blk-pm.c | 244 ++++++++++++++++++++++++++++++++++++ block/blk-pm.h | 69 ++++++++++ block/elevator.c | 22 +--- drivers/scsi/scsi_lib.c | 11 +- drivers/scsi/scsi_pm.c | 1 + drivers/scsi/sd.c | 1 + drivers/scsi/sr.c | 1 + include/linux/blk-pm.h | 24 ++++ include/linux/blkdev.h | 37 ++---- 14 files changed, 402 insertions(+), 295 deletions(-) create mode 100644 block/blk-pm.c create mode 100644 block/blk-pm.h create mode 100644 include/linux/blk-pm.h
Signature · 1 line
2.18.0
PATCH v7 1/6 block: Move power management code into a new source file
Move the code for runtime power management from blk-core.c into the new source file blk-pm.c. Move the corresponding declarations from <linux/blkdev.h> into <linux/blk-pm.h>. For CONFIG_PM=n, leave out the declarations of the functions that are not used in that mode. This patch not only reduces the number of #ifdefs in the block layer core code but also reduces the size of header file <linux/blkdev.h> and hence should help to reduce the build time of the Linux kernel if CONFIG_PM is not defined.
--- block/Kconfig | 3 + block/Makefile | 1 + block/blk-core.c | 196 +---------------------------------------- block/blk-pm.c | 188 +++++++++++++++++++++++++++++++++++++++ block/blk-pm.h | 43 +++++++++ block/elevator.c | 22 +---- drivers/scsi/scsi_pm.c | 1 + drivers/scsi/sd.c | 1 + drivers/scsi/sr.c | 1 + include/linux/blk-pm.h | 24 +++++ include/linux/blkdev.h | 23 ----- 11 files changed, 264 insertions(+), 239 deletions(-) create mode 100644 block/blk-pm.c create mode 100644 block/blk-pm.h create mode 100644 include/linux/blk-pm.h
block/Kconfig +3 -0
@@ -228,4 +228,7 @@ config BLK_MQ_RDMA depends on BLOCK && INFINIBAND default y +config BLK_PM + def_bool BLOCK && PM + source block/Kconfig.iosched
block/Makefile +1 -0
@@ -37,3 +37,4 @@ obj-$(CONFIG_BLK_WBT) += blk-wbt.o obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o +obj-$(CONFIG_BLK_PM) += blk-pm.o
block/blk-core.c +1 -195
214 lines not on this page · raw message
block/blk-pm.c new file +188 -0
188 lines not on this page · raw message
block/blk-pm.h new file +43 -0
43 lines not on this page · raw message
block/elevator.c +1 -21
@@ -41,6 +41,7 @@#include "blk.h" #include "blk-mq-sched.h" +#include "blk-pm.h" #include "blk-wbt.h" static DEFINE_SPINLOCK (elv_list_lock);
@@ -557,27 +558,6 @@ void elv_bio_merged(struct request_queue *q, struct request *rq, e->type->ops.sq.elevator_bio_merged_fn (q, rq, bio); } -#ifdef CONFIG_PM -static void blk_pm_requeue_request (struct request *rq) -{ -if (rq->q->dev && !(rq->rq_flags &RQF_PM )) - rq->q->nr_pending--; -} - -static void blk_pm_add_request (struct request_queue *q,struct request *rq) -{ -if (q->dev && !(rq->rq_flags &RQF_PM ) && q->nr_pending++ ==0 && - (q->rpm_status ==RPM_SUSPENDED || q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); -} -#else -static inline void blk_pm_requeue_request (struct request *rq) {} -static inline void blk_pm_add_request (struct request_queue *q, -struct request *rq) -{ -} -#endif -void elv_requeue_request (struct request_queue *q,struct request *rq) {/*
drivers/scsi/scsi_pm.c +1 -0
@@ -8,6 +8,7 @@#include <linux/pm_runtime.h> #include <linux/export.h> #include <linux/async.h> +#include <linux/blk-pm.h> #include <scsi/scsi.h> #include <scsi/scsi_device.h>
drivers/scsi/sd.c +1 -0
@@ -45,6 +45,7 @@#include <linux/init.h> #include <linux/blkdev.h> #include <linux/blkpg.h> +#include <linux/blk-pm.h> #include <linux/delay.h> #include <linux/mutex.h> #include <linux/string_helpers.h>
drivers/scsi/sr.c +1 -0
@@ -43,6 +43,7 @@#include <linux/interrupt.h> #include <linux/init.h> #include <linux/blkdev.h> +#include <linux/blk-pm.h> #include <linux/mutex.h> #include <linux/slab.h> #include <linux/pm_runtime.h>
include/linux/blk-pm.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _BLK_PM_H_ +#define _BLK_PM_H_ + +struct device ; +struct request_queue ; + +/* +* block layer runtime pm functions +*/ +#ifdef CONFIG_PM +extern void blk_pm_runtime_init (struct request_queue *q,struct device *dev); +extern int blk_pre_runtime_suspend (struct request_queue *q); +extern void blk_post_runtime_suspend (struct request_queue *q,int err); +extern void blk_pre_runtime_resume (struct request_queue *q); +extern void blk_post_runtime_resume (struct request_queue *q,int err); +extern void blk_set_runtime_active (struct request_queue *q); +#else +static inline void blk_pm_runtime_init (struct request_queue *q, +struct device *dev) {} +#endif + +#endif /* _BLK_PM_H_ */
include/linux/blkdev.h +0 -23
@@ -1280,29 +1280,6 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id,extern void blk_put_queue (struct request_queue *);extern void blk_set_queue_dying (struct request_queue *); -/* -* block layer runtime pm functions -*/ -#ifdef CONFIG_PM -extern void blk_pm_runtime_init (struct request_queue *q,struct device *dev); -extern int blk_pre_runtime_suspend (struct request_queue *q); -extern void blk_post_runtime_suspend (struct request_queue *q,int err); -extern void blk_pre_runtime_resume (struct request_queue *q); -extern void blk_post_runtime_resume (struct request_queue *q,int err); -extern void blk_set_runtime_active (struct request_queue *q); -#else -static inline void blk_pm_runtime_init (struct request_queue *q, -struct device *dev) {} -static inline int blk_pre_runtime_suspend (struct request_queue *q) -{ -return -ENOSYS ; -} -static inline void blk_post_runtime_suspend (struct request_queue *q,int err) {} -static inline void blk_pre_runtime_resume (struct request_queue *q) {} -static inline void blk_post_runtime_resume (struct request_queue *q,int err) {} -static inline void blk_set_runtime_active (struct request_queue *q) {} -#endif -/* * blk_plug permits building a queue of related requests by holding the I/O * fragments for a short period. This allows merging of sequential requests
Signature · 1 line
2.18.0
PATCH v7 2/6 block, scsi: Change the preempt-only flag into a counter
In reply to Bart Van AsscheThe RQF_PREEMPT flag is used for three purposes:
- In the SCSI core, for making sure that power management requests are executed even if a device is in the "quiesced" state.
- For domain validation by SCSI drivers that use the parallel port.
- In the IDE driver, for IDE preempt requests.
Rename "preempt-only" into "pm-only" because the primary purpose of this mode is power management. Since the power management core may but does not have to resume a runtime suspended device before performing system-wide suspend and since a later patch will set "pm-only" mode as long as a block device is runtime suspended, make it possible to set "pm-only" mode from more than one context. Since with this change scsi_device_quiesce() is no longer idempotent, make that function return early if it is called for a quiesced queue.
--- block/blk-core.c | 35 ++++++++++++++++++----------------- block/blk-mq-debugfs.c | 10 +++++++++- drivers/scsi/scsi_lib.c | 11 +++++++---- include/linux/blkdev.h | 14 +++++++++----- 4 files changed, 43 insertions(+), 27 deletions(-)
block/blk-core.c +18 -17
@@ -422,24 +422,25 @@ void blk_sync_queue(struct request_queue *q)EXPORT_SYMBOL (blk_sync_queue);/** -* blk_set_preempt_only - set QUEUE_FLAG_PREEMPT_ONLY +* blk_set_pm_only - increment pm_only counter * @q: request queue pointer -* -* Returns the previous value of the PREEMPT_ONLY flag - 0 if the flag was not -* set and 1 if the flag was already set. */ -int blk_set_preempt_only (struct request_queue *q) +void blk_set_pm_only (struct request_queue *q) { -return blk_queue_flag_test_and_set (QUEUE_FLAG_PREEMPT_ONLY , q); +atomic_inc (&q->pm_only); } -EXPORT_SYMBOL_GPL (blk_set_preempt_only); +EXPORT_SYMBOL_GPL (blk_set_pm_only); -void blk_clear_preempt_only (struct request_queue *q) +void blk_clear_pm_only (struct request_queue *q) { -blk_queue_flag_clear (QUEUE_FLAG_PREEMPT_ONLY , q); -wake_up_all (&q->mq_freeze_wq); +int pm_only; + + pm_only =atomic_dec_return (&q->pm_only); +WARN_ON_ONCE (pm_only <0 ); +if (pm_only ==0 ) +wake_up_all (&q->mq_freeze_wq); } -EXPORT_SYMBOL_GPL (blk_clear_preempt_only); +EXPORT_SYMBOL_GPL (blk_clear_pm_only);/** * __blk_run_queue_uncond - run a queue whether or not it has been stopped
@@ -918,7 +919,7 @@ EXPORT_SYMBOL(blk_alloc_queue);*/ int blk_queue_enter (struct request_queue *q,blk_mq_req_flags_t flags) { -const bool preempt = flags &BLK_MQ_REQ_PREEMPT ; +const bool pm = flags &BLK_MQ_REQ_PREEMPT ;while (true) {bool success = false;
@@ -926,11 +927,11 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)rcu_read_lock ();if (percpu_ref_tryget_live (&q->q_usage_counter)) {/* -* The code that sets the PREEMPT_ONLY flag is -* responsible for ensuring that that flag is globally -* visible before the queue is unfrozen. +* The code that increments the pm_only counter is +* responsible for ensuring that that counter is +* globally visible before the queue is unfrozen. */ -if (preempt || !blk_queue_preempt_only (q)) { +if (pm || !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);
@@ -955,7 +956,7 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)wait_event (q->mq_freeze_wq, (atomic_read (&q->mq_freeze_depth) ==0 && - (preempt || !blk_queue_preempt_only (q))) || + (pm || !blk_queue_pm_only (q))) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;
block/blk-mq-debugfs.c +9 -1
@@ -102,6 +102,14 @@ static int blk_flags_show(struct seq_file *m, const unsigned long flags,return 0 ; } +static int queue_pm_only_show (void *data,struct seq_file *m) +{ +struct request_queue *q = data; + +seq_printf (m,"%d\n" ,atomic_read (&q->pm_only)); +return 0 ; +} +#define QUEUE_FLAG_NAME (name) [QUEUE_FLAG_##name] = #namestatic const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (QUEUED ),
@@ -132,7 +140,6 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (REGISTERED ),QUEUE_FLAG_NAME (SCSI_PASSTHROUGH ), QUEUE_FLAG_NAME(QUIESCED), - QUEUE_FLAG_NAME(PREEMPT_ONLY), };#undef QUEUE_FLAG_NAME
@@ -209,6 +216,7 @@ static ssize_t queue_write_hint_store(void *data, const char __user *buf,static const struct blk_mq_debugfs_attr blk_mq_debugfs_queue_attrs[] = { {"poll_stat" ,0400 , queue_poll_stat_show }, {"requeue_list" ,0400 , .seq_ops = &queue_requeue_list_seq_ops }, + {"pm_only" ,0600 , queue_pm_only_show,NULL }, {"state" ,0600 , queue_state_show, queue_state_write }, {"write_hints" ,0600 , queue_write_hint_show, queue_write_hint_store }, {"zone_wlock" ,0400 , queue_zone_wlock_show,NULL },
drivers/scsi/scsi_lib.c +7 -4
@@ -3046,11 +3046,14 @@ scsi_device_quiesce(struct scsi_device *sdev)*/ WARN_ON_ONCE (sdev->quiesced_by && sdev->quiesced_by != current); -blk_set_preempt_only (q); +if (sdev->quiesced_by == current) +return 0 ; + +blk_set_pm_only (q);blk_mq_freeze_queue (q);/* -* Ensure that the effect of blk_set_preempt_only() will be visible +* Ensure that the effect of blk_set_pm_only() will be visible * for percpu_ref_tryget() callers that occur after the queue * unfreeze even if the queue was already frozen before this function * was called. See also https://lwn.net/Articles/573497/.
@@ -3063,7 +3066,7 @@ scsi_device_quiesce(struct scsi_device *sdev)if (err ==0 ) sdev->quiesced_by = current;else -blk_clear_preempt_only (q); +blk_clear_pm_only (q);mutex_unlock (&sdev->state_mutex);return err;
@@ -3088,7 +3091,7 @@ void scsi_device_resume(struct scsi_device *sdev)mutex_lock (&sdev->state_mutex);WARN_ON_ONCE (!sdev->quiesced_by); sdev->quiesced_by =NULL ; -blk_clear_preempt_only (sdev->request_queue); +blk_clear_pm_only (sdev->request_queue);if (sdev->sdev_state ==SDEV_QUIESCE )scsi_device_set_state (sdev,SDEV_RUNNING );mutex_unlock (&sdev->state_mutex);
include/linux/blkdev.h +9 -5
@@ -504,6 +504,12 @@ struct request_queue {* various queue flags, see QUEUE_* below */ unsigned long queue_flags; +/* +* Number of contexts that have called blk_set_pm_only(). If this +* counter is above zero then only RQF_PM and RQF_PREEMPT requests are +* processed. +*/ +atomic_t pm_only;/* * ida allocated id for this queue. Used to index queues from
@@ -698,7 +704,6 @@ struct request_queue {#define QUEUE_FLAG_REGISTERED 26/* queue has been registered to a disk */ #define QUEUE_FLAG_SCSI_PASSTHROUGH 27/* queue supports SCSI commands */ #define QUEUE_FLAG_QUIESCED 28/* queue has been quiesced */ -#define QUEUE_FLAG_PREEMPT_ONLY 29/* only process REQ_PREEMPT requests */ #define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \ (1 << QUEUE_FLAG_SAME_COMP) | \
@@ -736,12 +741,11 @@ bool blk_queue_flag_test_and_clear(unsigned int flag, struct request_queue *q); ((rq)->cmd_flags & (REQ_FAILFAST_DEV |REQ_FAILFAST_TRANSPORT | \ REQ_FAILFAST_DRIVER))#define blk_queue_quiesced (q) test_bit(QUEUE_FLAG_QUIESCED, &(q)->queue_flags) -#define blk_queue_preempt_only (q) \ - test_bit(QUEUE_FLAG_PREEMPT_ONLY, &(q)->queue_flags) +#define blk_queue_pm_only (q) atomic_read(&(q)->pm_only)#define blk_queue_fua (q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags) -extern int blk_set_preempt_only (struct request_queue *q); -extern void blk_clear_preempt_only (struct request_queue *q); +extern void blk_set_pm_only (struct request_queue *q); +extern void blk_clear_pm_only (struct request_queue *q);static inline int queue_in_flight (struct request_queue *q) {
Signature · 1 line
2.18.0
PATCH v7 3/6 block: Split blk_pm_add_request() and blk_pm_put_request()
In reply to Bart Van AsscheMove the pm_request_resume() and pm_runtime_mark_last_busy() calls into two new functions and thereby separate legacy block layer code from code that works for both the legacy block layer and blk-mq. A later patch will add calls to the new functions in the blk-mq code.
--- block/blk-core.c | 1 + block/blk-pm.h | 36 +++++++++++++++++++++++++++++++----- block/elevator.c | 1 + 3 files changed, 33 insertions(+), 5 deletions(-)
block/blk-core.c +1 -0
@@ -1744,6 +1744,7 @@ void __blk_put_request(struct request_queue *q, struct request *req)blk_req_zone_write_unlock (req);blk_pm_put_request (req); +blk_pm_mark_last_busy (req);elv_completed_request (q, req);
block/blk-pm.h +31 -5
@@ -6,8 +6,23 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM +static inline void blk_pm_request_resume (struct request_queue *q) +{ +if (q->dev && (q->rpm_status ==RPM_SUSPENDED || + q->rpm_status ==RPM_SUSPENDING )) +pm_request_resume (q->dev); +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) +pm_runtime_mark_last_busy (rq->q->dev); +} +static inline void blk_pm_requeue_request (struct request *rq) { +lockdep_assert_held (rq->q->queue_lock); +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) rq->q->nr_pending--; }
@@ -15,17 +30,28 @@ static inline void blk_pm_requeue_request(struct request *rq)static inline void blk_pm_add_request (struct request_queue *q,struct request *rq) { -if (q->dev && !(rq->rq_flags &RQF_PM ) && q->nr_pending++ ==0 && - (q->rpm_status ==RPM_SUSPENDED || q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +lockdep_assert_held (q->queue_lock); + +if (q->dev && !(rq->rq_flags &RQF_PM )) + q->nr_pending++; }static inline void blk_pm_put_request (struct request *rq) { -if (rq->q->dev && !(rq->rq_flags &RQF_PM ) && !--rq->q->nr_pending) -pm_runtime_mark_last_busy (rq->q->dev); +lockdep_assert_held (rq->q->queue_lock); + +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) + --rq->q->nr_pending; }#else +static inline void blk_pm_request_resume (struct request_queue *q) +{ +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +} +static inline void blk_pm_requeue_request (struct request *rq) { }
block/elevator.c +1 -0
@@ -601,6 +601,7 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); +blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.18.0
PATCH v7 4/6 block: Schedule runtime resume earlier
In reply to Bart Van AsscheInstead of scheduling runtime resume of a request queue after a request has been queued, schedule asynchronous resume during request allocation. The new pm_request_resume() calls occur after blk_queue_enter() has increased the q_usage_counter request queue member. This change is needed for a later patch that will make request allocation block while the queue status is not RPM_ACTIVE.
--- block/blk-core.c | 2 ++ block/elevator.c | 1 - 2 files changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +2 -0
@@ -942,6 +942,8 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)if (success)return 0 ; +blk_pm_request_resume (q); +if (flags &BLK_MQ_REQ_NOWAIT )return -EBUSY ;
block/elevator.c +0 -1
@@ -601,7 +601,6 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); -blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.18.0
PATCH v7 5/6 block: Make blk_get_request() block for non-PM requests while suspended
In reply to Bart Van AsscheInstead of allowing requests that are not power management requests to enter the queue in runtime suspended status (RPM_SUSPENDED), make the blk_get_request() caller block. This change fixes a starvation
issue: it is now guaranteed that power management requests will be executed no matter how many blk_get_request() callers are waiting. Instead of maintaining the q->nr_pending counter, rely on q->q_usage_counter. Call pm_runtime_mark_last_busy() every time a request finishes instead of only if the queue depth drops to zero.
--- block/blk-core.c | 37 ++++++------------------- block/blk-pm.c | 72 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 75 insertions(+), 34 deletions(-)
block/blk-core.c +8 -29
@@ -2747,30 +2747,6 @@ void blk_account_io_done(struct request *req, u64 now) } } -#ifdef CONFIG_PM -/* -* Don't process normal requests when queue is suspended -* or in the process of suspending/resuming -*/ -static bool blk_pm_allow_request (struct request *rq) -{ -switch (rq->q->rpm_status) { -case RPM_RESUMING : -case RPM_SUSPENDING : -return rq->rq_flags &RQF_PM ; -case RPM_SUSPENDED : -return false; -default : -return true; - } -} -#else -static bool blk_pm_allow_request (struct request *rq) -{ -return true; -} -#endif -void blk_account_io_start (struct request *rq,bool new_io) {struct hd_struct *part;
@@ -2816,11 +2792,14 @@ static struct request *elv_next_request(struct request_queue *q)while (1 ) {list_for_each_entry (rq, &q->queue_head, queuelist) { -if (blk_pm_allow_request (rq)) -return rq; - -if (rq->rq_flags &RQF_SOFTBARRIER ) -break ; +#ifdef CONFIG_PM +/* +* If a request gets queued in state RPM_SUSPENDED +* then that's a kernel bug. +*/ +WARN_ON_ONCE (q->rpm_status ==RPM_SUSPENDED ); +#endif +return rq; }/*
block/blk-pm.c +67 -5
114 lines not on this page · raw message
Signature · 1 line
2.18.0
Re: [PATCH v7 5/6] block: Make blk_get_request() block for non-PM requests while suspended
Hi Bart
Quoted 7 lines · Bart Van Assche
Instead of allowing requests that are not power management requests to enter the queue in runtime suspended status (
RPM_SUSPENDED), make theblk_get_request()caller block. This change fixes a starvation
issue: it is now guaranteed that power management requests will be executed no matter how manyblk_get_request()callers are waiting. Instead of maintaining theq->nr_pendingcounter, rely onq->q_usage_counter.
Looks like we still depend on this nr_pending for blk-legacy.
Call pm_runtime_mark_last_busy() every time a
Quoted 1 line
request finishes instead of only if the queue depth drops to zero.
...
Quoted 40 lines
(quoted diff) +0
-0/*block/blk-pm.c +22
-0@@ -1,8 +1,11 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/blk-mq.h> #include <linux/blk-pm.h> #include <linux/blkdev.h> #include <linux/pm_runtime.h> +#include "blk-mq.h" +#include "blk-mq-tag.h" /** * blk_pm_runtime_init - Block layer runtime PM initialization routine @@ -40,6 +43,36 @@ void blk_pm_runtime_init(struct request_queue *q, struct device *dev) }EXPORT_SYMBOL (blk_pm_runtime_init); +struct in_flight_data { +struct request_queue *q; +int in_flight; +}; + +static void blk_count_in_flight (struct blk_mq_hw_ctx *hctx,struct request *rq, +void *priv,bool reserved) +{ +struct in_flight_data *in_flight = priv; + +if (rq->q == in_flight->q) + in_flight->in_flight++; +} + +/* +* Count the number of requests that are in flight for request queue @q. Use +* @q->nr_pending for legacy queues. Iterate over the tag set for blk-mq +* queues. Use blk_mq_queue_tag_busy_iter() instead of +* blk_mq_tagset_busy_iter() because the latter only considers requests that
block/blk-pm.c void blk_pm_runtime_init(struct request_queue *q, struct device *dev) · line 74
Quoted 10 lines
block/blk-pm.c +10
-0+* have already been started. +*/ +static int blk_requests_in_flight (struct request_queue *q) +{ +struct in_flight_data in_flight = { .q = q }; + +if (q->mq_ops) +blk_mq_queue_tag_busy_iter (q, blk_count_in_flight, &in_flight); +return q->nr_pending + in_flight.in_flight; +}
blk_mq_queue_tag_busy_iter only accounts the driver tags. This will only work w/o io scheduler
Quoted 5 lines
block/blk-pm.c +1
-0+/** * blk_pre_runtime_suspend - Pre runtime suspend check * @q: the queue of the device
block/blk-pm.c int blk_pre_runtime_suspend(struct request_queue *q) · line 109
Quoted 9 lines
block/blk-pm.c +6
-0@@ -68,14 +101,38 @@ int blk_pre_runtime_suspend(struct request_queue *q)if (!q->dev)return ret; +WARN_ON_ONCE (q->rpm_status !=RPM_ACTIVE ); + +blk_set_pm_only (q); +/* +* This function only gets called if the most recent +* pm_request_resume() call occurred at least autosuspend_delay_ms
^^^^^^^^^^^^^^^^^^^
pm_runtime_mark_last_busy ?
block/blk-pm.c int blk_pre_runtime_suspend(struct request_queue *q) · line 122
Quoted 13 lines
block/blk-pm.c +13
-0+* ago. Since blk_queue_enter() is called by the request allocation +* code before pm_request_resume(), if no requests have a tag assigned +* it is safe to suspend the device. +*/ + ret = -EBUSY ; +if (blk_requests_in_flight (q) ==0 ) { +/* +* Call synchronize_rcu() such that later blk_queue_enter() +* calls see the pm-only state. See also +* https://urldefense.proofpoint.com/v2/url?u=http-3A__lwn.net_Articles_573497_&d=DwIDAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=7WdAxUBeiTUTCy8v-7zXyr4qk7sx26ATvfo6QSTvZyQ&m=o3yai95U5Ge5APoSiMJX64Z8wlI7gf3x0mFnfHpuA4E&s=VPnOWAeo4J4VB944La0uBcynCgVHE-qp52b_9pV8NH4&e=. +*/ +synchronize_rcu (); +if (blk_requests_in_flight (q) ==0 )
Seems not safe here.
For blk-mq path:
Someone may have escaped from the preempt checking, missed the blk_pm_request_resume there, entered into generic_make_request, but have not allocated request and occupied any tag.
There could be a similar scenario for blk-legacy path, the q->nr_pending is increased when request is queued.
So I guess the q_usage_counter checking is still needed here.
Quoted 48 lines
block/blk-pm.c +15
-5+ ret =0 ; + } +spin_lock_irq (q->queue_lock); -if (q->nr_pending) { - ret = -EBUSY ; +if (ret <0 )pm_runtime_mark_last_busy (q->dev); - }else { +else q->rpm_status =RPM_SUSPENDING ; - }spin_unlock_irq (q->queue_lock); + +if (ret) +blk_clear_pm_only (q); +return ret; }EXPORT_SYMBOL (blk_pre_runtime_suspend);@@ -106,6 +163,9 @@ void blk_post_runtime_suspend(struct request_queue *q, int err)pm_runtime_mark_last_busy (q->dev); }spin_unlock_irq (q->queue_lock); + +if (err) +blk_clear_pm_only (q); }EXPORT_SYMBOL (blk_post_runtime_suspend);@@ -153,13 +213,15 @@ void blk_post_runtime_resume(struct request_queue *q, int err)spin_lock_irq (q->queue_lock);if (!err) { q->rpm_status =RPM_ACTIVE ; -__blk_run_queue (q);pm_runtime_mark_last_busy (q->dev);pm_request_autosuspend (q->dev); }else { q->rpm_status =RPM_SUSPENDED ; }spin_unlock_irq (q->queue_lock); + +if (!err) +blk_clear_pm_only (q); }EXPORT_SYMBOL (blk_post_runtime_resume);
Re: [PATCH v7 5/6] block: Make blk_get_request() block for non-PM requests while suspended
Quoted 10 lines · jianchao.wang
Quoted 7 lines · Bart Van Assche
Instead of allowing requests that are not power management requests to enter the queue in runtime suspended status (
RPM_SUSPENDED), make theblk_get_request()caller block. This change fixes a starvation
issue: it is now guaranteed that power management requests will be executed no matter how manyblk_get_request()callers are waiting. Instead of maintaining theq->nr_pendingcounter, rely onq->q_usage_counter.Looks like we still depend on this
nr_pendingfor blk-legacy.
That's right. I will update the commit message.
Quoted 10 lines
blk_mq_queue_tag_busy_iteronly accounts the driver tags. This will only work w/o io schedulerQuoted 8 lines
block/blk-pm.c +1
-0+/** * blk_pre_runtime_suspend - Pre runtime suspend check * @q: the queue of the device @@ -68,14 +101,38 @@ int blk_pre_runtime_suspend(struct request_queue *q)if (!q->dev)return ret;
Quoted 8 lines
block/blk-pm.c int blk_pre_runtime_suspend(struct request_queue *q) · line 109
Quoted 6 lines
block/blk-pm.c +6
-0+WARN_ON_ONCE (q->rpm_status !=RPM_ACTIVE ); + +blk_set_pm_only (q); +/* +* This function only gets called if the most recent +* pm_request_resume() call occurred at least autosuspend_delay_ms ^^^^^^^^^^^^^^^^^^^
pm_runtime_mark_last_busy?
Since every pm_request_resume() call from the block layer is followed by a pm_runtime_mark_last_busy() call and since the latter is called later I think you are right. I will update the comment.
Quoted 16 lines
block/blk-pm.c int blk_pre_runtime_suspend(struct request_queue *q) · line 122
Quoted 13 lines
block/blk-pm.c +13
-0+* ago. Since blk_queue_enter() is called by the request allocation +* code before pm_request_resume(), if no requests have a tag assigned +* it is safe to suspend the device. +*/ + ret = -EBUSY ; +if (blk_requests_in_flight (q) ==0 ) { +/* +* Call synchronize_rcu() such that later blk_queue_enter() +* calls see the pm-only state. See also +* https://urldefense.proofpoint.com/v2/url?u=http-3A__lwn.net_Articles_573497_&d=DwIDAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=7WdAxUBeiTUTCy8v-7zXyr4qk7sx26ATvfo6QSTvZyQ&m=o3yai95U5Ge5APoSiMJX64Z8wlI7gf3x0mFnfHpuA4E&s=VPnOWAeo4J4VB944La0uBcynCgVHE-qp52b_9pV8NH4&e=. +*/ +synchronize_rcu (); +if (blk_requests_in_flight (q) ==0 )Seems not safe here.
Quoted 8 lines
For blk-mq path:
Someone may have escaped from the preempt checking, missed theblk_pm_request_resumethere, entered intogeneric_make_request, but have not allocated request and occupied any tag.There could be a similar scenario for blk-legacy path, the
q->nr_pendingis increased when request is queued.So I guess the
q_usage_counterchecking is still needed here.
There is only one blk_pm_request_resume() call and that call is inside blk_queue_enter() after the pm_only counter has been checked.
For the legacy block layer, nr_pending is increased after the blk_queue_enter() call from inside blk_old_get_request() succeeded.
So I don't see how blk_pm_request_resume() or q->nr_pending++ could escape from the preempt checking?
Thanks,
Bart.
Re: [PATCH v7 5/6] block: Make blk_get_request() block for non-PM requests while suspended
Hi Bart
Quoted 23 lines · Bart Van Assche
Quoted 23 lines
Quoted 5 lines
+ * ago. Since
blk_queue_enter()is called by the request allocation + * code beforepm_request_resume(), if no requests have a tag assigned + * it is safe to suspend the device.
+ */
+ ret =-EBUSY;Quoted 8 lines
+ if (
blk_requests_in_flight(q)== 0) {
+ /*
+ * Callsynchronize_rcu()such that laterblk_queue_enter()
+ * calls see the pm-only state. See also
+ * https://urldefense.proofpoint.com/v2/url?u=http-3A__lwn.net_Articles_573497_&d=DwIDAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=7WdAxUBeiTUTCy8v-7zXyr4qk7sx26ATvfo6QSTvZyQ&m=o3yai95U5Ge5APoSiMJX64Z8wlI7gf3x0mFnfHpuA4E&s=VPnOWAeo4J4VB944La0uBcynCgVHE-qp52b_9pV8NH4&e=.
+ */
+synchronize_rcu();
+ if (blk_requests_in_flight(q)== 0)Seems not safe here.
For blk-mq path:
Someone may have escaped from the preempt checking, missed theblk_pm_request_resumethere, entered intogeneric_make_request, but have not allocated request and occupied any tag.There could be a similar scenario for blk-legacy path, the
q->nr_pendingis increased when request is queued.
Quoted 7 lines
Quoted 1 line
So I guess the
q_usage_counterchecking is still needed here.There is only one
blk_pm_request_resume()call and that call is insideblk_queue_enter()after thepm_onlycounter has been checked.For the legacy block layer,
nr_pendingis increased after theblk_queue_enter()call from insideblk_old_get_request()succeeded.So I don't see how
blk_pm_request_resume()or q->nr_pending++ could escape from the preempt checking?
For example:
blk_pre_runtime_suspend generic_make_request
blk_queue_enter // success here, no blk_pm_request_resume here blk_mq_make_requsetblk_set_pm_only
if (blk_requests_in_flight(q) == 0) {
synchronize_rcu();
if (blk_requests_in_flight(q) == 0)
ret = 0;
}
blk_mq_get_request
Thanks
Jianchao
Re: [PATCH v7 5/6] block: Make blk_get_request() block for non-PM requests while suspended
Quoted 14 lines · jianchao.wang
Quoted 9 lines · Bart Van Assche
There is only one
blk_pm_request_resume()call and that call is insideblk_queue_enter()after thepm_onlycounter has been checked.For the legacy block layer,
nr_pendingis increased after theblk_queue_enter()call from insideblk_old_get_request()succeeded.So I don't see how
blk_pm_request_resume()or q->nr_pending+-+- could escape from the preempt checking?For example:
Quoted 8 lines
blk_pre_runtime_suspendgeneric_make_request
blk_queue_enter//
success here, noblk_pm_request_resumehere
blk_mq_make_requsetblk_set_pm_onlyif (
blk_requests_in_flight(q)==0) {>synchronize_rcu();> if (blk_requests_in_flight(q)==0)
ret =0;> }>blk_mq_get_request
Hello Jianchao,
I think you are right. I will add a q_usage_counter check before setting 'ret' to zero.
Bart.
PATCH v7 6/6 blk-mq: Enable support for runtime power management
In reply to Bart Van AsscheNow that the blk-mq core processes power management requests (marked with RQF_PREEMPT) in other states than RPM_ACTIVE, enable runtime power management for blk-mq.
--- block/blk-mq.c | 2 ++ block/blk-pm.c | 6 ------ 2 files changed, 2 insertions(+), 6 deletions(-)
block/blk-mq.c +2 -0
@@ -33,6 +33,7 @@#include "blk-mq.h" #include "blk-mq-debugfs.h" #include "blk-mq-tag.h" +#include "blk-pm.h" #include "blk-stat.h" #include "blk-mq-sched.h" #include "blk-rq-qos.h"
@@ -475,6 +476,7 @@ static void __blk_mq_free_request(struct request *rq)struct blk_mq_hw_ctx *hctx =blk_mq_map_queue (q, ctx->cpu);const int sched_tag = rq->internal_tag; +blk_pm_mark_last_busy (rq);if (rq->tag !=-1 )blk_mq_put_tag (hctx, hctx->tags, ctx, rq->tag);if (sched_tag !=-1 )
block/blk-pm.c +0 -6
@@ -30,12 +30,6 @@*/ void blk_pm_runtime_init (struct request_queue *q,struct device *dev) { -/* Don't enable runtime PM for blk-mq until it is ready */ -if (q->mq_ops) { -pm_runtime_disable (dev); -return ; - } - q->dev = dev; q->rpm_status =RPM_ACTIVE ;pm_runtime_set_autosuspend_delay (q->dev,-1 );
Signature · 1 line
2.18.0