blk-mq: Implement runtime power management
Subsystems (2)
v7 cover blk-mq: Implement runtime power management
Hello Jens,
This patch series not only implements runtime power management for blk-mq but also fixes a starvation issue in the power management code for the legacy block layer. Please consider this patch series for the upstream kernel.
Thanks,
Bart.
Changes compared to v6:
- Left out the patches that split RQF_PREEMPT in three flags.
- Left out the patch that introduces the SCSI device state SDEV_SUSPENDED.
- Left out the patch that introduces blk_pm_runtime_exit().
- Restored the patch that changes the PREEMPT_ONLY flag into a counter.
Changes compared to v5:
- Introduced a new flag RQF_DV that replaces RQF_PREEMPT for SCSI domain validation.
- Introduced a new request queue state QUEUE_FLAG_DV_ONLY for SCSI domain validation.
- Instead of using SDEV_QUIESCE for both runtime suspend and SCSI domain validation, use that state for domain validation only and introduce a new state for runtime suspend, namely SDEV_QUIESCE.
- Reallow system suspend during SCSI domain validation.
- Moved the runtime resume call from the request allocation code into blk_queue_enter().
- Instead of relying on q_usage_counter, iterate over the tag set to determine whether or not any requests are in flight.
Changes compared to v4:
- Dropped the patches "Give RQF_PREEMPT back its original meaning" and "Serialize queue freezing and blk_pre_runtime_suspend()".
- Replaced "percpu_ref_read()" with "percpu_is_in_use()".
- Inserted pm_request_resume() calls in the block layer request allocation code such that the context that submits a request no longer has to call pm_runtime_get().
Changes compared to v3:
- Avoid adverse interactions between system-wide suspend/resume and runtime power management by changing the PREEMPT_ONLY flag into a counter.
- Give RQF_PREEMPT back its original meaning, namely that it is only set for ide_preempt requests.
- Remove the flag BLK_MQ_REQ_PREEMPT.
- Removed the pm_request_resume() call.
Changes compared to v2:
- Fixed the build for CONFIG_BLOCK=n.
- Added a patch that introduces percpu_ref_read() in the percpu-counter code.
- Added a patch that makes it easier to detect missing pm_runtime_get*() calls.
- Addressed Jianchao's feedback including the comment about runtime overhead of switching a per-cpu counter to atomic mode.
Changes compared to v1:
- Moved the runtime power management code into a separate file.
- Addressed Ming's feedback.
Bart Van Assche (6):
block: Move power management code into a new source file block, scsi: Change the preempt-only flag into a counter
block: Split blk_pm_add_request() and blk_pm_put_request()
block: Schedule runtime resume earlier
block: Make blk_get_request() block for non-PM requests while
suspended
blk-mq: Enable support for runtime power management
block/Kconfig | 3 + block/Makefile | 1 + block/blk-core.c | 271 +++++----------------------------------- block/blk-mq-debugfs.c | 10 +- block/blk-mq.c | 2 + block/blk-pm.c | 244 ++++++++++++++++++++++++++++++++++++ block/blk-pm.h | 69 ++++++++++ block/elevator.c | 22 +--- drivers/scsi/scsi_lib.c | 11 +- drivers/scsi/scsi_pm.c | 1 + drivers/scsi/sd.c | 1 + drivers/scsi/sr.c | 1 + include/linux/blk-pm.h | 24 ++++ include/linux/blkdev.h | 37 ++---- 14 files changed, 402 insertions(+), 295 deletions(-) create mode 100644 block/blk-pm.c create mode 100644 block/blk-pm.h create mode 100644 include/linux/blk-pm.h
Signature · 1 line
2.18.0
PATCH v7 1/6 block: Move power management code into a new source file
Move the code for runtime power management from blk-core.c into the new source file blk-pm.c. Move the corresponding declarations from <linux/blkdev.h> into <linux/blk-pm.h>. For CONFIG_PM=n, leave out the declarations of the functions that are not used in that mode. This patch not only reduces the number of #ifdefs in the block layer core code but also reduces the size of header file <linux/blkdev.h> and hence should help to reduce the build time of the Linux kernel if CONFIG_PM is not defined.
--- block/Kconfig | 3 + block/Makefile | 1 + block/blk-core.c | 196 +---------------------------------------- block/blk-pm.c | 188 +++++++++++++++++++++++++++++++++++++++ block/blk-pm.h | 43 +++++++++ block/elevator.c | 22 +---- drivers/scsi/scsi_pm.c | 1 + drivers/scsi/sd.c | 1 + drivers/scsi/sr.c | 1 + include/linux/blk-pm.h | 24 +++++ include/linux/blkdev.h | 23 ----- 11 files changed, 264 insertions(+), 239 deletions(-) create mode 100644 block/blk-pm.c create mode 100644 block/blk-pm.h create mode 100644 include/linux/blk-pm.h
block/Kconfig +3 -0
@@ -228,4 +228,7 @@ config BLK_MQ_RDMA depends on BLOCK && INFINIBAND default y +config BLK_PM + def_bool BLOCK && PM + source block/Kconfig.iosched
block/Makefile +1 -0
@@ -37,3 +37,4 @@ obj-$(CONFIG_BLK_WBT) += blk-wbt.o obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o +obj-$(CONFIG_BLK_PM) += blk-pm.o
block/blk-core.c +1 -195
214 lines not on this page · raw message
block/blk-pm.c new file +188 -0
188 lines not on this page · raw message
block/blk-pm.h new file +43 -0
43 lines not on this page · raw message
block/elevator.c +1 -21
@@ -41,6 +41,7 @@#include "blk.h" #include "blk-mq-sched.h" +#include "blk-pm.h" #include "blk-wbt.h" static DEFINE_SPINLOCK (elv_list_lock);
@@ -557,27 +558,6 @@ void elv_bio_merged(struct request_queue *q, struct request *rq, e->type->ops.sq.elevator_bio_merged_fn (q, rq, bio); } -#ifdef CONFIG_PM -static void blk_pm_requeue_request (struct request *rq) -{ -if (rq->q->dev && !(rq->rq_flags &RQF_PM )) - rq->q->nr_pending--; -} - -static void blk_pm_add_request (struct request_queue *q,struct request *rq) -{ -if (q->dev && !(rq->rq_flags &RQF_PM ) && q->nr_pending++ ==0 && - (q->rpm_status ==RPM_SUSPENDED || q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); -} -#else -static inline void blk_pm_requeue_request (struct request *rq) {} -static inline void blk_pm_add_request (struct request_queue *q, -struct request *rq) -{ -} -#endif -void elv_requeue_request (struct request_queue *q,struct request *rq) {/*
drivers/scsi/scsi_pm.c +1 -0
@@ -8,6 +8,7 @@#include <linux/pm_runtime.h> #include <linux/export.h> #include <linux/async.h> +#include <linux/blk-pm.h> #include <scsi/scsi.h> #include <scsi/scsi_device.h>
drivers/scsi/sd.c +1 -0
@@ -45,6 +45,7 @@#include <linux/init.h> #include <linux/blkdev.h> #include <linux/blkpg.h> +#include <linux/blk-pm.h> #include <linux/delay.h> #include <linux/mutex.h> #include <linux/string_helpers.h>
drivers/scsi/sr.c +1 -0
@@ -43,6 +43,7 @@#include <linux/interrupt.h> #include <linux/init.h> #include <linux/blkdev.h> +#include <linux/blk-pm.h> #include <linux/mutex.h> #include <linux/slab.h> #include <linux/pm_runtime.h>
include/linux/blk-pm.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _BLK_PM_H_ +#define _BLK_PM_H_ + +struct device ; +struct request_queue ; + +/* +* block layer runtime pm functions +*/ +#ifdef CONFIG_PM +extern void blk_pm_runtime_init (struct request_queue *q,struct device *dev); +extern int blk_pre_runtime_suspend (struct request_queue *q); +extern void blk_post_runtime_suspend (struct request_queue *q,int err); +extern void blk_pre_runtime_resume (struct request_queue *q); +extern void blk_post_runtime_resume (struct request_queue *q,int err); +extern void blk_set_runtime_active (struct request_queue *q); +#else +static inline void blk_pm_runtime_init (struct request_queue *q, +struct device *dev) {} +#endif + +#endif /* _BLK_PM_H_ */
include/linux/blkdev.h +0 -23
@@ -1280,29 +1280,6 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id,extern void blk_put_queue (struct request_queue *);extern void blk_set_queue_dying (struct request_queue *); -/* -* block layer runtime pm functions -*/ -#ifdef CONFIG_PM -extern void blk_pm_runtime_init (struct request_queue *q,struct device *dev); -extern int blk_pre_runtime_suspend (struct request_queue *q); -extern void blk_post_runtime_suspend (struct request_queue *q,int err); -extern void blk_pre_runtime_resume (struct request_queue *q); -extern void blk_post_runtime_resume (struct request_queue *q,int err); -extern void blk_set_runtime_active (struct request_queue *q); -#else -static inline void blk_pm_runtime_init (struct request_queue *q, -struct device *dev) {} -static inline int blk_pre_runtime_suspend (struct request_queue *q) -{ -return -ENOSYS ; -} -static inline void blk_post_runtime_suspend (struct request_queue *q,int err) {} -static inline void blk_pre_runtime_resume (struct request_queue *q) {} -static inline void blk_post_runtime_resume (struct request_queue *q,int err) {} -static inline void blk_set_runtime_active (struct request_queue *q) {} -#endif -/* * blk_plug permits building a queue of related requests by holding the I/O * fragments for a short period. This allows merging of sequential requests
Signature · 1 line
2.18.0
PATCH v7 2/6 block, scsi: Change the preempt-only flag into a counter
In reply to Bart Van AsscheThe RQF_PREEMPT flag is used for three purposes:
- In the SCSI core, for making sure that power management requests are executed even if a device is in the "quiesced" state.
- For domain validation by SCSI drivers that use the parallel port.
- In the IDE driver, for IDE preempt requests.
Rename "preempt-only" into "pm-only" because the primary purpose of this mode is power management. Since the power management core may but does not have to resume a runtime suspended device before performing system-wide suspend and since a later patch will set "pm-only" mode as long as a block device is runtime suspended, make it possible to set "pm-only" mode from more than one context. Since with this change scsi_device_quiesce() is no longer idempotent, make that function return early if it is called for a quiesced queue.
--- block/blk-core.c | 35 ++++++++++++++++++----------------- block/blk-mq-debugfs.c | 10 +++++++++- drivers/scsi/scsi_lib.c | 11 +++++++---- include/linux/blkdev.h | 14 +++++++++----- 4 files changed, 43 insertions(+), 27 deletions(-)
block/blk-core.c +18 -17
@@ -422,24 +422,25 @@ void blk_sync_queue(struct request_queue *q)EXPORT_SYMBOL (blk_sync_queue);/** -* blk_set_preempt_only - set QUEUE_FLAG_PREEMPT_ONLY +* blk_set_pm_only - increment pm_only counter * @q: request queue pointer -* -* Returns the previous value of the PREEMPT_ONLY flag - 0 if the flag was not -* set and 1 if the flag was already set. */ -int blk_set_preempt_only (struct request_queue *q) +void blk_set_pm_only (struct request_queue *q) { -return blk_queue_flag_test_and_set (QUEUE_FLAG_PREEMPT_ONLY , q); +atomic_inc (&q->pm_only); } -EXPORT_SYMBOL_GPL (blk_set_preempt_only); +EXPORT_SYMBOL_GPL (blk_set_pm_only); -void blk_clear_preempt_only (struct request_queue *q) +void blk_clear_pm_only (struct request_queue *q) { -blk_queue_flag_clear (QUEUE_FLAG_PREEMPT_ONLY , q); -wake_up_all (&q->mq_freeze_wq); +int pm_only; + + pm_only =atomic_dec_return (&q->pm_only); +WARN_ON_ONCE (pm_only <0 ); +if (pm_only ==0 ) +wake_up_all (&q->mq_freeze_wq); } -EXPORT_SYMBOL_GPL (blk_clear_preempt_only); +EXPORT_SYMBOL_GPL (blk_clear_pm_only);/** * __blk_run_queue_uncond - run a queue whether or not it has been stopped
@@ -918,7 +919,7 @@ EXPORT_SYMBOL(blk_alloc_queue);*/ int blk_queue_enter (struct request_queue *q,blk_mq_req_flags_t flags) { -const bool preempt = flags &BLK_MQ_REQ_PREEMPT ; +const bool pm = flags &BLK_MQ_REQ_PREEMPT ;while (true) {bool success = false;
@@ -926,11 +927,11 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)rcu_read_lock ();if (percpu_ref_tryget_live (&q->q_usage_counter)) {/* -* The code that sets the PREEMPT_ONLY flag is -* responsible for ensuring that that flag is globally -* visible before the queue is unfrozen. +* The code that increments the pm_only counter is +* responsible for ensuring that that counter is +* globally visible before the queue is unfrozen. */ -if (preempt || !blk_queue_preempt_only (q)) { +if (pm || !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);
@@ -955,7 +956,7 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)wait_event (q->mq_freeze_wq, (atomic_read (&q->mq_freeze_depth) ==0 && - (preempt || !blk_queue_preempt_only (q))) || + (pm || !blk_queue_pm_only (q))) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;
block/blk-mq-debugfs.c +9 -1
@@ -102,6 +102,14 @@ static int blk_flags_show(struct seq_file *m, const unsigned long flags,return 0 ; } +static int queue_pm_only_show (void *data,struct seq_file *m) +{ +struct request_queue *q = data; + +seq_printf (m,"%d\n" ,atomic_read (&q->pm_only)); +return 0 ; +} +#define QUEUE_FLAG_NAME (name) [QUEUE_FLAG_##name] = #namestatic const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (QUEUED ),
@@ -132,7 +140,6 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (REGISTERED ),QUEUE_FLAG_NAME (SCSI_PASSTHROUGH ), QUEUE_FLAG_NAME(QUIESCED), - QUEUE_FLAG_NAME(PREEMPT_ONLY), };#undef QUEUE_FLAG_NAME
@@ -209,6 +216,7 @@ static ssize_t queue_write_hint_store(void *data, const char __user *buf,static const struct blk_mq_debugfs_attr blk_mq_debugfs_queue_attrs[] = { {"poll_stat" ,0400 , queue_poll_stat_show }, {"requeue_list" ,0400 , .seq_ops = &queue_requeue_list_seq_ops }, + {"pm_only" ,0600 , queue_pm_only_show,NULL }, {"state" ,0600 , queue_state_show, queue_state_write }, {"write_hints" ,0600 , queue_write_hint_show, queue_write_hint_store }, {"zone_wlock" ,0400 , queue_zone_wlock_show,NULL },
drivers/scsi/scsi_lib.c +7 -4
@@ -3046,11 +3046,14 @@ scsi_device_quiesce(struct scsi_device *sdev)*/ WARN_ON_ONCE (sdev->quiesced_by && sdev->quiesced_by != current); -blk_set_preempt_only (q); +if (sdev->quiesced_by == current) +return 0 ; + +blk_set_pm_only (q);blk_mq_freeze_queue (q);/* -* Ensure that the effect of blk_set_preempt_only() will be visible +* Ensure that the effect of blk_set_pm_only() will be visible * for percpu_ref_tryget() callers that occur after the queue * unfreeze even if the queue was already frozen before this function * was called. See also https://lwn.net/Articles/573497/.
@@ -3063,7 +3066,7 @@ scsi_device_quiesce(struct scsi_device *sdev)if (err ==0 ) sdev->quiesced_by = current;else -blk_clear_preempt_only (q); +blk_clear_pm_only (q);mutex_unlock (&sdev->state_mutex);return err;
@@ -3088,7 +3091,7 @@ void scsi_device_resume(struct scsi_device *sdev)mutex_lock (&sdev->state_mutex);WARN_ON_ONCE (!sdev->quiesced_by); sdev->quiesced_by =NULL ; -blk_clear_preempt_only (sdev->request_queue); +blk_clear_pm_only (sdev->request_queue);if (sdev->sdev_state ==SDEV_QUIESCE )scsi_device_set_state (sdev,SDEV_RUNNING );mutex_unlock (&sdev->state_mutex);
include/linux/blkdev.h +9 -5
@@ -504,6 +504,12 @@ struct request_queue {* various queue flags, see QUEUE_* below */ unsigned long queue_flags; +/* +* Number of contexts that have called blk_set_pm_only(). If this +* counter is above zero then only RQF_PM and RQF_PREEMPT requests are +* processed. +*/ +atomic_t pm_only;/* * ida allocated id for this queue. Used to index queues from
@@ -698,7 +704,6 @@ struct request_queue {#define QUEUE_FLAG_REGISTERED 26/* queue has been registered to a disk */ #define QUEUE_FLAG_SCSI_PASSTHROUGH 27/* queue supports SCSI commands */ #define QUEUE_FLAG_QUIESCED 28/* queue has been quiesced */ -#define QUEUE_FLAG_PREEMPT_ONLY 29/* only process REQ_PREEMPT requests */ #define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \ (1 << QUEUE_FLAG_SAME_COMP) | \
@@ -736,12 +741,11 @@ bool blk_queue_flag_test_and_clear(unsigned int flag, struct request_queue *q); ((rq)->cmd_flags & (REQ_FAILFAST_DEV |REQ_FAILFAST_TRANSPORT | \ REQ_FAILFAST_DRIVER))#define blk_queue_quiesced (q) test_bit(QUEUE_FLAG_QUIESCED, &(q)->queue_flags) -#define blk_queue_preempt_only (q) \ - test_bit(QUEUE_FLAG_PREEMPT_ONLY, &(q)->queue_flags) +#define blk_queue_pm_only (q) atomic_read(&(q)->pm_only)#define blk_queue_fua (q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags) -extern int blk_set_preempt_only (struct request_queue *q); -extern void blk_clear_preempt_only (struct request_queue *q); +extern void blk_set_pm_only (struct request_queue *q); +extern void blk_clear_pm_only (struct request_queue *q);static inline int queue_in_flight (struct request_queue *q) {
Signature · 1 line
2.18.0
PATCH v7 3/6 block: Split blk_pm_add_request() and blk_pm_put_request()
In reply to Bart Van AsscheMove the pm_request_resume() and pm_runtime_mark_last_busy() calls into two new functions and thereby separate legacy block layer code from code that works for both the legacy block layer and blk-mq. A later patch will add calls to the new functions in the blk-mq code.
--- block/blk-core.c | 1 + block/blk-pm.h | 36 +++++++++++++++++++++++++++++++----- block/elevator.c | 1 + 3 files changed, 33 insertions(+), 5 deletions(-)
block/blk-core.c +1 -0
@@ -1744,6 +1744,7 @@ void __blk_put_request(struct request_queue *q, struct request *req)blk_req_zone_write_unlock (req);blk_pm_put_request (req); +blk_pm_mark_last_busy (req);elv_completed_request (q, req);
block/blk-pm.h +31 -5
@@ -6,8 +6,23 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM +static inline void blk_pm_request_resume (struct request_queue *q) +{ +if (q->dev && (q->rpm_status ==RPM_SUSPENDED || + q->rpm_status ==RPM_SUSPENDING )) +pm_request_resume (q->dev); +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) +pm_runtime_mark_last_busy (rq->q->dev); +} +static inline void blk_pm_requeue_request (struct request *rq) { +lockdep_assert_held (rq->q->queue_lock); +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) rq->q->nr_pending--; }
@@ -15,17 +30,28 @@ static inline void blk_pm_requeue_request(struct request *rq)static inline void blk_pm_add_request (struct request_queue *q,struct request *rq) { -if (q->dev && !(rq->rq_flags &RQF_PM ) && q->nr_pending++ ==0 && - (q->rpm_status ==RPM_SUSPENDED || q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +lockdep_assert_held (q->queue_lock); + +if (q->dev && !(rq->rq_flags &RQF_PM )) + q->nr_pending++; }static inline void blk_pm_put_request (struct request *rq) { -if (rq->q->dev && !(rq->rq_flags &RQF_PM ) && !--rq->q->nr_pending) -pm_runtime_mark_last_busy (rq->q->dev); +lockdep_assert_held (rq->q->queue_lock); + +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) + --rq->q->nr_pending; }#else +static inline void blk_pm_request_resume (struct request_queue *q) +{ +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +} +static inline void blk_pm_requeue_request (struct request *rq) { }
block/elevator.c +1 -0
@@ -601,6 +601,7 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); +blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.18.0
PATCH v7 3/7 block, scsi: Introduce blk_pm_runtime_exit()
In reply to Bart Van AsscheSince it is possible to unbind a SCSI ULD and since unbinding removes the association between a request queue and struct device, the q->dev pointer has to be reset during unbind. Hence introduce a function in the block layer that clears request_queue.dev.
--- block/blk-pm.c | 18 ++++++++++++++++++ drivers/scsi/sd.c | 9 ++++----- drivers/scsi/sr.c | 3 ++- include/linux/blk-pm.h | 2 ++ 4 files changed, 26 insertions(+), 6 deletions(-)
block/blk-pm.c +18 -0
@@ -40,6 +40,24 @@ void blk_pm_runtime_init(struct request_queue *q, struct device *dev) }EXPORT_SYMBOL (blk_pm_runtime_init); +/** +* blk_pm_runtime_exit - runtime PM exit routine +* @q: the queue of the device +* +* This function should be called from the device_driver.remove() callback +* function to avoid that further calls to runtime power management functions +* occur. +*/ +void blk_pm_runtime_exit (struct request_queue *q) +{ +if (!q->dev) +return ; + +pm_runtime_get_sync (q->dev); + q->dev =NULL ; +} +EXPORT_SYMBOL (blk_pm_runtime_exit); +/** * blk_pre_runtime_suspend - Pre runtime suspend check * @q: the queue of the device
drivers/scsi/sd.c +4 -5
@@ -3408,12 +3408,11 @@ static int sd_probe(struct device *dev)**/ static int sd_remove (struct device *dev) { -struct scsi_disk *sdkp; -dev_t devt; +struct scsi_disk *sdkp =dev_get_drvdata (dev); +struct scsi_device *sdp = sdkp->device; +dev_t devt =disk_devt (sdkp->disk); - sdkp =dev_get_drvdata (dev); - devt =disk_devt (sdkp->disk); -scsi_autopm_get_device (sdkp->device); +blk_pm_runtime_exit (sdp->request_queue);async_synchronize_full_domain (&scsi_sd_pm_domain);async_synchronize_full_domain (&scsi_sd_probe_domain);
drivers/scsi/sr.c +2 -1
@@ -1015,8 +1015,9 @@ static void sr_kref_release(struct kref *kref)static int sr_remove (struct device *dev) {struct scsi_cd *cd =dev_get_drvdata (dev); +struct scsi_device *sdev = cd->device; -scsi_autopm_get_device (cd->device); +blk_pm_runtime_exit (sdev->request_queue);del_gendisk (cd->disk);dev_set_drvdata (dev,NULL );
include/linux/blk-pm.h +2 -0
@@ -11,6 +11,7 @@ struct request_queue;*/ #ifdef CONFIG_PM extern void blk_pm_runtime_init (struct request_queue *q,struct device *dev); +extern void blk_pm_runtime_exit (struct request_queue *q);extern int blk_pre_runtime_suspend (struct request_queue *q);extern void blk_post_runtime_suspend (struct request_queue *q,int err);extern void blk_pre_runtime_resume (struct request_queue *q);
@@ -19,6 +20,7 @@ extern void blk_set_runtime_active(struct request_queue *q);#else static inline void blk_pm_runtime_init (struct request_queue *q,struct device *dev) {} +static inline void blk_pm_runtime_exit (struct request_queue *q) {}#endif #endif /* _BLK_PM_H_ */
Signature · 1 line
2.18.0
PATCH v7 4/6 block: Schedule runtime resume earlier
In reply to Bart Van AsscheInstead of scheduling runtime resume of a request queue after a request has been queued, schedule asynchronous resume during request allocation. The new pm_request_resume() calls occur after blk_queue_enter() has increased the q_usage_counter request queue member. This change is needed for a later patch that will make request allocation block while the queue status is not RPM_ACTIVE.
--- block/blk-core.c | 2 ++ block/elevator.c | 1 - 2 files changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +2 -0
@@ -942,6 +942,8 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)if (success)return 0 ; +blk_pm_request_resume (q); +if (flags &BLK_MQ_REQ_NOWAIT )return -EBUSY ;
block/elevator.c +0 -1
@@ -601,7 +601,6 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); -blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.18.0
PATCH v7 4/7 block: Split blk_pm_add_request() and blk_pm_put_request()
In reply to Bart Van AsscheMove the pm_request_resume() and pm_runtime_mark_last_busy() calls into two new functions and thereby separate legacy block layer code from code that works for both the legacy block layer and blk-mq. A later patch will add calls to the new functions in the blk-mq code.
--- block/blk-core.c | 1 + block/blk-pm.h | 36 +++++++++++++++++++++++++++++++----- block/elevator.c | 1 + 3 files changed, 33 insertions(+), 5 deletions(-)
block/blk-core.c +1 -0
@@ -1744,6 +1744,7 @@ void __blk_put_request(struct request_queue *q, struct request *req)blk_req_zone_write_unlock (req);blk_pm_put_request (req); +blk_pm_mark_last_busy (req);elv_completed_request (q, req);
block/blk-pm.h +31 -5
@@ -6,8 +6,23 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM +static inline void blk_pm_request_resume (struct request_queue *q) +{ +if (q->dev && (q->rpm_status ==RPM_SUSPENDED || + q->rpm_status ==RPM_SUSPENDING )) +pm_request_resume (q->dev); +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) +pm_runtime_mark_last_busy (rq->q->dev); +} +static inline void blk_pm_requeue_request (struct request *rq) { +lockdep_assert_held (rq->q->queue_lock); +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) rq->q->nr_pending--; }
@@ -15,17 +30,28 @@ static inline void blk_pm_requeue_request(struct request *rq)static inline void blk_pm_add_request (struct request_queue *q,struct request *rq) { -if (q->dev && !(rq->rq_flags &RQF_PM ) && q->nr_pending++ ==0 && - (q->rpm_status ==RPM_SUSPENDED || q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +lockdep_assert_held (q->queue_lock); + +if (q->dev && !(rq->rq_flags &RQF_PM )) + q->nr_pending++; }static inline void blk_pm_put_request (struct request *rq) { -if (rq->q->dev && !(rq->rq_flags &RQF_PM ) && !--rq->q->nr_pending) -pm_runtime_mark_last_busy (rq->q->dev); +lockdep_assert_held (rq->q->queue_lock); + +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) + --rq->q->nr_pending; }#else +static inline void blk_pm_request_resume (struct request_queue *q) +{ +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +} +static inline void blk_pm_requeue_request (struct request *rq) { }
block/elevator.c +1 -0
@@ -601,6 +601,7 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); +blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.18.0
PATCH v7 5/6 block: Make blk_get_request() block for non-PM requests while suspended
In reply to Bart Van AsscheInstead of allowing requests that are not power management requests to enter the queue in runtime suspended status (RPM_SUSPENDED), make the blk_get_request() caller block. This change fixes a starvation
issue: it is now guaranteed that power management requests will be executed no matter how many blk_get_request() callers are waiting. Instead of maintaining the q->nr_pending counter, rely on q->q_usage_counter. Call pm_runtime_mark_last_busy() every time a request finishes instead of only if the queue depth drops to zero.
--- block/blk-core.c | 37 ++++++------------------- block/blk-pm.c | 72 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 75 insertions(+), 34 deletions(-)
block/blk-core.c +8 -29
@@ -2747,30 +2747,6 @@ void blk_account_io_done(struct request *req, u64 now) } } -#ifdef CONFIG_PM -/* -* Don't process normal requests when queue is suspended -* or in the process of suspending/resuming -*/ -static bool blk_pm_allow_request (struct request *rq) -{ -switch (rq->q->rpm_status) { -case RPM_RESUMING : -case RPM_SUSPENDING : -return rq->rq_flags &RQF_PM ; -case RPM_SUSPENDED : -return false; -default : -return true; - } -} -#else -static bool blk_pm_allow_request (struct request *rq) -{ -return true; -} -#endif -void blk_account_io_start (struct request *rq,bool new_io) {struct hd_struct *part;
@@ -2816,11 +2792,14 @@ static struct request *elv_next_request(struct request_queue *q)while (1 ) {list_for_each_entry (rq, &q->queue_head, queuelist) { -if (blk_pm_allow_request (rq)) -return rq; - -if (rq->rq_flags &RQF_SOFTBARRIER ) -break ; +#ifdef CONFIG_PM +/* +* If a request gets queued in state RPM_SUSPENDED +* then that's a kernel bug. +*/ +WARN_ON_ONCE (q->rpm_status ==RPM_SUSPENDED ); +#endif +return rq; }/*
block/blk-pm.c +67 -5
114 lines not on this page · raw message
Signature · 1 line
2.18.0
PATCH v7 5/7 block: Schedule runtime resume earlier
In reply to Bart Van AsscheInstead of scheduling runtime resume of a request queue after a request has been queued, schedule asynchronous resume during request allocation. The new pm_request_resume() calls occur after blk_queue_enter() has increased the q_usage_counter request queue member. This change is needed for a later patch that will make request allocation block while the queue status is not RPM_ACTIVE.
--- block/blk-core.c | 2 ++ block/elevator.c | 1 - 2 files changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +2 -0
@@ -942,6 +942,8 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)if (success)return 0 ; +blk_pm_request_resume (q); +if (flags &BLK_MQ_REQ_NOWAIT )return -EBUSY ;
block/elevator.c +0 -1
@@ -601,7 +601,6 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); -blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.18.0
PATCH v7 6/6 blk-mq: Enable support for runtime power management
In reply to Bart Van AsscheNow that the blk-mq core processes power management requests (marked with RQF_PREEMPT) in other states than RPM_ACTIVE, enable runtime power management for blk-mq.
--- block/blk-mq.c | 2 ++ block/blk-pm.c | 6 ------ 2 files changed, 2 insertions(+), 6 deletions(-)
block/blk-mq.c +2 -0
@@ -33,6 +33,7 @@#include "blk-mq.h" #include "blk-mq-debugfs.h" #include "blk-mq-tag.h" +#include "blk-pm.h" #include "blk-stat.h" #include "blk-mq-sched.h" #include "blk-rq-qos.h"
@@ -475,6 +476,7 @@ static void __blk_mq_free_request(struct request *rq)struct blk_mq_hw_ctx *hctx =blk_mq_map_queue (q, ctx->cpu);const int sched_tag = rq->internal_tag; +blk_pm_mark_last_busy (rq);if (rq->tag !=-1 )blk_mq_put_tag (hctx, hctx->tags, ctx, rq->tag);if (sched_tag !=-1 )
block/blk-pm.c +0 -6
@@ -30,12 +30,6 @@*/ void blk_pm_runtime_init (struct request_queue *q,struct device *dev) { -/* Don't enable runtime PM for blk-mq until it is ready */ -if (q->mq_ops) { -pm_runtime_disable (dev); -return ; - } - q->dev = dev; q->rpm_status =RPM_ACTIVE ;pm_runtime_set_autosuspend_delay (q->dev,-1 );
Signature · 1 line
2.18.0
PATCH v7 6/7 block: Make blk_get_request() block for non-PM requests while suspended
In reply to Bart Van AsscheInstead of allowing requests that are not power management requests to enter the queue in runtime suspended status (RPM_SUSPENDED), make the blk_get_request() caller block. This change fixes a starvation
issue: it is now guaranteed that power management requests will be executed no matter how many blk_get_request() callers are waiting. Instead of maintaining the q->nr_pending counter, rely on q->q_usage_counter. Call pm_runtime_mark_last_busy() every time a request finishes instead of only if the queue depth drops to zero.
--- block/blk-core.c | 37 ++++++------------------- block/blk-pm.c | 72 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 75 insertions(+), 34 deletions(-)
block/blk-core.c +8 -29
@@ -2747,30 +2747,6 @@ void blk_account_io_done(struct request *req, u64 now) } } -#ifdef CONFIG_PM -/* -* Don't process normal requests when queue is suspended -* or in the process of suspending/resuming -*/ -static bool blk_pm_allow_request (struct request *rq) -{ -switch (rq->q->rpm_status) { -case RPM_RESUMING : -case RPM_SUSPENDING : -return rq->rq_flags &RQF_PM ; -case RPM_SUSPENDED : -return false; -default : -return true; - } -} -#else -static bool blk_pm_allow_request (struct request *rq) -{ -return true; -} -#endif -void blk_account_io_start (struct request *rq,bool new_io) {struct hd_struct *part;
@@ -2816,11 +2792,14 @@ static struct request *elv_next_request(struct request_queue *q)while (1 ) {list_for_each_entry (rq, &q->queue_head, queuelist) { -if (blk_pm_allow_request (rq)) -return rq; - -if (rq->rq_flags &RQF_SOFTBARRIER ) -break ; +#ifdef CONFIG_PM +/* +* If a request gets queued in state RPM_SUSPENDED +* then that's a kernel bug. +*/ +WARN_ON_ONCE (q->rpm_status ==RPM_SUSPENDED ); +#endif +return rq; }/*
block/blk-pm.c +67 -5
114 lines not on this page · raw message
Signature · 1 line
2.18.0
PATCH v7 7/7 blk-mq: Enable support for runtime power management
In reply to Bart Van AsscheNow that the blk-mq core processes power management requests (marked with RQF_PREEMPT) in other states than RPM_ACTIVE, enable runtime power management for blk-mq.
--- block/blk-mq.c | 2 ++ block/blk-pm.c | 6 ------ 2 files changed, 2 insertions(+), 6 deletions(-)
block/blk-mq.c +2 -0
@@ -33,6 +33,7 @@#include "blk-mq.h" #include "blk-mq-debugfs.h" #include "blk-mq-tag.h" +#include "blk-pm.h" #include "blk-stat.h" #include "blk-mq-sched.h" #include "blk-rq-qos.h"
@@ -475,6 +476,7 @@ static void __blk_mq_free_request(struct request *rq)struct blk_mq_hw_ctx *hctx =blk_mq_map_queue (q, ctx->cpu);const int sched_tag = rq->internal_tag; +blk_pm_mark_last_busy (rq);if (rq->tag !=-1 )blk_mq_put_tag (hctx, hctx->tags, ctx, rq->tag);if (sched_tag !=-1 )
block/blk-pm.c +0 -6
@@ -30,12 +30,6 @@*/ void blk_pm_runtime_init (struct request_queue *q,struct device *dev) { -/* Don't enable runtime PM for blk-mq until it is ready */ -if (q->mq_ops) { -pm_runtime_disable (dev); -return ; - } - q->dev = dev; q->rpm_status =RPM_ACTIVE ;pm_runtime_set_autosuspend_delay (q->dev,-1 );
Signature · 1 line
2.18.0
Quoted 1 line · Bart Van Assche
[ ... ]
Please ignore this e-mail thread - it contains a mixup of two versions of this patch series.
Bart.