blk-mq: Implement runtime power management
Subsystems (4)
v11 cover blk-mq: Implement runtime power management
Hello Jens,
One of the pieces that is missing before blk-mq can be made the default is implementing runtime power management support for blk-mq. This patch series not only implements runtime power management for blk-mq but also fixes a starvation issue in the power management code for the legacy block layer. Please consider this patch series for the upstream kernel.
Thanks,
Bart.
Changes compared to v10:
- Added a comment in the percpu-refcount patch in this series as Tejun asked.
- Updated Acked-by / Reviewed-by tags.
Changes compared to v9:
- Left out the patches that document the functions that iterate over requests and also the patch that introduces blk_mq_queue_rq_iter().
- Simplified blk_pre_runtime_suspend(): left out the check whether no requests are in progress.
- Fixed the race between blk_queue_enter(), queue freezing and runtime power management that Ming had identified.
- Added a new patch that introduces percpu_ref_resurrect().
Changes compared to v8:
- Fixed the race that was reported by Jianchao.
- Fixed another spelling issue in a source code comment.
Changes compared to v7:
- Addressed Jianchao's feedback about patch "Make blk_get_request() block for non-PM requests while suspended".
- Added two new patches - one that documents the functions that iterate over requests and one that introduces a new function that iterates over all requests associated with a queue.
Changes compared to v6:
- Left out the patches that split RQF_PREEMPT in three flags.
- Left out the patch that introduces the SCSI device state SDEV_SUSPENDED.
- Left out the patch that introduces blk_pm_runtime_exit().
- Restored the patch that changes the PREEMPT_ONLY flag into a counter.
Changes compared to v5:
- Introduced a new flag RQF_DV that replaces RQF_PREEMPT for SCSI domain validation.
- Introduced a new request queue state QUEUE_FLAG_DV_ONLY for SCSI domain validation.
- Instead of using SDEV_QUIESCE for both runtime suspend and SCSI domain validation, use that state for domain validation only and introduce a new state for runtime suspend, namely SDEV_QUIESCE.
- Reallow system suspend during SCSI domain validation.
- Moved the runtime resume call from the request allocation code into blk_queue_enter().
- Instead of relying on q_usage_counter, iterate over the tag set to determine whether or not any requests are in flight.
Changes compared to v4:
- Dropped the patches "Give RQF_PREEMPT back its original meaning" and "Serialize queue freezing and blk_pre_runtime_suspend()".
- Replaced "percpu_ref_read()" with "percpu_is_in_use()".
- Inserted pm_request_resume() calls in the block layer request allocation code such that the context that submits a request no longer has to call pm_runtime_get().
Changes compared to v3:
- Avoid adverse interactions between system-wide suspend/resume and runtime power management by changing the PREEMPT_ONLY flag into a counter.
- Give RQF_PREEMPT back its original meaning, namely that it is only set for ide_preempt requests.
- Remove the flag BLK_MQ_REQ_PREEMPT.
- Removed the pm_request_resume() call.
Changes compared to v2:
- Fixed the build for CONFIG_BLOCK=n.
- Added a patch that introduces percpu_ref_read() in the percpu-counter code.
- Added a patch that makes it easier to detect missing pm_runtime_get*() calls.
- Addressed Jianchao's feedback including the comment about runtime overhead of switching a per-cpu counter to atomic mode.
Changes compared to v1:
- Moved the runtime power management code into a separate file.
- Addressed Ming's feedback.
Bart Van Assche (8):
block: Move power management code into a new source file block, scsi: Change the preempt-only flag into a counter
block: Split blk_pm_add_request() and blk_pm_put_request()
block: Schedule runtime resume earlier
percpu-refcount: Introduce percpu_ref_resurrect()
block: Allow unfreezing of a queue while requests are in progress
block: Make blk_get_request() block for non-PM requests while
suspended
blk-mq: Enable support for runtime power management
block/Kconfig | 3 + block/Makefile | 1 + block/blk-core.c | 270 ++++---------------------------- block/blk-mq-debugfs.c | 10 +- block/blk-mq.c | 4 +- block/blk-pm.c | 216 +++++++++++++++++++++++++ block/blk-pm.h | 69 ++++++++ block/elevator.c | 22 +-- drivers/scsi/scsi_lib.c | 11 +- drivers/scsi/scsi_pm.c | 1 + drivers/scsi/sd.c | 1 + drivers/scsi/sr.c | 1 + include/linux/blk-pm.h | 24 +++ include/linux/blkdev.h | 37 ++--- include/linux/percpu-refcount.h | 1 + lib/percpu-refcount.c | 28 +++- 16 files changed, 401 insertions(+), 298 deletions(-) create mode 100644 block/blk-pm.c create mode 100644 block/blk-pm.h create mode 100644 include/linux/blk-pm.h
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 1/8 block: Move power management code into a new source file
Move the code for runtime power management from blk-core.c into the new source file blk-pm.c. Move the corresponding declarations from <linux/blkdev.h> into <linux/blk-pm.h>. For CONFIG_PM=n, leave out the declarations of the functions that are not used in that mode. This patch not only reduces the number of #ifdefs in the block layer core code but also reduces the size of header file <linux/blkdev.h> and hence should help to reduce the build time of the Linux kernel if CONFIG_PM is not defined.
--- block/Kconfig | 3 + block/Makefile | 1 + block/blk-core.c | 196 +---------------------------------------- block/blk-pm.c | 188 +++++++++++++++++++++++++++++++++++++++ block/blk-pm.h | 43 +++++++++ block/elevator.c | 22 +---- drivers/scsi/scsi_pm.c | 1 + drivers/scsi/sd.c | 1 + drivers/scsi/sr.c | 1 + include/linux/blk-pm.h | 24 +++++ include/linux/blkdev.h | 23 ----- 11 files changed, 264 insertions(+), 239 deletions(-) create mode 100644 block/blk-pm.c create mode 100644 block/blk-pm.h create mode 100644 include/linux/blk-pm.h
block/Kconfig +3 -0
@@ -228,4 +228,7 @@ config BLK_MQ_RDMA depends on BLOCK && INFINIBAND default y +config BLK_PM + def_bool BLOCK && PM + source block/Kconfig.iosched
block/Makefile +1 -0
@@ -37,3 +37,4 @@ obj-$(CONFIG_BLK_WBT) += blk-wbt.o obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o +obj-$(CONFIG_BLK_PM) += blk-pm.o
block/blk-core.c +1 -195
214 lines not on this page · raw message
block/blk-pm.c new file +188 -0
188 lines not on this page · raw message
block/blk-pm.h new file +43 -0
43 lines not on this page · raw message
block/elevator.c +1 -21
@@ -41,6 +41,7 @@#include "blk.h" #include "blk-mq-sched.h" +#include "blk-pm.h" #include "blk-wbt.h" static DEFINE_SPINLOCK (elv_list_lock);
@@ -557,27 +558,6 @@ void elv_bio_merged(struct request_queue *q, struct request *rq, e->type->ops.sq.elevator_bio_merged_fn (q, rq, bio); } -#ifdef CONFIG_PM -static void blk_pm_requeue_request (struct request *rq) -{ -if (rq->q->dev && !(rq->rq_flags &RQF_PM )) - rq->q->nr_pending--; -} - -static void blk_pm_add_request (struct request_queue *q,struct request *rq) -{ -if (q->dev && !(rq->rq_flags &RQF_PM ) && q->nr_pending++ ==0 && - (q->rpm_status ==RPM_SUSPENDED || q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); -} -#else -static inline void blk_pm_requeue_request (struct request *rq) {} -static inline void blk_pm_add_request (struct request_queue *q, -struct request *rq) -{ -} -#endif -void elv_requeue_request (struct request_queue *q,struct request *rq) {/*
drivers/scsi/scsi_pm.c +1 -0
@@ -8,6 +8,7 @@#include <linux/pm_runtime.h> #include <linux/export.h> #include <linux/async.h> +#include <linux/blk-pm.h> #include <scsi/scsi.h> #include <scsi/scsi_device.h>
drivers/scsi/sd.c +1 -0
@@ -45,6 +45,7 @@#include <linux/init.h> #include <linux/blkdev.h> #include <linux/blkpg.h> +#include <linux/blk-pm.h> #include <linux/delay.h> #include <linux/mutex.h> #include <linux/string_helpers.h>
drivers/scsi/sr.c +1 -0
@@ -43,6 +43,7 @@#include <linux/interrupt.h> #include <linux/init.h> #include <linux/blkdev.h> +#include <linux/blk-pm.h> #include <linux/mutex.h> #include <linux/slab.h> #include <linux/pm_runtime.h>
include/linux/blk-pm.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _BLK_PM_H_ +#define _BLK_PM_H_ + +struct device ; +struct request_queue ; + +/* +* block layer runtime pm functions +*/ +#ifdef CONFIG_PM +extern void blk_pm_runtime_init (struct request_queue *q,struct device *dev); +extern int blk_pre_runtime_suspend (struct request_queue *q); +extern void blk_post_runtime_suspend (struct request_queue *q,int err); +extern void blk_pre_runtime_resume (struct request_queue *q); +extern void blk_post_runtime_resume (struct request_queue *q,int err); +extern void blk_set_runtime_active (struct request_queue *q); +#else +static inline void blk_pm_runtime_init (struct request_queue *q, +struct device *dev) {} +#endif + +#endif /* _BLK_PM_H_ */
include/linux/blkdev.h +0 -23
@@ -1280,29 +1280,6 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id,extern void blk_put_queue (struct request_queue *);extern void blk_set_queue_dying (struct request_queue *); -/* -* block layer runtime pm functions -*/ -#ifdef CONFIG_PM -extern void blk_pm_runtime_init (struct request_queue *q,struct device *dev); -extern int blk_pre_runtime_suspend (struct request_queue *q); -extern void blk_post_runtime_suspend (struct request_queue *q,int err); -extern void blk_pre_runtime_resume (struct request_queue *q); -extern void blk_post_runtime_resume (struct request_queue *q,int err); -extern void blk_set_runtime_active (struct request_queue *q); -#else -static inline void blk_pm_runtime_init (struct request_queue *q, -struct device *dev) {} -static inline int blk_pre_runtime_suspend (struct request_queue *q) -{ -return -ENOSYS ; -} -static inline void blk_post_runtime_suspend (struct request_queue *q,int err) {} -static inline void blk_pre_runtime_resume (struct request_queue *q) {} -static inline void blk_post_runtime_resume (struct request_queue *q,int err) {} -static inline void blk_set_runtime_active (struct request_queue *q) {} -#endif -/* * blk_plug permits building a queue of related requests by holding the I/O * fragments for a short period. This allows merging of sequential requests
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 2/8 block, scsi: Change the preempt-only flag into a counter
In reply to Bart Van AsscheThe RQF_PREEMPT flag is used for three purposes:
- In the SCSI core, for making sure that power management requests are executed even if a device is in the "quiesced" state.
- For domain validation by SCSI drivers that use the parallel port.
- In the IDE driver, for IDE preempt requests.
Rename "preempt-only" into "pm-only" because the primary purpose of this mode is power management. Since the power management core may but does not have to resume a runtime suspended device before performing system-wide suspend and since a later patch will set "pm-only" mode as long as a block device is runtime suspended, make it possible to set "pm-only" mode from more than one context. Since with this change scsi_device_quiesce() is no longer idempotent, make that function return early if it is called for a quiesced queue.
--- block/blk-core.c | 35 ++++++++++++++++++----------------- block/blk-mq-debugfs.c | 10 +++++++++- drivers/scsi/scsi_lib.c | 11 +++++++---- include/linux/blkdev.h | 14 +++++++++----- 4 files changed, 43 insertions(+), 27 deletions(-)
block/blk-core.c +18 -17
@@ -422,24 +422,25 @@ void blk_sync_queue(struct request_queue *q)EXPORT_SYMBOL (blk_sync_queue);/** -* blk_set_preempt_only - set QUEUE_FLAG_PREEMPT_ONLY +* blk_set_pm_only - increment pm_only counter * @q: request queue pointer -* -* Returns the previous value of the PREEMPT_ONLY flag - 0 if the flag was not -* set and 1 if the flag was already set. */ -int blk_set_preempt_only (struct request_queue *q) +void blk_set_pm_only (struct request_queue *q) { -return blk_queue_flag_test_and_set (QUEUE_FLAG_PREEMPT_ONLY , q); +atomic_inc (&q->pm_only); } -EXPORT_SYMBOL_GPL (blk_set_preempt_only); +EXPORT_SYMBOL_GPL (blk_set_pm_only); -void blk_clear_preempt_only (struct request_queue *q) +void blk_clear_pm_only (struct request_queue *q) { -blk_queue_flag_clear (QUEUE_FLAG_PREEMPT_ONLY , q); -wake_up_all (&q->mq_freeze_wq); +int pm_only; + + pm_only =atomic_dec_return (&q->pm_only); +WARN_ON_ONCE (pm_only <0 ); +if (pm_only ==0 ) +wake_up_all (&q->mq_freeze_wq); } -EXPORT_SYMBOL_GPL (blk_clear_preempt_only); +EXPORT_SYMBOL_GPL (blk_clear_pm_only);/** * __blk_run_queue_uncond - run a queue whether or not it has been stopped
@@ -918,7 +919,7 @@ EXPORT_SYMBOL(blk_alloc_queue);*/ int blk_queue_enter (struct request_queue *q,blk_mq_req_flags_t flags) { -const bool preempt = flags &BLK_MQ_REQ_PREEMPT ; +const bool pm = flags &BLK_MQ_REQ_PREEMPT ;while (true) {bool success = false;
@@ -926,11 +927,11 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)rcu_read_lock ();if (percpu_ref_tryget_live (&q->q_usage_counter)) {/* -* The code that sets the PREEMPT_ONLY flag is -* responsible for ensuring that that flag is globally -* visible before the queue is unfrozen. +* The code that increments the pm_only counter is +* responsible for ensuring that that counter is +* globally visible before the queue is unfrozen. */ -if (preempt || !blk_queue_preempt_only (q)) { +if (pm || !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);
@@ -955,7 +956,7 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)wait_event (q->mq_freeze_wq, (atomic_read (&q->mq_freeze_depth) ==0 && - (preempt || !blk_queue_preempt_only (q))) || + (pm || !blk_queue_pm_only (q))) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;
block/blk-mq-debugfs.c +9 -1
@@ -102,6 +102,14 @@ static int blk_flags_show(struct seq_file *m, const unsigned long flags,return 0 ; } +static int queue_pm_only_show (void *data,struct seq_file *m) +{ +struct request_queue *q = data; + +seq_printf (m,"%d\n" ,atomic_read (&q->pm_only)); +return 0 ; +} +#define QUEUE_FLAG_NAME (name) [QUEUE_FLAG_##name] = #namestatic const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (QUEUED ),
@@ -132,7 +140,6 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (REGISTERED ),QUEUE_FLAG_NAME (SCSI_PASSTHROUGH ), QUEUE_FLAG_NAME(QUIESCED), - QUEUE_FLAG_NAME(PREEMPT_ONLY), };#undef QUEUE_FLAG_NAME
@@ -209,6 +216,7 @@ static ssize_t queue_write_hint_store(void *data, const char __user *buf,static const struct blk_mq_debugfs_attr blk_mq_debugfs_queue_attrs[] = { {"poll_stat" ,0400 , queue_poll_stat_show }, {"requeue_list" ,0400 , .seq_ops = &queue_requeue_list_seq_ops }, + {"pm_only" ,0600 , queue_pm_only_show,NULL }, {"state" ,0600 , queue_state_show, queue_state_write }, {"write_hints" ,0600 , queue_write_hint_show, queue_write_hint_store }, {"zone_wlock" ,0400 , queue_zone_wlock_show,NULL },
drivers/scsi/scsi_lib.c +7 -4
@@ -3046,11 +3046,14 @@ scsi_device_quiesce(struct scsi_device *sdev)*/ WARN_ON_ONCE (sdev->quiesced_by && sdev->quiesced_by != current); -blk_set_preempt_only (q); +if (sdev->quiesced_by == current) +return 0 ; + +blk_set_pm_only (q);blk_mq_freeze_queue (q);/* -* Ensure that the effect of blk_set_preempt_only() will be visible +* Ensure that the effect of blk_set_pm_only() will be visible * for percpu_ref_tryget() callers that occur after the queue * unfreeze even if the queue was already frozen before this function * was called. See also https://lwn.net/Articles/573497/.
@@ -3063,7 +3066,7 @@ scsi_device_quiesce(struct scsi_device *sdev)if (err ==0 ) sdev->quiesced_by = current;else -blk_clear_preempt_only (q); +blk_clear_pm_only (q);mutex_unlock (&sdev->state_mutex);return err;
@@ -3088,7 +3091,7 @@ void scsi_device_resume(struct scsi_device *sdev)mutex_lock (&sdev->state_mutex);WARN_ON_ONCE (!sdev->quiesced_by); sdev->quiesced_by =NULL ; -blk_clear_preempt_only (sdev->request_queue); +blk_clear_pm_only (sdev->request_queue);if (sdev->sdev_state ==SDEV_QUIESCE )scsi_device_set_state (sdev,SDEV_RUNNING );mutex_unlock (&sdev->state_mutex);
include/linux/blkdev.h +9 -5
@@ -504,6 +504,12 @@ struct request_queue {* various queue flags, see QUEUE_* below */ unsigned long queue_flags; +/* +* Number of contexts that have called blk_set_pm_only(). If this +* counter is above zero then only RQF_PM and RQF_PREEMPT requests are +* processed. +*/ +atomic_t pm_only;/* * ida allocated id for this queue. Used to index queues from
@@ -698,7 +704,6 @@ struct request_queue {#define QUEUE_FLAG_REGISTERED 26/* queue has been registered to a disk */ #define QUEUE_FLAG_SCSI_PASSTHROUGH 27/* queue supports SCSI commands */ #define QUEUE_FLAG_QUIESCED 28/* queue has been quiesced */ -#define QUEUE_FLAG_PREEMPT_ONLY 29/* only process REQ_PREEMPT requests */ #define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \ (1 << QUEUE_FLAG_SAME_COMP) | \
@@ -736,12 +741,11 @@ bool blk_queue_flag_test_and_clear(unsigned int flag, struct request_queue *q); ((rq)->cmd_flags & (REQ_FAILFAST_DEV |REQ_FAILFAST_TRANSPORT | \ REQ_FAILFAST_DRIVER))#define blk_queue_quiesced (q) test_bit(QUEUE_FLAG_QUIESCED, &(q)->queue_flags) -#define blk_queue_preempt_only (q) \ - test_bit(QUEUE_FLAG_PREEMPT_ONLY, &(q)->queue_flags) +#define blk_queue_pm_only (q) atomic_read(&(q)->pm_only)#define blk_queue_fua (q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags) -extern int blk_set_preempt_only (struct request_queue *q); -extern void blk_clear_preempt_only (struct request_queue *q); +extern void blk_set_pm_only (struct request_queue *q); +extern void blk_clear_pm_only (struct request_queue *q);static inline int queue_in_flight (struct request_queue *q) {
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 3/8 block: Split blk_pm_add_request() and blk_pm_put_request()
In reply to Bart Van AsscheMove the pm_request_resume() and pm_runtime_mark_last_busy() calls into two new functions and thereby separate legacy block layer code from code that works for both the legacy block layer and blk-mq. A later patch will add calls to the new functions in the blk-mq code.
--- block/blk-core.c | 1 + block/blk-pm.h | 36 +++++++++++++++++++++++++++++++----- block/elevator.c | 1 + 3 files changed, 33 insertions(+), 5 deletions(-)
block/blk-core.c +1 -0
@@ -1744,6 +1744,7 @@ void __blk_put_request(struct request_queue *q, struct request *req)blk_req_zone_write_unlock (req);blk_pm_put_request (req); +blk_pm_mark_last_busy (req);elv_completed_request (q, req);
block/blk-pm.h +31 -5
@@ -6,8 +6,23 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM +static inline void blk_pm_request_resume (struct request_queue *q) +{ +if (q->dev && (q->rpm_status ==RPM_SUSPENDED || + q->rpm_status ==RPM_SUSPENDING )) +pm_request_resume (q->dev); +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) +pm_runtime_mark_last_busy (rq->q->dev); +} +static inline void blk_pm_requeue_request (struct request *rq) { +lockdep_assert_held (rq->q->queue_lock); +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) rq->q->nr_pending--; }
@@ -15,17 +30,28 @@ static inline void blk_pm_requeue_request(struct request *rq)static inline void blk_pm_add_request (struct request_queue *q,struct request *rq) { -if (q->dev && !(rq->rq_flags &RQF_PM ) && q->nr_pending++ ==0 && - (q->rpm_status ==RPM_SUSPENDED || q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +lockdep_assert_held (q->queue_lock); + +if (q->dev && !(rq->rq_flags &RQF_PM )) + q->nr_pending++; }static inline void blk_pm_put_request (struct request *rq) { -if (rq->q->dev && !(rq->rq_flags &RQF_PM ) && !--rq->q->nr_pending) -pm_runtime_mark_last_busy (rq->q->dev); +lockdep_assert_held (rq->q->queue_lock); + +if (rq->q->dev && !(rq->rq_flags &RQF_PM )) + --rq->q->nr_pending; }#else +static inline void blk_pm_request_resume (struct request_queue *q) +{ +} + +static inline void blk_pm_mark_last_busy (struct request *rq) +{ +} +static inline void blk_pm_requeue_request (struct request *rq) { }
block/elevator.c +1 -0
@@ -601,6 +601,7 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); +blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 4/8 block: Schedule runtime resume earlier
In reply to Bart Van AsscheInstead of scheduling runtime resume of a request queue after a request has been queued, schedule asynchronous resume during request allocation. The new pm_request_resume() calls occur after blk_queue_enter() has increased the q_usage_counter request queue member. This change is needed for a later patch that will make request allocation block while the queue status is not RPM_ACTIVE.
--- block/blk-core.c | 3 ++- block/elevator.c | 1 - 2 files changed, 2 insertions(+), 2 deletions(-)
block/blk-core.c +2 -1
@@ -956,7 +956,8 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)wait_event (q->mq_freeze_wq, (atomic_read (&q->mq_freeze_depth) ==0 && - (pm || !blk_queue_pm_only (q))) || + (pm || (blk_pm_request_resume (q), + !blk_queue_pm_only (q)))) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;
block/elevator.c +0 -1
@@ -601,7 +601,6 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)trace_block_rq_insert (q, rq);blk_pm_add_request (q, rq); -blk_pm_request_resume (q); rq->q = q;
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 5/8 percpu-refcount: Introduce percpu_ref_resurrect()
In reply to Bart Van AsscheThis function will be used in a later patch to switch the struct request_queue q_usage_counter from killed back to live. In contrast to percpu_ref_reinit(), this new function does not require that the refcount is zero.
--- include/linux/percpu-refcount.h | 1 + lib/percpu-refcount.c | 28 ++++++++++++++++++++++++++-- 2 files changed, 27 insertions(+), 2 deletions(-)
include/linux/percpu-refcount.h +1 -0
@@ -108,6 +108,7 @@ void percpu_ref_switch_to_atomic_sync(struct percpu_ref *ref);void percpu_ref_switch_to_percpu (struct percpu_ref *ref);void percpu_ref_kill_and_confirm (struct percpu_ref *ref,percpu_ref_func_t *confirm_kill); +void percpu_ref_resurrect (struct percpu_ref *ref);void percpu_ref_reinit (struct percpu_ref *ref);/**
lib/percpu-refcount.c +26 -2
@@ -356,11 +356,35 @@ EXPORT_SYMBOL_GPL(percpu_ref_kill_and_confirm);*/ void percpu_ref_reinit (struct percpu_ref *ref) { +WARN_ON_ONCE (!percpu_ref_is_zero (ref)); + +percpu_ref_resurrect (ref); +} +EXPORT_SYMBOL_GPL (percpu_ref_reinit); + +/** +* percpu_ref_resurrect - modify a percpu refcount from dead to live +* @ref: perpcu_ref to resurrect +* +* Modify @ref so that it's in the same state as before percpu_ref_kill() was +* called. @ref must be dead but must not yet have exited. +* +* If @ref->release() frees @ref then the caller is responsible for +* guaranteeing that @ref->release() does not get called while this +* function is in progress. +* +* Note that percpu_ref_tryget[_live]() are safe to perform on @ref while +* this function is in progress. +*/ +void percpu_ref_resurrect (struct percpu_ref *ref) +{ +unsigned long __percpu *percpu_count;unsigned long flags;spin_lock_irqsave (&percpu_ref_switch_lock, flags); -WARN_ON_ONCE (!percpu_ref_is_zero (ref)); +WARN_ON_ONCE (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)); +WARN_ON_ONCE (__ref_is_percpu (ref, &percpu_count)); ref->percpu_count_ptr &= ~__PERCPU_REF_DEAD;percpu_ref_get (ref);
@@ -368,4 +392,4 @@ void percpu_ref_reinit(struct percpu_ref *ref)spin_unlock_irqrestore (&percpu_ref_switch_lock, flags); } -EXPORT_SYMBOL_GPL (percpu_ref_reinit); +EXPORT_SYMBOL_GPL (percpu_ref_resurrect);
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 6/8 block: Allow unfreezing of a queue while requests are in progress
In reply to Bart Van AsscheA later patch will call blk_freeze_queue_start() followed by blk_mq_unfreeze_queue() without waiting for q_usage_counter to drop to zero. Make sure that this doesn't cause a kernel warning to appear by switching from percpu_ref_reinit() to percpu_ref_resurrect(). The former namely requires that the refcount it operates on is zero.
--- block/blk-mq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-mq.c +1 -1
@@ -198,7 +198,7 @@ void blk_mq_unfreeze_queue(struct request_queue *q) freeze_depth =atomic_dec_return (&q->mq_freeze_depth);WARN_ON_ONCE (freeze_depth <0 );if (!freeze_depth) { -percpu_ref_reinit (&q->q_usage_counter); +percpu_ref_resurrect (&q->q_usage_counter);wake_up_all (&q->mq_freeze_wq); } }
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 7/8 block: Make blk_get_request() block for non-PM requests while suspended
In reply to Bart Van AsscheInstead of allowing requests that are not power management requests to enter the queue in runtime suspended status (RPM_SUSPENDED), make the blk_get_request() caller block. This change fixes a starvation
issue: it is now guaranteed that power management requests will be executed no matter how many blk_get_request() callers are waiting. For blk-mq, instead of maintaining the q->nr_pending counter, rely on q->q_usage_counter. Call pm_runtime_mark_last_busy() every time a request finishes instead of only if the queue depth drops to zero.
--- block/blk-core.c | 37 ++++++++----------------------------- block/blk-pm.c | 44 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 47 insertions(+), 34 deletions(-)
block/blk-core.c +8 -29
@@ -2746,30 +2746,6 @@ void blk_account_io_done(struct request *req, u64 now) } } -#ifdef CONFIG_PM -/* -* Don't process normal requests when queue is suspended -* or in the process of suspending/resuming -*/ -static bool blk_pm_allow_request (struct request *rq) -{ -switch (rq->q->rpm_status) { -case RPM_RESUMING : -case RPM_SUSPENDING : -return rq->rq_flags &RQF_PM ; -case RPM_SUSPENDED : -return false; -default : -return true; - } -} -#else -static bool blk_pm_allow_request (struct request *rq) -{ -return true; -} -#endif -void blk_account_io_start (struct request *rq,bool new_io) {struct hd_struct *part;
@@ -2815,11 +2791,14 @@ static struct request *elv_next_request(struct request_queue *q)while (1 ) {list_for_each_entry (rq, &q->queue_head, queuelist) { -if (blk_pm_allow_request (rq)) -return rq; - -if (rq->rq_flags &RQF_SOFTBARRIER ) -break ; +#ifdef CONFIG_PM +/* +* If a request gets queued in state RPM_SUSPENDED +* then that's a kernel bug. +*/ +WARN_ON_ONCE (q->rpm_status ==RPM_SUSPENDED ); +#endif +return rq; }/*
block/blk-pm.c +39 -5
80 lines not on this page · raw message
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v11 8/8 blk-mq: Enable support for runtime power management
In reply to Bart Van AsscheNow that the blk-mq core processes power management requests (marked with RQF_PREEMPT) in other states than RPM_ACTIVE, enable runtime power management for blk-mq.
--- block/blk-mq.c | 2 ++ block/blk-pm.c | 6 ------ 2 files changed, 2 insertions(+), 6 deletions(-)
block/blk-mq.c +2 -0
@@ -33,6 +33,7 @@#include "blk-mq.h" #include "blk-mq-debugfs.h" #include "blk-mq-tag.h" +#include "blk-pm.h" #include "blk-stat.h" #include "blk-mq-sched.h" #include "blk-rq-qos.h"
@@ -475,6 +476,7 @@ static void __blk_mq_free_request(struct request *rq)struct blk_mq_hw_ctx *hctx =blk_mq_map_queue (q, ctx->cpu);const int sched_tag = rq->internal_tag; +blk_pm_mark_last_busy (rq);if (rq->tag !=-1 )blk_mq_put_tag (hctx, hctx->tags, ctx, rq->tag);if (sched_tag !=-1 )
block/blk-pm.c +0 -6
@@ -30,12 +30,6 @@*/ void blk_pm_runtime_init (struct request_queue *q,struct device *dev) { -/* Don't enable runtime PM for blk-mq until it is ready */ -if (q->mq_ops) { -pm_runtime_disable (dev); -return ; - } - q->dev = dev; q->rpm_status =RPM_ACTIVE ;pm_runtime_set_autosuspend_delay (q->dev,-1 );
Signature · 1 line
2.19.0.605.g01d371f741-goog
Jens Axboe
maintainerQuoted 7 lines · Bart Van Assche
Hello Jens,
One of the pieces that is missing before blk-mq can be made the default is implementing runtime power management support for blk-mq. This patch series not only implements runtime power management for blk-mq but also fixes a starvation issue in the power management code for the legacy block layer. Please consider this patch series for the upstream kernel.
Thanks Bart, applied for 4.20.
Signature · 1 line
Jens Axboe
Jens,
Quoted 7 lines
Quoted 5 lines
One of the pieces that is missing before blk-mq can be made the default is implementing runtime power management support for blk-mq. This patch series not only implements runtime power management for blk-mq but also fixes a starvation issue in the power management code for the legacy block layer. Please consider this patch series for the upstream kernel.
Thanks Bart, applied for 4.20.
I have made a note to back out the ufs change for 4.20.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Jens Axboe
maintainerQuoted 9 lines · Martin K. Petersen
Jens,
Quoted 6 lines
Quoted 5 lines
One of the pieces that is missing before blk-mq can be made the default is implementing runtime power management support for blk-mq. This patch series not only implements runtime power management for blk-mq but also fixes a starvation issue in the power management code for the legacy block layer. Please consider this patch series for the upstream kernel.
Quoted 3 lines
Quoted 1 line
Thanks Bart, applied for 4.20.
I have made a note to back out the ufs change for 4.20.
Thanks Martin, will be nice to have this finally all resolved.
Signature · 1 line
Jens Axboe