PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

PATCH In review Stephen Boyd 3 participants 10 messages Aug 8, 2013 – Aug 13, 2013

Subsystems (2)

Suspend to RAM Hibernation

lore

Share & tools

lore

Stephen Boyd

author

2013-08-08 20:14 UTC #0 lore raw

PATCH 1/1 PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

pm_qos_update_request_timeout() updates a qos and then schedules a delayed work item to bring the qos back down to the default after the timeout. When the work item runs, pm_qos_work_fn() will call pm_qos_update_request() and deadlock because it tries to cancel itself via cancel_delayed_work_sync(). Future callers of that qos will also hang waiting to cancel the work that is canceling itself. Before ed1ac6e (PM: don't use [delayed_]work_pending(), 2013-01-11) this didn't happen because the work function wouldn't try to cancel itself.

Let's just do the little bit of pm_qos_update_request() here so that we don't deadlock.

Cc Tejun Heo <[email protected]>

Signed-off-by Stephen Boyd <[email protected]>

---
 kernel/power/qos.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

kernel/power/qos.c +5 -1

index 06fe285..d52d314 100644
@@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);
 }
 
 /**

Signature · 2 lines

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Stephen Boyd

author

2013-08-13 16:38 UTC #1 lore raw

Quoted 10 lines · Stephen Boyd

pm_qos_update_request_timeout() updates a qos and then schedules a delayed work item to bring the qos back down to the default after the timeout. When the work item runs, pm_qos_work_fn() will call pm_qos_update_request() and deadlock because it tries to cancel itself via cancel_delayed_work_sync(). Future callers of that qos will also hang waiting to cancel the work that is canceling itself. Before ed1ac6e (PM: don't use [delayed_]work_pending(), 2013-01-11) this didn't happen because the work function wouldn't try to cancel itself.

Quoted 6 lines

Let's just do the little bit of pm_qos_update_request() here so that we don't deadlock.
Cc Tejun Heo <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>
---

Ping?

Quoted 20 lines

 kernel/power/qos.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

kernel/power/qos.c +5 -1

index 06fe285..d52d314 100644
@@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);
 }
 
 /**

Signature · 2 lines

Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Tejun Heo

2013-08-13 16:43 UTC #2 lore raw

In reply to Stephen Boyd

Hello, Stephen.

Quoted 9 lines · Stephen Boyd

pm_qos_update_request_timeout() updates a qos and then schedules a delayed work item to bring the qos back down to the default after the timeout. When the work item runs, pm_qos_work_fn() will call pm_qos_update_request() and deadlock because it tries to cancel itself via cancel_delayed_work_sync(). Future callers of that qos will also hang waiting to cancel the work that is canceling itself. Before ed1ac6e (PM: don't use [delayed_]work_pending(), 2013-01-11) this didn't happen because the work function wouldn't try to cancel itself.

I see. That must have been racy tho. If the work item execution races someone else queuing the work item, the same deadlock could happen, right?

Quoted 14 lines

Let's just do the little bit of pm_qos_update_request() here so that we don't deadlock.
Cc Tejun Heo <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>
---
 kernel/power/qos.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/power/qos.c +0 -0
index 06fe285..d52d314 100644

kernel/power/qos.c static void pm_qos_work_fn(struct work_struct *work) · line 315

Quoted 9 lines

kernel/power/qos.c +5 -1

@@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);

Maybe it'd be cleaner to add a param or internal variant of pm_qos_update_request()?

Thanks.

Signature · 1 line

tejun

Stephen Boyd

author

2013-08-13 16:46 UTC #3 lore raw

Quoted 12 lines · Tejun Heo

Hello, Stephen.
Quoted 9 lines · Stephen Boyd
pm_qos_update_request_timeout() updates a qos and then schedules a delayed work item to bring the qos back down to the default after the timeout. When the work item runs, pm_qos_work_fn() will call pm_qos_update_request() and deadlock because it tries to cancel itself via cancel_delayed_work_sync(). Future callers of that qos will also hang waiting to cancel the work that is canceling itself. Before ed1ac6e (PM: don't use [delayed_]work_pending(), 2013-01-11) this didn't happen because the work function wouldn't try to cancel itself.

Quoted 3 lines

I see. That must have been racy tho. If the work item execution races someone else queuing the work item, the same deadlock could happen, right?

Yes you're right. It was always racy.

Quoted 18 lines

Quoted 18 lines
Let's just do the little bit of pm_qos_update_request() here so that we don't deadlock.
Cc Tejun Heo <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>
---
 kernel/power/qos.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/power/qos.c +0 -0
index 06fe285..d52d314 100644
@@ -308,7 +308,11 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 

Quoted 8 lines

kernel/power/qos.c static void pm_qos_work_fn(struct work_struct *work) · line 315
Quoted 6 lines
kernel/power/qos.c +5 -1
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);
Maybe it'd be cleaner to add a param or internal variant of pm_qos_update_request()?

Maybe, but I was trying to make a minimal fix here.

Signature · 2 lines

Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Tejun Heo

2013-08-13 17:01 UTC #4 lore raw

Hello,

Quoted 9 lines · Stephen Boyd

Quoted 7 lines
kernel/power/qos.c static void pm_qos_work_fn(struct work_struct *work) · line 315
Quoted 5 lines
kernel/power/qos.c +5 -0
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);
Maybe it'd be cleaner to add a param or internal variant of pm_qos_update_request()?
Maybe, but I was trying to make a minimal fix here.

Hmmm.... it just looks like things can easily get out of sync with the complex function call. I don't think it'll be too invasive if you introduce an internal variant which doesn't do the canceling. Rafael, what do you think?

Thanks.

Signature · 1 line

tejun

Rafael J. Wysocki

2013-08-13 20:38 UTC #5 lore raw

Quoted 11 lines · Tejun Heo

Hello,
Quoted 8 lines · Stephen Boyd
Quoted 7 lines
kernel/power/qos.c static void pm_qos_work_fn(struct work_struct *work) · line 315
Quoted 5 lines
kernel/power/qos.c +5 -0
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);
Maybe it'd be cleaner to add a param or internal variant of pm_qos_update_request()?

Quoted 4 lines

Quoted 1 line
Maybe, but I was trying to make a minimal fix here.
Hmmm.... it just looks like things can easily get out of sync with the complex function call.

Yes, that's just duplicated code.

Quoted 2 lines

I don't think it'll be too invasive if you introduce an internal variant which doesn't do the canceling. Rafael, what do you think?

I'd move the part of pm_qos_update_request() below the cancel_delayed_work_sync() to a separate static function that'd be called from two places.

Thanks,
Rafael

Signature · 2 lines

I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Stephen Boyd

author

2013-08-13 20:49 UTC #6 lore raw

Quoted 17 lines · Rafael J. Wysocki

Quoted 11 lines · Tejun Heo
Hello,
Quoted 8 lines · Stephen Boyd
Quoted 7 lines
kernel/power/qos.c static void pm_qos_work_fn(struct work_struct *work) · line 315
Quoted 5 lines
kernel/power/qos.c +5 -0
+	if (PM_QOS_DEFAULT_VALUE != req->node.prio)
+		pm_qos_update_target(
+				pm_qos_array[req->pm_qos_class]->constraints,
+				&req->node, PM_QOS_UPDATE_REQ,
+				PM_QOS_DEFAULT_VALUE);
Maybe it'd be cleaner to add a param or internal variant of pm_qos_update_request()?
Quoted 4 lines
Quoted 1 line
Maybe, but I was trying to make a minimal fix here.
Hmmm.... it just looks like things can easily get out of sync with the complex function call.

Quoted 9 lines

Yes, that's just duplicated code.
Quoted 2 lines
I don't think it'll be too invasive if you introduce an internal variant which doesn't do the canceling. Rafael, what do you think?
I'd move the part of pm_qos_update_request() below the cancel_delayed_work_sync() to a separate static function that'd be called from two places.

Ok I will throw this all into one patch and resend.

Signature · 2 lines

Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Stephen Boyd

author

2013-08-13 21:12 UTC #7 lore raw

PATCH 1/1 PM / QoS: Fix workqueue deadlock when using pm_qos_update_request_timeout()

In reply to Rafael J. Wysocki

pm_qos_update_request_timeout() updates a qos and then schedules a delayed work item to bring the qos back down to the default after the timeout. When the work item runs, pm_qos_work_fn() will call pm_qos_update_request() and deadlock because it tries to cancel itself via cancel_delayed_work_sync(). Future callers of that qos will also hang waiting to cancel the work that is canceling itself. Let's extract the little bit of code that does the real work of pm_qos_update_request() and call it from the work function so that we don't deadlock.

Cc Tejun Heo <[email protected]>

Signed-off-by Stephen Boyd <[email protected]>

---
 kernel/power/qos.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

kernel/power/qos.c +13 -7

index 06fe285..a394297 100644
@@ -296,6 +296,17 @@ int pm_qos_request_active(struct pm_qos_request *req)
 }
 EXPORT_SYMBOL_GPL(pm_qos_request_active);
 
+static void __pm_qos_update_request(struct pm_qos_request *req,
+			   s32 new_value)
+{
+	trace_pm_qos_update_request(req->pm_qos_class, new_value);
+
+	if (new_value != req->node.prio)
+		pm_qos_update_target(
+			pm_qos_array[req->pm_qos_class]->constraints,
+			&req->node, PM_QOS_UPDATE_REQ, new_value);
+}
+
 /**
  * pm_qos_work_fn - the timeout handler of pm_qos_update_request_timeout
  * @work: work struct for the delayed work (timeout)

@@ -308,7 +319,7 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	__pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
 }
 
 /**

@@ -364,12 +375,7 @@ void pm_qos_update_request(struct pm_qos_request *req,
 	}
 
 	cancel_delayed_work_sync(&req->work);
-
-	trace_pm_qos_update_request(req->pm_qos_class, new_value);
-	if (new_value != req->node.prio)
-		pm_qos_update_target(
-			pm_qos_array[req->pm_qos_class]->constraints,
-			&req->node, PM_QOS_UPDATE_REQ, new_value);
+	__pm_qos_update_request(req, new_value);
 }
 EXPORT_SYMBOL_GPL(pm_qos_update_request);

Signature · 2 lines

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Tejun Heo

Reviewed-by on v1 1/1

2013-08-13 22:13 UTC #8 lore raw

Hello,

kernel/power/qos.c static void pm_qos_work_fn(struct work_struct *work) · line 322

Quoted 6 lines · Stephen Boyd

kernel/power/qos.c +1 -1

@@ -308,7 +319,7 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);
 
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	__pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);

Maybe a short comment explaining why this is different would be nice? Other than that,

Reviewed-by Tejun Heo <[email protected]>

Thanks.

Signature · 1 line

tejun

Rafael J. Wysocki

2013-08-13 22:43 UTC #9 lore raw

Quoted 7 lines · Tejun Heo

Hello,

Quoted 4 lines · Stephen Boyd

kernel/power/qos.c +0 -0

@@ -308,7 +319,7 @@ static void pm_qos_work_fn(struct work_struct *work)
 						  struct pm_qos_request,
 						  work);

Quoted 7 lines

kernel/power/qos.c static void pm_qos_work_fn(struct work_struct *work) · line 322
Quoted 2 lines
kernel/power/qos.c +1 -1
-	pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
+	__pm_qos_update_request(req, PM_QOS_DEFAULT_VALUE);
Maybe a short comment explaining why this is different would be nice? Other than that,
Reviewed-by Tejun Heo <[email protected]>

Thanks guys, I'm going to push that as a fix for 3.11-rc6 and stable.

Rafael