Skip to contentSkip to conversation

soc: qcom: rpmh: Remove serialization of TCS commands

PATCH Superseded Maulik Shah 2 participants 4 messages May 29, 2020 – Jun 25, 2020

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 soc: qcom: rpmh: Remove serialization of TCS commands

From: Lina Iyer <[email protected]>

Requests sent to RPMH can be sent as fire-n-forget or response required, with the latter ensuring the command has been completed by the hardware accelerator. Commands in a request with tcs_cmd::wait set, would ensure that those select commands are sent as response required, even though the actual TCS request may be fire-n-forget.

Also, commands with .wait flag were also guaranteed to be complete before the following command in the TCS is sent. This means that the next command of the same request blocked until the current request is completed. This could mean waiting for a voltage to settle or series of NOCs be configured before the next command is sent. But drivers using this feature have never cared about the serialization aspect. By not enforcing the serialization we can allow the hardware to run in parallel improving the performance.

Let's clarify the usage of this member in the tcs_cmd structure to mean only completion and not serialization. This should also improve the performance of bus requests where changes could happen in parallel. Also, CPU resume from deep idle may see benefits from certain wake requests.

---
 drivers/soc/qcom/rpmh-rsc.c | 19 ++++++++-----------
 include/soc/qcom/tcs.h      |  5 +++--
 2 files changed, 11 insertions(+), 13 deletions(-)
drivers/soc/qcom/rpmh-rsc.c +8 -11
index 076fd27..d99e639 100644
@@ -413,8 +413,7 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
 			cmd = &req->cmds[j];
 			sts = read_tcs_cmd(drv, RSC_DRV_CMD_STATUS, i, j);
 			if (!(sts & CMD_STATUS_ISSUED) ||
-			   ((req->wait_for_compl || cmd->wait) &&
-			   !(sts & CMD_STATUS_COMPL))) {
+			   (cmd->wait && !(sts & CMD_STATUS_COMPL))) {
 				pr_err("Incomplete request: %s: addr=%#x data=%#x",
 				       drv->name, cmd->addr, cmd->data);
 				err = -EIO;
@@ -433,7 +432,6 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
 skip:
 		/* Reclaim the TCS */
 		write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, i, 0);
-		write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, i, 0);
 		writel_relaxed(BIT(i), drv->tcs_base + RSC_DRV_IRQ_CLEAR);
 		spin_lock(&drv->lock);
 		clear_bit(i, drv->tcs_in_use);
@@ -465,23 +463,23 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
 static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
 			       const struct tcs_request *msg)
 {
-	u32 msgid, cmd_msgid;
+	u32 msgid;
+	u32 cmd_msgid = CMD_MSGID_LEN | CMD_MSGID_WRITE;
 	u32 cmd_enable = 0;
-	u32 cmd_complete;
 	struct tcs_cmd *cmd;
 	int i, j;
 
-	cmd_msgid = CMD_MSGID_LEN;
+	/* Convert all commands to RR when the request has wait_for_compl set */
 	cmd_msgid |= msg->wait_for_compl ? CMD_MSGID_RESP_REQ : 0;
-	cmd_msgid |= CMD_MSGID_WRITE;
-
-	cmd_complete = read_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id);
 
 	for (i = 0, j = cmd_id; i < msg->num_cmds; i++, j++) {
 		cmd = &msg->cmds[i];
 		cmd_enable |= BIT(j);
-		cmd_complete |= cmd->wait << j;
 		msgid = cmd_msgid;
+		/*
+		 * Additionally, if the cmd->wait is set, make the command
+		 * response reqd even if the overall request was fire-n-forget.
+		 */
 		msgid |= cmd->wait ? CMD_MSGID_RESP_REQ : 0;
 
 		write_tcs_cmd(drv, RSC_DRV_CMD_MSGID, tcs_id, j, msgid);
@@ -490,7 +488,6 @@ static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
 		trace_rpmh_send_msg_rcuidle(drv, tcs_id, j, msgid, cmd);
 	}
 
-	write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id, cmd_complete);
 	cmd_enable |= read_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id);
 	write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id, cmd_enable);
 }
include/soc/qcom/tcs.h +3 -2
index 7a2a055..d1c87fd 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 /*
- * Copyright (c) 2016-2019, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2016-2020, The Linux Foundation. All rights reserved.
  */
 
 #ifndef __SOC_QCOM_TCS_H__
@@ -30,7 +30,7 @@ enum rpmh_state {
  *
  * @addr: the address of the resource slv_id:18:16 | offset:0:15
  * @data: the resource state request
- * @wait: wait for this request to be complete before sending the next
+ * @wait: Ensure that this command is complete before returning
  */
 struct tcs_cmd {
 	u32 addr;
@@ -43,6 +43,7 @@ struct tcs_cmd {
  *
  * @state:          state for the request.
  * @wait_for_compl: wait until we get a response from the h/w accelerator
+ *                  (sets the cmd->wait for all commmands in the request)
  * @num_cmds:       the number of @cmds in this request
  * @cmds:           an array of tcs_cmds
  */
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
#1 lore raw

Hi,

Quoted 18 lines · Maulik Shah

From: Lina Iyer <[email protected]>

Requests sent to RPMH can be sent as fire-n-forget or response required, with the latter ensuring the command has been completed by the hardware accelerator. Commands in a request with tcs_cmd::wait set, would ensure that those select commands are sent as response required, even though the actual TCS request may be fire-n-forget.

Also, commands with .wait flag were also guaranteed to be complete before the following command in the TCS is sent. This means that the next command of the same request blocked until the current request is completed. This could mean waiting for a voltage to settle or series of NOCs be configured before the next command is sent. But drivers using this feature have never cared about the serialization aspect. By not enforcing the serialization we can allow the hardware to run in parallel improving the performance.

Quoted 7 lines

Let's clarify the usage of this member in the tcs_cmd structure to mean only completion and not serialization. This should also improve the performance of bus requests where changes could happen in parallel. Also, CPU resume from deep idle may see benefits from certain wake requests.

You are posting the patch and so you need your SoB too.

Quoted 10 lines
---
 drivers/soc/qcom/rpmh-rsc.c | 19 ++++++++-----------
 include/soc/qcom/tcs.h      |  5 +++--
 2 files changed, 11 insertions(+), 13 deletions(-)
drivers/soc/qcom/rpmh-rsc.c +0 -0
index 076fd27..d99e639 100644

drivers/soc/qcom/rpmh-rsc.c static irqreturn_t tcs_tx_done(int irq, void *p) · line 416

Quoted 6 lines
drivers/soc/qcom/rpmh-rsc.c +1 -2
@@ -413,8 +413,7 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
                        cmd = &req->cmds[j];
                        sts = read_tcs_cmd(drv, RSC_DRV_CMD_STATUS, i, j);
                        if (!(sts & CMD_STATUS_ISSUED) ||
-                          ((req->wait_for_compl || cmd->wait) &&
-                          !(sts & CMD_STATUS_COMPL))) {
+                          (cmd->wait && !(sts & CMD_STATUS_COMPL))) {

I don't quite understand this part of the change. Why don't you need to check "req->wait_for_compl" anymore? You are still setting "CMD_MSGID_RESP_REQ" if "req->wait_for_compl" is set and none of the code in your patch actually sets "cmd->wait" (unlike what's implied in your change to the header file). Maybe some previous version of the patch _was_ actually setting "cmd->wait" and when you stopped doing that you forgot to restore this part of the change?

drivers/soc/qcom/rpmh-rsc.c static irqreturn_t tcs_tx_done(int irq, void *p) · line 436

Quoted 8 lines
drivers/soc/qcom/rpmh-rsc.c +0 -1
                                pr_err("Incomplete request: %s: addr=%#x data=%#x",
                                       drv->name, cmd->addr, cmd->data);
                                err = -EIO;
@@ -433,7 +432,6 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
 skip:
                /* Reclaim the TCS */
                write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, i, 0);
-               write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, i, 0);

Should you also be removing the write to RSC_DRV_CMD_WAIT_FOR_CMPL in both tcs_write() and tcs_invalidate()? Those are the only two places left that set this register and they both always set it to 0.

Quoted 55 lines
drivers/soc/qcom/rpmh-rsc.c +7 -8
                writel_relaxed(BIT(i), drv->tcs_base + RSC_DRV_IRQ_CLEAR);
                spin_lock(&drv->lock);
                clear_bit(i, drv->tcs_in_use);
@@ -465,23 +463,23 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
 static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
                               const struct tcs_request *msg)
 {
-       u32 msgid, cmd_msgid;
+       u32 msgid;
+       u32 cmd_msgid = CMD_MSGID_LEN | CMD_MSGID_WRITE;
        u32 cmd_enable = 0;
-       u32 cmd_complete;
        struct tcs_cmd *cmd;
        int i, j;
 
-       cmd_msgid = CMD_MSGID_LEN;
+       /* Convert all commands to RR when the request has wait_for_compl set */
        cmd_msgid |= msg->wait_for_compl ? CMD_MSGID_RESP_REQ : 0;
-       cmd_msgid |= CMD_MSGID_WRITE;
-
-       cmd_complete = read_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id);
 
        for (i = 0, j = cmd_id; i < msg->num_cmds; i++, j++) {
                cmd = &msg->cmds[i];
                cmd_enable |= BIT(j);
-               cmd_complete |= cmd->wait << j;
                msgid = cmd_msgid;
+               /*
+                * Additionally, if the cmd->wait is set, make the command
+                * response reqd even if the overall request was fire-n-forget.
+                */
                msgid |= cmd->wait ? CMD_MSGID_RESP_REQ : 0;
 
                write_tcs_cmd(drv, RSC_DRV_CMD_MSGID, tcs_id, j, msgid);
@@ -490,7 +488,6 @@ static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
                trace_rpmh_send_msg_rcuidle(drv, tcs_id, j, msgid, cmd);
        }
 
-       write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id, cmd_complete);
        cmd_enable |= read_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id);
        write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id, cmd_enable);
 }
include/soc/qcom/tcs.h +1 -1
index 7a2a055..d1c87fd 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 /*
- * Copyright (c) 2016-2019, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2016-2020, The Linux Foundation. All rights reserved.
  */
 
 #ifndef __SOC_QCOM_TCS_H__

include/soc/qcom/tcs.h enum rpmh_state { · line 33

Quoted 5 lines
include/soc/qcom/tcs.h +1 -1
@@ -30,7 +30,7 @@ enum rpmh_state {
  *
  * @addr: the address of the resource slv_id:18:16 | offset:0:15
  * @data: the resource state request
- * @wait: wait for this request to be complete before sending the next
+ * @wait: Ensure that this command is complete before returning

I have a hard time understanding what you're trying to convey here. Do we even need the "wait" in this structure?

When you call rpmh_write() we use DEFINE_RPMH_MSG_ONSTACK which sets "wait_for_compl", right?

...so I guess you're expecting this to be used for rpmh_write_async()?
...but we never wait in that case, do we?

...or are you expecting rpmh_write_batch() to take advantage of this somehow?

include/soc/qcom/tcs.h struct tcs_cmd { · line 46

Quoted 8 lines
include/soc/qcom/tcs.h +1 -0
  */
 struct tcs_cmd {
        u32 addr;
@@ -43,6 +43,7 @@ struct tcs_cmd {
  *
  * @state:          state for the request.
  * @wait_for_compl: wait until we get a response from the h/w accelerator
+ *                  (sets the cmd->wait for all commmands in the request)

s/commmands/scommands

Also: I don't think this actually goes and modifies "cmd->wait" for all the commands in the request, does it? Maybe say "same as setting cmd->wait for all the commands in the request"?

-Doug

#2 lore raw

Hi,

Quoted 20 lines · Doug Anderson

Hi,

Quoted 17 lines · Maulik Shah

From: Lina Iyer <[email protected]>

Requests sent to RPMH can be sent as fire-n-forget or response required, with the latter ensuring the command has been completed by the hardware accelerator. Commands in a request with tcs_cmd::wait set, would ensure that those select commands are sent as response required, even though the actual TCS request may be fire-n-forget.

Also, commands with .wait flag were also guaranteed to be complete before the following command in the TCS is sent. This means that the next command of the same request blocked until the current request is completed. This could mean waiting for a voltage to settle or series of NOCs be configured before the next command is sent. But drivers using this feature have never cared about the serialization aspect. By not enforcing the serialization we can allow the hardware to run in parallel improving the performance.

Quoted 8 lines
Quoted 7 lines

Let's clarify the usage of this member in the tcs_cmd structure to mean only completion and not serialization. This should also improve the performance of bus requests where changes could happen in parallel. Also, CPU resume from deep idle may see benefits from certain wake requests.

You are posting the patch and so you need your SoB too.

sure, i will add and re-spin.

Quoted 17 lines
Quoted 17 lines

---
  drivers/soc/qcom/rpmh-rsc.c | 19 ++++++++-----------
  include/soc/qcom/tcs.h      |  5 +++--
  2 files changed, 11 insertions(+), 13 deletions(-)
drivers/soc/qcom/rpmh-rsc.c +0 -2
index 076fd27..d99e639 100644
@@ -413,8 +413,7 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
                         cmd = &req->cmds[j];
                         sts = read_tcs_cmd(drv, RSC_DRV_CMD_STATUS, i, j);
                         if (!(sts & CMD_STATUS_ISSUED) ||
-                          ((req->wait_for_compl || cmd->wait) &&
-                          !(sts & CMD_STATUS_COMPL))) {
Quoted 8 lines

drivers/soc/qcom/rpmh-rsc.c static irqreturn_t tcs_tx_done(int irq, void *p) · line 416

Quoted 1 line
drivers/soc/qcom/rpmh-rsc.c +1 -0
+                          (cmd->wait && !(sts & CMD_STATUS_COMPL))) {

I don't quite understand this part of the change. Why don't you need to check "req->wait_for_compl" anymore? You are still setting "CMD_MSGID_RESP_REQ" if "req->wait_for_compl" is set and none of the code in your patch actually sets "cmd->wait" (unlike what's implied in your change to the header file). Maybe some previous version of the patch _was_ actually setting "cmd->wait" and when you stopped doing that you forgot to restore this part of the change?

TCS cmd->wait, will be set by the callers of rpmh APIs. We don't set in our driver.

Quoted 13 lines

drivers/soc/qcom/rpmh-rsc.c static irqreturn_t tcs_tx_done(int irq, void *p) · line 436

Quoted 10 lines
drivers/soc/qcom/rpmh-rsc.c +0 -1
                                 pr_err("Incomplete request: %s: addr=%#x data=%#x",
                                        drv->name, cmd->addr, cmd->data);
                                 err = -EIO;
@@ -433,7 +432,6 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
  skip:
                 /* Reclaim the TCS */
                 write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, i, 0);
-               write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, i, 0);

Should you also be removing the write to RSC_DRV_CMD_WAIT_FOR_CMPL in both tcs_write() and tcs_invalidate()? Those are the only two places left that set this register and they both always set it to 0.

Thanks for pointing this, yes both of these places it can be removed.

Quoted 55 lines
Quoted 55 lines
drivers/soc/qcom/rpmh-rsc.c +7 -8
                 writel_relaxed(BIT(i), drv->tcs_base + RSC_DRV_IRQ_CLEAR);
                 spin_lock(&drv->lock);
                 clear_bit(i, drv->tcs_in_use);
@@ -465,23 +463,23 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
  static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
                                const struct tcs_request *msg)
  {
-       u32 msgid, cmd_msgid;
+       u32 msgid;
+       u32 cmd_msgid = CMD_MSGID_LEN | CMD_MSGID_WRITE;
         u32 cmd_enable = 0;
-       u32 cmd_complete;
         struct tcs_cmd *cmd;
         int i, j;
 
-       cmd_msgid = CMD_MSGID_LEN;
+       /* Convert all commands to RR when the request has wait_for_compl set */
         cmd_msgid |= msg->wait_for_compl ? CMD_MSGID_RESP_REQ : 0;
-       cmd_msgid |= CMD_MSGID_WRITE;
-
-       cmd_complete = read_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id);
 
         for (i = 0, j = cmd_id; i < msg->num_cmds; i++, j++) {
                 cmd = &msg->cmds[i];
                 cmd_enable |= BIT(j);
-               cmd_complete |= cmd->wait << j;
                 msgid = cmd_msgid;
+               /*
+                * Additionally, if the cmd->wait is set, make the command
+                * response reqd even if the overall request was fire-n-forget.
+                */
                 msgid |= cmd->wait ? CMD_MSGID_RESP_REQ : 0;
 
                 write_tcs_cmd(drv, RSC_DRV_CMD_MSGID, tcs_id, j, msgid);
@@ -490,7 +488,6 @@ static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
                 trace_rpmh_send_msg_rcuidle(drv, tcs_id, j, msgid, cmd);
         }
 
-       write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id, cmd_complete);
         cmd_enable |= read_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id);
         write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id, cmd_enable);
  }
include/soc/qcom/tcs.h +1 -1
index 7a2a055..d1c87fd 100644
@@ -1,6 +1,6 @@
  /* SPDX-License-Identifier: GPL-2.0 */
  /*
- * Copyright (c) 2016-2019, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2016-2020, The Linux Foundation. All rights reserved.
   */
 
Quoted 9 lines

include/soc/qcom/tcs.h enum rpmh_state { · line 33

Quoted 7 lines
include/soc/qcom/tcs.h +1 -1
  #ifndef __SOC_QCOM_TCS_H__
@@ -30,7 +30,7 @@ enum rpmh_state {
   *
   * @addr: the address of the resource slv_id:18:16 | offset:0:15
   * @data: the resource state request
- * @wait: wait for this request to be complete before sending the next
+ * @wait: Ensure that this command is complete before returning

I have a hard time understanding what you're trying to convey here. Do we even need the "wait" in this structure?

Yes we need.

Quoted 3 lines

When you call rpmh_write() we use DEFINE_RPMH_MSG_ONSTACK which sets "wait_for_compl", right?

Right.

Quoted 2 lines

...so I guess you're expecting this to be used for rpmh_write_async()?

Correct and also for rpmh_write_batch().

Quoted 1 line

...but we never wait in that case, do we?

tcs cmd->wait will be set by callers of rpmh API.

Quoted 2 lines

...or are you expecting rpmh_write_batch() to take advantage of this somehow?

Yes.

Lets me take an example,

a caller is sending below data in rpmh_write_batch()

CMD0: addr=a, data=x, wait=0,
CMD1: addr=b, data=y, wait=1,
CMD2: addr=c, data=z, wait=0,

now __tcs_buffer_write()  with current code is setting two things,

1. serialize sending the commands (by setting RSC_DRV_CMD_WAIT_FOR_CMPL for CMD1)
This ensured that CMD1 is waited to complete before triggering any new command (CMD2 of TCS) to the HW.

2. wait for completion of commands before returning (by setting CMD_MSGID_RESP_REQ for CMD1) This ensured that CMD1 is complete before generating completion interrupt.

This patch drops serialization part (1), so that all the commands can be triggered at once and can parallelly be finished. if wait=1 is set for some command, the completion interrupt will be generated once those commands are complete in HW.

Quoted 12 lines

include/soc/qcom/tcs.h struct tcs_cmd { · line 46

Quoted 10 lines
include/soc/qcom/tcs.h +1 -0
   */
  struct tcs_cmd {
         u32 addr;
@@ -43,6 +43,7 @@ struct tcs_cmd {
   *
   * @state:          state for the request.
   * @wait_for_compl: wait until we get a response from the h/w accelerator
+ *                  (sets the cmd->wait for all commmands in the request)

s/commmands/scommands

Quoted 3 lines

Also: I don't think this actually goes and modifies "cmd->wait" for all the commands in the request, does it? Maybe say "same as setting cmd->wait for all the commands in the request"?

ok, i will update in next revision.

Quoted 3 lines

-Doug

Signature · 1 line
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
#3 lore raw

Hi,

Quoted 58 lines · Maulik Shah

Hi,

Quoted 20 lines · Doug Anderson

Hi,

Quoted 17 lines · Maulik Shah

From: Lina Iyer <[email protected]>

Requests sent to RPMH can be sent as fire-n-forget or response required, with the latter ensuring the command has been completed by the hardware accelerator. Commands in a request with tcs_cmd::wait set, would ensure that those select commands are sent as response required, even though the actual TCS request may be fire-n-forget.

Also, commands with .wait flag were also guaranteed to be complete before the following command in the TCS is sent. This means that the next command of the same request blocked until the current request is completed. This could mean waiting for a voltage to settle or series of NOCs be configured before the next command is sent. But drivers using this feature have never cared about the serialization aspect. By not enforcing the serialization we can allow the hardware to run in parallel improving the performance.

Quoted 8 lines
Quoted 7 lines

Let's clarify the usage of this member in the tcs_cmd structure to mean only completion and not serialization. This should also improve the performance of bus requests where changes could happen in parallel. Also, CPU resume from deep idle may see benefits from certain wake requests.

You are posting the patch and so you need your SoB too.

sure, i will add and re-spin.

Quoted 17 lines
Quoted 17 lines

---
  drivers/soc/qcom/rpmh-rsc.c | 19 ++++++++-----------
  include/soc/qcom/tcs.h      |  5 +++--
  2 files changed, 11 insertions(+), 13 deletions(-)
drivers/soc/qcom/rpmh-rsc.c +0 -2
index 076fd27..d99e639 100644
@@ -413,8 +413,7 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
                         cmd = &req->cmds[j];
                         sts = read_tcs_cmd(drv, RSC_DRV_CMD_STATUS, i, j);
                         if (!(sts & CMD_STATUS_ISSUED) ||
-                          ((req->wait_for_compl || cmd->wait) &&
-                          !(sts & CMD_STATUS_COMPL))) {
Quoted 8 lines

drivers/soc/qcom/rpmh-rsc.c static irqreturn_t tcs_tx_done(int irq, void *p) · line 416

Quoted 1 line
drivers/soc/qcom/rpmh-rsc.c +1 -0
+                          (cmd->wait && !(sts & CMD_STATUS_COMPL))) {

I don't quite understand this part of the change. Why don't you need to check "req->wait_for_compl" anymore? You are still setting "CMD_MSGID_RESP_REQ" if "req->wait_for_compl" is set and none of the code in your patch actually sets "cmd->wait" (unlike what's implied in your change to the header file). Maybe some previous version of the patch _was_ actually setting "cmd->wait" and when you stopped doing that you forgot to restore this part of the change?

Quoted 2 lines

TCS cmd->wait, will be set by the callers of rpmh APIs. We don't set in our driver.

I still don't understand. In your patch in __tcs_buffer_write(), if "wait_for_compl" is set then "CMD_MSGID_RESP_REQ" will be added for every message in the request, right? That's because you have this bit of code:

/* Convert all commands to RR when the request has wait_for_compl set */ cmd_msgid |= msg->wait_for_compl ? CMD_MSGID_RESP_REQ : 0;

That means that if _either_ "cmd->wait" or "req->wait_for_compl" is set then you expect the "sts" to have "CMD_STATUS_COMPL", right? That's exactly the code that used to be there.

Said another way, if "req->wait_for_compl" was set then it's an error if any of our commands are missing the "CMD_STATUS_COMPL" bit, right?

Quoted 105 lines
Quoted 13 lines

drivers/soc/qcom/rpmh-rsc.c static irqreturn_t tcs_tx_done(int irq, void *p) · line 436

Quoted 10 lines
drivers/soc/qcom/rpmh-rsc.c +0 -1
                                 pr_err("Incomplete request: %s: addr=%#x data=%#x",
                                        drv->name, cmd->addr, cmd->data);
                                 err = -EIO;
@@ -433,7 +432,6 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
  skip:
                 /* Reclaim the TCS */
                 write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, i, 0);
-               write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, i, 0);

Should you also be removing the write to RSC_DRV_CMD_WAIT_FOR_CMPL in both tcs_write() and tcs_invalidate()? Those are the only two places left that set this register and they both always set it to 0.

Thanks for pointing this, yes both of these places it can be removed.

Quoted 55 lines
Quoted 55 lines
drivers/soc/qcom/rpmh-rsc.c +7 -8
                 writel_relaxed(BIT(i), drv->tcs_base + RSC_DRV_IRQ_CLEAR);
                 spin_lock(&drv->lock);
                 clear_bit(i, drv->tcs_in_use);
@@ -465,23 +463,23 @@ static irqreturn_t tcs_tx_done(int irq, void *p)
  static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
                                const struct tcs_request *msg)
  {
-       u32 msgid, cmd_msgid;
+       u32 msgid;
+       u32 cmd_msgid = CMD_MSGID_LEN | CMD_MSGID_WRITE;
         u32 cmd_enable = 0;
-       u32 cmd_complete;
         struct tcs_cmd *cmd;
         int i, j;
 
-       cmd_msgid = CMD_MSGID_LEN;
+       /* Convert all commands to RR when the request has wait_for_compl set */
         cmd_msgid |= msg->wait_for_compl ? CMD_MSGID_RESP_REQ : 0;
-       cmd_msgid |= CMD_MSGID_WRITE;
-
-       cmd_complete = read_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id);
 
         for (i = 0, j = cmd_id; i < msg->num_cmds; i++, j++) {
                 cmd = &msg->cmds[i];
                 cmd_enable |= BIT(j);
-               cmd_complete |= cmd->wait << j;
                 msgid = cmd_msgid;
+               /*
+                * Additionally, if the cmd->wait is set, make the command
+                * response reqd even if the overall request was fire-n-forget.
+                */
                 msgid |= cmd->wait ? CMD_MSGID_RESP_REQ : 0;
 
                 write_tcs_cmd(drv, RSC_DRV_CMD_MSGID, tcs_id, j, msgid);
@@ -490,7 +488,6 @@ static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,
                 trace_rpmh_send_msg_rcuidle(drv, tcs_id, j, msgid, cmd);
         }
 
-       write_tcs_reg(drv, RSC_DRV_CMD_WAIT_FOR_CMPL, tcs_id, cmd_complete);
         cmd_enable |= read_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id);
         write_tcs_reg(drv, RSC_DRV_CMD_ENABLE, tcs_id, cmd_enable);
  }
include/soc/qcom/tcs.h +1 -1
index 7a2a055..d1c87fd 100644
@@ -1,6 +1,6 @@
  /* SPDX-License-Identifier: GPL-2.0 */
  /*
- * Copyright (c) 2016-2019, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2016-2020, The Linux Foundation. All rights reserved.
   */
 
Quoted 9 lines

include/soc/qcom/tcs.h enum rpmh_state { · line 33

Quoted 7 lines
include/soc/qcom/tcs.h +1 -1
  #ifndef __SOC_QCOM_TCS_H__
@@ -30,7 +30,7 @@ enum rpmh_state {
   *
   * @addr: the address of the resource slv_id:18:16 | offset:0:15
   * @data: the resource state request
- * @wait: wait for this request to be complete before sending the next
+ * @wait: Ensure that this command is complete before returning

I have a hard time understanding what you're trying to convey here. Do we even need the "wait" in this structure?

Yes we need.

Quoted 3 lines

When you call rpmh_write() we use DEFINE_RPMH_MSG_ONSTACK which sets "wait_for_compl", right?

Right.

Quoted 2 lines

...so I guess you're expecting this to be used for rpmh_write_async()?

Correct and also for rpmh_write_batch().

Quoted 1 line

...but we never wait in that case, do we?

tcs cmd->wait will be set by callers of rpmh API.

Quoted 2 lines

...or are you expecting rpmh_write_batch() to take advantage of this somehow?

Yes.

Lets me take an example,

a caller is sending below data in rpmh_write_batch()

CMD0: addr=a, data=x, wait=0,
CMD1: addr=b, data=y, wait=1,
CMD2: addr=c, data=z, wait=0,

now __tcs_buffer_write() with current code is setting two things,

Quoted 8 lines

1. serialize sending the commands (by setting RSC_DRV_CMD_WAIT_FOR_CMPL for CMD1)
This ensured that CMD1 is waited to complete before triggering any new command (CMD2 of TCS) to the HW.

2. wait for completion of commands before returning (by setting CMD_MSGID_RESP_REQ for CMD1)
This ensured that CMD1 is complete before generating completion interrupt.

OK, but I guess my question is: who exactly cares about the completion interrupt?

For rpmh_write_async() as soon as the commands are in the queue the function returns. There's no callback and rpmh_write_async() doesn't set the "completion" to anything so there's nobody that cares at all, right? That means, effectively, it's not useful for anyone to set the "wait" member when calling rpmh_write_async().

Ah, I see. So the "wait" value _only_ matters for the batch case and only for "active only" transfers. I guess another thing that would be nice to have in a comment somewhere, assuming I understood it all correctly.

Quoted 27 lines

This patch drops serialization part (1), so that all the commands can be triggered at once and can parallelly be finished.
if wait=1 is set for some command, the completion interrupt will be generated once those commands are complete in HW.

Quoted 12 lines

include/soc/qcom/tcs.h struct tcs_cmd { · line 46

Quoted 10 lines
include/soc/qcom/tcs.h +1 -0
   */
  struct tcs_cmd {
         u32 addr;
@@ -43,6 +43,7 @@ struct tcs_cmd {
   *
   * @state:          state for the request.
   * @wait_for_compl: wait until we get a response from the h/w accelerator
+ *                  (sets the cmd->wait for all commmands in the request)

s/commmands/scommands

Quoted 3 lines

Also: I don't think this actually goes and modifies "cmd->wait" for all the commands in the request, does it? Maybe say "same as setting cmd->wait for all the commands in the request"?

ok, i will update in next revision.

Quoted 3 lines

-Doug

--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation