Skip to contentSkip to conversation

Fix hang due to CPU BW request as BPMP suspended

PATCH Applied Sumit Gupta 3 participants 9 messages Oct 9, 2023 – Oct 13, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover Fix hang due to CPU BW request as BPMP suspended

This patch set fixes hang during system resume which started coming after adding Memory Interconnect and OPP support to the Tegra194 CPUFREQ in below change:
f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth").

Tegra194 CPUFREQ driver uses 'CPUFREQ_NEED_INITIAL_FREQ_CHECK' flag which causes a CPU frequency set request from the 'cpuhp_cpufreq_online' hotplug notifier during resume. The CPU frequency set call also triggers a DRAM bandwidth set request but the BPMP driver hasn't resumed yet which results in hang during resume.

Fix this by resetting the BPMP IPC channels inside tegra_bpmp_transfer*() API if the bandwidth request came from CPU clusters and the BPMP driver is still suspended.

---
v[1] -> v2:
- add suspend hook instead of reset hook based approach.
- add suspended instead of needs_reset flag in 'struct tegra_bpmp'.
- add flags field to 'struct tegra_bpmp_message'.
- set TEGRA_BPMP_MESSAGE_RESET flag from MC driver for CPU BW request.
- use suspended and flags to reset BPMP IPC early on resume.

Sumit Gupta (1):
firmware: tegra: add suspend hook and reset BPMP IPC early on resume

Thierry Reding (1):
memory: tegra: set BPMP msg flags to reset IPC channels

 drivers/firmware/tegra/bpmp.c   | 30 ++++++++++++++++++++++++++++++
 drivers/memory/tegra/tegra234.c |  4 ++++
 include/soc/tegra/bpmp.h        |  6 ++++++
 3 files changed, 40 insertions(+)

[1] https://lore.kernel.org/linux-tegra/[email protected]/

Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/2 firmware: tegra: add suspend hook and reset BPMP IPC early on resume

Add suspend hook and a 'suspended' field in the 'struct tegra_bpmp' to mark if BPMP is suspended. Also, add a 'flags' field in the 'struct tegra_bpmp_message' whose 'TEGRA_BPMP_MESSAGE_RESET' bit can be set from the Tegra MC driver to signal that the reset of BPMP IPC channels is required before sending MRQ to the BPMP FW. Together both the fields allow us to handle any requests that might be sent too soon as they can cause hang during system resume. One case where we see BPMP requests being sent before the BPMP driver has resumed is the memory bandwidth requests which are triggered by onlining the CPUs during system resume. The CPUs are onlined before the BPMP has resumed and we need to reset the BPMP IPC channels to handle these requests.
The additional check for 'flags' is done to avoid any un-intended BPMP IPC reset if the tegra_bpmp_transfer*() API gets called during suspend sequence after the BPMP driver is suspended.

Fixes f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth")
---
 drivers/firmware/tegra/bpmp.c | 30 ++++++++++++++++++++++++++++++
 include/soc/tegra/bpmp.h      |  6 ++++++
 2 files changed, 36 insertions(+)
drivers/firmware/tegra/bpmp.c +30 -0
index 51d062e0c3f1..c1590d3aa9cb 100644
@@ -313,6 +313,8 @@ static ssize_t tegra_bpmp_channel_write(struct tegra_bpmp_channel *channel,
 	return __tegra_bpmp_channel_write(channel, mrq, flags, data, size);
 }
 
+static int __maybe_unused tegra_bpmp_resume(struct device *dev);
+
 int tegra_bpmp_transfer_atomic(struct tegra_bpmp *bpmp,
 			       struct tegra_bpmp_message *msg)
 {
@@ -325,6 +327,14 @@ int tegra_bpmp_transfer_atomic(struct tegra_bpmp *bpmp,
 	if (!tegra_bpmp_message_valid(msg))
 		return -EINVAL;
 
+	if (bpmp->suspended) {
+		/* Reset BPMP IPC channels during resume based on flags passed */
+		if (msg->flags & TEGRA_BPMP_MESSAGE_RESET)
+			tegra_bpmp_resume(bpmp->dev);
+		else
+			return -EAGAIN;
+	}
+
 	channel = bpmp->tx_channel;
 
 	spin_lock(&bpmp->atomic_tx_lock);
@@ -364,6 +374,14 @@ int tegra_bpmp_transfer(struct tegra_bpmp *bpmp,
 	if (!tegra_bpmp_message_valid(msg))
 		return -EINVAL;
 
+	if (bpmp->suspended) {
+		/* Reset BPMP IPC channels during resume based on flags passed */
+		if (msg->flags & TEGRA_BPMP_MESSAGE_RESET)
+			tegra_bpmp_resume(bpmp->dev);
+		else
+			return -EAGAIN;
+	}
+
 	channel = tegra_bpmp_write_threaded(bpmp, msg->mrq, msg->tx.data,
 					    msg->tx.size);
 	if (IS_ERR(channel))
@@ -796,10 +814,21 @@ static int tegra_bpmp_probe(struct platform_device *pdev)
 	return err;
 }
 
+static int __maybe_unused tegra_bpmp_suspend(struct device *dev)
+{
+	struct tegra_bpmp *bpmp = dev_get_drvdata(dev);
+
+	bpmp->suspended = true;
+
+	return 0;
+}
+
 static int __maybe_unused tegra_bpmp_resume(struct device *dev)
 {
 	struct tegra_bpmp *bpmp = dev_get_drvdata(dev);
 
+	bpmp->suspended = false;
+
 	if (bpmp->soc->ops->resume)
 		return bpmp->soc->ops->resume(bpmp);
 	else
@@ -807,6 +836,7 @@ static int __maybe_unused tegra_bpmp_resume(struct device *dev)
 }
 
 static const struct dev_pm_ops tegra_bpmp_pm_ops = {
+	.suspend_noirq = tegra_bpmp_suspend,
 	.resume_noirq = tegra_bpmp_resume,
 };
 
include/soc/tegra/bpmp.h +6 -0
index 5842e38bb288..f5e4ac5b8cce 100644
@@ -102,8 +102,12 @@ struct tegra_bpmp {
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debugfs_mirror;
 #endif
+
+	bool suspended;
 };
 
+#define TEGRA_BPMP_MESSAGE_RESET BIT(0)
+
 struct tegra_bpmp_message {
 	unsigned int mrq;
 
@@ -117,6 +121,8 @@ struct tegra_bpmp_message {
 		size_t size;
 		int ret;
 	} rx;
+
+	unsigned long flags;
 };
 
 #if IS_ENABLED(CONFIG_TEGRA_BPMP)
Signature · 1 line
2.17.1
Acked-by on v2 1/2
#2 lore raw

Re: [Patch v2 1/2] firmware: tegra: add suspend hook and reset BPMP IPC early on resume

Quoted 16 lines · Sumit Gupta

Add suspend hook and a 'suspended' field in the 'struct tegra_bpmp' to mark if BPMP is suspended. Also, add a 'flags' field in the 'struct tegra_bpmp_message' whose 'TEGRA_BPMP_MESSAGE_RESET' bit can be set from the Tegra MC driver to signal that the reset of BPMP IPC channels is required before sending MRQ to the BPMP FW. Together both the fields allow us to handle any requests that might be sent too soon as they can cause hang during system resume. One case where we see BPMP requests being sent before the BPMP driver has resumed is the memory bandwidth requests which are triggered by onlining the CPUs during system resume. The CPUs are onlined before the BPMP has resumed and we need to reset the BPMP IPC channels to handle these requests.
The additional check for 'flags' is done to avoid any un-intended BPMP IPC reset if the tegra_bpmp_transfer*() API gets called during suspend sequence after the BPMP driver is suspended.

Quoted 8 lines
Fixes f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth")
---
 drivers/firmware/tegra/bpmp.c | 30 ++++++++++++++++++++++++++++++
 include/soc/tegra/bpmp.h      |  6 ++++++
 2 files changed, 36 insertions(+)

Krzysztof,

if you want to pick these up instead of me taking them through the Tegra tree:

#3 lore raw

PATCH v2 2/2 memory: tegra: set BPMP msg flags to reset IPC channels

In reply to Sumit Gupta

From: Thierry Reding <[email protected]>

Set the 'TEGRA_BPMP_MESSAGE_RESET' bit in newly added 'flags' field of 'struct tegra_bpmp_message' to request for the reset of BPMP IPC channels. This is used along with the 'suspended' check in BPMP driver for handling early bandwidth requests due to the hotplug of CPU's during system resume before the driver gets resumed.

Fixes f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth")
---
 drivers/memory/tegra/tegra234.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/memory/tegra/tegra234.c +4 -0
index 9e5b5dbd9c8d..2845041f32d6 100644
@@ -986,6 +986,10 @@ static int tegra234_mc_icc_set(struct icc_node *src, struct icc_node *dst)
 	msg.rx.data = &bwmgr_resp;
 	msg.rx.size = sizeof(bwmgr_resp);
 
+	if (pclient->bpmp_id >= TEGRA_ICC_BPMP_CPU_CLUSTER0 &&
+	    pclient->bpmp_id <= TEGRA_ICC_BPMP_CPU_CLUSTER2)
+		msg.flags = TEGRA_BPMP_MESSAGE_RESET;
+
 	ret = tegra_bpmp_transfer(mc->bpmp, &msg);
 	if (ret < 0) {
 		dev_err(mc->dev, "BPMP transfer failed: %d\n", ret);
Signature · 1 line
2.17.1
Acked-by on v2 2/2
#4 lore raw

Re: [Patch v2 2/2] memory: tegra: set BPMP msg flags to reset IPC channels

Quoted 8 lines · Sumit Gupta

From: Thierry Reding <[email protected]>

Set the 'TEGRA_BPMP_MESSAGE_RESET' bit in newly added 'flags' field of 'struct tegra_bpmp_message' to request for the reset of BPMP IPC channels. This is used along with the 'suspended' check in BPMP driver for handling early bandwidth requests due to the hotplug of CPU's during system resume before the driver gets resumed.

Quoted 7 lines
Fixes f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth")
---
 drivers/memory/tegra/tegra234.c | 4 ++++
 1 file changed, 4 insertions(+)

Krzysztof,

this one has a build-time dependency on patch 1/2, so it'd make sense for me to pick this up into the Tegra tree along with patch 1/2. That is slightly easier because I already have a BPMP patch in the tree.

There should be no conflict between this and the Tegra tree, though, so if you feel strongly about it, you could also pick up both patches, in which case:

Quoted 19 lines
drivers/memory/tegra/tegra234.c +4 -0
index 9e5b5dbd9c8d..2845041f32d6 100644
@@ -986,6 +986,10 @@ static int tegra234_mc_icc_set(struct icc_node *src, struct icc_node *dst)
 	msg.rx.data = &bwmgr_resp;
 	msg.rx.size = sizeof(bwmgr_resp);
 
+	if (pclient->bpmp_id >= TEGRA_ICC_BPMP_CPU_CLUSTER0 &&
+	    pclient->bpmp_id <= TEGRA_ICC_BPMP_CPU_CLUSTER2)
+		msg.flags = TEGRA_BPMP_MESSAGE_RESET;
+
 	ret = tegra_bpmp_transfer(mc->bpmp, &msg);
 	if (ret < 0) {
 		dev_err(mc->dev, "BPMP transfer failed: %d\n", ret);
Signature · 1 line
2.17.1
#5 lore raw

Re: [Patch v2 2/2] memory: tegra: set BPMP msg flags to reset IPC channels

Quoted 17 lines · Thierry Reding
Quoted 8 lines · Sumit Gupta

From: Thierry Reding <[email protected]>

Set the 'TEGRA_BPMP_MESSAGE_RESET' bit in newly added 'flags' field of 'struct tegra_bpmp_message' to request for the reset of BPMP IPC channels. This is used along with the 'suspended' check in BPMP driver for handling early bandwidth requests due to the hotplug of CPU's during system resume before the driver gets resumed.

Quoted 7 lines
Fixes f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth")
---
 drivers/memory/tegra/tegra234.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines

Krzysztof,

this one has a build-time dependency on patch 1/2, so it'd make sense for me to pick this up into the Tegra tree along with patch 1/2. That is slightly easier because I already have a BPMP patch in the tree.

Sounds good.

Best regards,
Krzysztof

Acked-by on v2 2/2
#6 lore raw

Re: [Patch v2 2/2] memory: tegra: set BPMP msg flags to reset IPC channels

In reply to Sumit Gupta
Quoted 8 lines · Sumit Gupta

From: Thierry Reding <[email protected]>

Set the 'TEGRA_BPMP_MESSAGE_RESET' bit in newly added 'flags' field of 'struct tegra_bpmp_message' to request for the reset of BPMP IPC channels. This is used along with the 'suspended' check in BPMP driver for handling early bandwidth requests due to the hotplug of CPU's during system resume before the driver gets resumed.

Quoted 4 lines
Fixes f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth")

Best regards,
Krzysztof

#7 lore raw

Re: (subset) [Patch v2 0/2] Fix hang due to CPU BW request as BPMP suspended

In reply to Sumit Gupta

From: Thierry Reding <[email protected]>

Quoted 5 lines · Sumit Gupta

This patch set fixes hang during system resume which started coming after adding Memory Interconnect and OPP support to the Tegra194 CPUFREQ in below change:
f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth").

Quoted 7 lines

Tegra194 CPUFREQ driver uses 'CPUFREQ_NEED_INITIAL_FREQ_CHECK' flag which causes a CPU frequency set request from the 'cpuhp_cpufreq_online' hotplug notifier during resume. The CPU frequency set call also triggers a DRAM bandwidth set request but the BPMP driver hasn't resumed yet which results in hang during resume.

[...]

Applied, thanks!

[2/2] memory: tegra: set BPMP msg flags to reset IPC channels
(no commit info)

Best regards,

Signature · 1 line
Thierry Reding <[email protected]>
#8 lore raw

Re: (subset) [Patch v2 0/2] Fix hang due to CPU BW request as BPMP suspended

Quoted 15 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Quoted 11 lines · Sumit Gupta

This patch set fixes hang during system resume which started coming after adding Memory Interconnect and OPP support to the Tegra194 CPUFREQ in below change:
f41e1442ac5b ("cpufreq: tegra194: add OPP support and set bandwidth").

Tegra194 CPUFREQ driver uses 'CPUFREQ_NEED_INITIAL_FREQ_CHECK' flag which causes a CPU frequency set request from the 'cpuhp_cpufreq_online' hotplug notifier during resume. The CPU frequency set call also triggers a DRAM bandwidth set request but the BPMP driver hasn't resumed yet which results in hang during resume.

Quoted 6 lines
Quoted 1 line

[...]

Applied, thanks!

[2/2] memory: tegra: set BPMP msg flags to reset IPC channels
(no commit info)

For the record, I've actually applied both patches, but applying them to different branches (i.e. resulting in two subsets for the same series) seems to have confused b4.

Thierry