Skip to contentSkip to conversation

TI K3 R5F remoteproc support

PATCH In review Suman Anna 5 participants 32 messages Mar 24, 2020 – Apr 16, 2020

Subsystems (2)
lore
Share & tools
lore

Suman Anna

author
#0 lore raw

cover TI K3 R5F remoteproc support

Hi All,

The following series adds a new K3 R5F remoteproc driver for all the R5F processor clusters/subsystems on TI AM65x and J721E SoCs. The AM65x has a single R5FSS cluster, while J721E has three R5FSS clusters. All clusters are capable of supporting either LockStep (safety compliant providing fault tolerance) or Split (performance mode) mode. The modes themselves are limited on some SoC variants through an eFUSE setting. The IP version and SoC integration is slightly different between AM65x and J721E SoC families leading to couple of functional behavior differences.

The R5Fs are designed to boot out of TCMs with the initial boot-up code on the R5Fs configure the Memory Protection Unit (MPU) to run code from DDR. IPC is through the virtio-rpmsg transport. There is no error recovery or Power Management support at present. The J721E

The patches are on top of the current rproc-next branch, and uses one patch from the OMAP remoteproc series [1]. It also leverages the fixed memory carveout fixes series [2].

The following is the patch summary:
- Patch 1 is an old patch [3] from Loic posted previously to the upstream lists and allows the driver to perform the necessary sequencing w.r.t IP power-on and local resets and provide a balanced state machine across sysfs start/stop, and bind/unbind or module removal.
- Patch 2 is a minor enhancement in remoteproc core to allow dynamically created platform devices to be registered with remoteproc core.
- Patches 3 and 5 add the base binding and the the driver respectively.
- Patch 4 is a common helper that will also be used by a TI K3 DSP remoteproc driver (to be posted in the near future) providing the interface to the System Controller software over TI-SCI for performing device-specific operations.
- Patches 6 and 7 are couple of incremental features to the R5F driver.

regards
Suman

[1] https://patchwork.kernel.org/patch/11455135/
[2] https://patchwork.kernel.org/cover/11447649/
[3] https://patchwork.kernel.org/patch/10251897/

Loic Pallardy (1):
remoteproc: add prepare and unprepare ops

Suman Anna (6):
remoteproc: use a local copy for the name field
dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs remoteproc/k3-r5: Add TI-SCI processor control helper functions remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem remoteproc/k3-r5: Initialize TCM memories for ECC remoteproc/k3-r5: Add loading support for on-chip SRAM regions

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  |  338 ++++
 drivers/remoteproc/Kconfig                    |   16 +
 drivers/remoteproc/Makefile                   |    1 +
 drivers/remoteproc/remoteproc_core.c          |   29 +-
 drivers/remoteproc/ti_k3_r5_remoteproc.c      | 1461 +++++++++++++++++
 drivers/remoteproc/ti_sci_proc.h              |  102 ++
 include/linux/remoteproc.h                    |    6 +-
 7 files changed, 1950 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
 create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c
 create mode 100644 drivers/remoteproc/ti_sci_proc.h
Signature · 1 line
2.23.0
author
#1 lore raw

PATCH 1/7 remoteproc: add prepare and unprepare ops

From: Loic Pallardy <[email protected]>

On some SoC architecture, it is needed to enable HW like clock, bus, regulator, memory region... before loading co-processor firmware.

This patch introduces prepare and unprepare ops to execute platform specific function before firmware loading and after stop execution.

---
 drivers/remoteproc/remoteproc_core.c | 20 +++++++++++++++++++-
 include/linux/remoteproc.h           |  4 ++++
 2 files changed, 23 insertions(+), 1 deletion(-)
drivers/remoteproc/remoteproc_core.c +19 -1
index 26f6947267d2..aca6d022901a 100644
@@ -1394,12 +1394,22 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 		return ret;
 	}
 
+	/* Prepare rproc for firmware loading if needed */
+	if (rproc->ops->prepare) {
+		ret = rproc->ops->prepare(rproc);
+		if (ret) {
+			dev_err(dev, "can't prepare rproc %s: %d\n",
+				rproc->name, ret);
+			goto disable_iommu;
+		}
+	}
+
 	rproc->bootaddr = rproc_get_boot_addr(rproc, fw);
 
 	/* Load resource table, core dump segment list etc from the firmware */
 	ret = rproc_parse_fw(rproc, fw);
 	if (ret)
-		goto disable_iommu;
+		goto unprepare_rproc;
 
 	/* reset max_notifyid */
 	rproc->max_notifyid = -1;
@@ -1433,6 +1443,10 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 	kfree(rproc->cached_table);
 	rproc->cached_table = NULL;
 	rproc->table_ptr = NULL;
+unprepare_rproc:
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
 disable_iommu:
 	rproc_disable_iommu(rproc);
 	return ret;
@@ -1838,6 +1852,10 @@ void rproc_shutdown(struct rproc *rproc)
 	/* clean up all acquired resources */
 	rproc_resource_cleanup(rproc);
 
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
+
 	rproc_disable_iommu(rproc);
 
 	/* Free the copy of the resource table */
include/linux/remoteproc.h +4 -0
index 07bd73a6d72a..ddce7a7775d1 100644
@@ -355,6 +355,8 @@ enum rsc_handling_status {
 
 /**
  * struct rproc_ops - platform-specific device handlers
+ * @prepare:	prepare device for code loading
+ * @unprepare:	unprepare device after stop
  * @start:	power on the device and boot it
  * @stop:	power off the device
  * @kick:	kick a virtqueue (virtqueue id given as a parameter)
@@ -371,6 +373,8 @@ enum rsc_handling_status {
  * @get_boot_addr:	get boot address to entry point specified in firmware
  */
 struct rproc_ops {
+	int (*prepare)(struct rproc *rproc);
+	int (*unprepare)(struct rproc *rproc);
 	int (*start)(struct rproc *rproc);
 	int (*stop)(struct rproc *rproc);
 	void (*kick)(struct rproc *rproc, int vqid);
Signature · 1 line
2.23.0
Reviewed-by on v1 1/7
#2 lore raw

Re: [PATCH 1/7] remoteproc: add prepare and unprepare ops

Quoted 10 lines · Suman Anna

From: Loic Pallardy <[email protected]>

On some SoC architecture, it is needed to enable HW like clock, bus, regulator, memory region... before loading co-processor firmware.

This patch introduces prepare and unprepare ops to execute platform specific function before firmware loading and after stop execution.

Quoted 82 lines
---
 drivers/remoteproc/remoteproc_core.c | 20 +++++++++++++++++++-
 include/linux/remoteproc.h           |  4 ++++
 2 files changed, 23 insertions(+), 1 deletion(-)
drivers/remoteproc/remoteproc_core.c +19 -1
index 26f6947267d2..aca6d022901a 100644
@@ -1394,12 +1394,22 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 		return ret;
 	}
 
+	/* Prepare rproc for firmware loading if needed */
+	if (rproc->ops->prepare) {
+		ret = rproc->ops->prepare(rproc);
+		if (ret) {
+			dev_err(dev, "can't prepare rproc %s: %d\n",
+				rproc->name, ret);
+			goto disable_iommu;
+		}
+	}
+
 	rproc->bootaddr = rproc_get_boot_addr(rproc, fw);
 
 	/* Load resource table, core dump segment list etc from the firmware */
 	ret = rproc_parse_fw(rproc, fw);
 	if (ret)
-		goto disable_iommu;
+		goto unprepare_rproc;
 
 	/* reset max_notifyid */
 	rproc->max_notifyid = -1;
@@ -1433,6 +1443,10 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 	kfree(rproc->cached_table);
 	rproc->cached_table = NULL;
 	rproc->table_ptr = NULL;
+unprepare_rproc:
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
 disable_iommu:
 	rproc_disable_iommu(rproc);
 	return ret;
@@ -1838,6 +1852,10 @@ void rproc_shutdown(struct rproc *rproc)
 	/* clean up all acquired resources */
 	rproc_resource_cleanup(rproc);
 
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
+
 	rproc_disable_iommu(rproc);
 
 	/* Free the copy of the resource table */
include/linux/remoteproc.h +4 -0
index 07bd73a6d72a..ddce7a7775d1 100644
@@ -355,6 +355,8 @@ enum rsc_handling_status {
 
 /**
  * struct rproc_ops - platform-specific device handlers
+ * @prepare:	prepare device for code loading
+ * @unprepare:	unprepare device after stop
  * @start:	power on the device and boot it
  * @stop:	power off the device
  * @kick:	kick a virtqueue (virtqueue id given as a parameter)
@@ -371,6 +373,8 @@ enum rsc_handling_status {
  * @get_boot_addr:	get boot address to entry point specified in firmware
  */
 struct rproc_ops {
+	int (*prepare)(struct rproc *rproc);
+	int (*unprepare)(struct rproc *rproc);
 	int (*start)(struct rproc *rproc);
 	int (*stop)(struct rproc *rproc);
 	void (*kick)(struct rproc *rproc, int vqid);
Signature · 1 line
2.23.0
maintainer Reviewed-by on v1 1/7
#3 lore raw

Re: [PATCH 1/7] remoteproc: add prepare and unprepare ops

In reply to Suman Anna

Good morning Suman,

I have started to work on this set - comments will come in over the next few days...

Quoted 22 lines · Suman Anna

From: Loic Pallardy <[email protected]>

On some SoC architecture, it is needed to enable HW like clock, bus, regulator, memory region... before loading co-processor firmware.

This patch introduces prepare and unprepare ops to execute platform specific function before firmware loading and after stop execution.

---
 drivers/remoteproc/remoteproc_core.c | 20 +++++++++++++++++++-
 include/linux/remoteproc.h           |  4 ++++
 2 files changed, 23 insertions(+), 1 deletion(-)
drivers/remoteproc/remoteproc_core.c +0 -0
index 26f6947267d2..aca6d022901a 100644

drivers/remoteproc/remoteproc_core.c static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw) · line 1399

Quoted 6 lines
drivers/remoteproc/remoteproc_core.c +3 -0
@@ -1394,12 +1394,22 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 		return ret;
 	}
 
+	/* Prepare rproc for firmware loading if needed */
+	if (rproc->ops->prepare) {
+		ret = rproc->ops->prepare(rproc);

In my patchset on MCU synchronisation I have moved ops->{start/stop} to remoteproc_internal.h and called them rproc_start/stop_device() (after Loic's suggestion). In order to be consistent and remove boiler plate code in the core we could do the same, i.e have rproc_prepare/unprepare_device() in remoteproc_internal.h .

With the above:

Thanks,
Mathieu

Quoted 64 lines
drivers/remoteproc/remoteproc_core.c +16 -1
+		if (ret) {
+			dev_err(dev, "can't prepare rproc %s: %d\n",
+				rproc->name, ret);
+			goto disable_iommu;
+		}
+	}
+
 	rproc->bootaddr = rproc_get_boot_addr(rproc, fw);
 
 	/* Load resource table, core dump segment list etc from the firmware */
 	ret = rproc_parse_fw(rproc, fw);
 	if (ret)
-		goto disable_iommu;
+		goto unprepare_rproc;
 
 	/* reset max_notifyid */
 	rproc->max_notifyid = -1;
@@ -1433,6 +1443,10 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 	kfree(rproc->cached_table);
 	rproc->cached_table = NULL;
 	rproc->table_ptr = NULL;
+unprepare_rproc:
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
 disable_iommu:
 	rproc_disable_iommu(rproc);
 	return ret;
@@ -1838,6 +1852,10 @@ void rproc_shutdown(struct rproc *rproc)
 	/* clean up all acquired resources */
 	rproc_resource_cleanup(rproc);
 
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
+
 	rproc_disable_iommu(rproc);
 
 	/* Free the copy of the resource table */
include/linux/remoteproc.h +4 -0
index 07bd73a6d72a..ddce7a7775d1 100644
@@ -355,6 +355,8 @@ enum rsc_handling_status {
 
 /**
  * struct rproc_ops - platform-specific device handlers
+ * @prepare:	prepare device for code loading
+ * @unprepare:	unprepare device after stop
  * @start:	power on the device and boot it
  * @stop:	power off the device
  * @kick:	kick a virtqueue (virtqueue id given as a parameter)
@@ -371,6 +373,8 @@ enum rsc_handling_status {
  * @get_boot_addr:	get boot address to entry point specified in firmware
  */
 struct rproc_ops {
+	int (*prepare)(struct rproc *rproc);
+	int (*unprepare)(struct rproc *rproc);
 	int (*start)(struct rproc *rproc);
 	int (*stop)(struct rproc *rproc);
 	void (*kick)(struct rproc *rproc, int vqid);
Signature · 1 line
2.23.0
author
#4 lore raw

Re: [PATCH 1/7] remoteproc: add prepare and unprepare ops

Hi Mathieu,

Quoted 31 lines · Mathieu Poirier

Good morning Suman,

I have started to work on this set - comments will come in over the next few days...

Quoted 25 lines · Suman Anna

From: Loic Pallardy <[email protected]>

On some SoC architecture, it is needed to enable HW like clock, bus, regulator, memory region... before loading co-processor firmware.

This patch introduces prepare and unprepare ops to execute platform specific function before firmware loading and after stop execution.

---
 drivers/remoteproc/remoteproc_core.c | 20 +++++++++++++++++++-
 include/linux/remoteproc.h           |  4 ++++
 2 files changed, 23 insertions(+), 1 deletion(-)
drivers/remoteproc/remoteproc_core.c +0 -0
index 26f6947267d2..aca6d022901a 100644
@@ -1394,12 +1394,22 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 		return ret;
 	}
 
Quoted 9 lines

drivers/remoteproc/remoteproc_core.c static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw) · line 1399

Quoted 3 lines
drivers/remoteproc/remoteproc_core.c +3 -0
+	/* Prepare rproc for firmware loading if needed */
+	if (rproc->ops->prepare) {
+		ret = rproc->ops->prepare(rproc);

In my patchset on MCU synchronisation I have moved ops->{start/stop} to remoteproc_internal.h and called them rproc_start/stop_device() (after Loic's suggestion). In order to be consistent and remove boiler plate code in the core we could do the same, i.e have rproc_prepare/unprepare_device() in remoteproc_internal.h .

Yes, I will update so for v2. I plan to separate out this patch and the next patch for v2 to go alongside your cleanup and Alex's patch on the idr_init move.

regards
Suman

Quoted 72 lines

With the above:

Thanks,
Mathieu

Quoted 64 lines
drivers/remoteproc/remoteproc_core.c +16 -1
+		if (ret) {
+			dev_err(dev, "can't prepare rproc %s: %d\n",
+				rproc->name, ret);
+			goto disable_iommu;
+		}
+	}
+
 	rproc->bootaddr = rproc_get_boot_addr(rproc, fw);
 
 	/* Load resource table, core dump segment list etc from the firmware */
 	ret = rproc_parse_fw(rproc, fw);
 	if (ret)
-		goto disable_iommu;
+		goto unprepare_rproc;
 
 	/* reset max_notifyid */
 	rproc->max_notifyid = -1;
@@ -1433,6 +1443,10 @@ static int rproc_fw_boot(struct rproc *rproc, const struct firmware *fw)
 	kfree(rproc->cached_table);
 	rproc->cached_table = NULL;
 	rproc->table_ptr = NULL;
+unprepare_rproc:
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
 disable_iommu:
 	rproc_disable_iommu(rproc);
 	return ret;
@@ -1838,6 +1852,10 @@ void rproc_shutdown(struct rproc *rproc)
 	/* clean up all acquired resources */
 	rproc_resource_cleanup(rproc);
 
+	/* release HW resources if needed */
+	if (rproc->ops->unprepare)
+		rproc->ops->unprepare(rproc);
+
 	rproc_disable_iommu(rproc);
 
 	/* Free the copy of the resource table */
include/linux/remoteproc.h +4 -0
index 07bd73a6d72a..ddce7a7775d1 100644
@@ -355,6 +355,8 @@ enum rsc_handling_status {
 
 /**
  * struct rproc_ops - platform-specific device handlers
+ * @prepare:	prepare device for code loading
+ * @unprepare:	unprepare device after stop
  * @start:	power on the device and boot it
  * @stop:	power off the device
  * @kick:	kick a virtqueue (virtqueue id given as a parameter)
@@ -371,6 +373,8 @@ enum rsc_handling_status {
  * @get_boot_addr:	get boot address to entry point specified in firmware
  */
 struct rproc_ops {
+	int (*prepare)(struct rproc *rproc);
+	int (*unprepare)(struct rproc *rproc);
 	int (*start)(struct rproc *rproc);
 	int (*stop)(struct rproc *rproc);
 	void (*kick)(struct rproc *rproc, int vqid);
Signature · 1 line
2.23.0
author
#5 lore raw

PATCH 2/7 remoteproc: use a local copy for the name field

In reply to Suman Anna

The current name field used in the remoteproc structure is simply a pointer to a name field supplied during the rproc_alloc() call. The pointer passed in by remoteproc drivers during registration is typically a dev_name pointer, but it is possible that the pointer will no longer remain valid if the devices themselves were created at runtime like in the case of of_platform_populate(), and were deleted upon any failures within the respective remoteproc driver probe function.

So, allocate and maintain a local copy for this name field to keep it agnostic of the logic used in the remoteproc drivers.

---
 drivers/remoteproc/remoteproc_core.c | 9 ++++++++-
 include/linux/remoteproc.h           | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/remoteproc/remoteproc_core.c +8 -1
index aca6d022901a..6e0b91fa6f11 100644
@@ -1989,6 +1989,7 @@ static void rproc_type_release(struct device *dev)
 
 	kfree(rproc->firmware);
 	kfree(rproc->ops);
+	kfree(rproc->name);
 	kfree(rproc);
 }
 
@@ -2061,7 +2062,13 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 	}
 
 	rproc->firmware = p;
-	rproc->name = name;
+	rproc->name = kstrdup(name, GFP_KERNEL);
+	if (!rproc->name) {
+		kfree(p);
+		kfree(rproc->ops);
+		kfree(rproc);
+		return NULL;
+	}
 	rproc->priv = &rproc[1];
 	rproc->auto_boot = true;
 	rproc->elf_class = ELFCLASS32;
include/linux/remoteproc.h +1 -1
index ddce7a7775d1..77788a4bb94e 100644
@@ -490,7 +490,7 @@ struct rproc_dump_segment {
 struct rproc {
 	struct list_head node;
 	struct iommu_domain *domain;
-	const char *name;
+	char *name;
 	char *firmware;
 	void *priv;
 	struct rproc_ops *ops;
Signature · 1 line
2.23.0
#6 lore raw

Re: [PATCH 2/7] remoteproc: use a local copy for the name field

Quoted 31 lines · Suman Anna

The current name field used in the remoteproc structure is simply a pointer to a name field supplied during the rproc_alloc() call. The pointer passed in by remoteproc drivers during registration is typically a dev_name pointer, but it is possible that the pointer will no longer remain valid if the devices themselves were created at runtime like in the case of of_platform_populate(), and were deleted upon any failures within the respective remoteproc driver probe function.

So, allocate and maintain a local copy for this name field to keep it agnostic of the logic used in the remoteproc drivers.

---
 drivers/remoteproc/remoteproc_core.c | 9 ++++++++-
 include/linux/remoteproc.h           | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/remoteproc/remoteproc_core.c +1 -0
index aca6d022901a..6e0b91fa6f11 100644
@@ -1989,6 +1989,7 @@ static void rproc_type_release(struct device *dev)
 
 	kfree(rproc->firmware);
 	kfree(rproc->ops);
+	kfree(rproc->name);
 	kfree(rproc);
 }
 

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2065

Quoted 5 lines
drivers/remoteproc/remoteproc_core.c +1 -1
@@ -2061,7 +2062,13 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 	}
 
 	rproc->firmware = p;
-	rproc->name = name;
+	rproc->name = kstrdup(name, GFP_KERNEL);

Let's use kstrdup_const() instead here (and kfree_const() instead of kfree()), so that the cases where we are passed a constant we won't create a duplicate on the heap.

And the "name" in struct rproc can remain const.

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2070

Quoted 5 lines
drivers/remoteproc/remoteproc_core.c +5 -0
+	if (!rproc->name) {
+		kfree(p);
+		kfree(rproc->ops);
+		kfree(rproc);
+		return NULL;

Perhaps we can rearrange the hunks here slightly and get to a point where we can rely on the release function earlier?

Regards,
Bjorn

Quoted 20 lines
drivers/remoteproc/remoteproc_core.c +1 -0
+	}
 	rproc->priv = &rproc[1];
 	rproc->auto_boot = true;
 	rproc->elf_class = ELFCLASS32;
include/linux/remoteproc.h +1 -1
index ddce7a7775d1..77788a4bb94e 100644
@@ -490,7 +490,7 @@ struct rproc_dump_segment {
 struct rproc {
 	struct list_head node;
 	struct iommu_domain *domain;
-	const char *name;
+	char *name;
 	char *firmware;
 	void *priv;
 	struct rproc_ops *ops;
Signature · 1 line
2.23.0
author
#7 lore raw

Re: [PATCH 2/7] remoteproc: use a local copy for the name field

Hi Bjorn,

Quoted 43 lines · Bjorn Andersson
Quoted 31 lines · Suman Anna

The current name field used in the remoteproc structure is simply a pointer to a name field supplied during the rproc_alloc() call. The pointer passed in by remoteproc drivers during registration is typically a dev_name pointer, but it is possible that the pointer will no longer remain valid if the devices themselves were created at runtime like in the case of of_platform_populate(), and were deleted upon any failures within the respective remoteproc driver probe function.

So, allocate and maintain a local copy for this name field to keep it agnostic of the logic used in the remoteproc drivers.

---
 drivers/remoteproc/remoteproc_core.c | 9 ++++++++-
 include/linux/remoteproc.h           | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/remoteproc/remoteproc_core.c +1 -0
index aca6d022901a..6e0b91fa6f11 100644
@@ -1989,6 +1989,7 @@ static void rproc_type_release(struct device *dev)
 
 	kfree(rproc->firmware);
 	kfree(rproc->ops);
+	kfree(rproc->name);
 	kfree(rproc);
 }
 

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2065

Quoted 5 lines
drivers/remoteproc/remoteproc_core.c +1 -1
@@ -2061,7 +2062,13 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 	}
 
 	rproc->firmware = p;
-	rproc->name = name;
+	rproc->name = kstrdup(name, GFP_KERNEL);

Let's use kstrdup_const() instead here (and kfree_const() instead of kfree()), so that the cases where we are passed a constant we won't create a duplicate on the heap.

Quoted 1 line

And the "name" in struct rproc can remain const.

Agreed, that's better functions to use for this.

Quoted 9 lines

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2070

Quoted 6 lines
drivers/remoteproc/remoteproc_core.c +5 -0
+	if (!rproc->name) {
+		kfree(p);
+		kfree(rproc->ops);
+		kfree(rproc);
+		return NULL;

Perhaps we can rearrange the hunks here slightly and get to a point where we can rely on the release function earlier?

Not sure I understand. I don't see any release function, all failure paths in rproc_alloc() directly unwind the previous operations. You mean move this to before the alloc for rproc structure, something similar to what we are doing with firmware?

regards
Suman

Quoted 24 lines

Regards,
Bjorn

Quoted 20 lines
drivers/remoteproc/remoteproc_core.c +1 -0
+	}
 	rproc->priv = &rproc[1];
 	rproc->auto_boot = true;
 	rproc->elf_class = ELFCLASS32;
include/linux/remoteproc.h +1 -1
index ddce7a7775d1..77788a4bb94e 100644
@@ -490,7 +490,7 @@ struct rproc_dump_segment {
 struct rproc {
 	struct list_head node;
 	struct iommu_domain *domain;
-	const char *name;
+	char *name;
 	char *firmware;
 	void *priv;
 	struct rproc_ops *ops;
Signature · 1 line
2.23.0
#8 lore raw

Re: [PATCH 2/7] remoteproc: use a local copy for the name field

Quoted 60 lines · Suman Anna

Hi Bjorn,

Quoted 43 lines · Bjorn Andersson
Quoted 31 lines · Suman Anna

The current name field used in the remoteproc structure is simply a pointer to a name field supplied during the rproc_alloc() call. The pointer passed in by remoteproc drivers during registration is typically a dev_name pointer, but it is possible that the pointer will no longer remain valid if the devices themselves were created at runtime like in the case of of_platform_populate(), and were deleted upon any failures within the respective remoteproc driver probe function.

So, allocate and maintain a local copy for this name field to keep it agnostic of the logic used in the remoteproc drivers.

---
 drivers/remoteproc/remoteproc_core.c | 9 ++++++++-
 include/linux/remoteproc.h           | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/remoteproc/remoteproc_core.c +1 -0
index aca6d022901a..6e0b91fa6f11 100644
@@ -1989,6 +1989,7 @@ static void rproc_type_release(struct device *dev)
 
 	kfree(rproc->firmware);
 	kfree(rproc->ops);
+	kfree(rproc->name);
 	kfree(rproc);
 }
 

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2065

Quoted 5 lines
drivers/remoteproc/remoteproc_core.c +1 -1
@@ -2061,7 +2062,13 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 	}
 
 	rproc->firmware = p;
-	rproc->name = name;
+	rproc->name = kstrdup(name, GFP_KERNEL);

Let's use kstrdup_const() instead here (and kfree_const() instead of kfree()), so that the cases where we are passed a constant we won't create a duplicate on the heap.

Quoted 1 line

And the "name" in struct rproc can remain const.

Agreed, that's better functions to use for this.

Quoted 9 lines

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2070

Quoted 6 lines
drivers/remoteproc/remoteproc_core.c +5 -0
+	if (!rproc->name) {
+		kfree(p);
+		kfree(rproc->ops);
+		kfree(rproc);
+		return NULL;

Perhaps we can rearrange the hunks here slightly and get to a point where we can rely on the release function earlier?

Quoted 5 lines

Not sure I understand. I don't see any release function, all failure paths in rproc_alloc() directly unwind the previous operations. You mean move this to before the alloc for rproc structure, something similar to what we are doing with firmware?

Look at the failure for ida_simple_get(), there we're past the setup of rproc->dev.type, so the rproc_type->release function will be invoked as we call put_device().

So if you move the initialization of rproc->dev up right after the allocation of rproc we should be able to rely on that to clean up all these for us.

Regards,
Bjorn

Quoted 29 lines

regards
Suman

Quoted 24 lines

Regards,
Bjorn

Quoted 20 lines
drivers/remoteproc/remoteproc_core.c +1 -0
+	}
 	rproc->priv = &rproc[1];
 	rproc->auto_boot = true;
 	rproc->elf_class = ELFCLASS32;
include/linux/remoteproc.h +1 -1
index ddce7a7775d1..77788a4bb94e 100644
@@ -490,7 +490,7 @@ struct rproc_dump_segment {
 struct rproc {
 	struct list_head node;
 	struct iommu_domain *domain;
-	const char *name;
+	char *name;
 	char *firmware;
 	void *priv;
 	struct rproc_ops *ops;
Signature · 1 line
2.23.0
author
#9 lore raw

Re: [PATCH 2/7] remoteproc: use a local copy for the name field

Quoted 68 lines · Bjorn Andersson
Quoted 60 lines · Suman Anna

Hi Bjorn,

Quoted 43 lines · Bjorn Andersson
Quoted 31 lines · Suman Anna

The current name field used in the remoteproc structure is simply a pointer to a name field supplied during the rproc_alloc() call. The pointer passed in by remoteproc drivers during registration is typically a dev_name pointer, but it is possible that the pointer will no longer remain valid if the devices themselves were created at runtime like in the case of of_platform_populate(), and were deleted upon any failures within the respective remoteproc driver probe function.

So, allocate and maintain a local copy for this name field to keep it agnostic of the logic used in the remoteproc drivers.

---
 drivers/remoteproc/remoteproc_core.c | 9 ++++++++-
 include/linux/remoteproc.h           | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/remoteproc/remoteproc_core.c +1 -0
index aca6d022901a..6e0b91fa6f11 100644
@@ -1989,6 +1989,7 @@ static void rproc_type_release(struct device *dev)
 
 	kfree(rproc->firmware);
 	kfree(rproc->ops);
+	kfree(rproc->name);
 	kfree(rproc);
 }
 

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2065

Quoted 5 lines
drivers/remoteproc/remoteproc_core.c +1 -1
@@ -2061,7 +2062,13 @@ struct rproc *rproc_alloc(struct device *dev, const char *name,
 	}
 
 	rproc->firmware = p;
-	rproc->name = name;
+	rproc->name = kstrdup(name, GFP_KERNEL);

Let's use kstrdup_const() instead here (and kfree_const() instead of kfree()), so that the cases where we are passed a constant we won't create a duplicate on the heap.

Quoted 1 line

And the "name" in struct rproc can remain const.

Agreed, that's better functions to use for this.

Quoted 9 lines

drivers/remoteproc/remoteproc_core.c struct rproc *rproc_alloc(struct device *dev, const char *name, · line 2070

Quoted 6 lines
drivers/remoteproc/remoteproc_core.c +5 -0
+	if (!rproc->name) {
+		kfree(p);
+		kfree(rproc->ops);
+		kfree(rproc);
+		return NULL;

Perhaps we can rearrange the hunks here slightly and get to a point where we can rely on the release function earlier?

Quoted 5 lines

Not sure I understand. I don't see any release function, all failure paths in rproc_alloc() directly unwind the previous operations. You mean move this to before the alloc for rproc structure, something similar to what we are doing with firmware?

Quoted 7 lines

Look at the failure for ida_simple_get(), there we're past the setup of rproc->dev.type, so the rproc_type->release function will be invoked as we call put_device().

So if you move the initialization of rproc->dev up right after the allocation of rproc we should be able to rely on that to clean up all these for us.

Yeah ok. That's cleanup though, and probably a patch of its own, and not directly related to the subject of this patch. Yeah, I can rework this patch to sit on top of that cleanup patch.

regards
Suman

Quoted 33 lines

Regards,
Bjorn

Quoted 29 lines

regards
Suman

Quoted 24 lines

Regards,
Bjorn

Quoted 20 lines
drivers/remoteproc/remoteproc_core.c +1 -0
+	}
 	rproc->priv = &rproc[1];
 	rproc->auto_boot = true;
 	rproc->elf_class = ELFCLASS32;
include/linux/remoteproc.h +1 -1
index ddce7a7775d1..77788a4bb94e 100644
@@ -490,7 +490,7 @@ struct rproc_dump_segment {
 struct rproc {
 	struct list_head node;
 	struct iommu_domain *domain;
-	const char *name;
+	char *name;
 	char *firmware;
 	void *priv;
 	struct rproc_ops *ops;
Signature · 1 line
2.23.0
author
#10 lore raw

PATCH 3/7 dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

In reply to Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2. No issues are found with dtbs_check.

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +338 -0
new file mode 100644
index 000000000000..bbfc1e6ae884
@@ -0,0 +1,338 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/ti,k3-r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI K3 R5F processor subsystems
+
+maintainers:
+  - Suman Anna <[email protected]>
+
+description: |
+  The TI K3 family of SoCs usually have one or more dual-core Arm Cortex R5F
+  processor subsystems/clusters (R5FSS). The dual core cluster can be used
+  either in a LockStep mode providing safety/fault tolerance features or in a
+  Split mode providing two individual compute cores for doubling the compute
+  capacity. These are used together with other processors present on the SoC
+  to achieve various system level goals.
+
+  Each Dual-Core R5F sub-system is represented as a single DTS node
+  representing the cluster, with a pair of child DT nodes representing
+  the individual R5F cores. Each node has a number of required or optional
+  properties that enable the OS running on the host processor to perform
+  the device management of the remote processor and to communicate with the
+  remote processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:
+    enum:
+      - ti,am654-r5fss
+      - ti,j721e-r5fss
+
+  power-domains:
+    description: |
+      Should contain a phandle to a PM domain provider node and an args
+      specifier containing the R5FSS device id value. This property is
+      as per the binding,
+      Documentation/devicetree/bindings/soc/ti/sci-pm-domain.txt
+    maxItems: 1
+
+  "#address-cells":
+    const: 1
+
+  "#size-cells":
+    const: 1
+
+  ranges:
+    description: |
+      Standard ranges definition providing address translations for
+      local R5F TCM address spaces to bus addresses.
+
+# Optional properties:
+# --------------------
+
+  lockstep-mode:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F
+      cluster. Should be either a value of 1 (LockStep mode) or
+      0 (Split mode), default is LockStep mode if omitted.
+
+# R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f@[a-f0-9]+$":
+    type: object
+    description: |
+        The R5F Sub-System device node should define two R5F child nodes, each
+        node representing a TI instantiation of the Arm Cortex R5F core. There
+        are some specific integration differences for the IP like the usage of
+        a Region Address Translator (RAT) for translating the larger SoC bus
+        addresses into a 32-bit address space for the processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+    properties:
+      compatible:
+        enum:
+          - ti,am654-r5f
+          - ti,j721e-r5f
+
+      reg:
+        description: |
+          Should contain an entry for each value in 'reg-names'.
+          Each entry should have the memory region's start address
+          and the size of the region, the representation matching
+          the parent node's '#address-cells' and '#size-cells' values.
+        maxItems: 2
+
+      reg-names:
+        description: |
+          Should contain strings with the names of the specific internal
+          internal memory regions, and should be defined in this order
+        maxItems: 2
+        items:
+          - const: atcm
+          - const: btcm
+
+      ti,sci:
+        $ref: /schemas/types.yaml#/definitions/phandle
+        description:
+          Should be a phandle to the TI-SCI System Controller node
+
+      ti,sci-dev-id:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        description: |
+          Should contain the TI-SCI device id corresponding to the R5F core.
+          Please refer to the corresponding System Controller documentation
+          for valid values for the R5F cores.
+
+      ti,sci-proc-ids:
+        description: Should contain a single tuple of <proc_id host_id>.
+        allOf:
+          - $ref: /schemas/types.yaml#/definitions/uint32-matrix
+          - maxItems: 1
+            items:
+              items:
+                - description: TI-SCI processor id for the R5F core device
+                - description: TI-SCI host id to which processor control
+                               ownership should be transferred to
+
+      resets:
+        description: |
+          Should contain the phandle to the reset controller node
+          managing the resets for this device, and a reset
+          specifier. Please refer to the following reset bindings
+          for the reset argument specifier,
+          Documentation/devicetree/bindings/reset/ti,sci-reset.txt
+            for AM65x and J721E SoCs
+
+      firmware-name:
+        description: |
+          Should contain the name of the default firmware image
+          file located on the firmware search path
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split
+# modes, and are mandatory for R5F Core1 _only_ in Split mode. They are unused for
+# R5F Core1 in LockStep mode:
+
+      mboxes:
+        description: |
+          OMAP Mailbox specifier denoting the sub-mailbox, to be used for
+          communication with the remote processor. This property should match
+          with the sub-mailbox node used in the firmware image. The specifier
+          format is as per the bindings,
+          Documentation/devicetree/bindings/mailbox/omap-mailbox.txt
+
+      memory-region:
+        minItems: 2
+        description: |
+          phandle to the reserved memory nodes to be associated with the remoteproc
+          device. There should be atleast two reserved memory nodes defined - the
+          first one would be used for dynamic DMA allocations like vrings and vring
+          buffers, and the remaining ones used for the firmware image sections. The
+          reserved memory nodes should be carveout nodes, and should be defined as
+          per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+# Optional properties:
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      atcm-enable:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if ATCM should be enabled. R5F
+          view of ATCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is disabled if omitted.
+
+      btcm-enable:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if BTCM should be enabled. R5F
+          view of BTCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is enabled if omitted.
+
+      loczrama:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating which TCM should appear at
+          address 0 (from core's view). Should be either a value of 1 (ATCM
+          at 0x0) or 0 (BTCM at 0x0), default value is 1 if omitted.
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        description: |
+          pHandles to one or more reserved on-chip SRAM region. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in,
+          Documentation/devicetree/bindings/sram/sram.yaml
+
+    required:
+     - compatible
+     - reg
+     - reg-names
+     - ti,sci
+     - ti,sci-dev-id
+     - ti,sci-proc-ids
+     - resets
+     - firmware-name
+
+    additionalProperties: false
+
+required:
+ - compatible
+ - power-domains
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+
+    //Example: AM654 SoC
+    /* R5F DDR Carveout reserved memory nodes */
+    reserved-memory {
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges;
+
+        mcu_r5fss0_core1_dma_memory_region: r5f-dma-memory@9b000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core1_memory_region: r5f-memory@9b100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b100000 0x00 0xf00000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_dma_memory_region: r5f-dma-memory@9c000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_memory_region: r5f-memory@9c100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c100000 0x00 0x700000>;
+            no-map;
+        };
+    };
+
+    cbass_main: interconnect@100000 {
+        compatible = "simple-bus";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>,
+                 <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>,
+                 <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>;
+
+        cbass_mcu: interconnect@28380000 {
+            compatible = "simple-bus";
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>, /* MCU R5F Core0 */
+                     <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>, /* MCU R5F Core1 */
+                     <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>; /* MCU SRAM */
+
+            /* MCU domain SRAM node */
+            mcu_ram: mcu-ram@41c00000 {
+                compatible = "mmio-sram";
+                reg = <0x00 0x41c00000 0x00 0x80000>;
+                ranges = <0x0 0x00 0x41c00000 0x80000>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+
+                mcu_r5fss0_core0_sram: r5f-sram@0 {
+                    reg = <0x0 0x40000>;
+                };
+            };
+
+            /* AM65x MCU R5FSS node */
+            mcu_r5fss0: r5fss@41000000 {
+                compatible = "ti,am654-r5fss";
+                power-domains = <&k3_pds 129>;
+                lockstep-mode = <1>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+                ranges = <0x41000000 0x00 0x41000000 0x20000>,
+                         <0x41400000 0x00 0x41400000 0x20000>;
+
+                mcu_r5f0: r5f@41000000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41000000 0x00008000>,
+                          <0x41010000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <159>;
+                    ti,sci-proc-ids = <0x01 0xFF>;
+                    resets = <&k3_reset 159 1>;
+                    firmware-name = "am65x-mcu-r5f0_0-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox0 &mbox_mcu_r5fss0_core0>;
+                    memory-region = <&mcu_r5fss0_core0_dma_memory_region>,
+                                    <&mcu_r5fss0_core0_memory_region>;
+                    sram = <&mcu_r5fss0_core0_sram>;
+                };
+
+                mcu_r5f1: r5f@41400000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41400000 0x00008000>,
+                          <0x41410000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <245>;
+                    ti,sci-proc-ids = <0x02 0xFF>;
+                    resets = <&k3_reset 245 1>;
+                    firmware-name = "am65x-mcu-r5f0_1-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox1 &mbox_mcu_r5fss0_core1>;
+               };
+           };
+        };
+    };
Signature · 1 line
2.23.0
maintainer
#11 lore raw

Re: [PATCH 3/7] dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

Quoted 24 lines · Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2. No issues are found with dtbs_check.

Quoted 7 lines

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml

My bot found errors running 'make dt_binding_check' on your patch:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dts:23.13-20: Warning (ranges_format): /example-0/reserved-memory:ranges: empty "ranges" property but its #address-cells (2) differs from /example-0 (1) Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dts:23.13-20: Warning (ranges_format): /example-0/reserved-memory:ranges: empty "ranges" property but its #size-cells (2) differs from /example-0 (1) Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dts:54.13-56.72: Warning (ranges_format): /example-0/interconnect@100000:ranges: "ranges" property has invalid length (72 bytes) (parent #address-cells == 1, child #address-cells == 2, #size-cells == 2)

/builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: interconnect@100000: $nodename:0: 'interconnect@100000' does not match '^(bus|soc|axi|ahb|apb)(@[0-9a-f]+)?$'
/builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: interconnect@28380000: $nodename:0: 'interconnect@28380000' does not match '^(bus|soc|axi|ahb|apb)(@[0-9a-f]+)?$'

/builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: mcu-ram@41c00000: 'r5f-sram@0' does not match any of the regexes: '^([a-z]*-)?sram(-section)?@[a-f0-9]+$', 'pinctrl-[0-9]+' /builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: mcu-ram@41c00000: $nodename:0: 'mcu-ram@41c00000' does not match '^sram(@.*)?'

See https://patchwork.ozlabs.org/patch/1260966

If you already ran 'make dt_binding_check' and didn't see the above error(s), then make sure dt-schema is up to date:

pip3 install git+https://github.com/devicetree-org/dt-schema.git@master --upgrade

Please check and re-submit.

author
#12 lore raw

Re: [PATCH 3/7] dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

Hi Rob,

Quoted 33 lines · Rob Herring
Quoted 24 lines · Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2. No issues are found with dtbs_check.

Quoted 7 lines

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
Quoted 5 lines

My bot found errors running 'make dt_binding_check' on your patch:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dts:23.13-20: Warning (ranges_format): /example-0/reserved-memory:ranges: empty "ranges" property but its #address-cells (2) differs from /example-0 (1) Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dts:23.13-20: Warning (ranges_format): /example-0/reserved-memory:ranges: empty "ranges" property but its #size-cells (2) differs from /example-0 (1) Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dts:54.13-56.72: Warning (ranges_format): /example-0/interconnect@100000:ranges: "ranges" property has invalid length (72 bytes) (parent #address-cells == 1, child #address-cells == 2, #size-cells == 2)

I only saw these, because the generated example dts is using cell values of 1. I tried adding them using 2 in my example, but then it complains about duplicate properties. I do not know how to fix these though. Same with the K3 DSP bindings patches as well.

Quoted 4 lines
/builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: interconnect@100000: $nodename:0: 'interconnect@100000' does not match '^(bus|soc|axi|ahb|apb)(@[0-9a-f]+)?$'
/builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: interconnect@28380000: $nodename:0: 'interconnect@28380000' does not match '^(bus|soc|axi|ahb|apb)(@[0-9a-f]+)?$'

/builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: mcu-ram@41c00000: 'r5f-sram@0' does not match any of the regexes: '^([a-z]*-)?sram(-section)?@[a-f0-9]+$', 'pinctrl-[0-9]+' /builds/robherring/linux-dt-review/Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.example.dt.yaml: mcu-ram@41c00000: $nodename:0: 'mcu-ram@41c00000' does not match '^sram(@.*)?'

These node names are already upstream. Do I just update the example or the upstream dts also needs fixing?

Quoted 7 lines

See https://patchwork.ozlabs.org/patch/1260966

If you already ran 'make dt_binding_check' and didn't see the above error(s), then make sure dt-schema is up to date:

pip3 install git+https://github.com/devicetree-org/dt-schema.git@master --upgrade

Thanks, will upgrade my dt-schema.

regards
Suman

#13 lore raw

Re: [PATCH 3/7] dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

In reply to Suman Anna
Quoted 12 lines · Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

I'm inclined to say the system DT stuff should be sorted out before accepting this. Is the system DT stuff going to be useful for your R5 cores? Do you really want to be stuck with this binding?

Quoted 10 lines

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2.

Then change the default by specifying what you want. Or change the example to be 1 cell. It is *just* an example.

Quoted 1 line

No issues are found with dtbs_check.

I doubt that if your dts matches the example.

Quoted 49 lines

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +35 -0
new file mode 100644
index 000000000000..bbfc1e6ae884
@@ -0,0 +1,338 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/ti,k3-r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI K3 R5F processor subsystems
+
+maintainers:
+  - Suman Anna <[email protected]>
+
+description: |
+  The TI K3 family of SoCs usually have one or more dual-core Arm Cortex R5F
+  processor subsystems/clusters (R5FSS). The dual core cluster can be used
+  either in a LockStep mode providing safety/fault tolerance features or in a
+  Split mode providing two individual compute cores for doubling the compute
+  capacity. These are used together with other processors present on the SoC
+  to achieve various system level goals.
+
+  Each Dual-Core R5F sub-system is represented as a single DTS node
+  representing the cluster, with a pair of child DT nodes representing
+  the individual R5F cores. Each node has a number of required or optional
+  properties that enable the OS running on the host processor to perform
+  the device management of the remote processor and to communicate with the
+  remote processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 45

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+    enum:
+      - ti,am654-r5fss
+      - ti,j721e-r5fss
+
+  power-domains:
+    description: |
+      Should contain a phandle to a PM domain provider node and an args
+      specifier containing the R5FSS device id value. This property is
+      as per the binding,
+      Documentation/devicetree/bindings/soc/ti/sci-pm-domain.txt

What implementation of power domains is used is outside the scope of this binding. I'd just drop the whole description as it is pretty generic.

Quoted 7 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +7 -0
+    maxItems: 1
+
+  "#address-cells":
+    const: 1
+
+  "#size-cells":
+    const: 1

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 62

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+
+  ranges:
+    description: |
+      Standard ranges definition providing address translations for
+      local R5F TCM address spaces to bus addresses.
+
+# Optional properties:
+# --------------------
+
+  lockstep-mode:

Needs a vendor prefix.

Quoted 26 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +26 -0
+    $ref: /schemas/types.yaml#/definitions/uint32
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F
+      cluster. Should be either a value of 1 (LockStep mode) or
+      0 (Split mode), default is LockStep mode if omitted.
+
+# R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f@[a-f0-9]+$":
+    type: object
+    description: |
+        The R5F Sub-System device node should define two R5F child nodes, each
+        node representing a TI instantiation of the Arm Cortex R5F core. There
+        are some specific integration differences for the IP like the usage of
+        a Region Address Translator (RAT) for translating the larger SoC bus
+        addresses into a 32-bit address space for the processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+    properties:
+      compatible:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 98

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+        enum:
+          - ti,am654-r5f
+          - ti,j721e-r5f
+
+      reg:
+        description: |
+          Should contain an entry for each value in 'reg-names'.
+          Each entry should have the memory region's start address
+          and the size of the region, the representation matching
+          the parent node's '#address-cells' and '#size-cells' values.

That's every 'reg' property.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 99

Quoted 1 line
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +1 -0
+        maxItems: 2

You need to define what each one is:

items:
- description: ...
- description: ...

Quoted 16 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +16 -0
+
+      reg-names:
+        description: |
+          Should contain strings with the names of the specific internal
+          internal memory regions, and should be defined in this order
+        maxItems: 2
+        items:
+          - const: atcm
+          - const: btcm
+
+      ti,sci:
+        $ref: /schemas/types.yaml#/definitions/phandle
+        description:
+          Should be a phandle to the TI-SCI System Controller node
+
+      ti,sci-dev-id:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 125

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        description: |
+          Should contain the TI-SCI device id corresponding to the R5F core.
+          Please refer to the corresponding System Controller documentation
+          for valid values for the R5F cores.
+
+      ti,sci-proc-ids:
+        description: Should contain a single tuple of <proc_id host_id>.
+        allOf:
+          - $ref: /schemas/types.yaml#/definitions/uint32-matrix

Sounds more like an array.

Quoted 5 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +5 -0
+          - maxItems: 1
+            items:
+              items:
+                - description: TI-SCI processor id for the R5F core device
+                - description: TI-SCI host id to which processor control

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 140

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+                               ownership should be transferred to
+
+      resets:
+        description: |
+          Should contain the phandle to the reset controller node
+          managing the resets for this device, and a reset
+          specifier. Please refer to the following reset bindings
+          for the reset argument specifier,
+          Documentation/devicetree/bindings/reset/ti,sci-reset.txt
+            for AM65x and J721E SoCs

Drop. How many resets (maxItems or items list)?

Quoted 7 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +7 -0
+
+      firmware-name:
+        description: |
+          Should contain the name of the default firmware image
+          file located on the firmware search path
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 157

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+# modes, and are mandatory for R5F Core1 _only_ in Split mode. They are unused for
+# R5F Core1 in LockStep mode:
+
+      mboxes:
+        description: |
+          OMAP Mailbox specifier denoting the sub-mailbox, to be used for
+          communication with the remote processor. This property should match
+          with the sub-mailbox node used in the firmware image. The specifier
+          format is as per the bindings,
+          Documentation/devicetree/bindings/mailbox/omap-mailbox.txt

How many?

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 163

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+
+      memory-region:
+        minItems: 2
+        description: |
+          phandle to the reserved memory nodes to be associated with the remoteproc
+          device. There should be atleast two reserved memory nodes defined - the

What's the max number? As is, it will be 2.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 174

Quoted 11 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +11 -0
+          first one would be used for dynamic DMA allocations like vrings and vring
+          buffers, and the remaining ones used for the firmware image sections. The
+          reserved memory nodes should be carveout nodes, and should be defined as
+          per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+# Optional properties:
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      atcm-enable:

Vendor prefix needed.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 182

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if ATCM should be enabled. R5F
+          view of ATCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is disabled if omitted.
+
+      btcm-enable:

ditto

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 190

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if BTCM should be enabled. R5F
+          view of BTCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is enabled if omitted.
+
+      loczrama:

ditto

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 196

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating which TCM should appear at
+          address 0 (from core's view). Should be either a value of 1 (ATCM
+          at 0x0) or 0 (BTCM at 0x0), default value is 1 if omitted.

I can't decipher how you came up with 'loczrama' based on the description.

Quoted 57 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +57 -0
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        description: |
+          pHandles to one or more reserved on-chip SRAM region. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in,
+          Documentation/devicetree/bindings/sram/sram.yaml
+
+    required:
+     - compatible
+     - reg
+     - reg-names
+     - ti,sci
+     - ti,sci-dev-id
+     - ti,sci-proc-ids
+     - resets
+     - firmware-name
+
+    additionalProperties: false
+
+required:
+ - compatible
+ - power-domains
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+
+    //Example: AM654 SoC
+    /* R5F DDR Carveout reserved memory nodes */
+    reserved-memory {
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges;
+
+        mcu_r5fss0_core1_dma_memory_region: r5f-dma-memory@9b000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core1_memory_region: r5f-memory@9b100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b100000 0x00 0xf00000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_dma_memory_region: r5f-dma-memory@9c000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c000000 0x00 0x100000>;
+            no-map;

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 263

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+        };
+
+        mcu_r5fss0_core0_memory_region: r5f-memory@9c100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c100000 0x00 0x700000>;
+            no-map;
+        };
+    };
+
+    cbass_main: interconnect@100000 {

bus@...

Doesn't look like the right address either.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 271

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        compatible = "simple-bus";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>,
+                 <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>,
+                 <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>;
+
+        cbass_mcu: interconnect@28380000 {

Doesn't look like the right address.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 280

Quoted 9 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +9 -0
+            compatible = "simple-bus";
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>, /* MCU R5F Core0 */
+                     <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>, /* MCU R5F Core1 */
+                     <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>; /* MCU SRAM */
+
+            /* MCU domain SRAM node */
+            mcu_ram: mcu-ram@41c00000 {

I would omit this node from the example. Nothing special here really.

Quoted 61 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +58 -1
+                compatible = "mmio-sram";
+                reg = <0x00 0x41c00000 0x00 0x80000>;
+                ranges = <0x0 0x00 0x41c00000 0x80000>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+
+                mcu_r5fss0_core0_sram: r5f-sram@0 {
+                    reg = <0x0 0x40000>;
+                };
+            };
+
+            /* AM65x MCU R5FSS node */
+            mcu_r5fss0: r5fss@41000000 {
+                compatible = "ti,am654-r5fss";
+                power-domains = <&k3_pds 129>;
+                lockstep-mode = <1>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+                ranges = <0x41000000 0x00 0x41000000 0x20000>,
+                         <0x41400000 0x00 0x41400000 0x20000>;
+
+                mcu_r5f0: r5f@41000000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41000000 0x00008000>,
+                          <0x41010000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <159>;
+                    ti,sci-proc-ids = <0x01 0xFF>;
+                    resets = <&k3_reset 159 1>;
+                    firmware-name = "am65x-mcu-r5f0_0-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox0 &mbox_mcu_r5fss0_core0>;
+                    memory-region = <&mcu_r5fss0_core0_dma_memory_region>,
+                                    <&mcu_r5fss0_core0_memory_region>;
+                    sram = <&mcu_r5fss0_core0_sram>;
+                };
+
+                mcu_r5f1: r5f@41400000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41400000 0x00008000>,
+                          <0x41410000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <245>;
+                    ti,sci-proc-ids = <0x02 0xFF>;
+                    resets = <&k3_reset 245 1>;
+                    firmware-name = "am65x-mcu-r5f0_1-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox1 &mbox_mcu_r5fss0_core1>;
+               };
+           };
+        };
+    };
--

2.23.0

author
#14 lore raw

Re: [PATCH 3/7] dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

Hi Rob,

Quoted 14 lines · Rob Herring
Quoted 12 lines · Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

Quoted 3 lines

I'm inclined to say the system DT stuff should be sorted out before accepting this. Is the system DT stuff going to be useful for your R5 cores? Do you really want to be stuck with this binding?

Hmm, I am not dependent on System DT and prefer to be not gated by that. This is still all from the Linux host perspective, and we don't have any plans to use DT on the firmware-side.

Quoted 8 lines
Quoted 8 lines

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

Quoted 6 lines
Quoted 3 lines

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2.

Then change the default by specifying what you want. Or change the example to be 1 cell. It is *just* an example.

OK, was using the actual dts nodes as how they would be added in our dts files. The only way to get rid of the warnings is to use 1 cell. I can do that for the R5F bindings, but cannot really do that for the DSPs since the addresses need 2 cells.

Quoted 4 lines
Quoted 2 lines

No issues are found with dtbs_check.

I doubt that if your dts matches the example.

The top-level cells value is 2 in our dts files (See either of arm64/dts/ti/k3-am65.dtsi or k3-j721e.dtsi).

Quoted 56 lines
Quoted 56 lines

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +41 -0
new file mode 100644
index 000000000000..bbfc1e6ae884
@@ -0,0 +1,338 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/ti,k3-r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI K3 R5F processor subsystems
+
+maintainers:
+  - Suman Anna <[email protected]>
+
+description: |
+  The TI K3 family of SoCs usually have one or more dual-core Arm Cortex R5F
+  processor subsystems/clusters (R5FSS). The dual core cluster can be used
+  either in a LockStep mode providing safety/fault tolerance features or in a
+  Split mode providing two individual compute cores for doubling the compute
+  capacity. These are used together with other processors present on the SoC
+  to achieve various system level goals.
+
+  Each Dual-Core R5F sub-system is represented as a single DTS node
+  representing the cluster, with a pair of child DT nodes representing
+  the individual R5F cores. Each node has a number of required or optional
+  properties that enable the OS running on the host processor to perform
+  the device management of the remote processor and to communicate with the
+  remote processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:
+    enum:
+      - ti,am654-r5fss
+      - ti,j721e-r5fss
+
+  power-domains:
+    description: |
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 45

Quoted 4 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +4 -0
+      Should contain a phandle to a PM domain provider node and an args
+      specifier containing the R5FSS device id value. This property is
+      as per the binding,
+      Documentation/devicetree/bindings/soc/ti/sci-pm-domain.txt

What implementation of power domains is used is outside the scope of this binding. I'd just drop the whole description as it is pretty generic.

OK.

Quoted 12 lines
Quoted 12 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +11 -0
+    maxItems: 1
+
+  "#address-cells":
+    const: 1
+
+  "#size-cells":
+    const: 1
+
+  ranges:
+    description: |
+      Standard ranges definition providing address translations for
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 62

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+      local R5F TCM address spaces to bus addresses.
+
+# Optional properties:
+# --------------------
+
+  lockstep-mode:

Needs a vendor prefix.

Yep, will fix this one and all the others below.

Quoted 40 lines
Quoted 27 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +26 -0
+    $ref: /schemas/types.yaml#/definitions/uint32
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F
+      cluster. Should be either a value of 1 (LockStep mode) or
+      0 (Split mode), default is LockStep mode if omitted.
+
+# R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f@[a-f0-9]+$":
+    type: object
+    description: |
+        The R5F Sub-System device node should define two R5F child nodes, each
+        node representing a TI instantiation of the Arm Cortex R5F core. There
+        are some specific integration differences for the IP like the usage of
+        a Region Address Translator (RAT) for translating the larger SoC bus
+        addresses into a 32-bit address space for the processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+    properties:
+      compatible:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 98

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+        enum:
+          - ti,am654-r5f
+          - ti,j721e-r5f
+
+      reg:
+        description: |
+          Should contain an entry for each value in 'reg-names'.
+          Each entry should have the memory region's start address
+          and the size of the region, the representation matching
+          the parent node's '#address-cells' and '#size-cells' values.

That's every 'reg' property.

Quoted 7 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 99

Quoted 1 line
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +1 -0
+        maxItems: 2

You need to define what each one is:

items:
- description: ...
- description: ...

OK, will fix.

Quoted 21 lines
Quoted 21 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +20 -0
+
+      reg-names:
+        description: |
+          Should contain strings with the names of the specific internal
+          internal memory regions, and should be defined in this order
+        maxItems: 2
+        items:
+          - const: atcm
+          - const: btcm
+
+      ti,sci:
+        $ref: /schemas/types.yaml#/definitions/phandle
+        description:
+          Should be a phandle to the TI-SCI System Controller node
+
+      ti,sci-dev-id:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        description: |
+          Should contain the TI-SCI device id corresponding to the R5F core.
+          Please refer to the corresponding System Controller documentation
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 125

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+          for valid values for the R5F cores.
+
+      ti,sci-proc-ids:
+        description: Should contain a single tuple of <proc_id host_id>.
+        allOf:
+          - $ref: /schemas/types.yaml#/definitions/uint32-matrix

Sounds more like an array.

OK, I can modify this. Went with this originally to reflect the tuple, but guess both translate similarly for my usage.

Quoted 17 lines
Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +5 -0
+          - maxItems: 1
+            items:
+              items:
+                - description: TI-SCI processor id for the R5F core device
+                - description: TI-SCI host id to which processor control

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 140

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+                               ownership should be transferred to
+
+      resets:
+        description: |
+          Should contain the phandle to the reset controller node
+          managing the resets for this device, and a reset
+          specifier. Please refer to the following reset bindings
+          for the reset argument specifier,
+          Documentation/devicetree/bindings/reset/ti,sci-reset.txt
+            for AM65x and J721E SoCs
Quoted 1 line

Drop. How many resets (maxItems or items list)?

Yeah, this is 1, will update. Do you want me to drop just the specifier link or the entire description?

Quoted 12 lines
Quoted 12 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +11 -0
+
+      firmware-name:
+        description: |
+          Should contain the name of the default firmware image
+          file located on the firmware search path
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split
+# modes, and are mandatory for R5F Core1 _only_ in Split mode. They are unused for
+# R5F Core1 in LockStep mode:
+
+      mboxes:
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 157

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+        description: |
+          OMAP Mailbox specifier denoting the sub-mailbox, to be used for
+          communication with the remote processor. This property should match
+          with the sub-mailbox node used in the firmware image. The specifier
+          format is as per the bindings,
+          Documentation/devicetree/bindings/mailbox/omap-mailbox.txt

How many?

OK, will fix.

Quoted 9 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 163

Quoted 7 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+
+      memory-region:
+        minItems: 2
+        description: |
+          phandle to the reserved memory nodes to be associated with the remoteproc
+          device. There should be atleast two reserved memory nodes defined - the

What's the max number? As is, it will be 2.

Aah, I misinterpreted that not having would be open-ended. OK, I will have to give an arbitrary number here (maybe 4 or 8). Can we update this later on if a usecase really needs more?

Quoted 37 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 174

Quoted 12 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +11 -0
+          first one would be used for dynamic DMA allocations like vrings and vring
+          buffers, and the remaining ones used for the firmware image sections. The
+          reserved memory nodes should be carveout nodes, and should be defined as
+          per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+# Optional properties:
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      atcm-enable:

Vendor prefix needed.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 182

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if ATCM should be enabled. R5F
+          view of ATCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is disabled if omitted.
+
+      btcm-enable:

ditto

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 190

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if BTCM should be enabled. R5F
+          view of BTCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is enabled if omitted.
+
+      loczrama:

ditto

Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 196

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating which TCM should appear at
+          address 0 (from core's view). Should be either a value of 1 (ATCM
+          at 0x0) or 0 (BTCM at 0x0), default value is 1 if omitted.

I can't decipher how you came up with 'loczrama' based on the description.

That's actually the signal name from the Arm R5 specs.

Quoted 62 lines
Quoted 62 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +61 -0
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        description: |
+          pHandles to one or more reserved on-chip SRAM region. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in,
+          Documentation/devicetree/bindings/sram/sram.yaml
+
+    required:
+     - compatible
+     - reg
+     - reg-names
+     - ti,sci
+     - ti,sci-dev-id
+     - ti,sci-proc-ids
+     - resets
+     - firmware-name
+
+    additionalProperties: false
+
+required:
+ - compatible
+ - power-domains
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+
+    //Example: AM654 SoC
+    /* R5F DDR Carveout reserved memory nodes */
+    reserved-memory {
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges;
+
+        mcu_r5fss0_core1_dma_memory_region: r5f-dma-memory@9b000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core1_memory_region: r5f-memory@9b100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b100000 0x00 0xf00000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_dma_memory_region: r5f-dma-memory@9c000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_memory_region: r5f-memory@9c100000 {
+            compatible = "shared-dma-pool";
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 263

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+            reg = <0x00 0x9c100000 0x00 0x700000>;
+            no-map;
+        };
+    };
+
+    cbass_main: interconnect@100000 {

bus@...

Yeah, will update the example to use bus. The DTS nodes in the kernel are already using the interconnect name.

Quoted 2 lines

Doesn't look like the right address either.

Yeah, I skipped the actual first entry from the ranges, and only mentioned the ones that I am using in the nodes. Will fix this.

Quoted 11 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 271

Quoted 9 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        compatible = "simple-bus";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>,
+                 <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>,
+                 <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>;
+
+        cbass_mcu: interconnect@28380000 {

Doesn't look like the right address.

Same as above.

Quoted 11 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 280

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +9 -0
+            compatible = "simple-bus";
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>, /* MCU R5F Core0 */
+                     <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>, /* MCU R5F Core1 */
+                     <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>; /* MCU SRAM */
+
+            /* MCU domain SRAM node */
+            mcu_ram: mcu-ram@41c00000 {
Quoted 1 line

I would omit this node from the example. Nothing special here really.

Showcasing the optional sram property usage from the mcu_r5f0 node.

regards
Suman

Quoted 62 lines
Quoted 62 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +58 -1
+                compatible = "mmio-sram";
+                reg = <0x00 0x41c00000 0x00 0x80000>;
+                ranges = <0x0 0x00 0x41c00000 0x80000>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+
+                mcu_r5fss0_core0_sram: r5f-sram@0 {
+                    reg = <0x0 0x40000>;
+                };
+            };
+
+            /* AM65x MCU R5FSS node */
+            mcu_r5fss0: r5fss@41000000 {
+                compatible = "ti,am654-r5fss";
+                power-domains = <&k3_pds 129>;
+                lockstep-mode = <1>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+                ranges = <0x41000000 0x00 0x41000000 0x20000>,
+                         <0x41400000 0x00 0x41400000 0x20000>;
+
+                mcu_r5f0: r5f@41000000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41000000 0x00008000>,
+                          <0x41010000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <159>;
+                    ti,sci-proc-ids = <0x01 0xFF>;
+                    resets = <&k3_reset 159 1>;
+                    firmware-name = "am65x-mcu-r5f0_0-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox0 &mbox_mcu_r5fss0_core0>;
+                    memory-region = <&mcu_r5fss0_core0_dma_memory_region>,
+                                    <&mcu_r5fss0_core0_memory_region>;
+                    sram = <&mcu_r5fss0_core0_sram>;
+                };
+
+                mcu_r5f1: r5f@41400000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41400000 0x00008000>,
+                          <0x41410000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <245>;
+                    ti,sci-proc-ids = <0x02 0xFF>;
+                    resets = <&k3_reset 245 1>;
+                    firmware-name = "am65x-mcu-r5f0_1-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox1 &mbox_mcu_r5fss0_core1>;
+               };
+           };
+        };
+    };
--

2.23.0

author
#15 lore raw

Re: [PATCH 3/7] dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

Quoted 249 lines · Suman Anna

Hi Rob,

Quoted 14 lines · Rob Herring
Quoted 12 lines · Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

Quoted 3 lines

I'm inclined to say the system DT stuff should be sorted out before accepting this. Is the system DT stuff going to be useful for your R5 cores? Do you really want to be stuck with this binding?

Hmm, I am not dependent on System DT and prefer to be not gated by that. This is still all from the Linux host perspective, and we don't have any plans to use DT on the firmware-side.

Quoted 8 lines
Quoted 8 lines

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

Quoted 6 lines
Quoted 3 lines

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2.

Then change the default by specifying what you want. Or change the example to be 1 cell. It is *just* an example.

OK, was using the actual dts nodes as how they would be added in our dts files. The only way to get rid of the warnings is to use 1 cell. I can do that for the R5F bindings, but cannot really do that for the DSPs since the addresses need 2 cells.

Quoted 4 lines
Quoted 2 lines

No issues are found with dtbs_check.

I doubt that if your dts matches the example.

The top-level cells value is 2 in our dts files (See either of arm64/dts/ti/k3-am65.dtsi or k3-j721e.dtsi).

Quoted 56 lines
Quoted 56 lines

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +41 -0
new file mode 100644
index 000000000000..bbfc1e6ae884
@@ -0,0 +1,338 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/ti,k3-r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI K3 R5F processor subsystems
+
+maintainers:
+  - Suman Anna <[email protected]>
+
+description: |
+  The TI K3 family of SoCs usually have one or more dual-core Arm Cortex R5F
+  processor subsystems/clusters (R5FSS). The dual core cluster can be used
+  either in a LockStep mode providing safety/fault tolerance features or in a
+  Split mode providing two individual compute cores for doubling the compute
+  capacity. These are used together with other processors present on the SoC
+  to achieve various system level goals.
+
+  Each Dual-Core R5F sub-system is represented as a single DTS node
+  representing the cluster, with a pair of child DT nodes representing
+  the individual R5F cores. Each node has a number of required or optional
+  properties that enable the OS running on the host processor to perform
+  the device management of the remote processor and to communicate with the
+  remote processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:
+    enum:
+      - ti,am654-r5fss
+      - ti,j721e-r5fss
+
+  power-domains:
+    description: |
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 45

Quoted 4 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +4 -0
+      Should contain a phandle to a PM domain provider node and an args
+      specifier containing the R5FSS device id value. This property is
+      as per the binding,
+      Documentation/devicetree/bindings/soc/ti/sci-pm-domain.txt

What implementation of power domains is used is outside the scope of this binding. I'd just drop the whole description as it is pretty generic.

OK.

Quoted 12 lines
Quoted 12 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +11 -0
+    maxItems: 1
+
+  "#address-cells":
+    const: 1
+
+  "#size-cells":
+    const: 1
+
+  ranges:
+    description: |
+      Standard ranges definition providing address translations for
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 62

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+      local R5F TCM address spaces to bus addresses.
+
+# Optional properties:
+# --------------------
+
+  lockstep-mode:

Needs a vendor prefix.

Yep, will fix this one and all the others below.

Quoted 40 lines
Quoted 27 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +26 -0
+    $ref: /schemas/types.yaml#/definitions/uint32
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F
+      cluster. Should be either a value of 1 (LockStep mode) or
+      0 (Split mode), default is LockStep mode if omitted.
+
+# R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f@[a-f0-9]+$":
+    type: object
+    description: |
+        The R5F Sub-System device node should define two R5F child nodes, each
+        node representing a TI instantiation of the Arm Cortex R5F core. There
+        are some specific integration differences for the IP like the usage of
+        a Region Address Translator (RAT) for translating the larger SoC bus
+        addresses into a 32-bit address space for the processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+    properties:
+      compatible:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 98

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+        enum:
+          - ti,am654-r5f
+          - ti,j721e-r5f
+
+      reg:
+        description: |
+          Should contain an entry for each value in 'reg-names'.
+          Each entry should have the memory region's start address
+          and the size of the region, the representation matching
+          the parent node's '#address-cells' and '#size-cells' values.

That's every 'reg' property.

Quoted 7 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 99

Quoted 1 line
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +1 -0
+        maxItems: 2

You need to define what each one is:

items:
- description: ...
- description: ...

OK, will fix.

Quoted 21 lines
Quoted 21 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +20 -0
+
+      reg-names:
+        description: |
+          Should contain strings with the names of the specific internal
+          internal memory regions, and should be defined in this order
+        maxItems: 2
+        items:
+          - const: atcm
+          - const: btcm
+
+      ti,sci:
+        $ref: /schemas/types.yaml#/definitions/phandle
+        description:
+          Should be a phandle to the TI-SCI System Controller node
+
+      ti,sci-dev-id:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        description: |
+          Should contain the TI-SCI device id corresponding to the R5F core.
+          Please refer to the corresponding System Controller documentation
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 125

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+          for valid values for the R5F cores.
+
+      ti,sci-proc-ids:
+        description: Should contain a single tuple of <proc_id host_id>.
+        allOf:
+          - $ref: /schemas/types.yaml#/definitions/uint32-matrix

Sounds more like an array.

OK, I can modify this. Went with this originally to reflect the tuple, but guess both translate similarly for my usage.

Quoted 17 lines
Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +5 -0
+          - maxItems: 1
+            items:
+              items:
+                - description: TI-SCI processor id for the R5F core device
+                - description: TI-SCI host id to which processor control

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 140

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+                               ownership should be transferred to
+
+      resets:
+        description: |
+          Should contain the phandle to the reset controller node
+          managing the resets for this device, and a reset
+          specifier. Please refer to the following reset bindings
+          for the reset argument specifier,
+          Documentation/devicetree/bindings/reset/ti,sci-reset.txt
+            for AM65x and J721E SoCs
Quoted 1 line

Drop. How many resets (maxItems or items list)?

Yeah, this is 1, will update. Do you want me to drop just the specifier link or the entire description?

Quoted 1 line
Quoted 9 lines
Quoted 9 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 149

Quoted 9 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +9 -0
+
+      firmware-name:
+        description: |
+          Should contain the name of the default firmware image
+          file located on the firmware search path
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split
+# modes, and are mandatory for R5F Core1 _only_ in Split mode. They are unused for
+# R5F Core1 in LockStep mode:

Rob,

If I were to actually encode this in YAML, I need to distinguish Core0 from Core1. I have currently followed the DT generic node name convention when defining the two nodes and interpret in the driver based on addresses. What is the preferred mechanism for this - define node names as r5f0 and r5f1 or introduce a ti,core-id or ti,cpu-id property?

regards
Suman

Quoted 250 lines
Quoted 10 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 157

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+
+      mboxes:
+        description: |
+          OMAP Mailbox specifier denoting the sub-mailbox, to be used for
+          communication with the remote processor. This property should match
+          with the sub-mailbox node used in the firmware image. The specifier
+          format is as per the bindings,
+          Documentation/devicetree/bindings/mailbox/omap-mailbox.txt

How many?

OK, will fix.

Quoted 9 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 163

Quoted 7 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+
+      memory-region:
+        minItems: 2
+        description: |
+          phandle to the reserved memory nodes to be associated with the remoteproc
+          device. There should be atleast two reserved memory nodes defined - the

What's the max number? As is, it will be 2.

Aah, I misinterpreted that not having would be open-ended. OK, I will have to give an arbitrary number here (maybe 4 or 8). Can we update this later on if a usecase really needs more?

Quoted 37 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 174

Quoted 12 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +11 -0
+          first one would be used for dynamic DMA allocations like vrings and vring
+          buffers, and the remaining ones used for the firmware image sections. The
+          reserved memory nodes should be carveout nodes, and should be defined as
+          per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+# Optional properties:
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      atcm-enable:

Vendor prefix needed.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 182

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if ATCM should be enabled. R5F
+          view of ATCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is disabled if omitted.
+
+      btcm-enable:

ditto

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 190

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if BTCM should be enabled. R5F
+          view of BTCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is enabled if omitted.
+
+      loczrama:

ditto

Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 196

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating which TCM should appear at
+          address 0 (from core's view). Should be either a value of 1 (ATCM
+          at 0x0) or 0 (BTCM at 0x0), default value is 1 if omitted.

I can't decipher how you came up with 'loczrama' based on the description.

That's actually the signal name from the Arm R5 specs.

Quoted 62 lines
Quoted 62 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +61 -0
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        description: |
+          pHandles to one or more reserved on-chip SRAM region. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in,
+          Documentation/devicetree/bindings/sram/sram.yaml
+
+    required:
+     - compatible
+     - reg
+     - reg-names
+     - ti,sci
+     - ti,sci-dev-id
+     - ti,sci-proc-ids
+     - resets
+     - firmware-name
+
+    additionalProperties: false
+
+required:
+ - compatible
+ - power-domains
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+
+    //Example: AM654 SoC
+    /* R5F DDR Carveout reserved memory nodes */
+    reserved-memory {
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges;
+
+        mcu_r5fss0_core1_dma_memory_region: r5f-dma-memory@9b000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core1_memory_region: r5f-memory@9b100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b100000 0x00 0xf00000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_dma_memory_region: r5f-dma-memory@9c000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_memory_region: r5f-memory@9c100000 {
+            compatible = "shared-dma-pool";
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 263

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+            reg = <0x00 0x9c100000 0x00 0x700000>;
+            no-map;
+        };
+    };
+
+    cbass_main: interconnect@100000 {

bus@...

Yeah, will update the example to use bus. The DTS nodes in the kernel are already using the interconnect name.

Quoted 2 lines

Doesn't look like the right address either.

Yeah, I skipped the actual first entry from the ranges, and only mentioned the ones that I am using in the nodes. Will fix this.

Quoted 11 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 271

Quoted 9 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+        compatible = "simple-bus";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>,
+                 <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>,
+                 <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>;
+
+        cbass_mcu: interconnect@28380000 {

Doesn't look like the right address.

Same as above.

Quoted 11 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 280

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +9 -0
+            compatible = "simple-bus";
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>, /* MCU R5F Core0 */
+                     <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>, /* MCU R5F Core1 */
+                     <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>; /* MCU SRAM */
+
+            /* MCU domain SRAM node */
+            mcu_ram: mcu-ram@41c00000 {
Quoted 1 line

I would omit this node from the example. Nothing special here really.

Showcasing the optional sram property usage from the mcu_r5f0 node.

regards
Suman

Quoted 62 lines
Quoted 62 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +58 -1
+                compatible = "mmio-sram";
+                reg = <0x00 0x41c00000 0x00 0x80000>;
+                ranges = <0x0 0x00 0x41c00000 0x80000>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+
+                mcu_r5fss0_core0_sram: r5f-sram@0 {
+                    reg = <0x0 0x40000>;
+                };
+            };
+
+            /* AM65x MCU R5FSS node */
+            mcu_r5fss0: r5fss@41000000 {
+                compatible = "ti,am654-r5fss";
+                power-domains = <&k3_pds 129>;
+                lockstep-mode = <1>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+                ranges = <0x41000000 0x00 0x41000000 0x20000>,
+                         <0x41400000 0x00 0x41400000 0x20000>;
+
+                mcu_r5f0: r5f@41000000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41000000 0x00008000>,
+                          <0x41010000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <159>;
+                    ti,sci-proc-ids = <0x01 0xFF>;
+                    resets = <&k3_reset 159 1>;
+                    firmware-name = "am65x-mcu-r5f0_0-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox0 &mbox_mcu_r5fss0_core0>;
+                    memory-region = <&mcu_r5fss0_core0_dma_memory_region>,
+                                    <&mcu_r5fss0_core0_memory_region>;
+                    sram = <&mcu_r5fss0_core0_sram>;
+                };
+
+                mcu_r5f1: r5f@41400000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41400000 0x00008000>,
+                          <0x41410000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <245>;
+                    ti,sci-proc-ids = <0x02 0xFF>;
+                    resets = <&k3_reset 245 1>;
+                    firmware-name = "am65x-mcu-r5f0_1-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox1 &mbox_mcu_r5fss0_core1>;
+               };
+           };
+        };
+    };
--

2.23.0

maintainer
#16 lore raw

Re: [PATCH 3/7] dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

In reply to Suman Anna
Quoted 95 lines · Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2. No issues are found with dtbs_check.

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +58 -0
new file mode 100644
index 000000000000..bbfc1e6ae884
@@ -0,0 +1,338 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/ti,k3-r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI K3 R5F processor subsystems
+
+maintainers:
+  - Suman Anna <[email protected]>
+
+description: |
+  The TI K3 family of SoCs usually have one or more dual-core Arm Cortex R5F
+  processor subsystems/clusters (R5FSS). The dual core cluster can be used
+  either in a LockStep mode providing safety/fault tolerance features or in a
+  Split mode providing two individual compute cores for doubling the compute
+  capacity. These are used together with other processors present on the SoC
+  to achieve various system level goals.
+
+  Each Dual-Core R5F sub-system is represented as a single DTS node
+  representing the cluster, with a pair of child DT nodes representing
+  the individual R5F cores. Each node has a number of required or optional
+  properties that enable the OS running on the host processor to perform
+  the device management of the remote processor and to communicate with the
+  remote processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:
+    enum:
+      - ti,am654-r5fss
+      - ti,j721e-r5fss
+
+  power-domains:
+    description: |
+      Should contain a phandle to a PM domain provider node and an args
+      specifier containing the R5FSS device id value. This property is
+      as per the binding,
+      Documentation/devicetree/bindings/soc/ti/sci-pm-domain.txt
+    maxItems: 1
+
+  "#address-cells":
+    const: 1
+
+  "#size-cells":
+    const: 1
+
+  ranges:
+    description: |
+      Standard ranges definition providing address translations for
+      local R5F TCM address spaces to bus addresses.
+

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 68

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+# Optional properties:
+# --------------------
+
+  lockstep-mode:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F
+      cluster. Should be either a value of 1 (LockStep mode) or
+      0 (Split mode), default is LockStep mode if omitted.

Should I deduce lockstep means SMP and split mode two processors running in UP mode?

Quoted 26 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +26 -0
+
+# R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f@[a-f0-9]+$":
+    type: object
+    description: |
+        The R5F Sub-System device node should define two R5F child nodes, each
+        node representing a TI instantiation of the Arm Cortex R5F core. There
+        are some specific integration differences for the IP like the usage of
+        a Region Address Translator (RAT) for translating the larger SoC bus
+        addresses into a 32-bit address space for the processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+    properties:
+      compatible:
+        enum:
+          - ti,am654-r5f
+          - ti,j721e-r5f
+
+      reg:
+        description: |

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 104

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+          Should contain an entry for each value in 'reg-names'.
+          Each entry should have the memory region's start address
+          and the size of the region, the representation matching
+          the parent node's '#address-cells' and '#size-cells' values.
+        maxItems: 2
+
+      reg-names:
+        description: |
+          Should contain strings with the names of the specific internal
+          internal memory regions, and should be defined in this order

There is two "internal" in a row.

Quoted 43 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +43 -0
+        maxItems: 2
+        items:
+          - const: atcm
+          - const: btcm
+
+      ti,sci:
+        $ref: /schemas/types.yaml#/definitions/phandle
+        description:
+          Should be a phandle to the TI-SCI System Controller node
+
+      ti,sci-dev-id:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        description: |
+          Should contain the TI-SCI device id corresponding to the R5F core.
+          Please refer to the corresponding System Controller documentation
+          for valid values for the R5F cores.
+
+      ti,sci-proc-ids:
+        description: Should contain a single tuple of <proc_id host_id>.
+        allOf:
+          - $ref: /schemas/types.yaml#/definitions/uint32-matrix
+          - maxItems: 1
+            items:
+              items:
+                - description: TI-SCI processor id for the R5F core device
+                - description: TI-SCI host id to which processor control
+                               ownership should be transferred to
+
+      resets:
+        description: |
+          Should contain the phandle to the reset controller node
+          managing the resets for this device, and a reset
+          specifier. Please refer to the following reset bindings
+          for the reset argument specifier,
+          Documentation/devicetree/bindings/reset/ti,sci-reset.txt
+            for AM65x and J721E SoCs
+
+      firmware-name:
+        description: |
+          Should contain the name of the default firmware image
+          file located on the firmware search path
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 157

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+# modes, and are mandatory for R5F Core1 _only_ in Split mode. They are unused for
+# R5F Core1 in LockStep mode:
+
+      mboxes:
+        description: |
+          OMAP Mailbox specifier denoting the sub-mailbox, to be used for
+          communication with the remote processor. This property should match
+          with the sub-mailbox node used in the firmware image. The specifier
+          format is as per the bindings,
+          Documentation/devicetree/bindings/mailbox/omap-mailbox.txt

In the example below node "mcu_r5f1" has a "mboxes" property despite being in lockstep mode. I haven't delved in the code yet, perharps I'll find an answer in there.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 163

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+
+      memory-region:
+        minItems: 2
+        description: |
+          phandle to the reserved memory nodes to be associated with the remoteproc
+          device. There should be atleast two reserved memory nodes defined - the

s/atleast/at least

Quoted 7 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +7 -0
+          first one would be used for dynamic DMA allocations like vrings and vring
+          buffers, and the remaining ones used for the firmware image sections. The
+          reserved memory nodes should be carveout nodes, and should be defined as
+          per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+# Optional properties:

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 180

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +10 -0
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      atcm-enable:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if ATCM should be enabled. R5F
+          view of ATCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is disabled if omitted.

What is ATCM and why would one want to enable the feature?

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 188

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+
+      btcm-enable:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if BTCM should be enabled. R5F
+          view of BTCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is enabled if omitted.

Same here, there is no way to tell what BTCM is.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 196

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+
+      loczrama:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating which TCM should appear at
+          address 0 (from core's view). Should be either a value of 1 (ATCM
+          at 0x0) or 0 (BTCM at 0x0), default value is 1 if omitted.

loczrama: Location Zero Ram Address? Can both core show up at the address 0x0? In the example below both cores have the same setting for atcm, btcm and loczrama.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 205

Quoted 9 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +9 -0
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        description: |
+          pHandles to one or more reserved on-chip SRAM region. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in,
+          Documentation/devicetree/bindings/sram/sram.yaml

Is the behaviour the same whether operating in locksetup mode and split mode?

Thanks,
Mathieu

Quoted 136 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +133 -0
+
+    required:
+     - compatible
+     - reg
+     - reg-names
+     - ti,sci
+     - ti,sci-dev-id
+     - ti,sci-proc-ids
+     - resets
+     - firmware-name
+
+    additionalProperties: false
+
+required:
+ - compatible
+ - power-domains
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+
+    //Example: AM654 SoC
+    /* R5F DDR Carveout reserved memory nodes */
+    reserved-memory {
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges;
+
+        mcu_r5fss0_core1_dma_memory_region: r5f-dma-memory@9b000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core1_memory_region: r5f-memory@9b100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b100000 0x00 0xf00000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_dma_memory_region: r5f-dma-memory@9c000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_memory_region: r5f-memory@9c100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c100000 0x00 0x700000>;
+            no-map;
+        };
+    };
+
+    cbass_main: interconnect@100000 {
+        compatible = "simple-bus";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>,
+                 <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>,
+                 <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>;
+
+        cbass_mcu: interconnect@28380000 {
+            compatible = "simple-bus";
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>, /* MCU R5F Core0 */
+                     <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>, /* MCU R5F Core1 */
+                     <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>; /* MCU SRAM */
+
+            /* MCU domain SRAM node */
+            mcu_ram: mcu-ram@41c00000 {
+                compatible = "mmio-sram";
+                reg = <0x00 0x41c00000 0x00 0x80000>;
+                ranges = <0x0 0x00 0x41c00000 0x80000>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+
+                mcu_r5fss0_core0_sram: r5f-sram@0 {
+                    reg = <0x0 0x40000>;
+                };
+            };
+
+            /* AM65x MCU R5FSS node */
+            mcu_r5fss0: r5fss@41000000 {
+                compatible = "ti,am654-r5fss";
+                power-domains = <&k3_pds 129>;
+                lockstep-mode = <1>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+                ranges = <0x41000000 0x00 0x41000000 0x20000>,
+                         <0x41400000 0x00 0x41400000 0x20000>;
+
+                mcu_r5f0: r5f@41000000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41000000 0x00008000>,
+                          <0x41010000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <159>;
+                    ti,sci-proc-ids = <0x01 0xFF>;
+                    resets = <&k3_reset 159 1>;
+                    firmware-name = "am65x-mcu-r5f0_0-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox0 &mbox_mcu_r5fss0_core0>;
+                    memory-region = <&mcu_r5fss0_core0_dma_memory_region>,
+                                    <&mcu_r5fss0_core0_memory_region>;
+                    sram = <&mcu_r5fss0_core0_sram>;
+                };
+
+                mcu_r5f1: r5f@41400000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41400000 0x00008000>,
+                          <0x41410000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <245>;
+                    ti,sci-proc-ids = <0x02 0xFF>;
+                    resets = <&k3_reset 245 1>;
+                    firmware-name = "am65x-mcu-r5f0_1-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox1 &mbox_mcu_r5fss0_core1>;
+               };
+           };
+        };
+    };
Signature · 1 line
2.23.0
author
#17 lore raw

Re: [PATCH 3/7] dt-bindings: remoteproc: Add bindings for R5F subsystem on TI K3 SoCs

Hi Mathieu,

Quoted 101 lines · Mathieu Poirier
Quoted 100 lines · Suman Anna

The Texas Instruments K3 family of SoCs have one or more dual-core Arm Cortex R5F processor subsystems/clusters (R5FSS). The clusters can be split between multiple voltage domains as well. Add the device tree bindings document for these R5F subsystem devices. These R5F processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. The nodes require more than one memory region, with the first memory region used for DMA allocations at runtime. The remaining memory regions are reserved and are used for the loading and running of the R5F remote processors. The R5F processors can also optionally use any internal on-chip SRAM memories either for executing code or using it as fast-access data.

The added example illustrates the DT nodes for the single R5FSS device present on K3 AM65x family of SoCs.

---
Hi Rob,

The dt_bindings_check seems to throw couple of warnings around the usage of ranges because the tooling is adding the #address-cells and #size-cells of 1 by default, whereas our actual code uses 2. No issues are found with dtbs_check.

regards
Suman

 .../bindings/remoteproc/ti,k3-r5f-rproc.yaml  | 338 ++++++++++++++++++
 1 file changed, 338 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +63 -0
new file mode 100644
index 000000000000..bbfc1e6ae884
@@ -0,0 +1,338 @@
+# SPDX-License-Identifier: (GPL-2.0-only or BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/ti,k3-r5f-rproc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI K3 R5F processor subsystems
+
+maintainers:
+  - Suman Anna <[email protected]>
+
+description: |
+  The TI K3 family of SoCs usually have one or more dual-core Arm Cortex R5F
+  processor subsystems/clusters (R5FSS). The dual core cluster can be used
+  either in a LockStep mode providing safety/fault tolerance features or in a
+  Split mode providing two individual compute cores for doubling the compute
+  capacity. These are used together with other processors present on the SoC
+  to achieve various system level goals.
+
+  Each Dual-Core R5F sub-system is represented as a single DTS node
+  representing the cluster, with a pair of child DT nodes representing
+  the individual R5F cores. Each node has a number of required or optional
+  properties that enable the OS running on the host processor to perform
+  the device management of the remote processor and to communicate with the
+  remote processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+properties:
+  $nodename:
+    pattern: "^r5fss(@.*)?"
+
+  compatible:
+    enum:
+      - ti,am654-r5fss
+      - ti,j721e-r5fss
+
+  power-domains:
+    description: |
+      Should contain a phandle to a PM domain provider node and an args
+      specifier containing the R5FSS device id value. This property is
+      as per the binding,
+      Documentation/devicetree/bindings/soc/ti/sci-pm-domain.txt
+    maxItems: 1
+
+  "#address-cells":
+    const: 1
+
+  "#size-cells":
+    const: 1
+
+  ranges:
+    description: |
+      Standard ranges definition providing address translations for
+      local R5F TCM address spaces to bus addresses.
+
+# Optional properties:
+# --------------------
+
+  lockstep-mode:
+    $ref: /schemas/types.yaml#/definitions/uint32
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 68

Quoted 5 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +5 -0
+    enum: [0, 1]
+    description: |
+      Configuration Mode for the Dual R5F cores within the R5F
+      cluster. Should be either a value of 1 (LockStep mode) or
+      0 (Split mode), default is LockStep mode if omitted.

Should I deduce lockstep means SMP and split mode two processors running in UP mode?

No, slightly different from SMP. It is a safety/fault-tolerance mode, the 2 cores run the same code, and then the output is compared, and an error is thrown if they yield different results.

Quoted 31 lines
Quoted 31 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +30 -0
+
+# R5F Processor Child Nodes:
+# ==========================
+
+patternProperties:
+  "^r5f@[a-f0-9]+$":
+    type: object
+    description: |
+        The R5F Sub-System device node should define two R5F child nodes, each
+        node representing a TI instantiation of the Arm Cortex R5F core. There
+        are some specific integration differences for the IP like the usage of
+        a Region Address Translator (RAT) for translating the larger SoC bus
+        addresses into a 32-bit address space for the processor.
+
+# Required properties:
+# --------------------
+# The following are the mandatory properties:
+
+    properties:
+      compatible:
+        enum:
+          - ti,am654-r5f
+          - ti,j721e-r5f
+
+      reg:
+        description: |
+          Should contain an entry for each value in 'reg-names'.
+          Each entry should have the memory region's start address
+          and the size of the region, the representation matching
+          the parent node's '#address-cells' and '#size-cells' values.
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 104

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+        maxItems: 2
+
+      reg-names:
+        description: |
+          Should contain strings with the names of the specific internal
+          internal memory regions, and should be defined in this order

There is two "internal" in a row.

Thanks, will fix it up.

Quoted 49 lines
Quoted 49 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +48 -0
+        maxItems: 2
+        items:
+          - const: atcm
+          - const: btcm
+
+      ti,sci:
+        $ref: /schemas/types.yaml#/definitions/phandle
+        description:
+          Should be a phandle to the TI-SCI System Controller node
+
+      ti,sci-dev-id:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        description: |
+          Should contain the TI-SCI device id corresponding to the R5F core.
+          Please refer to the corresponding System Controller documentation
+          for valid values for the R5F cores.
+
+      ti,sci-proc-ids:
+        description: Should contain a single tuple of <proc_id host_id>.
+        allOf:
+          - $ref: /schemas/types.yaml#/definitions/uint32-matrix
+          - maxItems: 1
+            items:
+              items:
+                - description: TI-SCI processor id for the R5F core device
+                - description: TI-SCI host id to which processor control
+                               ownership should be transferred to
+
+      resets:
+        description: |
+          Should contain the phandle to the reset controller node
+          managing the resets for this device, and a reset
+          specifier. Please refer to the following reset bindings
+          for the reset argument specifier,
+          Documentation/devicetree/bindings/reset/ti,sci-reset.txt
+            for AM65x and J721E SoCs
+
+      firmware-name:
+        description: |
+          Should contain the name of the default firmware image
+          file located on the firmware search path
+
+# The following properties are mandatory for R5F Core0 in both LockStep and Split
+# modes, and are mandatory for R5F Core1 _only_ in Split mode. They are unused for
+# R5F Core1 in LockStep mode:
+
+      mboxes:
+        description: |
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 157

Quoted 5 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +5 -0
+          OMAP Mailbox specifier denoting the sub-mailbox, to be used for
+          communication with the remote processor. This property should match
+          with the sub-mailbox node used in the firmware image. The specifier
+          format is as per the bindings,
+          Documentation/devicetree/bindings/mailbox/omap-mailbox.txt

In the example below node "mcu_r5f1" has a "mboxes" property despite being in lockstep mode. I haven't delved in the code yet, perharps I'll find an answer in there.

Yeah, it is defined, but won't be parsed or used if the cluster is in lockstep mode as my comment above says. The lockstep capability is an efuse bit, so on some SoC variants, which do not have the capability, the cluster is hard-wired for Split-mode only. And we detect that in the driver irrespective of the lockstep-mode value.

Quoted 9 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 163

Quoted 7 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+
+      memory-region:
+        minItems: 2
+        description: |
+          phandle to the reserved memory nodes to be associated with the remoteproc
+          device. There should be atleast two reserved memory nodes defined - the

s/atleast/at least

OK

Quoted 12 lines
Quoted 12 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +11 -0
+          first one would be used for dynamic DMA allocations like vrings and vring
+          buffers, and the remaining ones used for the firmware image sections. The
+          reserved memory nodes should be carveout nodes, and should be defined as
+          per the bindings in
+          Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+# Optional properties:
+# --------------------
+# The following properties are optional properties for each of the R5F cores:
+
+      atcm-enable:
Quoted 8 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 180

Quoted 6 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +6 -0
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if ATCM should be enabled. R5F
+          view of ATCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is disabled if omitted.

What is ATCM and why would one want to enable the feature?

TCM stands for Tightly Coupled Memory (an internal RAM memory essentially), and R5 supports two such bank interfaces called A and B. Some of the specs use TCMA or TCMB as well.

Quoted 21 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 188

Quoted 9 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+
+      btcm-enable:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating if BTCM should be enabled. R5F
+          view of BTCM dictated by loczrama property. Should be either a value
+          of 1 (enabled) or 0 (disabled), default is enabled if omitted.

Same here, there is no way to tell what BTCM is.

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 196

Quoted 8 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +8 -0
+
+      loczrama:
+        $ref: /schemas/types.yaml#/definitions/uint32
+        enum: [0, 1]
+        description: |
+          R5F core configuration mode dictating which TCM should appear at
+          address 0 (from core's view). Should be either a value of 1 (ATCM
+          at 0x0) or 0 (BTCM at 0x0), default value is 1 if omitted.
Quoted 1 line

loczrama: Location Zero Ram Address?

Guess so. I never found the expansion.

Can both core show up at the address 0x0?

Quoted 2 lines

In the example below both cores have the same setting for atcm, btcm and loczrama.

Each core has their own ATCM, BTCM and loczrama settings and are effective in Split-mode. Only Core0 memories and settings are used in LockStep mode.

Quoted 11 lines

Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml line 205

Quoted 10 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +9 -0
+
+      sram:
+        $ref: /schemas/types.yaml#/definitions/phandle-array
+        minItems: 1
+        description: |
+          pHandles to one or more reserved on-chip SRAM region. The regions
+          should be defined as child nodes of the respective SRAM node, and
+          should be defined as per the generic bindings in,
+          Documentation/devicetree/bindings/sram/sram.yaml
Quoted 1 line

Is the behaviour the same whether operating in locksetup mode and split mode?

This is per core. So, just like mboxes, will be unused on Core1 if the cluster is configured for LockStep mode.

regards
Suman

Quoted 140 lines

Thanks,
Mathieu

Quoted 136 lines
Documentation/devicetree/bindings/remoteproc/ti,k3-r5f-rproc.yaml new file +133 -0
+
+    required:
+     - compatible
+     - reg
+     - reg-names
+     - ti,sci
+     - ti,sci-dev-id
+     - ti,sci-proc-ids
+     - resets
+     - firmware-name
+
+    additionalProperties: false
+
+required:
+ - compatible
+ - power-domains
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
+
+additionalProperties: false
+
+examples:
+  - |
+
+    //Example: AM654 SoC
+    /* R5F DDR Carveout reserved memory nodes */
+    reserved-memory {
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges;
+
+        mcu_r5fss0_core1_dma_memory_region: r5f-dma-memory@9b000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core1_memory_region: r5f-memory@9b100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9b100000 0x00 0xf00000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_dma_memory_region: r5f-dma-memory@9c000000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c000000 0x00 0x100000>;
+            no-map;
+        };
+
+        mcu_r5fss0_core0_memory_region: r5f-memory@9c100000 {
+            compatible = "shared-dma-pool";
+            reg = <0x00 0x9c100000 0x00 0x700000>;
+            no-map;
+        };
+    };
+
+    cbass_main: interconnect@100000 {
+        compatible = "simple-bus";
+        #address-cells = <2>;
+        #size-cells = <2>;
+        ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>,
+                 <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>,
+                 <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>;
+
+        cbass_mcu: interconnect@28380000 {
+            compatible = "simple-bus";
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x00 0x41000000 0x00 0x41000000 0x00 0x00020000>, /* MCU R5F Core0 */
+                     <0x00 0x41400000 0x00 0x41400000 0x00 0x00020000>, /* MCU R5F Core1 */
+                     <0x00 0x41c00000 0x00 0x41c00000 0x00 0x00080000>; /* MCU SRAM */
+
+            /* MCU domain SRAM node */
+            mcu_ram: mcu-ram@41c00000 {
+                compatible = "mmio-sram";
+                reg = <0x00 0x41c00000 0x00 0x80000>;
+                ranges = <0x0 0x00 0x41c00000 0x80000>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+
+                mcu_r5fss0_core0_sram: r5f-sram@0 {
+                    reg = <0x0 0x40000>;
+                };
+            };
+
+            /* AM65x MCU R5FSS node */
+            mcu_r5fss0: r5fss@41000000 {
+                compatible = "ti,am654-r5fss";
+                power-domains = <&k3_pds 129>;
+                lockstep-mode = <1>;
+                #address-cells = <1>;
+                #size-cells = <1>;
+                ranges = <0x41000000 0x00 0x41000000 0x20000>,
+                         <0x41400000 0x00 0x41400000 0x20000>;
+
+                mcu_r5f0: r5f@41000000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41000000 0x00008000>,
+                          <0x41010000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <159>;
+                    ti,sci-proc-ids = <0x01 0xFF>;
+                    resets = <&k3_reset 159 1>;
+                    firmware-name = "am65x-mcu-r5f0_0-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox0 &mbox_mcu_r5fss0_core0>;
+                    memory-region = <&mcu_r5fss0_core0_dma_memory_region>,
+                                    <&mcu_r5fss0_core0_memory_region>;
+                    sram = <&mcu_r5fss0_core0_sram>;
+                };
+
+                mcu_r5f1: r5f@41400000 {
+                    compatible = "ti,am654-r5f";
+                    reg = <0x41400000 0x00008000>,
+                          <0x41410000 0x00008000>;
+                    reg-names = "atcm", "btcm";
+                    ti,sci = <&dmsc>;
+                    ti,sci-dev-id = <245>;
+                    ti,sci-proc-ids = <0x02 0xFF>;
+                    resets = <&k3_reset 245 1>;
+                    firmware-name = "am65x-mcu-r5f0_1-fw";
+                    atcm-enable = <1>;
+                    btcm-enable = <1>;
+                    loczrama = <1>;
+                    mboxes = <&mailbox1 &mbox_mcu_r5fss0_core1>;
+               };
+           };
+        };
+    };
Signature · 1 line
2.23.0
author
#18 lore raw

PATCH 4/7 remoteproc/k3-r5: Add TI-SCI processor control helper functions

In reply to Suman Anna

Texas Instruments' K3 generation SoCs have specific modules/register spaces used for configuring the various aspects of a remote processor. These include power, reset, boot vector and other configuration features specific to each compute processor present on the SoC. These registers are managed by the System Controller such as DMSC on K3 AM65x SoCs.

The Texas Instrument's System Control Interface (TI-SCI) Message Protocol is used to communicate to the System Controller from various compute processors to invoke specific services provided by the firmware running on the System Controller.

Add a common processor control interface header file that can be used by multiple remoteproc drivers. The helper functions within this header file abstract the various TI SCI protocol ops for the remoteproc drivers, and allow them to request the System Controller to be able to program and manage various remote processors on the SoC. The remoteproc drivers are expected to manage the life-cycle of their ti_sci_proc_dev local structures.

---
 drivers/remoteproc/ti_sci_proc.h | 102 +++++++++++++++++++++++++++++++
 1 file changed, 102 insertions(+)
 create mode 100644 drivers/remoteproc/ti_sci_proc.h
drivers/remoteproc/ti_sci_proc.h new file +102 -0
new file mode 100644
index 000000000000..e42d8015b8e7
@@ -0,0 +1,102 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Texas Instruments TI-SCI Processor Controller Helper Functions
+ *
+ * Copyright (C) 2018-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *	Suman Anna
+ */
+
+#ifndef REMOTEPROC_TI_SCI_PROC_H
+#define REMOTEPROC_TI_SCI_PROC_H
+
+/**
+ * struct ti_sci_proc - structure representing a processor control client
+ * @sci: cached TI-SCI protocol handle
+ * @ops: cached TI-SCI proc ops
+ * @dev: cached client device pointer
+ * @proc_id: processor id for the consumer remoteproc device
+ * @host_id: host id to pass the control over for this consumer remoteproc
+ *	     device
+ */
+struct ti_sci_proc {
+	const struct ti_sci_handle *sci;
+	const struct ti_sci_proc_ops *ops;
+	struct device *dev;
+	u8 proc_id;
+	u8 host_id;
+};
+
+static inline int ti_sci_proc_request(struct ti_sci_proc *tsp)
+{
+	int ret;
+
+	ret = tsp->ops->request(tsp->sci, tsp->proc_id);
+	if (ret)
+		dev_err(tsp->dev, "ti-sci processor request failed: %d\n",
+			ret);
+	return ret;
+}
+
+static inline int ti_sci_proc_release(struct ti_sci_proc *tsp)
+{
+	int ret;
+
+	ret = tsp->ops->release(tsp->sci, tsp->proc_id);
+	if (ret)
+		dev_err(tsp->dev, "ti-sci processor release failed: %d\n",
+			ret);
+	return ret;
+}
+
+static inline int ti_sci_proc_handover(struct ti_sci_proc *tsp)
+{
+	int ret;
+
+	ret = tsp->ops->handover(tsp->sci, tsp->proc_id, tsp->host_id);
+	if (ret)
+		dev_err(tsp->dev, "ti-sci processor handover of %d to %d failed: %d\n",
+			tsp->proc_id, tsp->host_id, ret);
+	return ret;
+}
+
+static inline int ti_sci_proc_set_config(struct ti_sci_proc *tsp,
+					 u64 boot_vector,
+					 u32 cfg_set, u32 cfg_clr)
+{
+	int ret;
+
+	ret = tsp->ops->set_config(tsp->sci, tsp->proc_id, boot_vector,
+				   cfg_set, cfg_clr);
+	if (ret)
+		dev_err(tsp->dev, "ti-sci processor set_config failed: %d\n",
+			ret);
+	return ret;
+}
+
+static inline int ti_sci_proc_set_control(struct ti_sci_proc *tsp,
+					  u32 ctrl_set, u32 ctrl_clr)
+{
+	int ret;
+
+	ret = tsp->ops->set_control(tsp->sci, tsp->proc_id, ctrl_set, ctrl_clr);
+	if (ret)
+		dev_err(tsp->dev, "ti-sci processor set_control failed: %d\n",
+			ret);
+	return ret;
+}
+
+static inline int ti_sci_proc_get_status(struct ti_sci_proc *tsp,
+					 u64 *boot_vector, u32 *cfg_flags,
+					 u32 *ctrl_flags, u32 *status_flags)
+{
+	int ret;
+
+	ret = tsp->ops->get_status(tsp->sci, tsp->proc_id, boot_vector,
+				   cfg_flags, ctrl_flags, status_flags);
+	if (ret)
+		dev_err(tsp->dev, "ti-sci processor get_status failed: %d\n",
+			ret);
+	return ret;
+}
+
+#endif /* REMOTEPROC_TI_SCI_PROC_H */
Signature · 1 line
2.23.0
author
#19 lore raw

PATCH 5/7 remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

In reply to Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode. The driver uses various TI-SCI interfaces to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
 drivers/remoteproc/Kconfig               |   16 +
 drivers/remoteproc/Makefile              |    1 +
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
 3 files changed, 1363 insertions(+)
 create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c
drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
 	  This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+	tristate "TI K3 R5 remoteproc support"
+	depends on ARCH_K3
+	select MAILBOX
+	select OMAP2PLUS_MBOX
+	help
+	  Say y here to support TI's R5F remote processor subsystems
+	  on various TI K3 family of SoCs through the remote processor
+	  framework.
+
+	  You want to say y here in order to offload some processing
+	  tasks to these processors
+
+	  It's safe to say N here if you're not interested in utilizing
+	  a slave processor
+
 endif # REMOTEPROC
 
 endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y			+= qcom_wcnss_iris.o
 obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
 obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
 obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)	+= ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +1346 -0

1,346 lines not on this page · raw message

Signature · 1 line
2.23.0
maintainer
#20 lore raw

Re: [PATCH 5/7] remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

Quoted 49 lines · Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

Quoted 2 lines

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode.

Thank you for the detailed explanation, it really helps understand what is going on. From the above the assumption I made in the previous patch about lockstep and split mode is wrong. On the flip side it does highlight the need to add more details to the bindings.

Quoted 75 lines

The driver uses various TI-SCI interfaces
to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
 drivers/remoteproc/Kconfig               |   16 +
 drivers/remoteproc/Makefile              |    1 +
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
 3 files changed, 1363 insertions(+)
 create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c
drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
 	  This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+	tristate "TI K3 R5 remoteproc support"
+	depends on ARCH_K3
+	select MAILBOX
+	select OMAP2PLUS_MBOX
+	help
+	  Say y here to support TI's R5F remote processor subsystems
+	  on various TI K3 family of SoCs through the remote processor
+	  framework.
+
+	  You want to say y here in order to offload some processing
+	  tasks to these processors
+
+	  It's safe to say N here if you're not interested in utilizing
+	  a slave processor
+
 endif # REMOTEPROC
 
 endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y			+= qcom_wcnss_iris.o
 obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
 obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
 obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)	+= ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +11 -0
new file mode 100644
index 000000000000..655f8f14c37d
@@ -0,0 +1,1346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TI K3 R5F (MCU) Remote Processor driver
+ *
+ * Copyright (C) 2017-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *	Suman Anna <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>

drivers/remoteproc/ti_k3_r5_remoteproc.c line 21

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/omap-mailbox.h>

Please move this up after of_reserved_mem.h

Quoted 1,070 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +1070 -0
+#include <linux/reset.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+#include "omap_remoteproc.h"
+#include "remoteproc_internal.h"
+#include "ti_sci_proc.h"
+
+/* This address can either be for ATCM or BTCM with the other at address 0x0 */
+#define K3_R5_TCM_DEV_ADDR	0x41010000
+
+/* R5 TI-SCI Processor Configuration Flags */
+#define PROC_BOOT_CFG_FLAG_R5_DBG_EN			0x00000001
+#define PROC_BOOT_CFG_FLAG_R5_DBG_NIDEN			0x00000002
+#define PROC_BOOT_CFG_FLAG_R5_LOCKSTEP			0x00000100
+#define PROC_BOOT_CFG_FLAG_R5_TEINIT			0x00000200
+#define PROC_BOOT_CFG_FLAG_R5_NMFI_EN			0x00000400
+#define PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE		0x00000800
+#define PROC_BOOT_CFG_FLAG_R5_BTCM_EN			0x00001000
+#define PROC_BOOT_CFG_FLAG_R5_ATCM_EN			0x00002000
+
+/* R5 TI-SCI Processor Control Flags */
+#define PROC_BOOT_CTRL_FLAG_R5_CORE_HALT		0x00000001
+
+/* R5 TI-SCI Processor Status Flags */
+#define PROC_BOOT_STATUS_FLAG_R5_WFE			0x00000001
+#define PROC_BOOT_STATUS_FLAG_R5_WFI			0x00000002
+#define PROC_BOOT_STATUS_FLAG_R5_CLK_GATED		0x00000004
+#define PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED	0x00000100
+
+/**
+ * struct k3_r5_mem - internal memory structure
+ * @cpu_addr: MPU virtual address of the memory region
+ * @bus_addr: Bus address used to access the memory region
+ * @dev_addr: Device address from remoteproc view
+ * @size: Size of the memory region
+ */
+struct k3_r5_mem {
+	void __iomem *cpu_addr;
+	phys_addr_t bus_addr;
+	u32 dev_addr;
+	size_t size;
+};
+
+enum cluster_mode {
+	CLUSTER_MODE_SPLIT = 0,
+	CLUSTER_MODE_LOCKSTEP,
+};
+
+/**
+ * struct k3_r5_cluster - K3 R5F Cluster structure
+ * @dev: cached device pointer
+ * @mode: Mode to configure the Cluster - Split or LockStep
+ * @cores: list of R5 cores within the cluster
+ */
+struct k3_r5_cluster {
+	struct device *dev;
+	enum cluster_mode mode;
+	struct list_head cores;
+};
+
+/**
+ * struct k3_r5_core - K3 R5 core structure
+ * @elem: linked list item
+ * @dev: cached device pointer
+ * @rproc: rproc handle representing this core
+ * @mem: internal memory regions data
+ * @num_mems: number of internal memory regions
+ * @reset: reset control handle
+ * @tsp: TI-SCI processor control handle
+ * @ti_sci: TI-SCI handle
+ * @ti_sci_id: TI-SCI device identifier
+ * @atcm_enable: flag to control ATCM enablement
+ * @btcm_enable: flag to control BTCM enablement
+ * @loczrama: flag to dictate which TCM is at device address 0x0
+ */
+struct k3_r5_core {
+	struct list_head elem;
+	struct device *dev;
+	struct rproc *rproc;
+	struct k3_r5_mem *mem;
+	int num_mems;
+	struct reset_control *reset;
+	struct ti_sci_proc *tsp;
+	const struct ti_sci_handle *ti_sci;
+	u32 ti_sci_id;
+	u32 atcm_enable;
+	u32 btcm_enable;
+	u32 loczrama;
+};
+
+/**
+ * struct k3_r5_rproc - K3 remote processor state
+ * @dev: cached device pointer
+ * @cluster: cached pointer to parent cluster structure
+ * @mbox: mailbox channel handle
+ * @client: mailbox client to request the mailbox channel
+ * @rproc: rproc handle
+ * @core: cached pointer to r5 core structure being used
+ * @rmem: reserved memory regions data
+ * @num_rmems: number of reserved memory regions
+ */
+struct k3_r5_rproc {
+	struct device *dev;
+	struct k3_r5_cluster *cluster;
+	struct mbox_chan *mbox;
+	struct mbox_client client;
+	struct rproc *rproc;
+	struct k3_r5_core *core;
+	struct k3_r5_mem *rmem;
+	int num_rmems;
+};
+
+/**
+ * k3_r5_rproc_mbox_callback() - inbound mailbox message handler
+ * @client: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by the OMAP mailbox driver whenever a mailbox
+ * message is received. Usually, the mailbox payload simply contains
+ * the index of the virtqueue that is kicked by the remote processor,
+ * and we let remoteproc core handle it.
+ *
+ * In addition to virtqueue indices, we also have some out-of-band values
+ * that indicate different events. Those values are deliberately very
+ * large so they don't coincide with virtqueue indices.
+ */
+static void k3_r5_rproc_mbox_callback(struct mbox_client *client, void *data)
+{
+	struct k3_r5_rproc *kproc = container_of(client, struct k3_r5_rproc,
+						client);
+	struct device *dev = kproc->rproc->dev.parent;
+	const char *name = kproc->rproc->name;
+	u32 msg = omap_mbox_message(data);
+
+	dev_dbg(dev, "mbox msg: 0x%x\n", msg);
+
+	switch (msg) {
+	case RP_MBOX_CRASH:
+		/*
+		 * remoteproc detected an exception, but error recovery is not
+		 * supported. So, just log this for now
+		 */
+		dev_err(dev, "K3 R5F rproc %s crashed\n", name);
+		break;
+	case RP_MBOX_ECHO_REPLY:
+		dev_info(dev, "received echo reply from %s\n", name);
+		break;
+	default:
+		/* silently handle all other valid messages */
+		if (msg >= RP_MBOX_READY && msg < RP_MBOX_END_MSG)
+			return;
+		if (msg > kproc->rproc->max_notifyid) {
+			dev_dbg(dev, "dropping unknown message 0x%x", msg);
+			return;
+		}
+		/* msg contains the index of the triggered vring */
+		if (rproc_vq_interrupt(kproc->rproc, msg) == IRQ_NONE)
+			dev_dbg(dev, "no message was found in vqid %d\n", msg);
+	}
+}
+
+/* kick a virtqueue */
+static void k3_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	mbox_msg_t msg = (mbox_msg_t)vqid;
+	int ret;
+
+	/* send the index of the triggered virtqueue in the mailbox payload */
+	ret = mbox_send_message(kproc->mbox, (void *)msg);
+	if (ret < 0)
+		dev_err(dev, "failed to send mailbox message, status = %d\n",
+			ret);
+}
+
+static int k3_r5_split_reset(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = reset_control_assert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+			ret);
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_split_release(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = reset_control_deassert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset deassert failed, ret = %d\n",
+			ret);
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_reset(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* assert local reset on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = reset_control_assert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+				ret);
+			core = list_prev_entry(core, elem);
+			goto unroll_local_reset;
+		}
+	}
+
+	/* disable PSC modules on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+				ret);
+			goto unroll_module_reset;
+		}
+	}
+
+	return 0;
+
+unroll_module_reset:
+	list_for_each_entry_continue_reverse(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+	core = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_local_reset:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_release(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* enable PSC modules on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			core = list_next_entry(core, elem);
+			goto unroll_module_reset;
+		}
+	}
+
+	/* deassert local reset on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = reset_control_deassert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			goto unroll_local_reset;
+		}
+	}
+
+	return 0;
+
+unroll_local_reset:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (reset_control_assert(core->reset))
+			dev_warn(core->dev, "local-reset assert back failed\n");
+	}
+	core = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_module_reset:
+	list_for_each_entry_from(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static inline int k3_r5_core_halt(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       PROC_BOOT_CTRL_FLAG_R5_CORE_HALT, 0);
+}
+
+static inline int k3_r5_core_run(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       0, PROC_BOOT_CTRL_FLAG_R5_CORE_HALT);
+}
+
+/*
+ * The R5F cores have controls for both a reset and a halt/run. The code
+ * execution from DDR requires the initial boot-strapping code to be run
+ * from the internal TCMs. This function is used to release the resets on
+ * applicable cores to allow loading into the TCMs. The .prepare() ops is
+ * invoked by remoteproc core before any firmware loading, and is followed
+ * by the .start() ops after loading to actually let the R5 cores run.
+ */
+static int k3_r5_rproc_prepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_release(cluster) :
+			      k3_r5_split_release(core);
+	if (ret)
+		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
+			ret);
+
+	return ret;
+}
+
+/*
+ * This function implements the .unprepare() ops and performs the complimentary
+ * operations to that of the .prepare() ops. The function is used to assert the
+ * resets on all applicable cores for the rproc device (depending on LockStep
+ * or Split mode). This completes the second portion of powering down the R5F
+ * cores. The cores themselves are only halted in the .stop() ops, and the
+ * .unprepare() ops is invoked by the remoteproc core after the remoteproc is
+ * stopped.
+ */
+static int k3_r5_rproc_unprepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_reset(cluster) :
+			      k3_r5_split_reset(core);
+	if (ret)
+		dev_err(dev, "unable to disable cores, ret = %d\n", ret);
+
+	return ret;
+}
+
+/*
+ * The R5F start sequence includes two different operations
+ * 1. Configure the boot vector for R5F core(s)
+ * 2. Unhalt/Run the R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes. The LockStep
+ * mode requires the boot vector to be configured only for Core0, and then
+ * unhalt both the cores to start the execution - Core1 needs to be unhalted
+ * first followed by Core0. The Split-mode requires that Core0 to be maintained
+ * always in a higher power state that Core1 (implying Core1 needs to be started
+ * always only after Core0 is started).
+ */
+static int k3_r5_rproc_start(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct mbox_client *client = &kproc->client;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core;
+	u32 boot_addr;
+	int ret;
+
+	client->dev = dev;
+	client->tx_done = NULL;
+	client->rx_callback = k3_r5_rproc_mbox_callback;
+	client->tx_block = false;
+	client->knows_txdone = false;
+
+	kproc->mbox = mbox_request_channel(client, 0);
+	if (IS_ERR(kproc->mbox)) {
+		ret = -EBUSY;
+		dev_err(dev, "mbox_request_channel failed: %ld\n",
+			PTR_ERR(kproc->mbox));
+		return ret;
+	}
+
+	/*
+	 * Ping the remote processor, this is only for sanity-sake for now;
+	 * there is no functional effect whatsoever.
+	 *
+	 * Note that the reply will _not_ arrive immediately: this message
+	 * will wait in the mailbox fifo until the remote processor is booted.
+	 */
+	ret = mbox_send_message(kproc->mbox, (void *)RP_MBOX_ECHO_REQUEST);
+	if (ret < 0) {
+		dev_err(dev, "mbox_send_message failed: %d\n", ret);
+		goto put_mbox;
+	}
+
+	boot_addr = rproc->bootaddr;
+	/* TODO: add boot_addr sanity checking */
+	dev_err(dev, "booting R5F core using boot addr = 0x%x\n", boot_addr);
+
+	/* boot vector need not be programmed for Core1 in LockStep mode */
+	core = kproc->core;
+	ret = ti_sci_proc_set_config(core->tsp, boot_addr, 0, 0);
+	if (ret)
+		goto put_mbox;
+
+	/* unhalt/run all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry_reverse(core, &cluster->cores, elem) {
+			ret = k3_r5_core_run(core);
+			if (ret)
+				goto unroll_core_run;
+		}
+	} else {
+		ret = k3_r5_core_run(core);
+		if (ret)
+			goto put_mbox;
+	}
+
+	return 0;
+
+unroll_core_run:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (k3_r5_core_halt(core))
+			dev_warn(core->dev, "core halt back failed\n");
+	}
+put_mbox:
+	mbox_free_channel(kproc->mbox);
+	return ret;
+}
+
+/*
+ * The R5F stop function includes the following operations
+ * 1. Halt R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes, and the order
+ * of cores the operations are performed are also in general reverse to that
+ * of the start function. The LockStep mode requires each operation to be
+ * performed first on Core0 followed by Core1. The Split-mode requires that
+ * Core0 to be maintained always in a higher power state that Core1 (implying
+ * Core1 needs to be stopped first before Core0).
+ *
+ * Note that the R5F halt operation in general is not effective when the R5F
+ * core is running, but is needed to make sure the core won't run after
+ * deasserting the reset the subsequent time. The asserting of reset can
+ * be done here, but is preferred to be done in the .unprepare() ops - this
+ * maintains the symmetric behavior between the .start(), .stop(), .prepare()
+ * and .unprepare() ops, and also balances them well between sysfs 'state'
+ * flow and device bind/unbind or module removal.
+ */
+static int k3_r5_rproc_stop(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	int ret;
+
+	/* halt all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry(core, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret) {
+				core = list_prev_entry(core, elem);
+				goto unroll_core_halt;
+			}
+		}
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+	}
+
+	mbox_free_channel(kproc->mbox);
+
+	return 0;
+
+unroll_core_halt:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (k3_r5_core_run(core))
+			dev_warn(core->dev, "core run back failed\n");
+	}
+out:
+	return ret;
+}
+
+/*
+ * Internal Memory translation helper
+ *
+ * Custom function implementing the rproc .da_to_va ops to provide address
+ * translation (device address to kernel virtual address) for internal RAMs
+ * present in a DSP or IPU device). The translated addresses can be used
+ * either by the remoteproc core for loading, or by any rpmsg bus drivers.
+ */
+static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_core *core = kproc->core;
+	void __iomem *va = NULL;
+	phys_addr_t bus_addr;
+	u32 dev_addr, offset;
+	size_t size;
+	int i;
+
+	if (len == 0)
+		return NULL;
+
+	/* handle both R5 and SoC views of ATCM and BTCM */
+	for (i = 0; i < core->num_mems; i++) {
+		bus_addr = core->mem[i].bus_addr;
+		dev_addr = core->mem[i].dev_addr;
+		size = core->mem[i].size;
+
+		/* handle R5-view addresses of TCMs */
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+
+		/* handle SoC-view addresses of TCMs */
+		if (da >= bus_addr && ((da + len) <= (bus_addr + size))) {
+			offset = da - bus_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	/* handle static DDR reserved memory regions */
+	for (i = 0; i < kproc->num_rmems; i++) {
+		dev_addr = kproc->rmem[i].dev_addr;
+		size = kproc->rmem[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = kproc->rmem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	return NULL;
+}
+
+static const struct rproc_ops k3_r5_rproc_ops = {
+	.prepare	= k3_r5_rproc_prepare,
+	.unprepare	= k3_r5_rproc_unprepare,
+	.start		= k3_r5_rproc_start,
+	.stop		= k3_r5_rproc_stop,
+	.kick		= k3_r5_rproc_kick,
+	.da_to_va	= k3_r5_rproc_da_to_va,
+};
+
+static const char *k3_r5_rproc_get_firmware(struct device *dev)
+{
+	const char *fw_name;
+	int ret;
+
+	ret = of_property_read_string(dev->of_node, "firmware-name",
+				      &fw_name);
+	if (ret) {
+		dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+			ret);
+		return ERR_PTR(ret);
+	}
+
+	return fw_name;
+}
+
+static int k3_r5_rproc_configure(struct k3_r5_rproc *kproc)
+{
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core0, *core, *temp;
+	u32 ctrl = 0, cfg = 0, stat = 0;
+	u32 set_cfg = 0, clr_cfg = 0;
+	u64 boot_vec = 0;
+	bool lockstep_en;
+	int ret;
+
+	core0 = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+	core = cluster->mode ? core0 : kproc->core;
+
+	ret = ti_sci_proc_get_status(core->tsp, &boot_vec, &cfg, &ctrl,
+				     &stat);
+	if (ret < 0)
+		return ret;
+
+	dev_dbg(dev, "boot_vector = 0x%llx, cfg = 0x%x ctrl = 0x%x stat = 0x%x\n",
+		boot_vec, cfg, ctrl, stat);
+
+	lockstep_en = !!(stat & PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED);
+	if (!lockstep_en && cluster->mode) {
+		dev_err(cluster->dev, "lockstep mode not permitted, force configuring for split-mode\n");
+		cluster->mode = 0;
+	}
+
+	/* always enable ARM mode and set boot vector to 0 */
+	boot_vec = 0x0;
+	if (core == core0) {
+		clr_cfg = PROC_BOOT_CFG_FLAG_R5_TEINIT;
+		/*
+		 * LockStep configuration bit is Read-only on Split-mode _only_
+		 * devices and system firmware will NACK any requests with the
+		 * bit configured, so program it only on permitted devices
+		 */
+		if (lockstep_en)
+			clr_cfg |= PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+	}
+
+	if (core->atcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+
+	if (core->btcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+
+	if (core->loczrama)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+
+	if (cluster->mode) {
+		/*
+		 * work around system firmware limitations to make sure both
+		 * cores are programmed symmetrically in LockStep. LockStep
+		 * and TEINIT config is only allowed with Core0.
+		 */
+		list_for_each_entry(temp, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret)
+				goto out;
+
+			if (temp != core) {
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_TEINIT;
+			}
+			ret = ti_sci_proc_set_config(temp->tsp, boot_vec,
+						     set_cfg, clr_cfg);
+			if (ret)
+				goto out;
+		}
+
+		set_cfg = PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+		clr_cfg = 0;
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	}
+
+out:
+	return ret;
+}
+
+static int k3_r5_reserved_mem_init(struct k3_r5_rproc *kproc)
+{
+	struct device *dev = kproc->dev;
+	struct device_node *np = dev->of_node;
+	struct device_node *rmem_np;
+	struct reserved_mem *rmem;
+	int num_rmems;
+	int ret, i;
+
+	num_rmems = of_property_count_elems_of_size(np, "memory-region",
+						    sizeof(phandle));
+	if (num_rmems <= 0) {
+		dev_err(dev, "device does not have reserved memory regions, ret = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+	if (num_rmems < 2) {
+		dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+
+	/* use reserved memory region 0 for vring DMA allocations */
+	ret = of_reserved_mem_device_init_by_idx(dev, np, 0);
+	if (ret) {
+		dev_err(dev, "device cannot initialize DMA pool, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_rmems--;
+	kproc->rmem = kcalloc(num_rmems, sizeof(*kproc->rmem), GFP_KERNEL);
+	if (!kproc->rmem) {
+		ret = -ENOMEM;
+		goto release_rmem;
+	}
+
+	/* use remaining reserved memory regions for static carveouts */
+	for (i = 0; i < num_rmems; i++) {
+		rmem_np = of_parse_phandle(np, "memory-region", i + 1);
+		if (!rmem_np) {
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+
+		rmem = of_reserved_mem_lookup(rmem_np);
+		if (!rmem) {
+			of_node_put(rmem_np);
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+		of_node_put(rmem_np);
+
+		kproc->rmem[i].bus_addr = rmem->base;
+		/* 64-bit address regions currently not supported */
+		kproc->rmem[i].dev_addr = (u32)rmem->base;
+		kproc->rmem[i].size = rmem->size;
+		kproc->rmem[i].cpu_addr = ioremap_wc(rmem->base, rmem->size);
+		if (!kproc->rmem[i].cpu_addr) {
+			dev_err(dev, "failed to map reserved memory#%d at %pa of size %pa\n",
+				i + 1, &rmem->base, &rmem->size);
+			ret = -ENOMEM;
+			goto unmap_rmem;
+		}
+
+		dev_dbg(dev, "reserved memory%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			i + 1, &kproc->rmem[i].bus_addr,
+			kproc->rmem[i].size, kproc->rmem[i].cpu_addr,
+			kproc->rmem[i].dev_addr);
+	}
+	kproc->num_rmems = num_rmems;
+
+	return 0;
+
+unmap_rmem:
+	for (i--; i >= 0; i--) {
+		if (kproc->rmem[i].cpu_addr)
+			iounmap(kproc->rmem[i].cpu_addr);
+	}
+	kfree(kproc->rmem);
+release_rmem:
+	of_reserved_mem_device_release(dev);
+	return ret;
+}
+
+static void k3_r5_reserved_mem_exit(struct k3_r5_rproc *kproc)
+{
+	int i;
+
+	for (i = 0; i < kproc->num_rmems; i++)
+		iounmap(kproc->rmem[i].cpu_addr);
+	kfree(kproc->rmem);
+
+	of_reserved_mem_device_release(kproc->dev);
+}
+
+static int k3_r5_cluster_rproc_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core, *core1;
+	struct device *cdev;
+	const char *fw_name;
+	struct rproc *rproc;
+	int ret;
+
+	core1 = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+	list_for_each_entry(core, &cluster->cores, elem) {
+		cdev = core->dev;
+		fw_name = k3_r5_rproc_get_firmware(cdev);
+		if (IS_ERR(fw_name)) {
+			ret = PTR_ERR(fw_name);
+			goto out;
+		}
+
+		rproc = rproc_alloc(cdev, dev_name(cdev), &k3_r5_rproc_ops,
+				    fw_name, sizeof(*kproc));
+		if (!rproc) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		/* K3 R5s have a Region Address Translator (RAT) but no MMU */
+		rproc->has_iommu = false;
+		/* error recovery is not supported at present */
+		rproc->recovery_disabled = true;
+
+		kproc = rproc->priv;
+		kproc->cluster = cluster;
+		kproc->core = core;
+		kproc->dev = cdev;
+		kproc->rproc = rproc;
+		core->rproc = rproc;
+
+		ret = k3_r5_rproc_configure(kproc);
+		if (ret) {
+			dev_err(dev, "initial configure failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = k3_r5_reserved_mem_init(kproc);
+		if (ret) {
+			dev_err(dev, "reserved memory init failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = rproc_add(rproc);
+		if (ret) {
+			dev_err(dev, "rproc_add failed, ret = %d\n", ret);
+			goto err_add;
+		}
+
+		/* create only one rproc in lockstep mode */
+		if (cluster->mode)
+			break;
+	}
+
+	return 0;
+
+err_split:
+	rproc_del(rproc);
+err_add:
+	k3_r5_reserved_mem_exit(kproc);
+err_config:
+	rproc_free(rproc);
+	core->rproc = NULL;
+out:
+	/* undo core0 upon any failures on core1 in split-mode */
+	if (!cluster->mode && core == core1) {
+		core = list_prev_entry(core, elem);
+		rproc = core->rproc;
+		kproc = rproc->priv;
+		goto err_split;
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_rproc_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core;
+	struct rproc *rproc;
+
+	/*
+	 * lockstep mode has only one rproc associated with first core, whereas
+	 * split-mode has two rprocs associated with each core, and requires
+	 * that core1 be powered down first
+	 */
+	core = cluster->mode ?
+		list_first_entry(&cluster->cores, struct k3_r5_core, elem) :
+		list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		rproc = core->rproc;
+		kproc = rproc->priv;
+
+		rproc_del(rproc);
+
+		k3_r5_reserved_mem_exit(kproc);
+
+		rproc_free(rproc);
+		core->rproc = NULL;
+	}
+
+	return 0;
+}
+
+static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
+					       struct k3_r5_core *core)
+{
+	static const char * const mem_names[] = {"atcm", "btcm"};
+	struct device *dev = &pdev->dev;
+	struct resource *res;
+	int num_mems;
+	int i, ret;
+
+	num_mems = ARRAY_SIZE(mem_names);
+	core->mem = devm_kcalloc(dev, num_mems, sizeof(*core->mem), GFP_KERNEL);
+	if (!core->mem)
+		return -ENOMEM;
+
+	for (i = 0; i < num_mems; i++) {
+		res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						   mem_names[i]);
+		if (!res) {
+			dev_err(dev, "found no memory resource for %s\n",
+				mem_names[i]);
+			ret = -EINVAL;
+			goto fail;
+		}
+		if (!devm_request_mem_region(dev, res->start,
+					     resource_size(res),
+					     dev_name(dev))) {
+			dev_err(dev, "could not request %s region for resource\n",
+				mem_names[i]);
+			ret = -EBUSY;
+			goto fail;
+		}
+
+		/*
+		 * TCMs are designed in general to support RAM-like backing
+		 * memories. So, map these as Normal Non-Cached memories. This
+		 * also avoids/fixes any potential alignment faults due to
+		 * unaligned data accesses when using memcpy() or memset()
+		 * functions (normally seen with device type memory).
+		 */
+		core->mem[i].cpu_addr = devm_ioremap_wc(dev, res->start,
+							resource_size(res));
+		if (IS_ERR(core->mem[i].cpu_addr)) {
+			dev_err(dev, "failed to map %s memory\n", mem_names[i]);
+			ret = PTR_ERR(core->mem[i].cpu_addr);
+			devm_release_mem_region(dev, res->start,
+						resource_size(res));
+			goto fail;
+		}
+		core->mem[i].bus_addr = res->start;
+
+		/*
+		 * TODO:
+		 * The R5F cores can place ATCM & BTCM anywhere in its address
+		 * based on the corresponding Region Registers in the System
+		 * Control coprocessor. For now, place ATCM and BTCM at
+		 * addresses 0 and 0x41010000 (same as the bus address on AM65x
+		 * SoCs) based on loczrama setting
+		 */
+		if (!strcmp(mem_names[i], "atcm")) {
+			core->mem[i].dev_addr = core->loczrama ?
+							0 : K3_R5_TCM_DEV_ADDR;
+		} else {
+			core->mem[i].dev_addr = core->loczrama ?
+							K3_R5_TCM_DEV_ADDR : 0;
+		}
+		core->mem[i].size = resource_size(res);
+
+		dev_dbg(dev, "memory %8s: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			mem_names[i], &core->mem[i].bus_addr,
+			core->mem[i].size, core->mem[i].cpu_addr,
+			core->mem[i].dev_addr);
+	}
+	core->num_mems = num_mems;
+
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+	return ret;
+}
+
+static
+struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
+					  const struct ti_sci_handle *sci)
+{
+	struct ti_sci_proc *tsp;
+	u32 temp[2];
+	int ret;
+
+	ret = of_property_read_u32_array(dev->of_node, "ti,sci-proc-ids",
+					 temp, 2);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	tsp = kzalloc(sizeof(*tsp), GFP_KERNEL);
+	if (!tsp)
+		return ERR_PTR(-ENOMEM);
+
+	tsp->dev = dev;
+	tsp->sci = sci;
+	tsp->ops = &sci->ops.proc_ops;
+	tsp->proc_id = temp[0];
+	tsp->host_id = temp[1];
+
+	return tsp;
+}
+
+static int k3_r5_core_of_init(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_core *core;
+	int ret, ret1;
+
+	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
+	if (!core)
+		return -ENOMEM;
+
+	core->dev = dev;
+	core->atcm_enable = 0;
+	core->btcm_enable = 1;
+	core->loczrama = 1;
+
+	ret = of_property_read_u32(np, "atcm-enable", &core->atcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for atcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "btcm-enable", &core->btcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for btcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "loczrama", &core->loczrama);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for loczrama, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	core->ti_sci = ti_sci_get_by_phandle(np, "ti,sci");
+	if (IS_ERR(core->ti_sci)) {
+		ret = PTR_ERR(core->ti_sci);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get ti-sci handle, ret = %d\n",
+				ret);
+		}
+		core->ti_sci = NULL;
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "ti,sci-dev-id", &core->ti_sci_id);
+	if (ret) {
+		dev_err(dev, "missing 'ti,sci-dev-id' property\n");
+		goto err_sci_id;
+	}
+
+	core->reset = reset_control_get_exclusive(dev, NULL);
+	if (IS_ERR(core->reset)) {
+		ret = PTR_ERR(core->reset);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get reset handle, ret = %d\n",
+				ret);

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1101

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+		}
+		goto err_sci_id;
+	}
+
+	core->tsp = k3_r5_core_of_get_tsp(dev, core->ti_sci);
+	if (IS_ERR(core->tsp)) {
+		dev_err(dev, "failed to construct ti-sci proc control, ret = %d\n",
+			ret);
+		ret = PTR_ERR(core->tsp);
+		goto err_sci_proc;

Why not calling the label "err_reset"? I don't see what how "err_sci_proc" relates to what the code is doing...

I'll continue reviewing this patch tomorrow.

Thanks,
Mathieu

Quoted 100 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +100 -0
+	}
+
+	ret = ti_sci_proc_request(core->tsp);
+	if (ret < 0) {
+		dev_err(dev, "ti_sci_proc_request failed, ret = %d\n", ret);
+		goto err_proc;
+	}
+
+	ret = k3_r5_core_of_get_internal_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get internal memories, ret = %d\n",
+			ret);
+		goto err_intmem;
+	}
+
+	platform_set_drvdata(pdev, core);
+
+	return 0;
+
+err_intmem:
+	ret1 = ti_sci_proc_release(core->tsp);
+	if (ret1)
+		dev_err(dev, "failed to release proc, ret1 = %d\n", ret1);
+err_proc:
+	kfree(core->tsp);
+err_sci_proc:
+	reset_control_put(core->reset);
+err_sci_id:
+	ret1 = ti_sci_put_handle(core->ti_sci);
+	if (ret1)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret1);
+err_of:
+	devm_kfree(dev, core);
+	return ret;
+}
+
+/*
+ * free the resources explicitly since driver model is not being used
+ * for the child R5F devices
+ */
+static int k3_r5_core_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_core *core = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	int i, ret;
+
+	for (i = 0; i < core->num_mems; i++) {
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+
+	ret = ti_sci_proc_release(core->tsp);
+	if (ret)
+		dev_err(dev, "failed to release proc, ret = %d\n", ret);
+
+	kfree(core->tsp);
+	reset_control_put(core->reset);
+
+	ret = ti_sci_put_handle(core->ti_sci);
+	if (ret)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret);
+
+	platform_set_drvdata(pdev, NULL);
+	devm_kfree(dev, core);
+
+	return ret;
+}
+
+static int k3_r5_cluster_of_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct platform_device *cpdev;
+	struct device_node *child;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	for_each_available_child_of_node(np, child) {
+		cpdev = of_find_device_by_node(child);
+		if (!cpdev) {
+			ret = -ENODEV;
+			dev_err(dev, "could not get R5 core platform device\n");
+			goto fail;
+		}
+
+		ret = k3_r5_core_of_init(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_init failed, ret = %d\n",
+				ret);
+			put_device(&cpdev->dev);
+			goto fail;
+		}
+
+		core = platform_get_drvdata(cpdev);
+		put_device(&cpdev->dev);
+		list_add_tail(&core->elem, &cluster->cores);

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1211

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	}
+
+	return 0;
+
+fail:
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		if (k3_r5_core_of_exit(cpdev))
+			dev_err(dev, "k3_r5_core_of_exit cleanup failed\n");

I would move k3_r5_cluster_of_exit() just above k3_r5_cluster_of_init() and reuse the former here instead of repeating the same code.

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct platform_device *cpdev;
+	struct k3_r5_core *core, *temp;

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1231

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	int ret;
+
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		ret = k3_r5_core_of_exit(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_exit failed, ret = %d\n",
+				ret);
+			break;

So why stopping when encountering an error? Since this is the exit path I think it is better to call k3_r5_core_of_exit() on as many core as possible.

Quoted 118 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +115 -0
+		}
+	}
+
+	return ret;
+}
+
+static int k3_r5_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_cluster *cluster;
+	int ret, ret1;
+	int num_cores;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+
+	cluster->dev = dev;
+	cluster->mode = CLUSTER_MODE_LOCKSTEP;
+	INIT_LIST_HEAD(&cluster->cores);
+
+	ret = of_property_read_u32(np, "lockstep-mode", &cluster->mode);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for lockstep-mode, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_cores = of_get_available_child_count(np);
+	if (num_cores != 2) {
+		dev_err(dev, "MCU cluster requires both R5F cores to be enabled, num_cores = %d\n",
+			num_cores);
+		return -ENODEV;
+	}
+
+	platform_set_drvdata(pdev, cluster);
+
+	dev_dbg(dev, "creating child devices for R5F cores\n");
+	ret = of_platform_populate(np, NULL, NULL, dev);
+	if (ret) {
+		dev_err(dev, "of_platform_populate failed, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = k3_r5_cluster_of_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_init failed, ret = %d\n", ret);
+		goto fail_of;
+	}
+
+	ret = k3_r5_cluster_rproc_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_init failed, ret = %d\n",
+			ret);
+		goto fail_rproc;
+	}
+
+	return 0;
+
+fail_rproc:
+	ret1 = k3_r5_cluster_of_exit(pdev);
+	if (ret1)
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret1);
+fail_of:
+	of_platform_depopulate(dev);
+	return ret;
+}
+
+static int k3_r5_remove(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	ret = k3_r5_cluster_rproc_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_exit failed, ret = %d\n",
+			ret);
+		goto fail;
+	}
+
+	ret = k3_r5_cluster_of_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret);
+		goto fail;
+	}
+
+	dev_dbg(dev, "removing child devices for R5F cores\n");
+	of_platform_depopulate(dev);
+
+fail:
+	return ret;
+}
+
+static const struct of_device_id k3_r5_of_match[] = {
+	{ .compatible = "ti,am654-r5fss", },
+	{ .compatible = "ti,j721e-r5fss", },
+	{ /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, k3_r5_of_match);
+
+static struct platform_driver k3_r5_rproc_driver = {
+	.probe = k3_r5_probe,
+	.remove = k3_r5_remove,
+	.driver = {
+		.name = "k3_r5_rproc",
+		.of_match_table = k3_r5_of_match,
+	},
+};
+
+module_platform_driver(k3_r5_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("TI K3 R5F remote processor driver");
+MODULE_AUTHOR("Suman Anna <[email protected]>");
Signature · 1 line
2.23.0
author
#21 lore raw

Re: [PATCH 5/7] remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

Quoted 50 lines · Mathieu Poirier
Quoted 49 lines · Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

Quoted 7 lines
Quoted 2 lines

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode.

Thank you for the detailed explanation, it really helps understand what is going on. From the above the assumption I made in the previous patch about lockstep and split mode is wrong. On the flip side it does highlight the need to add more details to the bindings.

Yeah, I usually tend to lean on the verbose-side, but some of the details on bindings itself are deemed unnecessary.

Quoted 80 lines
Quoted 80 lines

The driver uses various TI-SCI interfaces
to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
 drivers/remoteproc/Kconfig               |   16 +
 drivers/remoteproc/Makefile              |    1 +
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
 3 files changed, 1363 insertions(+)
 create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c
drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
 	  This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+	tristate "TI K3 R5 remoteproc support"
+	depends on ARCH_K3
+	select MAILBOX
+	select OMAP2PLUS_MBOX
+	help
+	  Say y here to support TI's R5F remote processor subsystems
+	  on various TI K3 family of SoCs through the remote processor
+	  framework.
+
+	  You want to say y here in order to offload some processing
+	  tasks to these processors
+
+	  It's safe to say N here if you're not interested in utilizing
+	  a slave processor
+
 endif # REMOTEPROC
 
 endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y			+= qcom_wcnss_iris.o
 obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
 obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
 obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)	+= ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +15 -0
new file mode 100644
index 000000000000..655f8f14c37d
@@ -0,0 +1,1346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TI K3 R5F (MCU) Remote Processor driver
+ *
+ * Copyright (C) 2017-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *	Suman Anna <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 21

Quoted 6 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +6 -0
+#include <linux/of_address.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/omap-mailbox.h>

Please move this up after of_reserved_mem.h

Haa, I missed one after all :). Yep, will move it up.

Quoted 1,076 lines
Quoted 1,076 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +1075 -0
+#include <linux/reset.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+#include "omap_remoteproc.h"
+#include "remoteproc_internal.h"
+#include "ti_sci_proc.h"
+
+/* This address can either be for ATCM or BTCM with the other at address 0x0 */
+#define K3_R5_TCM_DEV_ADDR	0x41010000
+
+/* R5 TI-SCI Processor Configuration Flags */
+#define PROC_BOOT_CFG_FLAG_R5_DBG_EN			0x00000001
+#define PROC_BOOT_CFG_FLAG_R5_DBG_NIDEN			0x00000002
+#define PROC_BOOT_CFG_FLAG_R5_LOCKSTEP			0x00000100
+#define PROC_BOOT_CFG_FLAG_R5_TEINIT			0x00000200
+#define PROC_BOOT_CFG_FLAG_R5_NMFI_EN			0x00000400
+#define PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE		0x00000800
+#define PROC_BOOT_CFG_FLAG_R5_BTCM_EN			0x00001000
+#define PROC_BOOT_CFG_FLAG_R5_ATCM_EN			0x00002000
+
+/* R5 TI-SCI Processor Control Flags */
+#define PROC_BOOT_CTRL_FLAG_R5_CORE_HALT		0x00000001
+
+/* R5 TI-SCI Processor Status Flags */
+#define PROC_BOOT_STATUS_FLAG_R5_WFE			0x00000001
+#define PROC_BOOT_STATUS_FLAG_R5_WFI			0x00000002
+#define PROC_BOOT_STATUS_FLAG_R5_CLK_GATED		0x00000004
+#define PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED	0x00000100
+
+/**
+ * struct k3_r5_mem - internal memory structure
+ * @cpu_addr: MPU virtual address of the memory region
+ * @bus_addr: Bus address used to access the memory region
+ * @dev_addr: Device address from remoteproc view
+ * @size: Size of the memory region
+ */
+struct k3_r5_mem {
+	void __iomem *cpu_addr;
+	phys_addr_t bus_addr;
+	u32 dev_addr;
+	size_t size;
+};
+
+enum cluster_mode {
+	CLUSTER_MODE_SPLIT = 0,
+	CLUSTER_MODE_LOCKSTEP,
+};
+
+/**
+ * struct k3_r5_cluster - K3 R5F Cluster structure
+ * @dev: cached device pointer
+ * @mode: Mode to configure the Cluster - Split or LockStep
+ * @cores: list of R5 cores within the cluster
+ */
+struct k3_r5_cluster {
+	struct device *dev;
+	enum cluster_mode mode;
+	struct list_head cores;
+};
+
+/**
+ * struct k3_r5_core - K3 R5 core structure
+ * @elem: linked list item
+ * @dev: cached device pointer
+ * @rproc: rproc handle representing this core
+ * @mem: internal memory regions data
+ * @num_mems: number of internal memory regions
+ * @reset: reset control handle
+ * @tsp: TI-SCI processor control handle
+ * @ti_sci: TI-SCI handle
+ * @ti_sci_id: TI-SCI device identifier
+ * @atcm_enable: flag to control ATCM enablement
+ * @btcm_enable: flag to control BTCM enablement
+ * @loczrama: flag to dictate which TCM is at device address 0x0
+ */
+struct k3_r5_core {
+	struct list_head elem;
+	struct device *dev;
+	struct rproc *rproc;
+	struct k3_r5_mem *mem;
+	int num_mems;
+	struct reset_control *reset;
+	struct ti_sci_proc *tsp;
+	const struct ti_sci_handle *ti_sci;
+	u32 ti_sci_id;
+	u32 atcm_enable;
+	u32 btcm_enable;
+	u32 loczrama;
+};
+
+/**
+ * struct k3_r5_rproc - K3 remote processor state
+ * @dev: cached device pointer
+ * @cluster: cached pointer to parent cluster structure
+ * @mbox: mailbox channel handle
+ * @client: mailbox client to request the mailbox channel
+ * @rproc: rproc handle
+ * @core: cached pointer to r5 core structure being used
+ * @rmem: reserved memory regions data
+ * @num_rmems: number of reserved memory regions
+ */
+struct k3_r5_rproc {
+	struct device *dev;
+	struct k3_r5_cluster *cluster;
+	struct mbox_chan *mbox;
+	struct mbox_client client;
+	struct rproc *rproc;
+	struct k3_r5_core *core;
+	struct k3_r5_mem *rmem;
+	int num_rmems;
+};
+
+/**
+ * k3_r5_rproc_mbox_callback() - inbound mailbox message handler
+ * @client: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by the OMAP mailbox driver whenever a mailbox
+ * message is received. Usually, the mailbox payload simply contains
+ * the index of the virtqueue that is kicked by the remote processor,
+ * and we let remoteproc core handle it.
+ *
+ * In addition to virtqueue indices, we also have some out-of-band values
+ * that indicate different events. Those values are deliberately very
+ * large so they don't coincide with virtqueue indices.
+ */
+static void k3_r5_rproc_mbox_callback(struct mbox_client *client, void *data)
+{
+	struct k3_r5_rproc *kproc = container_of(client, struct k3_r5_rproc,
+						client);
+	struct device *dev = kproc->rproc->dev.parent;
+	const char *name = kproc->rproc->name;
+	u32 msg = omap_mbox_message(data);
+
+	dev_dbg(dev, "mbox msg: 0x%x\n", msg);
+
+	switch (msg) {
+	case RP_MBOX_CRASH:
+		/*
+		 * remoteproc detected an exception, but error recovery is not
+		 * supported. So, just log this for now
+		 */
+		dev_err(dev, "K3 R5F rproc %s crashed\n", name);
+		break;
+	case RP_MBOX_ECHO_REPLY:
+		dev_info(dev, "received echo reply from %s\n", name);
+		break;
+	default:
+		/* silently handle all other valid messages */
+		if (msg >= RP_MBOX_READY && msg < RP_MBOX_END_MSG)
+			return;
+		if (msg > kproc->rproc->max_notifyid) {
+			dev_dbg(dev, "dropping unknown message 0x%x", msg);
+			return;
+		}
+		/* msg contains the index of the triggered vring */
+		if (rproc_vq_interrupt(kproc->rproc, msg) == IRQ_NONE)
+			dev_dbg(dev, "no message was found in vqid %d\n", msg);
+	}
+}
+
+/* kick a virtqueue */
+static void k3_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	mbox_msg_t msg = (mbox_msg_t)vqid;
+	int ret;
+
+	/* send the index of the triggered virtqueue in the mailbox payload */
+	ret = mbox_send_message(kproc->mbox, (void *)msg);
+	if (ret < 0)
+		dev_err(dev, "failed to send mailbox message, status = %d\n",
+			ret);
+}
+
+static int k3_r5_split_reset(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = reset_control_assert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+			ret);
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_split_release(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = reset_control_deassert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset deassert failed, ret = %d\n",
+			ret);
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_reset(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* assert local reset on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = reset_control_assert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+				ret);
+			core = list_prev_entry(core, elem);
+			goto unroll_local_reset;
+		}
+	}
+
+	/* disable PSC modules on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+				ret);
+			goto unroll_module_reset;
+		}
+	}
+
+	return 0;
+
+unroll_module_reset:
+	list_for_each_entry_continue_reverse(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+	core = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_local_reset:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_release(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* enable PSC modules on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			core = list_next_entry(core, elem);
+			goto unroll_module_reset;
+		}
+	}
+
+	/* deassert local reset on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = reset_control_deassert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			goto unroll_local_reset;
+		}
+	}
+
+	return 0;
+
+unroll_local_reset:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (reset_control_assert(core->reset))
+			dev_warn(core->dev, "local-reset assert back failed\n");
+	}
+	core = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_module_reset:
+	list_for_each_entry_from(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static inline int k3_r5_core_halt(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       PROC_BOOT_CTRL_FLAG_R5_CORE_HALT, 0);
+}
+
+static inline int k3_r5_core_run(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       0, PROC_BOOT_CTRL_FLAG_R5_CORE_HALT);
+}
+
+/*
+ * The R5F cores have controls for both a reset and a halt/run. The code
+ * execution from DDR requires the initial boot-strapping code to be run
+ * from the internal TCMs. This function is used to release the resets on
+ * applicable cores to allow loading into the TCMs. The .prepare() ops is
+ * invoked by remoteproc core before any firmware loading, and is followed
+ * by the .start() ops after loading to actually let the R5 cores run.
+ */
+static int k3_r5_rproc_prepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_release(cluster) :
+			      k3_r5_split_release(core);
+	if (ret)
+		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
+			ret);
+
+	return ret;
+}
+
+/*
+ * This function implements the .unprepare() ops and performs the complimentary
+ * operations to that of the .prepare() ops. The function is used to assert the
+ * resets on all applicable cores for the rproc device (depending on LockStep
+ * or Split mode). This completes the second portion of powering down the R5F
+ * cores. The cores themselves are only halted in the .stop() ops, and the
+ * .unprepare() ops is invoked by the remoteproc core after the remoteproc is
+ * stopped.
+ */
+static int k3_r5_rproc_unprepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_reset(cluster) :
+			      k3_r5_split_reset(core);
+	if (ret)
+		dev_err(dev, "unable to disable cores, ret = %d\n", ret);
+
+	return ret;
+}
+
+/*
+ * The R5F start sequence includes two different operations
+ * 1. Configure the boot vector for R5F core(s)
+ * 2. Unhalt/Run the R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes. The LockStep
+ * mode requires the boot vector to be configured only for Core0, and then
+ * unhalt both the cores to start the execution - Core1 needs to be unhalted
+ * first followed by Core0. The Split-mode requires that Core0 to be maintained
+ * always in a higher power state that Core1 (implying Core1 needs to be started
+ * always only after Core0 is started).
+ */
+static int k3_r5_rproc_start(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct mbox_client *client = &kproc->client;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core;
+	u32 boot_addr;
+	int ret;
+
+	client->dev = dev;
+	client->tx_done = NULL;
+	client->rx_callback = k3_r5_rproc_mbox_callback;
+	client->tx_block = false;
+	client->knows_txdone = false;
+
+	kproc->mbox = mbox_request_channel(client, 0);
+	if (IS_ERR(kproc->mbox)) {
+		ret = -EBUSY;
+		dev_err(dev, "mbox_request_channel failed: %ld\n",
+			PTR_ERR(kproc->mbox));
+		return ret;
+	}
+
+	/*
+	 * Ping the remote processor, this is only for sanity-sake for now;
+	 * there is no functional effect whatsoever.
+	 *
+	 * Note that the reply will _not_ arrive immediately: this message
+	 * will wait in the mailbox fifo until the remote processor is booted.
+	 */
+	ret = mbox_send_message(kproc->mbox, (void *)RP_MBOX_ECHO_REQUEST);
+	if (ret < 0) {
+		dev_err(dev, "mbox_send_message failed: %d\n", ret);
+		goto put_mbox;
+	}
+
+	boot_addr = rproc->bootaddr;
+	/* TODO: add boot_addr sanity checking */
+	dev_err(dev, "booting R5F core using boot addr = 0x%x\n", boot_addr);
+
+	/* boot vector need not be programmed for Core1 in LockStep mode */
+	core = kproc->core;
+	ret = ti_sci_proc_set_config(core->tsp, boot_addr, 0, 0);
+	if (ret)
+		goto put_mbox;
+
+	/* unhalt/run all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry_reverse(core, &cluster->cores, elem) {
+			ret = k3_r5_core_run(core);
+			if (ret)
+				goto unroll_core_run;
+		}
+	} else {
+		ret = k3_r5_core_run(core);
+		if (ret)
+			goto put_mbox;
+	}
+
+	return 0;
+
+unroll_core_run:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (k3_r5_core_halt(core))
+			dev_warn(core->dev, "core halt back failed\n");
+	}
+put_mbox:
+	mbox_free_channel(kproc->mbox);
+	return ret;
+}
+
+/*
+ * The R5F stop function includes the following operations
+ * 1. Halt R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes, and the order
+ * of cores the operations are performed are also in general reverse to that
+ * of the start function. The LockStep mode requires each operation to be
+ * performed first on Core0 followed by Core1. The Split-mode requires that
+ * Core0 to be maintained always in a higher power state that Core1 (implying
+ * Core1 needs to be stopped first before Core0).
+ *
+ * Note that the R5F halt operation in general is not effective when the R5F
+ * core is running, but is needed to make sure the core won't run after
+ * deasserting the reset the subsequent time. The asserting of reset can
+ * be done here, but is preferred to be done in the .unprepare() ops - this
+ * maintains the symmetric behavior between the .start(), .stop(), .prepare()
+ * and .unprepare() ops, and also balances them well between sysfs 'state'
+ * flow and device bind/unbind or module removal.
+ */
+static int k3_r5_rproc_stop(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	int ret;
+
+	/* halt all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry(core, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret) {
+				core = list_prev_entry(core, elem);
+				goto unroll_core_halt;
+			}
+		}
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+	}
+
+	mbox_free_channel(kproc->mbox);
+
+	return 0;
+
+unroll_core_halt:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (k3_r5_core_run(core))
+			dev_warn(core->dev, "core run back failed\n");
+	}
+out:
+	return ret;
+}
+
+/*
+ * Internal Memory translation helper
+ *
+ * Custom function implementing the rproc .da_to_va ops to provide address
+ * translation (device address to kernel virtual address) for internal RAMs
+ * present in a DSP or IPU device). The translated addresses can be used
+ * either by the remoteproc core for loading, or by any rpmsg bus drivers.
+ */
+static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_core *core = kproc->core;
+	void __iomem *va = NULL;
+	phys_addr_t bus_addr;
+	u32 dev_addr, offset;
+	size_t size;
+	int i;
+
+	if (len == 0)
+		return NULL;
+
+	/* handle both R5 and SoC views of ATCM and BTCM */
+	for (i = 0; i < core->num_mems; i++) {
+		bus_addr = core->mem[i].bus_addr;
+		dev_addr = core->mem[i].dev_addr;
+		size = core->mem[i].size;
+
+		/* handle R5-view addresses of TCMs */
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+
+		/* handle SoC-view addresses of TCMs */
+		if (da >= bus_addr && ((da + len) <= (bus_addr + size))) {
+			offset = da - bus_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	/* handle static DDR reserved memory regions */
+	for (i = 0; i < kproc->num_rmems; i++) {
+		dev_addr = kproc->rmem[i].dev_addr;
+		size = kproc->rmem[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = kproc->rmem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	return NULL;
+}
+
+static const struct rproc_ops k3_r5_rproc_ops = {
+	.prepare	= k3_r5_rproc_prepare,
+	.unprepare	= k3_r5_rproc_unprepare,
+	.start		= k3_r5_rproc_start,
+	.stop		= k3_r5_rproc_stop,
+	.kick		= k3_r5_rproc_kick,
+	.da_to_va	= k3_r5_rproc_da_to_va,
+};
+
+static const char *k3_r5_rproc_get_firmware(struct device *dev)
+{
+	const char *fw_name;
+	int ret;
+
+	ret = of_property_read_string(dev->of_node, "firmware-name",
+				      &fw_name);
+	if (ret) {
+		dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+			ret);
+		return ERR_PTR(ret);
+	}
+
+	return fw_name;
+}
+
+static int k3_r5_rproc_configure(struct k3_r5_rproc *kproc)
+{
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core0, *core, *temp;
+	u32 ctrl = 0, cfg = 0, stat = 0;
+	u32 set_cfg = 0, clr_cfg = 0;
+	u64 boot_vec = 0;
+	bool lockstep_en;
+	int ret;
+
+	core0 = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+	core = cluster->mode ? core0 : kproc->core;
+
+	ret = ti_sci_proc_get_status(core->tsp, &boot_vec, &cfg, &ctrl,
+				     &stat);
+	if (ret < 0)
+		return ret;
+
+	dev_dbg(dev, "boot_vector = 0x%llx, cfg = 0x%x ctrl = 0x%x stat = 0x%x\n",
+		boot_vec, cfg, ctrl, stat);
+
+	lockstep_en = !!(stat & PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED);
+	if (!lockstep_en && cluster->mode) {
+		dev_err(cluster->dev, "lockstep mode not permitted, force configuring for split-mode\n");
+		cluster->mode = 0;
+	}
+
+	/* always enable ARM mode and set boot vector to 0 */
+	boot_vec = 0x0;
+	if (core == core0) {
+		clr_cfg = PROC_BOOT_CFG_FLAG_R5_TEINIT;
+		/*
+		 * LockStep configuration bit is Read-only on Split-mode _only_
+		 * devices and system firmware will NACK any requests with the
+		 * bit configured, so program it only on permitted devices
+		 */
+		if (lockstep_en)
+			clr_cfg |= PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+	}
+
+	if (core->atcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+
+	if (core->btcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+
+	if (core->loczrama)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+
+	if (cluster->mode) {
+		/*
+		 * work around system firmware limitations to make sure both
+		 * cores are programmed symmetrically in LockStep. LockStep
+		 * and TEINIT config is only allowed with Core0.
+		 */
+		list_for_each_entry(temp, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret)
+				goto out;
+
+			if (temp != core) {
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_TEINIT;
+			}
+			ret = ti_sci_proc_set_config(temp->tsp, boot_vec,
+						     set_cfg, clr_cfg);
+			if (ret)
+				goto out;
+		}
+
+		set_cfg = PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+		clr_cfg = 0;
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	}
+
+out:
+	return ret;
+}
+
+static int k3_r5_reserved_mem_init(struct k3_r5_rproc *kproc)
+{
+	struct device *dev = kproc->dev;
+	struct device_node *np = dev->of_node;
+	struct device_node *rmem_np;
+	struct reserved_mem *rmem;
+	int num_rmems;
+	int ret, i;
+
+	num_rmems = of_property_count_elems_of_size(np, "memory-region",
+						    sizeof(phandle));
+	if (num_rmems <= 0) {
+		dev_err(dev, "device does not have reserved memory regions, ret = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+	if (num_rmems < 2) {
+		dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+
+	/* use reserved memory region 0 for vring DMA allocations */
+	ret = of_reserved_mem_device_init_by_idx(dev, np, 0);
+	if (ret) {
+		dev_err(dev, "device cannot initialize DMA pool, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_rmems--;
+	kproc->rmem = kcalloc(num_rmems, sizeof(*kproc->rmem), GFP_KERNEL);
+	if (!kproc->rmem) {
+		ret = -ENOMEM;
+		goto release_rmem;
+	}
+
+	/* use remaining reserved memory regions for static carveouts */
+	for (i = 0; i < num_rmems; i++) {
+		rmem_np = of_parse_phandle(np, "memory-region", i + 1);
+		if (!rmem_np) {
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+
+		rmem = of_reserved_mem_lookup(rmem_np);
+		if (!rmem) {
+			of_node_put(rmem_np);
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+		of_node_put(rmem_np);
+
+		kproc->rmem[i].bus_addr = rmem->base;
+		/* 64-bit address regions currently not supported */
+		kproc->rmem[i].dev_addr = (u32)rmem->base;
+		kproc->rmem[i].size = rmem->size;
+		kproc->rmem[i].cpu_addr = ioremap_wc(rmem->base, rmem->size);
+		if (!kproc->rmem[i].cpu_addr) {
+			dev_err(dev, "failed to map reserved memory#%d at %pa of size %pa\n",
+				i + 1, &rmem->base, &rmem->size);
+			ret = -ENOMEM;
+			goto unmap_rmem;
+		}
+
+		dev_dbg(dev, "reserved memory%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			i + 1, &kproc->rmem[i].bus_addr,
+			kproc->rmem[i].size, kproc->rmem[i].cpu_addr,
+			kproc->rmem[i].dev_addr);
+	}
+	kproc->num_rmems = num_rmems;
+
+	return 0;
+
+unmap_rmem:
+	for (i--; i >= 0; i--) {
+		if (kproc->rmem[i].cpu_addr)
+			iounmap(kproc->rmem[i].cpu_addr);
+	}
+	kfree(kproc->rmem);
+release_rmem:
+	of_reserved_mem_device_release(dev);
+	return ret;
+}
+
+static void k3_r5_reserved_mem_exit(struct k3_r5_rproc *kproc)
+{
+	int i;
+
+	for (i = 0; i < kproc->num_rmems; i++)
+		iounmap(kproc->rmem[i].cpu_addr);
+	kfree(kproc->rmem);
+
+	of_reserved_mem_device_release(kproc->dev);
+}
+
+static int k3_r5_cluster_rproc_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core, *core1;
+	struct device *cdev;
+	const char *fw_name;
+	struct rproc *rproc;
+	int ret;
+
+	core1 = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+	list_for_each_entry(core, &cluster->cores, elem) {
+		cdev = core->dev;
+		fw_name = k3_r5_rproc_get_firmware(cdev);
+		if (IS_ERR(fw_name)) {
+			ret = PTR_ERR(fw_name);
+			goto out;
+		}
+
+		rproc = rproc_alloc(cdev, dev_name(cdev), &k3_r5_rproc_ops,
+				    fw_name, sizeof(*kproc));
+		if (!rproc) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		/* K3 R5s have a Region Address Translator (RAT) but no MMU */
+		rproc->has_iommu = false;
+		/* error recovery is not supported at present */
+		rproc->recovery_disabled = true;
+
+		kproc = rproc->priv;
+		kproc->cluster = cluster;
+		kproc->core = core;
+		kproc->dev = cdev;
+		kproc->rproc = rproc;
+		core->rproc = rproc;
+
+		ret = k3_r5_rproc_configure(kproc);
+		if (ret) {
+			dev_err(dev, "initial configure failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = k3_r5_reserved_mem_init(kproc);
+		if (ret) {
+			dev_err(dev, "reserved memory init failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = rproc_add(rproc);
+		if (ret) {
+			dev_err(dev, "rproc_add failed, ret = %d\n", ret);
+			goto err_add;
+		}
+
+		/* create only one rproc in lockstep mode */
+		if (cluster->mode)
+			break;
+	}
+
+	return 0;
+
+err_split:
+	rproc_del(rproc);
+err_add:
+	k3_r5_reserved_mem_exit(kproc);
+err_config:
+	rproc_free(rproc);
+	core->rproc = NULL;
+out:
+	/* undo core0 upon any failures on core1 in split-mode */
+	if (!cluster->mode && core == core1) {
+		core = list_prev_entry(core, elem);
+		rproc = core->rproc;
+		kproc = rproc->priv;
+		goto err_split;
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_rproc_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core;
+	struct rproc *rproc;
+
+	/*
+	 * lockstep mode has only one rproc associated with first core, whereas
+	 * split-mode has two rprocs associated with each core, and requires
+	 * that core1 be powered down first
+	 */
+	core = cluster->mode ?
+		list_first_entry(&cluster->cores, struct k3_r5_core, elem) :
+		list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		rproc = core->rproc;
+		kproc = rproc->priv;
+
+		rproc_del(rproc);
+
+		k3_r5_reserved_mem_exit(kproc);
+
+		rproc_free(rproc);
+		core->rproc = NULL;
+	}
+
+	return 0;
+}
+
+static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
+					       struct k3_r5_core *core)
+{
+	static const char * const mem_names[] = {"atcm", "btcm"};
+	struct device *dev = &pdev->dev;
+	struct resource *res;
+	int num_mems;
+	int i, ret;
+
+	num_mems = ARRAY_SIZE(mem_names);
+	core->mem = devm_kcalloc(dev, num_mems, sizeof(*core->mem), GFP_KERNEL);
+	if (!core->mem)
+		return -ENOMEM;
+
+	for (i = 0; i < num_mems; i++) {
+		res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						   mem_names[i]);
+		if (!res) {
+			dev_err(dev, "found no memory resource for %s\n",
+				mem_names[i]);
+			ret = -EINVAL;
+			goto fail;
+		}
+		if (!devm_request_mem_region(dev, res->start,
+					     resource_size(res),
+					     dev_name(dev))) {
+			dev_err(dev, "could not request %s region for resource\n",
+				mem_names[i]);
+			ret = -EBUSY;
+			goto fail;
+		}
+
+		/*
+		 * TCMs are designed in general to support RAM-like backing
+		 * memories. So, map these as Normal Non-Cached memories. This
+		 * also avoids/fixes any potential alignment faults due to
+		 * unaligned data accesses when using memcpy() or memset()
+		 * functions (normally seen with device type memory).
+		 */
+		core->mem[i].cpu_addr = devm_ioremap_wc(dev, res->start,
+							resource_size(res));
+		if (IS_ERR(core->mem[i].cpu_addr)) {
+			dev_err(dev, "failed to map %s memory\n", mem_names[i]);
+			ret = PTR_ERR(core->mem[i].cpu_addr);
+			devm_release_mem_region(dev, res->start,
+						resource_size(res));
+			goto fail;
+		}
+		core->mem[i].bus_addr = res->start;
+
+		/*
+		 * TODO:
+		 * The R5F cores can place ATCM & BTCM anywhere in its address
+		 * based on the corresponding Region Registers in the System
+		 * Control coprocessor. For now, place ATCM and BTCM at
+		 * addresses 0 and 0x41010000 (same as the bus address on AM65x
+		 * SoCs) based on loczrama setting
+		 */
+		if (!strcmp(mem_names[i], "atcm")) {
+			core->mem[i].dev_addr = core->loczrama ?
+							0 : K3_R5_TCM_DEV_ADDR;
+		} else {
+			core->mem[i].dev_addr = core->loczrama ?
+							K3_R5_TCM_DEV_ADDR : 0;
+		}
+		core->mem[i].size = resource_size(res);
+
+		dev_dbg(dev, "memory %8s: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			mem_names[i], &core->mem[i].bus_addr,
+			core->mem[i].size, core->mem[i].cpu_addr,
+			core->mem[i].dev_addr);
+	}
+	core->num_mems = num_mems;
+
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+	return ret;
+}
+
+static
+struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
+					  const struct ti_sci_handle *sci)
+{
+	struct ti_sci_proc *tsp;
+	u32 temp[2];
+	int ret;
+
+	ret = of_property_read_u32_array(dev->of_node, "ti,sci-proc-ids",
+					 temp, 2);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	tsp = kzalloc(sizeof(*tsp), GFP_KERNEL);
+	if (!tsp)
+		return ERR_PTR(-ENOMEM);
+
+	tsp->dev = dev;
+	tsp->sci = sci;
+	tsp->ops = &sci->ops.proc_ops;
+	tsp->proc_id = temp[0];
+	tsp->host_id = temp[1];
+
+	return tsp;
+}
+
+static int k3_r5_core_of_init(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_core *core;
+	int ret, ret1;
+
+	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
+	if (!core)
+		return -ENOMEM;
+
+	core->dev = dev;
+	core->atcm_enable = 0;
+	core->btcm_enable = 1;
+	core->loczrama = 1;
+
+	ret = of_property_read_u32(np, "atcm-enable", &core->atcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for atcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "btcm-enable", &core->btcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for btcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "loczrama", &core->loczrama);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for loczrama, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	core->ti_sci = ti_sci_get_by_phandle(np, "ti,sci");
+	if (IS_ERR(core->ti_sci)) {
+		ret = PTR_ERR(core->ti_sci);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get ti-sci handle, ret = %d\n",
+				ret);
+		}
+		core->ti_sci = NULL;
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "ti,sci-dev-id", &core->ti_sci_id);
+	if (ret) {
+		dev_err(dev, "missing 'ti,sci-dev-id' property\n");
+		goto err_sci_id;
+	}
+
+	core->reset = reset_control_get_exclusive(dev, NULL);
+	if (IS_ERR(core->reset)) {
+		ret = PTR_ERR(core->reset);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get reset handle, ret = %d\n",
+				ret);
+		}
+		goto err_sci_id;
+	}
+
+	core->tsp = k3_r5_core_of_get_tsp(dev, core->ti_sci);
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1101

Quoted 5 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +5 -0
+	if (IS_ERR(core->tsp)) {
+		dev_err(dev, "failed to construct ti-sci proc control, ret = %d\n",
+			ret);
+		ret = PTR_ERR(core->tsp);
+		goto err_sci_proc;

Why not calling the label "err_reset"? I don't see what how "err_sci_proc" relates to what the code is doing...

Hmm, I have been using the err labels based on what failed, and this has nothing to do with err_reset, which ideally would have been used above, but I didn't need any specific cleanup on that. tsp is short for ti_sci_proc, as I fill in all the relevant fields to talk to the centralized System Firmware.

Quoted 2 lines

I'll continue reviewing this patch tomorrow.

Thanks for the reviews.

regards
Suman

Quoted 260 lines

Thanks,
Mathieu

Quoted 100 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +100 -0
+	}
+
+	ret = ti_sci_proc_request(core->tsp);
+	if (ret < 0) {
+		dev_err(dev, "ti_sci_proc_request failed, ret = %d\n", ret);
+		goto err_proc;
+	}
+
+	ret = k3_r5_core_of_get_internal_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get internal memories, ret = %d\n",
+			ret);
+		goto err_intmem;
+	}
+
+	platform_set_drvdata(pdev, core);
+
+	return 0;
+
+err_intmem:
+	ret1 = ti_sci_proc_release(core->tsp);
+	if (ret1)
+		dev_err(dev, "failed to release proc, ret1 = %d\n", ret1);
+err_proc:
+	kfree(core->tsp);
+err_sci_proc:
+	reset_control_put(core->reset);
+err_sci_id:
+	ret1 = ti_sci_put_handle(core->ti_sci);
+	if (ret1)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret1);
+err_of:
+	devm_kfree(dev, core);
+	return ret;
+}
+
+/*
+ * free the resources explicitly since driver model is not being used
+ * for the child R5F devices
+ */
+static int k3_r5_core_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_core *core = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	int i, ret;
+
+	for (i = 0; i < core->num_mems; i++) {
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+
+	ret = ti_sci_proc_release(core->tsp);
+	if (ret)
+		dev_err(dev, "failed to release proc, ret = %d\n", ret);
+
+	kfree(core->tsp);
+	reset_control_put(core->reset);
+
+	ret = ti_sci_put_handle(core->ti_sci);
+	if (ret)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret);
+
+	platform_set_drvdata(pdev, NULL);
+	devm_kfree(dev, core);
+
+	return ret;
+}
+
+static int k3_r5_cluster_of_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct platform_device *cpdev;
+	struct device_node *child;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	for_each_available_child_of_node(np, child) {
+		cpdev = of_find_device_by_node(child);
+		if (!cpdev) {
+			ret = -ENODEV;
+			dev_err(dev, "could not get R5 core platform device\n");
+			goto fail;
+		}
+
+		ret = k3_r5_core_of_init(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_init failed, ret = %d\n",
+				ret);
+			put_device(&cpdev->dev);
+			goto fail;
+		}
+
+		core = platform_get_drvdata(cpdev);
+		put_device(&cpdev->dev);
+		list_add_tail(&core->elem, &cluster->cores);

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1211

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	}
+
+	return 0;
+
+fail:
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		if (k3_r5_core_of_exit(cpdev))
+			dev_err(dev, "k3_r5_core_of_exit cleanup failed\n");

I would move k3_r5_cluster_of_exit() just above k3_r5_cluster_of_init() and reuse the former here instead of repeating the same code.

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct platform_device *cpdev;
+	struct k3_r5_core *core, *temp;

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1231

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	int ret;
+
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		ret = k3_r5_core_of_exit(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_exit failed, ret = %d\n",
+				ret);
+			break;

So why stopping when encountering an error? Since this is the exit path I think it is better to call k3_r5_core_of_exit() on as many core as possible.

Quoted 118 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +115 -0
+		}
+	}
+
+	return ret;
+}
+
+static int k3_r5_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_cluster *cluster;
+	int ret, ret1;
+	int num_cores;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+
+	cluster->dev = dev;
+	cluster->mode = CLUSTER_MODE_LOCKSTEP;
+	INIT_LIST_HEAD(&cluster->cores);
+
+	ret = of_property_read_u32(np, "lockstep-mode", &cluster->mode);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for lockstep-mode, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_cores = of_get_available_child_count(np);
+	if (num_cores != 2) {
+		dev_err(dev, "MCU cluster requires both R5F cores to be enabled, num_cores = %d\n",
+			num_cores);
+		return -ENODEV;
+	}
+
+	platform_set_drvdata(pdev, cluster);
+
+	dev_dbg(dev, "creating child devices for R5F cores\n");
+	ret = of_platform_populate(np, NULL, NULL, dev);
+	if (ret) {
+		dev_err(dev, "of_platform_populate failed, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = k3_r5_cluster_of_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_init failed, ret = %d\n", ret);
+		goto fail_of;
+	}
+
+	ret = k3_r5_cluster_rproc_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_init failed, ret = %d\n",
+			ret);
+		goto fail_rproc;
+	}
+
+	return 0;
+
+fail_rproc:
+	ret1 = k3_r5_cluster_of_exit(pdev);
+	if (ret1)
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret1);
+fail_of:
+	of_platform_depopulate(dev);
+	return ret;
+}
+
+static int k3_r5_remove(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	ret = k3_r5_cluster_rproc_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_exit failed, ret = %d\n",
+			ret);
+		goto fail;
+	}
+
+	ret = k3_r5_cluster_of_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret);
+		goto fail;
+	}
+
+	dev_dbg(dev, "removing child devices for R5F cores\n");
+	of_platform_depopulate(dev);
+
+fail:
+	return ret;
+}
+
+static const struct of_device_id k3_r5_of_match[] = {
+	{ .compatible = "ti,am654-r5fss", },
+	{ .compatible = "ti,j721e-r5fss", },
+	{ /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, k3_r5_of_match);
+
+static struct platform_driver k3_r5_rproc_driver = {
+	.probe = k3_r5_probe,
+	.remove = k3_r5_remove,
+	.driver = {
+		.name = "k3_r5_rproc",
+		.of_match_table = k3_r5_of_match,
+	},
+};
+
+module_platform_driver(k3_r5_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("TI K3 R5F remote processor driver");
+MODULE_AUTHOR("Suman Anna <[email protected]>");
Signature · 1 line
2.23.0
maintainer
#22 lore raw

Re: [PATCH 5/7] remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

In reply to Suman Anna
Quoted 733 lines · Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode. The driver uses various TI-SCI interfaces to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
 drivers/remoteproc/Kconfig               |   16 +
 drivers/remoteproc/Makefile              |    1 +
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
 3 files changed, 1363 insertions(+)
 create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c
drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
 	  This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+	tristate "TI K3 R5 remoteproc support"
+	depends on ARCH_K3
+	select MAILBOX
+	select OMAP2PLUS_MBOX
+	help
+	  Say y here to support TI's R5F remote processor subsystems
+	  on various TI K3 family of SoCs through the remote processor
+	  framework.
+
+	  You want to say y here in order to offload some processing
+	  tasks to these processors
+
+	  It's safe to say N here if you're not interested in utilizing
+	  a slave processor
+
 endif # REMOTEPROC
 
 endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y			+= qcom_wcnss_iris.o
 obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
 obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
 obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)	+= ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +619 -0
new file mode 100644
index 000000000000..655f8f14c37d
@@ -0,0 +1,1346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TI K3 R5F (MCU) Remote Processor driver
+ *
+ * Copyright (C) 2017-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *	Suman Anna <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/omap-mailbox.h>
+#include <linux/reset.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+#include "omap_remoteproc.h"
+#include "remoteproc_internal.h"
+#include "ti_sci_proc.h"
+
+/* This address can either be for ATCM or BTCM with the other at address 0x0 */
+#define K3_R5_TCM_DEV_ADDR	0x41010000
+
+/* R5 TI-SCI Processor Configuration Flags */
+#define PROC_BOOT_CFG_FLAG_R5_DBG_EN			0x00000001
+#define PROC_BOOT_CFG_FLAG_R5_DBG_NIDEN			0x00000002
+#define PROC_BOOT_CFG_FLAG_R5_LOCKSTEP			0x00000100
+#define PROC_BOOT_CFG_FLAG_R5_TEINIT			0x00000200
+#define PROC_BOOT_CFG_FLAG_R5_NMFI_EN			0x00000400
+#define PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE		0x00000800
+#define PROC_BOOT_CFG_FLAG_R5_BTCM_EN			0x00001000
+#define PROC_BOOT_CFG_FLAG_R5_ATCM_EN			0x00002000
+
+/* R5 TI-SCI Processor Control Flags */
+#define PROC_BOOT_CTRL_FLAG_R5_CORE_HALT		0x00000001
+
+/* R5 TI-SCI Processor Status Flags */
+#define PROC_BOOT_STATUS_FLAG_R5_WFE			0x00000001
+#define PROC_BOOT_STATUS_FLAG_R5_WFI			0x00000002
+#define PROC_BOOT_STATUS_FLAG_R5_CLK_GATED		0x00000004
+#define PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED	0x00000100
+
+/**
+ * struct k3_r5_mem - internal memory structure
+ * @cpu_addr: MPU virtual address of the memory region
+ * @bus_addr: Bus address used to access the memory region
+ * @dev_addr: Device address from remoteproc view
+ * @size: Size of the memory region
+ */
+struct k3_r5_mem {
+	void __iomem *cpu_addr;
+	phys_addr_t bus_addr;
+	u32 dev_addr;
+	size_t size;
+};
+
+enum cluster_mode {
+	CLUSTER_MODE_SPLIT = 0,
+	CLUSTER_MODE_LOCKSTEP,
+};
+
+/**
+ * struct k3_r5_cluster - K3 R5F Cluster structure
+ * @dev: cached device pointer
+ * @mode: Mode to configure the Cluster - Split or LockStep
+ * @cores: list of R5 cores within the cluster
+ */
+struct k3_r5_cluster {
+	struct device *dev;
+	enum cluster_mode mode;
+	struct list_head cores;
+};
+
+/**
+ * struct k3_r5_core - K3 R5 core structure
+ * @elem: linked list item
+ * @dev: cached device pointer
+ * @rproc: rproc handle representing this core
+ * @mem: internal memory regions data
+ * @num_mems: number of internal memory regions
+ * @reset: reset control handle
+ * @tsp: TI-SCI processor control handle
+ * @ti_sci: TI-SCI handle
+ * @ti_sci_id: TI-SCI device identifier
+ * @atcm_enable: flag to control ATCM enablement
+ * @btcm_enable: flag to control BTCM enablement
+ * @loczrama: flag to dictate which TCM is at device address 0x0
+ */
+struct k3_r5_core {
+	struct list_head elem;
+	struct device *dev;
+	struct rproc *rproc;
+	struct k3_r5_mem *mem;
+	int num_mems;
+	struct reset_control *reset;
+	struct ti_sci_proc *tsp;
+	const struct ti_sci_handle *ti_sci;
+	u32 ti_sci_id;
+	u32 atcm_enable;
+	u32 btcm_enable;
+	u32 loczrama;
+};
+
+/**
+ * struct k3_r5_rproc - K3 remote processor state
+ * @dev: cached device pointer
+ * @cluster: cached pointer to parent cluster structure
+ * @mbox: mailbox channel handle
+ * @client: mailbox client to request the mailbox channel
+ * @rproc: rproc handle
+ * @core: cached pointer to r5 core structure being used
+ * @rmem: reserved memory regions data
+ * @num_rmems: number of reserved memory regions
+ */
+struct k3_r5_rproc {
+	struct device *dev;
+	struct k3_r5_cluster *cluster;
+	struct mbox_chan *mbox;
+	struct mbox_client client;
+	struct rproc *rproc;
+	struct k3_r5_core *core;
+	struct k3_r5_mem *rmem;
+	int num_rmems;
+};
+
+/**
+ * k3_r5_rproc_mbox_callback() - inbound mailbox message handler
+ * @client: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by the OMAP mailbox driver whenever a mailbox
+ * message is received. Usually, the mailbox payload simply contains
+ * the index of the virtqueue that is kicked by the remote processor,
+ * and we let remoteproc core handle it.
+ *
+ * In addition to virtqueue indices, we also have some out-of-band values
+ * that indicate different events. Those values are deliberately very
+ * large so they don't coincide with virtqueue indices.
+ */
+static void k3_r5_rproc_mbox_callback(struct mbox_client *client, void *data)
+{
+	struct k3_r5_rproc *kproc = container_of(client, struct k3_r5_rproc,
+						client);
+	struct device *dev = kproc->rproc->dev.parent;
+	const char *name = kproc->rproc->name;
+	u32 msg = omap_mbox_message(data);
+
+	dev_dbg(dev, "mbox msg: 0x%x\n", msg);
+
+	switch (msg) {
+	case RP_MBOX_CRASH:
+		/*
+		 * remoteproc detected an exception, but error recovery is not
+		 * supported. So, just log this for now
+		 */
+		dev_err(dev, "K3 R5F rproc %s crashed\n", name);
+		break;
+	case RP_MBOX_ECHO_REPLY:
+		dev_info(dev, "received echo reply from %s\n", name);
+		break;
+	default:
+		/* silently handle all other valid messages */
+		if (msg >= RP_MBOX_READY && msg < RP_MBOX_END_MSG)
+			return;
+		if (msg > kproc->rproc->max_notifyid) {
+			dev_dbg(dev, "dropping unknown message 0x%x", msg);
+			return;
+		}
+		/* msg contains the index of the triggered vring */
+		if (rproc_vq_interrupt(kproc->rproc, msg) == IRQ_NONE)
+			dev_dbg(dev, "no message was found in vqid %d\n", msg);
+	}
+}
+
+/* kick a virtqueue */
+static void k3_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	mbox_msg_t msg = (mbox_msg_t)vqid;
+	int ret;
+
+	/* send the index of the triggered virtqueue in the mailbox payload */
+	ret = mbox_send_message(kproc->mbox, (void *)msg);
+	if (ret < 0)
+		dev_err(dev, "failed to send mailbox message, status = %d\n",
+			ret);
+}
+
+static int k3_r5_split_reset(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = reset_control_assert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+			ret);
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_split_release(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = reset_control_deassert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset deassert failed, ret = %d\n",
+			ret);
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_reset(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* assert local reset on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = reset_control_assert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+				ret);
+			core = list_prev_entry(core, elem);
+			goto unroll_local_reset;
+		}
+	}
+
+	/* disable PSC modules on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+				ret);
+			goto unroll_module_reset;
+		}
+	}
+
+	return 0;
+
+unroll_module_reset:
+	list_for_each_entry_continue_reverse(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+	core = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_local_reset:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_release(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* enable PSC modules on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			core = list_next_entry(core, elem);
+			goto unroll_module_reset;
+		}
+	}
+
+	/* deassert local reset on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = reset_control_deassert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			goto unroll_local_reset;
+		}
+	}
+
+	return 0;
+
+unroll_local_reset:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (reset_control_assert(core->reset))
+			dev_warn(core->dev, "local-reset assert back failed\n");
+	}
+	core = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_module_reset:
+	list_for_each_entry_from(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static inline int k3_r5_core_halt(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       PROC_BOOT_CTRL_FLAG_R5_CORE_HALT, 0);
+}
+
+static inline int k3_r5_core_run(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       0, PROC_BOOT_CTRL_FLAG_R5_CORE_HALT);
+}
+
+/*
+ * The R5F cores have controls for both a reset and a halt/run. The code
+ * execution from DDR requires the initial boot-strapping code to be run
+ * from the internal TCMs. This function is used to release the resets on
+ * applicable cores to allow loading into the TCMs. The .prepare() ops is
+ * invoked by remoteproc core before any firmware loading, and is followed
+ * by the .start() ops after loading to actually let the R5 cores run.
+ */
+static int k3_r5_rproc_prepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_release(cluster) :
+			      k3_r5_split_release(core);
+	if (ret)
+		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
+			ret);
+
+	return ret;
+}
+
+/*
+ * This function implements the .unprepare() ops and performs the complimentary
+ * operations to that of the .prepare() ops. The function is used to assert the
+ * resets on all applicable cores for the rproc device (depending on LockStep
+ * or Split mode). This completes the second portion of powering down the R5F
+ * cores. The cores themselves are only halted in the .stop() ops, and the
+ * .unprepare() ops is invoked by the remoteproc core after the remoteproc is
+ * stopped.
+ */
+static int k3_r5_rproc_unprepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_reset(cluster) :
+			      k3_r5_split_reset(core);
+	if (ret)
+		dev_err(dev, "unable to disable cores, ret = %d\n", ret);
+
+	return ret;
+}
+
+/*
+ * The R5F start sequence includes two different operations
+ * 1. Configure the boot vector for R5F core(s)
+ * 2. Unhalt/Run the R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes. The LockStep
+ * mode requires the boot vector to be configured only for Core0, and then
+ * unhalt both the cores to start the execution - Core1 needs to be unhalted
+ * first followed by Core0. The Split-mode requires that Core0 to be maintained
+ * always in a higher power state that Core1 (implying Core1 needs to be started
+ * always only after Core0 is started).
+ */
+static int k3_r5_rproc_start(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct mbox_client *client = &kproc->client;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core;
+	u32 boot_addr;
+	int ret;
+
+	client->dev = dev;
+	client->tx_done = NULL;
+	client->rx_callback = k3_r5_rproc_mbox_callback;
+	client->tx_block = false;
+	client->knows_txdone = false;
+
+	kproc->mbox = mbox_request_channel(client, 0);
+	if (IS_ERR(kproc->mbox)) {
+		ret = -EBUSY;
+		dev_err(dev, "mbox_request_channel failed: %ld\n",
+			PTR_ERR(kproc->mbox));
+		return ret;
+	}
+
+	/*
+	 * Ping the remote processor, this is only for sanity-sake for now;
+	 * there is no functional effect whatsoever.
+	 *
+	 * Note that the reply will _not_ arrive immediately: this message
+	 * will wait in the mailbox fifo until the remote processor is booted.
+	 */
+	ret = mbox_send_message(kproc->mbox, (void *)RP_MBOX_ECHO_REQUEST);
+	if (ret < 0) {
+		dev_err(dev, "mbox_send_message failed: %d\n", ret);
+		goto put_mbox;
+	}
+
+	boot_addr = rproc->bootaddr;
+	/* TODO: add boot_addr sanity checking */
+	dev_err(dev, "booting R5F core using boot addr = 0x%x\n", boot_addr);
+
+	/* boot vector need not be programmed for Core1 in LockStep mode */
+	core = kproc->core;
+	ret = ti_sci_proc_set_config(core->tsp, boot_addr, 0, 0);
+	if (ret)
+		goto put_mbox;
+
+	/* unhalt/run all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry_reverse(core, &cluster->cores, elem) {
+			ret = k3_r5_core_run(core);
+			if (ret)
+				goto unroll_core_run;
+		}
+	} else {
+		ret = k3_r5_core_run(core);
+		if (ret)
+			goto put_mbox;
+	}
+
+	return 0;
+
+unroll_core_run:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (k3_r5_core_halt(core))
+			dev_warn(core->dev, "core halt back failed\n");
+	}
+put_mbox:
+	mbox_free_channel(kproc->mbox);
+	return ret;
+}
+
+/*
+ * The R5F stop function includes the following operations
+ * 1. Halt R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes, and the order
+ * of cores the operations are performed are also in general reverse to that
+ * of the start function. The LockStep mode requires each operation to be
+ * performed first on Core0 followed by Core1. The Split-mode requires that
+ * Core0 to be maintained always in a higher power state that Core1 (implying
+ * Core1 needs to be stopped first before Core0).
+ *
+ * Note that the R5F halt operation in general is not effective when the R5F
+ * core is running, but is needed to make sure the core won't run after
+ * deasserting the reset the subsequent time. The asserting of reset can
+ * be done here, but is preferred to be done in the .unprepare() ops - this
+ * maintains the symmetric behavior between the .start(), .stop(), .prepare()
+ * and .unprepare() ops, and also balances them well between sysfs 'state'
+ * flow and device bind/unbind or module removal.
+ */
+static int k3_r5_rproc_stop(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	int ret;
+
+	/* halt all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry(core, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret) {
+				core = list_prev_entry(core, elem);
+				goto unroll_core_halt;
+			}
+		}
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+	}
+
+	mbox_free_channel(kproc->mbox);
+
+	return 0;
+
+unroll_core_halt:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (k3_r5_core_run(core))
+			dev_warn(core->dev, "core run back failed\n");
+	}
+out:
+	return ret;
+}
+
+/*
+ * Internal Memory translation helper
+ *
+ * Custom function implementing the rproc .da_to_va ops to provide address
+ * translation (device address to kernel virtual address) for internal RAMs
+ * present in a DSP or IPU device). The translated addresses can be used
+ * either by the remoteproc core for loading, or by any rpmsg bus drivers.
+ */
+static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_core *core = kproc->core;
+	void __iomem *va = NULL;
+	phys_addr_t bus_addr;
+	u32 dev_addr, offset;
+	size_t size;
+	int i;
+
+	if (len == 0)
+		return NULL;
+
+	/* handle both R5 and SoC views of ATCM and BTCM */
+	for (i = 0; i < core->num_mems; i++) {
+		bus_addr = core->mem[i].bus_addr;
+		dev_addr = core->mem[i].dev_addr;
+		size = core->mem[i].size;
+
+		/* handle R5-view addresses of TCMs */
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+
+		/* handle SoC-view addresses of TCMs */
+		if (da >= bus_addr && ((da + len) <= (bus_addr + size))) {
+			offset = da - bus_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	/* handle static DDR reserved memory regions */
+	for (i = 0; i < kproc->num_rmems; i++) {
+		dev_addr = kproc->rmem[i].dev_addr;
+		size = kproc->rmem[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = kproc->rmem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	return NULL;
+}
+
+static const struct rproc_ops k3_r5_rproc_ops = {
+	.prepare	= k3_r5_rproc_prepare,
+	.unprepare	= k3_r5_rproc_unprepare,
+	.start		= k3_r5_rproc_start,
+	.stop		= k3_r5_rproc_stop,
+	.kick		= k3_r5_rproc_kick,
+	.da_to_va	= k3_r5_rproc_da_to_va,
+};
+
+static const char *k3_r5_rproc_get_firmware(struct device *dev)
+{
+	const char *fw_name;
+	int ret;
+
+	ret = of_property_read_string(dev->of_node, "firmware-name",
+				      &fw_name);
+	if (ret) {
+		dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+			ret);
+		return ERR_PTR(ret);
+	}
+
+	return fw_name;
+}
+
+static int k3_r5_rproc_configure(struct k3_r5_rproc *kproc)
+{
+	struct k3_r5_cluster *cluster = kproc->cluster;

drivers/remoteproc/ti_k3_r5_remoteproc.c line 629

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core0, *core, *temp;
+	u32 ctrl = 0, cfg = 0, stat = 0;
+	u32 set_cfg = 0, clr_cfg = 0;
+	u64 boot_vec = 0;
+	bool lockstep_en;
+	int ret;
+
+	core0 = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+	core = cluster->mode ? core0 : kproc->core;

The above two lines generated interesting mental gymnastic - please sprinkle with comments in order to disambiguate what is going on.

Quoted 41 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +41 -0
+
+	ret = ti_sci_proc_get_status(core->tsp, &boot_vec, &cfg, &ctrl,
+				     &stat);
+	if (ret < 0)
+		return ret;
+
+	dev_dbg(dev, "boot_vector = 0x%llx, cfg = 0x%x ctrl = 0x%x stat = 0x%x\n",
+		boot_vec, cfg, ctrl, stat);
+
+	lockstep_en = !!(stat & PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED);
+	if (!lockstep_en && cluster->mode) {
+		dev_err(cluster->dev, "lockstep mode not permitted, force configuring for split-mode\n");
+		cluster->mode = 0;
+	}
+
+	/* always enable ARM mode and set boot vector to 0 */
+	boot_vec = 0x0;
+	if (core == core0) {
+		clr_cfg = PROC_BOOT_CFG_FLAG_R5_TEINIT;
+		/*
+		 * LockStep configuration bit is Read-only on Split-mode _only_
+		 * devices and system firmware will NACK any requests with the
+		 * bit configured, so program it only on permitted devices
+		 */
+		if (lockstep_en)
+			clr_cfg |= PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+	}
+
+	if (core->atcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+
+	if (core->btcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+
+	if (core->loczrama)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+	else

drivers/remoteproc/ti_k3_r5_remoteproc.c line 680

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+
+	if (cluster->mode) {
+		/*
+		 * work around system firmware limitations to make sure both
+		 * cores are programmed symmetrically in LockStep. LockStep
+		 * and TEINIT config is only allowed with Core0.
+		 */
+		list_for_each_entry(temp, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);

When I first read this I thought this was an error and what was really needed was k3_r5_core_halt(temp)... But no, this is correct because k3_r5_rproc_configure() is called for each core in the cluster in function k3_r5_cluster_rproc_init(). But then again why halting the same for each of the cores found in the system?

So something seems wrong here. Either call k3_r5_core_halt(temp) or move k3_r5_core_halt(core) outside of the if (cluster->mode) to avoid more confusion.

Quoted 177 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +177 -0
+			if (ret)
+				goto out;
+
+			if (temp != core) {
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_TEINIT;
+			}
+			ret = ti_sci_proc_set_config(temp->tsp, boot_vec,
+						     set_cfg, clr_cfg);
+			if (ret)
+				goto out;
+		}
+
+		set_cfg = PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+		clr_cfg = 0;
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	}
+
+out:
+	return ret;
+}
+
+static int k3_r5_reserved_mem_init(struct k3_r5_rproc *kproc)
+{
+	struct device *dev = kproc->dev;
+	struct device_node *np = dev->of_node;
+	struct device_node *rmem_np;
+	struct reserved_mem *rmem;
+	int num_rmems;
+	int ret, i;
+
+	num_rmems = of_property_count_elems_of_size(np, "memory-region",
+						    sizeof(phandle));
+	if (num_rmems <= 0) {
+		dev_err(dev, "device does not have reserved memory regions, ret = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+	if (num_rmems < 2) {
+		dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+
+	/* use reserved memory region 0 for vring DMA allocations */
+	ret = of_reserved_mem_device_init_by_idx(dev, np, 0);
+	if (ret) {
+		dev_err(dev, "device cannot initialize DMA pool, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_rmems--;
+	kproc->rmem = kcalloc(num_rmems, sizeof(*kproc->rmem), GFP_KERNEL);
+	if (!kproc->rmem) {
+		ret = -ENOMEM;
+		goto release_rmem;
+	}
+
+	/* use remaining reserved memory regions for static carveouts */
+	for (i = 0; i < num_rmems; i++) {
+		rmem_np = of_parse_phandle(np, "memory-region", i + 1);
+		if (!rmem_np) {
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+
+		rmem = of_reserved_mem_lookup(rmem_np);
+		if (!rmem) {
+			of_node_put(rmem_np);
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+		of_node_put(rmem_np);
+
+		kproc->rmem[i].bus_addr = rmem->base;
+		/* 64-bit address regions currently not supported */
+		kproc->rmem[i].dev_addr = (u32)rmem->base;
+		kproc->rmem[i].size = rmem->size;
+		kproc->rmem[i].cpu_addr = ioremap_wc(rmem->base, rmem->size);
+		if (!kproc->rmem[i].cpu_addr) {
+			dev_err(dev, "failed to map reserved memory#%d at %pa of size %pa\n",
+				i + 1, &rmem->base, &rmem->size);
+			ret = -ENOMEM;
+			goto unmap_rmem;
+		}
+
+		dev_dbg(dev, "reserved memory%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			i + 1, &kproc->rmem[i].bus_addr,
+			kproc->rmem[i].size, kproc->rmem[i].cpu_addr,
+			kproc->rmem[i].dev_addr);
+	}
+	kproc->num_rmems = num_rmems;
+
+	return 0;
+
+unmap_rmem:
+	for (i--; i >= 0; i--) {
+		if (kproc->rmem[i].cpu_addr)
+			iounmap(kproc->rmem[i].cpu_addr);
+	}
+	kfree(kproc->rmem);
+release_rmem:
+	of_reserved_mem_device_release(dev);
+	return ret;
+}
+
+static void k3_r5_reserved_mem_exit(struct k3_r5_rproc *kproc)
+{
+	int i;
+
+	for (i = 0; i < kproc->num_rmems; i++)
+		iounmap(kproc->rmem[i].cpu_addr);
+	kfree(kproc->rmem);
+
+	of_reserved_mem_device_release(kproc->dev);
+}
+
+static int k3_r5_cluster_rproc_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core, *core1;
+	struct device *cdev;
+	const char *fw_name;
+	struct rproc *rproc;
+	int ret;
+
+	core1 = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+	list_for_each_entry(core, &cluster->cores, elem) {
+		cdev = core->dev;
+		fw_name = k3_r5_rproc_get_firmware(cdev);
+		if (IS_ERR(fw_name)) {
+			ret = PTR_ERR(fw_name);
+			goto out;
+		}
+
+		rproc = rproc_alloc(cdev, dev_name(cdev), &k3_r5_rproc_ops,
+				    fw_name, sizeof(*kproc));
+		if (!rproc) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		/* K3 R5s have a Region Address Translator (RAT) but no MMU */
+		rproc->has_iommu = false;
+		/* error recovery is not supported at present */
+		rproc->recovery_disabled = true;
+
+		kproc = rproc->priv;
+		kproc->cluster = cluster;
+		kproc->core = core;
+		kproc->dev = cdev;
+		kproc->rproc = rproc;
+		core->rproc = rproc;
+
+		ret = k3_r5_rproc_configure(kproc);
+		if (ret) {
+			dev_err(dev, "initial configure failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = k3_r5_reserved_mem_init(kproc);
+		if (ret) {
+			dev_err(dev, "reserved memory init failed, ret = %d\n",
+				ret);
+			goto err_config;

drivers/remoteproc/ti_k3_r5_remoteproc.c line 867

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+		}
+
+		ret = rproc_add(rproc);
+		if (ret) {
+			dev_err(dev, "rproc_add failed, ret = %d\n", ret);
+			goto err_add;
+		}
+
+		/* create only one rproc in lockstep mode */
+		if (cluster->mode)

Here and throughout the file, please use the cluster mode enumeration in order to improve readability, i.e

                if (cluster->mode == CLUSTER_MODE_LOCKSTEP)
Quoted 128 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +128 -0
+			break;
+	}
+
+	return 0;
+
+err_split:
+	rproc_del(rproc);
+err_add:
+	k3_r5_reserved_mem_exit(kproc);
+err_config:
+	rproc_free(rproc);
+	core->rproc = NULL;
+out:
+	/* undo core0 upon any failures on core1 in split-mode */
+	if (!cluster->mode && core == core1) {
+		core = list_prev_entry(core, elem);
+		rproc = core->rproc;
+		kproc = rproc->priv;
+		goto err_split;
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_rproc_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core;
+	struct rproc *rproc;
+
+	/*
+	 * lockstep mode has only one rproc associated with first core, whereas
+	 * split-mode has two rprocs associated with each core, and requires
+	 * that core1 be powered down first
+	 */
+	core = cluster->mode ?
+		list_first_entry(&cluster->cores, struct k3_r5_core, elem) :
+		list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		rproc = core->rproc;
+		kproc = rproc->priv;
+
+		rproc_del(rproc);
+
+		k3_r5_reserved_mem_exit(kproc);
+
+		rproc_free(rproc);
+		core->rproc = NULL;
+	}
+
+	return 0;
+}
+
+static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
+					       struct k3_r5_core *core)
+{
+	static const char * const mem_names[] = {"atcm", "btcm"};
+	struct device *dev = &pdev->dev;
+	struct resource *res;
+	int num_mems;
+	int i, ret;
+
+	num_mems = ARRAY_SIZE(mem_names);
+	core->mem = devm_kcalloc(dev, num_mems, sizeof(*core->mem), GFP_KERNEL);
+	if (!core->mem)
+		return -ENOMEM;
+
+	for (i = 0; i < num_mems; i++) {
+		res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						   mem_names[i]);
+		if (!res) {
+			dev_err(dev, "found no memory resource for %s\n",
+				mem_names[i]);
+			ret = -EINVAL;
+			goto fail;
+		}
+		if (!devm_request_mem_region(dev, res->start,
+					     resource_size(res),
+					     dev_name(dev))) {
+			dev_err(dev, "could not request %s region for resource\n",
+				mem_names[i]);
+			ret = -EBUSY;
+			goto fail;
+		}
+
+		/*
+		 * TCMs are designed in general to support RAM-like backing
+		 * memories. So, map these as Normal Non-Cached memories. This
+		 * also avoids/fixes any potential alignment faults due to
+		 * unaligned data accesses when using memcpy() or memset()
+		 * functions (normally seen with device type memory).
+		 */
+		core->mem[i].cpu_addr = devm_ioremap_wc(dev, res->start,
+							resource_size(res));
+		if (IS_ERR(core->mem[i].cpu_addr)) {
+			dev_err(dev, "failed to map %s memory\n", mem_names[i]);
+			ret = PTR_ERR(core->mem[i].cpu_addr);
+			devm_release_mem_region(dev, res->start,
+						resource_size(res));
+			goto fail;
+		}
+		core->mem[i].bus_addr = res->start;
+
+		/*
+		 * TODO:
+		 * The R5F cores can place ATCM & BTCM anywhere in its address
+		 * based on the corresponding Region Registers in the System
+		 * Control coprocessor. For now, place ATCM and BTCM at
+		 * addresses 0 and 0x41010000 (same as the bus address on AM65x
+		 * SoCs) based on loczrama setting
+		 */
+		if (!strcmp(mem_names[i], "atcm")) {
+			core->mem[i].dev_addr = core->loczrama ?
+							0 : K3_R5_TCM_DEV_ADDR;
+		} else {
+			core->mem[i].dev_addr = core->loczrama ?
+							K3_R5_TCM_DEV_ADDR : 0;
+		}
+		core->mem[i].size = resource_size(res);
+
+		dev_dbg(dev, "memory %8s: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			mem_names[i], &core->mem[i].bus_addr,
+			core->mem[i].size, core->mem[i].cpu_addr,
+			core->mem[i].dev_addr);
+	}
+	core->num_mems = num_mems;
+

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1005

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);

Since the devm_ API has been used for memory allocation all this should be called automatically when of_platform_depopulate() is called.

Quoted 7 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +7 -0
+	return ret;
+}
+
+static
+struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
+					  const struct ti_sci_handle *sci)
+{

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1022

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	struct ti_sci_proc *tsp;
+	u32 temp[2];
+	int ret;
+
+	ret = of_property_read_u32_array(dev->of_node, "ti,sci-proc-ids",
+					 temp, 2);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	tsp = kzalloc(sizeof(*tsp), GFP_KERNEL);

Here @dev is available, I would just call devm_kzalloc() and get rid of the kfree() in the error path of k3_r5_core_of_init() and k3_r5_core_of_exit().

Quoted 17 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +17 -0
+	if (!tsp)
+		return ERR_PTR(-ENOMEM);
+
+	tsp->dev = dev;
+	tsp->sci = sci;
+	tsp->ops = &sci->ops.proc_ops;
+	tsp->proc_id = temp[0];
+	tsp->host_id = temp[1];
+
+	return tsp;
+}
+
+static int k3_r5_core_of_init(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_core *core;

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1049

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	int ret, ret1;
+
+	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
+	if (!core)
+		return -ENOMEM;
+
+	core->dev = dev;
+	core->atcm_enable = 0;
+	core->btcm_enable = 1;
+	core->loczrama = 1;

Please add a comment that justifies the selection of default value. Otherwise this looks very esoteric.

Quoted 73 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +73 -0
+
+	ret = of_property_read_u32(np, "atcm-enable", &core->atcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for atcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "btcm-enable", &core->btcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for btcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "loczrama", &core->loczrama);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for loczrama, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	core->ti_sci = ti_sci_get_by_phandle(np, "ti,sci");
+	if (IS_ERR(core->ti_sci)) {
+		ret = PTR_ERR(core->ti_sci);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get ti-sci handle, ret = %d\n",
+				ret);
+		}
+		core->ti_sci = NULL;
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "ti,sci-dev-id", &core->ti_sci_id);
+	if (ret) {
+		dev_err(dev, "missing 'ti,sci-dev-id' property\n");
+		goto err_sci_id;
+	}
+
+	core->reset = reset_control_get_exclusive(dev, NULL);
+	if (IS_ERR(core->reset)) {
+		ret = PTR_ERR(core->reset);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get reset handle, ret = %d\n",
+				ret);
+		}
+		goto err_sci_id;
+	}
+
+	core->tsp = k3_r5_core_of_get_tsp(dev, core->ti_sci);
+	if (IS_ERR(core->tsp)) {
+		dev_err(dev, "failed to construct ti-sci proc control, ret = %d\n",
+			ret);
+		ret = PTR_ERR(core->tsp);
+		goto err_sci_proc;
+	}
+
+	ret = ti_sci_proc_request(core->tsp);
+	if (ret < 0) {
+		dev_err(dev, "ti_sci_proc_request failed, ret = %d\n", ret);
+		goto err_proc;
+	}
+
+	ret = k3_r5_core_of_get_internal_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get internal memories, ret = %d\n",
+			ret);
+		goto err_intmem;
+	}
+
+	platform_set_drvdata(pdev, core);
+
+	return 0;
+
+err_intmem:
+	ret1 = ti_sci_proc_release(core->tsp);

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1132

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	if (ret1)
+		dev_err(dev, "failed to release proc, ret1 = %d\n", ret1);
+err_proc:
+	kfree(core->tsp);
+err_sci_proc:
+	reset_control_put(core->reset);
+err_sci_id:
+	ret1 = ti_sci_put_handle(core->ti_sci);
+	if (ret1)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret1);

s/"ret = %d"/"ret1 = %d"

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1134

Quoted 2 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +2 -0
+err_of:
+	devm_kfree(dev, core);

Same comment as above, this should be called automatically.

Quoted 24 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +24 -0
+	return ret;
+}
+
+/*
+ * free the resources explicitly since driver model is not being used
+ * for the child R5F devices
+ */
+static int k3_r5_core_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_core *core = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	int i, ret;
+
+	for (i = 0; i < core->num_mems; i++) {
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+
+	ret = ti_sci_proc_release(core->tsp);
+	if (ret)
+		dev_err(dev, "failed to release proc, ret = %d\n", ret);

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1168

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+
+	kfree(core->tsp);
+	reset_control_put(core->reset);
+
+	ret = ti_sci_put_handle(core->ti_sci);
+	if (ret)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret);
+
+	platform_set_drvdata(pdev, NULL);
+	devm_kfree(dev, core);

Same comment regarding the devm_ API, this should be called automatically.

I will continue tomorrow,
Mathieu

Quoted 181 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +178 -0
+
+	return ret;
+}
+
+static int k3_r5_cluster_of_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct platform_device *cpdev;
+	struct device_node *child;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	for_each_available_child_of_node(np, child) {
+		cpdev = of_find_device_by_node(child);
+		if (!cpdev) {
+			ret = -ENODEV;
+			dev_err(dev, "could not get R5 core platform device\n");
+			goto fail;
+		}
+
+		ret = k3_r5_core_of_init(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_init failed, ret = %d\n",
+				ret);
+			put_device(&cpdev->dev);
+			goto fail;
+		}
+
+		core = platform_get_drvdata(cpdev);
+		put_device(&cpdev->dev);
+		list_add_tail(&core->elem, &cluster->cores);
+	}
+
+	return 0;
+
+fail:
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		if (k3_r5_core_of_exit(cpdev))
+			dev_err(dev, "k3_r5_core_of_exit cleanup failed\n");
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct platform_device *cpdev;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		ret = k3_r5_core_of_exit(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_exit failed, ret = %d\n",
+				ret);
+			break;
+		}
+	}
+
+	return ret;
+}
+
+static int k3_r5_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_cluster *cluster;
+	int ret, ret1;
+	int num_cores;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+
+	cluster->dev = dev;
+	cluster->mode = CLUSTER_MODE_LOCKSTEP;
+	INIT_LIST_HEAD(&cluster->cores);
+
+	ret = of_property_read_u32(np, "lockstep-mode", &cluster->mode);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for lockstep-mode, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_cores = of_get_available_child_count(np);
+	if (num_cores != 2) {
+		dev_err(dev, "MCU cluster requires both R5F cores to be enabled, num_cores = %d\n",
+			num_cores);
+		return -ENODEV;
+	}
+
+	platform_set_drvdata(pdev, cluster);
+
+	dev_dbg(dev, "creating child devices for R5F cores\n");
+	ret = of_platform_populate(np, NULL, NULL, dev);
+	if (ret) {
+		dev_err(dev, "of_platform_populate failed, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = k3_r5_cluster_of_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_init failed, ret = %d\n", ret);
+		goto fail_of;
+	}
+
+	ret = k3_r5_cluster_rproc_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_init failed, ret = %d\n",
+			ret);
+		goto fail_rproc;
+	}
+
+	return 0;
+
+fail_rproc:
+	ret1 = k3_r5_cluster_of_exit(pdev);
+	if (ret1)
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret1);
+fail_of:
+	of_platform_depopulate(dev);
+	return ret;
+}
+
+static int k3_r5_remove(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	ret = k3_r5_cluster_rproc_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_exit failed, ret = %d\n",
+			ret);
+		goto fail;
+	}
+
+	ret = k3_r5_cluster_of_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret);
+		goto fail;
+	}
+
+	dev_dbg(dev, "removing child devices for R5F cores\n");
+	of_platform_depopulate(dev);
+
+fail:
+	return ret;
+}
+
+static const struct of_device_id k3_r5_of_match[] = {
+	{ .compatible = "ti,am654-r5fss", },
+	{ .compatible = "ti,j721e-r5fss", },
+	{ /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, k3_r5_of_match);
+
+static struct platform_driver k3_r5_rproc_driver = {
+	.probe = k3_r5_probe,
+	.remove = k3_r5_remove,
+	.driver = {
+		.name = "k3_r5_rproc",
+		.of_match_table = k3_r5_of_match,
+	},
+};
+
+module_platform_driver(k3_r5_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("TI K3 R5F remote processor driver");
+MODULE_AUTHOR("Suman Anna <[email protected]>");
Signature · 1 line
2.23.0
author
#23 lore raw

Re: [PATCH 5/7] remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

Hi Mathieu,

Quoted 739 lines · Mathieu Poirier
Quoted 738 lines · Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode. The driver uses various TI-SCI interfaces to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
  drivers/remoteproc/Kconfig               |   16 +
  drivers/remoteproc/Makefile              |    1 +
  drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
  3 files changed, 1363 insertions(+)

create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c

drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
  	  This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+	tristate "TI K3 R5 remoteproc support"
+	depends on ARCH_K3
+	select MAILBOX
+	select OMAP2PLUS_MBOX
+	help
+	  Say y here to support TI's R5F remote processor subsystems
+	  on various TI K3 family of SoCs through the remote processor
+	  framework.
+
+	  You want to say y here in order to offload some processing
+	  tasks to these processors
+
+	  It's safe to say N here if you're not interested in utilizing
+	  a slave processor
+
  endif # REMOTEPROC
 
  endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y			+= qcom_wcnss_iris.o
  obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)	+= ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +624 -0
new file mode 100644
index 000000000000..655f8f14c37d
@@ -0,0 +1,1346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TI K3 R5F (MCU) Remote Processor driver
+ *
+ * Copyright (C) 2017-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *	Suman Anna <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/omap-mailbox.h>
+#include <linux/reset.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+#include "omap_remoteproc.h"
+#include "remoteproc_internal.h"
+#include "ti_sci_proc.h"
+
+/* This address can either be for ATCM or BTCM with the other at address 0x0 */
+#define K3_R5_TCM_DEV_ADDR	0x41010000
+
+/* R5 TI-SCI Processor Configuration Flags */
+#define PROC_BOOT_CFG_FLAG_R5_DBG_EN			0x00000001
+#define PROC_BOOT_CFG_FLAG_R5_DBG_NIDEN			0x00000002
+#define PROC_BOOT_CFG_FLAG_R5_LOCKSTEP			0x00000100
+#define PROC_BOOT_CFG_FLAG_R5_TEINIT			0x00000200
+#define PROC_BOOT_CFG_FLAG_R5_NMFI_EN			0x00000400
+#define PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE		0x00000800
+#define PROC_BOOT_CFG_FLAG_R5_BTCM_EN			0x00001000
+#define PROC_BOOT_CFG_FLAG_R5_ATCM_EN			0x00002000
+
+/* R5 TI-SCI Processor Control Flags */
+#define PROC_BOOT_CTRL_FLAG_R5_CORE_HALT		0x00000001
+
+/* R5 TI-SCI Processor Status Flags */
+#define PROC_BOOT_STATUS_FLAG_R5_WFE			0x00000001
+#define PROC_BOOT_STATUS_FLAG_R5_WFI			0x00000002
+#define PROC_BOOT_STATUS_FLAG_R5_CLK_GATED		0x00000004
+#define PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED	0x00000100
+
+/**
+ * struct k3_r5_mem - internal memory structure
+ * @cpu_addr: MPU virtual address of the memory region
+ * @bus_addr: Bus address used to access the memory region
+ * @dev_addr: Device address from remoteproc view
+ * @size: Size of the memory region
+ */
+struct k3_r5_mem {
+	void __iomem *cpu_addr;
+	phys_addr_t bus_addr;
+	u32 dev_addr;
+	size_t size;
+};
+
+enum cluster_mode {
+	CLUSTER_MODE_SPLIT = 0,
+	CLUSTER_MODE_LOCKSTEP,
+};
+
+/**
+ * struct k3_r5_cluster - K3 R5F Cluster structure
+ * @dev: cached device pointer
+ * @mode: Mode to configure the Cluster - Split or LockStep
+ * @cores: list of R5 cores within the cluster
+ */
+struct k3_r5_cluster {
+	struct device *dev;
+	enum cluster_mode mode;
+	struct list_head cores;
+};
+
+/**
+ * struct k3_r5_core - K3 R5 core structure
+ * @elem: linked list item
+ * @dev: cached device pointer
+ * @rproc: rproc handle representing this core
+ * @mem: internal memory regions data
+ * @num_mems: number of internal memory regions
+ * @reset: reset control handle
+ * @tsp: TI-SCI processor control handle
+ * @ti_sci: TI-SCI handle
+ * @ti_sci_id: TI-SCI device identifier
+ * @atcm_enable: flag to control ATCM enablement
+ * @btcm_enable: flag to control BTCM enablement
+ * @loczrama: flag to dictate which TCM is at device address 0x0
+ */
+struct k3_r5_core {
+	struct list_head elem;
+	struct device *dev;
+	struct rproc *rproc;
+	struct k3_r5_mem *mem;
+	int num_mems;
+	struct reset_control *reset;
+	struct ti_sci_proc *tsp;
+	const struct ti_sci_handle *ti_sci;
+	u32 ti_sci_id;
+	u32 atcm_enable;
+	u32 btcm_enable;
+	u32 loczrama;
+};
+
+/**
+ * struct k3_r5_rproc - K3 remote processor state
+ * @dev: cached device pointer
+ * @cluster: cached pointer to parent cluster structure
+ * @mbox: mailbox channel handle
+ * @client: mailbox client to request the mailbox channel
+ * @rproc: rproc handle
+ * @core: cached pointer to r5 core structure being used
+ * @rmem: reserved memory regions data
+ * @num_rmems: number of reserved memory regions
+ */
+struct k3_r5_rproc {
+	struct device *dev;
+	struct k3_r5_cluster *cluster;
+	struct mbox_chan *mbox;
+	struct mbox_client client;
+	struct rproc *rproc;
+	struct k3_r5_core *core;
+	struct k3_r5_mem *rmem;
+	int num_rmems;
+};
+
+/**
+ * k3_r5_rproc_mbox_callback() - inbound mailbox message handler
+ * @client: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by the OMAP mailbox driver whenever a mailbox
+ * message is received. Usually, the mailbox payload simply contains
+ * the index of the virtqueue that is kicked by the remote processor,
+ * and we let remoteproc core handle it.
+ *
+ * In addition to virtqueue indices, we also have some out-of-band values
+ * that indicate different events. Those values are deliberately very
+ * large so they don't coincide with virtqueue indices.
+ */
+static void k3_r5_rproc_mbox_callback(struct mbox_client *client, void *data)
+{
+	struct k3_r5_rproc *kproc = container_of(client, struct k3_r5_rproc,
+						client);
+	struct device *dev = kproc->rproc->dev.parent;
+	const char *name = kproc->rproc->name;
+	u32 msg = omap_mbox_message(data);
+
+	dev_dbg(dev, "mbox msg: 0x%x\n", msg);
+
+	switch (msg) {
+	case RP_MBOX_CRASH:
+		/*
+		 * remoteproc detected an exception, but error recovery is not
+		 * supported. So, just log this for now
+		 */
+		dev_err(dev, "K3 R5F rproc %s crashed\n", name);
+		break;
+	case RP_MBOX_ECHO_REPLY:
+		dev_info(dev, "received echo reply from %s\n", name);
+		break;
+	default:
+		/* silently handle all other valid messages */
+		if (msg >= RP_MBOX_READY && msg < RP_MBOX_END_MSG)
+			return;
+		if (msg > kproc->rproc->max_notifyid) {
+			dev_dbg(dev, "dropping unknown message 0x%x", msg);
+			return;
+		}
+		/* msg contains the index of the triggered vring */
+		if (rproc_vq_interrupt(kproc->rproc, msg) == IRQ_NONE)
+			dev_dbg(dev, "no message was found in vqid %d\n", msg);
+	}
+}
+
+/* kick a virtqueue */
+static void k3_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	mbox_msg_t msg = (mbox_msg_t)vqid;
+	int ret;
+
+	/* send the index of the triggered virtqueue in the mailbox payload */
+	ret = mbox_send_message(kproc->mbox, (void *)msg);
+	if (ret < 0)
+		dev_err(dev, "failed to send mailbox message, status = %d\n",
+			ret);
+}
+
+static int k3_r5_split_reset(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = reset_control_assert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+			ret);
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_split_release(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = reset_control_deassert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset deassert failed, ret = %d\n",
+			ret);
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_reset(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* assert local reset on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = reset_control_assert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+				ret);
+			core = list_prev_entry(core, elem);
+			goto unroll_local_reset;
+		}
+	}
+
+	/* disable PSC modules on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+				ret);
+			goto unroll_module_reset;
+		}
+	}
+
+	return 0;
+
+unroll_module_reset:
+	list_for_each_entry_continue_reverse(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+	core = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_local_reset:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_release(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* enable PSC modules on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			core = list_next_entry(core, elem);
+			goto unroll_module_reset;
+		}
+	}
+
+	/* deassert local reset on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = reset_control_deassert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			goto unroll_local_reset;
+		}
+	}
+
+	return 0;
+
+unroll_local_reset:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (reset_control_assert(core->reset))
+			dev_warn(core->dev, "local-reset assert back failed\n");
+	}
+	core = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_module_reset:
+	list_for_each_entry_from(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static inline int k3_r5_core_halt(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       PROC_BOOT_CTRL_FLAG_R5_CORE_HALT, 0);
+}
+
+static inline int k3_r5_core_run(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       0, PROC_BOOT_CTRL_FLAG_R5_CORE_HALT);
+}
+
+/*
+ * The R5F cores have controls for both a reset and a halt/run. The code
+ * execution from DDR requires the initial boot-strapping code to be run
+ * from the internal TCMs. This function is used to release the resets on
+ * applicable cores to allow loading into the TCMs. The .prepare() ops is
+ * invoked by remoteproc core before any firmware loading, and is followed
+ * by the .start() ops after loading to actually let the R5 cores run.
+ */
+static int k3_r5_rproc_prepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_release(cluster) :
+			      k3_r5_split_release(core);
+	if (ret)
+		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
+			ret);
+
+	return ret;
+}
+
+/*
+ * This function implements the .unprepare() ops and performs the complimentary
+ * operations to that of the .prepare() ops. The function is used to assert the
+ * resets on all applicable cores for the rproc device (depending on LockStep
+ * or Split mode). This completes the second portion of powering down the R5F
+ * cores. The cores themselves are only halted in the .stop() ops, and the
+ * .unprepare() ops is invoked by the remoteproc core after the remoteproc is
+ * stopped.
+ */
+static int k3_r5_rproc_unprepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_reset(cluster) :
+			      k3_r5_split_reset(core);
+	if (ret)
+		dev_err(dev, "unable to disable cores, ret = %d\n", ret);
+
+	return ret;
+}
+
+/*
+ * The R5F start sequence includes two different operations
+ * 1. Configure the boot vector for R5F core(s)
+ * 2. Unhalt/Run the R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes. The LockStep
+ * mode requires the boot vector to be configured only for Core0, and then
+ * unhalt both the cores to start the execution - Core1 needs to be unhalted
+ * first followed by Core0. The Split-mode requires that Core0 to be maintained
+ * always in a higher power state that Core1 (implying Core1 needs to be started
+ * always only after Core0 is started).
+ */
+static int k3_r5_rproc_start(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct mbox_client *client = &kproc->client;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core;
+	u32 boot_addr;
+	int ret;
+
+	client->dev = dev;
+	client->tx_done = NULL;
+	client->rx_callback = k3_r5_rproc_mbox_callback;
+	client->tx_block = false;
+	client->knows_txdone = false;
+
+	kproc->mbox = mbox_request_channel(client, 0);
+	if (IS_ERR(kproc->mbox)) {
+		ret = -EBUSY;
+		dev_err(dev, "mbox_request_channel failed: %ld\n",
+			PTR_ERR(kproc->mbox));
+		return ret;
+	}
+
+	/*
+	 * Ping the remote processor, this is only for sanity-sake for now;
+	 * there is no functional effect whatsoever.
+	 *
+	 * Note that the reply will _not_ arrive immediately: this message
+	 * will wait in the mailbox fifo until the remote processor is booted.
+	 */
+	ret = mbox_send_message(kproc->mbox, (void *)RP_MBOX_ECHO_REQUEST);
+	if (ret < 0) {
+		dev_err(dev, "mbox_send_message failed: %d\n", ret);
+		goto put_mbox;
+	}
+
+	boot_addr = rproc->bootaddr;
+	/* TODO: add boot_addr sanity checking */
+	dev_err(dev, "booting R5F core using boot addr = 0x%x\n", boot_addr);
+
+	/* boot vector need not be programmed for Core1 in LockStep mode */
+	core = kproc->core;
+	ret = ti_sci_proc_set_config(core->tsp, boot_addr, 0, 0);
+	if (ret)
+		goto put_mbox;
+
+	/* unhalt/run all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry_reverse(core, &cluster->cores, elem) {
+			ret = k3_r5_core_run(core);
+			if (ret)
+				goto unroll_core_run;
+		}
+	} else {
+		ret = k3_r5_core_run(core);
+		if (ret)
+			goto put_mbox;
+	}
+
+	return 0;
+
+unroll_core_run:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (k3_r5_core_halt(core))
+			dev_warn(core->dev, "core halt back failed\n");
+	}
+put_mbox:
+	mbox_free_channel(kproc->mbox);
+	return ret;
+}
+
+/*
+ * The R5F stop function includes the following operations
+ * 1. Halt R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes, and the order
+ * of cores the operations are performed are also in general reverse to that
+ * of the start function. The LockStep mode requires each operation to be
+ * performed first on Core0 followed by Core1. The Split-mode requires that
+ * Core0 to be maintained always in a higher power state that Core1 (implying
+ * Core1 needs to be stopped first before Core0).
+ *
+ * Note that the R5F halt operation in general is not effective when the R5F
+ * core is running, but is needed to make sure the core won't run after
+ * deasserting the reset the subsequent time. The asserting of reset can
+ * be done here, but is preferred to be done in the .unprepare() ops - this
+ * maintains the symmetric behavior between the .start(), .stop(), .prepare()
+ * and .unprepare() ops, and also balances them well between sysfs 'state'
+ * flow and device bind/unbind or module removal.
+ */
+static int k3_r5_rproc_stop(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	int ret;
+
+	/* halt all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry(core, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret) {
+				core = list_prev_entry(core, elem);
+				goto unroll_core_halt;
+			}
+		}
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+	}
+
+	mbox_free_channel(kproc->mbox);
+
+	return 0;
+
+unroll_core_halt:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (k3_r5_core_run(core))
+			dev_warn(core->dev, "core run back failed\n");
+	}
+out:
+	return ret;
+}
+
+/*
+ * Internal Memory translation helper
+ *
+ * Custom function implementing the rproc .da_to_va ops to provide address
+ * translation (device address to kernel virtual address) for internal RAMs
+ * present in a DSP or IPU device). The translated addresses can be used
+ * either by the remoteproc core for loading, or by any rpmsg bus drivers.
+ */
+static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_core *core = kproc->core;
+	void __iomem *va = NULL;
+	phys_addr_t bus_addr;
+	u32 dev_addr, offset;
+	size_t size;
+	int i;
+
+	if (len == 0)
+		return NULL;
+
+	/* handle both R5 and SoC views of ATCM and BTCM */
+	for (i = 0; i < core->num_mems; i++) {
+		bus_addr = core->mem[i].bus_addr;
+		dev_addr = core->mem[i].dev_addr;
+		size = core->mem[i].size;
+
+		/* handle R5-view addresses of TCMs */
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+
+		/* handle SoC-view addresses of TCMs */
+		if (da >= bus_addr && ((da + len) <= (bus_addr + size))) {
+			offset = da - bus_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	/* handle static DDR reserved memory regions */
+	for (i = 0; i < kproc->num_rmems; i++) {
+		dev_addr = kproc->rmem[i].dev_addr;
+		size = kproc->rmem[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = kproc->rmem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	return NULL;
+}
+
+static const struct rproc_ops k3_r5_rproc_ops = {
+	.prepare	= k3_r5_rproc_prepare,
+	.unprepare	= k3_r5_rproc_unprepare,
+	.start		= k3_r5_rproc_start,
+	.stop		= k3_r5_rproc_stop,
+	.kick		= k3_r5_rproc_kick,
+	.da_to_va	= k3_r5_rproc_da_to_va,
+};
+
+static const char *k3_r5_rproc_get_firmware(struct device *dev)
+{
+	const char *fw_name;
+	int ret;
+
+	ret = of_property_read_string(dev->of_node, "firmware-name",
+				      &fw_name);
+	if (ret) {
+		dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+			ret);
+		return ERR_PTR(ret);
+	}
+
+	return fw_name;
+}
+
+static int k3_r5_rproc_configure(struct k3_r5_rproc *kproc)
+{
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core0, *core, *temp;
+	u32 ctrl = 0, cfg = 0, stat = 0;
+	u32 set_cfg = 0, clr_cfg = 0;
+	u64 boot_vec = 0;
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 629

Quoted 5 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +5 -0
+	bool lockstep_en;
+	int ret;
+
+	core0 = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+	core = cluster->mode ? core0 : kproc->core;

The above two lines generated interesting mental gymnastic - please sprinkle with comments in order to disambiguate what is going on.

Yeah, I will add some comments or add some top-level description above the function for v2.

Quoted 59 lines
Quoted 42 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +41 -0
+
+	ret = ti_sci_proc_get_status(core->tsp, &boot_vec, &cfg, &ctrl,
+				     &stat);
+	if (ret < 0)
+		return ret;
+
+	dev_dbg(dev, "boot_vector = 0x%llx, cfg = 0x%x ctrl = 0x%x stat = 0x%x\n",
+		boot_vec, cfg, ctrl, stat);
+
+	lockstep_en = !!(stat & PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED);
+	if (!lockstep_en && cluster->mode) {
+		dev_err(cluster->dev, "lockstep mode not permitted, force configuring for split-mode\n");
+		cluster->mode = 0;
+	}
+
+	/* always enable ARM mode and set boot vector to 0 */
+	boot_vec = 0x0;
+	if (core == core0) {
+		clr_cfg = PROC_BOOT_CFG_FLAG_R5_TEINIT;
+		/*
+		 * LockStep configuration bit is Read-only on Split-mode _only_
+		 * devices and system firmware will NACK any requests with the
+		 * bit configured, so program it only on permitted devices
+		 */
+		if (lockstep_en)
+			clr_cfg |= PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+	}
+
+	if (core->atcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+
+	if (core->btcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+
+	if (core->loczrama)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+	else

drivers/remoteproc/ti_k3_r5_remoteproc.c line 680

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+
+	if (cluster->mode) {
+		/*
+		 * work around system firmware limitations to make sure both
+		 * cores are programmed symmetrically in LockStep. LockStep
+		 * and TEINIT config is only allowed with Core0.
+		 */
+		list_for_each_entry(temp, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);

When I first read this I thought this was an error and what was really needed was k3_r5_core_halt(temp)... But no, this is correct because k3_r5_rproc_configure() is called for each core in the cluster in function k3_r5_cluster_rproc_init(). But then again why halting the same for each of the cores found in the system?

Quoted 2 lines

So something seems wrong here. Either call k3_r5_core_halt(temp) or move k3_r5_core_halt(core) outside of the if (cluster->mode) to avoid more confusion.

LockStep-mode in general still requires operating/managing both the cores. Let me double-check this again, the halt should be done on both the cores for sure, so it does appear that I should be using k3_r5_core_halt(temp) indeed.

This function is not called a second time if the cluster is in LockStep mode. I break out of the loop at the end in k3_r5_cluster_rproc_init() if the cluster is in LockStep mode.

Quoted 192 lines
Quoted 178 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +177 -0
+			if (ret)
+				goto out;
+
+			if (temp != core) {
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_TEINIT;
+			}
+			ret = ti_sci_proc_set_config(temp->tsp, boot_vec,
+						     set_cfg, clr_cfg);
+			if (ret)
+				goto out;
+		}
+
+		set_cfg = PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+		clr_cfg = 0;
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	}
+
+out:
+	return ret;
+}
+
+static int k3_r5_reserved_mem_init(struct k3_r5_rproc *kproc)
+{
+	struct device *dev = kproc->dev;
+	struct device_node *np = dev->of_node;
+	struct device_node *rmem_np;
+	struct reserved_mem *rmem;
+	int num_rmems;
+	int ret, i;
+
+	num_rmems = of_property_count_elems_of_size(np, "memory-region",
+						    sizeof(phandle));
+	if (num_rmems <= 0) {
+		dev_err(dev, "device does not have reserved memory regions, ret = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+	if (num_rmems < 2) {
+		dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+
+	/* use reserved memory region 0 for vring DMA allocations */
+	ret = of_reserved_mem_device_init_by_idx(dev, np, 0);
+	if (ret) {
+		dev_err(dev, "device cannot initialize DMA pool, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_rmems--;
+	kproc->rmem = kcalloc(num_rmems, sizeof(*kproc->rmem), GFP_KERNEL);
+	if (!kproc->rmem) {
+		ret = -ENOMEM;
+		goto release_rmem;
+	}
+
+	/* use remaining reserved memory regions for static carveouts */
+	for (i = 0; i < num_rmems; i++) {
+		rmem_np = of_parse_phandle(np, "memory-region", i + 1);
+		if (!rmem_np) {
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+
+		rmem = of_reserved_mem_lookup(rmem_np);
+		if (!rmem) {
+			of_node_put(rmem_np);
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+		of_node_put(rmem_np);
+
+		kproc->rmem[i].bus_addr = rmem->base;
+		/* 64-bit address regions currently not supported */
+		kproc->rmem[i].dev_addr = (u32)rmem->base;
+		kproc->rmem[i].size = rmem->size;
+		kproc->rmem[i].cpu_addr = ioremap_wc(rmem->base, rmem->size);
+		if (!kproc->rmem[i].cpu_addr) {
+			dev_err(dev, "failed to map reserved memory#%d at %pa of size %pa\n",
+				i + 1, &rmem->base, &rmem->size);
+			ret = -ENOMEM;
+			goto unmap_rmem;
+		}
+
+		dev_dbg(dev, "reserved memory%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			i + 1, &kproc->rmem[i].bus_addr,
+			kproc->rmem[i].size, kproc->rmem[i].cpu_addr,
+			kproc->rmem[i].dev_addr);
+	}
+	kproc->num_rmems = num_rmems;
+
+	return 0;
+
+unmap_rmem:
+	for (i--; i >= 0; i--) {
+		if (kproc->rmem[i].cpu_addr)
+			iounmap(kproc->rmem[i].cpu_addr);
+	}
+	kfree(kproc->rmem);
+release_rmem:
+	of_reserved_mem_device_release(dev);
+	return ret;
+}
+
+static void k3_r5_reserved_mem_exit(struct k3_r5_rproc *kproc)
+{
+	int i;
+
+	for (i = 0; i < kproc->num_rmems; i++)
+		iounmap(kproc->rmem[i].cpu_addr);
+	kfree(kproc->rmem);
+
+	of_reserved_mem_device_release(kproc->dev);
+}
+
+static int k3_r5_cluster_rproc_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core, *core1;
+	struct device *cdev;
+	const char *fw_name;
+	struct rproc *rproc;
+	int ret;
+
+	core1 = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+	list_for_each_entry(core, &cluster->cores, elem) {
+		cdev = core->dev;
+		fw_name = k3_r5_rproc_get_firmware(cdev);
+		if (IS_ERR(fw_name)) {
+			ret = PTR_ERR(fw_name);
+			goto out;
+		}
+
+		rproc = rproc_alloc(cdev, dev_name(cdev), &k3_r5_rproc_ops,
+				    fw_name, sizeof(*kproc));
+		if (!rproc) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		/* K3 R5s have a Region Address Translator (RAT) but no MMU */
+		rproc->has_iommu = false;
+		/* error recovery is not supported at present */
+		rproc->recovery_disabled = true;
+
+		kproc = rproc->priv;
+		kproc->cluster = cluster;
+		kproc->core = core;
+		kproc->dev = cdev;
+		kproc->rproc = rproc;
+		core->rproc = rproc;
+
+		ret = k3_r5_rproc_configure(kproc);
+		if (ret) {
+			dev_err(dev, "initial configure failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = k3_r5_reserved_mem_init(kproc);
+		if (ret) {
+			dev_err(dev, "reserved memory init failed, ret = %d\n",
+				ret);
+			goto err_config;

drivers/remoteproc/ti_k3_r5_remoteproc.c line 867

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+		}
+
+		ret = rproc_add(rproc);
+		if (ret) {
+			dev_err(dev, "rproc_add failed, ret = %d\n", ret);
+			goto err_add;
+		}
+
+		/* create only one rproc in lockstep mode */
+		if (cluster->mode)

Here and throughout the file, please use the cluster mode enumeration in order to improve readability, i.e

Quoted 1 line
                 if (cluster->mode == CLUSTER_MODE_LOCKSTEP)

OK.

Quoted 134 lines
Quoted 134 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +133 -0
+			break;
+	}
+
+	return 0;
+
+err_split:
+	rproc_del(rproc);
+err_add:
+	k3_r5_reserved_mem_exit(kproc);
+err_config:
+	rproc_free(rproc);
+	core->rproc = NULL;
+out:
+	/* undo core0 upon any failures on core1 in split-mode */
+	if (!cluster->mode && core == core1) {
+		core = list_prev_entry(core, elem);
+		rproc = core->rproc;
+		kproc = rproc->priv;
+		goto err_split;
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_rproc_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core;
+	struct rproc *rproc;
+
+	/*
+	 * lockstep mode has only one rproc associated with first core, whereas
+	 * split-mode has two rprocs associated with each core, and requires
+	 * that core1 be powered down first
+	 */
+	core = cluster->mode ?
+		list_first_entry(&cluster->cores, struct k3_r5_core, elem) :
+		list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		rproc = core->rproc;
+		kproc = rproc->priv;
+
+		rproc_del(rproc);
+
+		k3_r5_reserved_mem_exit(kproc);
+
+		rproc_free(rproc);
+		core->rproc = NULL;
+	}
+
+	return 0;
+}
+
+static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
+					       struct k3_r5_core *core)
+{
+	static const char * const mem_names[] = {"atcm", "btcm"};
+	struct device *dev = &pdev->dev;
+	struct resource *res;
+	int num_mems;
+	int i, ret;
+
+	num_mems = ARRAY_SIZE(mem_names);
+	core->mem = devm_kcalloc(dev, num_mems, sizeof(*core->mem), GFP_KERNEL);
+	if (!core->mem)
+		return -ENOMEM;
+
+	for (i = 0; i < num_mems; i++) {
+		res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						   mem_names[i]);
+		if (!res) {
+			dev_err(dev, "found no memory resource for %s\n",
+				mem_names[i]);
+			ret = -EINVAL;
+			goto fail;
+		}
+		if (!devm_request_mem_region(dev, res->start,
+					     resource_size(res),
+					     dev_name(dev))) {
+			dev_err(dev, "could not request %s region for resource\n",
+				mem_names[i]);
+			ret = -EBUSY;
+			goto fail;
+		}
+
+		/*
+		 * TCMs are designed in general to support RAM-like backing
+		 * memories. So, map these as Normal Non-Cached memories. This
+		 * also avoids/fixes any potential alignment faults due to
+		 * unaligned data accesses when using memcpy() or memset()
+		 * functions (normally seen with device type memory).
+		 */
+		core->mem[i].cpu_addr = devm_ioremap_wc(dev, res->start,
+							resource_size(res));
+		if (IS_ERR(core->mem[i].cpu_addr)) {
+			dev_err(dev, "failed to map %s memory\n", mem_names[i]);
+			ret = PTR_ERR(core->mem[i].cpu_addr);
+			devm_release_mem_region(dev, res->start,
+						resource_size(res));
+			goto fail;
+		}
+		core->mem[i].bus_addr = res->start;
+
+		/*
+		 * TODO:
+		 * The R5F cores can place ATCM & BTCM anywhere in its address
+		 * based on the corresponding Region Registers in the System
+		 * Control coprocessor. For now, place ATCM and BTCM at
+		 * addresses 0 and 0x41010000 (same as the bus address on AM65x
+		 * SoCs) based on loczrama setting
+		 */
+		if (!strcmp(mem_names[i], "atcm")) {
+			core->mem[i].dev_addr = core->loczrama ?
+							0 : K3_R5_TCM_DEV_ADDR;
+		} else {
+			core->mem[i].dev_addr = core->loczrama ?
+							K3_R5_TCM_DEV_ADDR : 0;
+		}
+		core->mem[i].size = resource_size(res);
+
+		dev_dbg(dev, "memory %8s: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			mem_names[i], &core->mem[i].bus_addr,
+			core->mem[i].size, core->mem[i].cpu_addr,
+			core->mem[i].dev_addr);
+	}
+	core->num_mems = num_mems;
+
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1005

Quoted 5 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +5 -0
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);

Since the devm_ API has been used for memory allocation all this should be called automatically when of_platform_depopulate() is called.

The per-core platform devices do not use a dedicated platform driver associated with them, so there were no probe/remove functions associated with them. I added these since some of them do not have devm_ equivalents, and it was easier to balance everything. Let me retest these and get back to you, if of_platform_depopulate() takes care automatically and without any ordering issues.

Quoted 44 lines
Quoted 8 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +7 -0
+	return ret;
+}
+
+static
+struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
+					  const struct ti_sci_handle *sci)
+{

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1022

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	struct ti_sci_proc *tsp;
+	u32 temp[2];
+	int ret;
+
+	ret = of_property_read_u32_array(dev->of_node, "ti,sci-proc-ids",
+					 temp, 2);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	tsp = kzalloc(sizeof(*tsp), GFP_KERNEL);

Here @dev is available, I would just call devm_kzalloc() and get rid of the kfree() in the error path of k3_r5_core_of_init() and k3_r5_core_of_exit().

Quoted 22 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +22 -0
+	if (!tsp)
+		return ERR_PTR(-ENOMEM);
+
+	tsp->dev = dev;
+	tsp->sci = sci;
+	tsp->ops = &sci->ops.proc_ops;
+	tsp->proc_id = temp[0];
+	tsp->host_id = temp[1];
+
+	return tsp;
+}
+
+static int k3_r5_core_of_init(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_core *core;
+	int ret, ret1;
+
+	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
+	if (!core)
+		return -ENOMEM;
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1049

Quoted 5 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +5 -0
+
+	core->dev = dev;
+	core->atcm_enable = 0;
+	core->btcm_enable = 1;
+	core->loczrama = 1;

Please add a comment that justifies the selection of default value. Otherwise this looks very esoteric.

Yeah sure. These are default values on the SoC.

Quoted 78 lines
Quoted 78 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +77 -0
+
+	ret = of_property_read_u32(np, "atcm-enable", &core->atcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for atcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "btcm-enable", &core->btcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for btcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "loczrama", &core->loczrama);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for loczrama, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	core->ti_sci = ti_sci_get_by_phandle(np, "ti,sci");
+	if (IS_ERR(core->ti_sci)) {
+		ret = PTR_ERR(core->ti_sci);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get ti-sci handle, ret = %d\n",
+				ret);
+		}
+		core->ti_sci = NULL;
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "ti,sci-dev-id", &core->ti_sci_id);
+	if (ret) {
+		dev_err(dev, "missing 'ti,sci-dev-id' property\n");
+		goto err_sci_id;
+	}
+
+	core->reset = reset_control_get_exclusive(dev, NULL);
+	if (IS_ERR(core->reset)) {
+		ret = PTR_ERR(core->reset);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get reset handle, ret = %d\n",
+				ret);
+		}
+		goto err_sci_id;
+	}
+
+	core->tsp = k3_r5_core_of_get_tsp(dev, core->ti_sci);
+	if (IS_ERR(core->tsp)) {
+		dev_err(dev, "failed to construct ti-sci proc control, ret = %d\n",
+			ret);
+		ret = PTR_ERR(core->tsp);
+		goto err_sci_proc;
+	}
+
+	ret = ti_sci_proc_request(core->tsp);
+	if (ret < 0) {
+		dev_err(dev, "ti_sci_proc_request failed, ret = %d\n", ret);
+		goto err_proc;
+	}
+
+	ret = k3_r5_core_of_get_internal_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get internal memories, ret = %d\n",
+			ret);
+		goto err_intmem;
+	}
+
+	platform_set_drvdata(pdev, core);
+
+	return 0;
+
+err_intmem:
+	ret1 = ti_sci_proc_release(core->tsp);
+	if (ret1)
+		dev_err(dev, "failed to release proc, ret1 = %d\n", ret1);
+err_proc:
+	kfree(core->tsp);
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1132

Quoted 6 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +6 -0
+err_sci_proc:
+	reset_control_put(core->reset);
+err_sci_id:
+	ret1 = ti_sci_put_handle(core->ti_sci);
+	if (ret1)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret1);

s/"ret = %d"/"ret1 = %d"

OK

Quoted 34 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1134

Quoted 3 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +2 -0
+err_of:
+	devm_kfree(dev, core);

Same comment as above, this should be called automatically.

Quoted 28 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +28 -0
+	return ret;
+}
+
+/*
+ * free the resources explicitly since driver model is not being used
+ * for the child R5F devices
+ */
+static int k3_r5_core_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_core *core = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	int i, ret;
+
+	for (i = 0; i < core->num_mems; i++) {
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+
+	ret = ti_sci_proc_release(core->tsp);
+	if (ret)
+		dev_err(dev, "failed to release proc, ret = %d\n", ret);
+
+	kfree(core->tsp);
+	reset_control_put(core->reset);
+
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 1168

Quoted 6 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +6 -0
+	ret = ti_sci_put_handle(core->ti_sci);
+	if (ret)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret);
+
+	platform_set_drvdata(pdev, NULL);
+	devm_kfree(dev, core);

Same comment regarding the devm_ API, this should be called automatically.

Yeah, will relook at the devm_ usage at all places.

Thanks for the review.

regards,
Suman

Quoted 185 lines

I will continue tomorrow,
Mathieu

Quoted 181 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +178 -0
+
+	return ret;
+}
+
+static int k3_r5_cluster_of_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct platform_device *cpdev;
+	struct device_node *child;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	for_each_available_child_of_node(np, child) {
+		cpdev = of_find_device_by_node(child);
+		if (!cpdev) {
+			ret = -ENODEV;
+			dev_err(dev, "could not get R5 core platform device\n");
+			goto fail;
+		}
+
+		ret = k3_r5_core_of_init(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_init failed, ret = %d\n",
+				ret);
+			put_device(&cpdev->dev);
+			goto fail;
+		}
+
+		core = platform_get_drvdata(cpdev);
+		put_device(&cpdev->dev);
+		list_add_tail(&core->elem, &cluster->cores);
+	}
+
+	return 0;
+
+fail:
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		if (k3_r5_core_of_exit(cpdev))
+			dev_err(dev, "k3_r5_core_of_exit cleanup failed\n");
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct platform_device *cpdev;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		ret = k3_r5_core_of_exit(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_exit failed, ret = %d\n",
+				ret);
+			break;
+		}
+	}
+
+	return ret;
+}
+
+static int k3_r5_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_cluster *cluster;
+	int ret, ret1;
+	int num_cores;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+
+	cluster->dev = dev;
+	cluster->mode = CLUSTER_MODE_LOCKSTEP;
+	INIT_LIST_HEAD(&cluster->cores);
+
+	ret = of_property_read_u32(np, "lockstep-mode", &cluster->mode);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for lockstep-mode, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_cores = of_get_available_child_count(np);
+	if (num_cores != 2) {
+		dev_err(dev, "MCU cluster requires both R5F cores to be enabled, num_cores = %d\n",
+			num_cores);
+		return -ENODEV;
+	}
+
+	platform_set_drvdata(pdev, cluster);
+
+	dev_dbg(dev, "creating child devices for R5F cores\n");
+	ret = of_platform_populate(np, NULL, NULL, dev);
+	if (ret) {
+		dev_err(dev, "of_platform_populate failed, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = k3_r5_cluster_of_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_init failed, ret = %d\n", ret);
+		goto fail_of;
+	}
+
+	ret = k3_r5_cluster_rproc_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_init failed, ret = %d\n",
+			ret);
+		goto fail_rproc;
+	}
+
+	return 0;
+
+fail_rproc:
+	ret1 = k3_r5_cluster_of_exit(pdev);
+	if (ret1)
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret1);
+fail_of:
+	of_platform_depopulate(dev);
+	return ret;
+}
+
+static int k3_r5_remove(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	ret = k3_r5_cluster_rproc_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_exit failed, ret = %d\n",
+			ret);
+		goto fail;
+	}
+
+	ret = k3_r5_cluster_of_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret);
+		goto fail;
+	}
+
+	dev_dbg(dev, "removing child devices for R5F cores\n");
+	of_platform_depopulate(dev);
+
+fail:
+	return ret;
+}
+
+static const struct of_device_id k3_r5_of_match[] = {
+	{ .compatible = "ti,am654-r5fss", },
+	{ .compatible = "ti,j721e-r5fss", },
+	{ /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, k3_r5_of_match);
+
+static struct platform_driver k3_r5_rproc_driver = {
+	.probe = k3_r5_probe,
+	.remove = k3_r5_remove,
+	.driver = {
+		.name = "k3_r5_rproc",
+		.of_match_table = k3_r5_of_match,
+	},
+};
+
+module_platform_driver(k3_r5_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("TI K3 R5F remote processor driver");
+MODULE_AUTHOR("Suman Anna <[email protected]>");
Signature · 1 line
2.23.0
maintainer
#24 lore raw

Re: [PATCH 5/7] remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

In reply to Suman Anna
Quoted 400 lines · Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode. The driver uses various TI-SCI interfaces to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
 drivers/remoteproc/Kconfig               |   16 +
 drivers/remoteproc/Makefile              |    1 +
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
 3 files changed, 1363 insertions(+)
 create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c
drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
 	  This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+	tristate "TI K3 R5 remoteproc support"
+	depends on ARCH_K3
+	select MAILBOX
+	select OMAP2PLUS_MBOX
+	help
+	  Say y here to support TI's R5F remote processor subsystems
+	  on various TI K3 family of SoCs through the remote processor
+	  framework.
+
+	  You want to say y here in order to offload some processing
+	  tasks to these processors
+
+	  It's safe to say N here if you're not interested in utilizing
+	  a slave processor
+
 endif # REMOTEPROC
 
 endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y			+= qcom_wcnss_iris.o
 obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
 obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
 obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)	+= ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +286 -0
new file mode 100644
index 000000000000..655f8f14c37d
@@ -0,0 +1,1346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TI K3 R5F (MCU) Remote Processor driver
+ *
+ * Copyright (C) 2017-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *	Suman Anna <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/omap-mailbox.h>
+#include <linux/reset.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+#include "omap_remoteproc.h"
+#include "remoteproc_internal.h"
+#include "ti_sci_proc.h"
+
+/* This address can either be for ATCM or BTCM with the other at address 0x0 */
+#define K3_R5_TCM_DEV_ADDR	0x41010000
+
+/* R5 TI-SCI Processor Configuration Flags */
+#define PROC_BOOT_CFG_FLAG_R5_DBG_EN			0x00000001
+#define PROC_BOOT_CFG_FLAG_R5_DBG_NIDEN			0x00000002
+#define PROC_BOOT_CFG_FLAG_R5_LOCKSTEP			0x00000100
+#define PROC_BOOT_CFG_FLAG_R5_TEINIT			0x00000200
+#define PROC_BOOT_CFG_FLAG_R5_NMFI_EN			0x00000400
+#define PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE		0x00000800
+#define PROC_BOOT_CFG_FLAG_R5_BTCM_EN			0x00001000
+#define PROC_BOOT_CFG_FLAG_R5_ATCM_EN			0x00002000
+
+/* R5 TI-SCI Processor Control Flags */
+#define PROC_BOOT_CTRL_FLAG_R5_CORE_HALT		0x00000001
+
+/* R5 TI-SCI Processor Status Flags */
+#define PROC_BOOT_STATUS_FLAG_R5_WFE			0x00000001
+#define PROC_BOOT_STATUS_FLAG_R5_WFI			0x00000002
+#define PROC_BOOT_STATUS_FLAG_R5_CLK_GATED		0x00000004
+#define PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED	0x00000100
+
+/**
+ * struct k3_r5_mem - internal memory structure
+ * @cpu_addr: MPU virtual address of the memory region
+ * @bus_addr: Bus address used to access the memory region
+ * @dev_addr: Device address from remoteproc view
+ * @size: Size of the memory region
+ */
+struct k3_r5_mem {
+	void __iomem *cpu_addr;
+	phys_addr_t bus_addr;
+	u32 dev_addr;
+	size_t size;
+};
+
+enum cluster_mode {
+	CLUSTER_MODE_SPLIT = 0,
+	CLUSTER_MODE_LOCKSTEP,
+};
+
+/**
+ * struct k3_r5_cluster - K3 R5F Cluster structure
+ * @dev: cached device pointer
+ * @mode: Mode to configure the Cluster - Split or LockStep
+ * @cores: list of R5 cores within the cluster
+ */
+struct k3_r5_cluster {
+	struct device *dev;
+	enum cluster_mode mode;
+	struct list_head cores;
+};
+
+/**
+ * struct k3_r5_core - K3 R5 core structure
+ * @elem: linked list item
+ * @dev: cached device pointer
+ * @rproc: rproc handle representing this core
+ * @mem: internal memory regions data
+ * @num_mems: number of internal memory regions
+ * @reset: reset control handle
+ * @tsp: TI-SCI processor control handle
+ * @ti_sci: TI-SCI handle
+ * @ti_sci_id: TI-SCI device identifier
+ * @atcm_enable: flag to control ATCM enablement
+ * @btcm_enable: flag to control BTCM enablement
+ * @loczrama: flag to dictate which TCM is at device address 0x0
+ */
+struct k3_r5_core {
+	struct list_head elem;
+	struct device *dev;
+	struct rproc *rproc;
+	struct k3_r5_mem *mem;
+	int num_mems;
+	struct reset_control *reset;
+	struct ti_sci_proc *tsp;
+	const struct ti_sci_handle *ti_sci;
+	u32 ti_sci_id;
+	u32 atcm_enable;
+	u32 btcm_enable;
+	u32 loczrama;
+};
+
+/**
+ * struct k3_r5_rproc - K3 remote processor state
+ * @dev: cached device pointer
+ * @cluster: cached pointer to parent cluster structure
+ * @mbox: mailbox channel handle
+ * @client: mailbox client to request the mailbox channel
+ * @rproc: rproc handle
+ * @core: cached pointer to r5 core structure being used
+ * @rmem: reserved memory regions data
+ * @num_rmems: number of reserved memory regions
+ */
+struct k3_r5_rproc {
+	struct device *dev;
+	struct k3_r5_cluster *cluster;
+	struct mbox_chan *mbox;
+	struct mbox_client client;
+	struct rproc *rproc;
+	struct k3_r5_core *core;
+	struct k3_r5_mem *rmem;
+	int num_rmems;
+};
+
+/**
+ * k3_r5_rproc_mbox_callback() - inbound mailbox message handler
+ * @client: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by the OMAP mailbox driver whenever a mailbox
+ * message is received. Usually, the mailbox payload simply contains
+ * the index of the virtqueue that is kicked by the remote processor,
+ * and we let remoteproc core handle it.
+ *
+ * In addition to virtqueue indices, we also have some out-of-band values
+ * that indicate different events. Those values are deliberately very
+ * large so they don't coincide with virtqueue indices.
+ */
+static void k3_r5_rproc_mbox_callback(struct mbox_client *client, void *data)
+{
+	struct k3_r5_rproc *kproc = container_of(client, struct k3_r5_rproc,
+						client);
+	struct device *dev = kproc->rproc->dev.parent;
+	const char *name = kproc->rproc->name;
+	u32 msg = omap_mbox_message(data);
+
+	dev_dbg(dev, "mbox msg: 0x%x\n", msg);
+
+	switch (msg) {
+	case RP_MBOX_CRASH:
+		/*
+		 * remoteproc detected an exception, but error recovery is not
+		 * supported. So, just log this for now
+		 */
+		dev_err(dev, "K3 R5F rproc %s crashed\n", name);
+		break;
+	case RP_MBOX_ECHO_REPLY:
+		dev_info(dev, "received echo reply from %s\n", name);
+		break;
+	default:
+		/* silently handle all other valid messages */
+		if (msg >= RP_MBOX_READY && msg < RP_MBOX_END_MSG)
+			return;
+		if (msg > kproc->rproc->max_notifyid) {
+			dev_dbg(dev, "dropping unknown message 0x%x", msg);
+			return;
+		}
+		/* msg contains the index of the triggered vring */
+		if (rproc_vq_interrupt(kproc->rproc, msg) == IRQ_NONE)
+			dev_dbg(dev, "no message was found in vqid %d\n", msg);
+	}
+}
+
+/* kick a virtqueue */
+static void k3_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	mbox_msg_t msg = (mbox_msg_t)vqid;
+	int ret;
+
+	/* send the index of the triggered virtqueue in the mailbox payload */
+	ret = mbox_send_message(kproc->mbox, (void *)msg);
+	if (ret < 0)
+		dev_err(dev, "failed to send mailbox message, status = %d\n",
+			ret);
+}
+
+static int k3_r5_split_reset(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = reset_control_assert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+			ret);
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_split_release(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = reset_control_deassert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset deassert failed, ret = %d\n",
+			ret);
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_reset(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* assert local reset on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = reset_control_assert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+				ret);
+			core = list_prev_entry(core, elem);
+			goto unroll_local_reset;
+		}
+	}
+
+	/* disable PSC modules on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+				ret);
+			goto unroll_module_reset;
+		}
+	}
+
+	return 0;
+
+unroll_module_reset:
+	list_for_each_entry_continue_reverse(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+	core = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_local_reset:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+

drivers/remoteproc/ti_k3_r5_remoteproc.c line 296

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	return ret;
+}
+
+static int k3_r5_lockstep_release(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* enable PSC modules on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {

Out of curiosity, any HW specific reason to start with the last core?

Quoted 125 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +125 -0
+		ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			core = list_next_entry(core, elem);
+			goto unroll_module_reset;
+		}
+	}
+
+	/* deassert local reset on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = reset_control_deassert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			goto unroll_local_reset;
+		}
+	}
+
+	return 0;
+
+unroll_local_reset:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (reset_control_assert(core->reset))
+			dev_warn(core->dev, "local-reset assert back failed\n");
+	}
+	core = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_module_reset:
+	list_for_each_entry_from(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static inline int k3_r5_core_halt(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       PROC_BOOT_CTRL_FLAG_R5_CORE_HALT, 0);
+}
+
+static inline int k3_r5_core_run(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       0, PROC_BOOT_CTRL_FLAG_R5_CORE_HALT);
+}
+
+/*
+ * The R5F cores have controls for both a reset and a halt/run. The code
+ * execution from DDR requires the initial boot-strapping code to be run
+ * from the internal TCMs. This function is used to release the resets on
+ * applicable cores to allow loading into the TCMs. The .prepare() ops is
+ * invoked by remoteproc core before any firmware loading, and is followed
+ * by the .start() ops after loading to actually let the R5 cores run.
+ */
+static int k3_r5_rproc_prepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_release(cluster) :
+			      k3_r5_split_release(core);
+	if (ret)
+		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
+			ret);
+
+	return ret;
+}
+
+/*
+ * This function implements the .unprepare() ops and performs the complimentary
+ * operations to that of the .prepare() ops. The function is used to assert the
+ * resets on all applicable cores for the rproc device (depending on LockStep
+ * or Split mode). This completes the second portion of powering down the R5F
+ * cores. The cores themselves are only halted in the .stop() ops, and the
+ * .unprepare() ops is invoked by the remoteproc core after the remoteproc is
+ * stopped.
+ */
+static int k3_r5_rproc_unprepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_reset(cluster) :
+			      k3_r5_split_reset(core);
+	if (ret)
+		dev_err(dev, "unable to disable cores, ret = %d\n", ret);
+
+	return ret;
+}
+
+/*
+ * The R5F start sequence includes two different operations
+ * 1. Configure the boot vector for R5F core(s)
+ * 2. Unhalt/Run the R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes. The LockStep
+ * mode requires the boot vector to be configured only for Core0, and then
+ * unhalt both the cores to start the execution - Core1 needs to be unhalted
+ * first followed by Core0. The Split-mode requires that Core0 to be maintained
+ * always in a higher power state that Core1 (implying Core1 needs to be started
+ * always only after Core0 is started).
+ */
+static int k3_r5_rproc_start(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct mbox_client *client = &kproc->client;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core;
+	u32 boot_addr;
+	int ret;
+
+	client->dev = dev;
+	client->tx_done = NULL;
+	client->rx_callback = k3_r5_rproc_mbox_callback;

drivers/remoteproc/ti_k3_r5_remoteproc.c line 431

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	client->tx_block = false;
+	client->knows_txdone = false;
+
+	kproc->mbox = mbox_request_channel(client, 0);
+	if (IS_ERR(kproc->mbox)) {
+		ret = -EBUSY;
+		dev_err(dev, "mbox_request_channel failed: %ld\n",
+			PTR_ERR(kproc->mbox));
+		return ret;
+	}

Does this needs to be done every time a remote processor is booted or could it be done just once in k3_r5_core_of_init()?

Quoted 7 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +7 -0
+
+	/*
+	 * Ping the remote processor, this is only for sanity-sake for now;
+	 * there is no functional effect whatsoever.
+	 *
+	 * Note that the reply will _not_ arrive immediately: this message
+	 * will wait in the mailbox fifo until the remote processor is booted.

drivers/remoteproc/ti_k3_r5_remoteproc.c line 448

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+	 */
+	ret = mbox_send_message(kproc->mbox, (void *)RP_MBOX_ECHO_REQUEST);
+	if (ret < 0) {
+		dev_err(dev, "mbox_send_message failed: %d\n", ret);
+		goto put_mbox;
+	}
+
+	boot_addr = rproc->bootaddr;
+	/* TODO: add boot_addr sanity checking */
+	dev_err(dev, "booting R5F core using boot addr = 0x%x\n", boot_addr);

s/dev_err()/dev_dbg()

Quoted 308 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +308 -0
+
+	/* boot vector need not be programmed for Core1 in LockStep mode */
+	core = kproc->core;
+	ret = ti_sci_proc_set_config(core->tsp, boot_addr, 0, 0);
+	if (ret)
+		goto put_mbox;
+
+	/* unhalt/run all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry_reverse(core, &cluster->cores, elem) {
+			ret = k3_r5_core_run(core);
+			if (ret)
+				goto unroll_core_run;
+		}
+	} else {
+		ret = k3_r5_core_run(core);
+		if (ret)
+			goto put_mbox;
+	}
+
+	return 0;
+
+unroll_core_run:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (k3_r5_core_halt(core))
+			dev_warn(core->dev, "core halt back failed\n");
+	}
+put_mbox:
+	mbox_free_channel(kproc->mbox);
+	return ret;
+}
+
+/*
+ * The R5F stop function includes the following operations
+ * 1. Halt R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes, and the order
+ * of cores the operations are performed are also in general reverse to that
+ * of the start function. The LockStep mode requires each operation to be
+ * performed first on Core0 followed by Core1. The Split-mode requires that
+ * Core0 to be maintained always in a higher power state that Core1 (implying
+ * Core1 needs to be stopped first before Core0).
+ *
+ * Note that the R5F halt operation in general is not effective when the R5F
+ * core is running, but is needed to make sure the core won't run after
+ * deasserting the reset the subsequent time. The asserting of reset can
+ * be done here, but is preferred to be done in the .unprepare() ops - this
+ * maintains the symmetric behavior between the .start(), .stop(), .prepare()
+ * and .unprepare() ops, and also balances them well between sysfs 'state'
+ * flow and device bind/unbind or module removal.
+ */
+static int k3_r5_rproc_stop(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	int ret;
+
+	/* halt all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry(core, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret) {
+				core = list_prev_entry(core, elem);
+				goto unroll_core_halt;
+			}
+		}
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+	}
+
+	mbox_free_channel(kproc->mbox);
+
+	return 0;
+
+unroll_core_halt:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (k3_r5_core_run(core))
+			dev_warn(core->dev, "core run back failed\n");
+	}
+out:
+	return ret;
+}
+
+/*
+ * Internal Memory translation helper
+ *
+ * Custom function implementing the rproc .da_to_va ops to provide address
+ * translation (device address to kernel virtual address) for internal RAMs
+ * present in a DSP or IPU device). The translated addresses can be used
+ * either by the remoteproc core for loading, or by any rpmsg bus drivers.
+ */
+static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_core *core = kproc->core;
+	void __iomem *va = NULL;
+	phys_addr_t bus_addr;
+	u32 dev_addr, offset;
+	size_t size;
+	int i;
+
+	if (len == 0)
+		return NULL;
+
+	/* handle both R5 and SoC views of ATCM and BTCM */
+	for (i = 0; i < core->num_mems; i++) {
+		bus_addr = core->mem[i].bus_addr;
+		dev_addr = core->mem[i].dev_addr;
+		size = core->mem[i].size;
+
+		/* handle R5-view addresses of TCMs */
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+
+		/* handle SoC-view addresses of TCMs */
+		if (da >= bus_addr && ((da + len) <= (bus_addr + size))) {
+			offset = da - bus_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	/* handle static DDR reserved memory regions */
+	for (i = 0; i < kproc->num_rmems; i++) {
+		dev_addr = kproc->rmem[i].dev_addr;
+		size = kproc->rmem[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = kproc->rmem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	return NULL;
+}
+
+static const struct rproc_ops k3_r5_rproc_ops = {
+	.prepare	= k3_r5_rproc_prepare,
+	.unprepare	= k3_r5_rproc_unprepare,
+	.start		= k3_r5_rproc_start,
+	.stop		= k3_r5_rproc_stop,
+	.kick		= k3_r5_rproc_kick,
+	.da_to_va	= k3_r5_rproc_da_to_va,
+};
+
+static const char *k3_r5_rproc_get_firmware(struct device *dev)
+{
+	const char *fw_name;
+	int ret;
+
+	ret = of_property_read_string(dev->of_node, "firmware-name",
+				      &fw_name);
+	if (ret) {
+		dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+			ret);
+		return ERR_PTR(ret);
+	}
+
+	return fw_name;
+}
+
+static int k3_r5_rproc_configure(struct k3_r5_rproc *kproc)
+{
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core0, *core, *temp;
+	u32 ctrl = 0, cfg = 0, stat = 0;
+	u32 set_cfg = 0, clr_cfg = 0;
+	u64 boot_vec = 0;
+	bool lockstep_en;
+	int ret;
+
+	core0 = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+	core = cluster->mode ? core0 : kproc->core;
+
+	ret = ti_sci_proc_get_status(core->tsp, &boot_vec, &cfg, &ctrl,
+				     &stat);
+	if (ret < 0)
+		return ret;
+
+	dev_dbg(dev, "boot_vector = 0x%llx, cfg = 0x%x ctrl = 0x%x stat = 0x%x\n",
+		boot_vec, cfg, ctrl, stat);
+
+	lockstep_en = !!(stat & PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED);
+	if (!lockstep_en && cluster->mode) {
+		dev_err(cluster->dev, "lockstep mode not permitted, force configuring for split-mode\n");
+		cluster->mode = 0;
+	}
+
+	/* always enable ARM mode and set boot vector to 0 */
+	boot_vec = 0x0;
+	if (core == core0) {
+		clr_cfg = PROC_BOOT_CFG_FLAG_R5_TEINIT;
+		/*
+		 * LockStep configuration bit is Read-only on Split-mode _only_
+		 * devices and system firmware will NACK any requests with the
+		 * bit configured, so program it only on permitted devices
+		 */
+		if (lockstep_en)
+			clr_cfg |= PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+	}
+
+	if (core->atcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+
+	if (core->btcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+
+	if (core->loczrama)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+
+	if (cluster->mode) {
+		/*
+		 * work around system firmware limitations to make sure both
+		 * cores are programmed symmetrically in LockStep. LockStep
+		 * and TEINIT config is only allowed with Core0.
+		 */
+		list_for_each_entry(temp, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret)
+				goto out;
+
+			if (temp != core) {
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_TEINIT;
+			}
+			ret = ti_sci_proc_set_config(temp->tsp, boot_vec,
+						     set_cfg, clr_cfg);
+			if (ret)
+				goto out;
+		}
+
+		set_cfg = PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+		clr_cfg = 0;
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	}
+
+out:
+	return ret;
+}
+
+static int k3_r5_reserved_mem_init(struct k3_r5_rproc *kproc)
+{
+	struct device *dev = kproc->dev;
+	struct device_node *np = dev->of_node;
+	struct device_node *rmem_np;
+	struct reserved_mem *rmem;
+	int num_rmems;
+	int ret, i;
+
+	num_rmems = of_property_count_elems_of_size(np, "memory-region",
+						    sizeof(phandle));
+	if (num_rmems <= 0) {
+		dev_err(dev, "device does not have reserved memory regions, ret = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+	if (num_rmems < 2) {
+		dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+
+	/* use reserved memory region 0 for vring DMA allocations */
+	ret = of_reserved_mem_device_init_by_idx(dev, np, 0);
+	if (ret) {
+		dev_err(dev, "device cannot initialize DMA pool, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_rmems--;
+	kproc->rmem = kcalloc(num_rmems, sizeof(*kproc->rmem), GFP_KERNEL);
+	if (!kproc->rmem) {
+		ret = -ENOMEM;
+		goto release_rmem;
+	}
+
+	/* use remaining reserved memory regions for static carveouts */
+	for (i = 0; i < num_rmems; i++) {
+		rmem_np = of_parse_phandle(np, "memory-region", i + 1);
+		if (!rmem_np) {
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+
+		rmem = of_reserved_mem_lookup(rmem_np);

drivers/remoteproc/ti_k3_r5_remoteproc.c line 766

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +10 -0
+		if (!rmem) {
+			of_node_put(rmem_np);
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+		of_node_put(rmem_np);
+
+		kproc->rmem[i].bus_addr = rmem->base;
+		/* 64-bit address regions currently not supported */
+		kproc->rmem[i].dev_addr = (u32)rmem->base;

Because the bus and the device addresses are the same I have to deduce the AP and the R5 have the same view of the memory. Please add a comment to assert that is really the case.

Quoted 583 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +580 -0
+		kproc->rmem[i].size = rmem->size;
+		kproc->rmem[i].cpu_addr = ioremap_wc(rmem->base, rmem->size);
+		if (!kproc->rmem[i].cpu_addr) {
+			dev_err(dev, "failed to map reserved memory#%d at %pa of size %pa\n",
+				i + 1, &rmem->base, &rmem->size);
+			ret = -ENOMEM;
+			goto unmap_rmem;
+		}
+
+		dev_dbg(dev, "reserved memory%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			i + 1, &kproc->rmem[i].bus_addr,
+			kproc->rmem[i].size, kproc->rmem[i].cpu_addr,
+			kproc->rmem[i].dev_addr);
+	}
+	kproc->num_rmems = num_rmems;
+
+	return 0;
+
+unmap_rmem:
+	for (i--; i >= 0; i--) {
+		if (kproc->rmem[i].cpu_addr)
+			iounmap(kproc->rmem[i].cpu_addr);
+	}
+	kfree(kproc->rmem);
+release_rmem:
+	of_reserved_mem_device_release(dev);
+	return ret;
+}
+
+static void k3_r5_reserved_mem_exit(struct k3_r5_rproc *kproc)
+{
+	int i;
+
+	for (i = 0; i < kproc->num_rmems; i++)
+		iounmap(kproc->rmem[i].cpu_addr);
+	kfree(kproc->rmem);
+
+	of_reserved_mem_device_release(kproc->dev);
+}
+
+static int k3_r5_cluster_rproc_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core, *core1;
+	struct device *cdev;
+	const char *fw_name;
+	struct rproc *rproc;
+	int ret;
+
+	core1 = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+	list_for_each_entry(core, &cluster->cores, elem) {
+		cdev = core->dev;
+		fw_name = k3_r5_rproc_get_firmware(cdev);
+		if (IS_ERR(fw_name)) {
+			ret = PTR_ERR(fw_name);
+			goto out;
+		}
+
+		rproc = rproc_alloc(cdev, dev_name(cdev), &k3_r5_rproc_ops,
+				    fw_name, sizeof(*kproc));
+		if (!rproc) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		/* K3 R5s have a Region Address Translator (RAT) but no MMU */
+		rproc->has_iommu = false;
+		/* error recovery is not supported at present */
+		rproc->recovery_disabled = true;
+
+		kproc = rproc->priv;
+		kproc->cluster = cluster;
+		kproc->core = core;
+		kproc->dev = cdev;
+		kproc->rproc = rproc;
+		core->rproc = rproc;
+
+		ret = k3_r5_rproc_configure(kproc);
+		if (ret) {
+			dev_err(dev, "initial configure failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = k3_r5_reserved_mem_init(kproc);
+		if (ret) {
+			dev_err(dev, "reserved memory init failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = rproc_add(rproc);
+		if (ret) {
+			dev_err(dev, "rproc_add failed, ret = %d\n", ret);
+			goto err_add;
+		}
+
+		/* create only one rproc in lockstep mode */
+		if (cluster->mode)
+			break;
+	}
+
+	return 0;
+
+err_split:
+	rproc_del(rproc);
+err_add:
+	k3_r5_reserved_mem_exit(kproc);
+err_config:
+	rproc_free(rproc);
+	core->rproc = NULL;
+out:
+	/* undo core0 upon any failures on core1 in split-mode */
+	if (!cluster->mode && core == core1) {
+		core = list_prev_entry(core, elem);
+		rproc = core->rproc;
+		kproc = rproc->priv;
+		goto err_split;
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_rproc_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core;
+	struct rproc *rproc;
+
+	/*
+	 * lockstep mode has only one rproc associated with first core, whereas
+	 * split-mode has two rprocs associated with each core, and requires
+	 * that core1 be powered down first
+	 */
+	core = cluster->mode ?
+		list_first_entry(&cluster->cores, struct k3_r5_core, elem) :
+		list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		rproc = core->rproc;
+		kproc = rproc->priv;
+
+		rproc_del(rproc);
+
+		k3_r5_reserved_mem_exit(kproc);
+
+		rproc_free(rproc);
+		core->rproc = NULL;
+	}
+
+	return 0;
+}
+
+static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
+					       struct k3_r5_core *core)
+{
+	static const char * const mem_names[] = {"atcm", "btcm"};
+	struct device *dev = &pdev->dev;
+	struct resource *res;
+	int num_mems;
+	int i, ret;
+
+	num_mems = ARRAY_SIZE(mem_names);
+	core->mem = devm_kcalloc(dev, num_mems, sizeof(*core->mem), GFP_KERNEL);
+	if (!core->mem)
+		return -ENOMEM;
+
+	for (i = 0; i < num_mems; i++) {
+		res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						   mem_names[i]);
+		if (!res) {
+			dev_err(dev, "found no memory resource for %s\n",
+				mem_names[i]);
+			ret = -EINVAL;
+			goto fail;
+		}
+		if (!devm_request_mem_region(dev, res->start,
+					     resource_size(res),
+					     dev_name(dev))) {
+			dev_err(dev, "could not request %s region for resource\n",
+				mem_names[i]);
+			ret = -EBUSY;
+			goto fail;
+		}
+
+		/*
+		 * TCMs are designed in general to support RAM-like backing
+		 * memories. So, map these as Normal Non-Cached memories. This
+		 * also avoids/fixes any potential alignment faults due to
+		 * unaligned data accesses when using memcpy() or memset()
+		 * functions (normally seen with device type memory).
+		 */
+		core->mem[i].cpu_addr = devm_ioremap_wc(dev, res->start,
+							resource_size(res));
+		if (IS_ERR(core->mem[i].cpu_addr)) {
+			dev_err(dev, "failed to map %s memory\n", mem_names[i]);
+			ret = PTR_ERR(core->mem[i].cpu_addr);
+			devm_release_mem_region(dev, res->start,
+						resource_size(res));
+			goto fail;
+		}
+		core->mem[i].bus_addr = res->start;
+
+		/*
+		 * TODO:
+		 * The R5F cores can place ATCM & BTCM anywhere in its address
+		 * based on the corresponding Region Registers in the System
+		 * Control coprocessor. For now, place ATCM and BTCM at
+		 * addresses 0 and 0x41010000 (same as the bus address on AM65x
+		 * SoCs) based on loczrama setting
+		 */
+		if (!strcmp(mem_names[i], "atcm")) {
+			core->mem[i].dev_addr = core->loczrama ?
+							0 : K3_R5_TCM_DEV_ADDR;
+		} else {
+			core->mem[i].dev_addr = core->loczrama ?
+							K3_R5_TCM_DEV_ADDR : 0;
+		}
+		core->mem[i].size = resource_size(res);
+
+		dev_dbg(dev, "memory %8s: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			mem_names[i], &core->mem[i].bus_addr,
+			core->mem[i].size, core->mem[i].cpu_addr,
+			core->mem[i].dev_addr);
+	}
+	core->num_mems = num_mems;
+
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+	return ret;
+}
+
+static
+struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
+					  const struct ti_sci_handle *sci)
+{
+	struct ti_sci_proc *tsp;
+	u32 temp[2];
+	int ret;
+
+	ret = of_property_read_u32_array(dev->of_node, "ti,sci-proc-ids",
+					 temp, 2);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	tsp = kzalloc(sizeof(*tsp), GFP_KERNEL);
+	if (!tsp)
+		return ERR_PTR(-ENOMEM);
+
+	tsp->dev = dev;
+	tsp->sci = sci;
+	tsp->ops = &sci->ops.proc_ops;
+	tsp->proc_id = temp[0];
+	tsp->host_id = temp[1];
+
+	return tsp;
+}
+
+static int k3_r5_core_of_init(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_core *core;
+	int ret, ret1;
+
+	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
+	if (!core)
+		return -ENOMEM;
+
+	core->dev = dev;
+	core->atcm_enable = 0;
+	core->btcm_enable = 1;
+	core->loczrama = 1;
+
+	ret = of_property_read_u32(np, "atcm-enable", &core->atcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for atcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "btcm-enable", &core->btcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for btcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "loczrama", &core->loczrama);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for loczrama, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	core->ti_sci = ti_sci_get_by_phandle(np, "ti,sci");
+	if (IS_ERR(core->ti_sci)) {
+		ret = PTR_ERR(core->ti_sci);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get ti-sci handle, ret = %d\n",
+				ret);
+		}
+		core->ti_sci = NULL;
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "ti,sci-dev-id", &core->ti_sci_id);
+	if (ret) {
+		dev_err(dev, "missing 'ti,sci-dev-id' property\n");
+		goto err_sci_id;
+	}
+
+	core->reset = reset_control_get_exclusive(dev, NULL);
+	if (IS_ERR(core->reset)) {
+		ret = PTR_ERR(core->reset);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get reset handle, ret = %d\n",
+				ret);
+		}
+		goto err_sci_id;
+	}
+
+	core->tsp = k3_r5_core_of_get_tsp(dev, core->ti_sci);
+	if (IS_ERR(core->tsp)) {
+		dev_err(dev, "failed to construct ti-sci proc control, ret = %d\n",
+			ret);
+		ret = PTR_ERR(core->tsp);
+		goto err_sci_proc;
+	}
+
+	ret = ti_sci_proc_request(core->tsp);
+	if (ret < 0) {
+		dev_err(dev, "ti_sci_proc_request failed, ret = %d\n", ret);
+		goto err_proc;
+	}
+
+	ret = k3_r5_core_of_get_internal_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get internal memories, ret = %d\n",
+			ret);
+		goto err_intmem;
+	}
+
+	platform_set_drvdata(pdev, core);
+
+	return 0;
+
+err_intmem:
+	ret1 = ti_sci_proc_release(core->tsp);
+	if (ret1)
+		dev_err(dev, "failed to release proc, ret1 = %d\n", ret1);
+err_proc:
+	kfree(core->tsp);
+err_sci_proc:
+	reset_control_put(core->reset);
+err_sci_id:
+	ret1 = ti_sci_put_handle(core->ti_sci);
+	if (ret1)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret1);
+err_of:
+	devm_kfree(dev, core);
+	return ret;
+}
+
+/*
+ * free the resources explicitly since driver model is not being used
+ * for the child R5F devices
+ */
+static int k3_r5_core_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_core *core = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	int i, ret;
+
+	for (i = 0; i < core->num_mems; i++) {
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+
+	ret = ti_sci_proc_release(core->tsp);
+	if (ret)
+		dev_err(dev, "failed to release proc, ret = %d\n", ret);
+
+	kfree(core->tsp);
+	reset_control_put(core->reset);
+
+	ret = ti_sci_put_handle(core->ti_sci);
+	if (ret)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret);
+
+	platform_set_drvdata(pdev, NULL);
+	devm_kfree(dev, core);
+
+	return ret;
+}
+
+static int k3_r5_cluster_of_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct platform_device *cpdev;
+	struct device_node *child;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	for_each_available_child_of_node(np, child) {
+		cpdev = of_find_device_by_node(child);
+		if (!cpdev) {
+			ret = -ENODEV;
+			dev_err(dev, "could not get R5 core platform device\n");
+			goto fail;
+		}
+
+		ret = k3_r5_core_of_init(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_init failed, ret = %d\n",
+				ret);
+			put_device(&cpdev->dev);
+			goto fail;
+		}
+
+		core = platform_get_drvdata(cpdev);
+		put_device(&cpdev->dev);
+		list_add_tail(&core->elem, &cluster->cores);
+	}
+
+	return 0;
+
+fail:
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		if (k3_r5_core_of_exit(cpdev))
+			dev_err(dev, "k3_r5_core_of_exit cleanup failed\n");
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct platform_device *cpdev;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		ret = k3_r5_core_of_exit(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_exit failed, ret = %d\n",
+				ret);
+			break;
+		}
+	}
+
+	return ret;
+}
+
+static int k3_r5_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_cluster *cluster;
+	int ret, ret1;
+	int num_cores;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+
+	cluster->dev = dev;
+	cluster->mode = CLUSTER_MODE_LOCKSTEP;
+	INIT_LIST_HEAD(&cluster->cores);
+
+	ret = of_property_read_u32(np, "lockstep-mode", &cluster->mode);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for lockstep-mode, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_cores = of_get_available_child_count(np);
+	if (num_cores != 2) {
+		dev_err(dev, "MCU cluster requires both R5F cores to be enabled, num_cores = %d\n",
+			num_cores);
+		return -ENODEV;
+	}
+
+	platform_set_drvdata(pdev, cluster);
+
+	dev_dbg(dev, "creating child devices for R5F cores\n");
+	ret = of_platform_populate(np, NULL, NULL, dev);
+	if (ret) {
+		dev_err(dev, "of_platform_populate failed, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = k3_r5_cluster_of_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_init failed, ret = %d\n", ret);
+		goto fail_of;
+	}
+
+	ret = k3_r5_cluster_rproc_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_init failed, ret = %d\n",
+			ret);
+		goto fail_rproc;
+	}
+
+	return 0;
+
+fail_rproc:
+	ret1 = k3_r5_cluster_of_exit(pdev);
+	if (ret1)
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret1);
+fail_of:
+	of_platform_depopulate(dev);
+	return ret;
+}
+
+static int k3_r5_remove(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	ret = k3_r5_cluster_rproc_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_exit failed, ret = %d\n",
+			ret);
+		goto fail;
+	}
+
+	ret = k3_r5_cluster_of_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret);
+		goto fail;
+	}
+
+	dev_dbg(dev, "removing child devices for R5F cores\n");
+	of_platform_depopulate(dev);
+
+fail:
+	return ret;
+}
+
+static const struct of_device_id k3_r5_of_match[] = {
+	{ .compatible = "ti,am654-r5fss", },
+	{ .compatible = "ti,j721e-r5fss", },
+	{ /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, k3_r5_of_match);
+
+static struct platform_driver k3_r5_rproc_driver = {
+	.probe = k3_r5_probe,
+	.remove = k3_r5_remove,
+	.driver = {
+		.name = "k3_r5_rproc",
+		.of_match_table = k3_r5_of_match,
+	},
+};
+
+module_platform_driver(k3_r5_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("TI K3 R5F remote processor driver");
+MODULE_AUTHOR("Suman Anna <[email protected]>");
Signature · 1 line
2.23.0
author
#25 lore raw

Re: [PATCH 5/7] remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

Quoted 405 lines · Mathieu Poirier
Quoted 404 lines · Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode. The driver uses various TI-SCI interfaces to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
  drivers/remoteproc/Kconfig               |   16 +
  drivers/remoteproc/Makefile              |    1 +
  drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
  3 files changed, 1363 insertions(+)

create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c

drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
  	  This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+	tristate "TI K3 R5 remoteproc support"
+	depends on ARCH_K3
+	select MAILBOX
+	select OMAP2PLUS_MBOX
+	help
+	  Say y here to support TI's R5F remote processor subsystems
+	  on various TI K3 family of SoCs through the remote processor
+	  framework.
+
+	  You want to say y here in order to offload some processing
+	  tasks to these processors
+
+	  It's safe to say N here if you're not interested in utilizing
+	  a slave processor
+
  endif # REMOTEPROC
 
  endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y			+= qcom_wcnss_iris.o
  obj-$(CONFIG_ST_REMOTEPROC)		+= st_remoteproc.o
  obj-$(CONFIG_ST_SLIM_REMOTEPROC)	+= st_slim_rproc.o
  obj-$(CONFIG_STM32_RPROC)		+= stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)	+= ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +290 -0
new file mode 100644
index 000000000000..655f8f14c37d
@@ -0,0 +1,1346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TI K3 R5F (MCU) Remote Processor driver
+ *
+ * Copyright (C) 2017-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *	Suman Anna <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/omap-mailbox.h>
+#include <linux/reset.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+#include "omap_remoteproc.h"
+#include "remoteproc_internal.h"
+#include "ti_sci_proc.h"
+
+/* This address can either be for ATCM or BTCM with the other at address 0x0 */
+#define K3_R5_TCM_DEV_ADDR	0x41010000
+
+/* R5 TI-SCI Processor Configuration Flags */
+#define PROC_BOOT_CFG_FLAG_R5_DBG_EN			0x00000001
+#define PROC_BOOT_CFG_FLAG_R5_DBG_NIDEN			0x00000002
+#define PROC_BOOT_CFG_FLAG_R5_LOCKSTEP			0x00000100
+#define PROC_BOOT_CFG_FLAG_R5_TEINIT			0x00000200
+#define PROC_BOOT_CFG_FLAG_R5_NMFI_EN			0x00000400
+#define PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE		0x00000800
+#define PROC_BOOT_CFG_FLAG_R5_BTCM_EN			0x00001000
+#define PROC_BOOT_CFG_FLAG_R5_ATCM_EN			0x00002000
+
+/* R5 TI-SCI Processor Control Flags */
+#define PROC_BOOT_CTRL_FLAG_R5_CORE_HALT		0x00000001
+
+/* R5 TI-SCI Processor Status Flags */
+#define PROC_BOOT_STATUS_FLAG_R5_WFE			0x00000001
+#define PROC_BOOT_STATUS_FLAG_R5_WFI			0x00000002
+#define PROC_BOOT_STATUS_FLAG_R5_CLK_GATED		0x00000004
+#define PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED	0x00000100
+
+/**
+ * struct k3_r5_mem - internal memory structure
+ * @cpu_addr: MPU virtual address of the memory region
+ * @bus_addr: Bus address used to access the memory region
+ * @dev_addr: Device address from remoteproc view
+ * @size: Size of the memory region
+ */
+struct k3_r5_mem {
+	void __iomem *cpu_addr;
+	phys_addr_t bus_addr;
+	u32 dev_addr;
+	size_t size;
+};
+
+enum cluster_mode {
+	CLUSTER_MODE_SPLIT = 0,
+	CLUSTER_MODE_LOCKSTEP,
+};
+
+/**
+ * struct k3_r5_cluster - K3 R5F Cluster structure
+ * @dev: cached device pointer
+ * @mode: Mode to configure the Cluster - Split or LockStep
+ * @cores: list of R5 cores within the cluster
+ */
+struct k3_r5_cluster {
+	struct device *dev;
+	enum cluster_mode mode;
+	struct list_head cores;
+};
+
+/**
+ * struct k3_r5_core - K3 R5 core structure
+ * @elem: linked list item
+ * @dev: cached device pointer
+ * @rproc: rproc handle representing this core
+ * @mem: internal memory regions data
+ * @num_mems: number of internal memory regions
+ * @reset: reset control handle
+ * @tsp: TI-SCI processor control handle
+ * @ti_sci: TI-SCI handle
+ * @ti_sci_id: TI-SCI device identifier
+ * @atcm_enable: flag to control ATCM enablement
+ * @btcm_enable: flag to control BTCM enablement
+ * @loczrama: flag to dictate which TCM is at device address 0x0
+ */
+struct k3_r5_core {
+	struct list_head elem;
+	struct device *dev;
+	struct rproc *rproc;
+	struct k3_r5_mem *mem;
+	int num_mems;
+	struct reset_control *reset;
+	struct ti_sci_proc *tsp;
+	const struct ti_sci_handle *ti_sci;
+	u32 ti_sci_id;
+	u32 atcm_enable;
+	u32 btcm_enable;
+	u32 loczrama;
+};
+
+/**
+ * struct k3_r5_rproc - K3 remote processor state
+ * @dev: cached device pointer
+ * @cluster: cached pointer to parent cluster structure
+ * @mbox: mailbox channel handle
+ * @client: mailbox client to request the mailbox channel
+ * @rproc: rproc handle
+ * @core: cached pointer to r5 core structure being used
+ * @rmem: reserved memory regions data
+ * @num_rmems: number of reserved memory regions
+ */
+struct k3_r5_rproc {
+	struct device *dev;
+	struct k3_r5_cluster *cluster;
+	struct mbox_chan *mbox;
+	struct mbox_client client;
+	struct rproc *rproc;
+	struct k3_r5_core *core;
+	struct k3_r5_mem *rmem;
+	int num_rmems;
+};
+
+/**
+ * k3_r5_rproc_mbox_callback() - inbound mailbox message handler
+ * @client: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by the OMAP mailbox driver whenever a mailbox
+ * message is received. Usually, the mailbox payload simply contains
+ * the index of the virtqueue that is kicked by the remote processor,
+ * and we let remoteproc core handle it.
+ *
+ * In addition to virtqueue indices, we also have some out-of-band values
+ * that indicate different events. Those values are deliberately very
+ * large so they don't coincide with virtqueue indices.
+ */
+static void k3_r5_rproc_mbox_callback(struct mbox_client *client, void *data)
+{
+	struct k3_r5_rproc *kproc = container_of(client, struct k3_r5_rproc,
+						client);
+	struct device *dev = kproc->rproc->dev.parent;
+	const char *name = kproc->rproc->name;
+	u32 msg = omap_mbox_message(data);
+
+	dev_dbg(dev, "mbox msg: 0x%x\n", msg);
+
+	switch (msg) {
+	case RP_MBOX_CRASH:
+		/*
+		 * remoteproc detected an exception, but error recovery is not
+		 * supported. So, just log this for now
+		 */
+		dev_err(dev, "K3 R5F rproc %s crashed\n", name);
+		break;
+	case RP_MBOX_ECHO_REPLY:
+		dev_info(dev, "received echo reply from %s\n", name);
+		break;
+	default:
+		/* silently handle all other valid messages */
+		if (msg >= RP_MBOX_READY && msg < RP_MBOX_END_MSG)
+			return;
+		if (msg > kproc->rproc->max_notifyid) {
+			dev_dbg(dev, "dropping unknown message 0x%x", msg);
+			return;
+		}
+		/* msg contains the index of the triggered vring */
+		if (rproc_vq_interrupt(kproc->rproc, msg) == IRQ_NONE)
+			dev_dbg(dev, "no message was found in vqid %d\n", msg);
+	}
+}
+
+/* kick a virtqueue */
+static void k3_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct device *dev = rproc->dev.parent;
+	mbox_msg_t msg = (mbox_msg_t)vqid;
+	int ret;
+
+	/* send the index of the triggered virtqueue in the mailbox payload */
+	ret = mbox_send_message(kproc->mbox, (void *)msg);
+	if (ret < 0)
+		dev_err(dev, "failed to send mailbox message, status = %d\n",
+			ret);
+}
+
+static int k3_r5_split_reset(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = reset_control_assert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+			ret);
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_split_release(struct k3_r5_core *core)
+{
+	int ret;
+
+	ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+						   core->ti_sci_id);
+	if (ret) {
+		dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	ret = reset_control_deassert(core->reset);
+	if (ret) {
+		dev_err(core->dev, "local-reset deassert failed, ret = %d\n",
+			ret);
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_reset(struct k3_r5_cluster *cluster)
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* assert local reset on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = reset_control_assert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+				ret);
+			core = list_prev_entry(core, elem);
+			goto unroll_local_reset;
+		}
+	}
+
+	/* disable PSC modules on all applicable cores */
+	list_for_each_entry(core, &cluster->cores, elem) {
+		ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+				ret);
+			goto unroll_module_reset;
+		}
+	}
+
+	return 0;
+
+unroll_module_reset:
+	list_for_each_entry_continue_reverse(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+	core = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_local_reset:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (reset_control_deassert(core->reset))
+			dev_warn(core->dev, "local-reset deassert back failed\n");
+	}
+
+	return ret;
+}
+
+static int k3_r5_lockstep_release(struct k3_r5_cluster *cluster)
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 296

Quoted 6 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +6 -0
+{
+	struct k3_r5_core *core;
+	int ret;
+
+	/* enable PSC modules on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {

Out of curiosity, any HW specific reason to start with the last core?

Yeah, that is the order required by HW. We have different sequencing between LockStep and Split-modes. Please see the comments added in the descriptions for k3_r5_rproc_start() and k3_r5_rproc_stop() functions below.

Quoted 131 lines
Quoted 131 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +130 -0
+		ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+							   core->ti_sci_id);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			core = list_next_entry(core, elem);
+			goto unroll_module_reset;
+		}
+	}
+
+	/* deassert local reset on all applicable cores */
+	list_for_each_entry_reverse(core, &cluster->cores, elem) {
+		ret = reset_control_deassert(core->reset);
+		if (ret) {
+			dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+				ret);
+			goto unroll_local_reset;
+		}
+	}
+
+	return 0;
+
+unroll_local_reset:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (reset_control_assert(core->reset))
+			dev_warn(core->dev, "local-reset assert back failed\n");
+	}
+	core = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_module_reset:
+	list_for_each_entry_from(core, &cluster->cores, elem) {
+		if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+							 core->ti_sci_id))
+			dev_warn(core->dev, "module-reset assert back failed\n");
+	}
+
+	return ret;
+}
+
+static inline int k3_r5_core_halt(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       PROC_BOOT_CTRL_FLAG_R5_CORE_HALT, 0);
+}
+
+static inline int k3_r5_core_run(struct k3_r5_core *core)
+{
+	return ti_sci_proc_set_control(core->tsp,
+				       0, PROC_BOOT_CTRL_FLAG_R5_CORE_HALT);
+}
+
+/*
+ * The R5F cores have controls for both a reset and a halt/run. The code
+ * execution from DDR requires the initial boot-strapping code to be run
+ * from the internal TCMs. This function is used to release the resets on
+ * applicable cores to allow loading into the TCMs. The .prepare() ops is
+ * invoked by remoteproc core before any firmware loading, and is followed
+ * by the .start() ops after loading to actually let the R5 cores run.
+ */
+static int k3_r5_rproc_prepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_release(cluster) :
+			      k3_r5_split_release(core);
+	if (ret)
+		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
+			ret);
+
+	return ret;
+}
+
+/*
+ * This function implements the .unprepare() ops and performs the complimentary
+ * operations to that of the .prepare() ops. The function is used to assert the
+ * resets on all applicable cores for the rproc device (depending on LockStep
+ * or Split mode). This completes the second portion of powering down the R5F
+ * cores. The cores themselves are only halted in the .stop() ops, and the
+ * .unprepare() ops is invoked by the remoteproc core after the remoteproc is
+ * stopped.
+ */
+static int k3_r5_rproc_unprepare(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	struct device *dev = kproc->dev;
+	int ret;
+
+	ret = cluster->mode ? k3_r5_lockstep_reset(cluster) :
+			      k3_r5_split_reset(core);
+	if (ret)
+		dev_err(dev, "unable to disable cores, ret = %d\n", ret);
+
+	return ret;
+}
+
+/*
+ * The R5F start sequence includes two different operations
+ * 1. Configure the boot vector for R5F core(s)
+ * 2. Unhalt/Run the R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes. The LockStep
+ * mode requires the boot vector to be configured only for Core0, and then
+ * unhalt both the cores to start the execution - Core1 needs to be unhalted
+ * first followed by Core0. The Split-mode requires that Core0 to be maintained
+ * always in a higher power state that Core1 (implying Core1 needs to be started
+ * always only after Core0 is started).
+ */
+static int k3_r5_rproc_start(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct mbox_client *client = &kproc->client;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core;
+	u32 boot_addr;
+	int ret;
+
+	client->dev = dev;
+	client->tx_done = NULL;
+	client->rx_callback = k3_r5_rproc_mbox_callback;
+	client->tx_block = false;
+	client->knows_txdone = false;
+
+	kproc->mbox = mbox_request_channel(client, 0);
+	if (IS_ERR(kproc->mbox)) {
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 431

Quoted 5 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +5 -0
+		ret = -EBUSY;
+		dev_err(dev, "mbox_request_channel failed: %ld\n",
+			PTR_ERR(kproc->mbox));
+		return ret;
+	}

Does this needs to be done every time a remote processor is booted or could it be done just once in k3_r5_core_of_init()?

This is to ensure that we are not registering any mailbox callbacks until the resource table is parsed and allocated.

Quoted 12 lines
Quoted 12 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +11 -0
+
+	/*
+	 * Ping the remote processor, this is only for sanity-sake for now;
+	 * there is no functional effect whatsoever.
+	 *
+	 * Note that the reply will _not_ arrive immediately: this message
+	 * will wait in the mailbox fifo until the remote processor is booted.
+	 */
+	ret = mbox_send_message(kproc->mbox, (void *)RP_MBOX_ECHO_REQUEST);
+	if (ret < 0) {
+		dev_err(dev, "mbox_send_message failed: %d\n", ret);
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 448

Quoted 6 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +6 -0
+		goto put_mbox;
+	}
+
+	boot_addr = rproc->bootaddr;
+	/* TODO: add boot_addr sanity checking */
+	dev_err(dev, "booting R5F core using boot addr = 0x%x\n", boot_addr);

s/dev_err()/dev_dbg()

Yes, ok.

Quoted 315 lines
Quoted 315 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +314 -0
+
+	/* boot vector need not be programmed for Core1 in LockStep mode */
+	core = kproc->core;
+	ret = ti_sci_proc_set_config(core->tsp, boot_addr, 0, 0);
+	if (ret)
+		goto put_mbox;
+
+	/* unhalt/run all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry_reverse(core, &cluster->cores, elem) {
+			ret = k3_r5_core_run(core);
+			if (ret)
+				goto unroll_core_run;
+		}
+	} else {
+		ret = k3_r5_core_run(core);
+		if (ret)
+			goto put_mbox;
+	}
+
+	return 0;
+
+unroll_core_run:
+	list_for_each_entry_continue(core, &cluster->cores, elem) {
+		if (k3_r5_core_halt(core))
+			dev_warn(core->dev, "core halt back failed\n");
+	}
+put_mbox:
+	mbox_free_channel(kproc->mbox);
+	return ret;
+}
+
+/*
+ * The R5F stop function includes the following operations
+ * 1. Halt R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes, and the order
+ * of cores the operations are performed are also in general reverse to that
+ * of the start function. The LockStep mode requires each operation to be
+ * performed first on Core0 followed by Core1. The Split-mode requires that
+ * Core0 to be maintained always in a higher power state that Core1 (implying
+ * Core1 needs to be stopped first before Core0).
+ *
+ * Note that the R5F halt operation in general is not effective when the R5F
+ * core is running, but is needed to make sure the core won't run after
+ * deasserting the reset the subsequent time. The asserting of reset can
+ * be done here, but is preferred to be done in the .unprepare() ops - this
+ * maintains the symmetric behavior between the .start(), .stop(), .prepare()
+ * and .unprepare() ops, and also balances them well between sysfs 'state'
+ * flow and device bind/unbind or module removal.
+ */
+static int k3_r5_rproc_stop(struct rproc *rproc)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct k3_r5_core *core = kproc->core;
+	int ret;
+
+	/* halt all applicable cores */
+	if (cluster->mode) {
+		list_for_each_entry(core, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret) {
+				core = list_prev_entry(core, elem);
+				goto unroll_core_halt;
+			}
+		}
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+	}
+
+	mbox_free_channel(kproc->mbox);
+
+	return 0;
+
+unroll_core_halt:
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		if (k3_r5_core_run(core))
+			dev_warn(core->dev, "core run back failed\n");
+	}
+out:
+	return ret;
+}
+
+/*
+ * Internal Memory translation helper
+ *
+ * Custom function implementing the rproc .da_to_va ops to provide address
+ * translation (device address to kernel virtual address) for internal RAMs
+ * present in a DSP or IPU device). The translated addresses can be used
+ * either by the remoteproc core for loading, or by any rpmsg bus drivers.
+ */
+static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
+{
+	struct k3_r5_rproc *kproc = rproc->priv;
+	struct k3_r5_core *core = kproc->core;
+	void __iomem *va = NULL;
+	phys_addr_t bus_addr;
+	u32 dev_addr, offset;
+	size_t size;
+	int i;
+
+	if (len == 0)
+		return NULL;
+
+	/* handle both R5 and SoC views of ATCM and BTCM */
+	for (i = 0; i < core->num_mems; i++) {
+		bus_addr = core->mem[i].bus_addr;
+		dev_addr = core->mem[i].dev_addr;
+		size = core->mem[i].size;
+
+		/* handle R5-view addresses of TCMs */
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+
+		/* handle SoC-view addresses of TCMs */
+		if (da >= bus_addr && ((da + len) <= (bus_addr + size))) {
+			offset = da - bus_addr;
+			va = core->mem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	/* handle static DDR reserved memory regions */
+	for (i = 0; i < kproc->num_rmems; i++) {
+		dev_addr = kproc->rmem[i].dev_addr;
+		size = kproc->rmem[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = kproc->rmem[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
+	return NULL;
+}
+
+static const struct rproc_ops k3_r5_rproc_ops = {
+	.prepare	= k3_r5_rproc_prepare,
+	.unprepare	= k3_r5_rproc_unprepare,
+	.start		= k3_r5_rproc_start,
+	.stop		= k3_r5_rproc_stop,
+	.kick		= k3_r5_rproc_kick,
+	.da_to_va	= k3_r5_rproc_da_to_va,
+};
+
+static const char *k3_r5_rproc_get_firmware(struct device *dev)
+{
+	const char *fw_name;
+	int ret;
+
+	ret = of_property_read_string(dev->of_node, "firmware-name",
+				      &fw_name);
+	if (ret) {
+		dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+			ret);
+		return ERR_PTR(ret);
+	}
+
+	return fw_name;
+}
+
+static int k3_r5_rproc_configure(struct k3_r5_rproc *kproc)
+{
+	struct k3_r5_cluster *cluster = kproc->cluster;
+	struct device *dev = kproc->dev;
+	struct k3_r5_core *core0, *core, *temp;
+	u32 ctrl = 0, cfg = 0, stat = 0;
+	u32 set_cfg = 0, clr_cfg = 0;
+	u64 boot_vec = 0;
+	bool lockstep_en;
+	int ret;
+
+	core0 = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+	core = cluster->mode ? core0 : kproc->core;
+
+	ret = ti_sci_proc_get_status(core->tsp, &boot_vec, &cfg, &ctrl,
+				     &stat);
+	if (ret < 0)
+		return ret;
+
+	dev_dbg(dev, "boot_vector = 0x%llx, cfg = 0x%x ctrl = 0x%x stat = 0x%x\n",
+		boot_vec, cfg, ctrl, stat);
+
+	lockstep_en = !!(stat & PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED);
+	if (!lockstep_en && cluster->mode) {
+		dev_err(cluster->dev, "lockstep mode not permitted, force configuring for split-mode\n");
+		cluster->mode = 0;
+	}
+
+	/* always enable ARM mode and set boot vector to 0 */
+	boot_vec = 0x0;
+	if (core == core0) {
+		clr_cfg = PROC_BOOT_CFG_FLAG_R5_TEINIT;
+		/*
+		 * LockStep configuration bit is Read-only on Split-mode _only_
+		 * devices and system firmware will NACK any requests with the
+		 * bit configured, so program it only on permitted devices
+		 */
+		if (lockstep_en)
+			clr_cfg |= PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+	}
+
+	if (core->atcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+
+	if (core->btcm_enable)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+
+	if (core->loczrama)
+		set_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+	else
+		clr_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+
+	if (cluster->mode) {
+		/*
+		 * work around system firmware limitations to make sure both
+		 * cores are programmed symmetrically in LockStep. LockStep
+		 * and TEINIT config is only allowed with Core0.
+		 */
+		list_for_each_entry(temp, &cluster->cores, elem) {
+			ret = k3_r5_core_halt(core);
+			if (ret)
+				goto out;
+
+			if (temp != core) {
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+				clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_TEINIT;
+			}
+			ret = ti_sci_proc_set_config(temp->tsp, boot_vec,
+						     set_cfg, clr_cfg);
+			if (ret)
+				goto out;
+		}
+
+		set_cfg = PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+		clr_cfg = 0;
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	} else {
+		ret = k3_r5_core_halt(core);
+		if (ret)
+			goto out;
+
+		ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+					     set_cfg, clr_cfg);
+	}
+
+out:
+	return ret;
+}
+
+static int k3_r5_reserved_mem_init(struct k3_r5_rproc *kproc)
+{
+	struct device *dev = kproc->dev;
+	struct device_node *np = dev->of_node;
+	struct device_node *rmem_np;
+	struct reserved_mem *rmem;
+	int num_rmems;
+	int ret, i;
+
+	num_rmems = of_property_count_elems_of_size(np, "memory-region",
+						    sizeof(phandle));
+	if (num_rmems <= 0) {
+		dev_err(dev, "device does not have reserved memory regions, ret = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+	if (num_rmems < 2) {
+		dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+			num_rmems);
+		return -EINVAL;
+	}
+
+	/* use reserved memory region 0 for vring DMA allocations */
+	ret = of_reserved_mem_device_init_by_idx(dev, np, 0);
+	if (ret) {
+		dev_err(dev, "device cannot initialize DMA pool, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_rmems--;
+	kproc->rmem = kcalloc(num_rmems, sizeof(*kproc->rmem), GFP_KERNEL);
+	if (!kproc->rmem) {
+		ret = -ENOMEM;
+		goto release_rmem;
+	}
+
+	/* use remaining reserved memory regions for static carveouts */
+	for (i = 0; i < num_rmems; i++) {
+		rmem_np = of_parse_phandle(np, "memory-region", i + 1);
+		if (!rmem_np) {
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+
+		rmem = of_reserved_mem_lookup(rmem_np);
+		if (!rmem) {
+			of_node_put(rmem_np);
+			ret = -EINVAL;
+			goto unmap_rmem;
+		}
+		of_node_put(rmem_np);
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 766

Quoted 4 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +4 -0
+
+		kproc->rmem[i].bus_addr = rmem->base;
+		/* 64-bit address regions currently not supported */
+		kproc->rmem[i].dev_addr = (u32)rmem->base;

Because the bus and the device addresses are the same I have to deduce the AP and the R5 have the same view of the memory. Please add a comment to assert that is really the case.

Yes for now, since we are not using the Region Address Translator yet which would have provided address extension from 32-bit device addresses to 64-bit bus addresses.

regards
Suman

Quoted 584 lines
Quoted 584 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +580 -0
+		kproc->rmem[i].size = rmem->size;
+		kproc->rmem[i].cpu_addr = ioremap_wc(rmem->base, rmem->size);
+		if (!kproc->rmem[i].cpu_addr) {
+			dev_err(dev, "failed to map reserved memory#%d at %pa of size %pa\n",
+				i + 1, &rmem->base, &rmem->size);
+			ret = -ENOMEM;
+			goto unmap_rmem;
+		}
+
+		dev_dbg(dev, "reserved memory%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			i + 1, &kproc->rmem[i].bus_addr,
+			kproc->rmem[i].size, kproc->rmem[i].cpu_addr,
+			kproc->rmem[i].dev_addr);
+	}
+	kproc->num_rmems = num_rmems;
+
+	return 0;
+
+unmap_rmem:
+	for (i--; i >= 0; i--) {
+		if (kproc->rmem[i].cpu_addr)
+			iounmap(kproc->rmem[i].cpu_addr);
+	}
+	kfree(kproc->rmem);
+release_rmem:
+	of_reserved_mem_device_release(dev);
+	return ret;
+}
+
+static void k3_r5_reserved_mem_exit(struct k3_r5_rproc *kproc)
+{
+	int i;
+
+	for (i = 0; i < kproc->num_rmems; i++)
+		iounmap(kproc->rmem[i].cpu_addr);
+	kfree(kproc->rmem);
+
+	of_reserved_mem_device_release(kproc->dev);
+}
+
+static int k3_r5_cluster_rproc_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core, *core1;
+	struct device *cdev;
+	const char *fw_name;
+	struct rproc *rproc;
+	int ret;
+
+	core1 = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+	list_for_each_entry(core, &cluster->cores, elem) {
+		cdev = core->dev;
+		fw_name = k3_r5_rproc_get_firmware(cdev);
+		if (IS_ERR(fw_name)) {
+			ret = PTR_ERR(fw_name);
+			goto out;
+		}
+
+		rproc = rproc_alloc(cdev, dev_name(cdev), &k3_r5_rproc_ops,
+				    fw_name, sizeof(*kproc));
+		if (!rproc) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		/* K3 R5s have a Region Address Translator (RAT) but no MMU */
+		rproc->has_iommu = false;
+		/* error recovery is not supported at present */
+		rproc->recovery_disabled = true;
+
+		kproc = rproc->priv;
+		kproc->cluster = cluster;
+		kproc->core = core;
+		kproc->dev = cdev;
+		kproc->rproc = rproc;
+		core->rproc = rproc;
+
+		ret = k3_r5_rproc_configure(kproc);
+		if (ret) {
+			dev_err(dev, "initial configure failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = k3_r5_reserved_mem_init(kproc);
+		if (ret) {
+			dev_err(dev, "reserved memory init failed, ret = %d\n",
+				ret);
+			goto err_config;
+		}
+
+		ret = rproc_add(rproc);
+		if (ret) {
+			dev_err(dev, "rproc_add failed, ret = %d\n", ret);
+			goto err_add;
+		}
+
+		/* create only one rproc in lockstep mode */
+		if (cluster->mode)
+			break;
+	}
+
+	return 0;
+
+err_split:
+	rproc_del(rproc);
+err_add:
+	k3_r5_reserved_mem_exit(kproc);
+err_config:
+	rproc_free(rproc);
+	core->rproc = NULL;
+out:
+	/* undo core0 upon any failures on core1 in split-mode */
+	if (!cluster->mode && core == core1) {
+		core = list_prev_entry(core, elem);
+		rproc = core->rproc;
+		kproc = rproc->priv;
+		goto err_split;
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_rproc_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct k3_r5_rproc *kproc;
+	struct k3_r5_core *core;
+	struct rproc *rproc;
+
+	/*
+	 * lockstep mode has only one rproc associated with first core, whereas
+	 * split-mode has two rprocs associated with each core, and requires
+	 * that core1 be powered down first
+	 */
+	core = cluster->mode ?
+		list_first_entry(&cluster->cores, struct k3_r5_core, elem) :
+		list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+
+	list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+		rproc = core->rproc;
+		kproc = rproc->priv;
+
+		rproc_del(rproc);
+
+		k3_r5_reserved_mem_exit(kproc);
+
+		rproc_free(rproc);
+		core->rproc = NULL;
+	}
+
+	return 0;
+}
+
+static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
+					       struct k3_r5_core *core)
+{
+	static const char * const mem_names[] = {"atcm", "btcm"};
+	struct device *dev = &pdev->dev;
+	struct resource *res;
+	int num_mems;
+	int i, ret;
+
+	num_mems = ARRAY_SIZE(mem_names);
+	core->mem = devm_kcalloc(dev, num_mems, sizeof(*core->mem), GFP_KERNEL);
+	if (!core->mem)
+		return -ENOMEM;
+
+	for (i = 0; i < num_mems; i++) {
+		res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+						   mem_names[i]);
+		if (!res) {
+			dev_err(dev, "found no memory resource for %s\n",
+				mem_names[i]);
+			ret = -EINVAL;
+			goto fail;
+		}
+		if (!devm_request_mem_region(dev, res->start,
+					     resource_size(res),
+					     dev_name(dev))) {
+			dev_err(dev, "could not request %s region for resource\n",
+				mem_names[i]);
+			ret = -EBUSY;
+			goto fail;
+		}
+
+		/*
+		 * TCMs are designed in general to support RAM-like backing
+		 * memories. So, map these as Normal Non-Cached memories. This
+		 * also avoids/fixes any potential alignment faults due to
+		 * unaligned data accesses when using memcpy() or memset()
+		 * functions (normally seen with device type memory).
+		 */
+		core->mem[i].cpu_addr = devm_ioremap_wc(dev, res->start,
+							resource_size(res));
+		if (IS_ERR(core->mem[i].cpu_addr)) {
+			dev_err(dev, "failed to map %s memory\n", mem_names[i]);
+			ret = PTR_ERR(core->mem[i].cpu_addr);
+			devm_release_mem_region(dev, res->start,
+						resource_size(res));
+			goto fail;
+		}
+		core->mem[i].bus_addr = res->start;
+
+		/*
+		 * TODO:
+		 * The R5F cores can place ATCM & BTCM anywhere in its address
+		 * based on the corresponding Region Registers in the System
+		 * Control coprocessor. For now, place ATCM and BTCM at
+		 * addresses 0 and 0x41010000 (same as the bus address on AM65x
+		 * SoCs) based on loczrama setting
+		 */
+		if (!strcmp(mem_names[i], "atcm")) {
+			core->mem[i].dev_addr = core->loczrama ?
+							0 : K3_R5_TCM_DEV_ADDR;
+		} else {
+			core->mem[i].dev_addr = core->loczrama ?
+							K3_R5_TCM_DEV_ADDR : 0;
+		}
+		core->mem[i].size = resource_size(res);
+
+		dev_dbg(dev, "memory %8s: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			mem_names[i], &core->mem[i].bus_addr,
+			core->mem[i].size, core->mem[i].cpu_addr,
+			core->mem[i].dev_addr);
+	}
+	core->num_mems = num_mems;
+
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+	return ret;
+}
+
+static
+struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
+					  const struct ti_sci_handle *sci)
+{
+	struct ti_sci_proc *tsp;
+	u32 temp[2];
+	int ret;
+
+	ret = of_property_read_u32_array(dev->of_node, "ti,sci-proc-ids",
+					 temp, 2);
+	if (ret < 0)
+		return ERR_PTR(ret);
+
+	tsp = kzalloc(sizeof(*tsp), GFP_KERNEL);
+	if (!tsp)
+		return ERR_PTR(-ENOMEM);
+
+	tsp->dev = dev;
+	tsp->sci = sci;
+	tsp->ops = &sci->ops.proc_ops;
+	tsp->proc_id = temp[0];
+	tsp->host_id = temp[1];
+
+	return tsp;
+}
+
+static int k3_r5_core_of_init(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_core *core;
+	int ret, ret1;
+
+	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
+	if (!core)
+		return -ENOMEM;
+
+	core->dev = dev;
+	core->atcm_enable = 0;
+	core->btcm_enable = 1;
+	core->loczrama = 1;
+
+	ret = of_property_read_u32(np, "atcm-enable", &core->atcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for atcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "btcm-enable", &core->btcm_enable);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for btcm-enable, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "loczrama", &core->loczrama);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for loczrama, ret = %d\n", ret);
+		goto err_of;
+	}
+
+	core->ti_sci = ti_sci_get_by_phandle(np, "ti,sci");
+	if (IS_ERR(core->ti_sci)) {
+		ret = PTR_ERR(core->ti_sci);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get ti-sci handle, ret = %d\n",
+				ret);
+		}
+		core->ti_sci = NULL;
+		goto err_of;
+	}
+
+	ret = of_property_read_u32(np, "ti,sci-dev-id", &core->ti_sci_id);
+	if (ret) {
+		dev_err(dev, "missing 'ti,sci-dev-id' property\n");
+		goto err_sci_id;
+	}
+
+	core->reset = reset_control_get_exclusive(dev, NULL);
+	if (IS_ERR(core->reset)) {
+		ret = PTR_ERR(core->reset);
+		if (ret != -EPROBE_DEFER) {
+			dev_err(dev, "failed to get reset handle, ret = %d\n",
+				ret);
+		}
+		goto err_sci_id;
+	}
+
+	core->tsp = k3_r5_core_of_get_tsp(dev, core->ti_sci);
+	if (IS_ERR(core->tsp)) {
+		dev_err(dev, "failed to construct ti-sci proc control, ret = %d\n",
+			ret);
+		ret = PTR_ERR(core->tsp);
+		goto err_sci_proc;
+	}
+
+	ret = ti_sci_proc_request(core->tsp);
+	if (ret < 0) {
+		dev_err(dev, "ti_sci_proc_request failed, ret = %d\n", ret);
+		goto err_proc;
+	}
+
+	ret = k3_r5_core_of_get_internal_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get internal memories, ret = %d\n",
+			ret);
+		goto err_intmem;
+	}
+
+	platform_set_drvdata(pdev, core);
+
+	return 0;
+
+err_intmem:
+	ret1 = ti_sci_proc_release(core->tsp);
+	if (ret1)
+		dev_err(dev, "failed to release proc, ret1 = %d\n", ret1);
+err_proc:
+	kfree(core->tsp);
+err_sci_proc:
+	reset_control_put(core->reset);
+err_sci_id:
+	ret1 = ti_sci_put_handle(core->ti_sci);
+	if (ret1)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret1);
+err_of:
+	devm_kfree(dev, core);
+	return ret;
+}
+
+/*
+ * free the resources explicitly since driver model is not being used
+ * for the child R5F devices
+ */
+static int k3_r5_core_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_core *core = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	int i, ret;
+
+	for (i = 0; i < core->num_mems; i++) {
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+	}
+	if (core->mem)
+		devm_kfree(dev, core->mem);
+
+	ret = ti_sci_proc_release(core->tsp);
+	if (ret)
+		dev_err(dev, "failed to release proc, ret = %d\n", ret);
+
+	kfree(core->tsp);
+	reset_control_put(core->reset);
+
+	ret = ti_sci_put_handle(core->ti_sci);
+	if (ret)
+		dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret);
+
+	platform_set_drvdata(pdev, NULL);
+	devm_kfree(dev, core);
+
+	return ret;
+}
+
+static int k3_r5_cluster_of_init(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct platform_device *cpdev;
+	struct device_node *child;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	for_each_available_child_of_node(np, child) {
+		cpdev = of_find_device_by_node(child);
+		if (!cpdev) {
+			ret = -ENODEV;
+			dev_err(dev, "could not get R5 core platform device\n");
+			goto fail;
+		}
+
+		ret = k3_r5_core_of_init(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_init failed, ret = %d\n",
+				ret);
+			put_device(&cpdev->dev);
+			goto fail;
+		}
+
+		core = platform_get_drvdata(cpdev);
+		put_device(&cpdev->dev);
+		list_add_tail(&core->elem, &cluster->cores);
+	}
+
+	return 0;
+
+fail:
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		if (k3_r5_core_of_exit(cpdev))
+			dev_err(dev, "k3_r5_core_of_exit cleanup failed\n");
+	}
+	return ret;
+}
+
+static int k3_r5_cluster_of_exit(struct platform_device *pdev)
+{
+	struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+	struct device *dev = &pdev->dev;
+	struct platform_device *cpdev;
+	struct k3_r5_core *core, *temp;
+	int ret;
+
+	list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+		list_del(&core->elem);
+		cpdev = to_platform_device(core->dev);
+		ret = k3_r5_core_of_exit(cpdev);
+		if (ret) {
+			dev_err(dev, "k3_r5_core_of_exit failed, ret = %d\n",
+				ret);
+			break;
+		}
+	}
+
+	return ret;
+}
+
+static int k3_r5_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	struct k3_r5_cluster *cluster;
+	int ret, ret1;
+	int num_cores;
+
+	cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+	if (!cluster)
+		return -ENOMEM;
+
+	cluster->dev = dev;
+	cluster->mode = CLUSTER_MODE_LOCKSTEP;
+	INIT_LIST_HEAD(&cluster->cores);
+
+	ret = of_property_read_u32(np, "lockstep-mode", &cluster->mode);
+	if (ret < 0 && ret != -EINVAL) {
+		dev_err(dev, "invalid format for lockstep-mode, ret = %d\n",
+			ret);
+		return ret;
+	}
+
+	num_cores = of_get_available_child_count(np);
+	if (num_cores != 2) {
+		dev_err(dev, "MCU cluster requires both R5F cores to be enabled, num_cores = %d\n",
+			num_cores);
+		return -ENODEV;
+	}
+
+	platform_set_drvdata(pdev, cluster);
+
+	dev_dbg(dev, "creating child devices for R5F cores\n");
+	ret = of_platform_populate(np, NULL, NULL, dev);
+	if (ret) {
+		dev_err(dev, "of_platform_populate failed, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = k3_r5_cluster_of_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_init failed, ret = %d\n", ret);
+		goto fail_of;
+	}
+
+	ret = k3_r5_cluster_rproc_init(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_init failed, ret = %d\n",
+			ret);
+		goto fail_rproc;
+	}
+
+	return 0;
+
+fail_rproc:
+	ret1 = k3_r5_cluster_of_exit(pdev);
+	if (ret1)
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret1);
+fail_of:
+	of_platform_depopulate(dev);
+	return ret;
+}
+
+static int k3_r5_remove(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	ret = k3_r5_cluster_rproc_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_rproc_exit failed, ret = %d\n",
+			ret);
+		goto fail;
+	}
+
+	ret = k3_r5_cluster_of_exit(pdev);
+	if (ret) {
+		dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret);
+		goto fail;
+	}
+
+	dev_dbg(dev, "removing child devices for R5F cores\n");
+	of_platform_depopulate(dev);
+
+fail:
+	return ret;
+}
+
+static const struct of_device_id k3_r5_of_match[] = {
+	{ .compatible = "ti,am654-r5fss", },
+	{ .compatible = "ti,j721e-r5fss", },
+	{ /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, k3_r5_of_match);
+
+static struct platform_driver k3_r5_rproc_driver = {
+	.probe = k3_r5_probe,
+	.remove = k3_r5_remove,
+	.driver = {
+		.name = "k3_r5_rproc",
+		.of_match_table = k3_r5_of_match,
+	},
+};
+
+module_platform_driver(k3_r5_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("TI K3 R5F remote processor driver");
+MODULE_AUTHOR("Suman Anna <[email protected]>");
Signature · 1 line
2.23.0
maintainer
#26 lore raw

Re: [PATCH 5/7] remoteproc/k3-r5: Add a remoteproc driver for R5F subsystem

Quoted 416 lines · Suman Anna
Quoted 405 lines · Mathieu Poirier
Quoted 404 lines · Suman Anna

The TI K3 family of SoCs typically have one or more dual-core Arm Cortex R5F processor clusters/subsystems (R5FSS). This R5F subsystem/cluster can be configured at boot time to be either run in a LockStep mode or in an Asymmetric Multi Processing (AMP) fashion in Split-mode. This subsystem has 64 KB each Tightly-Coupled Memory (TCM) internal memories for each core split between two banks - TCMA and TCMB (further interleaved into two banks). The subsystem does not have an MMU, but has a Region Address Translater (RAT) module that is accessible only from the R5Fs for providing translations between 32-bit CPU addresses into larger system bus addresses.

Add a remoteproc driver to support this subsystem to be able to load and boot the R5F cores primarily in LockStep mode. The code also includes the base support for Split mode. Error Recovery and Power Management features are not currently supported. Loading support includes the internal TCMs and DDR. RAT support is left for a future patch, and as such the reserved memory carveout regions are all expected to be using memory regions within the first 2 GB.

The R5F remote processors do not have an MMU, and so require fixed memory carveout regions matching the firmware image addresses. Support for this is provided by mandating multiple memory regions to be attached to the remoteproc device. The first memory region will be used to serve as the DMA pool for all dynamic allocations like the vrings and vring buffers. The remaining memory regions are mapped into the kernel at device probe time, and are used to provide address translations for firmware image segments without the need for any RSC_CARVEOUT entries. Any firmware image using memory outside of the supplied reserved memory carveout regions will be errored out.

The R5F processors on TI K3 SoCs require a specific sequence for booting and shutting down the processors. This sequence is also dependent on the mode (LockStep or Split) the R5F cluster is configured for. The R5F cores have a Memory Protection Unit (MPU) that has a default configuration that does not allow the cores to run out of DDR out of reset. This is resolved by using the TCMs for boot-strapping code that applies the appropriate executable permissions on desired DDR memory. The loading into the TCMs requires that the resets be released first with the cores in halted state. The Power Sleep Controller (PSC) module on K3 SoCs requires that the cores be in WFI/WFE states with no active bus transactions before the cores can be put back into reset. Support for this is provided by using the newly introduced .prepare() and .unprepare() ops in the remoteproc core. The .prepare() ops is invoked before any loading, and the .unprepare() ops is invoked after the remoteproc resource cleanup. The R5F core resets are deasserted in .prepare() and asserted in .unprepare(), and the cores themselves are started and halted in .start() and .stop() ops. This ensures symmetric usage and allows the R5F cores state machine to be maintained properly between using the sysfs 'state' variable, bind/unbind and regular module load/unload flows.

The subsystem is represented as a single remoteproc in LockStep mode, and as two remoteprocs in Split mode. The driver uses various TI-SCI interfaces to talk to the System Controller (DMSC) for managing configuration, power and reset management of these cores. IPC between the A53 cores and the R5 cores is supported through the virtio rpmsg stack using shared memory and OMAP Mailboxes.

The AM65x SoCs typically have a single R5FSS in the MCU voltage domain. The J721E SoCs uses a slightly revised IP and typically have three R5FSSs, with one cluster present within the MCU voltage domain (MCU_R5FSS0), and the remaining two clusters present in the MAIN voltage domain (MAIN_R5FSS0 and MAIN_R5FSS1). The integration of these clusters on J721E SoC is also slightly different in that these IPs do support an actual local reset line, while they are a no-op on AM65x SoCs.

---
  drivers/remoteproc/Kconfig               |   16 +
  drivers/remoteproc/Makefile              |    1 +
  drivers/remoteproc/ti_k3_r5_remoteproc.c | 1346 ++++++++++++++++++++++
  3 files changed, 1363 insertions(+)

create mode 100644 drivers/remoteproc/ti_k3_r5_remoteproc.c

drivers/remoteproc/Kconfig +16 -0
index de3862c15fcc..073048b4c0fb 100644
@@ -224,6 +224,22 @@ config STM32_RPROC
 
       This can be either built-in or a loadable module.
 
+config TI_K3_R5_REMOTEPROC
+    tristate "TI K3 R5 remoteproc support"
+    depends on ARCH_K3
+    select MAILBOX
+    select OMAP2PLUS_MBOX
+    help
+      Say y here to support TI's R5F remote processor subsystems
+      on various TI K3 family of SoCs through the remote processor
+      framework.
+
+      You want to say y here in order to offload some processing
+      tasks to these processors
+
+      It's safe to say N here if you're not interested in utilizing
+      a slave processor
+
  endif # REMOTEPROC
 
  endmenu
drivers/remoteproc/Makefile +1 -0
index e30a1b15fbac..00ba826818af 100644
@@ -28,3 +28,4 @@ qcom_wcnss_pil-y                   += qcom_wcnss_iris.o
  obj-$(CONFIG_ST_REMOTEPROC)                += st_remoteproc.o
  obj-$(CONFIG_ST_SLIM_REMOTEPROC)   += st_slim_rproc.o
  obj-$(CONFIG_STM32_RPROC)          += stm32_rproc.o
+obj-$(CONFIG_TI_K3_R5_REMOTEPROC)   += ti_k3_r5_remoteproc.o
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +290 -0
new file mode 100644
index 000000000000..655f8f14c37d
@@ -0,0 +1,1346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * TI K3 R5F (MCU) Remote Processor driver
+ *
+ * Copyright (C) 2017-2020 Texas Instruments Incorporated - http://www.ti.com/
+ *  Suman Anna <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/of_address.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/remoteproc.h>
+#include <linux/omap-mailbox.h>
+#include <linux/reset.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+#include "omap_remoteproc.h"
+#include "remoteproc_internal.h"
+#include "ti_sci_proc.h"
+
+/* This address can either be for ATCM or BTCM with the other at address 0x0 */
+#define K3_R5_TCM_DEV_ADDR  0x41010000
+
+/* R5 TI-SCI Processor Configuration Flags */
+#define PROC_BOOT_CFG_FLAG_R5_DBG_EN                        0x00000001
+#define PROC_BOOT_CFG_FLAG_R5_DBG_NIDEN                     0x00000002
+#define PROC_BOOT_CFG_FLAG_R5_LOCKSTEP                      0x00000100
+#define PROC_BOOT_CFG_FLAG_R5_TEINIT                        0x00000200
+#define PROC_BOOT_CFG_FLAG_R5_NMFI_EN                       0x00000400
+#define PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE           0x00000800
+#define PROC_BOOT_CFG_FLAG_R5_BTCM_EN                       0x00001000
+#define PROC_BOOT_CFG_FLAG_R5_ATCM_EN                       0x00002000
+
+/* R5 TI-SCI Processor Control Flags */
+#define PROC_BOOT_CTRL_FLAG_R5_CORE_HALT            0x00000001
+
+/* R5 TI-SCI Processor Status Flags */
+#define PROC_BOOT_STATUS_FLAG_R5_WFE                        0x00000001
+#define PROC_BOOT_STATUS_FLAG_R5_WFI                        0x00000002
+#define PROC_BOOT_STATUS_FLAG_R5_CLK_GATED          0x00000004
+#define PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED 0x00000100
+
+/**
+ * struct k3_r5_mem - internal memory structure
+ * @cpu_addr: MPU virtual address of the memory region
+ * @bus_addr: Bus address used to access the memory region
+ * @dev_addr: Device address from remoteproc view
+ * @size: Size of the memory region
+ */
+struct k3_r5_mem {
+    void __iomem *cpu_addr;
+    phys_addr_t bus_addr;
+    u32 dev_addr;
+    size_t size;
+};
+
+enum cluster_mode {
+    CLUSTER_MODE_SPLIT = 0,
+    CLUSTER_MODE_LOCKSTEP,
+};
+
+/**
+ * struct k3_r5_cluster - K3 R5F Cluster structure
+ * @dev: cached device pointer
+ * @mode: Mode to configure the Cluster - Split or LockStep
+ * @cores: list of R5 cores within the cluster
+ */
+struct k3_r5_cluster {
+    struct device *dev;
+    enum cluster_mode mode;
+    struct list_head cores;
+};
+
+/**
+ * struct k3_r5_core - K3 R5 core structure
+ * @elem: linked list item
+ * @dev: cached device pointer
+ * @rproc: rproc handle representing this core
+ * @mem: internal memory regions data
+ * @num_mems: number of internal memory regions
+ * @reset: reset control handle
+ * @tsp: TI-SCI processor control handle
+ * @ti_sci: TI-SCI handle
+ * @ti_sci_id: TI-SCI device identifier
+ * @atcm_enable: flag to control ATCM enablement
+ * @btcm_enable: flag to control BTCM enablement
+ * @loczrama: flag to dictate which TCM is at device address 0x0
+ */
+struct k3_r5_core {
+    struct list_head elem;
+    struct device *dev;
+    struct rproc *rproc;
+    struct k3_r5_mem *mem;
+    int num_mems;
+    struct reset_control *reset;
+    struct ti_sci_proc *tsp;
+    const struct ti_sci_handle *ti_sci;
+    u32 ti_sci_id;
+    u32 atcm_enable;
+    u32 btcm_enable;
+    u32 loczrama;
+};
+
+/**
+ * struct k3_r5_rproc - K3 remote processor state
+ * @dev: cached device pointer
+ * @cluster: cached pointer to parent cluster structure
+ * @mbox: mailbox channel handle
+ * @client: mailbox client to request the mailbox channel
+ * @rproc: rproc handle
+ * @core: cached pointer to r5 core structure being used
+ * @rmem: reserved memory regions data
+ * @num_rmems: number of reserved memory regions
+ */
+struct k3_r5_rproc {
+    struct device *dev;
+    struct k3_r5_cluster *cluster;
+    struct mbox_chan *mbox;
+    struct mbox_client client;
+    struct rproc *rproc;
+    struct k3_r5_core *core;
+    struct k3_r5_mem *rmem;
+    int num_rmems;
+};
+
+/**
+ * k3_r5_rproc_mbox_callback() - inbound mailbox message handler
+ * @client: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by the OMAP mailbox driver whenever a mailbox
+ * message is received. Usually, the mailbox payload simply contains
+ * the index of the virtqueue that is kicked by the remote processor,
+ * and we let remoteproc core handle it.
+ *
+ * In addition to virtqueue indices, we also have some out-of-band values
+ * that indicate different events. Those values are deliberately very
+ * large so they don't coincide with virtqueue indices.
+ */
+static void k3_r5_rproc_mbox_callback(struct mbox_client *client, void *data)
+{
+    struct k3_r5_rproc *kproc = container_of(client, struct k3_r5_rproc,
+                                            client);
+    struct device *dev = kproc->rproc->dev.parent;
+    const char *name = kproc->rproc->name;
+    u32 msg = omap_mbox_message(data);
+
+    dev_dbg(dev, "mbox msg: 0x%x\n", msg);
+
+    switch (msg) {
+    case RP_MBOX_CRASH:
+            /*
+             * remoteproc detected an exception, but error recovery is not
+             * supported. So, just log this for now
+             */
+            dev_err(dev, "K3 R5F rproc %s crashed\n", name);
+            break;
+    case RP_MBOX_ECHO_REPLY:
+            dev_info(dev, "received echo reply from %s\n", name);
+            break;
+    default:
+            /* silently handle all other valid messages */
+            if (msg >= RP_MBOX_READY && msg < RP_MBOX_END_MSG)
+                    return;
+            if (msg > kproc->rproc->max_notifyid) {
+                    dev_dbg(dev, "dropping unknown message 0x%x", msg);
+                    return;
+            }
+            /* msg contains the index of the triggered vring */
+            if (rproc_vq_interrupt(kproc->rproc, msg) == IRQ_NONE)
+                    dev_dbg(dev, "no message was found in vqid %d\n", msg);
+    }
+}
+
+/* kick a virtqueue */
+static void k3_r5_rproc_kick(struct rproc *rproc, int vqid)
+{
+    struct k3_r5_rproc *kproc = rproc->priv;
+    struct device *dev = rproc->dev.parent;
+    mbox_msg_t msg = (mbox_msg_t)vqid;
+    int ret;
+
+    /* send the index of the triggered virtqueue in the mailbox payload */
+    ret = mbox_send_message(kproc->mbox, (void *)msg);
+    if (ret < 0)
+            dev_err(dev, "failed to send mailbox message, status = %d\n",
+                    ret);
+}
+
+static int k3_r5_split_reset(struct k3_r5_core *core)
+{
+    int ret;
+
+    ret = reset_control_assert(core->reset);
+    if (ret) {
+            dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+                    ret);
+            return ret;
+    }
+
+    ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+                                               core->ti_sci_id);
+    if (ret) {
+            dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+                    ret);
+            if (reset_control_deassert(core->reset))
+                    dev_warn(core->dev, "local-reset deassert back failed\n");
+    }
+
+    return ret;
+}
+
+static int k3_r5_split_release(struct k3_r5_core *core)
+{
+    int ret;
+
+    ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+                                               core->ti_sci_id);
+    if (ret) {
+            dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+                    ret);
+            return ret;
+    }
+
+    ret = reset_control_deassert(core->reset);
+    if (ret) {
+            dev_err(core->dev, "local-reset deassert failed, ret = %d\n",
+                    ret);
+            if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+                                                     core->ti_sci_id))
+                    dev_warn(core->dev, "module-reset assert back failed\n");
+    }
+
+    return ret;
+}
+
+static int k3_r5_lockstep_reset(struct k3_r5_cluster *cluster)
+{
+    struct k3_r5_core *core;
+    int ret;
+
+    /* assert local reset on all applicable cores */
+    list_for_each_entry(core, &cluster->cores, elem) {
+            ret = reset_control_assert(core->reset);
+            if (ret) {
+                    dev_err(core->dev, "local-reset assert failed, ret = %d\n",
+                            ret);
+                    core = list_prev_entry(core, elem);
+                    goto unroll_local_reset;
+            }
+    }
+
+    /* disable PSC modules on all applicable cores */
+    list_for_each_entry(core, &cluster->cores, elem) {
+            ret = core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+                                                       core->ti_sci_id);
+            if (ret) {
+                    dev_err(core->dev, "module-reset assert failed, ret = %d\n",
+                            ret);
+                    goto unroll_module_reset;
+            }
+    }
+
+    return 0;
+
+unroll_module_reset:
+    list_for_each_entry_continue_reverse(core, &cluster->cores, elem) {
+            if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+                                                     core->ti_sci_id))
+                    dev_warn(core->dev, "module-reset assert back failed\n");
+    }
+    core = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_local_reset:
+    list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+            if (reset_control_deassert(core->reset))
+                    dev_warn(core->dev, "local-reset deassert back failed\n");
+    }
+
+    return ret;
+}
+
+static int k3_r5_lockstep_release(struct k3_r5_cluster *cluster)
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 296

Quoted 6 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +6 -0
+{
+    struct k3_r5_core *core;
+    int ret;
+
+    /* enable PSC modules on all applicable cores */
+    list_for_each_entry_reverse(core, &cluster->cores, elem) {

Out of curiosity, any HW specific reason to start with the last core?

Quoted 4 lines

Yeah, that is the order required by HW. We have different sequencing between LockStep and Split-modes. Please see the comments added in the descriptions for k3_r5_rproc_start() and k3_r5_rproc_stop() functions below.

Thanks for the explanation.

Quoted 140 lines
Quoted 131 lines
Quoted 131 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +130 -0
+            ret = core->ti_sci->ops.dev_ops.get_device(core->ti_sci,
+                                                       core->ti_sci_id);
+            if (ret) {
+                    dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+                            ret);
+                    core = list_next_entry(core, elem);
+                    goto unroll_module_reset;
+            }
+    }
+
+    /* deassert local reset on all applicable cores */
+    list_for_each_entry_reverse(core, &cluster->cores, elem) {
+            ret = reset_control_deassert(core->reset);
+            if (ret) {
+                    dev_err(core->dev, "module-reset deassert failed, ret = %d\n",
+                            ret);
+                    goto unroll_local_reset;
+            }
+    }
+
+    return 0;
+
+unroll_local_reset:
+    list_for_each_entry_continue(core, &cluster->cores, elem) {
+            if (reset_control_assert(core->reset))
+                    dev_warn(core->dev, "local-reset assert back failed\n");
+    }
+    core = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+unroll_module_reset:
+    list_for_each_entry_from(core, &cluster->cores, elem) {
+            if (core->ti_sci->ops.dev_ops.put_device(core->ti_sci,
+                                                     core->ti_sci_id))
+                    dev_warn(core->dev, "module-reset assert back failed\n");
+    }
+
+    return ret;
+}
+
+static inline int k3_r5_core_halt(struct k3_r5_core *core)
+{
+    return ti_sci_proc_set_control(core->tsp,
+                                   PROC_BOOT_CTRL_FLAG_R5_CORE_HALT, 0);
+}
+
+static inline int k3_r5_core_run(struct k3_r5_core *core)
+{
+    return ti_sci_proc_set_control(core->tsp,
+                                   0, PROC_BOOT_CTRL_FLAG_R5_CORE_HALT);
+}
+
+/*
+ * The R5F cores have controls for both a reset and a halt/run. The code
+ * execution from DDR requires the initial boot-strapping code to be run
+ * from the internal TCMs. This function is used to release the resets on
+ * applicable cores to allow loading into the TCMs. The .prepare() ops is
+ * invoked by remoteproc core before any firmware loading, and is followed
+ * by the .start() ops after loading to actually let the R5 cores run.
+ */
+static int k3_r5_rproc_prepare(struct rproc *rproc)
+{
+    struct k3_r5_rproc *kproc = rproc->priv;
+    struct k3_r5_cluster *cluster = kproc->cluster;
+    struct k3_r5_core *core = kproc->core;
+    struct device *dev = kproc->dev;
+    int ret;
+
+    ret = cluster->mode ? k3_r5_lockstep_release(cluster) :
+                          k3_r5_split_release(core);
+    if (ret)
+            dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
+                    ret);
+
+    return ret;
+}
+
+/*
+ * This function implements the .unprepare() ops and performs the complimentary
+ * operations to that of the .prepare() ops. The function is used to assert the
+ * resets on all applicable cores for the rproc device (depending on LockStep
+ * or Split mode). This completes the second portion of powering down the R5F
+ * cores. The cores themselves are only halted in the .stop() ops, and the
+ * .unprepare() ops is invoked by the remoteproc core after the remoteproc is
+ * stopped.
+ */
+static int k3_r5_rproc_unprepare(struct rproc *rproc)
+{
+    struct k3_r5_rproc *kproc = rproc->priv;
+    struct k3_r5_cluster *cluster = kproc->cluster;
+    struct k3_r5_core *core = kproc->core;
+    struct device *dev = kproc->dev;
+    int ret;
+
+    ret = cluster->mode ? k3_r5_lockstep_reset(cluster) :
+                          k3_r5_split_reset(core);
+    if (ret)
+            dev_err(dev, "unable to disable cores, ret = %d\n", ret);
+
+    return ret;
+}
+
+/*
+ * The R5F start sequence includes two different operations
+ * 1. Configure the boot vector for R5F core(s)
+ * 2. Unhalt/Run the R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes. The LockStep
+ * mode requires the boot vector to be configured only for Core0, and then
+ * unhalt both the cores to start the execution - Core1 needs to be unhalted
+ * first followed by Core0. The Split-mode requires that Core0 to be maintained
+ * always in a higher power state that Core1 (implying Core1 needs to be started
+ * always only after Core0 is started).
+ */
+static int k3_r5_rproc_start(struct rproc *rproc)
+{
+    struct k3_r5_rproc *kproc = rproc->priv;
+    struct k3_r5_cluster *cluster = kproc->cluster;
+    struct mbox_client *client = &kproc->client;
+    struct device *dev = kproc->dev;
+    struct k3_r5_core *core;
+    u32 boot_addr;
+    int ret;
+
+    client->dev = dev;
+    client->tx_done = NULL;
+    client->rx_callback = k3_r5_rproc_mbox_callback;
+    client->tx_block = false;
+    client->knows_txdone = false;
+
+    kproc->mbox = mbox_request_channel(client, 0);
+    if (IS_ERR(kproc->mbox)) {
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 431

Quoted 5 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +5 -0
+            ret = -EBUSY;
+            dev_err(dev, "mbox_request_channel failed: %ld\n",
+                    PTR_ERR(kproc->mbox));
+            return ret;
+    }

Does this needs to be done every time a remote processor is booted or could it be done just once in k3_r5_core_of_init()?

Quoted 2 lines

This is to ensure that we are not registering any mailbox callbacks until the resource table is parsed and allocated.

I don't see how the callbacks would be used if the resource table isn't parsed... And if that is somehow and issue callbacks should simply test if resources are available before proceeding. In the end there is nothing wrong with this code, I just find it cumbersome. But this is a platform driver so the decision is entirely yours.

Quoted 348 lines
Quoted 13 lines
Quoted 13 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +11 -0
+
+    /*
+     * Ping the remote processor, this is only for sanity-sake for now;
+     * there is no functional effect whatsoever.
+     *
+     * Note that the reply will _not_ arrive immediately: this message
+     * will wait in the mailbox fifo until the remote processor is booted.
+     */
+    ret = mbox_send_message(kproc->mbox, (void *)RP_MBOX_ECHO_REQUEST);
+    if (ret < 0) {
+            dev_err(dev, "mbox_send_message failed: %d\n", ret);
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 448

Quoted 6 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +6 -0
+            goto put_mbox;
+    }
+
+    boot_addr = rproc->bootaddr;
+    /* TODO: add boot_addr sanity checking */
+    dev_err(dev, "booting R5F core using boot addr = 0x%x\n", boot_addr);

s/dev_err()/dev_dbg()

Yes, ok.

Quoted 315 lines
Quoted 315 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +314 -0
+
+    /* boot vector need not be programmed for Core1 in LockStep mode */
+    core = kproc->core;
+    ret = ti_sci_proc_set_config(core->tsp, boot_addr, 0, 0);
+    if (ret)
+            goto put_mbox;
+
+    /* unhalt/run all applicable cores */
+    if (cluster->mode) {
+            list_for_each_entry_reverse(core, &cluster->cores, elem) {
+                    ret = k3_r5_core_run(core);
+                    if (ret)
+                            goto unroll_core_run;
+            }
+    } else {
+            ret = k3_r5_core_run(core);
+            if (ret)
+                    goto put_mbox;
+    }
+
+    return 0;
+
+unroll_core_run:
+    list_for_each_entry_continue(core, &cluster->cores, elem) {
+            if (k3_r5_core_halt(core))
+                    dev_warn(core->dev, "core halt back failed\n");
+    }
+put_mbox:
+    mbox_free_channel(kproc->mbox);
+    return ret;
+}
+
+/*
+ * The R5F stop function includes the following operations
+ * 1. Halt R5F core(s)
+ *
+ * The sequence is different between LockStep and Split modes, and the order
+ * of cores the operations are performed are also in general reverse to that
+ * of the start function. The LockStep mode requires each operation to be
+ * performed first on Core0 followed by Core1. The Split-mode requires that
+ * Core0 to be maintained always in a higher power state that Core1 (implying
+ * Core1 needs to be stopped first before Core0).
+ *
+ * Note that the R5F halt operation in general is not effective when the R5F
+ * core is running, but is needed to make sure the core won't run after
+ * deasserting the reset the subsequent time. The asserting of reset can
+ * be done here, but is preferred to be done in the .unprepare() ops - this
+ * maintains the symmetric behavior between the .start(), .stop(), .prepare()
+ * and .unprepare() ops, and also balances them well between sysfs 'state'
+ * flow and device bind/unbind or module removal.
+ */
+static int k3_r5_rproc_stop(struct rproc *rproc)
+{
+    struct k3_r5_rproc *kproc = rproc->priv;
+    struct k3_r5_cluster *cluster = kproc->cluster;
+    struct k3_r5_core *core = kproc->core;
+    int ret;
+
+    /* halt all applicable cores */
+    if (cluster->mode) {
+            list_for_each_entry(core, &cluster->cores, elem) {
+                    ret = k3_r5_core_halt(core);
+                    if (ret) {
+                            core = list_prev_entry(core, elem);
+                            goto unroll_core_halt;
+                    }
+            }
+    } else {
+            ret = k3_r5_core_halt(core);
+            if (ret)
+                    goto out;
+    }
+
+    mbox_free_channel(kproc->mbox);
+
+    return 0;
+
+unroll_core_halt:
+    list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+            if (k3_r5_core_run(core))
+                    dev_warn(core->dev, "core run back failed\n");
+    }
+out:
+    return ret;
+}
+
+/*
+ * Internal Memory translation helper
+ *
+ * Custom function implementing the rproc .da_to_va ops to provide address
+ * translation (device address to kernel virtual address) for internal RAMs
+ * present in a DSP or IPU device). The translated addresses can be used
+ * either by the remoteproc core for loading, or by any rpmsg bus drivers.
+ */
+static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
+{
+    struct k3_r5_rproc *kproc = rproc->priv;
+    struct k3_r5_core *core = kproc->core;
+    void __iomem *va = NULL;
+    phys_addr_t bus_addr;
+    u32 dev_addr, offset;
+    size_t size;
+    int i;
+
+    if (len == 0)
+            return NULL;
+
+    /* handle both R5 and SoC views of ATCM and BTCM */
+    for (i = 0; i < core->num_mems; i++) {
+            bus_addr = core->mem[i].bus_addr;
+            dev_addr = core->mem[i].dev_addr;
+            size = core->mem[i].size;
+
+            /* handle R5-view addresses of TCMs */
+            if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+                    offset = da - dev_addr;
+                    va = core->mem[i].cpu_addr + offset;
+                    return (__force void *)va;
+            }
+
+            /* handle SoC-view addresses of TCMs */
+            if (da >= bus_addr && ((da + len) <= (bus_addr + size))) {
+                    offset = da - bus_addr;
+                    va = core->mem[i].cpu_addr + offset;
+                    return (__force void *)va;
+            }
+    }
+
+    /* handle static DDR reserved memory regions */
+    for (i = 0; i < kproc->num_rmems; i++) {
+            dev_addr = kproc->rmem[i].dev_addr;
+            size = kproc->rmem[i].size;
+
+            if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+                    offset = da - dev_addr;
+                    va = kproc->rmem[i].cpu_addr + offset;
+                    return (__force void *)va;
+            }
+    }
+
+    return NULL;
+}
+
+static const struct rproc_ops k3_r5_rproc_ops = {
+    .prepare        = k3_r5_rproc_prepare,
+    .unprepare      = k3_r5_rproc_unprepare,
+    .start          = k3_r5_rproc_start,
+    .stop           = k3_r5_rproc_stop,
+    .kick           = k3_r5_rproc_kick,
+    .da_to_va       = k3_r5_rproc_da_to_va,
+};
+
+static const char *k3_r5_rproc_get_firmware(struct device *dev)
+{
+    const char *fw_name;
+    int ret;
+
+    ret = of_property_read_string(dev->of_node, "firmware-name",
+                                  &fw_name);
+    if (ret) {
+            dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+                    ret);
+            return ERR_PTR(ret);
+    }
+
+    return fw_name;
+}
+
+static int k3_r5_rproc_configure(struct k3_r5_rproc *kproc)
+{
+    struct k3_r5_cluster *cluster = kproc->cluster;
+    struct device *dev = kproc->dev;
+    struct k3_r5_core *core0, *core, *temp;
+    u32 ctrl = 0, cfg = 0, stat = 0;
+    u32 set_cfg = 0, clr_cfg = 0;
+    u64 boot_vec = 0;
+    bool lockstep_en;
+    int ret;
+
+    core0 = list_first_entry(&cluster->cores, struct k3_r5_core, elem);
+    core = cluster->mode ? core0 : kproc->core;
+
+    ret = ti_sci_proc_get_status(core->tsp, &boot_vec, &cfg, &ctrl,
+                                 &stat);
+    if (ret < 0)
+            return ret;
+
+    dev_dbg(dev, "boot_vector = 0x%llx, cfg = 0x%x ctrl = 0x%x stat = 0x%x\n",
+            boot_vec, cfg, ctrl, stat);
+
+    lockstep_en = !!(stat & PROC_BOOT_STATUS_FLAG_R5_LOCKSTEP_PERMITTED);
+    if (!lockstep_en && cluster->mode) {
+            dev_err(cluster->dev, "lockstep mode not permitted, force configuring for split-mode\n");
+            cluster->mode = 0;
+    }
+
+    /* always enable ARM mode and set boot vector to 0 */
+    boot_vec = 0x0;
+    if (core == core0) {
+            clr_cfg = PROC_BOOT_CFG_FLAG_R5_TEINIT;
+            /*
+             * LockStep configuration bit is Read-only on Split-mode _only_
+             * devices and system firmware will NACK any requests with the
+             * bit configured, so program it only on permitted devices
+             */
+            if (lockstep_en)
+                    clr_cfg |= PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+    }
+
+    if (core->atcm_enable)
+            set_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+    else
+            clr_cfg |= PROC_BOOT_CFG_FLAG_R5_ATCM_EN;
+
+    if (core->btcm_enable)
+            set_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+    else
+            clr_cfg |= PROC_BOOT_CFG_FLAG_R5_BTCM_EN;
+
+    if (core->loczrama)
+            set_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+    else
+            clr_cfg |= PROC_BOOT_CFG_FLAG_R5_TCM_RSTBASE;
+
+    if (cluster->mode) {
+            /*
+             * work around system firmware limitations to make sure both
+             * cores are programmed symmetrically in LockStep. LockStep
+             * and TEINIT config is only allowed with Core0.
+             */
+            list_for_each_entry(temp, &cluster->cores, elem) {
+                    ret = k3_r5_core_halt(core);
+                    if (ret)
+                            goto out;
+
+                    if (temp != core) {
+                            clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+                            clr_cfg &= ~PROC_BOOT_CFG_FLAG_R5_TEINIT;
+                    }
+                    ret = ti_sci_proc_set_config(temp->tsp, boot_vec,
+                                                 set_cfg, clr_cfg);
+                    if (ret)
+                            goto out;
+            }
+
+            set_cfg = PROC_BOOT_CFG_FLAG_R5_LOCKSTEP;
+            clr_cfg = 0;
+            ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+                                         set_cfg, clr_cfg);
+    } else {
+            ret = k3_r5_core_halt(core);
+            if (ret)
+                    goto out;
+
+            ret = ti_sci_proc_set_config(core->tsp, boot_vec,
+                                         set_cfg, clr_cfg);
+    }
+
+out:
+    return ret;
+}
+
+static int k3_r5_reserved_mem_init(struct k3_r5_rproc *kproc)
+{
+    struct device *dev = kproc->dev;
+    struct device_node *np = dev->of_node;
+    struct device_node *rmem_np;
+    struct reserved_mem *rmem;
+    int num_rmems;
+    int ret, i;
+
+    num_rmems = of_property_count_elems_of_size(np, "memory-region",
+                                                sizeof(phandle));
+    if (num_rmems <= 0) {
+            dev_err(dev, "device does not have reserved memory regions, ret = %d\n",
+                    num_rmems);
+            return -EINVAL;
+    }
+    if (num_rmems < 2) {
+            dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+                    num_rmems);
+            return -EINVAL;
+    }
+
+    /* use reserved memory region 0 for vring DMA allocations */
+    ret = of_reserved_mem_device_init_by_idx(dev, np, 0);
+    if (ret) {
+            dev_err(dev, "device cannot initialize DMA pool, ret = %d\n",
+                    ret);
+            return ret;
+    }
+
+    num_rmems--;
+    kproc->rmem = kcalloc(num_rmems, sizeof(*kproc->rmem), GFP_KERNEL);
+    if (!kproc->rmem) {
+            ret = -ENOMEM;
+            goto release_rmem;
+    }
+
+    /* use remaining reserved memory regions for static carveouts */
+    for (i = 0; i < num_rmems; i++) {
+            rmem_np = of_parse_phandle(np, "memory-region", i + 1);
+            if (!rmem_np) {
+                    ret = -EINVAL;
+                    goto unmap_rmem;
+            }
+
+            rmem = of_reserved_mem_lookup(rmem_np);
+            if (!rmem) {
+                    of_node_put(rmem_np);
+                    ret = -EINVAL;
+                    goto unmap_rmem;
+            }
+            of_node_put(rmem_np);
Quoted 8 lines

drivers/remoteproc/ti_k3_r5_remoteproc.c line 766

Quoted 4 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +4 -0
+
+            kproc->rmem[i].bus_addr = rmem->base;
+            /* 64-bit address regions currently not supported */
+            kproc->rmem[i].dev_addr = (u32)rmem->base;

Because the bus and the device addresses are the same I have to deduce the AP and the R5 have the same view of the memory. Please add a comment to assert that is really the case.

Quoted 3 lines

Yes for now, since we are not using the Region Address Translator yet which would have provided address extension from 32-bit device addresses to 64-bit bus addresses.

Ok, that makes sense. Just add a comment so that nobody thinks it's a mistake.

Thanks,
Mathieu

Quoted 589 lines

regards
Suman

Quoted 584 lines
Quoted 584 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c new file +580 -1
+            kproc->rmem[i].size = rmem->size;
+            kproc->rmem[i].cpu_addr = ioremap_wc(rmem->base, rmem->size);
+            if (!kproc->rmem[i].cpu_addr) {
+                    dev_err(dev, "failed to map reserved memory#%d at %pa of size %pa\n",
+                            i + 1, &rmem->base, &rmem->size);
+                    ret = -ENOMEM;
+                    goto unmap_rmem;
+            }
+
+            dev_dbg(dev, "reserved memory%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+                    i + 1, &kproc->rmem[i].bus_addr,
+                    kproc->rmem[i].size, kproc->rmem[i].cpu_addr,
+                    kproc->rmem[i].dev_addr);
+    }
+    kproc->num_rmems = num_rmems;
+
+    return 0;
+
+unmap_rmem:
+    for (i--; i >= 0; i--) {
+            if (kproc->rmem[i].cpu_addr)
+                    iounmap(kproc->rmem[i].cpu_addr);
+    }
+    kfree(kproc->rmem);
+release_rmem:
+    of_reserved_mem_device_release(dev);
+    return ret;
+}
+
+static void k3_r5_reserved_mem_exit(struct k3_r5_rproc *kproc)
+{
+    int i;
+
+    for (i = 0; i < kproc->num_rmems; i++)
+            iounmap(kproc->rmem[i].cpu_addr);
+    kfree(kproc->rmem);
+
+    of_reserved_mem_device_release(kproc->dev);
+}
+
+static int k3_r5_cluster_rproc_init(struct platform_device *pdev)
+{
+    struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+    struct device *dev = &pdev->dev;
+    struct k3_r5_rproc *kproc;
+    struct k3_r5_core *core, *core1;
+    struct device *cdev;
+    const char *fw_name;
+    struct rproc *rproc;
+    int ret;
+
+    core1 = list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+    list_for_each_entry(core, &cluster->cores, elem) {
+            cdev = core->dev;
+            fw_name = k3_r5_rproc_get_firmware(cdev);
+            if (IS_ERR(fw_name)) {
+                    ret = PTR_ERR(fw_name);
+                    goto out;
+            }
+
+            rproc = rproc_alloc(cdev, dev_name(cdev), &k3_r5_rproc_ops,
+                                fw_name, sizeof(*kproc));
+            if (!rproc) {
+                    ret = -ENOMEM;
+                    goto out;
+            }
+
+            /* K3 R5s have a Region Address Translator (RAT) but no MMU */
+            rproc->has_iommu = false;
+            /* error recovery is not supported at present */
+            rproc->recovery_disabled = true;
+
+            kproc = rproc->priv;
+            kproc->cluster = cluster;
+            kproc->core = core;
+            kproc->dev = cdev;
+            kproc->rproc = rproc;
+            core->rproc = rproc;
+
+            ret = k3_r5_rproc_configure(kproc);
+            if (ret) {
+                    dev_err(dev, "initial configure failed, ret = %d\n",
+                            ret);
+                    goto err_config;
+            }
+
+            ret = k3_r5_reserved_mem_init(kproc);
+            if (ret) {
+                    dev_err(dev, "reserved memory init failed, ret = %d\n",
+                            ret);
+                    goto err_config;
+            }
+
+            ret = rproc_add(rproc);
+            if (ret) {
+                    dev_err(dev, "rproc_add failed, ret = %d\n", ret);
+                    goto err_add;
+            }
+
+            /* create only one rproc in lockstep mode */
+            if (cluster->mode)
+                    break;
+    }
+
+    return 0;
+
+err_split:
+    rproc_del(rproc);
+err_add:
+    k3_r5_reserved_mem_exit(kproc);
+err_config:
+    rproc_free(rproc);
+    core->rproc = NULL;
+out:
+    /* undo core0 upon any failures on core1 in split-mode */
+    if (!cluster->mode && core == core1) {
+            core = list_prev_entry(core, elem);
+            rproc = core->rproc;
+            kproc = rproc->priv;
+            goto err_split;
+    }
+    return ret;
+}
+
+static int k3_r5_cluster_rproc_exit(struct platform_device *pdev)
+{
+    struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+    struct k3_r5_rproc *kproc;
+    struct k3_r5_core *core;
+    struct rproc *rproc;
+
+    /*
+     * lockstep mode has only one rproc associated with first core, whereas
+     * split-mode has two rprocs associated with each core, and requires
+     * that core1 be powered down first
+     */
+    core = cluster->mode ?
+            list_first_entry(&cluster->cores, struct k3_r5_core, elem) :
+            list_last_entry(&cluster->cores, struct k3_r5_core, elem);
+
+    list_for_each_entry_from_reverse(core, &cluster->cores, elem) {
+            rproc = core->rproc;
+            kproc = rproc->priv;
+
+            rproc_del(rproc);
+
+            k3_r5_reserved_mem_exit(kproc);
+
+            rproc_free(rproc);
+            core->rproc = NULL;
+    }
+
+    return 0;
+}
+
+static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
+                                           struct k3_r5_core *core)
+{
+    static const char * const mem_names[] = {"atcm", "btcm"};
+    struct device *dev = &pdev->dev;
+    struct resource *res;
+    int num_mems;
+    int i, ret;
+
+    num_mems = ARRAY_SIZE(mem_names);
+    core->mem = devm_kcalloc(dev, num_mems, sizeof(*core->mem), GFP_KERNEL);
+    if (!core->mem)
+            return -ENOMEM;
+
+    for (i = 0; i < num_mems; i++) {
+            res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+                                               mem_names[i]);
+            if (!res) {
+                    dev_err(dev, "found no memory resource for %s\n",
+                            mem_names[i]);
+                    ret = -EINVAL;
+                    goto fail;
+            }
+            if (!devm_request_mem_region(dev, res->start,
+                                         resource_size(res),
+                                         dev_name(dev))) {
+                    dev_err(dev, "could not request %s region for resource\n",
+                            mem_names[i]);
+                    ret = -EBUSY;
+                    goto fail;
+            }
+
+            /*
+             * TCMs are designed in general to support RAM-like backing
+             * memories. So, map these as Normal Non-Cached memories. This
+             * also avoids/fixes any potential alignment faults due to
+             * unaligned data accesses when using memcpy() or memset()
+             * functions (normally seen with device type memory).
+             */
+            core->mem[i].cpu_addr = devm_ioremap_wc(dev, res->start,
+                                                    resource_size(res));
+            if (IS_ERR(core->mem[i].cpu_addr)) {
+                    dev_err(dev, "failed to map %s memory\n", mem_names[i]);
+                    ret = PTR_ERR(core->mem[i].cpu_addr);
+                    devm_release_mem_region(dev, res->start,
+                                            resource_size(res));
+                    goto fail;
+            }
+            core->mem[i].bus_addr = res->start;
+
+            /*
+             * TODO:
+             * The R5F cores can place ATCM & BTCM anywhere in its address
+             * based on the corresponding Region Registers in the System
+             * Control coprocessor. For now, place ATCM and BTCM at
+             * addresses 0 and 0x41010000 (same as the bus address on AM65x
+             * SoCs) based on loczrama setting
+             */
+            if (!strcmp(mem_names[i], "atcm")) {
+                    core->mem[i].dev_addr = core->loczrama ?
+                                                    0 : K3_R5_TCM_DEV_ADDR;
+            } else {
+                    core->mem[i].dev_addr = core->loczrama ?
+                                                    K3_R5_TCM_DEV_ADDR : 0;
+            }
+            core->mem[i].size = resource_size(res);
+
+            dev_dbg(dev, "memory %8s: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+                    mem_names[i], &core->mem[i].bus_addr,
+                    core->mem[i].size, core->mem[i].cpu_addr,
+                    core->mem[i].dev_addr);
+    }
+    core->num_mems = num_mems;
+
+    return 0;
+
+fail:
+    for (i--; i >= 0; i--) {
+            devm_iounmap(dev, core->mem[i].cpu_addr);
+            devm_release_mem_region(dev, core->mem[i].bus_addr,
+                                    core->mem[i].size);
+    }
+    if (core->mem)
+            devm_kfree(dev, core->mem);
+    return ret;
+}
+
+static
+struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
+                                      const struct ti_sci_handle *sci)
+{
+    struct ti_sci_proc *tsp;
+    u32 temp[2];
+    int ret;
+
+    ret = of_property_read_u32_array(dev->of_node, "ti,sci-proc-ids",
+                                     temp, 2);
+    if (ret < 0)
+            return ERR_PTR(ret);
+
+    tsp = kzalloc(sizeof(*tsp), GFP_KERNEL);
+    if (!tsp)
+            return ERR_PTR(-ENOMEM);
+
+    tsp->dev = dev;
+    tsp->sci = sci;
+    tsp->ops = &sci->ops.proc_ops;
+    tsp->proc_id = temp[0];
+    tsp->host_id = temp[1];
+
+    return tsp;
+}
+
+static int k3_r5_core_of_init(struct platform_device *pdev)
+{
+    struct device *dev = &pdev->dev;
+    struct device_node *np = dev->of_node;
+    struct k3_r5_core *core;
+    int ret, ret1;
+
+    core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
+    if (!core)
+            return -ENOMEM;
+
+    core->dev = dev;
+    core->atcm_enable = 0;
+    core->btcm_enable = 1;
+    core->loczrama = 1;
+
+    ret = of_property_read_u32(np, "atcm-enable", &core->atcm_enable);
+    if (ret < 0 && ret != -EINVAL) {
+            dev_err(dev, "invalid format for atcm-enable, ret = %d\n", ret);
+            goto err_of;
+    }
+
+    ret = of_property_read_u32(np, "btcm-enable", &core->btcm_enable);
+    if (ret < 0 && ret != -EINVAL) {
+            dev_err(dev, "invalid format for btcm-enable, ret = %d\n", ret);
+            goto err_of;
+    }
+
+    ret = of_property_read_u32(np, "loczrama", &core->loczrama);
+    if (ret < 0 && ret != -EINVAL) {
+            dev_err(dev, "invalid format for loczrama, ret = %d\n", ret);
+            goto err_of;
+    }
+
+    core->ti_sci = ti_sci_get_by_phandle(np, "ti,sci");
+    if (IS_ERR(core->ti_sci)) {
+            ret = PTR_ERR(core->ti_sci);
+            if (ret != -EPROBE_DEFER) {
+                    dev_err(dev, "failed to get ti-sci handle, ret = %d\n",
+                            ret);
+            }
+            core->ti_sci = NULL;
+            goto err_of;
+    }
+
+    ret = of_property_read_u32(np, "ti,sci-dev-id", &core->ti_sci_id);
+    if (ret) {
+            dev_err(dev, "missing 'ti,sci-dev-id' property\n");
+            goto err_sci_id;
+    }
+
+    core->reset = reset_control_get_exclusive(dev, NULL);
+    if (IS_ERR(core->reset)) {
+            ret = PTR_ERR(core->reset);
+            if (ret != -EPROBE_DEFER) {
+                    dev_err(dev, "failed to get reset handle, ret = %d\n",
+                            ret);
+            }
+            goto err_sci_id;
+    }
+
+    core->tsp = k3_r5_core_of_get_tsp(dev, core->ti_sci);
+    if (IS_ERR(core->tsp)) {
+            dev_err(dev, "failed to construct ti-sci proc control, ret = %d\n",
+                    ret);
+            ret = PTR_ERR(core->tsp);
+            goto err_sci_proc;
+    }
+
+    ret = ti_sci_proc_request(core->tsp);
+    if (ret < 0) {
+            dev_err(dev, "ti_sci_proc_request failed, ret = %d\n", ret);
+            goto err_proc;
+    }
+
+    ret = k3_r5_core_of_get_internal_memories(pdev, core);
+    if (ret) {
+            dev_err(dev, "failed to get internal memories, ret = %d\n",
+                    ret);
+            goto err_intmem;
+    }
+
+    platform_set_drvdata(pdev, core);
+
+    return 0;
+
+err_intmem:
+    ret1 = ti_sci_proc_release(core->tsp);
+    if (ret1)
+            dev_err(dev, "failed to release proc, ret1 = %d\n", ret1);
+err_proc:
+    kfree(core->tsp);
+err_sci_proc:
+    reset_control_put(core->reset);
+err_sci_id:
+    ret1 = ti_sci_put_handle(core->ti_sci);
+    if (ret1)
+            dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret1);
+err_of:
+    devm_kfree(dev, core);
+    return ret;
+}
+
+/*
+ * free the resources explicitly since driver model is not being used
+ * for the child R5F devices
+ */
+static int k3_r5_core_of_exit(struct platform_device *pdev)
+{
+    struct k3_r5_core *core = platform_get_drvdata(pdev);
+    struct device *dev = &pdev->dev;
+    int i, ret;
+
+    for (i = 0; i < core->num_mems; i++) {
+            devm_release_mem_region(dev, core->mem[i].bus_addr,
+                                    core->mem[i].size);
+            devm_iounmap(dev, core->mem[i].cpu_addr);
+    }
+    if (core->mem)
+            devm_kfree(dev, core->mem);
+
+    ret = ti_sci_proc_release(core->tsp);
+    if (ret)
+            dev_err(dev, "failed to release proc, ret = %d\n", ret);
+
+    kfree(core->tsp);
+    reset_control_put(core->reset);
+
+    ret = ti_sci_put_handle(core->ti_sci);
+    if (ret)
+            dev_err(dev, "failed to put ti_sci handle, ret = %d\n", ret);
+
+    platform_set_drvdata(pdev, NULL);
+    devm_kfree(dev, core);
+
+    return ret;
+}
+
+static int k3_r5_cluster_of_init(struct platform_device *pdev)
+{
+    struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+    struct device *dev = &pdev->dev;
+    struct device_node *np = dev->of_node;
+    struct platform_device *cpdev;
+    struct device_node *child;
+    struct k3_r5_core *core, *temp;
+    int ret;
+
+    for_each_available_child_of_node(np, child) {
+            cpdev = of_find_device_by_node(child);
+            if (!cpdev) {
+                    ret = -ENODEV;
+                    dev_err(dev, "could not get R5 core platform device\n");
+                    goto fail;
+            }
+
+            ret = k3_r5_core_of_init(cpdev);
+            if (ret) {
+                    dev_err(dev, "k3_r5_core_of_init failed, ret = %d\n",
+                            ret);
+                    put_device(&cpdev->dev);
+                    goto fail;
+            }
+
+            core = platform_get_drvdata(cpdev);
+            put_device(&cpdev->dev);
+            list_add_tail(&core->elem, &cluster->cores);
+    }
+
+    return 0;
+
+fail:
+    list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+            list_del(&core->elem);
+            cpdev = to_platform_device(core->dev);
+            if (k3_r5_core_of_exit(cpdev))
+                    dev_err(dev, "k3_r5_core_of_exit cleanup failed\n");
+    }
+    return ret;
+}
+
+static int k3_r5_cluster_of_exit(struct platform_device *pdev)
+{
+    struct k3_r5_cluster *cluster = platform_get_drvdata(pdev);
+    struct device *dev = &pdev->dev;
+    struct platform_device *cpdev;
+    struct k3_r5_core *core, *temp;
+    int ret;
+
+    list_for_each_entry_safe_reverse(core, temp, &cluster->cores, elem) {
+            list_del(&core->elem);
+            cpdev = to_platform_device(core->dev);
+            ret = k3_r5_core_of_exit(cpdev);
+            if (ret) {
+                    dev_err(dev, "k3_r5_core_of_exit failed, ret = %d\n",
+                            ret);
+                    break;
+            }
+    }
+
+    return ret;
+}
+
+static int k3_r5_probe(struct platform_device *pdev)
+{
+    struct device *dev = &pdev->dev;
+    struct device_node *np = dev->of_node;
+    struct k3_r5_cluster *cluster;
+    int ret, ret1;
+    int num_cores;
+
+    cluster = devm_kzalloc(dev, sizeof(*cluster), GFP_KERNEL);
+    if (!cluster)
+            return -ENOMEM;
+
+    cluster->dev = dev;
+    cluster->mode = CLUSTER_MODE_LOCKSTEP;
+    INIT_LIST_HEAD(&cluster->cores);
+
+    ret = of_property_read_u32(np, "lockstep-mode", &cluster->mode);
+    if (ret < 0 && ret != -EINVAL) {
+            dev_err(dev, "invalid format for lockstep-mode, ret = %d\n",
+                    ret);
+            return ret;
+    }
+
+    num_cores = of_get_available_child_count(np);
+    if (num_cores != 2) {
+            dev_err(dev, "MCU cluster requires both R5F cores to be enabled, num_cores = %d\n",
+                    num_cores);
+            return -ENODEV;
+    }
+
+    platform_set_drvdata(pdev, cluster);
+
+    dev_dbg(dev, "creating child devices for R5F cores\n");
+    ret = of_platform_populate(np, NULL, NULL, dev);
+    if (ret) {
+            dev_err(dev, "of_platform_populate failed, ret = %d\n", ret);
+            return ret;
+    }
+
+    ret = k3_r5_cluster_of_init(pdev);
+    if (ret) {
+            dev_err(dev, "k3_r5_cluster_of_init failed, ret = %d\n", ret);
+            goto fail_of;
+    }
+
+    ret = k3_r5_cluster_rproc_init(pdev);
+    if (ret) {
+            dev_err(dev, "k3_r5_cluster_rproc_init failed, ret = %d\n",
+                    ret);
+            goto fail_rproc;
+    }
+
+    return 0;
+
+fail_rproc:
+    ret1 = k3_r5_cluster_of_exit(pdev);
+    if (ret1)
+            dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret1);
+fail_of:
+    of_platform_depopulate(dev);
+    return ret;
+}
+
+static int k3_r5_remove(struct platform_device *pdev)
+{
+    struct device *dev = &pdev->dev;
+    int ret;
+
+    ret = k3_r5_cluster_rproc_exit(pdev);
+    if (ret) {
+            dev_err(dev, "k3_r5_cluster_rproc_exit failed, ret = %d\n",
+                    ret);
+            goto fail;
+    }
+
+    ret = k3_r5_cluster_of_exit(pdev);
+    if (ret) {
+            dev_err(dev, "k3_r5_cluster_of_exit failed, ret = %d\n", ret);
+            goto fail;
+    }
+
+    dev_dbg(dev, "removing child devices for R5F cores\n");
+    of_platform_depopulate(dev);
+
+fail:
+    return ret;
+}
+
+static const struct of_device_id k3_r5_of_match[] = {
+    { .compatible = "ti,am654-r5fss", },
+    { .compatible = "ti,j721e-r5fss", },
+    { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, k3_r5_of_match);
+
+static struct platform_driver k3_r5_rproc_driver = {
+    .probe = k3_r5_probe,
+    .remove = k3_r5_remove,
+    .driver = {
+            .name = "k3_r5_rproc",
+            .of_match_table = k3_r5_of_match,
+    },
+};
+
+module_platform_driver(k3_r5_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("TI K3 R5F remote processor driver");
+MODULE_AUTHOR("Suman Anna <[email protected]>");
--

2.23.0

author
#27 lore raw

PATCH 6/7 remoteproc/k3-r5: Initialize TCM memories for ECC

In reply to Suman Anna

The R5F processors on K3 SoCs all have two TCMs (ATCM and BTCM) that support 32-bit ECC. The TCMs are typically loaded with some boot-up code to initialize the R5 MPUs to further execute code out of DDR. The ECC for the TCMs is enabled by default on K3 SoCs due to internal default tie-off values, but the TCM memories are not initialized on device power up. Any read access without the corresponding TCM memory location initialized will generate an ECC error, and any such access from a A72 or A53 core will trigger a SError.

So, zero initialize both the TCM memories before loading any firmware onto a R5F in remoteproc mode. Any R5F booted from U-Boot/SPL would require a similar initialization in the bootloader. Note that both the TCMs are initialized unconditionally as the TCM enable config bits only manage the access and visibility from R5. The Core1 TCMs are not used and accessible in LockStep mode, so they are only initialized in Split-mode.

---
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/remoteproc/ti_k3_r5_remoteproc.c +11 -0
index 655f8f14c37d..8c9b7ae5d8b7 100644
@@ -366,6 +366,17 @@ static int k3_r5_rproc_prepare(struct rproc *rproc)
 		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
 			ret);
 
+	/*
+	 * Zero out both TCMs unconditionally (access from v8 Arm core is not
+	 * affected by ATCM & BTCM enable configuration values) so that ECC
+	 * can be effective on all TCM addresses.
+	 */
+	dev_dbg(dev, "zeroing out ATCM memory\n");
+	memset(core->mem[0].cpu_addr, 0x00, core->mem[0].size);
+
+	dev_dbg(dev, "zeroing out BTCM memory\n");
+	memset(core->mem[1].cpu_addr, 0x00, core->mem[1].size);
+
 	return ret;
 }
 
Signature · 1 line
2.23.0
maintainer Acked-by on v1 6/7
#28 lore raw

Re: [PATCH 6/7] remoteproc/k3-r5: Initialize TCM memories for ECC

Quoted 9 lines · Suman Anna

The R5F processors on K3 SoCs all have two TCMs (ATCM and BTCM) that support 32-bit ECC. The TCMs are typically loaded with some boot-up code to initialize the R5 MPUs to further execute code out of DDR. The ECC for the TCMs is enabled by default on K3 SoCs due to internal default tie-off values, but the TCM memories are not initialized on device power up. Any read access without the corresponding TCM memory location initialized will generate an ECC error, and any such access from a A72 or A53 core will trigger a SError.

Quoted 7 lines

So, zero initialize both the TCM memories before loading any firmware onto a R5F in remoteproc mode. Any R5F booted from U-Boot/SPL would require a similar initialization in the bootloader. Note that both the TCMs are initialized unconditionally as the TCM enable config bits only manage the access and visibility from R5. The Core1 TCMs are not used and accessible in LockStep mode, so they are only initialized in Split-mode.

Everything was going well with this changelog until the last sentence. Intuitively one is looking for the code that avoids the initialisation for "Core1" in the patch but it is not there, and rightly so. In locksetup mode the second core is not registered with the remoteproc core and as such the associated TCMs won't be initialised.

Simply put, I would just remove the last sentence as all it does (at least for me) is add confusion.

With that:

Quoted 31 lines
---
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/remoteproc/ti_k3_r5_remoteproc.c +11 -0
index 655f8f14c37d..8c9b7ae5d8b7 100644
@@ -366,6 +366,17 @@ static int k3_r5_rproc_prepare(struct rproc *rproc)
 		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
 			ret);
 
+	/*
+	 * Zero out both TCMs unconditionally (access from v8 Arm core is not
+	 * affected by ATCM & BTCM enable configuration values) so that ECC
+	 * can be effective on all TCM addresses.
+	 */
+	dev_dbg(dev, "zeroing out ATCM memory\n");
+	memset(core->mem[0].cpu_addr, 0x00, core->mem[0].size);
+
+	dev_dbg(dev, "zeroing out BTCM memory\n");
+	memset(core->mem[1].cpu_addr, 0x00, core->mem[1].size);
+
 	return ret;
 }
 
Signature · 1 line
2.23.0
author
#29 lore raw

Re: [PATCH 6/7] remoteproc/k3-r5: Initialize TCM memories for ECC

Quoted 18 lines · Mathieu Poirier
Quoted 9 lines · Suman Anna

The R5F processors on K3 SoCs all have two TCMs (ATCM and BTCM) that support 32-bit ECC. The TCMs are typically loaded with some boot-up code to initialize the R5 MPUs to further execute code out of DDR. The ECC for the TCMs is enabled by default on K3 SoCs due to internal default tie-off values, but the TCM memories are not initialized on device power up. Any read access without the corresponding TCM memory location initialized will generate an ECC error, and any such access from a A72 or A53 core will trigger a SError.

Quoted 7 lines

So, zero initialize both the TCM memories before loading any firmware onto a R5F in remoteproc mode. Any R5F booted from U-Boot/SPL would require a similar initialization in the bootloader. Note that both the TCMs are initialized unconditionally as the TCM enable config bits only manage the access and visibility from R5. The Core1 TCMs are not used and accessible in LockStep mode, so they are only initialized in Split-mode.

Quoted 8 lines

Everything was going well with this changelog until the last sentence. Intuitively one is looking for the code that avoids the initialisation for "Core1" in the patch but it is not there, and rightly so. In locksetup mode the second core is not registered with the remoteproc core and as such the associated TCMs won't be initialised.

Simply put, I would just remove the last sentence as all it does (at least for me) is add confusion.

Yep, that was more of a "NOTE: " type comment on overall behavior. I will drop the sentence for v2.

regards
Suman

Quoted 36 lines

With that:

Quoted 31 lines
---
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/remoteproc/ti_k3_r5_remoteproc.c +11 -0
index 655f8f14c37d..8c9b7ae5d8b7 100644
@@ -366,6 +366,17 @@ static int k3_r5_rproc_prepare(struct rproc *rproc)
 		dev_err(dev, "unable to enable cores for TCM loading, ret = %d\n",
 			ret);
 
+	/*
+	 * Zero out both TCMs unconditionally (access from v8 Arm core is not
+	 * affected by ATCM & BTCM enable configuration values) so that ECC
+	 * can be effective on all TCM addresses.
+	 */
+	dev_dbg(dev, "zeroing out ATCM memory\n");
+	memset(core->mem[0].cpu_addr, 0x00, core->mem[0].size);
+
+	dev_dbg(dev, "zeroing out BTCM memory\n");
+	memset(core->mem[1].cpu_addr, 0x00, core->mem[1].size);
+
 	return ret;
 }
 
Signature · 1 line
2.23.0
author
#30 lore raw

PATCH 7/7 remoteproc/k3-r5: Add loading support for on-chip SRAM regions

In reply to Suman Anna

The K3 SoCs has various internal on-chip SRAM memories like the SRAM within the MCU domain or the shared MSMC RAM within NavSS that can be used for multiple purposes. One such purpose is to have the R5F cores use a portion of such on-chip SRAM for fast-access data or to directly execute code.

Add support to the K3 R5 remoteproc driver to parse and support loading into such memories. The SRAM regions need to be mapped as normal non-cacheable memory to avoid kernel crashes when the remoteproc loader code uses the Arm64 memset library function (the "DC ZVA" instruction throws a alignment fault on device type memory).

These SRAM regions are completely optional as not all firmware images require these memories, and any such memory has to be reserved as such in the DTS files.

---
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 106 ++++++++++++++++++++++-
 1 file changed, 105 insertions(+), 1 deletion(-)
drivers/remoteproc/ti_k3_r5_remoteproc.c +105 -1
index 8c9b7ae5d8b7..0ae0b66ec9eb 100644
@@ -85,7 +85,9 @@ struct k3_r5_cluster {
  * @dev: cached device pointer
  * @rproc: rproc handle representing this core
  * @mem: internal memory regions data
+ * @sram: on-chip SRAM memory regions data
  * @num_mems: number of internal memory regions
+ * @num_sram: number of on-chip SRAM memory regions
  * @reset: reset control handle
  * @tsp: TI-SCI processor control handle
  * @ti_sci: TI-SCI handle
@@ -99,7 +101,9 @@ struct k3_r5_core {
 	struct device *dev;
 	struct rproc *rproc;
 	struct k3_r5_mem *mem;
+	struct k3_r5_mem *sram;
 	int num_mems;
+	int num_sram;
 	struct reset_control *reset;
 	struct ti_sci_proc *tsp;
 	const struct ti_sci_handle *ti_sci;
@@ -585,6 +589,18 @@ static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
 		}
 	}
 
+	/* handle any SRAM regions using SoC-view addresses */
+	for (i = 0; i < core->num_sram; i++) {
+		dev_addr = core->sram[i].dev_addr;
+		size = core->sram[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->sram[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
 	/* handle static DDR reserved memory regions */
 	for (i = 0; i < kproc->num_rmems; i++) {
 		dev_addr = kproc->rmem[i].dev_addr;
@@ -1017,6 +1033,77 @@ static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
 	return ret;
 }
 
+static int k3_r5_core_of_get_sram_memories(struct platform_device *pdev,
+					   struct k3_r5_core *core)
+{
+	struct device_node *np = pdev->dev.of_node;
+	struct device *dev = &pdev->dev;
+	struct device_node *sram_np;
+	struct resource res;
+	int num_sram;
+	int i, ret;
+
+	num_sram = of_property_count_elems_of_size(np, "sram", sizeof(phandle));
+	if (num_sram <= 0) {
+		dev_dbg(dev, "device does not use reserved on-chip memories, num_sram = %d\n",
+			num_sram);
+		return 0;
+	}
+
+	core->sram = kcalloc(num_sram, sizeof(*core->sram), GFP_KERNEL);
+	if (!core->sram)
+		return -ENOMEM;
+
+	for (i = 0; i < num_sram; i++) {
+		sram_np = of_parse_phandle(np, "sram", i);
+		if (!sram_np) {
+			ret = -EINVAL;
+			goto fail;
+		}
+
+		if (!of_device_is_available(sram_np)) {
+			of_node_put(sram_np);
+			ret = -EINVAL;
+			goto fail;
+		}
+
+		ret = of_address_to_resource(sram_np, 0, &res);
+		of_node_put(sram_np);
+		if (ret) {
+			ret = -EINVAL;
+			goto fail;
+		}
+		core->sram[i].bus_addr = res.start;
+		core->sram[i].dev_addr = res.start;
+		core->sram[i].size = resource_size(&res);
+		core->sram[i].cpu_addr = ioremap_wc(res.start,
+						    resource_size(&res));
+		if (!core->sram[i].cpu_addr) {
+			dev_err(dev, "failed to parse and map sram%d memory at %pad\n",
+				i, &res.start);
+			ret = -ENOMEM;
+			goto fail;
+		}
+
+		dev_dbg(dev, "memory    sram%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",
+			i, &core->sram[i].bus_addr,
+			core->sram[i].size, core->sram[i].cpu_addr,
+			core->sram[i].dev_addr);
+	}
+	core->num_sram = num_sram;
+
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		if (core->sram[i].cpu_addr)
+			iounmap(core->sram[i].cpu_addr);
+	}
+	kfree(core->sram);
+
+	return ret;
+}
+
 static
 struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
 					  const struct ti_sci_handle *sci)
@@ -1048,7 +1135,7 @@ static int k3_r5_core_of_init(struct platform_device *pdev)
 	struct device *dev = &pdev->dev;
 	struct device_node *np = dev->of_node;
 	struct k3_r5_core *core;
-	int ret, ret1;
+	int ret, ret1, i;
 
 	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
 	if (!core)
@@ -1125,10 +1212,23 @@ static int k3_r5_core_of_init(struct platform_device *pdev)
 		goto err_intmem;
 	}
 
+	ret = k3_r5_core_of_get_sram_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get sram memories, ret = %d\n", ret);
+		goto err_sram;
+	}
+
 	platform_set_drvdata(pdev, core);
 
 	return 0;
 
+err_sram:
+	for (i = 0; i < core->num_mems; i++) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	devm_kfree(dev, core->mem);
 err_intmem:
 	ret1 = ti_sci_proc_release(core->tsp);
 	if (ret1)
@@ -1156,6 +1256,10 @@ static int k3_r5_core_of_exit(struct platform_device *pdev)
 	struct device *dev = &pdev->dev;
 	int i, ret;
 
+	for (i = 0; i < core->num_sram; i++)
+		iounmap(core->sram[i].cpu_addr);
+	kfree(core->sram);
+
 	for (i = 0; i < core->num_mems; i++) {
 		devm_release_mem_region(dev, core->mem[i].bus_addr,
 					core->mem[i].size);
Signature · 1 line
2.23.0
maintainer Acked-by on v1 7/7
#31 lore raw

Re: [PATCH 7/7] remoteproc/k3-r5: Add loading support for on-chip SRAM regions

Quoted 111 lines · Suman Anna

The K3 SoCs has various internal on-chip SRAM memories like the SRAM within the MCU domain or the shared MSMC RAM within NavSS that can be used for multiple purposes. One such purpose is to have the R5F cores use a portion of such on-chip SRAM for fast-access data or to directly execute code.

Add support to the K3 R5 remoteproc driver to parse and support loading into such memories. The SRAM regions need to be mapped as normal non-cacheable memory to avoid kernel crashes when the remoteproc loader code uses the Arm64 memset library function (the "DC ZVA" instruction throws a alignment fault on device type memory).

These SRAM regions are completely optional as not all firmware images require these memories, and any such memory has to be reserved as such in the DTS files.

---
 drivers/remoteproc/ti_k3_r5_remoteproc.c | 106 ++++++++++++++++++++++-
 1 file changed, 105 insertions(+), 1 deletion(-)
drivers/remoteproc/ti_k3_r5_remoteproc.c +59 -0
index 8c9b7ae5d8b7..0ae0b66ec9eb 100644
@@ -85,7 +85,9 @@ struct k3_r5_cluster {
  * @dev: cached device pointer
  * @rproc: rproc handle representing this core
  * @mem: internal memory regions data
+ * @sram: on-chip SRAM memory regions data
  * @num_mems: number of internal memory regions
+ * @num_sram: number of on-chip SRAM memory regions
  * @reset: reset control handle
  * @tsp: TI-SCI processor control handle
  * @ti_sci: TI-SCI handle
@@ -99,7 +101,9 @@ struct k3_r5_core {
 	struct device *dev;
 	struct rproc *rproc;
 	struct k3_r5_mem *mem;
+	struct k3_r5_mem *sram;
 	int num_mems;
+	int num_sram;
 	struct reset_control *reset;
 	struct ti_sci_proc *tsp;
 	const struct ti_sci_handle *ti_sci;
@@ -585,6 +589,18 @@ static void *k3_r5_rproc_da_to_va(struct rproc *rproc, u64 da, size_t len)
 		}
 	}
 
+	/* handle any SRAM regions using SoC-view addresses */
+	for (i = 0; i < core->num_sram; i++) {
+		dev_addr = core->sram[i].dev_addr;
+		size = core->sram[i].size;
+
+		if (da >= dev_addr && ((da + len) <= (dev_addr + size))) {
+			offset = da - dev_addr;
+			va = core->sram[i].cpu_addr + offset;
+			return (__force void *)va;
+		}
+	}
+
 	/* handle static DDR reserved memory regions */
 	for (i = 0; i < kproc->num_rmems; i++) {
 		dev_addr = kproc->rmem[i].dev_addr;
@@ -1017,6 +1033,77 @@ static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev,
 	return ret;
 }
 
+static int k3_r5_core_of_get_sram_memories(struct platform_device *pdev,
+					   struct k3_r5_core *core)
+{
+	struct device_node *np = pdev->dev.of_node;
+	struct device *dev = &pdev->dev;
+	struct device_node *sram_np;
+	struct resource res;
+	int num_sram;
+	int i, ret;
+
+	num_sram = of_property_count_elems_of_size(np, "sram", sizeof(phandle));
+	if (num_sram <= 0) {
+		dev_dbg(dev, "device does not use reserved on-chip memories, num_sram = %d\n",
+			num_sram);
+		return 0;
+	}
+
+	core->sram = kcalloc(num_sram, sizeof(*core->sram), GFP_KERNEL);
+	if (!core->sram)
+		return -ENOMEM;
+
+	for (i = 0; i < num_sram; i++) {
+		sram_np = of_parse_phandle(np, "sram", i);
+		if (!sram_np) {
+			ret = -EINVAL;
+			goto fail;
+		}
+
+		if (!of_device_is_available(sram_np)) {
+			of_node_put(sram_np);
+			ret = -EINVAL;
+			goto fail;
+		}
+
+		ret = of_address_to_resource(sram_np, 0, &res);
+		of_node_put(sram_np);
+		if (ret) {
+			ret = -EINVAL;
+			goto fail;
+		}
+		core->sram[i].bus_addr = res.start;
+		core->sram[i].dev_addr = res.start;
+		core->sram[i].size = resource_size(&res);

drivers/remoteproc/ti_k3_r5_remoteproc.c static int k3_r5_core_of_get_internal_memories(struct platform_device *pdev, · line 1088

Quoted 10 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c +10 -0
+		core->sram[i].cpu_addr = ioremap_wc(res.start,
+						    resource_size(&res));
+		if (!core->sram[i].cpu_addr) {
+			dev_err(dev, "failed to parse and map sram%d memory at %pad\n",
+				i, &res.start);
+			ret = -ENOMEM;
+			goto fail;
+		}
+
+		dev_dbg(dev, "memory    sram%d: bus addr %pa size 0x%zx va %pK da 0x%x\n",

s/"memory sram%d:..."/"memory sram%d:..."

With the above:

I am done reviewing this set. Despite the significant amount of material to process and the comments here and there, I haven't found anything to really complain about.

Thanks,
Mathieu

Quoted 68 lines
drivers/remoteproc/ti_k3_r5_remoteproc.c +36 -1
+			i, &core->sram[i].bus_addr,
+			core->sram[i].size, core->sram[i].cpu_addr,
+			core->sram[i].dev_addr);
+	}
+	core->num_sram = num_sram;
+
+	return 0;
+
+fail:
+	for (i--; i >= 0; i--) {
+		if (core->sram[i].cpu_addr)
+			iounmap(core->sram[i].cpu_addr);
+	}
+	kfree(core->sram);
+
+	return ret;
+}
+
 static
 struct ti_sci_proc *k3_r5_core_of_get_tsp(struct device *dev,
 					  const struct ti_sci_handle *sci)
@@ -1048,7 +1135,7 @@ static int k3_r5_core_of_init(struct platform_device *pdev)
 	struct device *dev = &pdev->dev;
 	struct device_node *np = dev->of_node;
 	struct k3_r5_core *core;
-	int ret, ret1;
+	int ret, ret1, i;
 
 	core = devm_kzalloc(dev, sizeof(*core), GFP_KERNEL);
 	if (!core)
@@ -1125,10 +1212,23 @@ static int k3_r5_core_of_init(struct platform_device *pdev)
 		goto err_intmem;
 	}
 
+	ret = k3_r5_core_of_get_sram_memories(pdev, core);
+	if (ret) {
+		dev_err(dev, "failed to get sram memories, ret = %d\n", ret);
+		goto err_sram;
+	}
+
 	platform_set_drvdata(pdev, core);
 
 	return 0;
 
+err_sram:
+	for (i = 0; i < core->num_mems; i++) {
+		devm_iounmap(dev, core->mem[i].cpu_addr);
+		devm_release_mem_region(dev, core->mem[i].bus_addr,
+					core->mem[i].size);
+	}
+	devm_kfree(dev, core->mem);
 err_intmem:
 	ret1 = ti_sci_proc_release(core->tsp);
 	if (ret1)
@@ -1156,6 +1256,10 @@ static int k3_r5_core_of_exit(struct platform_device *pdev)
 	struct device *dev = &pdev->dev;
 	int i, ret;
 
+	for (i = 0; i < core->num_sram; i++)
+		iounmap(core->sram[i].cpu_addr);
+	kfree(core->sram);
+
 	for (i = 0; i < core->num_mems; i++) {
 		devm_release_mem_region(dev, core->mem[i].bus_addr,
 					core->mem[i].size);
Signature · 1 line
2.23.0