Skip to contentSkip to conversation

Create infrastructure for running C code from SRAM.

RFC New Russ Dill 7 participants 21 messages Sep 3, 2013 – Sep 9, 2013

Subsystems (7)
lore
Share & tools
lore

Russ Dill

author
#0 lore raw

cover Create infrastructure for running C code from SRAM.

This RFC patchset explores an idea for loading C code into SRAM. Currently, all the code I'm aware of that needs to run from SRAM is written in assembler. The most common reason for code needing to run from SRAM is that the memory controller is being disabled/ enabled or is already disabled. arch/arm has by far the most examples, but code also exists in powerpc and sh.

The code is written in asm for two primary reasons. First so that markers can be put in indicating the size of the code they it can be copied. Second so that data can be placed along with text and accessed in a position independant manner.

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
__sram_am33xx_start = .;
*(.sram.am33xx.*)
__sram_am33xx_end = .;
}

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

[1] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4984c6
[2] http://www.spinics.net/lists/linux-omap/msg96504.html

Russ Dill (4):
Misc: SRAM: Create helpers for loading C code into SRAM
ARM: SRAM: Add macro for generating SRAM resume trampoline
Misc: SRAM: Hack for allowing executable code in SRAM.
ARM: AM33XX: Move suspend/resume assembly to C

 arch/arm/include/asm/suspend.h    |  14 ++
 arch/arm/kernel/vmlinux.lds.S     |   2 +
 arch/arm/mach-omap2/Makefile      |   2 +-
 arch/arm/mach-omap2/pm33xx.c      |  50 ++---
 arch/arm/mach-omap2/pm33xx.h      |  23 +--
 arch/arm/mach-omap2/sleep33xx.S   | 394 --------------------------------------
 arch/arm/mach-omap2/sleep33xx.c   | 309 ++++++++++++++++++++++++++++++
 arch/arm/mach-omap2/sram.c        |  15 --
 drivers/misc/sram.c               | 106 +++++++++-
 include/asm-generic/vmlinux.lds.h |   7 +
 include/linux/sram.h              |  44 +++++
 11 files changed, 509 insertions(+), 457 deletions(-)
 delete mode 100644 arch/arm/mach-omap2/sleep33xx.S
 create mode 100644 arch/arm/mach-omap2/sleep33xx.c
 create mode 100644 include/linux/sram.h
Signature · 1 line
1.8.3.2
author
#1 lore raw

RFC 1/4 Misc: SRAM: Create helpers for loading C code into SRAM

This commit adds helpers for loading, tracking, and running code from SRAM. Some platforms need to run code from SRAM, such as during suspend/resume when the access to SDRAM is disabled.

Currently, this is done in assembly. By using the assembler, labels can be added in order to mark the size of the code, and areas can be added for position independent access to data for when the code is copied into SRAM. As more and more such code is added, the maintenance burden grows.

This commit adds mechanisms to allow C code to be run from SRAM. The first step is to put code and data that will be needed when running from SRAM into special sections. The sections should follow the naming convention of .sram.<sram_user>.<section_name>. For example:

#define __sram_am33xx __section(.sram.am33xx.text)
#define __sram_am33xxdata __section(.sram.am33xx.data)

Functions can then be marked with __sram_am33xx and data with __sram_am33xxdata. These sections and markers to them can be added to the kernel image by inserting a "SRAM_SECTIONS(<sram_user>)" into your architecture's vmlinux.lds.S file.

A call to sram_load_sections("<sram_dt_path>", <sram_user>) will copy the sections into SRAM:

sram_load_sections("/ocp/ocmcram@40300000", am33xx);

Because the code is now relocated to SRAM, special accessors must be used when accessing functions or variables.

- kern_to_sram(ptr): Translate a pointer to a function or variable that
is contained in a .sram.<sram_user>.* section to the address it has been loaded to in SRAM. For instance, when calling a function in sram, one would use: "ret = kern_to_sram(&am33xx_suspend)(flags);"

- sram_to_phys(addr): Translate an pointer into SRAM to a physical address. This is useful for obtaining a physical pointer to a resume function contained in SRAM.

Any compilation unit that will be copied to SRAM that accesses other functions or variables that are also copied to SRAM should be compiled with -fPIC as calling the kern_to_sram accessors requires reads from SDRAM.

Functions within a SRAM section group can access functions and data also within the same SRAM section group without using kern_to_sram accessors so long as the unit is compiled with -fPIC. Care should be taken to use a method for obtaining an absolute address when accessing functions or data outside of the SRAM section group.

---
 drivers/misc/sram.c               | 103 ++++++++++++++++++++++++++++++++++++++
 include/asm-generic/vmlinux.lds.h |   7 +++
 include/linux/sram.h              |  44 ++++++++++++++++
 3 files changed, 154 insertions(+)
 create mode 100644 include/linux/sram.h
drivers/misc/sram.c +103 -0
index d87cc91..08baaab 100644
@@ -25,17 +25,110 @@
 #include <linux/io.h>
 #include <linux/of.h>
 #include <linux/platform_device.h>
+#include <linux/of_platform.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/genalloc.h>
+#include <linux/sram.h>
+#include <asm-generic/cacheflush.h>
 
 #define SRAM_GRANULARITY	32
 
+static DEFINE_SPINLOCK(sram_lock);
+static LIST_HEAD(chunk_list);
+
 struct sram_dev {
 	struct gen_pool *pool;
 	struct clk *clk;
 };
 
+struct sram_chunk {
+	struct gen_pool *pool;
+	struct list_head node;
+	struct rcu_head head;
+	void *kern;
+	unsigned long addr;
+	size_t sz;
+};
+
+int sram_load_data(const char *of_path, void *data, size_t sz)
+{
+	struct device_node *np;
+	struct platform_device *pdev;
+	struct sram_chunk *chunk;
+	struct sram_dev *sram;
+
+	np = of_find_node_by_path(of_path);
+	if (!np)
+		return -ENODEV;
+
+	pdev = of_find_device_by_node(np);
+	if (!pdev)
+		return -ENODEV;
+
+	sram = platform_get_drvdata(pdev);
+
+	chunk = kzalloc(sizeof(*chunk), GFP_KERNEL);
+	if (!chunk)
+		return -ENOMEM;
+
+	chunk->pool = sram->pool;
+	chunk->kern = data;
+	chunk->sz = sz;
+
+	chunk->addr = gen_pool_alloc(sram->pool, sz);
+	if (!chunk->addr) {
+		kfree(chunk);
+		return -ENOMEM;
+	}
+
+	memcpy((void *) chunk->addr, data, sz);
+	flush_icache_range(chunk->addr, chunk->addr + sz);
+
+	spin_lock(&sram_lock);
+	list_add_rcu(&chunk->node, &chunk_list);
+	spin_unlock(&sram_lock);
+
+	return 0;
+
+}
+EXPORT_SYMBOL_GPL(sram_load_data);
+
+phys_addr_t sram_to_phys(unsigned long addr)
+{
+	struct sram_chunk *chunk;
+	phys_addr_t ret = -1;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(chunk, &chunk_list, node) {
+		ret = gen_pool_virt_to_phys(chunk->pool, addr);
+		if (ret != -1)
+			break;
+	}
+	rcu_read_unlock();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(sram_to_phys);
+
+void __iomem *__kern_to_sram(void *ptr)
+{
+	struct sram_chunk *chunk;
+	void __iomem *ret = NULL;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(chunk, &chunk_list, node) {
+		if (ptr >= chunk->kern && ptr < chunk->kern + chunk->sz) {
+			ret = (void *) (ptr - chunk->kern + chunk->addr);
+			break;
+		}
+	}
+	rcu_read_unlock();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(__kern_to_sram);
+
 static int sram_probe(struct platform_device *pdev)
 {
 	void __iomem *virt_base;
@@ -82,6 +175,16 @@ static int sram_probe(struct platform_device *pdev)
 static int sram_remove(struct platform_device *pdev)
 {
 	struct sram_dev *sram = platform_get_drvdata(pdev);
+	struct sram_chunk *chunk;
+
+	list_for_each_entry(chunk, &chunk_list, node) {
+		if (chunk->pool == sram->pool) {
+			spin_lock(&sram_lock);
+			list_del_rcu(&chunk->node);
+			spin_unlock(&sram_lock);
+			kfree_rcu(chunk, head);
+		}
+	}
 
 	if (gen_pool_avail(sram->pool) < gen_pool_size(sram->pool))
 		dev_dbg(&pdev->dev, "removed while SRAM allocated\n");
include/asm-generic/vmlinux.lds.h +7 -0
index 69732d2..4817732 100644
@@ -797,3 +797,10 @@
 	BSS(bss_align)							\
 	. = ALIGN(stop_align);						\
 	VMLINUX_SYMBOL(__bss_stop) = .;
+
+#define SRAM_SECTIONS(name)						\
+	.sram.##name : AT(ADDR(.sram.##name) - LOAD_OFFSET) {		\
+		VMLINUX_SYMBOL(__sram_##name##_start) = .;		\
+		*(.sram.##name##.*)					\
+		VMLINUX_SYMBOL(__sram_##name##_end) = .;		\
+	}
include/linux/sram.h new file +44 -0

44 lines not on this page · raw message

Signature · 1 line
1.8.3.2
author
#2 lore raw

RFC 2/4 ARM: SRAM: Add macro for generating SRAM resume trampoline

In reply to Russ Dill

Add a helper that generates a short snippet of code that loads the stack pointer and calls a c (or asm) function. The code gets placed into a SRAM section for loading into SRAM.

---
 arch/arm/include/asm/suspend.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/arm/include/asm/suspend.h +14 -0
index cd20029..d23d004 100644
@@ -9,4 +9,18 @@ struct sleep_save_sp {
 extern void cpu_resume(void);
 extern int cpu_suspend(unsigned long, int (*)(unsigned long));
 
+/*
+ * Generate a SRAM trampoline for resume.
+ * @proc: SoC, should match argument used with SRAM_SECTIONS().
+ * @func: C or asm function to call at resume.
+ * @stack: Stack to use before calling func.
+ */
+#define ARM_SRAM_RESUME(proc, func, stack)				\
+void __naked __noreturn __sram_##proc proc##_resume_trampoline(void)	\
+{									\
+	__asm__ __volatile__("mov sp, %0" : : "r"((stack)) : "sp");	\
+	func();								\
+}
+
+
 #endif
Signature · 1 line
1.8.3.2
author
#3 lore raw

RFC 3/4 Misc: SRAM: Hack for allowing executable code in SRAM.

In reply to Russ Dill

The generic SRAM mechanism does not ioremap memory in a manner that allows code to be executed from SRAM. There is currently no generic way to request ioremap to return a memory area with execution allowed.

Insert a temporary hack for proof of concept on ARM.

---
 drivers/misc/sram.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/misc/sram.c +2 -1
index 08baaab..e059a23 100644
@@ -31,6 +31,7 @@
 #include <linux/genalloc.h>
 #include <linux/sram.h>
 #include <asm-generic/cacheflush.h>
+#include <asm/io.h>
 
 #define SRAM_GRANULARITY	32
 
@@ -138,7 +139,7 @@ static int sram_probe(struct platform_device *pdev)
 	int ret;
 
 	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
-	virt_base = devm_ioremap_resource(&pdev->dev, res);
+	virt_base = __arm_ioremap_exec(res->start, resource_size(res), false);
 	if (IS_ERR(virt_base))
 		return PTR_ERR(virt_base);
 
Signature · 1 line
1.8.3.2
maintainer
#4 lore raw

RFC 3/4 Misc: SRAM: Hack for allowing executable code in SRAM.

Quoted 25 lines · Russ Dill

The generic SRAM mechanism does not ioremap memory in a manner that allows code to be executed from SRAM. There is currently no generic way to request ioremap to return a memory area with execution allowed.

Insert a temporary hack for proof of concept on ARM.

---
 drivers/misc/sram.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/misc/sram.c +1 -0
index 08baaab..e059a23 100644
@@ -31,6 +31,7 @@
 #include <linux/genalloc.h>
 #include <linux/sram.h>
 #include <asm-generic/cacheflush.h>
+#include <asm/io.h>
 
 #define SRAM_GRANULARITY	32
 

drivers/misc/sram.c static int sram_probe(struct platform_device *pdev) · line 144

Quoted 7 lines
drivers/misc/sram.c +1 -1
@@ -138,7 +139,7 @@ static int sram_probe(struct platform_device *pdev)
 	int ret;
 
 	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
-	virt_base = devm_ioremap_resource(&pdev->dev, res);
+	virt_base = __arm_ioremap_exec(res->start, resource_size(res), false);
 	if (IS_ERR(virt_base))
 		return PTR_ERR(virt_base);

You can get rid of this hack by defining ioremap_exec in include/asm-generic/io.h the same way as ioremap_nocache is done:

#ifndef ioremap_exec
#define ioremap_exec ioremap
#endif

Then the arch that need ioremap_exec can define and implement it. Needs to be reviewed on LKML naturally :)

Regards,

Tony

author
#5 lore raw

RFC 3/4 Misc: SRAM: Hack for allowing executable code in SRAM.

Quoted 38 lines · Tony Lindgren
Quoted 25 lines · Russ Dill

The generic SRAM mechanism does not ioremap memory in a manner that allows code to be executed from SRAM. There is currently no generic way to request ioremap to return a memory area with execution allowed.

Insert a temporary hack for proof of concept on ARM.

---
 drivers/misc/sram.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/misc/sram.c +1 -0
index 08baaab..e059a23 100644
@@ -31,6 +31,7 @@
 #include <linux/genalloc.h>
 #include <linux/sram.h>
 #include <asm-generic/cacheflush.h>
+#include <asm/io.h>
 
 #define SRAM_GRANULARITY     32
 

drivers/misc/sram.c static int sram_probe(struct platform_device *pdev) · line 144

Quoted 7 lines
drivers/misc/sram.c +1 -1
@@ -138,7 +139,7 @@ static int sram_probe(struct platform_device *pdev)
      int ret;
 
      res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
-     virt_base = devm_ioremap_resource(&pdev->dev, res);
+     virt_base = __arm_ioremap_exec(res->start, resource_size(res), false);
      if (IS_ERR(virt_base))
              return PTR_ERR(virt_base);

You can get rid of this hack by defining ioremap_exec in include/asm-generic/io.h the same way as ioremap_nocache is done:

Quoted 6 lines

#ifndef ioremap_exec
#define ioremap_exec ioremap
#endif

Then the arch that need ioremap_exec can define and implement it. Needs to be reviewed on LKML naturally :)

The similar statement for nocache in asm-generic/io.h appears in an #ifndef CONFIG_MMU block. I think the better example would be ioremap_wc, which looks like:

#ifndef ARCH_HAS_IOREMAP_WC
#define ioremap_wc ioremap_nocache
#endif

Course, ioremap_exec on ARM has a slight complication since it has an extra bool nocache argument.

author
#6 lore raw

RFC 4/4 ARM: AM33XX: Move suspend/resume assembly to C

In reply to Russ Dill

This provides a proof of concept of the SRAM helpers. The SRAM_SECTIONS macro is added to vmlinux.lds.S, the code in sleep33xx.S is translated to C and moved to sleep33xx.c. In the process, a struct used for passing arguments is removed and instead the fixed arguments are only passed once at init time, only an argument with some flags remains.

---
 arch/arm/kernel/vmlinux.lds.S   |   2 +
 arch/arm/mach-omap2/Makefile    |   2 +-
 arch/arm/mach-omap2/pm33xx.c    |  50 ++---
 arch/arm/mach-omap2/pm33xx.h    |  23 +--
 arch/arm/mach-omap2/sleep33xx.S | 394 ----------------------------------------
 arch/arm/mach-omap2/sleep33xx.c | 309 +++++++++++++++++++++++++++++++
 arch/arm/mach-omap2/sram.c      |  15 --
 7 files changed, 339 insertions(+), 456 deletions(-)
 delete mode 100644 arch/arm/mach-omap2/sleep33xx.S
 create mode 100644 arch/arm/mach-omap2/sleep33xx.c
arch/arm/kernel/vmlinux.lds.S +2 -0
index 7bcee5c..21c3b64 100644
@@ -112,6 +112,8 @@ SECTIONS
 			ARM_CPU_KEEP(PROC_INFO)
 	}
 
+	SRAM_SECTIONS(am33xx)
+
 	RO_DATA(PAGE_SIZE)
 
 	. = ALIGN(4);
arch/arm/mach-omap2/Makefile +1 -1
index dcf5d89..195485c 100644
@@ -95,7 +95,7 @@ obj-$(CONFIG_POWER_AVS_OMAP_CLASS3)    += smartreflex-class3.o
 
 AFLAGS_sleep24xx.o			:=-Wa,-march=armv6
 AFLAGS_sleep34xx.o			:=-Wa,-march=armv7-a$(plus_sec)
-AFLAGS_sleep33xx.o			:=-Wa,-march=armv7-a$(plus_sec)
+CFLAGS_sleep33xx.o			+= -fPIC -march=armv7-a
 
 endif
 
arch/arm/mach-omap2/pm33xx.c +20 -30

112 lines not on this page · raw message

arch/arm/mach-omap2/pm33xx.h +7 -16
index b9c3a3f..aaa5f71 100644
@@ -16,10 +16,9 @@
 #ifndef __ARCH_ARM_MACH_OMAP2_PM33XX_H
 #define __ARCH_ARM_MACH_OMAP2_PM33XX_H
 
+#include <linux/kernel.h>
 #include "control.h"
 
-#ifndef __ASSEMBLER__
-
 struct am33xx_pm_context {
 	struct am33xx_ipc_data	ipc;
 	struct firmware		*firmware;
@@ -28,19 +27,6 @@ struct am33xx_pm_context {
 	u32			ver;
 };
 
-/*
- * Params passed to suspend routine
- *
- * Since these are used to load into registers by suspend code,
- * entries here must always be in sync with the suspend code
- * in arm/mach-omap2/sleep33xx.S
- */
-struct am33xx_suspend_params {
-	void __iomem *emif_addr_virt;
-	u32 wfi_flags;
-	void __iomem *dram_sync;
-};
-
 struct wakeup_src {
 	int irq_nr;
 	char src[10];
@@ -55,8 +41,13 @@ int wkup_m3_copy_code(const u8 *data, size_t size);
 int wkup_m3_prepare(void);
 void wkup_m3_register_txev_handler(void (*txev_handler)(void));
 int am33xx_do_sram_cpuidle(u32, u32);
+int am33xx_suspend(long unsigned int flags);
+void am33xx_resume_trampoline(void);
+void am33xx_sram_init(void __iomem *emif_base, void __iomem *dram_sync);
 
-#endif
+#define __sram_am33xx		__section(.sram.am33xx.text)
+#define __sram_am33xxdata	__section(.sram.am33xx.data)
+#define __sram_am33xxconst	__section(.sram.am33xx.rodata)
 
 #define	IPC_CMD_DS0			0x4
 #define	IPC_CMD_IDLE			0xd
arch/arm/mach-omap2/sleep33xx.S deleted +0 -394

394 lines not on this page · raw message

arch/arm/mach-omap2/sleep33xx.c new file +309 -0

309 lines not on this page · raw message

arch/arm/mach-omap2/sram.c +0 -15
index 303c562..97b7f77 100644
@@ -285,19 +285,6 @@ static inline int omap34xx_sram_init(void)
 }
 #endif /* CONFIG_ARCH_OMAP3 */
 
-#ifdef CONFIG_SOC_AM33XX
-static inline int am33xx_sram_init(void)
-{
-	am33xx_push_sram_idle();
-	return 0;
-}
-#else
-static inline int am33xx_sram_init(void)
-{
-	return 0;
-}
-#endif
-
 int __init omap_sram_init(void)
 {
 	omap_detect_sram();
@@ -307,8 +294,6 @@ int __init omap_sram_init(void)
 		omap242x_sram_init();
 	else if (cpu_is_omap2430())
 		omap243x_sram_init();
-	else if (soc_is_am33xx())
-		am33xx_sram_init();
 	else if (cpu_is_omap34xx())
 		omap34xx_sram_init();
 
Signature · 1 line
1.8.3.2
#7 lore raw

cover Create infrastructure for running C code from SRAM.

In reply to Russ Dill

Hi,

El 03/09/13 13:44, Russ Dill escribi?:

Quoted 59 lines

This RFC patchset explores an idea for loading C code into SRAM. Currently, all the code I'm aware of that needs to run from SRAM is written in assembler. The most common reason for code needing to run from SRAM is that the memory controller is being disabled/ enabled or is already disabled. arch/arm has by far the most examples, but code also exists in powerpc and sh.

The code is written in asm for two primary reasons. First so that markers can be put in indicating the size of the code they it can be copied. Second so that data can be placed along with text and accessed in a position independant manner.

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
   __sram_am33xx_start = .;
   *(.sram.am33xx.*)
   __sram_am33xx_end = .;
}

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

[1] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4984c6
[2] http://www.spinics.net/lists/linux-omap/msg96504.html

Russ Dill (4):
Misc: SRAM: Create helpers for loading C code into SRAM
ARM: SRAM: Add macro for generating SRAM resume trampoline
Misc: SRAM: Hack for allowing executable code in SRAM.
ARM: AM33XX: Move suspend/resume assembly to C

  arch/arm/include/asm/suspend.h    |  14 ++
  arch/arm/kernel/vmlinux.lds.S     |   2 +
  arch/arm/mach-omap2/Makefile      |   2 +-
  arch/arm/mach-omap2/pm33xx.c      |  50 ++---
  arch/arm/mach-omap2/pm33xx.h      |  23 +--
  arch/arm/mach-omap2/sleep33xx.S   | 394 --------------------------------------
  arch/arm/mach-omap2/sleep33xx.c   | 309 ++++++++++++++++++++++++++++++
  arch/arm/mach-omap2/sram.c        |  15 --
  drivers/misc/sram.c               | 106 +++++++++-
  include/asm-generic/vmlinux.lds.h |   7 +
  include/linux/sram.h              |  44 +++++
  11 files changed, 509 insertions(+), 457 deletions(-)
Quoted 4 lines

delete mode 100644 arch/arm/mach-omap2/sleep33xx.S create mode 100644 arch/arm/mach-omap2/sleep33xx.c create mode 100644 include/linux/sram.h

I'm interested in this, as I'll need something like it for suspend/resume on sunxi. Unfortunately, I only got the cover letter on my email, and the web lakml archives don't seem to have the rest either. After a bit of searching on Google I found a copy on linux-omap[1], but it'd be great if I didn't have to hunt for the patches :)

I only have one comment, from a quick look at the code

+ memcpy((void *) chunk->addr, data, sz);
+ flush_icache_range(chunk->addr, chunk->addr + sz);

How would that behave on Thumb-2 mode? I believe that's the reason why fncpy() got introduced[2] some time ago.

Thanks for working on this!

Emilio

[1] http://www.mail-archive.com/linux-omap at vger.kernel.org/msg94995.html
[2] http://www.spinics.net/lists/arm-kernel/msg110706.html

author
#8 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 75 lines · Emilio L?pez

Hi,

El 03/09/13 13:44, Russ Dill escribi?:

Quoted 66 lines

This RFC patchset explores an idea for loading C code into SRAM. Currently, all the code I'm aware of that needs to run from SRAM is written
in assembler. The most common reason for code needing to run from SRAM is that the memory controller is being disabled/ enabled or is already disabled. arch/arm has by far the most examples, but code also exists in powerpc and sh.

The code is written in asm for two primary reasons. First so that markers can be put in indicating the size of the code they it can be copied. Second
so that data can be placed along with text and accessed in a position independant manner.

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile
time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
   __sram_am33xx_start = .;
   *(.sram.am33xx.*)
   __sram_am33xx_end = .;
}

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for
translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

[1]
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4984c6
[2] http://www.spinics.net/lists/linux-omap/msg96504.html

Russ Dill (4):
Misc: SRAM: Create helpers for loading C code into SRAM
ARM: SRAM: Add macro for generating SRAM resume trampoline
Misc: SRAM: Hack for allowing executable code in SRAM.
ARM: AM33XX: Move suspend/resume assembly to C

  arch/arm/include/asm/suspend.h    |  14 ++
  arch/arm/kernel/vmlinux.lds.S     |   2 +
  arch/arm/mach-omap2/Makefile      |   2 +-
  arch/arm/mach-omap2/pm33xx.c      |  50 ++---
  arch/arm/mach-omap2/pm33xx.h      |  23 +--
  arch/arm/mach-omap2/sleep33xx.S   | 394
--------------------------------------
  arch/arm/mach-omap2/sleep33xx.c   | 309 ++++++++++++++++++++++++++++++
  arch/arm/mach-omap2/sram.c        |  15 --
  drivers/misc/sram.c               | 106 +++++++++-
  include/asm-generic/vmlinux.lds.h |   7 +
  include/linux/sram.h              |  44 +++++
  11 files changed, 509 insertions(+), 457 deletions(-)
Quoted 4 lines

delete mode 100644 arch/arm/mach-omap2/sleep33xx.S create mode 100644 arch/arm/mach-omap2/sleep33xx.c create mode 100644 include/linux/sram.h

Quoted 5 lines

I'm interested in this, as I'll need something like it for suspend/resume on sunxi. Unfortunately, I only got the cover letter on my email, and the web lakml archives don't seem to have the rest either. After a bit of searching on Google I found a copy on linux-omap[1], but it'd be great if I didn't have to hunt for the patches :)

The mails to arm-kernel are "awaiting moderation".

Quoted 9 lines

I only have one comment, from a quick look at the code

(quoted diff) +2 -0
+       memcpy((void *) chunk->addr, data, sz);
+       flush_icache_range(chunk->addr, chunk->addr + sz);

How would that behave on Thumb-2 mode? I believe that's the reason why fncpy() got introduced[2] some time ago.

Thanks for working on this!

I think this is already taken care of by the way sram.c is using genalloc. The allocation returned should be aligned to 32 bytes. The thumb bit shouldn't be an issue as code is copied based on the start and end makers made by the linker. I may need to add .align statements in the linker so that the start and end markers for the copied code are aligned to at least 8 bytes.

Thanks!

Quoted 9 lines

Emilio

[1] http://www.mail-archive.com/linux-omap at vger.kernel.org/msg94995.html
[2] http://www.spinics.net/lists/arm-kernel/msg110706.html

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#9 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 9 lines · Russ Dill
Quoted 5 lines · Emilio L?pez

I'm interested in this, as I'll need something like it for suspend/resume on sunxi. Unfortunately, I only got the cover letter on my email, and the web lakml archives don't seem to have the rest either. After a bit of searching on Google I found a copy on linux-omap[1], but it'd be great if I didn't have to hunt for the patches :)

The mails to arm-kernel are "awaiting moderation".

That is because you have RFC in the subject line instead of PATCH RFC

Thanks,
Sekhar

#10 lore raw

cover Create infrastructure for running C code from SRAM.

In reply to Russ Dill
Quoted 6 lines · Russ Dill

I think this is already taken care of by the way sram.c is using genalloc. The allocation returned should be aligned to 32 bytes. The thumb bit shouldn't be an issue as code is copied based on the start and end makers made by the linker. I may need to add .align statements in the linker so that the start and end markers for the copied code are aligned to at least 8 bytes.

I think you need to read up on what fncpy does... there's more to it than just merely copying code at an appropriate alignment.

#11 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 10 lines · Russell King - ARM Linux
Quoted 6 lines · Russ Dill

I think this is already taken care of by the way sram.c is using genalloc. The allocation returned should be aligned to 32 bytes. The thumb bit shouldn't be an issue as code is copied based on the start and end makers made by the linker. I may need to add .align statements in the linker so that the start and end markers for the copied code are aligned to at least 8 bytes.

I think you need to read up on what fncpy does... there's more to it than just merely copying code at an appropriate alignment.

The technique of putting each loadable blob in a specific vmlinux section, and then adjusting entry-point symbols by adding/subtracting the appropriate offset, probably does work.

This relies on the functions' code alignment requirement being honoured by both the vmlinux link map, and the allocator used to find SRAM space to copy the functions to.

Searching the entire list of known blobs every time we want to convert a symbol seems unnecessary though. Surely the caller could know the blob<->symbol mapping anyway?

One thing fncpy() doesn't provide is a way to copy groups of functions that call each other, if vmlinux needs to know about any symbol other than the one at the start. We might need a better mechanism if that is needed.

I actually wonder whether fncpy() contains a buglet, whereby flush_icache_range() is used instead of coherent_kern_range(). The SRAM is probably not mapped cached, but at least a DSB would be needed before flushing the relevant lines from the I-cache.

However, flush_icache_range() seems to be implemented by a call to coherent_kern_range() anyway, so perhaps that's not a problem.

Cheers
---Dave

author
#12 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 16 lines · Dave Martin
Quoted 10 lines · Russell King - ARM Linux
Quoted 6 lines · Russ Dill

I think this is already taken care of by the way sram.c is using genalloc. The allocation returned should be aligned to 32 bytes. The thumb bit shouldn't be an issue as code is copied based on the start and end makers made by the linker. I may need to add .align statements in the linker so that the start and end markers for the copied code are aligned to at least 8 bytes.

I think you need to read up on what fncpy does... there's more to it than just merely copying code at an appropriate alignment.

The technique of putting each loadable blob in a specific vmlinux section, and then adjusting entry-point symbols by adding/subtracting the appropriate offset, probably does work.

Quoted 7 lines

This relies on the functions' code alignment requirement being honoured by both the vmlinux link map, and the allocator used to find SRAM space to copy the functions to.

Searching the entire list of known blobs every time we want to convert a symbol seems unnecessary though. Surely the caller could know the blob<->symbol mapping anyway?

It doesn't search the list of known blobs, only loaded blobs. On all the platforms I'm aware of, only one SRAM section is loaded with code.

Quoted 10 lines

One thing fncpy() doesn't provide is a way to copy groups of functions that call each other, if vmlinux needs to know about any symbol other than the one at the start. We might need a better mechanism if that is needed.

I actually wonder whether fncpy() contains a buglet, whereby flush_icache_range() is used instead of coherent_kern_range(). The SRAM is probably not mapped cached, but at least a DSB would be needed before flushing the relevant lines from the I-cache.

It is mapped cached on most platforms.

Quoted 10 lines

However, flush_icache_range() seems to be implemented by a call to coherent_kern_range() anyway, so perhaps that's not a problem.

Cheers
---Dave

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#13 lore raw

cover Create infrastructure for running C code from SRAM.

In reply to Dave Martin
Quoted 4 lines · Dave Martin

I actually wonder whether fncpy() contains a buglet, whereby flush_icache_range() is used instead of coherent_kern_range(). The SRAM is probably not mapped cached, but at least a DSB would be needed before flushing the relevant lines from the I-cache.

flush_icache_range() is correct - it's there to ensure that memory which has been written will be readable to the instruction stream. That's it's whole purpose, and it's used when modules are loaded too.

You're reading too much into the name: it doesn't just touch the I cache.

author
#14 lore raw

cover Create infrastructure for running C code from SRAM.

In reply to Russell King - ARM Linux
Quoted 10 lines · Russell King - ARM Linux
Quoted 6 lines · Russ Dill

I think this is already taken care of by the way sram.c is using genalloc. The allocation returned should be aligned to 32 bytes. The thumb bit shouldn't be an issue as code is copied based on the start and end makers made by the linker. I may need to add .align statements in the linker so that the start and end markers for the copied code are aligned to at least 8 bytes.

I think you need to read up on what fncpy does... there's more to it than just merely copying code at an appropriate alignment.

Yes, I need to add a pair of inlines that do the asm trickery to/from function addresses. Thanks.

#15 lore raw

cover Create infrastructure for running C code from SRAM.

In reply to Russ Dill
Quoted 7 lines · Russ Dill

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

What are you doing about the various gcc utility functions that may be implicitly called from C code such as memcpy and memset?

Quoted 10 lines

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
__sram_am33xx_start = .;
*(.sram.am33xx.*)
__sram_am33xx_end = .;
}

Quoted 7 lines

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

What about the relocations which will need to be fixed up - eg, addresses in the literal pool, the GOT table contents, etc? You say nothing about this.

#16 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 30 lines · Russell King - ARM Linux
Quoted 7 lines · Russ Dill

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

What are you doing about the various gcc utility functions that may be implicitly called from C code such as memcpy and memset?

Quoted 10 lines

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
__sram_am33xx_start = .;
*(.sram.am33xx.*)
__sram_am33xx_end = .;
}

Quoted 7 lines

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

Quoted 3 lines

What about the relocations which will need to be fixed up - eg, addresses in the literal pool, the GOT table contents, etc? You say nothing about this.

I was also thinking about this, and there are more problems.

As well as what has already been mentioned:

* Calls from inside the SRAM code to vmlinux (including lib1funcs etc.) will typically break, except on architectures where function calls are (absolute by default not ARM).

* The compiler/linker won't detect unsafe constructs or code generation, because it assumes that anything built with -fPIC is going to be patched up later by ld.so or equivalent.

* The GOT is generated by the linker, and is a single table. Yet each SRAM blob needs to be able to refer to its own GOT entries position- independently. Moving the blobs independently won't work.

In other words: -fPIC does not generate position-independent code.

It generates position-dependent code that is easier to move around than non-fPIC code, but you still need a dynamic linker (or equivalent) to make it all work.

There are various "correct" ways to handle this, the simplest of which is probably to build each SRAM blob as a kernel module, embed the result in the kernel somehow, and then use the module loader infrastructure to handle fixing the module up to the right address.

But this is still likely to be overkill, given the small scale of the SRAM code.

Restricting such code to carefully-written assembler (as now) may be the more practical approrach, unless there's a good example of somewhere that C code would provide a big benefit.

Cheers
---Dave

author
#17 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 35 lines · Dave Martin
Quoted 30 lines · Russell King - ARM Linux
Quoted 7 lines · Russ Dill

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

What are you doing about the various gcc utility functions that may be implicitly called from C code such as memcpy and memset?

Quoted 10 lines

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
__sram_am33xx_start = .;
*(.sram.am33xx.*)
__sram_am33xx_end = .;
}

Quoted 7 lines

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

Quoted 3 lines

What about the relocations which will need to be fixed up - eg, addresses in the literal pool, the GOT table contents, etc? You say nothing about this.

Quoted 7 lines

I was also thinking about this, and there are more problems.

As well as what has already been mentioned:

* Calls from inside the SRAM code to vmlinux (including lib1funcs etc.) will typically break, except on architectures where function calls are (absolute by default not ARM).

As in the response to RMK, I think compiler flags are enough to prevent implicit memcpy/memset calls. The code would not be allowed to do divisions, module, or 64 bit multiplication. A make rule would check the sram sections for any dynamically relocatable symbols.

Quoted 3 lines

* The compiler/linker won't detect unsafe constructs or code generation, because it assumes that anything built with -fPIC is going to be patched up later by ld.so or equivalent.

Can you provide examples of what some of these other unsafe constructs might be?

Quoted 3 lines

* The GOT is generated by the linker, and is a single table. Yet each SRAM blob needs to be able to refer to its own GOT entries position- independently. Moving the blobs independently won't work.

Would GOT entries only exist if there are accesses to .data or .bss? The SRAM C code would not support such a thing, only access to data and text within the SRAM grouping is allowed. Is there a way to make the compiler or linker complain if such an access is done? If not, it'd be another make rule as above.

Quoted 5 lines

In other words: -fPIC does not generate position-independent code.

It generates position-dependent code that is easier to move around than non-fPIC code, but you still need a dynamic linker (or equivalent) to make it all work.

arch/arm/boot/compressed/ seems to manage it. Hopefully, by allowing only more limited code, I can get by with less tricks.

Quoted 7 lines

There are various "correct" ways to handle this, the simplest of which is probably to build each SRAM blob as a kernel module, embed the result in the kernel somehow, and then use the module loader infrastructure to handle fixing the module up to the right address.

But this is still likely to be overkill, given the small scale of the SRAM code.

Yes, I'm pretty sure several people would scream rather loudly if getting suspend/resume support on their platform required CONFIG_MODULES=y.

Quoted 3 lines

Restricting such code to carefully-written assembler (as now) may be the more practical approrach, unless there's a good example of somewhere that C code would provide a big benefit.

There are currently about 5000 or so lines of assembly code in arch/arm that are used for suspend/resume stubs. In one stage of am335x development, the sleep/resume stub for am335x was about 1200 lines long. Since then, a lot of that code has been moved to a firmware blob, but there has been some pushback on that, which is why I'm investigating this path. Especially given that there are some future platforms that will follow the am335x pm model.

author
#18 lore raw

cover Create infrastructure for running C code from SRAM.

In reply to Russell King - ARM Linux
Quoted 11 lines · Russell King - ARM Linux
Quoted 7 lines · Russ Dill

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

What are you doing about the various gcc utility functions that may be implicitly called from C code such as memcpy and memset?

That would create a problem. Would '-ffreestanding' be the correct flag to add? As far as the family of __aeabi_*, I need to add documentation stating that on ARM, you can't divide, perform modulo, and can't do 64 bit multiplications. I can then add a make rule that will grep the symbol lists of .sram sections for ^__aeabi_. Is this enough?

Quoted 18 lines
Quoted 10 lines

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
__sram_am33xx_start = .;
*(.sram.am33xx.*)
__sram_am33xx_end = .;
}

Quoted 7 lines

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

Quoted 3 lines

What about the relocations which will need to be fixed up - eg, addresses in the literal pool, the GOT table contents, etc? You say nothing about this.

The C code would need to be written so that such accesses do not occur. From functions that are in the sram text section, only accesses to other sram sections in their group would be allowed. And above, a compilation step could be added to make the compilation fail when such things happen.

The direction you are going though is good, if this is fragile, it doesn't put us in a better place then having fragile asm that at least complies reliably. As I'm looking towards the future and platforms that are similar to am335x in that they are placing more and more of the PM state machine burden on the CPU, I'd really like to try to make this happen.

#19 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 11 lines · Russ Dill
Quoted 9 lines · Russell King - ARM Linux
Quoted 7 lines · Russ Dill

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

Quoted 5 lines
Quoted 2 lines

What are you doing about the various gcc utility functions that may be implicitly called from C code such as memcpy and memset?

That would create a problem. Would '-ffreestanding' be the correct flag to add?

No, unfortunately, -ffreestanding won't prevent GCC from generating implicit calls to memzero() et al. These are mainly issued when using initialized non-POD stack variables so avoiding those might help you there.

Quoted 6 lines

As far as the family of __aeabi_*, I need to add documentation stating that on ARM, you can't divide, perform modulo, and can't do 64 bit multiplications. I can then add a make rule that will grep the symbol lists of .sram sections for ^__aeabi_. Is this enough?

Well, even printk() needs integer division for its %d/%u modifiers, so this is really not so easy to achieve.

Quoted 22 lines
Quoted 18 lines
Quoted 10 lines

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
__sram_am33xx_start = .;
*(.sram.am33xx.*)
__sram_am33xx_end = .;
}

Quoted 7 lines

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

Quoted 3 lines

What about the relocations which will need to be fixed up - eg, addresses in the literal pool, the GOT table contents, etc? You say nothing about this.

Quoted 6 lines

The C code would need to be written so that such accesses do not occur. From functions that are in the sram text section, only accesses to other sram sections in their group would be allowed. And above, a compilation step could be added to make the compilation fail when such things happen.

The point is that, sadly, GCC is just not very good at generating relocatable code for embedded targets. Playing with -fvisibility may result in code that contains fewer dynamic relocations, but you will always end up with a few that need to be fixed up before the code can run. Another thing to note is that usually, these relocations can only be fixed up once, as the addend is overwritten by the fixed-up address. This means that the code can only run in SRAM, and you should probably best avoid the module loader machinery as it may clobber the addends before you get to process them.

One thing that remains implicit in this discussion is that you are executing from SRAM because DRAM is not available (I presume). Wouldn't it be better to treat the code that lives in the SRAM as a completely separate executable? You can generate a PIE executable that supplies minimal memzero et al, fixup the relocations yourself (look at the uboot sources for an example of this) and you will be absolutely sure that the code can run completely autonomously. In fact, some of this stuff could potentially be reused for other disjoint execution domains such as TZ secure world.

Regards,
Ard.

Quoted 11 lines

The direction you are going though is good, if this is fragile, it doesn't put us in a better place then having fragile asm that at least complies reliably. As I'm looking towards the future and platforms that are similar to am335x in that they are placing more and more of the PM state machine burden on the CPU, I'd really like to try to make this happen.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

author
#20 lore raw

cover Create infrastructure for running C code from SRAM.

Quoted 71 lines · Ard Biesheuvel
Quoted 11 lines · Russ Dill
Quoted 9 lines · Russell King - ARM Linux
Quoted 7 lines · Russ Dill

SRAM handling code is in the process of being moved from arch directories into drivers/misc/sram.c using device tree and genalloc [1] [2]. This RFC patchset builds on that, including the limitation that the SRAM address is not known at compile time. Because the SRAM address is not known at compile time, the code that runs from SRAM must be compiled with -fPIC. Even if the code were loaded to a fixed virtual address, portions of the code must often be run with the MMU disabled.

Quoted 5 lines
Quoted 2 lines

What are you doing about the various gcc utility functions that may be implicitly called from C code such as memcpy and memset?

That would create a problem. Would '-ffreestanding' be the correct flag to add?

No, unfortunately, -ffreestanding won't prevent GCC from generating implicit calls to memzero() et al. These are mainly issued when using initialized non-POD stack variables so avoiding those might help you there.

Quoted 6 lines

As far as the family of __aeabi_*, I need to add documentation stating that on ARM, you can't divide, perform modulo, and can't do 64 bit multiplications. I can then add a make rule that will grep the symbol lists of .sram sections for ^__aeabi_. Is this enough?

Well, even printk() needs integer division for its %d/%u modifiers, so this is really not so easy to achieve.

Quoted 22 lines
Quoted 18 lines
Quoted 10 lines

The general idea is that for each SRAM user (such as an SoC specific suspend/resume mechanism) to create a group of sections. The section group is created with a single macro for each user, but end up looking like this:

.sram.am33xx : AT(ADDR(.sram.am33xx) - 0) {
__sram_am33xx_start = .;
*(.sram.am33xx.*)
__sram_am33xx_end = .;
}

Quoted 7 lines

Any data or functions that should be copied to SRAM for this use should be maked with an appropriate __section() attribute. A helper is then added for translating between the original kernel symbol, and the address of that function or variable once it has been copied into SRAM. Once control is passed to a function within the SRAM section grouping, it can access any variables or functions within that same SRAM section grouping without translation.

Quoted 3 lines

What about the relocations which will need to be fixed up - eg, addresses in the literal pool, the GOT table contents, etc? You say nothing about this.

Quoted 6 lines

The C code would need to be written so that such accesses do not occur. From functions that are in the sram text section, only accesses to other sram sections in their group would be allowed. And above, a compilation step could be added to make the compilation fail when such things happen.

The point is that, sadly, GCC is just not very good at generating relocatable code for embedded targets. Playing with -fvisibility may result in code that contains fewer dynamic relocations, but you will always end up with a few that need to be fixed up before the code can run. Another thing to note is that usually, these relocations can only be fixed up once, as the addend is overwritten by the fixed-up address. This means that the code can only run in SRAM, and you should probably best avoid the module loader machinery as it may clobber the addends before you get to process them.

Quoted 9 lines

One thing that remains implicit in this discussion is that you are executing from SRAM because DRAM is not available (I presume). Wouldn't it be better to treat the code that lives in the SRAM as a completely separate executable? You can generate a PIE executable that supplies minimal memzero et al, fixup the relocations yourself (look at the uboot sources for an example of this) and you will be absolutely sure that the code can run completely autonomously. In fact, some of this stuff could potentially be reused for other disjoint execution domains such as TZ secure world.

This is the path I'm going down, but I'm trying to do it without relocations. I'm following the model of arch/arm/boot/compressed and generating a relocatable gcc builtin library with weak symbols containing lib1funcs.S, string.c, ashldi3.S, and some stubs for div0 and the unwind symbols, call in sramlib.o.

I'm then doing an objcopy of the .sramlib section, and the .sram.* sections into a single object file and performing a link with a linker script like:

SECTIONS

{
    .text : { *(.sramlib) }

    OVERLAY ALIGN(32) : NOCROSSREFS
    {
        .sram.am33xx { *(.sram.am33xx.*) }
        .sram.am437x { *(.sram.am437x.*) }
    }
}

It produces output without any relocations, but from there I'm a little fuzzy on how to get the symbols of functions and variables into the kernel. In the meantime, I'll look into the u-boot methods.