Skip to contentSkip to conversation

arm64: improve debuggability

PATCH Superseded Nadav Amit 2 participants 11 messages Jun 25, 2023 – Jun 26, 2023

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Nadav Amit

author
#0 lore raw

v2 cover arm64: improve debuggability

From: Nadav Amit <[email protected]>

My recent experience in debugging ARM64 tests on EFI was not as fun as I expected it to be.

There were several reasons for that besides the questionable definition of "fun":

1. ARM64 is not compiled with frame pointers and there is no stack unwinder when the stack is dumped.

2. Building an EFI drops the debug information.

3. The addresses that are printed on dump_stack() and the use of GDB are hard because taking code relocation into account is non trivial.

The patches help both ARM64 and EFI for this matter. The image address is printed when EFI is used to allow the use of GDB. Symbols are emitted into a separate debug file. The frame pointer is included and special entry is added upon an exception to allow backtracing across exceptions.

v1->v2:
* Andrew comments [see in individual patches]
* Few cleanups

Nadav Amit (6):
efi: keep efi debug information in a separate file lib/stack: print base addresses on efi
arm64: enable frame pointer and support stack unwinding
arm64: stack: update trace stack on exception
efi: print address of image
arm64: dump stack on bad exception

 arm/Makefile.arm        |  3 --
 arm/Makefile.arm64      |  1 +
 arm/Makefile.common     |  8 +++++-
 arm/cstart64.S          | 13 +++++++++
 configure               |  3 ++
 lib/arm64/asm-offsets.c |  6 +++-
 lib/arm64/asm/stack.h   |  3 ++
 lib/arm64/processor.c   |  1 +
 lib/arm64/stack.c       | 62 +++++++++++++++++++++++++++++++++++++++++
 lib/efi.c               |  4 +++
 lib/stack.c             | 31 +++++++++++++++++++--
 x86/Makefile.common     |  5 +++-
 12 files changed, 132 insertions(+), 8 deletions(-)
 create mode 100644 lib/arm64/stack.c
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v2 1/6 efi: keep efi debug information in a separate file

From: Nadav Amit <[email protected]>

Debugging tests that run on EFI is hard because the debug information is not included in the EFI file. Dump it into a separeate .debug file to allow the use of gdb or pretty_print_stacks script.

---

v1->v2:
* Making clean should remove .debug [Andrew]
* x86 EFI support [Andrew]

---
 arm/Makefile.common | 5 ++++-
 x86/Makefile.common | 5 ++++-
 2 files changed, 8 insertions(+), 2 deletions(-)
arm/Makefile.common +4 -1
index d60cf8c..9b45a8f 100644
@@ -78,6 +78,9 @@ ifeq ($(CONFIG_EFI),y)
 
 %.efi: %.so
 	$(call arch_elf_check, $^)
+	$(OBJCOPY) --only-keep-debug $^ [email protected]
+	$(OBJCOPY) --strip-debug $^
+	$(OBJCOPY) [email protected] $^
 	$(OBJCOPY) \
 		-j .text -j .sdata -j .data -j .dynamic -j .dynsym \
 		-j .rel -j .rela -j .rel.* -j .rela.* -j .rel* -j .rela* \
@@ -103,7 +106,7 @@ $(libeabi): $(eabiobjs)
 	$(AR) rcs $@ $^
 
 arm_clean: asm_offsets_clean
-	$(RM) $(TEST_DIR)/*.{o,flat,elf,so,efi} $(libeabi) $(eabiobjs) \
+	$(RM) $(TEST_DIR)/*.{o,flat,elf,so,efi,debug} $(libeabi) $(eabiobjs) \
 	      $(TEST_DIR)/.*.d $(TEST_DIR)/efi/.*.d lib/arm/.*.d
 
 generated-files = $(asm-offsets)
x86/Makefile.common +4 -1
index 9f2bc93..c42c3e4 100644
@@ -54,6 +54,9 @@ ifeq ($(CONFIG_EFI),y)
 	@chmod a-x $@
 
 %.efi: %.so
+	$(OBJCOPY) --only-keep-debug $^ [email protected]
+	$(OBJCOPY) --strip-debug $^
+	$(OBJCOPY) [email protected] $^
 	$(OBJCOPY) \
 		-j .text -j .sdata -j .data -j .dynamic -j .dynsym -j .rel \
 		-j .rela -j .reloc -S --target=$(FORMAT) $< $@
@@ -124,4 +127,4 @@ arch_clean:
 	$(RM) $(TEST_DIR)/*.o $(TEST_DIR)/*.flat $(TEST_DIR)/*.elf \
 	$(TEST_DIR)/.*.d lib/x86/.*.d \
 	$(TEST_DIR)/efi/*.o $(TEST_DIR)/efi/.*.d \
-	$(TEST_DIR)/*.so $(TEST_DIR)/*.efi
+	$(TEST_DIR)/*.so $(TEST_DIR)/*.efi $(TEST_DIR)/*.debug
Signature · 1 line
2.34.1
#2 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] efi: keep efi debug information in a separate file

Quoted 5 lines · Nadav Amit

From: Nadav Amit <[email protected]>

Debugging tests that run on EFI is hard because the debug information is not included in the EFI file. Dump it into a separeate .debug file to allow the use of gdb or pretty_print_stacks script.

We're still missing the run_tests.sh change needed for pretty_print_stacks, but I can post that myself.

Thanks,
drew

Quoted 59 lines

---

v1->v2:
* Making clean should remove .debug [Andrew]
* x86 EFI support [Andrew]

---
 arm/Makefile.common | 5 ++++-
 x86/Makefile.common | 5 ++++-
 2 files changed, 8 insertions(+), 2 deletions(-)
arm/Makefile.common +4 -1
index d60cf8c..9b45a8f 100644
@@ -78,6 +78,9 @@ ifeq ($(CONFIG_EFI),y)
 
 %.efi: %.so
 	$(call arch_elf_check, $^)
+	$(OBJCOPY) --only-keep-debug $^ [email protected]
+	$(OBJCOPY) --strip-debug $^
+	$(OBJCOPY) [email protected] $^
 	$(OBJCOPY) \
 		-j .text -j .sdata -j .data -j .dynamic -j .dynsym \
 		-j .rel -j .rela -j .rel.* -j .rela.* -j .rel* -j .rela* \
@@ -103,7 +106,7 @@ $(libeabi): $(eabiobjs)
 	$(AR) rcs $@ $^
 
 arm_clean: asm_offsets_clean
-	$(RM) $(TEST_DIR)/*.{o,flat,elf,so,efi} $(libeabi) $(eabiobjs) \
+	$(RM) $(TEST_DIR)/*.{o,flat,elf,so,efi,debug} $(libeabi) $(eabiobjs) \
 	      $(TEST_DIR)/.*.d $(TEST_DIR)/efi/.*.d lib/arm/.*.d
 
 generated-files = $(asm-offsets)
x86/Makefile.common +4 -1
index 9f2bc93..c42c3e4 100644
@@ -54,6 +54,9 @@ ifeq ($(CONFIG_EFI),y)
 	@chmod a-x $@
 
 %.efi: %.so
+	$(OBJCOPY) --only-keep-debug $^ [email protected]
+	$(OBJCOPY) --strip-debug $^
+	$(OBJCOPY) [email protected] $^
 	$(OBJCOPY) \
 		-j .text -j .sdata -j .data -j .dynamic -j .dynsym -j .rel \
 		-j .rela -j .reloc -S --target=$(FORMAT) $< $@
@@ -124,4 +127,4 @@ arch_clean:
 	$(RM) $(TEST_DIR)/*.o $(TEST_DIR)/*.flat $(TEST_DIR)/*.elf \
 	$(TEST_DIR)/.*.d lib/x86/.*.d \
 	$(TEST_DIR)/efi/*.o $(TEST_DIR)/efi/.*.d \
-	$(TEST_DIR)/*.so $(TEST_DIR)/*.efi
+	$(TEST_DIR)/*.so $(TEST_DIR)/*.efi $(TEST_DIR)/*.debug
Signature · 1 line
2.34.1
#3 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] efi: keep efi debug information in a separate file

In reply to Nadav Amit
Quoted 6 lines · Nadav Amit

From: Nadav Amit <[email protected]>

Debugging tests that run on EFI is hard because the debug information is not included in the EFI file. Dump it into a separeate .debug file to allow the use of gdb or pretty_print_stacks script.

Quoted 6 lines

---

v1->v2:
* Making clean should remove .debug [Andrew]

I forgot to point out in the previous review that not only do we need to clean but also add *.debug to .gitignore

Thanks,
drew

author
#4 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] efi: keep efi debug information in a separate file

Quoted 15 lines · Andrew Jones

!! External Email

Quoted 8 lines · Nadav Amit

From: Nadav Amit <[email protected]>

Debugging tests that run on EFI is hard because the debug information is not included in the EFI file. Dump it into a separeate .debug file to allow the use of gdb or pretty_print_stacks script.

Quoted 7 lines
Quoted 4 lines

---

v1->v2:
* Making clean should remove .debug [Andrew]

I forgot to point out in the previous review that not only do we need to clean but also add *.debug to .gitignore

I will do that for v3.

I did not (and still do not) understand what change you want for run_tests.sh so I would indeed appreciate if you do this one.

author
#5 lore raw

PATCH v2 2/6 lib/stack: print base addresses on efi

In reply to Nadav Amit

From: Nadav Amit <[email protected]>

Making sense from dumped stacks when running EFI tests is very hard due to the relocation. Fix it by adjusting the address back to the original address.

Introduce CONFIG_RELOC, which would be set on arm64 and on EFI configs.

---

v1->v2: Introduce CONFIG_RELOC to support ARM64 [Andrew]

---
 configure   |  3 +++
 lib/stack.c | 31 +++++++++++++++++++++++++++++--
 2 files changed, 32 insertions(+), 2 deletions(-)
configure +3 -0
index b665f7d..8a3c8fe 100755
@@ -416,6 +416,9 @@ EOF
 if [ "$arch" = "arm" ] || [ "$arch" = "arm64" ]; then
     echo "TARGET=$target" >> config.mak
 fi
+if [ "$efi" = "y" ] || [ "$arch" = "arm64" ]; then
+    echo "CONFIG_RELOC=y" >> config.mak
+fi
 
 cat <<EOF > lib/config.h
 #ifndef _CONFIG_H_
lib/stack.c +29 -2
index bdb23fd..dd6bfa8 100644
@@ -6,13 +6,38 @@
  */
 
 #include <libcflat.h>
+#include <stdbool.h>
 #include <stack.h>
 
 #define MAX_DEPTH 20
 
+#ifdef CONFIG_RELOC
+extern char _text, _etext;
+
+static bool base_address(const void *rebased_addr, unsigned long *addr)
+{
+	unsigned long ra = (unsigned long)rebased_addr;
+	unsigned long start = (unsigned long)&_text;
+	unsigned long end = (unsigned long)&_etext;
+
+	if (ra < start || ra >= end)
+		return false;
+
+	*addr = ra - start;
+	return true;
+}
+#else
+static bool base_address(const void *rebased_addr, unsigned long *addr)
+{
+	*addr = (unsigned long)rebased_addr;
+	return true;
+}
+#endif
+
 static void print_stack(const void **return_addrs, int depth,
 			bool top_is_return_address)
 {
+	unsigned long addr;
 	int i = 0;
 
 	printf("\tSTACK:");
@@ -20,12 +45,14 @@ static void print_stack(const void **return_addrs, int depth,
 	/* @addr indicates a non-return address, as expected by the stack
 	 * pretty printer script. */
 	if (depth > 0 && !top_is_return_address) {
-		printf(" @%lx", (unsigned long) return_addrs[0]);
+		if (base_address(return_addrs[0], &addr))
+			printf(" @%lx", addr);
 		i++;
 	}
 
 	for (; i < depth; i++) {
-		printf(" %lx", (unsigned long) return_addrs[i]);
+		if (base_address(return_addrs[i], &addr))
+			printf(" %lx", addr);
 	}
 	printf("\n");
 }
Signature · 1 line
2.34.1
#6 lore raw

Re: [kvm-unit-tests PATCH v2 2/6] lib/stack: print base addresses on efi

Quoted 23 lines · Nadav Amit

From: Nadav Amit <[email protected]>

Making sense from dumped stacks when running EFI tests is very hard due to the relocation. Fix it by adjusting the address back to the original address.

Introduce CONFIG_RELOC, which would be set on arm64 and on EFI configs.

---

v1->v2: Introduce CONFIG_RELOC to support ARM64 [Andrew]

---
 configure   |  3 +++
 lib/stack.c | 31 +++++++++++++++++++++++++++++--
 2 files changed, 32 insertions(+), 2 deletions(-)
configure +0 -0
index b665f7d..8a3c8fe 100755

configure EOF · line 420

Quoted 5 lines
configure +2 -0
@@ -416,6 +416,9 @@ EOF
 if [ "$arch" = "arm" ] || [ "$arch" = "arm64" ]; then
     echo "TARGET=$target" >> config.mak
 fi
+if [ "$efi" = "y" ] || [ "$arch" = "arm64" ]; then
+    echo "CONFIG_RELOC=y" >> config.mak

This won't work, because, unlike CONFIG_EFI, nothing is defining CONFIG_RELOC in the build. We need CONFIG_RELOC in lib/config.h or to append -DCONFIG_RELOC to CFLAGS in a makefile when some config.mak variable is set. I think my preference would be something like

Makefile +1 -1
index 307bc291844a..ae79059e7e6f 100644
@@ -40,7 +40,7 @@ OBJDIRS += $(LIBFDT_objdir)
 
 # EFI App
 ifeq ($(CONFIG_EFI),y)
-EFI_CFLAGS := -DCONFIG_EFI
+EFI_CFLAGS := -DCONFIG_EFI -DCONFIG_RELOC
 # The following CFLAGS and LDFLAGS come from:
 #   - GNU-EFI/Makefile.defaults
 #   - GNU-EFI/apps/Makefile
arm/Makefile.arm64 +1 -0
index 60385e2d2b2b..960880f1c09f 100644
@@ -12,6 +12,7 @@ CFLAGS += -mstrict-align
 
 mno_outline_atomics := $(call cc-option, -mno-outline-atomics, "")
 CFLAGS += $(mno_outline_atomics)
+CFLAGS += -DCONFIG_RELOC
 
 define arch_elf_check =
        $(if $(shell ! $(READELF) -rW $(1) >&/dev/null && echo "nok"),
powerpc/Makefile.common +1 -0
index 8ce00340b6be..c2e976e41f20 100644
@@ -24,6 +24,7 @@ CFLAGS += -ffreestanding
 CFLAGS += -O2 -msoft-float -mno-altivec $(mabi_no_altivec)
 CFLAGS += -I $(SRCDIR)/lib -I $(SRCDIR)/lib/libfdt -I lib
 CFLAGS += -Wa,-mregnames
+CFLAGS += -DCONFIG_RELOC
 
 # We want to keep intermediate files
 .PRECIOUS: %.o

(I also threw in ppc64 since it also relocates.)

Thanks,
drew

author
#7 lore raw

PATCH v2 3/6 arm64: enable frame pointer and support stack unwinding

In reply to Nadav Amit

From: Nadav Amit <[email protected]>

Enable frame pointers for arm64 and perform stack unwinding based on arm64 convention.

---
v1->v2:
* Adding SPDX [checkpatch]
* Moving some unused declarations to next patch [Andrew]
* Adding recursion prevention

---
 arm/Makefile.arm      |  3 ---
 arm/Makefile.arm64    |  1 +
 arm/Makefile.common   |  3 +++
 lib/arm64/asm/stack.h |  3 +++
 lib/arm64/stack.c     | 44 +++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 51 insertions(+), 3 deletions(-)
 create mode 100644 lib/arm64/stack.c
arm/Makefile.arm +0 -3
index 2ce00f5..7fd39f3 100644
@@ -11,9 +11,6 @@ ifeq ($(CONFIG_EFI),y)
 $(error Cannot build arm32 tests as EFI apps)
 endif
 
-# stack.o relies on frame pointers.
-KEEP_FRAME_POINTER := y
-
 CFLAGS += $(machine)
 CFLAGS += -mcpu=$(PROCESSOR)
 CFLAGS += -mno-unaligned-access
arm/Makefile.arm64 +1 -0
index eada7f9..60385e2 100644
@@ -21,6 +21,7 @@ define arch_elf_check =
 endef
 
 cstart.o = $(TEST_DIR)/cstart64.o
+cflatobjs += lib/arm64/stack.o
 cflatobjs += lib/arm64/processor.o
 cflatobjs += lib/arm64/spinlock.o
 cflatobjs += lib/arm64/gic-v3-its.o lib/arm64/gic-v3-its-cmd.o
arm/Makefile.common +3 -0
index 9b45a8f..bc86e44 100644
@@ -22,6 +22,9 @@ $(TEST_DIR)/sieve.elf: AUXFLAGS = 0x1
 ##################################################################
 AUXFLAGS ?= 0x0
 
+# stack.o relies on frame pointers.
+KEEP_FRAME_POINTER := y
+
 CFLAGS += -std=gnu99
 CFLAGS += -ffreestanding
 CFLAGS += -O2
lib/arm64/asm/stack.h +3 -0
index d000624..be486cf 100644
@@ -5,4 +5,7 @@
 #error Do not directly include <asm/stack.h>. Just use <stack.h>.
 #endif
 
+#define HAVE_ARCH_BACKTRACE_FRAME
+#define HAVE_ARCH_BACKTRACE
+
 #endif
lib/arm64/stack.c new file +44 -0

44 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#8 lore raw

PATCH v2 4/6 arm64: stack: update trace stack on exception

In reply to Nadav Amit

From: Nadav Amit <[email protected]>

Using gdb for backtracing or dumping the stack following an exception is not very helpful as the exact location of the exception is not saved.

Add an additional frame to save the location of the exception.

One delicate point is dealing with the pretty_print_stacks script. When the stack is dumped, the script would not print the right address for the exception address: for every return address it deducts "1" before looking for the instruction location in the code (using addr2line). As a somewhat hacky solution add "1" for the exception address when dumping the stack.

---
v1->v2:
* .globl before label [Andrew]
* Some comments [Andrew]

---
 arm/cstart64.S          | 13 +++++++++++++
 lib/arm64/asm-offsets.c |  6 +++++-
 lib/arm64/stack.c       | 18 ++++++++++++++++++
 3 files changed, 36 insertions(+), 1 deletion(-)
arm/cstart64.S +13 -0
index cbd6b51..865a96d 100644
@@ -314,6 +314,13 @@ exceptions_init:
 	mrs	x2, spsr_el1
 	stp	x1, x2, [sp, #S_PC]
 
+	/*
+	 * Save a frame pointer using the link to allow unwinding of
+	 * exceptions.
+	 */
+	stp	x29, x1, [sp, #S_FP]
+	add 	x29, sp, #S_FP
+
 	mov	x0, \vec
 	mov	x1, sp
 	mrs	x2, esr_el1
@@ -349,6 +356,9 @@ exceptions_init:
 	eret
 .endm
 
+.globl vector_stub_start
+vector_stub_start:
+
 vector_stub	el1t_sync,     0
 vector_stub	el1t_irq,      1
 vector_stub	el1t_fiq,      2
@@ -369,6 +379,9 @@ vector_stub	el0_irq_32,   13
 vector_stub	el0_fiq_32,   14
 vector_stub	el0_error_32, 15
 
+.globl vector_stub_end
+vector_stub_end:
+
 .section .text.ex
 
 .macro ventry, label
lib/arm64/asm-offsets.c +5 -1
index 53a1277..80de023 100644
@@ -25,6 +25,10 @@ int main(void)
 	OFFSET(S_PSTATE, pt_regs, pstate);
 	OFFSET(S_ORIG_X0, pt_regs, orig_x0);
 	OFFSET(S_SYSCALLNO, pt_regs, syscallno);
-	DEFINE(S_FRAME_SIZE, sizeof(struct pt_regs));
+
+	/* FP and LR (16 bytes) go on the frame above pt_regs */
+	DEFINE(S_FP, sizeof(struct pt_regs));
+	DEFINE(S_FRAME_SIZE, (sizeof(struct pt_regs) + 16));
+
 	return 0;
 }
lib/arm64/stack.c +18 -0
index a2024e8..82611f4 100644
@@ -6,12 +6,16 @@
 #include <stdbool.h>
 #include <stack.h>
 
+extern char vector_stub_start, vector_stub_end;
+
 int backtrace_frame(const void *frame, const void **return_addrs, int max_depth)
 {
 	const void *fp = frame;
 	static bool walking;
 	void *lr;
 	int depth;
+	bool is_exception = false;
+	unsigned long addr;
 
 	if (walking) {
 		printf("RECURSIVE STACK WALK!!!\n");
@@ -31,6 +35,20 @@ int backtrace_frame(const void *frame, const void **return_addrs, int max_depth)
 				  : );
 
 		return_addrs[depth] = lr;
+
+		/*
+		 * If this is an exception, add 1 to the pointer so when the
+		 * pretty_print_stacks script is run it would get the right
+		 * address (it deducts 1 to find the call address, but we want
+		 * the actual address).
+		 */
+		if (is_exception)
+			return_addrs[depth] += 1;
+
+		/* Check if we are in the exception handlers for the next entry */
+		addr = (unsigned long)lr;
+		is_exception = (addr >= (unsigned long)&vector_stub_start &&
+				addr < (unsigned long)&vector_stub_end);
 	}
 
 	walking = false;
Signature · 1 line
2.34.1
author
#9 lore raw

PATCH v2 5/6 efi: print address of image

In reply to Nadav Amit

From: Nadav Amit <[email protected]>

Using gdb to debug tests that are run from efi is very hard without knowing the base address in which the image was loaded. Print the address so it can be used while debugging.

---
 lib/efi.c | 4 ++++
 1 file changed, 4 insertions(+)
lib/efi.c +4 -0
index 2e127a4..2091771 100644
@@ -19,6 +19,8 @@ extern int __argc, __envc;
 extern char *__argv[100];
 extern char *__environ[200];
 
+extern char _text;
+
 extern int main(int argc, char **argv, char **envp);
 
 efi_system_table_t *efi_system_table = NULL;
@@ -363,6 +365,8 @@ efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
 		goto efi_main_error;
 	}
 
+	printf("Address of image is: 0x%lx\n", (unsigned long)&_text);
+
 	/* Run the test case */
 	ret = main(__argc, __argv, __environ);
 
Signature · 1 line
2.34.1
author
#10 lore raw

PATCH v2 6/6 arm64: dump stack on bad exception

In reply to Nadav Amit

From: Nadav Amit <[email protected]>

Upon a bad exception, the stack is rather useful for debugging, splat it out.

---
 lib/arm64/processor.c | 1 +
 1 file changed, 1 insertion(+)
lib/arm64/processor.c +1 -0
index 831207c..5bcad67 100644
@@ -130,6 +130,7 @@ static void bad_exception(enum vector v, struct pt_regs *regs,
 	printf("Vector: %d (%s)\n", v, vector_names[v]);
 	printf("ESR_EL1: %8s%08x, ec=%#x (%s)\n", "", esr, ec, ec_names[ec]);
 	printf("FAR_EL1: %016lx (%svalid)\n", far, far_valid ? "" : "not ");
+	dump_stack();
 	printf("Exception frame registers:\n");
 	show_regs(regs);
 	abort();
Signature · 1 line
2.34.1