Skip to contentSkip to conversation

pahole: Fix arm64 argument register allocation

PATCH In review Yonghong Song 3 participants 8 messages Sep 17 – Sep 25 BPF

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v2 cover pahole: Fix arm64 argument register allocation

The arm64 ABI starts an argument twice the register size on an even-numbered argument register, leaving a hole when the next free register is an odd one:

  u64 f(u64 a, __int128 v, u64 b);

passes a in x0, v in x2:x3 -- skipping x1 -- and b in x4. pahole did not model the hole, so b looked like it was in an unexpected register and the whole function was dropped from BTF. Patch 1 skips the register the ABI leaves behind, patch 3 adds a test for it. Only arm64 has this rule; the other architectures are unaffected.

Patch 2 is a small follow-up. Floating point and vector arguments are allocated from a register bank of their own -- v0-v7 on arm64, xmm0-xmm7 on x86-64 -- so a wide one never occupies a general purpose register pair and must not trigger the even-register rule either. It does not change the BTF emitted today, because such an argument still advances the general purpose register index by its own size, so it comes with no test of its own; it keeps the alignment rule scoped to the bank it applies to.

Changelog:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- add commit "Do not align the argument register index for FP arguments".

Yonghong Song (3):
dwarf_loader: Skip the argument register the arm64 ABI leaves as a
hole
dwarf_loader: Do not align the argument register index for FP
arguments
tests: tests: Add test for 16-byte aligned arguments on arm64

 dwarf_loader.c              | 72 ++++++++++++++++++++++++++++++++--
 dwarves.h                   | 11 ++++++
 tests/clang_parm_align16.sh | 77 +++++++++++++++++++++++++++++++++++++
 tests/test_lib.sh           |  3 +-
 4 files changed, 158 insertions(+), 5 deletions(-)
 create mode 100755 tests/clang_parm_align16.sh
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH v2 1/3 dwarf_loader: Skip the argument register the arm64 ABI leaves as a hole

arm64 requires an argument whose alignment is twice the register size to start on an even-numbered argument register, so such an argument arriving when the next free register is an odd one leaves that register unused:

  u64 f_odd(u64 a, __int128 v, u64 b);

passes a in x0, v in x2:x3 -- skipping x1 -- and b in x4. The x86-64 ABI has no such rule and packs v into rsi:rdx instead.

---
 dwarf_loader.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++---
 dwarves.h      |  1 +
 2 files changed, 51 insertions(+), 3 deletions(-)
dwarf_loader.c +50 -3
index 61ef52f..81c2076 100644
@@ -1501,6 +1501,23 @@ static bool arch__agg_use_two_regs(const GElf_Ehdr *ehdr)
 	}
 }
 
+/*
+ * Some ABIs require an argument whose alignment is twice the register size to
+ * start on an even-numbered argument register, leaving a hole when the next
+ * free register is an odd one. For example, on arm64,
+ *	u64 f(u64 a, __int128 v, u64 b)
+ * passes a in x0, v in x2:x3 -- skipping x1 -- and b in x4.
+ */
+static bool arch__arg_align_two_regs(const GElf_Ehdr *ehdr)
+{
+	switch (ehdr->e_machine) {
+	case EM_AARCH64:
+		return true;
+	default:
+		return false;
+	}
+}
+
 static struct template_type_param *template_type_param__new(Dwarf_Die *die, struct cu *cu, struct conf_load *conf)
 {
 	struct template_type_param *ttparm = tag__alloc(cu, sizeof(*ttparm));
@@ -3634,6 +3651,28 @@ static int parameter__abi_slots(const struct parameter *parm, const struct cu *c
 	return slots > 0 ? slots : 1;
 }
 
+static int parameter__abi_reg_align(const struct parameter *parm, const struct cu *cu)
+{
+	struct tag *type;
+
+	if (!cu->arg_align_two_regs || parm->type_byte_size <= cu->addr_size)
+		return 1;
+
+	type = tag__strip_typedefs_and_modifiers(&parm->tag, cu);
+	if (type == NULL)
+		return 1;
+
+	return tag__natural_alignment(type, cu) > cu->addr_size ? 2 : 1;
+}
+
+static int parameter__align_reg_idx(const struct parameter *parm, int reg_idx,
+				    const struct cu *cu)
+{
+	int align = parameter__abi_reg_align(parm, cu);
+
+	return (reg_idx + align - 1) & ~(align - 1);
+}
+
 static bool parameter__has_piece_info(const struct parameter *parm)
 {
 	return parm->first_reg_fields || parm->second_reg_fields;
@@ -3653,7 +3692,7 @@ static bool ftype__next_parameter_preserves_slots(struct ftype *ftype, struct pa
 	if (!next || next->loc_reg == PARAMETER_UNKNOWN_REG)
 		return false;
 
-	next_reg_idx = reg_idx + slots;
+	next_reg_idx = parameter__align_reg_idx(next, reg_idx + slots, cu);
 	return next_reg_idx < cu->nr_register_params &&
 	       next->loc_reg == cu->register_params[next_reg_idx];
 }
@@ -3702,6 +3741,7 @@ static void function__match_clang_parameter_locations(struct ftype *ftype, struc
 		if (pos->passed_in_memory)
 			continue;
 
+		reg_idx = parameter__align_reg_idx(pos, reg_idx, cu);
 		if (reg_idx >= cu->nr_register_params)
 			break;
 
@@ -3732,11 +3772,17 @@ static void function__analyze_parameter_locations(struct function *fn, struct cu
 
 	ftype__for_each_parameter(ftype, pos) {
 		bool consumes_register = true;
-		bool regs_available = reg_idx < cu->nr_register_params;
+		bool regs_available;
 		int slots = parameter__abi_slots(pos, cu);
-		int expected_reg = regs_available ? cu->register_params[reg_idx] : -1;
+		int expected_reg;
 		int reg_slots = pos->passed_in_memory ? 1 : slots;
 
+		if (!pos->passed_in_memory)
+			reg_idx = parameter__align_reg_idx(pos, reg_idx, cu);
+
+		regs_available = reg_idx < cu->nr_register_params;
+		expected_reg = regs_available ? cu->register_params[reg_idx] : -1;
+
 		if (pos->has_loc) {
 			if (true_sig_enabled && pos->loc_const_value) {
 				pos->optimized = 1;
@@ -4467,6 +4513,7 @@ static int cu__set_common(struct cu *cu, struct conf_load *conf,
 	cu->little_endian = ehdr.e_ident[EI_DATA] == ELFDATA2LSB;
 	cu->nr_register_params = arch__nr_register_params(&ehdr);
 	cu->agg_use_two_regs = arch__agg_use_two_regs(&ehdr);
+	cu->arg_align_two_regs = arch__arg_align_two_regs(&ehdr);
 	arch__set_register_params(&ehdr, cu);
 	return 0;
 }
dwarves.h +1 -0
index df77f1e..70adbbf 100644
@@ -304,6 +304,7 @@ struct cu {
 	uint8_t		 little_endian:1;
 	uint8_t		 producer_clang:1;
 	uint8_t		 agg_use_two_regs:1;	/* An aggregate like {long a; long b;} */
+	uint8_t		 arg_align_two_regs:1;	/* An over-aligned arg starts on an even register */
 	uint8_t		 nr_register_params;
 	int		 register_params[ARCH_MAX_REGISTER_PARAMS];
 	int		 functions_saved;
Signature · 1 line
2.53.0-Meta
author reviewer
#2 lore raw

PATCH v2 2/3 dwarf_loader: Do not align the argument register index for FP arguments

In reply to Yonghong Song

The arm64 ABI aligns a general purpose argument that occupies two registers to an even register number, leaving a hole when the next free register is an odd one. parameter__abi_reg_align() applies that rule to every argument wider than a pointer, but floating point and vector arguments are allocated from a register bank of their own -- v0-v7 on arm64, xmm0-xmm7 on x86-64 -- so a wide one never takes a general purpose register pair and must not trigger the rule:

u64 f_fp(u64 a, long double v, u64 b);

Recognize floating point and vector arguments and leave the general purpose register index alone for them.

This does not change the BTF that is emitted today: such an argument still advances the general purpose register index by its own size, so the alignment is not the only thing standing between the model and the ABI. It does keep parameter__abi_reg_align() honest about which bank the rule applies to.

---
 dwarf_loader.c | 21 ++++++++++++++++++++-
 dwarves.h      | 10 ++++++++++
 2 files changed, 30 insertions(+), 1 deletion(-)
dwarf_loader.c +20 -1
index 81c2076..1e5363a 100644
@@ -3651,6 +3651,25 @@ static int parameter__abi_slots(const struct parameter *parm, const struct cu *c
 	return slots > 0 ? slots : 1;
 }
 
+/*
+ * Floating point and vector arguments are allocated from a register bank of
+ * their own -- v0-v7 on arm64, xmm0-xmm7 on x86-64 -- so a wide one does not
+ * take a general purpose register pair and must not trigger the even-register
+ * alignment rule, which applies to the general purpose bank only.
+ */
+static bool tag__uses_gpr_bank(const struct tag *type, const struct cu *cu)
+{
+	/* Scalars represented by FP/SIMD registers have their own allocation bank. */
+	if (tag__is_base_type(type, cu) && base_type__is_float(tag__base_type(type)))
+		return false;
+
+	/* Also exclude vector/SIMD types if dwarves represents them distinctly. */
+	if (tag__is_vector(type))
+		return false;
+
+	return true;
+}
+
 static int parameter__abi_reg_align(const struct parameter *parm, const struct cu *cu)
 {
 	struct tag *type;
@@ -3659,7 +3678,7 @@ static int parameter__abi_reg_align(const struct parameter *parm, const struct c
 		return 1;
 
 	type = tag__strip_typedefs_and_modifiers(&parm->tag, cu);
-	if (type == NULL)
+	if (type == NULL || !tag__uses_gpr_bank(type, cu))
 		return 1;
 
 	return tag__natural_alignment(type, cu) > cu->addr_size ? 2 : 1;
dwarves.h +10 -0
index 70adbbf..f3453ed 100644
@@ -1607,6 +1607,11 @@ static inline uint16_t base_type__size(const struct tag *tag)
 	return tag__base_type(tag)->bit_size / 8;
 }
 
+static inline bool base_type__is_float(const struct base_type *bt)
+{
+	return bt->float_type != 0;
+}
+
 const char *__base_type__name(const struct base_type *bt);
 
 const char *base_type__name(const struct base_type *btype, char *bf, size_t len);
@@ -1625,6 +1630,11 @@ static inline struct array_type *tag__array_type(const struct tag *tag)
 	return (struct array_type *)tag;
 }
 
+static inline bool tag__is_vector(const struct tag *tag)
+{
+	return tag->tag == DW_TAG_array_type && tag__array_type(tag)->is_vector;
+}
+
 struct string_type {
 	struct tag      tag;
 	uint32_t        nr_entries;
Signature · 1 line
2.53.0-Meta
author reviewer
#3 lore raw

PATCH v2 3/3 tests: tests: Add test for 16-byte aligned arguments on arm64

In reply to Yonghong Song

Cover the arm64 even-register rule for 16-byte aligned arguments:
f_even() and f_stack() where the rule does not bite, f_odd() and f_odd_tail() where it leaves a register hole, and f_box() where the alignment comes from a struct member rather than from the parameter type itself.

The test results on arm64:

  $ VERBOSE=1 ./clang_parm_align16.sh
  Validation of BTF encoding of over-aligned arguments.
     BTF: u64 f_even(u64 a, u64 b, __int128 v);
     BTF: u64 f_odd(u64 a, __int128 v, u64 b);
     BTF: u64 f_odd_tail(u64 a, __int128 v, u64 b, u64 c, u64 d);
     BTF: u64 f_stack(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, __int128 v);
     BTF: u64 f_box(u64 a, struct box s, u64 b);
  Test ./clang_parm_align16.sh passed
---
 tests/clang_parm_align16.sh | 77 +++++++++++++++++++++++++++++++++++++
 tests/test_lib.sh           |  3 +-
 2 files changed, 78 insertions(+), 2 deletions(-)
 create mode 100755 tests/clang_parm_align16.sh
tests/clang_parm_align16.sh new file +77 -0
new file mode 100755
index 0000000..484aeb6
@@ -0,0 +1,77 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0-only
+
+source test_lib.sh
+
+outdir=$(make_tmpdir)
+
+# Comment this out to save test data.
+trap cleanup EXIT
+
+title_log "Validation of BTF encoding of over-aligned arguments."
+
+align16="${outdir}/align16"
+CC=$(which clang 2>/dev/null)
+
+if [[ -z "$CC" ]]; then
+	info_log "skip: clang not available"
+	test_skip
+fi
+
+arch=$(uname -m)
+if [[ "$arch" != "aarch64" ]]; then
+	info_log "skip: test is arm64 only, running on $arch"
+	test_skip
+fi
+
+# arm64 makes an argument whose alignment is 16 start on an even-numbered
+# argument register, so f_odd() passes a in x0, v in x2:x3 -- leaving x1 as a
+# hole -- and b in x4.  pahole has to account for that hole, otherwise every
+# parameter after v looks like it is in an unexpected register and the whole
+# function is dropped from BTF.
+cat > ${align16}.c << EOF
+typedef unsigned long long u64;
+struct box { __int128 v; };
+
+__attribute__((noinline)) u64 f_even(u64 a, u64 b, __int128 v)
+{ return a + b + (u64)v; }
+
+__attribute__((noinline)) u64 f_odd(u64 a, __int128 v, u64 b)
+{ return a + b + (u64)v; }
+
+__attribute__((noinline)) u64 f_odd_tail(u64 a, __int128 v, u64 b, u64 c, u64 d)
+{ return a + b + c + d + (u64)v; }
+
+__attribute__((noinline)) u64 f_stack(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f,
+				      u64 g, __int128 v)
+{ return a + b + c + d + e + f + g + (u64)v; }
+
+__attribute__((noinline)) u64 f_box(u64 a, struct box s, u64 b)
+{ return a + b + (u64)s.v; }
+
+u64 (*keep[])() = { (u64(*)())f_even, (u64(*)())f_odd, (u64(*)())f_odd_tail,
+		    (u64(*)())f_stack, (u64(*)())f_box };
+EOF
+
+${CC} -g -O2 -c -o ${align16}.o ${align16}.c 2>/dev/null
+if [[ $? -ne 0 ]]; then
+	info_log "skip: clang could not compile ${align16}.c"
+	test_skip
+fi
+
+LLVM_OBJCOPY=objcopy pahole -J --btf_features=consistent_func ${align16}.o
+if [[ $? -ne 0 ]]; then
+	error_log "Could not encode BTF for ${align16}.o"
+	test_fail
+fi
+
+for fn in f_even f_odd f_odd_tail f_stack f_box; do
+	encoded=$(pfunct --all --format_path=btf ${align16}.o | grep " ${fn}(")
+	verbose_log "BTF: $encoded"
+	if [[ -z "$encoded" ]]; then
+		error_log "${fn}() is missing from BTF"
+		test_fail
+	fi
+done
+
+test_pass
tests/test_lib.sh +1 -2
index e2a7218..e2d0b0c 100755
@@ -661,8 +661,7 @@ check_bpftool_btf_support()
 cleanup()
 {
 	if [ -n "$outdir" ] && [ -d "$outdir" ]; then
-		rm ${outdir}/*
-		rmdir $outdir
+		rm -rf "$outdir"
 	fi
 	return 0
 }
Signature · 1 line
2.53.0-Meta
#4 lore raw
In reply to Yonghong Song
Quoted 12 lines · Yonghong Song

The arm64 ABI starts an argument twice the register size on an even-numbered argument register, leaving a hole when the next free register is an odd one:

  u64 f(u64 a, __int128 v, u64 b);

passes a in x0, v in x2:x3 -- skipping x1 -- and b in x4. pahole did not model the hole, so b looked like it was in an unexpected register and the whole function was dropped from BTF. Patch 1 skips the register the ABI leaves behind, patch 3 adds a test for it. Only arm64 has this rule; the other architectures are unaffected.

Quoted 8 lines

Patch 2 is a small follow-up. Floating point and vector arguments are allocated from a register bank of their own -- v0-v7 on arm64, xmm0-xmm7 on x86-64 -- so a wide one never occupies a general purpose register pair and must not trigger the even-register rule either. It does not change the BTF emitted today, because such an argument still advances the general purpose register index by its own size, so it comes with no test of its own; it keeps the alignment rule scoped to the bank it applies to.

one small issue here; we record alignment for structs from DW_AT_align in associated tags; we should ensure that we utilize that alignment where available. However it's a bit broken in that we don't appear to record alignment for typedefs, just structs.

I think we can tackle that with a followup though; unless anyone objects I'll land these patches later today.

Quoted 22 lines

Changelog:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- add commit "Do not align the argument register index for FP arguments".

Yonghong Song (3):
dwarf_loader: Skip the argument register the arm64 ABI leaves as a
hole
dwarf_loader: Do not align the argument register index for FP
arguments
tests: tests: Add test for 16-byte aligned arguments on arm64

 dwarf_loader.c              | 72 ++++++++++++++++++++++++++++++++--
 dwarves.h                   | 11 ++++++
 tests/clang_parm_align16.sh | 77 +++++++++++++++++++++++++++++++++++++
 tests/test_lib.sh           |  3 +-
 4 files changed, 158 insertions(+), 5 deletions(-)
 create mode 100755 tests/clang_parm_align16.sh
Signature · 1 line
2.53.0-Meta
#5 lore raw
Quoted 22 lines · Alan Maguire
Quoted 12 lines · Yonghong Song

The arm64 ABI starts an argument twice the register size on an even-numbered argument register, leaving a hole when the next free register is an odd one:

  u64 f(u64 a, __int128 v, u64 b);

passes a in x0, v in x2:x3 -- skipping x1 -- and b in x4. pahole did not model the hole, so b looked like it was in an unexpected register and the whole function was dropped from BTF. Patch 1 skips the register the ABI leaves behind, patch 3 adds a test for it. Only arm64 has this rule; the other architectures are unaffected.

Quoted 8 lines

Patch 2 is a small follow-up. Floating point and vector arguments are allocated from a register bank of their own -- v0-v7 on arm64, xmm0-xmm7 on x86-64 -- so a wide one never occupies a general purpose register pair and must not trigger the even-register rule either. It does not change the BTF emitted today, because such an argument still advances the general purpose register index by its own size, so it comes with no test of its own; it keeps the alignment rule scoped to the bank it applies to.

Quoted 7 lines

one small issue here; we record alignment for structs from DW_AT_align in associated tags; we should ensure that we utilize that alignment where available. However it's a bit broken in that we don't appear to record alignment for typedefs, just structs.

I think we can tackle that with a followup though; unless anyone objects I'll land these patches later today.

Pls go ahead.
Also pls land LOC* things too, so you can post the rest of LOC* kernel bits. So that everything lands and ready for the merge window in a couple weeks.

People will be traveling soon, so the sooner the better.

#6 lore raw
Quoted 29 lines · Alexei Starovoitov
Quoted 27 lines · Alan Maguire
Quoted 12 lines · Yonghong Song

The arm64 ABI starts an argument twice the register size on an even-numbered argument register, leaving a hole when the next free register is an odd one:

  u64 f(u64 a, __int128 v, u64 b);

passes a in x0, v in x2:x3 -- skipping x1 -- and b in x4. pahole did not model the hole, so b looked like it was in an unexpected register and the whole function was dropped from BTF. Patch 1 skips the register the ABI leaves behind, patch 3 adds a test for it. Only arm64 has this rule; the other architectures are unaffected.

Quoted 8 lines

Patch 2 is a small follow-up. Floating point and vector arguments are allocated from a register bank of their own -- v0-v7 on arm64, xmm0-xmm7 on x86-64 -- so a wide one never occupies a general purpose register pair and must not trigger the even-register rule either. It does not change the BTF emitted today, because such an argument still advances the general purpose register index by its own size, so it comes with no test of its own; it keeps the alignment rule scoped to the bank it applies to.

one small issue here; we record alignment for structs from DW_AT_align in associated tags; we should ensure that we utilize that alignment where available. However it's a bit broken in that we don't appear to record alignment for typedefs, just structs.

Quoted 9 lines
Quoted 2 lines

I think we can tackle that with a followup though; unless anyone objects I'll land these patches later today.

Pls go ahead.
Also pls land LOC* things too, so you can post the rest of LOC* kernel bits. So that everything lands and ready for the merge window in a couple weeks.

People will be traveling soon, so the sooner the better.

Sure; this series is applied; location patches will go out shortly. We will need a libbpf sync commit with latest libbpf for pahole too; Andrii would you mind doing that when you get a chance? (The pahole stuff will still build without latest libbpf but the inline feature will not work without the APIs to add location data).

#7 lore raw
In reply to Alan Maguire
Quoted 22 lines · Alan Maguire
Quoted 12 lines · Yonghong Song

The arm64 ABI starts an argument twice the register size on an even-numbered argument register, leaving a hole when the next free register is an odd one:

  u64 f(u64 a, __int128 v, u64 b);

passes a in x0, v in x2:x3 -- skipping x1 -- and b in x4. pahole did not model the hole, so b looked like it was in an unexpected register and the whole function was dropped from BTF. Patch 1 skips the register the ABI leaves behind, patch 3 adds a test for it. Only arm64 has this rule; the other architectures are unaffected.

Quoted 8 lines

Patch 2 is a small follow-up. Floating point and vector arguments are allocated from a register bank of their own -- v0-v7 on arm64, xmm0-xmm7 on x86-64 -- so a wide one never occupies a general purpose register pair and must not trigger the even-register rule either. It does not change the BTF emitted today, because such an argument still advances the general purpose register index by its own size, so it comes with no test of its own; it keeps the alignment rule scoped to the bank it applies to.

Quoted 8 lines

one small issue here; we record alignment for structs from DW_AT_align in associated tags; we should ensure that we utilize that alignment where available. However it's a bit broken in that we don't appear to record alignment for typedefs, just structs.

I think we can tackle that with a followup though; unless anyone objects I'll land these patches later today.

Something along the lines of [1] would make sense I think; i.e. factoring in DW_AT_alignment into alignment calculation where available. Not a big rush on this though as it's a less likely scenario in the argument-passing case.

[1] https://github.com/acmel/dwarves/commit/e6bad3dfd334438d6d835084b0e832c95b6d1302

Quoted 25 lines
Quoted 23 lines

Changelog:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- add commit "Do not align the argument register index for FP arguments".

Yonghong Song (3):
dwarf_loader: Skip the argument register the arm64 ABI leaves as a
hole
dwarf_loader: Do not align the argument register index for FP
arguments
tests: tests: Add test for 16-byte aligned arguments on arm64

 dwarf_loader.c              | 72 ++++++++++++++++++++++++++++++++--
 dwarves.h                   | 11 ++++++
 tests/clang_parm_align16.sh | 77 +++++++++++++++++++++++++++++++++++++
 tests/test_lib.sh           |  3 +-
 4 files changed, 158 insertions(+), 5 deletions(-)
 create mode 100755 tests/clang_parm_align16.sh
Signature · 1 line
2.53.0-Meta