Skip to contentSkip to conversation

libbpf: generate more efficient BPF_CORE_READ code

PATCH Applied Andrii Nakryiko 2 participants 2 messages Oct 11, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libbpf: generate more efficient BPF_CORE_READ code

Existing BPF_CORE_READ() macro generates slightly suboptimal code. If there are intermediate pointers to be read, initial source pointer is going to be assigned into a temporary variable and then temporary variable is going to be uniformly used as a "source" pointer for all intermediate pointer reads. Schematically (ignoring all the type casts), BPF_CORE_READ(s, a, b, c) is expanded into:

({
	const void *__t = src;
	bpf_probe_read(&__t, sizeof(*__t), &__t->a);
	bpf_probe_read(&__t, sizeof(*__t), &__t->b);

	typeof(s->a->b->c) __r;
	bpf_probe_read(&__r, sizeof(*__r), &__t->c);
})

This initial __t = src makes calls more uniform, but causes slightly less optimal register usage sometimes when compiled with Clang. This can cascase into, e.g., more register spills.

This patch fixes this issue by generating more optimal sequence:

({
	const void *__t;
	bpf_probe_read(&__t, sizeof(*__t), &src->a); /* <-- src here */
	bpf_probe_read(&__t, sizeof(*__t), &__t->b);

	typeof(s->a->b->c) __r;
	bpf_probe_read(&__r, sizeof(*__r), &__t->c);
})
Fixes 7db3822ab991 ("libbpf: Add BPF_CORE_READ/BPF_CORE_READ_INTO helpers")
---
 tools/lib/bpf/bpf_core_read.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf_core_read.h +2 -2
index ae877e3ffb51..4daf04c25493 100644
@@ -88,11 +88,11 @@
 	read_fn((void *)(dst), sizeof(*(dst)), &((src_type)(src))->accessor)
 
 /* "recursively" read a sequence of inner pointers using local __t var */
+#define ___rd_first(src, a) ___read(bpf_core_read, &__t, ___type(src), src, a);
 #define ___rd_last(...)							    \
 	___read(bpf_core_read, &__t,					    \
 		___type(___nolast(__VA_ARGS__)), __t, ___last(__VA_ARGS__));
-#define ___rd_p0(src) const void *__t = src;
-#define ___rd_p1(...) ___rd_p0(___nolast(__VA_ARGS__)) ___rd_last(__VA_ARGS__)
+#define ___rd_p1(...) const void *__t; ___rd_first(__VA_ARGS__)
 #define ___rd_p2(...) ___rd_p1(___nolast(__VA_ARGS__)) ___rd_last(__VA_ARGS__)
 #define ___rd_p3(...) ___rd_p2(___nolast(__VA_ARGS__)) ___rd_last(__VA_ARGS__)
 #define ___rd_p4(...) ___rd_p3(___nolast(__VA_ARGS__)) ___rd_last(__VA_ARGS__)
Signature · 1 line
2.17.1
maintainer Applied
#1 lore raw
Quoted 29 lines · Andrii Nakryiko

Existing BPF_CORE_READ() macro generates slightly suboptimal code. If there are intermediate pointers to be read, initial source pointer is going to be assigned into a temporary variable and then temporary variable is going to be uniformly used as a "source" pointer for all intermediate pointer reads. Schematically (ignoring all the type casts), BPF_CORE_READ(s, a, b, c) is expanded into:

({
	const void *__t = src;
	bpf_probe_read(&__t, sizeof(*__t), &__t->a);
	bpf_probe_read(&__t, sizeof(*__t), &__t->b);

	typeof(s->a->b->c) __r;
	bpf_probe_read(&__r, sizeof(*__r), &__t->c);
})

This initial __t = src makes calls more uniform, but causes slightly less optimal register usage sometimes when compiled with Clang. This can cascase into, e.g., more register spills.

This patch fixes this issue by generating more optimal sequence:

({
	const void *__t;
	bpf_probe_read(&__t, sizeof(*__t), &src->a); /* <-- src here */
	bpf_probe_read(&__t, sizeof(*__t), &__t->b);

	typeof(s->a->b->c) __r;
	bpf_probe_read(&__r, sizeof(*__r), &__t->c);
})
Quoted 2 lines
Fixes 7db3822ab991 ("libbpf: Add BPF_CORE_READ/BPF_CORE_READ_INTO helpers")

Applied, thanks!