Skip to contentSkip to conversation

kbuild: bpf: Do not run pahole with -j on 32bit userspace

RFC In review Jiri Slaby (SUSE) 7 participants 19 messages Aug 20, 2024 – Sep 25, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 kbuild: bpf: Do not run pahole with -j on 32bit userspace

From: Jiri Slaby <[email protected]>

== WARNING ==
This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM are completely unhandled.

The simple version is just do there:
ifeq ($(CONFIG_64BIT,y)
but it has its own deficiencies, of course.

So any ideas, inputs?
== WARNING ==

When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:

Quoted 2 lines

btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
btf_encoder__encode: failed to tag kfuncs!

or simply SIGSEGV (failed to allocate the btf encoder).

It very depends on how many threads are created.

So do not invoke pahole with -j on 32bit.

Fixes b4f72786429c ("scripts/pahole-flags.sh: Parse DWARF and generate BTF with multithreading.")
Cc Masahiro Yamada <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Nicolas Schier <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Eduard Zingerman <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Cc John Fastabend <[email protected]>
Cc KP Singh <[email protected]>
Cc Stanislav Fomichev <[email protected]>
Cc Hao Luo <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 init/Kconfig            |  4 ++++
 scripts/Makefile.btf    |  2 ++
 scripts/pahole-class.sh | 21 +++++++++++++++++++++
 3 files changed, 27 insertions(+)
 create mode 100644 scripts/pahole-class.sh
init/Kconfig +4 -0
index f36ca8a0e209..f5e80497eef0 100644
@@ -113,6 +113,10 @@ config PAHOLE_VERSION
 	int
 	default $(shell,$(srctree)/scripts/pahole-version.sh $(PAHOLE))
 
+config PAHOLE_CLASS
+	string
+	default $(shell,$(srctree)/scripts/pahole-class.sh $(PAHOLE))
+
 config CONSTRUCTORS
 	bool
 
scripts/Makefile.btf +2 -0
index b75f09f3f424..f7de8e922bce 100644
@@ -12,7 +12,9 @@ endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
 
+ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
 pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
+endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
scripts/pahole-class.sh new file +21 -0
new file mode 100644
index 000000000000..d15a92077f76
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+# Usage: $ ./pahole-class.sh pahole
+#
+# Prints pahole's ELF class, such as ELF64
+
+if [ ! -x "$(command -v "$@")" ]; then
+	echo 0
+	exit 1
+fi
+
+PAHOLE="$(which "$@")"
+CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
+
+# Scripts like scripts/dummy-tools/pahole
+if [ -n "$CLASS" ]; then
+	echo "$CLASS"
+else
+	echo ELF64
+fi
Signature · 1 line
2.46.0
author
#1 lore raw
Quoted 11 lines · Jiri Slaby (SUSE)

From: Jiri Slaby <[email protected]>

== WARNING ==
This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM are completely unhandled.

The simple version is just do there:

   ifeq ($(CONFIG_64BIT,y)

but it has its own deficiencies, of course.

So any ideas, inputs?

Also as Shung-Hsi Yu suggests, we can cap -j to 1 in pahole proper when sizeof(long) == 4.

Quoted 10 lines

== WARNING ==

When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:

Quoted 2 lines

btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
btf_encoder__encode: failed to tag kfuncs!

or simply SIGSEGV (failed to allocate the btf encoder).

It very depends on how many threads are created.

I forgot to add, that it depends on the kernel version too. It happens much often with 6.11-rc now (vmlinux got big enough, apparently).

thanks,

Signature · 2 lines
js
suse labs
#2 lore raw
In reply to Jiri Slaby (SUSE)
Quoted 15 lines · Jiri Slaby (SUSE)

From: Jiri Slaby <[email protected]>

== WARNING ==
This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM are completely unhandled.

The simple version is just do there:
ifeq ($(CONFIG_64BIT,y)
but it has its own deficiencies, of course.

So any ideas, inputs?
== WARNING ==

When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:

Quoted 8 lines
Quoted 2 lines

btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
btf_encoder__encode: failed to tag kfuncs!

or simply SIGSEGV (failed to allocate the btf encoder).

It very depends on how many threads are created.

So do not invoke pahole with -j on 32bit.

could you share more details about your setup?

does it need to run on pure 32bit to reproduce? I can't reproduce when doing cross build and running 32 bit pahole on x86_64.. I do see some errors though

[667939] STRUCT bpf_prog_aux Error emitting BTF type Encountered error while encoding BTF.

thanks,
jirka

Quoted 90 lines
Fixes b4f72786429c ("scripts/pahole-flags.sh: Parse DWARF and generate BTF with multithreading.")
Cc Masahiro Yamada <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Nicolas Schier <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Eduard Zingerman <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Cc John Fastabend <[email protected]>
Cc KP Singh <[email protected]>
Cc Stanislav Fomichev <[email protected]>
Cc Hao Luo <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 init/Kconfig            |  4 ++++
 scripts/Makefile.btf    |  2 ++
 scripts/pahole-class.sh | 21 +++++++++++++++++++++
 3 files changed, 27 insertions(+)
 create mode 100644 scripts/pahole-class.sh
init/Kconfig +4 -0
index f36ca8a0e209..f5e80497eef0 100644
@@ -113,6 +113,10 @@ config PAHOLE_VERSION
 	int
 	default $(shell,$(srctree)/scripts/pahole-version.sh $(PAHOLE))
 
+config PAHOLE_CLASS
+	string
+	default $(shell,$(srctree)/scripts/pahole-class.sh $(PAHOLE))
+
 config CONSTRUCTORS
 	bool
 
scripts/Makefile.btf +2 -0
index b75f09f3f424..f7de8e922bce 100644
@@ -12,7 +12,9 @@ endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
 
+ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
 pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
+endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
scripts/pahole-class.sh new file +21 -0
new file mode 100644
index 000000000000..d15a92077f76
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+# Usage: $ ./pahole-class.sh pahole
+#
+# Prints pahole's ELF class, such as ELF64
+
+if [ ! -x "$(command -v "$@")" ]; then
+	echo 0
+	exit 1
+fi
+
+PAHOLE="$(which "$@")"
+CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
+
+# Scripts like scripts/dummy-tools/pahole
+if [ -n "$CLASS" ]; then
+	echo "$CLASS"
+else
+	echo ELF64
+fi
Signature · 1 line
2.46.0
author
#3 lore raw
Quoted 21 lines · Jiri Olsa
Quoted 20 lines · Jiri Slaby (SUSE)

From: Jiri Slaby <[email protected]>

== WARNING ==
This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM are completely unhandled.

The simple version is just do there:

   ifeq ($(CONFIG_64BIT,y)

but it has its own deficiencies, of course.

So any ideas, inputs?
== WARNING ==

When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:

Quoted 2 lines

btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
btf_encoder__encode: failed to tag kfuncs!

or simply SIGSEGV (failed to allocate the btf encoder).

Quoted 7 lines
Quoted 3 lines

It very depends on how many threads are created.

So do not invoke pahole with -j on 32bit.

could you share more details about your setup?

does it need to run on pure 32bit to reproduce?

armv7l builds are 32bit only.

Quoted 2 lines

I can't reproduce when
doing cross build and running 32 bit pahole on x86_64..

i586 is built using 64bit kernel. It is enough to have 32bit userspace. As written in the linked bug:
https://bugzilla.suse.com/show_bug.cgi?id=1229450#c6

FWIW, steps to reproduce locally:
docker pull jirislaby/pahole_crash
docker run -it jirislaby/pahole_crash

The VM space of pahole is exhausted:
process map: https://bugzilla.suse.com/attachment.cgi?id=876821
strace of mmaps: https://bugzilla.suse.com/attachment.cgi?id=876822

You need to run with large enough -j on a fast machine. Note that this happens on build hosts even with -j4, but they are under heavy load, so parallelism of held memory is high.

On my box with 16 cores, it is (likely far) enough to run with -j32.

Quoted 5 lines

I do see some
errors though

[667939] STRUCT bpf_prog_aux Error emitting BTF type Encountered error while encoding BTF.

It's possible that it is one of the errors. There are different ones. As I wrote above, sometimes it is a crash, sometimes it is the failure I mentioned above. But it always ends up with a failed build:

Quoted 7 lines

libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available
make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
make[2]: *** Deleting file 'vmlinux'
make[1]: *** [/home/abuild/rpmbuild/BUILD/kernel-vanilla-6.11~rc3.338.gc3f2d783a459/linux-6.11-rc3-338-gc3f2d783a459/Makefile:1158: vmlinux] Error 2
make: *** [../Makefile:224: __sub-make] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.olf5Nu (%build)

thanks,

Signature · 2 lines
js
suse labs
author
#4 lore raw
Quoted 43 lines · Jiri Slaby
Quoted 21 lines · Jiri Olsa
Quoted 20 lines · Jiri Slaby (SUSE)

From: Jiri Slaby <[email protected]>

== WARNING ==
This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM are completely unhandled.

The simple version is just do there:
   ifeq ($(CONFIG_64BIT,y)
but it has its own deficiencies, of course.

So any ideas, inputs?
== WARNING ==

When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:

Quoted 2 lines

btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
btf_encoder__encode: failed to tag kfuncs!

or simply SIGSEGV (failed to allocate the btf encoder).

Quoted 7 lines
Quoted 3 lines

It very depends on how many threads are created.

So do not invoke pahole with -j on 32bit.

could you share more details about your setup?

does it need to run on pure 32bit to reproduce?

armv7l builds are 32bit only.

Quoted 2 lines

I can't reproduce when
doing cross build and running 32 bit pahole on x86_64..

i586 is built using 64bit kernel. It is enough to have 32bit userspace. As written in the linked bug:
https://bugzilla.suse.com/show_bug.cgi?id=1229450#c6

FWIW, steps to reproduce locally:
docker pull jirislaby/pahole_crash
docker run -it jirislaby/pahole_crash

Quoted 5 lines

The VM space of pahole is exhausted:
process map: https://bugzilla.suse.com/attachment.cgi?id=876821
strace of mmaps: https://bugzilla.suse.com/attachment.cgi?id=876822

You need to run with large enough -j on a fast machine. Note that this happens on build hosts even with -j4, but they are under heavy load, so parallelism of held memory is high.

From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB

On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GB

Look there for full massif report.

So now I think we should disable BTF generation with 32bit pahole completely. Or someone debugs it and improves debug info loading to not eat that much.

thanks,

Signature · 2 lines
js
suse labs
author
#5 lore raw
Quoted 6 lines · Jiri Slaby

From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB

Quoted 6 lines

On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GB

Look there for full massif report.

From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
(In reply to Jiri Slaby from comment #20)

Quoted 1 line

| | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun (dwarf_loader.c:959)

So given this struct class_member is the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.

dwarves.h +3 -3
@@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,
  */
 struct tag {
        struct list_head node;
+       const char       *attribute;
+       void             *priv;
        type_id_t        type;
        uint16_t         tag;
+       uint16_t         recursivity_level;
        bool             visited;
        bool             top_level;
        bool             has_btf_type_tag;
-       uint16_t         recursivity_level;
-       const char       *attribute;
-       void             *priv;
 };
 
 // To use with things like type->type_enum == 

perf_event_type+perf_user_event_type
@@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct function *func)

 struct class_member {
        struct tag       tag;
        const char       *name;
+       uint64_t         const_value;
        uint32_t         bit_offset;
        uint32_t         bit_size;
        uint32_t         byte_offset;
        int              hole;
        size_t           byte_size;
+       uint32_t         alignment;
        int8_t           bitfield_offset;
        uint8_t          bitfield_size;
        uint8_t          bit_hole;
        uint8_t          bitfield_end:1;
-       uint64_t         const_value;
-       uint32_t         alignment;
        uint8_t          visited:1;
        uint8_t          is_static:1;
        uint8_t          has_bit_offset:1;
Signature · 2 lines
js
suse labs
#6 lore raw

(Add pahole maintainer and mailing list)

Hi Arnaldo,

We're running into kernel build failure on 32-bit (both full 32-bit and 32-bit userspace on 64-bit kernel) because pahole crashed due to virtual memory exhaustion[1]. As a workaround we currently limit pahole's parallel job count to 1 on such system[2]:

On Tue, 20 Aug 2024 10:59:50AM +0200, Jiri Slaby wrote:
[...]

Quoted 32 lines
scripts/Makefile.btf +2 -0
index b75f09f3f424..f7de8e922bce 100644
@@ -12,7 +12,9 @@ endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
 
+ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
 pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
+endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
scripts/pahole-class.sh new file +11 -0
new file mode 100644
index 000000000000..d15a92077f76
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+# Usage: $ ./pahole-class.sh pahole
+#
+# Prints pahole's ELF class, such as ELF64
+
+if [ ! -x "$(command -v "$@")" ]; then
+	echo 0
+	exit 1
+fi

scripts/pahole-class.sh line 21

Quoted 10 lines
scripts/pahole-class.sh new file +10 -0
+
+PAHOLE="$(which "$@")"
+CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
+
+# Scripts like scripts/dummy-tools/pahole
+if [ -n "$CLASS" ]; then
+	echo "$CLASS"
+else
+	echo ELF64
+fi

This helped lowered the memory usage enough so pahole no longer crash:

Quoted 11 lines · Jiri Slaby
Quoted 10 lines · Jiri Slaby

From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB

On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GB

Jiri (Slaby) in the meanwhile has also proposed structure packing to further reduce memory usage. (Note: I think the numbers below are from a 64-bit machine)

Quoted 50 lines

From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
(In reply to Jiri Slaby from comment #20)

Quoted 1 line

| | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun

(dwarf_loader.c:959)

So given this struct class_member is the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.

dwarves.h +3 -3
@@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,
  */
 struct tag {
        struct list_head node;
+       const char       *attribute;
+       void             *priv;
        type_id_t        type;
        uint16_t         tag;
+       uint16_t         recursivity_level;
        bool             visited;
        bool             top_level;
        bool             has_btf_type_tag;
-       uint16_t         recursivity_level;
-       const char       *attribute;
-       void             *priv;
 };
 
 // To use with things like type->type_enum ==

perf_event_type+perf_user_event_type

dwarves.h +2 -2
@@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct
 function *func)
 struct class_member {
        struct tag       tag;
        const char       *name;
+       uint64_t         const_value;
        uint32_t         bit_offset;
        uint32_t         bit_size;
        uint32_t         byte_offset;
        int              hole;
        size_t           byte_size;
+       uint32_t         alignment;
        int8_t           bitfield_offset;
        uint8_t          bitfield_size;
        uint8_t          bit_hole;
        uint8_t          bitfield_end:1;
-       uint64_t         const_value;
-       uint32_t         alignment;
        uint8_t          visited:1;
        uint8_t          is_static:1;
        uint8_t          has_bit_offset:1;
Quoted 1 line

--

What do you think?

IIUC pahole's memory usage is largely tied to the number of entries in vmlinux/kmodule DWARF, and there probably isn't much we could do about that.

Shung-Hsi

1: https://bugzilla.suse.com/show_bug.cgi?id=1229450
2: https://lore.kernel.org/all/[email protected]/

#7 lore raw
Quoted 64 lines · Shung-Hsi Yu

(Add pahole maintainer and mailing list)

Hi Arnaldo,

We're running into kernel build failure on 32-bit (both full 32-bit and 32-bit userspace on 64-bit kernel) because pahole crashed due to virtual memory exhaustion[1]. As a workaround we currently limit pahole's parallel job count to 1 on such system[2]:

On Tue, 20 Aug 2024 10:59:50AM +0200, Jiri Slaby wrote:
[...]

Quoted 32 lines
scripts/Makefile.btf +2 -0
index b75f09f3f424..f7de8e922bce 100644
@@ -12,7 +12,9 @@ endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
 
+ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
 pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
+endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
scripts/pahole-class.sh new file +11 -0
new file mode 100644
index 000000000000..d15a92077f76
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+# Usage: $ ./pahole-class.sh pahole
+#
+# Prints pahole's ELF class, such as ELF64
+
+if [ ! -x "$(command -v "$@")" ]; then
+	echo 0
+	exit 1
+fi

scripts/pahole-class.sh line 21

Quoted 10 lines
scripts/pahole-class.sh new file +10 -0
+
+PAHOLE="$(which "$@")"
+CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
+
+# Scripts like scripts/dummy-tools/pahole
+if [ -n "$CLASS" ]; then
+	echo "$CLASS"
+else
+	echo ELF64
+fi

This helped lowered the memory usage enough so pahole no longer crash:

Quoted 7 lines · Jiri Slaby
Quoted 6 lines · Jiri Slaby

From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GB

Jiri (Slaby) in the meanwhile has also proposed structure packing to further reduce memory usage. (Note: I think the numbers below are from a 64-bit machine)

That is interesting, packing pahole data structures ;-) :-)

Also a coincidence is that I did some packing on what is in the next branch:

3ef508ad94012933 dwarf_loader: Make 'struct dwarf_tag' more compact by getting rid of 'struct dwarf_off_ref 70febc8858588348 core: Reorganize 'struct class_member' to save 8 bytes
76bcb88a67556468 core: Make tag->recursivity_level a uint8_t
b8b9e04d177d8eb7 core: Make tag->top_level a single bit flag
539acefcdd5b0f71 core: Make tag->has_btf_type_tag a single bit flag
dba2c2c1aa5dfa05 core: Make tag->visited a single bit flag
7409cfadcae0253b core: Shrink 'struct namespace' a bit by using a hole in its embedded 'tag'

Also I did more work to reduce the number of allocations:

cbecc3785266f0c5 dwarf_loader: Do just one alloc for 'struct dwarf_tag + struct tag'

With it we get:

⬢[acme@toolbox pahole]$ pahole -C class_member build/libdwarves.so.1.0.0 struct class_member {
struct tag tag; /* 0 32 */

/* XXX last struct has 1 bit hole */

const char * name; /* 32 8 */
uint32_t bit_offset; /* 40 4 */
uint32_t bit_size; /* 44 4 */
uint32_t byte_offset; /* 48 4 */
int hole; /* 52 4 */
size_t byte_size; /* 56 8 */
/* --- cacheline 1 boundary (64 bytes) --- */
int8_t bitfield_offset; /* 64 1 */
uint8_t bitfield_size; /* 65 1 */
uint8_t bit_hole; /* 66 1 */
uint8_t bitfield_end:1; /* 67: 0 1 */
uint8_t visited:1; /* 67: 1 1 */
uint8_t is_static:1; /* 67: 2 1 */
uint8_t has_bit_offset:1; /* 67: 3 1 */
uint8_t accessibility:2; /* 67: 4 1 */
uint8_t virtuality:2; /* 67: 6 1 */
uint32_t alignment; /* 68 4 */
uint64_t const_value; /* 72 8 */

/* size: 80, cachelines: 2, members: 18 */ /* member types with bit holes: 1, total: 1 */ /* last cacheline: 16 bytes */
};

⬢[acme@toolbox pahole]$

And also the dwarf_tag, that is allocated for each tag coming from DWARF got smaller:

⬢[acme@toolbox pahole]$ pahole -C dwarf_tag build/libdwarves.so.1.0.0 struct dwarf_tag {

	struct hlist_node          hash_node;            /*     0    16 */
	Dwarf_Off                  type;                 /*    16     8 */
	Dwarf_Off                  id;                   /*    24     8 */

union {
Dwarf_Off abstract_origin; /* 32 8 */
Dwarf_Off containing_type; /* 32 8 */
}; /* 32 8 */
Dwarf_Off specification; /* 40 8 */
struct {
_Bool type:1; /* 48: 0 1 */
_Bool abstract_origin:1; /* 48: 1 1 */
_Bool containing_type:1; /* 48: 2 1 */
_Bool specification:1; /* 48: 3 1 */
} from_types_section; /* 48 1 */

/* XXX last struct has 4 bits of padding */ /* XXX 1 byte hole, try to pack */

uint16_t decl_line; /* 50 2 */
uint32_t small_id; /* 52 4 */
const char * decl_file; /* 56 8 */

/* size: 64, cachelines: 1, members: 9 */
/* sum members: 63, holes: 1, sum holes: 1 */ /* member types with bit paddings: 1, total: 4 bits */
};

⬢[acme@toolbox pahole]$

I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

Thanks!

- Arnaldo

Quoted 62 lines
Quoted 50 lines

From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
(In reply to Jiri Slaby from comment #20)

Quoted 1 line

| | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun

(dwarf_loader.c:959)

So given this struct class_member is the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.

dwarves.h +3 -3
@@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,
  */
 struct tag {
        struct list_head node;
+       const char       *attribute;
+       void             *priv;
        type_id_t        type;
        uint16_t         tag;
+       uint16_t         recursivity_level;
        bool             visited;
        bool             top_level;
        bool             has_btf_type_tag;
-       uint16_t         recursivity_level;
-       const char       *attribute;
-       void             *priv;
 };
 
 // To use with things like type->type_enum ==

perf_event_type+perf_user_event_type

dwarves.h +2 -2
@@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct
 function *func)
 struct class_member {
        struct tag       tag;
        const char       *name;
+       uint64_t         const_value;
        uint32_t         bit_offset;
        uint32_t         bit_size;
        uint32_t         byte_offset;
        int              hole;
        size_t           byte_size;
+       uint32_t         alignment;
        int8_t           bitfield_offset;
        uint8_t          bitfield_size;
        uint8_t          bit_hole;
        uint8_t          bitfield_end:1;
-       uint64_t         const_value;
-       uint32_t         alignment;
        uint8_t          visited:1;
        uint8_t          is_static:1;
        uint8_t          has_bit_offset:1;
Quoted 1 line

--

What do you think?

IIUC pahole's memory usage is largely tied to the number of entries in vmlinux/kmodule DWARF, and there probably isn't much we could do about that.

Shung-Hsi

1: https://bugzilla.suse.com/show_bug.cgi?id=1229450
2: https://lore.kernel.org/all/[email protected]/

author
#8 lore raw
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

thanks,

Signature · 2 lines
js
suse labs
#9 lore raw
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory, not address space.

Quoted 20 lines
Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

Quoted 3 lines

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:

⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs

Excerpt of the above:

This leads to deleting ~90 CUs during parallel vmlinux BTF generation versus deleting just 1 prior to this change.

c7ec9200caa7d485 btf_encoder: Add "distilled_base" BTF feature to split BTF generation bc4e6a9adfc72758 pahole: Sync with libbpf-1.5
5e3ed3ec2947c69f pahole: Do --lang_exclude CU filtering earlier c46455bb0379fa38 dwarf_loader: Allow filtering CUs early in loading ⬢[acme@toolbox pahole]$

- Arnaldo

#10 lore raw
Quoted 38 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory, not address space.

Quoted 20 lines
Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

Quoted 3 lines

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Quoted 6 lines

Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:

⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs

*R*edzce? memory usage ...

-Sedat-

Quoted 13 lines

Excerpt of the above:

This leads to deleting ~90 CUs during parallel vmlinux BTF generation versus deleting just 1 prior to this change.

c7ec9200caa7d485 btf_encoder: Add "distilled_base" BTF feature to split BTF generation bc4e6a9adfc72758 pahole: Sync with libbpf-1.5
5e3ed3ec2947c69f pahole: Do --lang_exclude CU filtering earlier c46455bb0379fa38 dwarf_loader: Allow filtering CUs early in loading ⬢[acme@toolbox pahole]$

- Arnaldo

#11 lore raw
Quoted 40 lines · On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
Quoted 38 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory, not address space.

Quoted 23 lines
Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Quoted 9 lines
Quoted 6 lines

Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:

⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs

*R*edzce? memory usage ...

If you meant that further typo it's golden, and if not the irony is rich :)

Either way this is my favorite email of the day!

Cheers,
Phil

Quoted 16 lines

-Sedat-

Quoted 13 lines

Excerpt of the above:

This leads to deleting ~90 CUs during parallel vmlinux BTF generation versus deleting just 1 prior to this change.

c7ec9200caa7d485 btf_encoder: Add "distilled_base" BTF feature to split BTF generation bc4e6a9adfc72758 pahole: Sync with libbpf-1.5
5e3ed3ec2947c69f pahole: Do --lang_exclude CU filtering earlier c46455bb0379fa38 dwarf_loader: Allow filtering CUs early in loading ⬢[acme@toolbox pahole]$

- Arnaldo

#12 lore raw
Quoted 48 lines · Phil Auld
Quoted 47 lines · On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
Quoted 44 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory, not address space.

Quoted 20 lines
Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

Quoted 3 lines

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:

⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs

Quoted 6 lines
Quoted 2 lines

*R*edzce? memory usage ...

If you meant that further typo it's golden, and if not the irony is rich :)

Either way this is my favorite email of the day!

Hahaha, I went to uppercase what comes after the colon and introduced that typo ;-)

Faxing it....

- Arnaldo

#13 lore raw
Quoted 53 lines · Arnaldo Carvalho de Melo
Quoted 51 lines · Phil Auld
Quoted 49 lines · On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
Quoted 38 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory, not address space.

Quoted 20 lines
Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

Quoted 3 lines

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Quoted 6 lines

Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:

⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs

*R*edzce? memory usage ...

Quoted 8 lines
Quoted 3 lines

If you meant that further typo it's golden, and if not the irony is rich :)

Either way this is my favorite email of the day!

Hahaha, I went to uppercase what comes after the colon and introduced that typo ;-)

Faxing it....

typo-fest continues ;)

but it's great to see that a new pahole release is coming (cc'ing Usama), we are eagerly waiting for one of the bug fixes that will go into 1.28 (one of the first commits after 1.27 was cut). Note, libbpf CI's for pahole-staging is failing, but that seems a one-off test and has nothing to do with pahole being broken.

Quoted 3 lines

- Arnaldo

author
#14 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 33 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory, not address space.

Quoted 20 lines
Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

Quoted 6 lines
Quoted 3 lines

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:

Not much helping.

On my box (as all previous runs):
next-j1 2.242
next-j16 2.808
next-j32 2.646

On a build host:
next-j1: 2.242
next-j16: 2.824
next-j20: 2.902 (crash)
next-j32: 2.902 (crash)

Signature · 2 lines
js
suse labs
author
#15 lore raw
Quoted 46 lines · Jiri Slaby
Quoted 31 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory, not address space.

Quoted 19 lines
Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB
on 32bit:
  * master-j1: 2.445 GB
  * master-j16: 2.608 GB
  * master-j32: 2.811 GB
  * next-j1: 2.256 GB
  * next-j16: 2.401 GB
  * next-j32: 2.613 GB

Quoted 6 lines
Quoted 3 lines

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:

Not much helping.

On my box (as all previous runs):
next-j1 2.242
next-j16 2.808
next-j32 2.646

Quoted 5 lines

On a build host:
next-j1: 2.242
next-j16: 2.824
next-j20: 2.902 (crash)
next-j32: 2.902 (crash)

So I disabled BTF on all 32bits (arm+i586) in openSUSE Tumbleweed for now. Better kernel without BTF, than no kernel at all 8-).

If you have ideas to force -j1 on 32bit, I can reenable/retest...

thanks,

Signature · 2 lines
js
suse labs
#16 lore raw
In reply to Jiri Slaby
Quoted 30 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.

Here, it hits the VM space limit (3 G).

Quoted 6 lines

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB

on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB

Quoted 3 lines

It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.

Great! I'm now processing a patch by Alan Maguire that should help with parallel loading, I'll add it o the next branch, together with his work on distilled BTF.

If you could test it, it would be great to see how much it helps with teh serial case and if it allows for at least some parallel processing on 32-bit architectures.

Thanks for all your effort on this, appreciated.

- Arnaldo

#17 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 6 lines · Arnaldo Carvalho de Melo

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

For pahole version 1.28 - Please, Go Go Go.

-Sedat-

pahole 1.27 segfaults when generating BTF for modules built with LTO #2032
https://github.com/ClangBuiltLinux/linux/issues/2032

#18 lore raw
Quoted 11 lines · Sedat Dilek
Quoted 6 lines · Arnaldo Carvalho de Melo

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master branch and cut 1.28.

Quoted 6 lines

For pahole version 1.28 - Please, Go Go Go.

-Sedat-

pahole 1.27 segfaults when generating BTF for modules built with LTO #2032
https://github.com/ClangBuiltLinux/linux/issues/2032

Hi Arnaldo,

Any news for pahole version 1.28 release?

Thanks.

Best regards,
-Sedat-