kbuild: bpf: Do not run pahole with -j on 32bit userspace
RFC 1/1 kbuild: bpf: Do not run pahole with -j on 32bit userspace
From: Jiri Slaby <[email protected]>
== WARNING ==
This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM are completely unhandled.
The simple version is just do there:
ifeq ($(CONFIG_64BIT,y)
but it has its own deficiencies, of course.
So any ideas, inputs?
== WARNING ==
When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:
Quoted 2 lines
btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.btf_encoder__encode: failed to tag kfuncs!
or simply SIGSEGV (failed to allocate the btf encoder).
It very depends on how many threads are created.
So do not invoke pahole with -j on 32bit.
--- init/Kconfig | 4 ++++ scripts/Makefile.btf | 2 ++ scripts/pahole-class.sh | 21 +++++++++++++++++++++ 3 files changed, 27 insertions(+) create mode 100644 scripts/pahole-class.sh
init/Kconfig +4 -0
@@ -113,6 +113,10 @@ config PAHOLE_VERSION int default $(shell,$(srctree)/scripts/pahole-version.sh $(PAHOLE)) +config PAHOLE_CLASS + string + default $(shell,$(srctree)/scripts/pahole-class.sh $(PAHOLE)) + config CONSTRUCTORS bool
scripts/Makefile.btf +2 -0
@@ -12,7 +12,9 @@ endif pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64) pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j +endif pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
scripts/pahole-class.sh new file +21 -0
@@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Usage: $ ./pahole-class.sh pahole +# +# Prints pahole's ELF class, such as ELF64 + +if [ ! -x "$(command -v "$@")" ]; then + echo 0 + exit 1 +fi + +PAHOLE="$(which "$@")" +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')" + +# Scripts like scripts/dummy-tools/pahole +if [ -n "$CLASS" ]; then + echo "$CLASS" +else + echo ELF64 +fi
Signature · 1 line
2.46.0
Quoted 11 lines · Jiri Slaby (SUSE)
From: Jiri Slaby <[email protected]>
== WARNING ==
This is only a PoC. There are deficiencies likeCROSS_COMPILEor LLVM are completely unhandled.The simple version is just do there:
ifeq ($(CONFIG_64BIT,y)but it has its own deficiencies, of course.
So any ideas, inputs?
Also as Shung-Hsi Yu suggests, we can cap -j to 1 in pahole proper when sizeof(long) == 4.
Quoted 10 lines
== WARNING ==
When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:
Quoted 2 lines
btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.btf_encoder__encode: failed to tag kfuncs!or simply SIGSEGV (failed to allocate the btf encoder).
It very depends on how many threads are created.
I forgot to add, that it depends on the kernel version too. It happens much often with 6.11-rc now (vmlinux got big enough, apparently).
thanks,
Signature · 2 lines
js suse labs
Quoted 15 lines · Jiri Slaby (SUSE)
From: Jiri Slaby <[email protected]>
== WARNING ==
This is only a PoC. There are deficiencies likeCROSS_COMPILEor LLVM are completely unhandled.The simple version is just do there:
ifeq ($(CONFIG_64BIT,y)
but it has its own deficiencies, of course.So any ideas, inputs?
== WARNING ==When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:
Quoted 8 lines
Quoted 2 lines
btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.btf_encoder__encode: failed to tag kfuncs!or simply SIGSEGV (failed to allocate the btf encoder).
It very depends on how many threads are created.
So do not invoke pahole with -j on 32bit.
could you share more details about your setup?
does it need to run on pure 32bit to reproduce? I can't reproduce when doing cross build and running 32 bit pahole on x86_64.. I do see some errors though
[667939] STRUCT bpf_prog_aux Error emitting BTF type Encountered error while encoding BTF.
thanks,
jirka
Quoted 90 lines
Signed-off-by Jiri Slaby <[email protected]>Fixes b4f72786429c ("scripts/pahole-flags.sh: Parse DWARF and generate BTF with multithreading.")Cc Masahiro Yamada <[email protected]>Cc Nathan Chancellor <[email protected]>Cc Nicolas Schier <[email protected]>Cc Alexei Starovoitov <[email protected]>Cc Daniel Borkmann <[email protected]>Cc Andrii Nakryiko <[email protected]>Cc Martin KaFai Lau <[email protected]>Cc Eduard Zingerman <[email protected]>Cc Song Liu <[email protected]>Cc Yonghong Song <[email protected]>Cc John Fastabend <[email protected]>Cc KP Singh <[email protected]>Cc Stanislav Fomichev <[email protected]>Cc Hao Luo <[email protected]>Cc Jiri Olsa <[email protected]>--- init/Kconfig | 4 ++++ scripts/Makefile.btf | 2 ++ scripts/pahole-class.sh | 21 +++++++++++++++++++++ 3 files changed, 27 insertions(+) create mode 100644 scripts/pahole-class.shinit/Kconfig +4
-0@@ -113,6 +113,10 @@ config PAHOLE_VERSION int default $(shell,$(srctree)/scripts/pahole-version.sh $(PAHOLE)) +config PAHOLE_CLASS + string + default $(shell,$(srctree)/scripts/pahole-class.sh $(PAHOLE)) + config CONSTRUCTORS boolscripts/Makefile.btf +2
-0@@ -12,7 +12,9 @@ endif pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64) pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j +endif pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimizedscripts/pahole-class.sh new file +21
-0@@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Usage: $ ./pahole-class.sh pahole +# +# Prints pahole's ELF class, such as ELF64 + +if [ ! -x "$(command -v "$@")" ]; then + echo 0 + exit 1 +fi + +PAHOLE="$(which "$@")" +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')" + +# Scripts like scripts/dummy-tools/pahole +if [ -n "$CLASS" ]; then + echo "$CLASS" +else + echo ELF64 +fiSignature · 1 line
2.46.0
Quoted 21 lines · Jiri Olsa
Quoted 20 lines · Jiri Slaby (SUSE)
From: Jiri Slaby <[email protected]>
== WARNING ==
This is only a PoC. There are deficiencies likeCROSS_COMPILEor LLVM are completely unhandled.The simple version is just do there:
ifeq ($(CONFIG_64BIT,y)but it has its own deficiencies, of course.
So any ideas, inputs?
== WARNING ==When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:
Quoted 2 lines
btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.btf_encoder__encode: failed to tag kfuncs!or simply SIGSEGV (failed to allocate the btf encoder).
Quoted 7 lines
Quoted 3 lines
It very depends on how many threads are created.
So do not invoke pahole with -j on 32bit.
could you share more details about your setup?
does it need to run on pure 32bit to reproduce?
armv7l builds are 32bit only.
Quoted 2 lines
I can't reproduce when
doing cross build and running 32 bit pahole onx86_64..
i586 is built using 64bit kernel. It is enough to have 32bit userspace. As written in the linked bug:
https://bugzilla.suse.com/show_bug.cgi?id=1229450#c6
FWIW, steps to reproduce locally:
docker pull jirislaby/pahole_crash
docker run -it jirislaby/pahole_crash
The VM space of pahole is exhausted:
process map: https://bugzilla.suse.com/attachment.cgi?id=876821
strace of mmaps: https://bugzilla.suse.com/attachment.cgi?id=876822
You need to run with large enough -j on a fast machine. Note that this happens on build hosts even with -j4, but they are under heavy load, so parallelism of held memory is high.
On my box with 16 cores, it is (likely far) enough to run with -j32.
Quoted 5 lines
I do see some
errors though[667939] STRUCT
bpf_prog_auxError emitting BTF type Encountered error while encoding BTF.
It's possible that it is one of the errors. There are different ones. As I wrote above, sometimes it is a crash, sometimes it is the failure I mentioned above. But it always ends up with a failed build:
Quoted 7 lines
libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available
make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
make[2]: *** Deleting file 'vmlinux'
make[1]: *** [/home/abuild/rpmbuild/BUILD/kernel-vanilla-6.11~rc3.338.gc3f2d783a459/linux-6.11-rc3-338-gc3f2d783a459/Makefile:1158: vmlinux] Error 2
make: *** [../Makefile:224: __sub-make] Error 2
error: Bad exit status from/var/tmp/rpm-tmp.olf5Nu(%build)
thanks,
Signature · 2 lines
js suse labs
Quoted 43 lines · Jiri Slaby
Quoted 21 lines · Jiri Olsa
Quoted 20 lines · Jiri Slaby (SUSE)
From: Jiri Slaby <[email protected]>
== WARNING ==
This is only a PoC. There are deficiencies likeCROSS_COMPILEor LLVM are completely unhandled.The simple version is just do there:
ifeq ($(CONFIG_64BIT,y)
but it has its own deficiencies, of course.So any ideas, inputs?
== WARNING ==When pahole is run with -j on 32bit userspace (32bit pahole in particular), it randomly fails with OOM:
Quoted 2 lines
btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.btf_encoder__encode: failed to tag kfuncs!or simply SIGSEGV (failed to allocate the btf encoder).
Quoted 7 lines
Quoted 3 lines
It very depends on how many threads are created.
So do not invoke pahole with -j on 32bit.
could you share more details about your setup?
does it need to run on pure 32bit to reproduce?
armv7l builds are 32bit only.
Quoted 2 lines
I can't reproduce when
doing cross build and running 32 bit pahole onx86_64..i586 is built using 64bit kernel. It is enough to have 32bit userspace. As written in the linked bug:
https://bugzilla.suse.com/show_bug.cgi?id=1229450#c6FWIW, steps to reproduce locally:
docker pull jirislaby/pahole_crash
docker run -it jirislaby/pahole_crash
Quoted 5 lines
The VM space of pahole is exhausted:
process map: https://bugzilla.suse.com/attachment.cgi?id=876821
strace of mmaps: https://bugzilla.suse.com/attachment.cgi?id=876822You need to run with large enough -j on a fast machine. Note that this happens on build hosts even with -j4, but they are under heavy load, so parallelism of held memory is high.
From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB
On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GB
Look there for full massif report.
So now I think we should disable BTF generation with 32bit pahole completely. Or someone debugs it and improves debug info loading to not eat that much.
thanks,
Signature · 2 lines
js suse labs
Quoted 6 lines · Jiri Slaby
From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB
Quoted 6 lines
On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GBLook there for full massif report.
From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
(In reply to Jiri Slaby from comment #20)
Quoted 1 line
| | | ->24.01% (954,816,480B)
0x489B4AB: UnknownInlinedFun (dwarf_loader.c:959)
So given this struct class_member is the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.
dwarves.h +3 -3
@@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,*/ struct tag {struct list_head node; +const char *attribute; +void *priv;type_id_t type;uint16_t tag; +uint16_t recursivity_level;bool visited;bool top_level;bool has_btf_type_tag; -uint16_t recursivity_level; -const char *attribute; -void *priv; };// To use with things like type->type_enum ==
perf_event_type+perf_user_event_type
@@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct function *func)
struct class_member {
struct tag tag;
const char *name;
+ uint64_t const_value;
uint32_t bit_offset;
uint32_t bit_size;
uint32_t byte_offset;
int hole;
size_t byte_size;
+ uint32_t alignment;
int8_t bitfield_offset;
uint8_t bitfield_size;
uint8_t bit_hole;
uint8_t bitfield_end:1;
- uint64_t const_value;
- uint32_t alignment;
uint8_t visited:1;
uint8_t is_static:1;
uint8_t has_bit_offset:1;Signature · 2 lines
js suse labs
(Add pahole maintainer and mailing list)
Hi Arnaldo,
We're running into kernel build failure on 32-bit (both full 32-bit and 32-bit userspace on 64-bit kernel) because pahole crashed due to virtual memory exhaustion[1]. As a workaround we currently limit pahole's parallel job count to 1 on such system[2]:
On Tue, 20 Aug 2024 10:59:50AM +0200, Jiri Slaby wrote:
[...]
Quoted 32 lines
scripts/Makefile.btf +2
-0@@ -12,7 +12,9 @@ endif pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64) pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j +endif pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimizedscripts/pahole-class.sh new file +11
-0@@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Usage: $ ./pahole-class.sh pahole +# +# Prints pahole's ELF class, such as ELF64 + +if [ ! -x "$(command -v "$@")" ]; then + echo 0 + exit 1 +fi
scripts/pahole-class.sh line 21
Quoted 10 lines
scripts/pahole-class.sh new file +10
-0+ +PAHOLE="$(which "$@")" +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')" + +# Scripts like scripts/dummy-tools/pahole +if [ -n "$CLASS" ]; then + echo "$CLASS" +else + echo ELF64 +fi
This helped lowered the memory usage enough so pahole no longer crash:
Quoted 11 lines · Jiri Slaby
Quoted 10 lines · Jiri Slaby
From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GBOn 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GB
Jiri (Slaby) in the meanwhile has also proposed structure packing to further reduce memory usage. (Note: I think the numbers below are from a 64-bit machine)
Quoted 50 lines
From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
(In reply to Jiri Slaby from comment #20)Quoted 1 line
| | | ->24.01% (954,816,480B)
0x489B4AB: UnknownInlinedFun(dwarf_loader.c:959)
So given this struct
class_memberis the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.dwarves.h +3
-3@@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,*/ struct tag {struct list_head node; +const char *attribute; +void *priv;type_id_t type;uint16_t tag; +uint16_t recursivity_level;bool visited;bool top_level;bool has_btf_type_tag; -uint16_t recursivity_level; -const char *attribute; -void *priv; };// To use with things like type->type_enum == perf_event_type+perf_user_event_type
dwarves.h +2
-2@@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct function *func)struct class_member {struct tag tag;const char *name; +uint64_t const_value;uint32_t bit_offset;uint32_t bit_size;uint32_t byte_offset;int hole;size_t byte_size; +uint32_t alignment;int8_t bitfield_offset;uint8_t bitfield_size;uint8_t bit_hole;uint8_t bitfield_end:1 ; -uint64_t const_value; -uint32_t alignment;uint8_t visited:1 ;uint8_t is_static:1 ;uint8_t has_bit_offset:1;
Quoted 1 line
--
What do you think?
IIUC pahole's memory usage is largely tied to the number of entries in vmlinux/kmodule DWARF, and there probably isn't much we could do about that.
Shung-Hsi
1: https://bugzilla.suse.com/show_bug.cgi?id=1229450
2: https://lore.kernel.org/all/[email protected]/
Quoted 64 lines · Shung-Hsi Yu
(Add pahole maintainer and mailing list)
Hi Arnaldo,
We're running into kernel build failure on 32-bit (both full 32-bit and 32-bit userspace on 64-bit kernel) because pahole crashed due to virtual memory exhaustion[1]. As a workaround we currently limit pahole's parallel job count to 1 on such system[2]:
On Tue, 20 Aug 2024 10:59:50AM +0200, Jiri Slaby wrote:
[...]Quoted 32 lines
scripts/Makefile.btf +2
-0@@ -12,7 +12,9 @@ endif pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64) pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j +endif pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimizedscripts/pahole-class.sh new file +11
-0@@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Usage: $ ./pahole-class.sh pahole +# +# Prints pahole's ELF class, such as ELF64 + +if [ ! -x "$(command -v "$@")" ]; then + echo 0 + exit 1 +fiscripts/pahole-class.sh line 21
Quoted 10 lines
scripts/pahole-class.sh new file +10
-0+ +PAHOLE="$(which "$@")" +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')" + +# Scripts like scripts/dummy-tools/pahole +if [ -n "$CLASS" ]; then + echo "$CLASS" +else + echo ELF64 +fiThis helped lowered the memory usage enough so pahole no longer crash:
Quoted 7 lines · Jiri Slaby
Quoted 6 lines · Jiri Slaby
From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GBJiri (Slaby) in the meanwhile has also proposed structure packing to further reduce memory usage. (Note: I think the numbers below are from a 64-bit machine)
That is interesting, packing pahole data structures ;-) :-)
Also a coincidence is that I did some packing on what is in the next branch:
3ef508ad94012933 dwarf_loader: Make 'struct dwarf_tag' more compact by getting rid of 'struct dwarf_off_ref 70febc8858588348 core: Reorganize 'struct class_member' to save 8 bytes76bcb88a67556468 core: Make tag->recursivity_level a uint8_tb8b9e04d177d8eb7 core: Make tag->top_level a single bit flag539acefcdd5b0f71 core: Make tag->has_btf_type_tag a single bit flagdba2c2c1aa5dfa05 core: Make tag->visited a single bit flag7409cfadcae0253b core: Shrink 'struct namespace' a bit by using a hole in its embedded 'tag'
Also I did more work to reduce the number of allocations:
cbecc3785266f0c5 dwarf_loader: Do just one alloc for 'struct dwarf_tag + struct tag'
With it we get:
⬢[acme@toolbox pahole]$ pahole -C class_member build/libdwarves.so.1.0.0 struct class_member {
struct tag tag; /* 0 32 */
/* XXX last struct has 1 bit hole */
const char * name; /* 32 8 */
uint32_t bit_offset; /* 40 4 */
uint32_t bit_size; /* 44 4 */
uint32_t byte_offset; /* 48 4 */
int hole; /* 52 4 */
size_t byte_size; /* 56 8 */
/* --- cacheline 1 boundary (64 bytes) --- */
int8_t bitfield_offset; /* 64 1 */
uint8_t bitfield_size; /* 65 1 */
uint8_t bit_hole; /* 66 1 */
uint8_t bitfield_end:1; /* 67: 0 1 */
uint8_t visited:1; /* 67: 1 1 */
uint8_t is_static:1; /* 67: 2 1 */
uint8_t has_bit_offset:1; /* 67: 3 1 */
uint8_t accessibility:2; /* 67: 4 1 */
uint8_t virtuality:2; /* 67: 6 1 */
uint32_t alignment; /* 68 4 */
uint64_t const_value; /* 72 8 */
/* size: 80, cachelines: 2, members: 18 */ /* member types with bit holes: 1, total: 1 */ /* last cacheline: 16 bytes */
};
⬢[acme@toolbox pahole]$
And also the dwarf_tag, that is allocated for each tag coming from DWARF got smaller:
⬢[acme@toolbox pahole]$ pahole -C dwarf_tag build/libdwarves.so.1.0.0 struct dwarf_tag {
struct hlist_node hash_node; /* 0 16 */ Dwarf_Off type; /* 16 8 */ Dwarf_Off id; /* 24 8 */
union {
Dwarf_Off abstract_origin; /* 32 8 */
Dwarf_Off containing_type; /* 32 8 */
}; /* 32 8 */
Dwarf_Off specification; /* 40 8 */
struct {
_Bool type:1; /* 48: 0 1 */
_Bool abstract_origin:1; /* 48: 1 1 */
_Bool containing_type:1; /* 48: 2 1 */
_Bool specification:1; /* 48: 3 1 */
} from_types_section; /* 48 1 */
/* XXX last struct has 4 bits of padding */ /* XXX 1 byte hole, try to pack */
uint16_t decl_line; /* 50 2 */
uint32_t small_id; /* 52 4 */
const char * decl_file; /* 56 8 */
/* size: 64, cachelines: 1, members: 9 */
/* sum members: 63, holes: 1, sum holes: 1 */ /* member types with bit paddings: 1, total: 4 bits */
};
⬢[acme@toolbox pahole]$
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
Thanks!
- Arnaldo
Quoted 62 lines
Quoted 50 lines
From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
(In reply to Jiri Slaby from comment #20)Quoted 1 line
| | | ->24.01% (954,816,480B)
0x489B4AB: UnknownInlinedFun(dwarf_loader.c:959)
So given this struct
class_memberis the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.dwarves.h +3
-3@@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,*/ struct tag {struct list_head node; +const char *attribute; +void *priv;type_id_t type;uint16_t tag; +uint16_t recursivity_level;bool visited;bool top_level;bool has_btf_type_tag; -uint16_t recursivity_level; -const char *attribute; -void *priv; };// To use with things like type->type_enum == perf_event_type+perf_user_event_type
dwarves.h +2
-2@@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct function *func)struct class_member {struct tag tag;const char *name; +uint64_t const_value;uint32_t bit_offset;uint32_t bit_size;uint32_t byte_offset;int hole;size_t byte_size; +uint32_t alignment;int8_t bitfield_offset;uint8_t bitfield_size;uint8_t bit_hole;uint8_t bitfield_end:1 ; -uint64_t const_value; -uint32_t alignment;uint8_t visited:1 ;uint8_t is_static:1 ;uint8_t has_bit_offset:1;Quoted 1 line
--
What do you think?
IIUC pahole's memory usage is largely tied to the number of entries in vmlinux/kmodule DWARF, and there probably isn't much we could do about that.
Shung-Hsi
1: https://bugzilla.suse.com/show_bug.cgi?id=1229450
2: https://lore.kernel.org/all/[email protected]/
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.
Here, it hits the VM space limit (3 G).
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB
on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
thanks,
Signature · 2 lines
js suse labs
Quoted 6 lines · Arnaldo Carvalho de Melo
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
For pahole version 1.28 - Please, Go Go Go.
-Sedat-
pahole 1.27 segfaults when generating BTF for modules built with LTO #2032
https://github.com/ClangBuiltLinux/linux/issues/2032
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
right, in my case it was on a 64-bit system, so just not enough memory, not address space.
Quoted 20 lines
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GBon 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB
Quoted 3 lines
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:
⬢[acme@toolbox pahole]$ git log --oneline -5f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs
Excerpt of the above:
This leads to deleting ~90 CUs during parallel vmlinux BTF generation versus deleting just 1 prior to this change.
c7ec9200caa7d485 btf_encoder: Add "distilled_base" BTF feature to split BTF generation bc4e6a9adfc72758 pahole: Sync with libbpf-1.55e3ed3ec2947c69f pahole: Do --lang_exclude CU filtering earlier c46455bb0379fa38 dwarf_loader: Allow filtering CUs early in loading ⬢[acme@toolbox pahole]$
- Arnaldo
Quoted 38 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
right, in my case it was on a 64-bit system, so just not enough memory, not address space.
Quoted 20 lines
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GBon 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GBQuoted 3 lines
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Quoted 6 lines
Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:
⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2(HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs
*R*edzce? memory usage ...
-Sedat-
Quoted 13 lines
Excerpt of the above:
This leads to deleting ~90 CUs during parallel vmlinux BTF generation versus deleting just 1 prior to this change.
c7ec9200caa7d485btf_encoder: Add "distilled_base" BTF feature to split BTF generationbc4e6a9adfc72758pahole: Sync with libbpf-1.55e3ed3ec2947c69fpahole: Do --lang_exclude CU filtering earlierc46455bb0379fa38dwarf_loader: Allow filtering CUs early in loading ⬢[acme@toolbox pahole]$- Arnaldo
Quoted 40 lines · On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
Quoted 38 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
right, in my case it was on a 64-bit system, so just not enough memory, not address space.
Quoted 23 lines
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GBon 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GBIt's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Quoted 9 lines
Quoted 6 lines
Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:
⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2(HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs*R*edzce? memory usage ...
If you meant that further typo it's golden, and if not the irony is rich :)
Either way this is my favorite email of the day!
Cheers,
Phil
Quoted 16 lines
-Sedat-
Quoted 13 lines
Excerpt of the above:
This leads to deleting ~90 CUs during parallel vmlinux BTF generation versus deleting just 1 prior to this change.
c7ec9200caa7d485btf_encoder: Add "distilled_base" BTF feature to split BTF generationbc4e6a9adfc72758pahole: Sync with libbpf-1.55e3ed3ec2947c69fpahole: Do --lang_exclude CU filtering earlierc46455bb0379fa38dwarf_loader: Allow filtering CUs early in loading ⬢[acme@toolbox pahole]$- Arnaldo
Quoted 30 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GBon 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB
Quoted 3 lines
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Great! I'm now processing a patch by Alan Maguire that should help with parallel loading, I'll add it o the next branch, together with his work on distilled BTF.
If you could test it, it would be great to see how much it helps with teh serial case and if it allows for at least some parallel processing on 32-bit architectures.
Thanks for all your effort on this, appreciated.
- Arnaldo
Quoted 48 lines · Phil Auld
Quoted 47 lines · On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
Quoted 44 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
right, in my case it was on a 64-bit system, so just not enough memory, not address space.
Quoted 20 lines
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GBon 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GBQuoted 3 lines
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:
⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2(HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs
Quoted 6 lines
Quoted 2 lines
*R*edzce? memory usage ...
If you meant that further typo it's golden, and if not the irony is rich :)
Either way this is my favorite email of the day!
Hahaha, I went to uppercase what comes after the colon and introduced that typo ;-)
Faxing it....
- Arnaldo
Quoted 53 lines · Arnaldo Carvalho de Melo
Quoted 51 lines · Phil Auld
Quoted 49 lines · On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
Quoted 38 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
right, in my case it was on a 64-bit system, so just not enough memory, not address space.
Quoted 20 lines
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GBon 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GBQuoted 3 lines
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Quoted 6 lines
Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:
⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2(HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs*R*edzce? memory usage ...
Quoted 8 lines
Quoted 3 lines
If you meant that further typo it's golden, and if not the irony is rich :)
Either way this is my favorite email of the day!
Hahaha, I went to uppercase what comes after the colon and introduced that typo ;-)
Faxing it....
typo-fest continues ;)
but it's great to see that a new pahole release is coming (cc'ing Usama), we are eagerly waiting for one of the bug fixes that will go into 1.28 (one of the first commits after 1.27 was cut). Note, libbpf CI's for pahole-staging is failing, but that seems a one-off test and has nothing to do with pahole being broken.
Quoted 3 lines
- Arnaldo
Quoted 33 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
right, in my case it was on a 64-bit system, so just not enough memory, not address space.
Quoted 20 lines
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GBon 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GB
Quoted 6 lines
Quoted 3 lines
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:
Not much helping.
On my box (as all previous runs):
next-j1 2.242
next-j16 2.808
next-j32 2.646
On a build host:
next-j1: 2.242
next-j16: 2.824
next-j20: 2.902 (crash)
next-j32: 2.902 (crash)
Signature · 2 lines
js suse labs
Quoted 46 lines · Jiri Slaby
Quoted 31 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Jiri Slaby
Quoted 5 lines · Arnaldo Carvalho de Melo
On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
I stumbled on this limitation as well when trying to build the kernel on a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just created a swapfile and it managed to proceed, a bit slowly, but worked as well.Here, it hits the VM space limit (3 G).
right, in my case it was on a 64-bit system, so just not enough memory, not address space.
Quoted 19 lines
Quoted 6 lines
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB
on 32bit:
* master-j1: 2.445 GB
* master-j16: 2.608 GB
* master-j32: 2.811 GB
* next-j1: 2.256 GB
* next-j16: 2.401 GB
* next-j32: 2.613 GBQuoted 6 lines
Quoted 3 lines
It's definitely better. So I think it could work now, if the thread count was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on random machines (32bit processes only of course). Unlike -j1.
Cool, I just merged a patch from Alan Maguire that should help with the parallel case, would be able to test it? It is in the 'next' branch:
Not much helping.
On my box (as all previous runs):
next-j1 2.242
next-j16 2.808
next-j32 2.646
Quoted 5 lines
On a build host:
next-j1: 2.242
next-j16: 2.824
next-j20: 2.902 (crash)
next-j32: 2.902 (crash)
So I disabled BTF on all 32bits (arm+i586) in openSUSE Tumbleweed for now. Better kernel without BTF, than no kernel at all 8-).
If you have ideas to force -j1 on 32bit, I can reenable/retest...
thanks,
Signature · 2 lines
js suse labs
Quoted 11 lines · Sedat Dilek
Quoted 6 lines · Arnaldo Carvalho de Melo
Please let me know if what is in the 'next' branch of:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Works for you, that will be extra motivation to move it to the master branch and cut 1.28.
Quoted 6 lines
For pahole version 1.28 - Please, Go Go Go.
-Sedat-
pahole 1.27 segfaults when generating BTF for modules built with LTO #2032
https://github.com/ClangBuiltLinux/linux/issues/2032
Hi Arnaldo,
Any news for pahole version 1.28 release?
Thanks.
Best regards,
-Sedat-