Skip to contentSkip to conversation

ANNOUNCE: pahole v1.27 (reproducible builds, BTF kfuncs)

PATCH New Arnaldo Carvalho de Melo 4 participants 13 messages Jun 11, 2024 – Jul 10, 2024 BPF

lore
Share & tools
lore
#0 lore raw

Hi,

The v1.27 release of pahole and its friends is out, supporting
parallel reproducible builds and encoding kernel kfuncs in BTF, allowing tools such as bpftrace to enumerate the available kfuncs and obtain its function signatures and return types.

Main git repo:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Mirror git repo:

https://github.com/acmel/dwarves.git

tarball + gpg signature:

https://fedorapeople.org/~acme/dwarves/dwarves-1.27.tar.xz
https://fedorapeople.org/~acme/dwarves/dwarves-1.27.tar.bz2
https://fedorapeople.org/~acme/dwarves/dwarves-1.27.tar.sign

Thanks a lot to all the contributors and distro packagers, you're on the
CC list, I appreciate a lot the work you put into these tools,

Best Regards,

- Arnaldo

BTF encoder:

- Inject kfunc decl tags into BTF from the BTF IDs ELF section in the Linux kernel vmlinux file.

This allows tools such as bpftools and pfunct to enumerate the available kfuncs and to gets its function signature, the type of its return and of its arguments. See the example in the BTF loader changes description, below.

- Support parallel reproducible builds, where it doesn't matter how many threads are used, the end BTF encoding result is the same.

- Sanitize unsupported DWARF int type with greater-than-16 byte, as BTF doesn't support it.

BTF loader:

- Initial support for BTF_KIND_DECL_TAG:

  $ pfunct --prototypes -F btf vmlinux.btf.decl_tag,decl_tag_kfuncs | grep ^bpf_kfunc | head
  bpf_kfunc void cubictcp_init(struct sock * sk);
  bpf_kfunc void cubictcp_cwnd_event(struct sock * sk, enum tcp_ca_event event);
  bpf_kfunc void cubictcp_cong_avoid(struct sock * sk, u32 ack, u32 acked);
  bpf_kfunc u32 cubictcp_recalc_ssthresh(struct sock * sk);
  bpf_kfunc void cubictcp_state(struct sock * sk, u8 new_state);
  bpf_kfunc void cubictcp_acked(struct sock * sk, const struct ack_sample  * sample);
  bpf_kfunc int bpf_iter_css_new(struct bpf_iter_css * it, struct cgroup_subsys_state * start, unsigned int flags);
  bpf_kfunc struct cgroup_subsys_state * bpf_iter_css_next(struct bpf_iter_css * it);
  bpf_kfunc void bpf_iter_css_destroy(struct bpf_iter_css * it);
  bpf_kfunc s64 bpf_map_sum_elem_count(const struct bpf_map  * map);
  $ pfunct --prototypes -F btf vmlinux.btf.decl_tag,decl_tag_kfuncs | grep ^bpf_kfunc | wc -l
  116
  $

pretty printing:

- Fix hole discovery with inheritance in C++.

#1 lore raw

Hi,

Arnaldo Carvalho de Melo wrote on Tue, Jun 11, 2024 at 06:26:53PM -0300:

Quoted 8 lines

The v1.27 release of pahole and its friends is out, supporting
parallel reproducible builds and encoding kernel kfuncs in BTF, allowing tools such as bpftrace to enumerate the available kfuncs and obtain its function signatures and return types.

Main git repo:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

It looks like the v1.27 tag has not been pushed to the git repos (either this or github), we're using git snapshots for nixpkgs, so it'd be great if a tag could be pushed out.
(I think some release monitoring tools left and right also use tags, even if that's less important if you Cc other distro maintainers... I just happened to see the mail on bpf@vger.)

Thanks,

Signature · 1 line
Dominique Martinet | Asmadeus
#2 lore raw
Quoted 3 lines · Dominique Martinet

It looks like the v1.27 tag has not been pushed to the git repos (either this or github), we're using git snapshots for nixpkgs, so it'd be great if a tag could be pushed out.

Done.

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tag/?h=v1.27
https://github.com/acmel/dwarves/releases/tag/v1.27

Quoted 3 lines

(I think some release monitoring tools left and right also use tags, even if that's less important if you Cc other distro maintainers... I just happened to see the mail on bpf@vger.)

May I add your e-mail here:

acme@x1:~/git/pahole$ cat PKG-MAINTAINERS 
# Please let me know if I should remove/update/add more distro package maintainers here
# I'm keeping this so that I CC them when releasing new versions, thanks!

Jan Alexander Steffens <[email protected]>
Domenico Andreoli <[email protected]>
Matthias Schwarzott <[email protected]>
Dominique Leuenberger <[email protected]>
acme@x1:~/git/pahole$

So that on the next release I CC you?

Thanks for reporting!

- Arnaldo

#3 lore raw

Arnaldo Carvalho de Melo wrote on Tue, Jun 11, 2024 at 07:52:12PM -0300:

Quoted 9 lines · Dominique Martinet
Quoted 3 lines

It looks like the v1.27 tag has not been pushed to the git repos (either this or github), we're using git snapshots for nixpkgs, so it'd be great if a tag could be pushed out.

Done.

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tag/?h=v1.27
https://github.com/acmel/dwarves/releases/tag/v1.27

Thank you!

Quoted 7 lines
Quoted 3 lines

(I think some release monitoring tools left and right also use tags, even if that's less important if you Cc other distro maintainers... I just happened to see the mail on bpf@vger.)

May I add your e-mail here:

acme@x1:~/git/pahole$ cat PKG-MAINTAINERS 

[..]

Quoted 2 lines

So that on the next release I CC you?

Sure; in practice this shouldn't change much as package update is semi-automated within a few days, but getting a heads up cannot hurt.

Signature · 1 line
Dominique Martinet | Asmadeus
#4 lore raw
In reply to Arnaldo Carvalho de Melo

Am 11.06.24 um 23:26 schrieb Arnaldo Carvalho de Melo:

Quoted 7 lines

Hi,

The v1.27 release of pahole and its friends is out, supporting
parallel reproducible builds and encoding kernel kfuncs in BTF, allowing tools such as bpftrace to enumerate the available kfuncs and obtain its function signatures and return types.

Regarding packaging of pahole:
What is the state of the contained ostra-cg?
I have no clue what it is and how to use it. Is there still a use-case for it?

Starting it without arguments only shows the usage string. Running it with two dummy arguments:

$ ostra-cg x y
Traceback (most recent call last):
  File "/usr/bin/ostra-cg", line 404, in <module>
    class_def = ostra.class_definition(class_def_file = "%s.fields" % 
traced_class,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/dwarves/runtime/python/ostra.py", line 154, in __init__
f = file(class_def_file)
^^^^
NameError: name 'file' is not defined. Did you mean: 'field'?

According to https://stackoverflow.com/questions/32131230/python-file-function the function file() does not exist in python3.

This part could be fixed by replacing it with open() but I wonder if this is worth it.

As nobody has complained about it being broken:
Should ostra just be removed?

Regards
Matthias

#5 lore raw
Quoted 10 lines · Matthias Schwarzott

Am 11.06.24 um 23:26 schrieb Arnaldo Carvalho de Melo:

Quoted 6 lines

Hi,
The v1.27 release of pahole and its friends is out, supporting
parallel reproducible builds and encoding kernel kfuncs in BTF, allowing tools such as bpftrace to enumerate the available kfuncs and obtain its function signatures and return types.

Regarding packaging of pahole:
What is the state of the contained ostra-cg?

I need to make a decision on that, it is used to produce things like:

http://vger.kernel.org/~acme/ostra/callgraphs/sock/0xf61bf500/

As documented in:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/README.ctracer

But yes, it needs to get retested after all these years to see how difficult it would be to try and get it back working.

- Arnaldo

Quoted 29 lines

I have no clue what it is and how to use it. Is there still a use-case for it?

Starting it without arguments only shows the usage string. Running it with two dummy arguments:

$ ostra-cg x y
Traceback (most recent call last):
  File "/usr/bin/ostra-cg", line 404, in <module>
    class_def = ostra.class_definition(class_def_file = "%s.fields" %
traced_class,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/dwarves/runtime/python/ostra.py", line 154, in __init__
f = file(class_def_file)
^^^^
NameError: name 'file' is not defined. Did you mean: 'field'?

According to
https://stackoverflow.com/questions/32131230/python-file-function the function file() does not exist in python3.

This part could be fixed by replacing it with open() but I wonder if this is worth it.

As nobody has complained about it being broken:
Should ostra just be removed?

Regards
Matthias

#6 lore raw
In reply to Arnaldo Carvalho de Melo

Hi Arnaldo,

Quoted 4 lines · Arnaldo Carvalho de Melo

The v1.27 release of pahole and its friends is out, supporting
parallel reproducible builds and encoding kernel kfuncs in BTF, allowing tools such as bpftrace to enumerate the available kfuncs and obtain its function signatures and return types.

After commit f632e75 ("dwarf_loader: Add the cu to the cus list early, remove on LSK_DELETE"), I (and others[1]) notice a crash when running pahole on modules built with Clang when CONFIG_LTO_CLANG is enabled:

  $ curl -LSso .config https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/raw/main/config
  $ scripts/config -d LTO_NONE -e LTO_CLANG_THIN
  $ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig vmlinux crypto/cast_common.ko
  make[3]: *** [scripts/Makefile.modfinal:59: crypto/cast_common.ko] Error 139

I've isolated this to the following commands using the files available at [2] (these were built with LLVM 18 but I could reproduce it with LLVM 17 and LLVM 19, so it appears to impact a number of versions):

  $ tar -tf clang-lto-pahole-1.27-crash.tar.zst
  clang-lto-pahole-1.27-crash/
  clang-lto-pahole-1.27-crash/cast_common.mod.o
  clang-lto-pahole-1.27-crash/module.lds
  clang-lto-pahole-1.27-crash/cast_common.o
  clang-lto-pahole-1.27-crash/cast_common.ko.bak
  clang-lto-pahole-1.27-crash/vmlinux
  clang-lto-pahole-1.27-crash/cast_common.ko
  $ tar -axf clang-lto-pahole-1.27-crash.tar.zst
  $ cd clang-lto-pahole-1.27-crash
  $ LLVM_OBJCOPY="llvm-objcopy" pahole-1.26 -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust --btf_base vmlinux cast_common.ko
  $ cp cast_common.ko{.bak,}
  $ LLVM_OBJCOPY="llvm-objcopy" pahole-1.27 -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust --btf_base vmlinux cast_common.ko
  fish: Job 1, '...' terminated by signal SIGSEGV (Address boundary error)

If there is any more information I can provide or patches I can test, I am more than happy to do so.

[1]: https://gitlab.archlinux.org/archlinux/packaging/packages/pahole/-/issues/1
[2]: https://1drv.ms/u/s!AsQNYeB-IEbqqC2F28JuLy__Q7Vd?e=KsraMU

Cheers,
Nathan

#7 lore raw
Quoted 38 lines · Nathan Chancellor

Hi Arnaldo,

Quoted 4 lines · Arnaldo Carvalho de Melo

The v1.27 release of pahole and its friends is out, supporting
parallel reproducible builds and encoding kernel kfuncs in BTF, allowing tools such as bpftrace to enumerate the available kfuncs and obtain its function signatures and return types.

After commit f632e75 ("dwarf_loader: Add the cu to the cus list early, remove on LSK_DELETE"), I (and others[1]) notice a crash when running pahole on modules built with Clang when CONFIG_LTO_CLANG is enabled:

  $ curl -LSso .config https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/raw/main/config
  $ scripts/config -d LTO_NONE -e LTO_CLANG_THIN
  $ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig vmlinux crypto/cast_common.ko
  make[3]: *** [scripts/Makefile.modfinal:59: crypto/cast_common.ko] Error 139

I've isolated this to the following commands using the files available at [2] (these were built with LLVM 18 but I could reproduce it with LLVM 17 and LLVM 19, so it appears to impact a number of versions):

  $ tar -tf clang-lto-pahole-1.27-crash.tar.zst
  clang-lto-pahole-1.27-crash/
  clang-lto-pahole-1.27-crash/cast_common.mod.o
  clang-lto-pahole-1.27-crash/module.lds
  clang-lto-pahole-1.27-crash/cast_common.o
  clang-lto-pahole-1.27-crash/cast_common.ko.bak
  clang-lto-pahole-1.27-crash/vmlinux
  clang-lto-pahole-1.27-crash/cast_common.ko
  $ tar -axf clang-lto-pahole-1.27-crash.tar.zst
  $ cd clang-lto-pahole-1.27-crash
  $ LLVM_OBJCOPY="llvm-objcopy" pahole-1.26 -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust --btf_base vmlinux cast_common.ko
Quoted 7 lines
  $ cp cast_common.ko{.bak,}
  $ LLVM_OBJCOPY="llvm-objcopy" pahole-1.27 -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust --btf_base vmlinux cast_common.ko
  fish: Job 1, '...' terminated by signal SIGSEGV (Address boundary error)

If there is any more information I can provide or patches I can test, I am more than happy to do so.

I reproduced the problem by just running 'pahole cast_common.ko", so this isn't even related to the BTF parts, its about the DWARF loader, I'm on it, thanks for the detailed report and for providing the files.

- Arnaldo

Quoted 6 lines

[1]: https://gitlab.archlinux.org/archlinux/packaging/packages/pahole/-/issues/1
[2]: https://1drv.ms/u/s!AsQNYeB-IEbqqC2F28JuLy__Q7Vd?e=KsraMU

Cheers,
Nathan

#8 lore raw
Quoted 42 lines · Arnaldo Carvalho de Melo
Quoted 41 lines · Nathan Chancellor
Quoted 4 lines · Arnaldo Carvalho de Melo

The v1.27 release of pahole and its friends is out, supporting
parallel reproducible builds and encoding kernel kfuncs in BTF, allowing tools such as bpftrace to enumerate the available kfuncs and obtain its function signatures and return types.

After commit f632e75 ("dwarf_loader: Add the cu to the cus list early, remove on LSK_DELETE"), I (and others[1]) notice a crash when running pahole on modules built with Clang when CONFIG_LTO_CLANG is enabled:

  $ curl -LSso .config https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/raw/main/config
  $ scripts/config -d LTO_NONE -e LTO_CLANG_THIN
  $ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig vmlinux crypto/cast_common.ko
  make[3]: *** [scripts/Makefile.modfinal:59: crypto/cast_common.ko] Error 139

I've isolated this to the following commands using the files available at [2] (these were built with LLVM 18 but I could reproduce it with LLVM 17 and LLVM 19, so it appears to impact a number of versions):

  $ tar -tf clang-lto-pahole-1.27-crash.tar.zst
  clang-lto-pahole-1.27-crash/
  clang-lto-pahole-1.27-crash/cast_common.mod.o
  clang-lto-pahole-1.27-crash/module.lds
  clang-lto-pahole-1.27-crash/cast_common.o
  clang-lto-pahole-1.27-crash/cast_common.ko.bak
  clang-lto-pahole-1.27-crash/vmlinux
  clang-lto-pahole-1.27-crash/cast_common.ko
  $ tar -axf clang-lto-pahole-1.27-crash.tar.zst
  $ cd clang-lto-pahole-1.27-crash
  $ LLVM_OBJCOPY="llvm-objcopy" pahole-1.26 -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust --btf_base vmlinux cast_common.ko
  $ cp cast_common.ko{.bak,}
  $ LLVM_OBJCOPY="llvm-objcopy" pahole-1.27 -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust --btf_base vmlinux cast_common.ko
  fish: Job 1, '...' terminated by signal SIGSEGV (Address boundary error)
Quoted 6 lines
Quoted 2 lines

If there is any more information I can provide or patches I can test, I am more than happy to do so.

I reproduced the problem by just running 'pahole cast_common.ko", so this isn't even related to the BTF parts, its about the DWARF loader, I'm on it, thanks for the detailed report and for providing the files.

One liner + explanation at the end, with it both DWARF and BTF loaders work:

⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ pahole -F btf -C task_struct cast_common.ko | tail

	/* XXX last struct has 1 hole, 1 bit hole */

	/* size: 11584, cachelines: 181, members: 265 */
	/* sum members: 11483, holes: 22, sum holes: 85 */
	/* sum bitfield members: 82 bits, bit holes: 2, sum bit holes: 46 bits */
	/* member types with holes: 4, total: 6, bit holes: 2, total: 2 */
	/* paddings: 6, sum paddings: 49 */
	/* forced alignments: 1, forced holes: 1, sum forced holes: 8 */
};

⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ pahole -F dwarf -C task_struct cast_common.ko | tail

	/* XXX last struct has 1 hole, 1 bit hole */

	/* size: 11584, cachelines: 181, members: 265 */
	/* sum members: 11483, holes: 22, sum holes: 85 */
	/* sum bitfield members: 82 bits, bit holes: 2, sum bit holes: 46 bits */
	/* member types with holes: 4, total: 6, bit holes: 2, total: 2 */
	/* paddings: 6, sum paddings: 49 */
	/* forced alignments: 8, forced holes: 1, sum forced holes: 8 */
};

⬢[acme@toolbox clang-lto-pahole-1.27-crash]$

Also the BTF encoder (that uses the DWARF loader), etc.

⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ pahole -j --btf_encode cast_common.ko ⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ readelf -SW cast_common.ko | grep BTF
[30] .BTF PROGBITS 0000000000000000 0223c9 01172c 00 0 0 1
⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ pahole -F btf cast_common.ko | wc -l 4302
⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ set -o vi
⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ pahole -F btf cast_common.ko | head struct elf32_note {

	Elf32_Word                 n_namesz;             /*     0     4 */
	Elf32_Word                 n_descsz;             /*     4     4 */
	Elf32_Word                 n_type;               /*     8     4 */
	/* size: 12, cachelines: 1, members: 3 */
	/* last cacheline: 12 bytes */
};
struct module {
	enum module_state          state;                /*     0     4 */
⬢[acme@toolbox clang-lto-pahole-1.27-crash]$ pahole -F btf cast_common.ko | tail
	const char  *              mod_name;             /*    24     8 */
	const char  * *            class_names;          /*    32     8 */
	const int                  length;               /*    40     4 */
	const int                  base;                 /*    44     4 */
	enum class_map_type        map_type;             /*    48     4 */
	/* size: 56, cachelines: 1, members: 7 */
	/* padding: 4 */
	/* last cacheline: 56 bytes */
};

⬢[acme@toolbox clang-lto-pahole-1.27-crash]$

Can you try with the one liner below? We remove it from the cus list unconditionally, and since we alloc space with zalloc/calloc in cu__new() and missed initializing that list_head (cu->node) we ended up hitting list_del with a zeroed 'struct list_head' :-\

I'll try and get this cast_common.ko checked into a test repo for pahole so that this gets regression tested.

Please test this patch so that we see if this is the only problem and your kernel build with clang completes successfully.

Thanks,

- Arnaldo

dwarves.c +1 -0
index 1ec259f50dbd3778..823a01524a12bb37 100644
@@ -739,6 +739,7 @@ struct cu *cu__new(const char *name, uint8_t addr_size,
 		cu->dfops	= NULL;
 		INIT_LIST_HEAD(&cu->tags);
 		INIT_LIST_HEAD(&cu->tool_list);
+		INIT_LIST_HEAD(&cu->node);
 
 		cu->addr_size = addr_size;
 		cu->extra_dbg_info = 0;
#9 lore raw
Quoted 10 lines · Arnaldo Carvalho de Melo

Can you try with the one liner below? We remove it from the cus list unconditionally, and since we alloc space with zalloc/calloc in cu__new() and missed initializing that list_head (cu->node) we ended up hitting list_del with a zeroed 'struct list_head' :-\

I'll try and get this cast_common.ko checked into a test repo for pahole so that this gets regression tested.

Please test this patch so that we see if this is the only problem and your kernel build with clang completes successfully.

Thanks, I rebuilt pahole with the following diff and both my build and the other configuration I tested for this regression successfully complete.

Quoted 12 lines
dwarves.c +1 -0
index 1ec259f50dbd3778..823a01524a12bb37 100644
@@ -739,6 +739,7 @@ struct cu *cu__new(const char *name, uint8_t addr_size,
 		cu->dfops	= NULL;
 		INIT_LIST_HEAD(&cu->tags);
 		INIT_LIST_HEAD(&cu->tool_list);
+		INIT_LIST_HEAD(&cu->node);
 
 		cu->addr_size = addr_size;
 		cu->extra_dbg_info = 0;
author Tested-by Nathan Chancellor
#10 lore raw
Quoted 9 lines · Nathan Chancellor
Quoted 8 lines · Arnaldo Carvalho de Melo

Can you try with the one liner below? We remove it from the cus list unconditionally, and since we alloc space with zalloc/calloc in cu__new() and missed initializing that list_head (cu->node) we ended up hitting list_del with a zeroed 'struct list_head' :-\

I'll try and get this cast_common.ko checked into a test repo for pahole so that this gets regression tested.

Quoted 8 lines
Quoted 2 lines

Please test this patch so that we see if this is the only problem and your kernel build with clang completes successfully.

Thanks, I rebuilt pahole with the following diff and both my build and the other configuration I tested for this regression successfully complete.

Great, I just added this:

From 6a2b27c0f512619b0e7a769a18a0fb05bb3789a5 Mon Sep 17 00:00:00 2001
From: Arnaldo Carvalho de Melo <[email protected]>
Date: Tue, 18 Jun 2024 10:37:30 -0300
Subject: [PATCH 1/1] core: Initialize cu->node with INIT_LIST_HEAD()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

In cu__new() zalloc() is used defensively, and that helped catch this problem where we assume that a cu us in the cus list of cu instances, but that is not the case when we use cus__merge_and_process_cu(), for instance when loading files created by clang with LTO, as reported by Peter Jung and narrowed down by Nathan Chancellor.

If we use INIT_LIST_HEAD() in cu__new() to initialize cu->node, which is what we do with other lists and nodes there, then the unconditional removal using list_del_init() will be a no-op and removing something not on the cus list of cu instances will not cause problems, just keep an unconsistent cus->nr_entries field.

So lets just have this fix in first, keeping Nathan's Tested-by and then do the a bit more involved fix of either adding that cu to the cus list or checking at removal time if it is there.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7f1e13e in __list_del (prev=0x0, next=0x0) at /home/acme/git/pahole/list.h:106
106 next->prev = prev;
(gdb) bt

  #0  0x00007ffff7f1e13e in __list_del (prev=0x0, next=0x0) at /home/acme/git/pahole/list.h:106
  #1  0x00007ffff7f1e176 in list_del_init (entry=0x417980) at /home/acme/git/pahole/list.h:165
  #2  0x00007ffff7f1f8f9 in __cus__remove (cus=0x4142a0, cu=0x417980) at /home/acme/git/pahole/dwarves.c:527
  #3  0x00007ffff7f1f92b in cus__remove (cus=0x4142a0, cu=0x417980) at /home/acme/git/pahole/dwarves.c:533
  #4  0x00007ffff7f3d01c in cus__finalize (cus=0x4142a0, cu=0x417980, conf=0x4133c0 <conf_load>, thr_data=0x0)
      at /home/acme/git/pahole/dwarf_loader.c:3040
  #5  0x00007ffff7f3e05c in cus__merge_and_process_cu (cus=0x4142a0, conf=0x4133c0 <conf_load>, mod=0x415cf0, dw=0x416110, elf=0x414380,
      filename=0x7fffffffe3f7 "cast_common.ko", build_id=0x416680 "\265D\371U\213\373u|\037\250\242\032\271\365⒜]y\023", build_id_len=20,
      type_dcu=0x0) at /home/acme/git/pahole/dwarf_loader.c:3482
  #6  0x00007ffff7f3e218 in cus__load_module (cus=0x4142a0, conf=0x4133c0 <conf_load>, mod=0x415cf0, dw=0x416110, elf=0x414380,
      filename=0x7fffffffe3f7 "cast_common.ko") at /home/acme/git/pahole/dwarf_loader.c:3521
  #7  0x00007ffff7f3e396 in cus__process_dwflmod (dwflmod=0x415cf0, userdata=0x415d00, name=0x415ea0 "cast_common.ko", base=65536,
      arg=0x7fffffffde40) at /home/acme/git/pahole/dwarf_loader.c:3581
  #8  0x00007ffff7eb4609 in dwfl_getmodules (dwfl=0x414300, callback=0x7ffff7f3e2ec <cus__process_dwflmod>, arg=0x7fffffffde40, offset=0)
      at ../libdwfl/dwfl_getmodules.c:86
  #9  0x00007ffff7f3e4c5 in cus__process_file (cus=0x4142a0, conf=0x4133c0 <conf_load>, fd=3, filename=0x7fffffffe3f7 "cast_common.ko")
      at /home/acme/git/pahole/dwarf_loader.c:3647
  #10 0x00007ffff7f3e5cd in dwarf__load_file (cus=0x4142a0, conf=0x4133c0 <conf_load>, filename=0x7fffffffe3f7 "cast_common.ko")
      at /home/acme/git/pahole/dwarf_loader.c:3684
  #11 0x00007ffff7f232df in cus__load_file (cus=0x4142a0, conf=0x4133c0 <conf_load>, filename=0x7fffffffe3f7 "cast_common.ko")
      at /home/acme/git/pahole/dwarves.c:2134
  #12 0x00007ffff7f23e8b in cus__load_files (cus=0x4142a0, conf=0x4133c0 <conf_load>, filenames=0x7fffffffe0f0)
      at /home/acme/git/pahole/dwarves.c:2637
  #13 0x000000000040aec0 in main (argc=2, argv=0x7fffffffe0e8) at /home/acme/git/pahole/pahole.c:3805
  (gdb) fr 1
  #1  0x00007ffff7f1e176 in list_del_init (entry=0x417980) at /home/acme/git/pahole/list.h:165
  165		__list_del(entry->prev, entry->next);
  (gdb) p entry
  $1 = (struct list_head *) 0x417980
  (gdb) p entry->next
  $2 = (struct list_head *) 0x0
  (gdb) p entry->prev
  $3 = (struct list_head *) 0x0
---
 dwarves.c | 1 +
 1 file changed, 1 insertion(+)
dwarves.c +1 -0
index 1ec259f50dbd3778..823a01524a12bb37 100644
@@ -739,6 +739,7 @@ struct cu *cu__new(const char *name, uint8_t addr_size,
 		cu->dfops	= NULL;
 		INIT_LIST_HEAD(&cu->tags);
 		INIT_LIST_HEAD(&cu->tool_list);
+		INIT_LIST_HEAD(&cu->node);
 
 		cu->addr_size = addr_size;
 		cu->extra_dbg_info = 0;
Signature · 1 line
2.45.0
#11 lore raw

PATCH 1/1 dwarf_loader: Add missing cus__add(cus, cu) to cus__merge_and_process_cu()

Just FYI, I added this on top of the previous one initializing cu->node.

Tested with that cast_common.ko and the vmlinux you made available.

- Arnaldo

---

In cus__finalize() if cu__finalize() returns LSK__DELETE, i.e. if the tool processing the cu is done with it, we will assume that it is in the cus list of cu instances, remove it and then delete it.

This was not being done by cus__merge_and_process_cu(), used when merging all DWARF CUs into a single 'struct cu', such as when processing binaries generated by clang using LTO. Add the missing cus__add() to keep cus->nr_entries consistent.

---
 dwarf_loader.c | 1 +
 1 file changed, 1 insertion(+)
dwarf_loader.c +1 -0
index b832c93cc2194eaf..3caf32457f42ec2e 100644
@@ -3452,6 +3452,7 @@ static int cus__merge_and_process_cu(struct cus *cus, struct conf_load *conf,
 			cu->priv = dcu;
 			cu->dfops = &dwarf__ops;
 			cu->language = attr_numeric(cu_die, DW_AT_language);
+			cus__add(cus, cu);
 		}
 
 		Dwarf_Die child;
Signature · 1 line
2.45.0
#12 lore raw
In reply to Arnaldo Carvalho de Melo

Hi Arnaldo,

Quoted 4 lines · Arnaldo Carvalho de Melo
Quoted 1 line

From 6a2b27c0f512619b0e7a769a18a0fb05bb3789a5 Mon Sep 17 00:00:00 2001

From: Arnaldo Carvalho de Melo <[email protected]>
Date: Tue, 18 Jun 2024 10:37:30 -0300
Subject: [PATCH 1/1] core: Initialize cu->node with INIT_LIST_HEAD()

Could a new release be cut for this issue? Several people have been bit by this (including CI systems) and two distributions have talked about backporting this change on top of 1.27 to resolve it:

https://gitlab.archlinux.org/archlinux/packaging/packages/pahole/-/issues/1
https://src.fedoraproject.org/rpms/dwarves/pull-request/4

Cheers,
Nathan