list corruption while enabling multi call uprobes via perf
Masami, ideas?
Trying to show a way to use perf tooling to show that libbpf functions statically linked with perf were being used by 'perf test bpf' regression tests and count how many times those functions were used led me to a crash, steps to reproduce:
[root@quaco ~]# ldd ~/bin/perf | grep bpf
[root@quaco ~]# nm ~/bin/perf | grep libbpf_ | head 00000000005dc088 t bpf_object__section_to_libbpf_map_type 00000000005e4101 T libbpf_attach_type_by_name00000000005e3fa7 t libbpf_find_attach_btf_id00000000005e3e7a t libbpf_find_prog_btf_id00000000005e3d09 T libbpf_find_vmlinux_btf_id00000000005e46b1 T libbpf_get_error00000000005e3aae t libbpf_get_type_names0000000000b810a9 b libbpf_initialized00000000005f0119 t libbpf__load_raw_btf00000000005eea9b t libbpf_netlink_open
[root@quaco ~]# set -o vi
[root@quaco ~]# perf probe -x ~/bin/perf libbpf_*
Added new events:
probe_perf:libbpf_perf_print (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_num_possible_cpus (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_get_error (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_attach_type_by_name (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_find_attach_btf_id (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_find_prog_btf_id (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_find_vmlinux_btf_id (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_prog_type_by_name (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_get_type_names (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_print (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_set_print (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_validate_opts (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_dump_errormsg (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_parse_nested (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_parse (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_len (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_data (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_strerror (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_strerror_r (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_filter (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_qdisc (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_class (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_link (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_netlink_open (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_getattr_u32 (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_getattr_u8 (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_data (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf__load_raw_btf (on libbpf_* in /home/acme/bin/perf)
You can now use it in all perf tools, such as:
perf record -e probe_perf:libbpf__load_raw_btf -aR sleep 1
[root@quaco ~]# perf stat -e probe_perf:libbpf* perf test bpf
41: BPF filter :
41.1: Basic BPF filtering :
And it gets stuck here:
[acme@quaco ~]$ uname -a
Linux quaco 5.5.0-rc4+ #2 SMP Thu Jan 2 11:17:21 -03 2020 x86_64 x86_64 x86_64 GNU/Linux [acme@quaco ~]$
[228215.294840] list_add corruption. next->prev should be prev (ffff90786c3c8028), but was ffff90786c3c8568. (next=ffff9079403fb960). [228215.294852] ------------[ cut here ]------------ [228215.294853] kernel BUG at lib/list_debug.c:23! [228215.294860] invalid opcode: 0000 [#1] SMP PTI [228215.294863] CPU: 3 PID: 9890 Comm: perf Not tainted 5.5.0-rc4+ #2 [228215.294864] Hardware name: LENOVO 20L8S2N70H/20L8S2N70H, BIOS N22ET48W (1.25 ) 07/18/2018 [228215.294868] RIP: 0010:__list_add_valid.cold+0xf/0x3f [228215.294870] Code: 39 fe 0f 85 71 00 00 00 48 8b 52 08 48 39 f2 0f 85 56 00 00 00 b8 01 00 00 00 c3 4c 89 c1 48 c7 c7 d0 8f 3a b6 e8 63 9c c4 ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 80 90 3a b6 e8 4c 9c c4 [228215.294871] RSP: 0018:ffffa9ec818efbc0 EFLAGS: 00010246 [228215.294873] RAX: 0000000000000075 RBX: ffff9079403fb960 RCX: 0000000000000000 [228215.294874] RDX: 0000000000000000 RSI: ffff9079464d9ce8 RDI: ffff9079464d9ce8 [228215.294875] RBP: ffff90786c3c8000 R08: 0000000000000001 R09: 0000000000000785 [228215.294876] R10: 000000000002b7d0 R11: 0000000000000003 R12: ffff90793b8d3000 [228215.294877] R13: ffff90793b8d3160 R14: 0000000000000001 R15: ffff90786c3c8028 [228215.294879] FS: 00007fa6f1f376c0(0000) GS:ffff9079464c0000(0000) knlGS:0000000000000000 [228215.294880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [228215.294881] CR2: 00000000004aafe0 CR3: 0000000125692001 CR4: 00000000003626e0 [228215.294882] DR0: 000000000059ae50 DR1: 0000000000000000 DR2: 0000000000000000 [228215.294883] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [228215.294884] Call Trace: [228215.294889] uprobe_perf_open+0x58/0x110 [228215.294892] ? uprobe_perf_close+0xc0/0xc0 [228215.294894] ? perf_prepare_sample+0x630/0x630 [228215.294896] uprobe_perf_multi_call+0x51/0x80 [228215.294898] perf_trace_event_init+0x63/0x290 [228215.294901] perf_trace_init+0x69/0xa0 [228215.294903] perf_tp_event_init+0x1b/0x40 [228215.294905] perf_try_init_event+0x45/0x120 [228215.294908] perf_event_alloc+0x493/0xbf0 [228215.294911] inherit_event.isra.0+0x47/0x250 [228215.294915] inherit_task_group.isra.0.part.0+0x3a/0x100 [228215.294917] perf_event_init_task+0x18a/0x2f0 [228215.294919] copy_process+0x6e1/0x1a70 [228215.294922] _do_fork+0x70/0x390 [228215.294924] __x64_sys_clone+0x81/0xa0 [228215.294927] do_syscall_64+0x5b/0x1b0 [228215.294932] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [228215.294934] RIP: 0033:0x7fa6f25d189f [228215.294936] Code: ed 0f 85 f4 00 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 8d 00 00 00 41 89 c5 85 c0 0f 85 9a 00 00 [228215.294937] RSP: 002b:00007ffe12998c60 EFLAGS: 00000246 ORIG_RAX: 0000000000000038 [228215.294939] RAX: ffffffffffffffda RBX: 00000000009d4b80 RCX: 00007fa6f25d189f [228215.294940] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 [228215.294941] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fa6f1f376c0 [228215.294942] R10: 00007fa6f1f37990 R11: 0000000000000246 R12: 0000000000000000 [228215.294943] R13: 0000000000000000 R14: 0000000000000029 R15: 0000000000000001 [228215.294944] Modules linked in: rpcsec_gss_krb5 sctp cdc_ether usbnet uinput xt_CHECKSUM xt_MASQUERADE tun bridge stp llc rfcomm nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables cmac bnep binfmt_misc vfat fat x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_soc_skl irqbypass iwlmvm typec_displayport snd_soc_sst_ipc snd_soc_sst_dsp elan_i2c snd_hda_ext_core snd_hda_codec_hdmi mac80211 snd_soc_acpi_intel_match crct10dif_pclmul snd_soc_acpi crc32_pclmul snd_soc_core libarc4 snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic intel_cstate snd_compress iwlwifi ac97_bus snd_pcm_dmaengine snd_hda_intel snd_usb_audio snd_intel_dspcfg snd_hda_codec [228215.294972] snd_hda_core uvcvideo snd_usbmidi_lib snd_rawmidi snd_hwdep snd_seq snd_seq_device btusb videobuf2_vmalloc snd_pcm btrtl videobuf2_memops btbcm videobuf2_v4l2 btintel mei_hdcp videobuf2_common mei_wdt bluetooth intel_uncore videodev intel_rapl_msr thinkpad_acpi ucsi_acpi snd_timer mei_me processor_thermal_device cfg80211 ledtrig_audio intel_lpss_pci intel_rapl_perf intel_wmi_thunderbolt iTCO_wdt wmi_bmof joydev typec_ucsi intel_rapl_common iTCO_vendor_support pcspkr thunderbolt ecdh_generic snd mc intel_xhci_usb_role_switch intel_lpss mei intel_soc_dts_iosf i2c_i801 intel_pch_thermal idma64 roles typec soundcore ecc rfkill int3403_thermal int340x_thermal_zone int3400_thermal acpi_thermal_rel acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc i915 cec rc_core i2c_algo_bit drm_kms_helper uas nvme r8152 drm e1000e usb_storage crc32c_intel serio_raw nvme_core mii wmi video fuse [228215.295002] ---[ end trace ff70b427cb0ab835 ]--- [228215.295004] RIP: 0010:__list_add_valid.cold+0xf/0x3f [228215.295006] Code: 39 fe 0f 85 71 00 00 00 48 8b 52 08 48 39 f2 0f 85 56 00 00 00 b8 01 00 00 00 c3 4c 89 c1 48 c7 c7 d0 8f 3a b6 e8 63 9c c4 ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 80 90 3a b6 e8 4c 9c c4 [228215.295007] RSP: 0018:ffffa9ec818efbc0 EFLAGS: 00010246 [228215.295008] RAX: 0000000000000075 RBX: ffff9079403fb960 RCX: 0000000000000000 [228215.295009] RDX: 0000000000000000 RSI: ffff9079464d9ce8 RDI: ffff9079464d9ce8 [228215.295010] RBP: ffff90786c3c8000 R08: 0000000000000001 R09: 0000000000000785 [228215.295011] R10: 000000000002b7d0 R11: 0000000000000003 R12: ffff90793b8d3000 [228215.295012] R13: ffff90793b8d3160 R14: 0000000000000001 R15: ffff90786c3c8028 [228215.295013] FS: 00007fa6f1f376c0(0000) GS:ffff9079464c0000(0000) knlGS:0000000000000000 [228215.295014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [228215.295015] CR2: 00000000004aafe0 CR3: 0000000125692001 CR4: 00000000003626e0 [228215.295016] DR0: 000000000059ae50 DR1: 0000000000000000 DR2: 0000000000000000 [228215.295017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[acme@quaco ~]$
- Arnaldo
Hi Arnaldo,
Quoted 5 lines · On Wed, 8 Jan 2020 14:16:11 -0300 Arnaldo Carvalho de Melo <[email protected]>…
Masami, ideas?
Trying to show a way to use perf tooling to show that libbpf functions statically linked with perf were being used by 'perf test bpf' regression tests and count how many times those functions were used led me to a crash, steps to reproduce:
Thanks for reporting!
Quoted 97 lines
[root@quaco ~]# ldd ~/bin/perf | grep bpf
[root@quaco ~]# nm ~/bin/perf | greplibbpf_| head00000000005dc088tbpf_object__section_to_libbpf_map_type00000000005e4101Tlibbpf_attach_type_by_name00000000005e3fa7tlibbpf_find_attach_btf_id00000000005e3e7atlibbpf_find_prog_btf_id00000000005e3d09Tlibbpf_find_vmlinux_btf_id00000000005e46b1Tlibbpf_get_error00000000005e3aaetlibbpf_get_type_names0000000000b810a9blibbpf_initialized00000000005f0119tlibbpf__load_raw_btf00000000005eea9btlibbpf_netlink_open
[root@quaco ~]# set -o vi
[root@quaco ~]# perf probe -x ~/bin/perf libbpf_*
Added new events:
probe_perf:libbpf_perf_print (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_num_possible_cpus (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_get_error (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_attach_type_by_name (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_find_attach_btf_id (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_find_prog_btf_id (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_find_vmlinux_btf_id (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_prog_type_by_name (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_get_type_names (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_print (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_set_print (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_validate_opts (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_dump_errormsg (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_parse_nested (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_parse (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_len (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_data (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_strerror (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_strerror_r (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nl_get_filter (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nl_get_qdisc (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nl_get_class (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nl_get_link (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_netlink_open (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_getattr_u32 (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_getattr_u8 (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf_nla_data (on libbpf_* in/home/acme/bin/perf) probe_perf:libbpf__load_raw_btf (on libbpf_* in/home/acme/bin/perf)You can now use it in all perf tools, such as:
perf record -e probe_perf:libbpf__load_raw_btf -aR sleep 1
[root@quaco ~]# perf stat -e probe_perf:libbpf* perf test bpf
41: BPF filter :
41.1: Basic BPF filtering :And it gets stuck here:
[acme@quaco ~]$ uname -a
Linux quaco 5.5.0-rc4+ #2 SMP Thu Jan 2 11:17:21 -03 2020x86_64x86_64x86_64GNU/Linux [acme@quaco ~]$[228215.294840] list_add corruption. next->prev should be prev (ffff90786c3c8028), but was ffff90786c3c8568. (next=ffff9079403fb960). [228215.294852] ------------[ cut here ]------------ [228215.294853] kernel BUG at lib/list_debug.c:23! [228215.294860] invalid opcode: 0000 [#1] SMP PTI [228215.294863] CPU: 3 PID: 9890 Comm: perf Not tainted 5.5.0-rc4+ #2 [228215.294864] Hardware name: LENOVO 20L8S2N70H/20L8S2N70H, BIOS N22ET48W (1.25 ) 07/18/2018 [228215.294868] RIP: 0010:__list_add_valid.cold+0xf/0x3f [228215.294870] Code: 39 fe 0f 85 71 00 00 00 48 8b 52 08 48 39 f2 0f 85 56 00 00 00 b8 01 00 00 00 c3 4c 89 c1 48 c7 c7 d0 8f 3a b6 e8 63 9c c4 ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 80 90 3a b6 e8 4c 9c c4 [228215.294871] RSP: 0018:ffffa9ec818efbc0 EFLAGS: 00010246 [228215.294873] RAX: 0000000000000075 RBX: ffff9079403fb960 RCX: 0000000000000000 [228215.294874] RDX: 0000000000000000 RSI: ffff9079464d9ce8 RDI: ffff9079464d9ce8 [228215.294875] RBP: ffff90786c3c8000 R08: 0000000000000001 R09: 0000000000000785 [228215.294876] R10: 000000000002b7d0 R11: 0000000000000003 R12: ffff90793b8d3000 [228215.294877] R13: ffff90793b8d3160 R14: 0000000000000001 R15: ffff90786c3c8028 [228215.294879] FS: 00007fa6f1f376c0(0000) GS:ffff9079464c0000(0000) knlGS:0000000000000000 [228215.294880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [228215.294881] CR2: 00000000004aafe0 CR3: 0000000125692001 CR4: 00000000003626e0 [228215.294882] DR0: 000000000059ae50 DR1: 0000000000000000 DR2: 0000000000000000 [228215.294883] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [228215.294884] Call Trace: [228215.294889] uprobe_perf_open+0x58/0x110 [228215.294892] ? uprobe_perf_close+0xc0/0xc0 [228215.294894] ? perf_prepare_sample+0x630/0x630 [228215.294896] uprobe_perf_multi_call+0x51/0x80 [228215.294898] perf_trace_event_init+0x63/0x290 [228215.294901] perf_trace_init+0x69/0xa0 [228215.294903] perf_tp_event_init+0x1b/0x40 [228215.294905] perf_try_init_event+0x45/0x120 [228215.294908] perf_event_alloc+0x493/0xbf0 [228215.294911] inherit_event.isra.0+0x47/0x250 [228215.294915] inherit_task_group.isra.0.part.0+0x3a/0x100 [228215.294917] perf_event_init_task+0x18a/0x2f0 [228215.294919] copy_process+0x6e1/0x1a70 [228215.294922] _do_fork+0x70/0x390 [228215.294924] __x64_sys_clone+0x81/0xa0 [228215.294927] do_syscall_64+0x5b/0x1b0 [228215.294932] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [228215.294934] RIP: 0033:0x7fa6f25d189f
Quoted 8 lines
[228215.294936] Code: ed 0f 85 f4 00 00 00 64 4c 8b 0c 25 10 00 00 00 45 31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 8d 00 00 00 41 89 c5 85 c0 0f 85 9a 00 00 [228215.294937] RSP: 002b:00007ffe12998c60 EFLAGS: 00000246 ORIG_RAX: 0000000000000038 [228215.294939] RAX: ffffffffffffffda RBX: 00000000009d4b80 RCX: 00007fa6f25d189f [228215.294940] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 [228215.294941] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fa6f1f376c0 [228215.294942] R10: 00007fa6f1f37990 R11: 0000000000000246 R12: 0000000000000000 [228215.294943] R13: 0000000000000000 R14: 0000000000000029 R15: 0000000000000001 [228215.294944] Modules linked in: rpcsec_gss_krb5 sctp cdc_ether usbnet uinput xt_CHECKSUM xt_MASQUERADE tun bridge stp llc rfcomm nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables cmac bnep binfmt_misc vfat fat x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_soc_skl irqbypass iwlmvm typec_displayport snd_soc_sst_ipc snd_soc_sst_dsp elan_i2c snd_hda_ext_core snd_hda_codec_hdmi mac80211 snd_soc_acpi_intel_match crct10dif_pclmul snd_soc_acpi crc32_pclmul snd_soc_core libarc4 snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic intel_cstate snd_compress iwlwifi ac97_bus snd_pcm_dmaengine snd_hda_intel snd_usb_audio snd_intel_dspcfg sn
d_hda_codec
Quoted 15 lines
[228215.294972] snd_hda_core uvcvideo snd_usbmidi_lib snd_rawmidi snd_hwdep snd_seq snd_seq_device btusb videobuf2_vmalloc snd_pcm btrtl videobuf2_memops btbcm videobuf2_v4l2 btintel mei_hdcp videobuf2_common mei_wdt bluetooth intel_uncore videodev intel_rapl_msr thinkpad_acpi ucsi_acpi snd_timer mei_me processor_thermal_device cfg80211 ledtrig_audio intel_lpss_pci intel_rapl_perf intel_wmi_thunderbolt iTCO_wdt wmi_bmof joydev typec_ucsi intel_rapl_common iTCO_vendor_support pcspkr thunderbolt ecdh_generic snd mc intel_xhci_usb_role_switch intel_lpss mei intel_soc_dts_iosf i2c_i801 intel_pch_thermal idma64 roles typec soundcore ecc rfkill int3403_thermal int340x_thermal_zone int3400_thermal acpi_thermal_rel acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc i915 cec rc_core i2c_algo_bit drm_kms_helper uas nvme r8152 drm e1000e usb_storage crc32c_intel serio_raw nvme_core mii wmi video fuse [228215.295002] ---[ end trace ff70b427cb0ab835 ]--- [228215.295004] RIP: 0010:__list_add_valid.cold+0xf/0x3f [228215.295006] Code: 39 fe 0f 85 71 00 00 00 48 8b 52 08 48 39 f2 0f 85 56 00 00 00 b8 01 00 00 00 c3 4c 89 c1 48 c7 c7 d0 8f 3a b6 e8 63 9c c4 ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 80 90 3a b6 e8 4c 9c c4 [228215.295007] RSP: 0018:ffffa9ec818efbc0 EFLAGS: 00010246 [228215.295008] RAX: 0000000000000075 RBX: ffff9079403fb960 RCX: 0000000000000000 [228215.295009] RDX: 0000000000000000 RSI: ffff9079464d9ce8 RDI: ffff9079464d9ce8 [228215.295010] RBP: ffff90786c3c8000 R08: 0000000000000001 R09: 0000000000000785 [228215.295011] R10: 000000000002b7d0 R11: 0000000000000003 R12: ffff90793b8d3000 [228215.295012] R13: ffff90793b8d3160 R14: 0000000000000001 R15: ffff90786c3c8028 [228215.295013] FS: 00007fa6f1f376c0(0000) GS:ffff9079464c0000(0000) knlGS:0000000000000000 [228215.295014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [228215.295015] CR2: 00000000004aafe0 CR3: 0000000125692001 CR4: 00000000003626e0 [228215.295016] DR0: 000000000059ae50 DR1: 0000000000000000 DR2: 0000000000000000 [228215.295017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Quoted 1 line
[acme@quaco ~]$
Hmm, this seems that the event->hw.tp_list is not initialized when removing from the list in uprobe_perf_close().
Could you try attached patch?
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Hi,
Quoted 2 lines · On Thu, 9 Jan 2020 11:10:56 +0900 Masami Hiramatsu <[email protected]> wrote:
Hmm, this seems that the
event->hw.tp_listis not initialized when removing from the list inuprobe_perf_close().
Oops, that's wrong. Of course my patch can ease (avoid kernel panic) the issue, but not fixing the root cause.
The root cause is that the uprobe event tries to open multiple probes with one perf_event. So the perf_event is reused on different probes.
In the reported case, if we remove the multiple probe event before perf-stat, no problem happens.
I'll try to fix it.
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Em Thu, Jan 09, 2020 at 06:33:56PM +0900, Masami Hiramatsu escreveu:
Quoted 13 lines
Hi,
Quoted 2 lines · On Thu, 9 Jan 2020 11:10:56 +0900 Masami Hiramatsu <[email protected]> wrote:
Hmm, this seems that the
event->hw.tp_listis not initialized when removing from the list inuprobe_perf_close().Oops, that's wrong. Of course my patch can ease (avoid kernel panic) the issue, but not fixing the root cause.
The root cause is that the uprobe event tries to open multiple probes with oneperf_event. So theperf_eventis reused on different probes.
Quoted 4 lines
In the reported case, if we remove the multiple probe event before perf-stat, no problem happens.
I'll try to fix it.
Ok!
For reference, I rebooted it with a fedora kernel, 5.3ish and it seems to work:
[root@quaco ~]# uname -a
Linux quaco 5.3.18-200.fc30.x86_64 #1 SMP Wed Dec 18 20:26:50 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux [root@quaco ~]# perf -v
perf version 5.5.rc3.g6c4798d3f08b
[root@quaco ~]# perf probe -x ~/bin/perf libbpf_*
Added new events:
probe_perf:libbpf_perf_print (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_num_possible_cpus (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_get_error (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_attach_type_by_name (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_find_attach_btf_id (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_find_prog_btf_id (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_find_vmlinux_btf_id (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_prog_type_by_name (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_get_type_names (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_print (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_set_print (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_validate_opts (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_dump_errormsg (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_parse_nested (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_parse (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_len (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_data (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_strerror (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_strerror_r (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_filter (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_qdisc (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_class (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nl_get_link (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_netlink_open (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_getattr_u32 (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_getattr_u8 (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf_nla_data_1 (on libbpf_* in /home/acme/bin/perf) probe_perf:libbpf__load_raw_btf (on libbpf_* in /home/acme/bin/perf)
You can now use it in all perf tools, such as:
perf record -e probe_perf:libbpf__load_raw_btf -aR sleep 1
[root@quaco ~]# perf stat -e probe_perf:libbpf* perf test bpf
41: BPF filter : 41.1: Basic BPF filtering : Ok 41.2: BPF pinning : Ok 41.3: BPF prologue generation : Ok 41.4: BPF relocation checker : Ok
Performance counter stats for 'perf test bpf':
8 probe_perf:libbpf__load_raw_btf
0 probe_perf:libbpf_nla_data_1
0 probe_perf:libbpf_nla_getattr_u8
0 probe_perf:libbpf_nla_getattr_u32
0 probe_perf:libbpf_netlink_open
0 probe_perf:libbpf_nl_get_link
0 probe_perf:libbpf_nl_get_class
0 probe_perf:libbpf_nl_get_qdisc
0 probe_perf:libbpf_nl_get_filter
0 probe_perf:libbpf_strerror_r
0 probe_perf:libbpf_strerror
0 probe_perf:libbpf_nla_data
0 probe_perf:libbpf_nla_len
0 probe_perf:libbpf_nla_parse
0 probe_perf:libbpf_nla_parse_nested
0 probe_perf:libbpf_nla_dump_errormsg
4 probe_perf:libbpf_validate_opts
4 probe_perf:libbpf_set_print
200 probe_perf:libbpf_print
3 probe_perf:libbpf_get_type_names
3 probe_perf:libbpf_prog_type_by_name
0 probe_perf:libbpf_find_vmlinux_btf_id
0 probe_perf:libbpf_find_prog_btf_id
0 probe_perf:libbpf_find_attach_btf_id
0 probe_perf:libbpf_attach_type_by_name
0 probe_perf:libbpf_get_error
0 probe_perf:libbpf_num_possible_cpus
200 probe_perf:libbpf_perf_print 9.239808031 seconds time elapsed
5.689898000 seconds user
3.156735000 seconds sys[root@quaco ~]#
[root@quaco ~]# perf stat -e probe_perf:libbpf* perf test llvm
39: LLVM search and compile : 39.1: Basic BPF llvm compile : Ok 39.2: kbuild searching : Ok 39.3: Compile source for BPF prologue generation : Ok 39.4: Compile source for BPF relocation : Ok
Performance counter stats for 'perf test llvm':
6 probe_perf:libbpf__load_raw_btf
0 probe_perf:libbpf_nla_data_1
0 probe_perf:libbpf_nla_getattr_u8
0 probe_perf:libbpf_nla_getattr_u32
0 probe_perf:libbpf_netlink_open
0 probe_perf:libbpf_nl_get_link
0 probe_perf:libbpf_nl_get_class
0 probe_perf:libbpf_nl_get_qdisc
0 probe_perf:libbpf_nl_get_filter
0 probe_perf:libbpf_strerror_r
0 probe_perf:libbpf_strerror
0 probe_perf:libbpf_nla_data
0 probe_perf:libbpf_nla_len
0 probe_perf:libbpf_nla_parse
0 probe_perf:libbpf_nla_parse_nested
0 probe_perf:libbpf_nla_dump_errormsg
3 probe_perf:libbpf_validate_opts
0 probe_perf:libbpf_set_print
134 probe_perf:libbpf_print
3 probe_perf:libbpf_get_type_names
3 probe_perf:libbpf_prog_type_by_name
0 probe_perf:libbpf_find_vmlinux_btf_id
0 probe_perf:libbpf_find_prog_btf_id
0 probe_perf:libbpf_find_attach_btf_id
0 probe_perf:libbpf_attach_type_by_name
3 probe_perf:libbpf_get_error
0 probe_perf:libbpf_num_possible_cpus
0 probe_perf:libbpf_perf_print 5.016205398 seconds time elapsed
2.713370000 seconds user
2.398449000 seconds sys[root@quaco ~]#
Then another pony to ask for, more than 128 probe points, please:
[root@quaco ~]# perf probe -x ~/bin/perf -F bpf*__* | wc -l 171
[root@quaco ~]# perf probe -x ~/bin/perf -F bpf*__* | head bpf__apply_obj_configbpf__clearbpf__config_objbpf__foreach_eventbpf__gen_prologuebpf__loadbpf__prepare_loadbpf__prepare_load_bufferbpf__probebpf__setup_output_event
[root@quaco ~]# perf probe -x ~/bin/perf bpf*__* Too many( > 128) probe point found.
Too many functions matched in /home/acme/bin/perf
Error: Failed to add events.
[root@quaco ~]#
Lets artificially reduce this:
[root@quaco ~]# perf probe -x ~/bin/perf -F bpf_[op]*__* | head -5 bpf_obj_config__map_funcsbpf_object__add_mapbpf_object__add_programbpf_object__btfbpf_object__btf_fd
[root@quaco ~]# perf probe -x ~/bin/perf -F bpf_[op]*__* | tail -5 bpf_program__sizebpf_program__titlebpf_program__unloadbpf_program__unpinbpf_program__unpin_instance
[root@quaco ~]# perf probe -x ~/bin/perf -F bpf_[op]*__* | tail -5
Now it works, scroll down harder to see the continuation...
[root@quaco ~]# perf probe -x ~/bin/perf bpf_[op]*__*
Added new events:
probe_perf:bpf_output__fprintf (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_output__printer (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__bpil_offs_to_addr (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__bpil_addr_to_offs (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__get_prog_info_linear (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__attach_trace (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__attach_raw_tracepoint (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__attach_tracepoint (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__attach_uprobe (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__attach_kprobe (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__attach_perf_event (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__find_map_by_offset (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__find_map_fd_by_name (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__find_map_by_name (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_expected_attach_type (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__get_expected_attach_type (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_tracing (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_tracing (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_perf_event (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_perf_event (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_xdp (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_xdp (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_raw_tracepoint (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_raw_tracepoint (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_tracepoint (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_tracepoint (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_sched_act (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_sched_act (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_sched_cls (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_sched_cls (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_kprobe (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_kprobe (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_socket_filter (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_socket_filter (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_type (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_type (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__get_type (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__nth_fd (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_prep (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__size (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__fd (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__title (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_ifindex (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__priv (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__set_priv (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__prev (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__next (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__priv (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__set_priv (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__btf_fd (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__btf (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__kversion (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__name (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__next (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__close (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__pin (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__unpin_programs (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__pin_programs (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__unpin_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__pin_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__unpin (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__pin (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__unpin_instance (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__pin_instance (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__load (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__load_xattr (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__unload (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__open_buffer (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__open_mem (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__open_file (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__open (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__open_xattr (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__load_progs (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__is_function_storage (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__load (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__collect_reloc (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__relocate (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__relocate (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__reloc_text (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__relocate_core (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__create_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__populate_internal_map (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__reuse_map (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__probe_caps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__probe_array_mmap (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__probe_btf_datasec (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__probe_btf_func (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__probe_global_data (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__probe_name (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__collect_reloc (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__record_reloc (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__section_to_libbpf_map_type (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__shndx_is_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__shndx_is_data (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__find_program_by_title (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__find_prog_by_idx (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__elf_collect (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__sanitize_and_load_btf (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_btf (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__is_btf_mandatory (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__sanitize_btf_ext (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__sanitize_btf (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_user_btf_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_user_btf_map (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_user_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_global_data_maps (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_internal_map (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__add_map (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__variable_offset (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__section_size (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_kversion (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_license (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__check_endianness (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__elf_init (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__elf_finish (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__new (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__init_prog_names (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_object__add_program (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__init (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__exit (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_program__unload (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_prog_linfo__lfind (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_prog_linfo__lfind_addr_func (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_prog_linfo__new (on bpf_[op]*__* in /home/acme/bin/perf) probe_perf:bpf_prog_linfo__free (on bpf_[op]*__* in /home/acme/bin/perf)
You can now use it in all perf tools, such as:
perf record -e probe_perf:bpf_prog_linfo__free -aR sleep 1
[root@quaco ~]#
It works, I think I'll add some option to 'perf stat' to suppress showing zeroed counters...
[root@quaco ~]# perf stat -e probe_perf:bpf* perf trace -e ~acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c -e openat --max-events=4
0.000 ( 0.065 ms): gpm/1040 openat(dfd: CWD, filename: "/dev/tty0") = 4
2000.487 ( 0.096 ms): gpm/1040 openat(dfd: CWD, filename: "/dev/tty0") = 4
3842.601 ( 0.054 ms): pickup/1440 openat(dfd: CWD, filename: "maildrop", flags: RDONLY|CLOEXEC|DIRECTORY|NONBLOCK) = 16
4001.040 ( 0.078 ms): gpm/1040 openat(dfd: CWD, filename: "/dev/tty0") = 4
Performance counter stats for 'perf trace -e /home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c -e openat --max-events=4':
0 probe_perf:bpf_prog_linfo__free
0 probe_perf:bpf_prog_linfo__new
0 probe_perf:bpf_prog_linfo__lfind_addr_func
0 probe_perf:bpf_prog_linfo__lfind
18 probe_perf:bpf_program__unload
9 probe_perf:bpf_program__exit
9 probe_perf:bpf_program__init
9 probe_perf:bpf_object__add_program
1 probe_perf:bpf_object__init_prog_names
1 probe_perf:bpf_object__new
2 probe_perf:bpf_object__elf_finish
1 probe_perf:bpf_object__elf_init
1 probe_perf:bpf_object__check_endianness
1 probe_perf:bpf_object__init_license
1 probe_perf:bpf_object__init_kversion
0 probe_perf:bpf_object__section_size
0 probe_perf:bpf_object__variable_offset
6 probe_perf:bpf_object__add_map
0 probe_perf:bpf_object__init_internal_map
1 probe_perf:bpf_object__init_global_data_maps
1 probe_perf:bpf_object__init_user_maps
0 probe_perf:bpf_object__init_user_btf_map
1 probe_perf:bpf_object__init_user_btf_maps
1 probe_perf:bpf_object__init_maps
0 probe_perf:bpf_object__sanitize_btf
0 probe_perf:bpf_object__sanitize_btf_ext
1 probe_perf:bpf_object__is_btf_mandatory
1 probe_perf:bpf_object__init_btf
1 probe_perf:bpf_object__sanitize_and_load_btf
1 probe_perf:bpf_object__elf_collect
8 probe_perf:bpf_object__find_prog_by_idx
5 probe_perf:bpf_object__find_program_by_title
0 probe_perf:bpf_object__shndx_is_data
17 probe_perf:bpf_object__shndx_is_maps
17 probe_perf:bpf_object__section_to_libbpf_map_type
17 probe_perf:bpf_program__record_reloc
8 probe_perf:bpf_program__collect_reloc
1 probe_perf:bpf_object__probe_name
1 probe_perf:bpf_object__probe_global_data
1 probe_perf:bpf_object__probe_btf_func
1 probe_perf:bpf_object__probe_btf_datasec
1 probe_perf:bpf_object__probe_array_mmap
1 probe_perf:bpf_object__probe_caps
0 probe_perf:bpf_object__reuse_map
0 probe_perf:bpf_object__populate_internal_map
1 probe_perf:bpf_object__create_maps
0 probe_perf:bpf_object__relocate_core
0 probe_perf:bpf_program__reloc_text
9 probe_perf:bpf_program__relocate
1 probe_perf:bpf_object__relocate
1 probe_perf:bpf_object__collect_reloc
9 probe_perf:bpf_program__load
74 probe_perf:bpf_program__is_function_storage
1 probe_perf:bpf_object__load_progs
0 probe_perf:bpf_object__open_xattr
0 probe_perf:bpf_object__open
0 probe_perf:bpf_object__open_file
1 probe_perf:bpf_object__open_mem
1 probe_perf:bpf_object__open_buffer
1 probe_perf:bpf_object__unload
1 probe_perf:bpf_object__load_xattr
1 probe_perf:bpf_object__load
0 probe_perf:bpf_program__pin_instance
0 probe_perf:bpf_program__unpin_instance
0 probe_perf:bpf_program__pin
0 probe_perf:bpf_program__unpin
0 probe_perf:bpf_object__pin_maps
0 probe_perf:bpf_object__unpin_maps
0 probe_perf:bpf_object__pin_programs
0 probe_perf:bpf_object__unpin_programs
0 probe_perf:bpf_object__pin
1 probe_perf:bpf_object__close
17 probe_perf:bpf_object__next
0 probe_perf:bpf_object__name
0 probe_perf:bpf_object__kversion
0 probe_perf:bpf_object__btf
0 probe_perf:bpf_object__btf_fd
0 probe_perf:bpf_object__set_priv
0 probe_perf:bpf_object__priv
71 probe_perf:bpf_program__next
0 probe_perf:bpf_program__prev
9 probe_perf:bpf_program__set_priv
27 probe_perf:bpf_program__priv
0 probe_perf:bpf_program__set_ifindex
9 probe_perf:bpf_program__title
11 probe_perf:bpf_program__fd
0 probe_perf:bpf_program__size
0 probe_perf:bpf_program__set_prep
11 probe_perf:bpf_program__nth_fd
0 probe_perf:bpf_program__get_type
9 probe_perf:bpf_program__set_type
0 probe_perf:bpf_program__is_type
0 probe_perf:bpf_program__set_socket_filter
0 probe_perf:bpf_program__is_socket_filter
0 probe_perf:bpf_program__set_kprobe
0 probe_perf:bpf_program__is_kprobe
0 probe_perf:bpf_program__set_sched_cls
0 probe_perf:bpf_program__is_sched_cls
0 probe_perf:bpf_program__set_sched_act
0 probe_perf:bpf_program__is_sched_act
9 probe_perf:bpf_program__set_tracepoint
0 probe_perf:bpf_program__is_tracepoint
0 probe_perf:bpf_program__set_raw_tracepoint
0 probe_perf:bpf_program__is_raw_tracepoint
0 probe_perf:bpf_program__set_xdp
0 probe_perf:bpf_program__is_xdp
0 probe_perf:bpf_program__set_perf_event
0 probe_perf:bpf_program__is_perf_event
0 probe_perf:bpf_program__set_tracing
0 probe_perf:bpf_program__is_tracing
0 probe_perf:bpf_program__get_expected_attach_type
0 probe_perf:bpf_program__set_expected_attach_type
4 probe_perf:bpf_object__find_map_by_name
0 probe_perf:bpf_object__find_map_fd_by_name
0 probe_perf:bpf_object__find_map_by_offset
0 probe_perf:bpf_program__attach_perf_event
0 probe_perf:bpf_program__attach_kprobe
0 probe_perf:bpf_program__attach_uprobe
0 probe_perf:bpf_program__attach_tracepoint
0 probe_perf:bpf_program__attach_raw_tracepoint
0 probe_perf:bpf_program__attach_trace
0 probe_perf:bpf_program__get_prog_info_linear
0 probe_perf:bpf_program__bpil_addr_to_offs
0 probe_perf:bpf_program__bpil_offs_to_addr
0 probe_perf:bpf_output__printer
0 probe_perf:bpf_output__fprintf 7.345027900 seconds time elapsed
0.841158000 seconds user
1.013929000 seconds sys[root@quaco ~]#
Quoted 17 lines · On Thu, 9 Jan 2020 10:38:46 -0300 Arnaldo Carvalho de Melo <[email protected]>…
Em Thu, Jan 09, 2020 at 06:33:56PM +0900, Masami Hiramatsu escreveu:
Quoted 16 lines
Hi,
Quoted 2 lines · On Thu, 9 Jan 2020 11:10:56 +0900 Masami Hiramatsu <[email protected]> wrote:
Hmm, this seems that the
event->hw.tp_listis not initialized when removing from the list inuprobe_perf_close().Oops, that's wrong. Of course my patch can ease (avoid kernel panic) the issue, but not fixing the root cause.
The root cause is that the uprobe event tries to open multiple probes with oneperf_event. So theperf_eventis reused on different probes.In the reported case, if we remove the multiple probe event before perf-stat, no problem happens.
Quoted 6 lines
Quoted 1 line
I'll try to fix it.
Ok!
For reference, I rebooted it with a fedora kernel, 5.3ish and it seems to work:
Yes, since it was my commit 60d53e2c3b75 ("tracing/probe: Split trace_event related data from trace_probe") broke list operation...
I'll send a fix soon.
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>