perf probe fails to add events due to "File exists"
Hi all,
My apologies in advance if this has been addressed in newer kernels. Likewise, I realize that this might be a problem in kernel/trace instead of in the perf userland. But I will appreciate any pointer.
I'm trying to use perf record to trace USDT events in OpenJDK (in libjvm.so). See the below entries in .note.stapsdt section.
stapsdt 0x0000002f NT_STAPSDT (SystemTap probe descriptors)
Provider: hotspot
Name: gc__begin
Location: 0x0000000000874144, Base: 0x00000000012014f8, Semaphore: 0x0000000000000000
Arguments: 1@$1
stapsdt 0x0000002f NT_STAPSDT (SystemTap probe descriptors)
Provider: hotspot
Name: gc__begin
Location: 0x00000000008741c2, Base: 0x00000000012014f8, Semaphore: 0x0000000000000000
Arguments: 1@$0
stapsdt 0x00000031 NT_STAPSDT (SystemTap probe descriptors)
Provider: hotspot
Name: gc__begin
Location: 0x0000000000874274, Base: 0x00000000012014f8, Semaphore: 0x0000000000000000
Arguments: 1@%dil
stapsdt 0x0000002f NT_STAPSDT (SystemTap probe descriptors)
Provider: hotspot
Name: gc__begin
Location: 0x0000000000874634, Base: 0x00000000012014f8, Semaphore: 0x0000000000000000
Arguments: 1@$1I tried
1. perf buildid-cache --add libjvm.so first, and then perf probe --add sdt_hotspot:gc__begin, or
2. perf probe -x libjvm.so --add sdt_hotspot:gc__begin.
Both result in
Failed to write event: File exists
Error: Failed to add events.
Here is the verbose log with some removed for brevity.
perf probe --verbose --add sdt_hotspot:gc__begin
probe-definition(0): sdt_hotspot:gc__begin
0 arguments
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
<snip>
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
Parsing probe_events: p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x874144 Group:sdt_hotspot Event:gc__begin probe:p
Parsing probe_events: p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x8741c2 Group:sdt_hotspot Event:gc__begin probe:p
Parsing probe_events: p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x874274 arg1=%di:u8 Group:sdt_hotspot Event:gc__begin probe:p
Parsing probe_events: p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x874634 Group:sdt_hotspot Event:gc__begin probe:p
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
Opening /sys/kernel/tracing//uprobe_events write=1
Opening /sys/kernel/tracing//README write=0
Writing event: p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x874144 Writing event: p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x8741c2 Writing event: p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x874274 arg1=%di:u8 Failed to write event: File exists
Error: Failed to add events. Reason: File exists (Code: -17)
Here is the content of /sys/kernel/tracing/uprobe_events after running perf probe.
p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x0000000000874144 p:sdt_hotspot/gc__begin /home/zixianc/jdk-21.0.9/lib/server/libjvm.so:0x00000000008741c2