Using a new perf tool against an older kernel
I ran into a few strange issues today using a new perf tool (3.0-rcX) against a 2.6.38 based kernel. Basic things seem to work:
# perf record -aR -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.443 MB perf.data (~19334 samples) ]
But:
# perf record -agR -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.558 MB perf.data (~68077 samples) ] Can't find id 9's machine Found 1 unknown events!
Is this an older tool processing a perf.data file generated by a more recent tool?
If that is not the case, consider reporting to [email protected].
I couldn't tell if this is kernel/user space incompatibility or tighter error checking in the new perf tool. Before I start digging further, thought I'd ask if this combination is supported. The error message above doesn't ask me:
Are you using an old kernel?
-Arun
Quoted 22 lines · Arun Sharma
I ran into a few strange issues today using a new perf tool (3.0-rcX) against a 2.6.38 based kernel. Basic things seem to work:
# perf record -aR -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.443 MB perf.data (~19334 samples) ]But:
# perf record -agR -- sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.558 MB perf.data (~68077 samples) ] Can't find id 9's machine Found 1 unknown events!Is this an older tool processing a perf.data file generated by a more recent tool?
If that is not the case, consider reporting to
[email protected].
Quoted 6 lines
I couldn't tell if this is kernel/user space incompatibility or tighter error checking in the new perf tool. Before I start digging further, thought I'd ask if this combination is supported. The error message above doesn't ask me:
Are you using an old kernel?
I have not seen issues like this using newer perf userspace against older kernels. For example, my laptop was running Fedora 14 (2.6.35) and now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g., testing patches).
David
Quoted 8 lines
-Arun
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · David Ahern
I have not seen issues like this using newer perf userspace against older kernels. For example, my laptop was running Fedora 14 (2.6.35) and now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g., testing patches).
I narrowed it down to PERF_SAMPLE_RAW:
perf record -ag -- sleep 1
is fine, but:
perf record -agR -- sleep 1
fails for me most of the time. The reason I needed to use the -R in the first place is that "perf script" fails on older kernels with:
Samples do not contain timestamps.
With the newer perf, I don't get errors, but the timestamp field is invalid. So I need to use the -R flag to get valid timestamps + stacktraces out of "perf script".
-Arun
Quoted 15 lines · Arun Sharma
Quoted 4 lines · David Ahern
I have not seen issues like this using newer perf userspace against older kernels. For example, my laptop was running Fedora 14 (2.6.35) and now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g., testing patches).
I narrowed it down to
PERF_SAMPLE_RAW:perf record -ag -- sleep 1
is fine, but:
perf record -agR -- sleep 1
Quoted 8 lines
fails for me most of the time. The reason I needed to use the -R in the first place is that "perf script" fails on older kernels with:
Samples do not contain timestamps.
With the newer perf, I don't get errors, but the timestamp field is invalid. So I need to use the -R flag to get valid timestamps + stacktraces out of "perf script".
That should have been fixed.
And -T on record gets the timestamps.
David
Quoted 2 lines
-Arun
Quoted 27 lines · David Ahern
Quoted 15 lines · Arun Sharma
Quoted 4 lines · David Ahern
I have not seen issues like this using newer perf userspace against older kernels. For example, my laptop was running Fedora 14 (2.6.35) and now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g., testing patches).
I narrowed it down to
PERF_SAMPLE_RAW:perf record -ag -- sleep 1
is fine, but:
perf record -agR -- sleep 1
Quoted 8 lines
fails for me most of the time. The reason I needed to use the -R in the first place is that "perf script" fails on older kernels with:
Samples do not contain timestamps.
With the newer perf, I don't get errors, but the timestamp field is invalid. So I need to use the -R flag to get valid timestamps + stacktraces out of "perf script".
Quoted 5 lines
That should have been fixed.
And -T on record gets the timestamps.
David
Right, it would be nice to suggest that from perf script when timestamps are not recorded.
Quoted 28 lines · Frederic Weisbecker
Quoted 27 lines · David Ahern
Quoted 23 lines · Arun Sharma
Quoted 4 lines · David Ahern
I have not seen issues like this using newer perf userspace against older kernels. For example, my laptop was running Fedora 14 (2.6.35) and now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g., testing patches).
I narrowed it down to
PERF_SAMPLE_RAW:perf record -ag -- sleep 1
is fine, but:
perf record -agR -- sleep 1
fails for me most of the time. The reason I needed to use the -R in the first place is that "perf script" fails on older kernels with:
Samples do not contain timestamps.
With the newer perf, I don't get errors, but the timestamp field is invalid. So I need to use the -R flag to get valid timestamps + stacktraces out of "perf script".
Quoted 8 lines
Quoted 5 lines
That should have been fixed.
And -T on record gets the timestamps.
David
Right, it would be nice to suggest that from perf script when timestamps are not recorded.
Timestamps are enabled by default, but that output option is removed if the samples do not have timestamps. That message is generated if the user requests timestamps (-f time) in the perf-script output and the samples do not have timestamps, but Arun did not request that. Arun's mileage with perf-3.0 definitely varies from what I've seen.
David
Quoted 15 lines · Arun Sharma
Quoted 4 lines · David Ahern
I have not seen issues like this using newer perf userspace against older kernels. For example, my laptop was running Fedora 14 (2.6.35) and now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g., testing patches).
I narrowed it down to
PERF_SAMPLE_RAW:perf record -ag -- sleep 1
is fine, but:
perf record -agR -- sleep 1
Quoted 8 lines
fails for me most of the time. The reason I needed to use the -R in the first place is that "perf script" fails on older kernels with:
Samples do not contain timestamps.
With the newer perf, I don't get errors, but the timestamp field is invalid. So I need to use the -R flag to get valid timestamps + stacktraces out of "perf script".
I'm confused, you first said it happens with new tools on older kernel.
Can you tell us which combination of kernel/user raises the error? and which error.
Thanks!
Quoted 6 lines · Frederic Weisbecker
I'm confused, you first said it happens with new tools on older kernel.
Can you tell us which combination of kernel/user raises the error? and which error.
perf-3.0 + 2.6.38
perf record -agR gives:
Can't find id 9's machine
Found 1 unknown events!
perf-3.0 + 2.6.32
perf record -ag
perf script
samples have bogus timestamps
perf-3.0 + 2.6.32
perf record -agR
perf script
error: Samples do not contain timestamps
perf-3.0 + any kernel
perf record -agT
perf script
is happy everywhere. Thanks!
-Arun
Quoted 8 lines · Arun Sharma
Quoted 6 lines · Frederic Weisbecker
I'm confused, you first said it happens with new tools on older kernel.
Can you tell us which combination of kernel/user raises the error? and which error.
I tried to repeat your examples below using perf-core tip (3.0-rc3, af07ce3e77).
Quoted 7 lines
perf-3.0 + 2.6.38
perf record -agR gives:
Can't find id 9's machine
Found 1 unknown events!
This does not reproduce on 2.6.38.8-32.fc15.x86_64 running bare metal or 2.6.38 in a VM. If you are running bare metal try -e cpu-clock and see if it is cycles related. I doubt it, but that should be the only difference in the tests.
That said, I did see that message while working on the gettimeofday patches a few weeks ago. I believe the root cause was the early mixture of cycles and tracepoints -- something I fixed before sending out the patches (sample_type hack as well as how the time-of-day trace events were added to the event list).
Quoted 7 lines
perf-3.0 + 2.6.32
perf record -ag
perf scriptsamples have bogus timestamps
I'm surprised by this one. I tried with an older Fedora 12 VM (2.6.32.21 kernel). I don't get timestamps with perf-script and perf-report -D shows -1 which is what I expect given that SAMPLE_TIME attribute is not set by default. One difference here is that VM's default to cpu-clock for the event.
Quoted 7 lines
perf-3.0 + 2.6.32
perf record -agR
perf scripterror: Samples do not contain timestamps
And this one did not work out so well with the F-12 VM. It caused a kernel panic - top line on console is perf_swevent_hrtimer (lines scrolled off the top).
Quoted 7 lines
perf-3.0 + any kernel
perf record -agT
perf scriptis happy everywhere. Thanks!
Which is what I would expect. Glad to see my version of reality apply elsewhere. ;-)
David
Quoted 3 lines
-Arun