Skip to contentSkip to conversation

perf inject: Flush ordered events on FINISHED_ROUND

PATCH In review Namhyung Kim 2 participants 7 messages Oct 2, 2020 – Oct 12, 2020 perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

PATCH 1/1 perf inject: Flush ordered events on FINISHED_ROUND

Currently perf inject just repipes the event without any flush. It makes an issue that it changes the order of events processed.

Normally we want to process events in time order, but events are recorded from several cpus and they weren't sorted with each other. So we put them into the ordered event queue, sorted by time, and flush them when we see a next FINISHED_ROUND event.

But this is for events from kernel, user space events (like the FINISHED_ROUND) are processed without queueing. So during the perf inject, it writes all the FINISHED_ROUND events directly while SAMPLE (and other) events are kept in the queue. This makes the user space events are put before any kernel events.

You can see that with the following command:

  $ perf record -a -o- sleep 1 | perf inject -b | perf script -i- --show-round-events
  PERF_RECORD_FINISHED_ROUND
  PERF_RECORD_FINISHED_ROUND
  PERF_RECORD_FINISHED_ROUND
  ...

Omitting perf inject, you can see the events are placed in the middle of the data.

You might argue that the whole point of the FINISHED_ROUND event is to sort (kernel) events. And after perf inject, all events are written in a proper time order so we don't care about the FINISHED_ROUND event anymore.

But the problem is we don't know whether the input data is sorted or not (maybe we can add a feature bit for this?) so it should use an ordered event queue when processing the input like in perf report.

Remember all the FINISHED_ROUND events now come before other events so the tool cannot know when it can flush the data so everything will be queued until it meets the end of the input. Actually it's same for perf inject itself as it doesn't flush the queue.

Below measures time and memory usage during the perf inject and report using ~190MB data file.

Before:
perf inject: 11.09 s, 382148 KB
perf report: 8.05 s, 397440 KB

After:
perf inject: 16.24 s, 83376 KB
perf report: 7.96 s, 216184 KB

As you can see, it used 2x memory of the input size. I guess it's because it needs to keep the copy for the whole input. But I don't understand why processing time of perf inject increased..

I'm not sure how it affects the auxtrace, but it should be fine IMHO.

Cc Al Grant <[email protected]>
Cc Adrian Hunter <[email protected]>
---
 tools/perf/builtin-inject.c | 1 +
 1 file changed, 1 insertion(+)
tools/perf/builtin-inject.c +1 -0
index 6d2f410d773a..9fa78a9edfc5 100644
@@ -79,6 +79,7 @@ static int perf_event__repipe_oe_synth(struct perf_tool *tool,
 				       union perf_event *event,
 				       struct ordered_events *oe __maybe_unused)
 {
+	ordered_events__flush(oe, OE_FLUSH__ROUND);
 	return perf_event__repipe_synth(tool, event);
 }
 
Signature · 1 line
2.28.0.806.g8561365e88-goog
#1 lore raw
Quoted 25 lines · Namhyung Kim

Currently perf inject just repipes the event without any flush. It makes an issue that it changes the order of events processed.

Normally we want to process events in time order, but events are recorded from several cpus and they weren't sorted with each other. So we put them into the ordered event queue, sorted by time, and flush them when we see a next FINISHED_ROUND event.

But this is for events from kernel, user space events (like the FINISHED_ROUND) are processed without queueing. So during the perf inject, it writes all the FINISHED_ROUND events directly while SAMPLE (and other) events are kept in the queue. This makes the user space events are put before any kernel events.

You can see that with the following command:

  $ perf record -a -o- sleep 1 | perf inject -b | perf script -i- --show-round-events
  PERF_RECORD_FINISHED_ROUND
  PERF_RECORD_FINISHED_ROUND
  PERF_RECORD_FINISHED_ROUND
  ...

Omitting perf inject, you can see the events are placed in the middle of the data.

Quoted 8 lines

You might argue that the whole point of the FINISHED_ROUND event is to sort (kernel) events. And after perf inject, all events are written in a proper time order so we don't care about the FINISHED_ROUND event anymore.

But the problem is we don't know whether the input data is sorted or not (maybe we can add a feature bit for this?) so it should use an ordered event queue when processing the input like in perf report.

I like the idea of storing the information that the data is sorted, and when it's there, let's not use ordered_oevets

Quoted 13 lines

Remember all the FINISHED_ROUND events now come before other events so the tool cannot know when it can flush the data so everything will be queued until it meets the end of the input. Actually it's same for perf inject itself as it doesn't flush the queue.

Below measures time and memory usage during the perf inject and report using ~190MB data file.

Before:
perf inject: 11.09 s, 382148 KB
perf report: 8.05 s, 397440 KB

Quoted 7 lines

After:
perf inject: 16.24 s, 83376 KB
perf report: 7.96 s, 216184 KB

As you can see, it used 2x memory of the input size. I guess it's because it needs to keep the copy for the whole input. But I don't understand why processing time of perf inject increased..

would be good to find out first

thanks,
jirka

Quoted 25 lines

I'm not sure how it affects the auxtrace, but it should be fine IMHO.

Cc Al Grant <[email protected]>
Cc Adrian Hunter <[email protected]>
---
 tools/perf/builtin-inject.c | 1 +
 1 file changed, 1 insertion(+)
tools/perf/builtin-inject.c +1 -0
index 6d2f410d773a..9fa78a9edfc5 100644
@@ -79,6 +79,7 @@ static int perf_event__repipe_oe_synth(struct perf_tool *tool,
 				       union perf_event *event,
 				       struct ordered_events *oe __maybe_unused)
 {
+	ordered_events__flush(oe, OE_FLUSH__ROUND);
 	return perf_event__repipe_synth(tool, event);
 }
 
Signature · 1 line
2.28.0.806.g8561365e88-goog
author maintainer
#2 lore raw

Hi Jiri,

Quoted 32 lines · Jiri Olsa
Quoted 30 lines · Namhyung Kim

Currently perf inject just repipes the event without any flush. It makes an issue that it changes the order of events processed.

Normally we want to process events in time order, but events are recorded from several cpus and they weren't sorted with each other. So we put them into the ordered event queue, sorted by time, and flush them when we see a next FINISHED_ROUND event.

But this is for events from kernel, user space events (like the FINISHED_ROUND) are processed without queueing. So during the perf inject, it writes all the FINISHED_ROUND events directly while SAMPLE (and other) events are kept in the queue. This makes the user space events are put before any kernel events.

You can see that with the following command:

  $ perf record -a -o- sleep 1 | perf inject -b | perf script -i- --show-round-events
  PERF_RECORD_FINISHED_ROUND
  PERF_RECORD_FINISHED_ROUND
  PERF_RECORD_FINISHED_ROUND
  ...

Omitting perf inject, you can see the events are placed in the middle of the data.

You might argue that the whole point of the FINISHED_ROUND event is to sort (kernel) events. And after perf inject, all events are written in a proper time order so we don't care about the FINISHED_ROUND event anymore.

Quoted 6 lines
Quoted 3 lines

But the problem is we don't know whether the input data is sorted or not (maybe we can add a feature bit for this?) so it should use an ordered event queue when processing the input like in perf report.

I like the idea of storing the information that the data is sorted, and when it's there, let's not use ordered_oevets

Thanks for your input. Yeah, I think it's better not to use it if possible.

Quoted 18 lines
Quoted 18 lines

Remember all the FINISHED_ROUND events now come before other events so the tool cannot know when it can flush the data so everything will be queued until it meets the end of the input. Actually it's same for perf inject itself as it doesn't flush the queue.

Below measures time and memory usage during the perf inject and report using ~190MB data file.

Before:
perf inject: 11.09 s, 382148 KB
perf report: 8.05 s, 397440 KB

After:
perf inject: 16.24 s, 83376 KB
perf report: 7.96 s, 216184 KB

Quoted 5 lines
Quoted 3 lines

As you can see, it used 2x memory of the input size. I guess it's because it needs to keep the copy for the whole input. But I don't understand why processing time of perf inject increased..

would be good to find out first

Will do!

Thanks
Namhyung

author maintainer
#3 lore raw
Quoted 8 lines
Quoted 8 lines · Namhyung Kim
Quoted 7 lines

Below measures time and memory usage during the perf inject and report using ~190MB data file.

Before:
perf inject: 11.09 s, 382148 KB
perf report: 8.05 s, 397440 KB

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

After:
perf inject: 16.24 s, 83376 KB
perf report: 7.96 s, 216184 KB

As you can see, it used 2x memory of the input size. I guess it's because it needs to keep the copy for the whole input. But I don't understand why processing time of perf inject increased..

Measuring it with time shows:

before after
real 11.309s 17.040s
user 8.084s 13.940s
sys 6.535s 6.732s

So it's user space to make the difference. I've run perf record on both (with cycles:U) and the dominant function is same: queue_event. (46.98% vs 65.87%)

It seems the flushing the queue makes more overhead on sorting.

Thanks
Namhyung

author maintainer
#4 lore raw
Quoted 23 lines · On Tue, Oct 06, 2020 at 11:39:49AM +0900, [email protected] wrote:
Quoted 8 lines
Quoted 8 lines · Namhyung Kim
Quoted 7 lines

Below measures time and memory usage during the perf inject and report using ~190MB data file.

Before:
perf inject: 11.09 s, 382148 KB
perf report: 8.05 s, 397440 KB

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

After:
perf inject: 16.24 s, 83376 KB
perf report: 7.96 s, 216184 KB

As you can see, it used 2x memory of the input size. I guess it's because it needs to keep the copy for the whole input. But I don't understand why processing time of perf inject increased..

Measuring it with time shows:

before after
real 11.309s 17.040s
user 8.084s 13.940s
sys 6.535s 6.732s

Quoted 5 lines

So it's user space to make the difference. I've run perf record on both (with cycles:U) and the dominant function is same: queue_event. (46.98% vs 65.87%)

It seems the flushing the queue makes more overhead on sorting.

So I suspect the cache-miss ratio affects the performance. With flushing, data is processed in the middle and all the entries are reused after flush so it would invalidate all the cache lines occasionally.

This is the perf stat result:

* Before

     7,167,414,019      L1-dcache-loads                                             
       337,471,761      L1-dcache-read-misses     #    4.71% of all L1-dcache hits  

      11.011224671 seconds time elapsed

* After

     7,075,556,792      L1-dcache-loads                                             
       771,810,388      L1-dcache-read-misses     #   10.91% of all L1-dcache hits  

      17.015901863 seconds time elapsed

Hmm.. it's a memory & time trade-off then. Maybe we need a switch to select which one?

Thanks
Namhyung

#5 lore raw
Quoted 47 lines · On Tue, Oct 06, 2020 at 02:40:32PM +0900, [email protected] wrote:
Quoted 23 lines · On Tue, Oct 06, 2020 at 11:39:49AM +0900, [email protected] wrote:
Quoted 8 lines
Quoted 8 lines · Namhyung Kim
Quoted 7 lines

Below measures time and memory usage during the perf inject and report using ~190MB data file.

Before:
perf inject: 11.09 s, 382148 KB
perf report: 8.05 s, 397440 KB

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

After:
perf inject: 16.24 s, 83376 KB
perf report: 7.96 s, 216184 KB

As you can see, it used 2x memory of the input size. I guess it's because it needs to keep the copy for the whole input. But I don't understand why processing time of perf inject increased..

Measuring it with time shows:

before after
real 11.309s 17.040s
user 8.084s 13.940s
sys 6.535s 6.732s

Quoted 5 lines

So it's user space to make the difference. I've run perf record on both (with cycles:U) and the dominant function is same: queue_event. (46.98% vs 65.87%)

It seems the flushing the queue makes more overhead on sorting.

So I suspect the cache-miss ratio affects the performance. With flushing, data is processed in the middle and all the entries are reused after flush so it would invalidate all the cache lines occasionally.

This is the perf stat result:

* Before

     7,167,414,019      L1-dcache-loads                                             
       337,471,761      L1-dcache-read-misses     #    4.71% of all L1-dcache hits  

      11.011224671 seconds time elapsed

* After

Quoted 8 lines
     7,075,556,792      L1-dcache-loads                                             
       771,810,388      L1-dcache-read-misses     #   10.91% of all L1-dcache hits  

      17.015901863 seconds time elapsed

Hmm.. it's a memory & time trade-off then. Maybe we need a switch to select which one?

I'd keep the faster one ;-) so the one before

jirka

author maintainer
#6 lore raw

Hi Jiri,

Quoted 17 lines · Jiri Olsa
Quoted 15 lines · On Tue, Oct 06, 2020 at 02:40:32PM +0900, [email protected] wrote:

This is the perf stat result:

* Before

     7,167,414,019      L1-dcache-loads
       337,471,761      L1-dcache-read-misses     #    4.71% of all L1-dcache hits

      11.011224671 seconds time elapsed

* After

     7,075,556,792      L1-dcache-loads
       771,810,388      L1-dcache-read-misses     #   10.91% of all L1-dcache hits
Quoted 7 lines
Quoted 5 lines
      17.015901863 seconds time elapsed

Hmm.. it's a memory & time trade-off then. Maybe we need a switch to select which one?

I'd keep the faster one ;-) so the one before

Yeah, it's hard to argue when it slows things down much. But I'd like to note that memory consumption also can be a big issue in some environments. I'll bring this back later..

Thanks
Namhyung