Skip to contentSkip to conversation

drm/msm: add trace_dma_fence_emit to msm_gpu_submit

PATCH In review Chia-I Wu 3 participants 13 messages Apr 12, 2022 – Apr 27, 2022

Subsystems (4)
lore
Share & tools
lore

Chia-I Wu

author
#0 lore raw

PATCH v2 1/1 drm/msm: add trace_dma_fence_emit to msm_gpu_submit

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

v2: improve commit message (Dmitry)

---
 drivers/gpu/drm/msm/msm_gpu.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/gpu/drm/msm/msm_gpu.c +2 -0
index faf0c242874e..a82193f41ea2 100644
@@ -15,6 +15,7 @@
 #include <linux/string_helpers.h>
 #include <linux/devcoredump.h>
 #include <linux/sched/task.h>
+#include <trace/events/dma_fence.h>
 
 /*
  * Power Management:
@@ -769,6 +770,7 @@ void msm_gpu_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit)
 	gpu->active_submits++;
 	mutex_unlock(&gpu->active_lock);
 
+	trace_dma_fence_emit(submit->hw_fence);
 	gpu->funcs->submit(gpu, submit);
 	gpu->cur_ctx_seqno = submit->queue->ctx->seqno;
 
Signature · 1 line
2.35.1.1178.g4f1659d476-goog
author
#1 lore raw
Quoted 8 lines · Chia-I Wu

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

Quoted 5 lines

v2: improve commit message (Dmitry)

This has been reviewed. Should we land it?

(Or, how do I check if it has landed?)

#2 lore raw

Am 26.04.22 um 18:32 schrieb Chia-I Wu:

Quoted 8 lines · Chia-I Wu
Quoted 7 lines

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

Quoted 6 lines
Quoted 5 lines

v2: improve commit message (Dmitry)

This has been reviewed. Should we land it?

No, there are still open discussions about it.

Regards,
Christian.

Quoted 2 lines

(Or, how do I check if it has landed?)

#3 lore raw
Quoted 10 lines · Christian König

Am 26.04.22 um 18:32 schrieb Chia-I Wu:

Quoted 8 lines · Chia-I Wu
Quoted 7 lines

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

v2: improve commit message (Dmitry)

This has been reviewed. Should we land it?

No, there are still open discussions about it.

I think if it is needed for trace visualization, that is justification enough for me

I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.

BR,
-R

Quoted 6 lines

Regards,
Christian.

Quoted 2 lines

(Or, how do I check if it has landed?)

#4 lore raw

Am 26.04.22 um 19:05 schrieb Rob Clark:

Quoted 16 lines
Quoted 14 lines · Christian König

Am 26.04.22 um 18:32 schrieb Chia-I Wu:

Quoted 13 lines · Chia-I Wu
Quoted 12 lines

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

v2: improve commit message (Dmitry)

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

This has been reviewed. Should we land it?

No, there are still open discussions about it.

I think if it is needed for trace visualization, that is justification enough for me

I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.

Regards,
Christian.

Quoted 8 lines

BR,
-R

Quoted 4 lines

Regards,
Christian.

Quoted 1 line

(Or, how do I check if it has landed?)

#5 lore raw
Quoted 27 lines · Christian König

Am 26.04.22 um 19:05 schrieb Rob Clark:

Quoted 16 lines
Quoted 14 lines · Christian König

Am 26.04.22 um 18:32 schrieb Chia-I Wu:

Quoted 13 lines · Chia-I Wu
Quoted 12 lines

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

v2: improve commit message (Dmitry)

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

This has been reviewed. Should we land it?

No, there are still open discussions about it.

I think if it is needed for trace visualization, that is justification enough for me

I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.

Quoted 8 lines

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.

Hmm, AFAIU from Chia-I, virtgpu has a separation of init and emit.. OTOH if using separate events in these special cases is better, then I'm ok with that and can revert this patch. Chia-I is more familiar with the visualization end of it, so I'll let him comment on whether that is a workable approach.

BR,
-R

Quoted 12 lines

Regards,
Christian.

Quoted 8 lines

BR,
-R

Quoted 4 lines

Regards,
Christian.

Quoted 1 line

(Or, how do I check if it has landed?)

#6 lore raw

Am 26.04.22 um 19:16 schrieb Rob Clark:

Quoted 32 lines
Quoted 30 lines · Christian König

Am 26.04.22 um 19:05 schrieb Rob Clark:

Quoted 16 lines
Quoted 14 lines · Christian König

Am 26.04.22 um 18:32 schrieb Chia-I Wu:

Quoted 13 lines · Chia-I Wu
Quoted 12 lines

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

v2: improve commit message (Dmitry)

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

This has been reviewed. Should we land it?

No, there are still open discussions about it.

I think if it is needed for trace visualization, that is justification enough for me

I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

Quoted 8 lines
Quoted 3 lines

The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.

Hmm, AFAIU from Chia-I, virtgpu has a separation of init and emit.. OTOH if using separate events in these special cases is better, then I'm ok with that and can revert this patch. Chia-I is more familiar with the visualization end of it, so I'll let him comment on whether that is a workable approach.

Interesting, I wasn't aware of the virtgpu separation of init and emit.

But yes if there is really an use case for tracing this time stamp as well then we should probably have that use case specific.

I just looked into the scheduler case a bit and found that even there we already have a different trace point for it, which is probably the reason why we never used trace_dma_fence_emit there.

So yes, there really isn't much reason I can see two have two separate trace points for every driver.

Christian.

Quoted 14 lines

BR,
-R

Quoted 10 lines

Regards,
Christian.

Quoted 7 lines

BR,
-R

Quoted 4 lines

Regards,
Christian.

Quoted 1 line

(Or, how do I check if it has landed?)

author
#7 lore raw
Quoted 33 lines · Christian König

Am 26.04.22 um 19:16 schrieb Rob Clark:

Quoted 31 lines
Quoted 29 lines · Christian König

Am 26.04.22 um 19:05 schrieb Rob Clark:

Quoted 16 lines
Quoted 14 lines · Christian König

Am 26.04.22 um 18:32 schrieb Chia-I Wu:

Quoted 13 lines · Chia-I Wu
Quoted 12 lines

In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.

v2: improve commit message (Dmitry)

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

This has been reviewed. Should we land it?

No, there are still open discussions about it.

I think if it is needed for trace visualization, that is justification enough for me

I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

I am a bit confused by this. _emit and _signaled are a great way to see how many fences are pending from cpu's point of view. How does _emit make no sense and _init is good instead?

Or is this just that _init is good enough most of the time? (More below)

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.

Hmm, AFAIU from Chia-I, virtgpu has a separation of init and emit.. OTOH if using separate events in these special cases is better, then I'm ok with that and can revert this patch. Chia-I is more familiar with the visualization end of it, so I'll let him comment on whether that is a workable approach.

Quoted 8 lines

Interesting, I wasn't aware of the virtgpu separation of init and emit.

But yes if there is really an use case for tracing this time stamp as well then we should probably have that use case specific.

I just looked into the scheduler case a bit and found that even there we already have a different trace point for it, which is probably the reason why we never used trace_dma_fence_emit there.

Yeah, I am using drm_sched tracepoints in that case.

Quoted 3 lines

So yes, there really isn't much reason I can see two have two separate trace points for every driver.

That sounds fair. In any tool, it should be easy to see if a fence timeline has _emit in addition to _init, and adapt accordingly. We can drop this patch.

A clarification that _emit is optional/redundant for most fence timelines should be nicer than removing it though.

Quoted 18 lines

Christian.

Quoted 14 lines

BR,
-R

Quoted 10 lines

Regards,
Christian.

Quoted 7 lines

BR,
-R

Quoted 4 lines

Regards,
Christian.

Quoted 1 line

(Or, how do I check if it has landed?)

#8 lore raw

Am 26.04.22 um 19:40 schrieb Chia-I Wu:

Quoted 8 lines

[SNIP]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

I am a bit confused by this. _emit and _signaled are a great way to see how many fences are pending from cpu's point of view. How does _emit make no sense and _init is good instead?

We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the _emit trace point.

See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.

The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.

On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.

So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.

Regards,
Christian.

author
#9 lore raw
Quoted 14 lines · Christian König

Am 26.04.22 um 19:40 schrieb Chia-I Wu:

Quoted 8 lines

[SNIP]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

I am a bit confused by this. _emit and _signaled are a great way to see how many fences are pending from cpu's point of view. How does _emit make no sense and _init is good instead?

We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the _emit trace point.

Quoted 7 lines

See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.

The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.

I am more interested in pending on the GPU.

Quoted 4 lines

On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.

_signaled alone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.

For all drivers except virtio-gpu, _init and "ring head update" always happen close enough that I can see why _emit is redundant. But I like having _emit as a generic tracepoint for timelines where _init and _emit can be apart, instead of requiring a special case tracepoint for each special case timeline.

Quoted 9 lines

So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.

Regards,
Christian.

#10 lore raw

Am 26.04.22 um 20:50 schrieb Chia-I Wu:

Quoted 21 lines
Quoted 12 lines · Christian König

Am 26.04.22 um 19:40 schrieb Chia-I Wu:

Quoted 8 lines

[SNIP]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

I am a bit confused by this. _emit and _signaled are a great way to see how many fences are pending from cpu's point of view. How does _emit make no sense and _init is good instead?

We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the _emit trace point.

Quoted 7 lines

See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.

The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.

Quoted 8 lines

I am more interested in pending on the GPU.

Quoted 3 lines

On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.

_signaled alone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.

Yeah, I'm perfectly aware of that.

Quoted 5 lines

For all drivers except virtio-gpu, _init and "ring head update" always happen close enough that I can see why _emit is redundant. But I like having _emit as a generic tracepoint for timelines where _init and _emit can be apart, instead of requiring a special case tracepoint for each special case timeline.

And I'm certainly not going to add _emit to all drivers just because of that. As you said it is a special case for virtio-gpu and the GPU scheduler.

And as I explained before the difference between _init and _emit shouldn't matter to your visualization. The background is that as soon as an dma_fence is initialized with _init it is "live" regarding the dependency and memory management and exactly that's what matters for the visualization.

The latency between _init and _emit is just interesting for debugging the scheduler and surprisingly virtio-gpu as well, for all other use cases it is irrelevant.

Regards,
Christian.

Quoted 8 lines
Quoted 8 lines

So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.

Regards,
Christian.

#11 lore raw
Quoted 49 lines · Christian König

Am 26.04.22 um 20:50 schrieb Chia-I Wu:

Quoted 21 lines
Quoted 12 lines · Christian König

Am 26.04.22 um 19:40 schrieb Chia-I Wu:

Quoted 8 lines

[SNIP]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

I am a bit confused by this. _emit and _signaled are a great way to see how many fences are pending from cpu's point of view. How does _emit make no sense and _init is good instead?

We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the _emit trace point.

Quoted 7 lines

See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.

The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.

Quoted 8 lines

I am more interested in pending on the GPU.

Quoted 3 lines

On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.

_signaled alone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.

Yeah, I'm perfectly aware of that.

Quoted 5 lines

For all drivers except virtio-gpu, _init and "ring head update" always happen close enough that I can see why _emit is redundant. But I like having _emit as a generic tracepoint for timelines where _init and _emit can be apart, instead of requiring a special case tracepoint for each special case timeline.

And I'm certainly not going to add _emit to all drivers just because of that. As you said it is a special case for virtio-gpu and the GPU scheduler.

And as I explained before the difference between _init and _emit shouldn't matter to your visualization. The background is that as soon as an dma_fence is initialized with _init it is "live" regarding the dependency and memory management and exactly that's what matters for the visualization.

Quoted 3 lines

The latency between _init and _emit is just interesting for debugging the scheduler and surprisingly virtio-gpu as well, for all other use cases it is irrelevant.

It might actually be *more* interesting for virtio-gpu.. unless there is some other way to link guest and host fences to see what the potential latency of guest->host is

re: adding the tracepoint to other drivers, I'm fine with folks doing that as needed. Unless you have a better proposal about how to visualize init vs emit latency, I think it's fine to have an extra tracepoint even if it is redundant in some cases. The visualization tool is the customer here, we have to give it what it wants/needs.

BR,
-R

Quoted 13 lines

Regards,
Christian.

Quoted 8 lines
Quoted 8 lines

So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.

Regards,
Christian.

author
#12 lore raw
Quoted 60 lines · Rob Clark
Quoted 49 lines · Christian König

Am 26.04.22 um 20:50 schrieb Chia-I Wu:

Quoted 21 lines
Quoted 12 lines · Christian König

Am 26.04.22 um 19:40 schrieb Chia-I Wu:

Quoted 8 lines

[SNIP]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Well I just send a patch to completely remove the trace point.

As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.

I am a bit confused by this. _emit and _signaled are a great way to see how many fences are pending from cpu's point of view. How does _emit make no sense and _init is good instead?

We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the _emit trace point.

Quoted 7 lines

See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.

The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.

Quoted 8 lines

I am more interested in pending on the GPU.

Quoted 3 lines

On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.

_signaled alone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.

Yeah, I'm perfectly aware of that.

Quoted 5 lines

For all drivers except virtio-gpu, _init and "ring head update" always happen close enough that I can see why _emit is redundant. But I like having _emit as a generic tracepoint for timelines where _init and _emit can be apart, instead of requiring a special case tracepoint for each special case timeline.

And I'm certainly not going to add _emit to all drivers just because of that. As you said it is a special case for virtio-gpu and the GPU scheduler.

And as I explained before the difference between _init and _emit shouldn't matter to your visualization. The background is that as soon as an dma_fence is initialized with _init it is "live" regarding the dependency and memory management and exactly that's what matters for the visualization.

Quoted 3 lines

The latency between _init and _emit is just interesting for debugging the scheduler and surprisingly virtio-gpu as well, for all other use cases it is irrelevant.

It might actually be *more* interesting for virtio-gpu.. unless there is some other way to link guest and host fences to see what the potential latency of guest->host is

Quoted 5 lines

re: adding the tracepoint to other drivers, I'm fine with folks doing that as needed. Unless you have a better proposal about how to visualize init vs emit latency, I think it's fine to have an extra tracepoint even if it is redundant in some cases. The visualization tool is the customer here, we have to give it what it wants/needs.

As far as perfetto is concerned, I just use either _init or _emit on a per-timeline basis. We can drop this patch for msm, and do not need to change drivers whose latencies between _init/_emit are ignorable.

init vs emit latency is still interesting. I prefer keeping _init / _emit as generic events that tools can parse, rather than adding per-driver special cases that tools need to understand.

Quoted 17 lines

BR,
-R

Quoted 13 lines

Regards,
Christian.

Quoted 8 lines
Quoted 8 lines

So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.

Regards,
Christian.