drm/msm: add trace_dma_fence_emit to msm_gpu_submit
Subsystems (4)
PATCH v2 1/1 drm/msm: add trace_dma_fence_emit to msm_gpu_submit
In practice, trace_dma_fence_init called from dma_fence_init is good enough and almost no driver calls trace_dma_fence_emit. But drm_sched and virtio both have cases where trace_dma_fence_init and trace_dma_fence_emit can be apart. It is easier for visualization tools to always use the more correct trace_dma_fence_emit when visualizing fence timelines.
v2: improve commit message (Dmitry)
--- drivers/gpu/drm/msm/msm_gpu.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/gpu/drm/msm/msm_gpu.c +2 -0
@@ -15,6 +15,7 @@#include <linux/string_helpers.h> #include <linux/devcoredump.h> #include <linux/sched/task.h> +#include <trace/events/dma_fence.h> /* * Power Management:
@@ -769,6 +770,7 @@ void msm_gpu_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) gpu->active_submits++;mutex_unlock (&gpu->active_lock); +trace_dma_fence_emit (submit->hw_fence); gpu->funcs->submit (gpu, submit); gpu->cur_ctx_seqno = submit->queue->ctx->seqno;
Signature · 1 line
2.35.1.1178.g4f1659d476-goog
Quoted 8 lines · Chia-I Wu
In practice,
trace_dma_fence_initcalled fromdma_fence_initis good enough and almost no driver callstrace_dma_fence_emit. Butdrm_schedand virtio both have cases wheretrace_dma_fence_initandtrace_dma_fence_emitcan be apart. It is easier for visualization tools to always use the more correcttrace_dma_fence_emitwhen visualizing fence timelines.
Quoted 5 lines
v2: improve commit message (Dmitry)
Signed-off-by Chia-I Wu <[email protected]>Cc Rob Clark <[email protected]>Reviewed-by Dmitry Baryshkov <[email protected]>
This has been reviewed. Should we land it?
(Or, how do I check if it has landed?)
Am 26.04.22 um 18:32 schrieb Chia-I Wu:
Quoted 8 lines · Chia-I Wu
Quoted 7 lines
In practice,
trace_dma_fence_initcalled fromdma_fence_initis good enough and almost no driver callstrace_dma_fence_emit. Butdrm_schedand virtio both have cases wheretrace_dma_fence_initandtrace_dma_fence_emitcan be apart. It is easier for visualization tools to always use the more correcttrace_dma_fence_emitwhen visualizing fence timelines.
Quoted 6 lines
Quoted 5 lines
v2: improve commit message (Dmitry)
Signed-off-by Chia-I Wu <[email protected]>Cc Rob Clark <[email protected]>Reviewed-by Dmitry Baryshkov <[email protected]>This has been reviewed. Should we land it?
No, there are still open discussions about it.
Regards,
Christian.
Quoted 2 lines
(Or, how do I check if it has landed?)
Quoted 10 lines · Christian König
Am 26.04.22 um 18:32 schrieb Chia-I Wu:
Quoted 8 lines · Chia-I Wu
Quoted 7 lines
In practice,
trace_dma_fence_initcalled fromdma_fence_initis good enough and almost no driver callstrace_dma_fence_emit. Butdrm_schedand virtio both have cases wheretrace_dma_fence_initandtrace_dma_fence_emitcan be apart. It is easier for visualization tools to always use the more correcttrace_dma_fence_emitwhen visualizing fence timelines.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
v2: improve commit message (Dmitry)
Signed-off-by Chia-I Wu <[email protected]>Cc Rob Clark <[email protected]>Reviewed-by Dmitry Baryshkov <[email protected]>This has been reviewed. Should we land it?
No, there are still open discussions about it.
I think if it is needed for trace visualization, that is justification enough for me
I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.
BR,
-R
Quoted 6 lines
Regards,
Christian.Quoted 2 lines
(Or, how do I check if it has landed?)
Am 26.04.22 um 19:05 schrieb Rob Clark:
Quoted 16 lines
Quoted 14 lines · Christian König
Am 26.04.22 um 18:32 schrieb Chia-I Wu:
Quoted 13 lines · Chia-I Wu
Quoted 12 lines
In practice,
trace_dma_fence_initcalled fromdma_fence_initis good enough and almost no driver callstrace_dma_fence_emit. Butdrm_schedand virtio both have cases wheretrace_dma_fence_initandtrace_dma_fence_emitcan be apart. It is easier for visualization tools to always use the more correcttrace_dma_fence_emitwhen visualizing fence timelines.v2: improve commit message (Dmitry)
Signed-off-by Chia-I Wu <[email protected]>Cc Rob Clark <[email protected]>Reviewed-by Dmitry Baryshkov <[email protected]>
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
This has been reviewed. Should we land it?
No, there are still open discussions about it.
I think if it is needed for trace visualization, that is justification enough for me
I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the trace_dma_fence_init trace point is good for.
The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.
Regards,
Christian.
Quoted 8 lines
BR,
-RQuoted 4 lines
Regards,
Christian.Quoted 1 line
(Or, how do I check if it has landed?)
Quoted 27 lines · Christian König
Am 26.04.22 um 19:05 schrieb Rob Clark:
Quoted 16 lines
Quoted 14 lines · Christian König
Am 26.04.22 um 18:32 schrieb Chia-I Wu:
Quoted 13 lines · Chia-I Wu
Quoted 12 lines
In practice,
trace_dma_fence_initcalled fromdma_fence_initis good enough and almost no driver callstrace_dma_fence_emit. Butdrm_schedand virtio both have cases wheretrace_dma_fence_initandtrace_dma_fence_emitcan be apart. It is easier for visualization tools to always use the more correcttrace_dma_fence_emitwhen visualizing fence timelines.v2: improve commit message (Dmitry)
Signed-off-by Chia-I Wu <[email protected]>Cc Rob Clark <[email protected]>Reviewed-by Dmitry Baryshkov <[email protected]>Quoted 8 lines
Quoted 2 lines
Quoted 1 line
This has been reviewed. Should we land it?
No, there are still open discussions about it.
I think if it is needed for trace visualization, that is justification enough for me
I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.
Quoted 8 lines
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.
Hmm, AFAIU from Chia-I, virtgpu has a separation of init and emit.. OTOH if using separate events in these special cases is better, then I'm ok with that and can revert this patch. Chia-I is more familiar with the visualization end of it, so I'll let him comment on whether that is a workable approach.
BR,
-R
Quoted 12 lines
Regards,
Christian.Quoted 8 lines
BR,
-RQuoted 4 lines
Regards,
Christian.Quoted 1 line
(Or, how do I check if it has landed?)
Am 26.04.22 um 19:16 schrieb Rob Clark:
Quoted 32 lines
Quoted 30 lines · Christian König
Am 26.04.22 um 19:05 schrieb Rob Clark:
Quoted 16 lines
Quoted 14 lines · Christian König
Am 26.04.22 um 18:32 schrieb Chia-I Wu:
Quoted 13 lines · Chia-I Wu
Quoted 12 lines
In practice,
trace_dma_fence_initcalled fromdma_fence_initis good enough and almost no driver callstrace_dma_fence_emit. Butdrm_schedand virtio both have cases wheretrace_dma_fence_initandtrace_dma_fence_emitcan be apart. It is easier for visualization tools to always use the more correcttrace_dma_fence_emitwhen visualizing fence timelines.v2: improve commit message (Dmitry)
Signed-off-by Chia-I Wu <[email protected]>Cc Rob Clark <[email protected]>Reviewed-by Dmitry Baryshkov <[email protected]>Quoted 8 lines
Quoted 2 lines
Quoted 1 line
This has been reviewed. Should we land it?
No, there are still open discussions about it.
I think if it is needed for trace visualization, that is justification enough for me
I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.
Quoted 8 lines
Quoted 3 lines
The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.
Hmm, AFAIU from Chia-I, virtgpu has a separation of init and emit.. OTOH if using separate events in these special cases is better, then I'm ok with that and can revert this patch. Chia-I is more familiar with the visualization end of it, so I'll let him comment on whether that is a workable approach.
Interesting, I wasn't aware of the virtgpu separation of init and emit.
But yes if there is really an use case for tracing this time stamp as well then we should probably have that use case specific.
I just looked into the scheduler case a bit and found that even there we already have a different trace point for it, which is probably the reason why we never used trace_dma_fence_emit there.
So yes, there really isn't much reason I can see two have two separate trace points for every driver.
Christian.
Quoted 14 lines
BR,
-RQuoted 10 lines
Regards,
Christian.Quoted 7 lines
BR,
-RQuoted 4 lines
Regards,
Christian.Quoted 1 line
(Or, how do I check if it has landed?)
Quoted 33 lines · Christian König
Am 26.04.22 um 19:16 schrieb Rob Clark:
Quoted 31 lines
Quoted 29 lines · Christian König
Am 26.04.22 um 19:05 schrieb Rob Clark:
Quoted 16 lines
Quoted 14 lines · Christian König
Am 26.04.22 um 18:32 schrieb Chia-I Wu:
Quoted 13 lines · Chia-I Wu
Quoted 12 lines
In practice,
trace_dma_fence_initcalled fromdma_fence_initis good enough and almost no driver callstrace_dma_fence_emit. Butdrm_schedand virtio both have cases wheretrace_dma_fence_initandtrace_dma_fence_emitcan be apart. It is easier for visualization tools to always use the more correcttrace_dma_fence_emitwhen visualizing fence timelines.v2: improve commit message (Dmitry)
Signed-off-by Chia-I Wu <[email protected]>Cc Rob Clark <[email protected]>Reviewed-by Dmitry Baryshkov <[email protected]>Quoted 8 lines
Quoted 2 lines
Quoted 1 line
This has been reviewed. Should we land it?
No, there are still open discussions about it.
I think if it is needed for trace visualization, that is justification enough for me
I don't really see otherwise how a generic trace visualization tool like perfetto would handle the case that some fence timelines have separate events but others do not.
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.
I am a bit confused by this. _emit and _signaled are a great way to see how many fences are pending from cpu's point of view. How does _emit make no sense and _init is good instead?
Or is this just that _init is good enough most of the time? (More below)
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
The only use case is for debugging the GPU scheduler and we should probably introduce a separate GPU scheduler specific trace point for this instead if we should ever need it.
Hmm, AFAIU from Chia-I, virtgpu has a separation of init and emit.. OTOH if using separate events in these special cases is better, then I'm ok with that and can revert this patch. Chia-I is more familiar with the visualization end of it, so I'll let him comment on whether that is a workable approach.
Quoted 8 lines
Interesting, I wasn't aware of the virtgpu separation of init and emit.
But yes if there is really an use case for tracing this time stamp as well then we should probably have that use case specific.
I just looked into the scheduler case a bit and found that even there we already have a different trace point for it, which is probably the reason why we never used
trace_dma_fence_emitthere.
Yeah, I am using drm_sched tracepoints in that case.
Quoted 3 lines
So yes, there really isn't much reason I can see two have two separate trace points for every driver.
That sounds fair. In any tool, it should be easy to see if a fence timeline has _emit in addition to _init, and adapt accordingly. We can drop this patch.
A clarification that _emit is optional/redundant for most fence timelines should be nicer than removing it though.
Quoted 18 lines
Christian.
Quoted 14 lines
BR,
-RQuoted 10 lines
Regards,
Christian.Quoted 7 lines
BR,
-RQuoted 4 lines
Regards,
Christian.Quoted 1 line
(Or, how do I check if it has landed?)
Am 26.04.22 um 19:40 schrieb Chia-I Wu:
Quoted 8 lines
[SNIP]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.I am a bit confused by this.
_emitand_signaledare a great way to see how many fences are pending from cpu's point of view. How does_emitmake no sense and_initis good instead?
We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the _emit trace point.
See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.
The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.
On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.
So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.
Regards,
Christian.
Quoted 14 lines · Christian König
Am 26.04.22 um 19:40 schrieb Chia-I Wu:
Quoted 8 lines
[SNIP]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.I am a bit confused by this.
_emitand_signaledare a great way to see how many fences are pending from cpu's point of view. How does_emitmake no sense and_initis good instead?We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the
_emittrace point.
Quoted 7 lines
See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.
The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.
I am more interested in pending on the GPU.
Quoted 4 lines
On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.
_signaled alone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.
For all drivers except virtio-gpu, _init and "ring head update" always happen close enough that I can see why _emit is redundant. But I like having _emit as a generic tracepoint for timelines where _init and _emit can be apart, instead of requiring a special case tracepoint for each special case timeline.
Quoted 9 lines
So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.
Regards,
Christian.
Am 26.04.22 um 20:50 schrieb Chia-I Wu:
Quoted 21 lines
Quoted 12 lines · Christian König
Am 26.04.22 um 19:40 schrieb Chia-I Wu:
Quoted 8 lines
[SNIP]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.I am a bit confused by this.
_emitand_signaledare a great way to see how many fences are pending from cpu's point of view. How does_emitmake no sense and_initis good instead?We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the
_emittrace point.Quoted 7 lines
See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.
The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.
Quoted 8 lines
I am more interested in pending on the GPU.
Quoted 3 lines
On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.
_signaledalone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.
Yeah, I'm perfectly aware of that.
Quoted 5 lines
For all drivers except virtio-gpu,
_initand "ring head update" always happen close enough that I can see why_emitis redundant. But I like having_emitas a generic tracepoint for timelines where_initand_emitcan be apart, instead of requiring a special case tracepoint for each special case timeline.
And I'm certainly not going to add _emit to all drivers just because of that. As you said it is a special case for virtio-gpu and the GPU scheduler.
And as I explained before the difference between _init and _emit shouldn't matter to your visualization. The background is that as soon as an dma_fence is initialized with _init it is "live" regarding the dependency and memory management and exactly that's what matters for the visualization.
The latency between _init and _emit is just interesting for debugging the scheduler and surprisingly virtio-gpu as well, for all other use cases it is irrelevant.
Regards,
Christian.
Quoted 8 lines
Quoted 8 lines
So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.
Regards,
Christian.
Quoted 49 lines · Christian König
Am 26.04.22 um 20:50 schrieb Chia-I Wu:
Quoted 21 lines
Quoted 12 lines · Christian König
Am 26.04.22 um 19:40 schrieb Chia-I Wu:
Quoted 8 lines
[SNIP]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.I am a bit confused by this.
_emitand_signaledare a great way to see how many fences are pending from cpu's point of view. How does_emitmake no sense and_initis good instead?We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the
_emittrace point.Quoted 7 lines
See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.
The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.
Quoted 8 lines
I am more interested in pending on the GPU.
Quoted 3 lines
On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.
_signaledalone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.Yeah, I'm perfectly aware of that.
Quoted 5 lines
For all drivers except virtio-gpu,
_initand "ring head update" always happen close enough that I can see why_emitis redundant. But I like having_emitas a generic tracepoint for timelines where_initand_emitcan be apart, instead of requiring a special case tracepoint for each special case timeline.And I'm certainly not going to add
_emitto all drivers just because of that. As you said it is a special case for virtio-gpu and the GPU scheduler.And as I explained before the difference between
_initand_emitshouldn't matter to your visualization. The background is that as soon as andma_fenceis initialized with_initit is "live" regarding the dependency and memory management and exactly that's what matters for the visualization.
Quoted 3 lines
The latency between
_initand_emitis just interesting for debugging the scheduler and surprisingly virtio-gpu as well, for all other use cases it is irrelevant.
It might actually be *more* interesting for virtio-gpu.. unless there is some other way to link guest and host fences to see what the potential latency of guest->host is
re: adding the tracepoint to other drivers, I'm fine with folks doing that as needed. Unless you have a better proposal about how to visualize init vs emit latency, I think it's fine to have an extra tracepoint even if it is redundant in some cases. The visualization tool is the customer here, we have to give it what it wants/needs.
BR,
-R
Quoted 13 lines
Regards,
Christian.Quoted 8 lines
Quoted 8 lines
So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.
Regards,
Christian.
Quoted 60 lines · Rob Clark
Quoted 49 lines · Christian König
Am 26.04.22 um 20:50 schrieb Chia-I Wu:
Quoted 21 lines
Quoted 12 lines · Christian König
Am 26.04.22 um 19:40 schrieb Chia-I Wu:
Quoted 8 lines
[SNIP]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Well I just send a patch to completely remove the trace point.
As I said it absolutely doesn't make sense to use this for visualization, that's what the
trace_dma_fence_inittrace point is good for.I am a bit confused by this.
_emitand_signaledare a great way to see how many fences are pending from cpu's point of view. How does_emitmake no sense and_initis good instead?We had exactly that confusion now multiple times and it's one of the main reasons why I want to remove the
_emittrace point.Quoted 7 lines
See the when you want to know how many fences are pending you need to watch out for init/destroy and *NOT* emit.
The reason is that in the special case where emit makes sense (e.g. the GPU scheduler fences) emit comes later than init, but pending on the CPU and taking up resources are all fences and not just the one emitted to the hardware.
Quoted 8 lines
I am more interested in pending on the GPU.
Quoted 3 lines
On the other hand when you want to measure how much time each operation took on the hardware you need to take a look at the differences of the signal events on each timeline.
_signaledalone is not enough when the GPU is not always busy. After the last pending fence signals but before the following _init/_emit, nothing is pending.Yeah, I'm perfectly aware of that.
Quoted 5 lines
For all drivers except virtio-gpu,
_initand "ring head update" always happen close enough that I can see why_emitis redundant. But I like having_emitas a generic tracepoint for timelines where_initand_emitcan be apart, instead of requiring a special case tracepoint for each special case timeline.And I'm certainly not going to add
_emitto all drivers just because of that. As you said it is a special case for virtio-gpu and the GPU scheduler.And as I explained before the difference between
_initand_emitshouldn't matter to your visualization. The background is that as soon as andma_fenceis initialized with_initit is "live" regarding the dependency and memory management and exactly that's what matters for the visualization.Quoted 3 lines
The latency between
_initand_emitis just interesting for debugging the scheduler and surprisingly virtio-gpu as well, for all other use cases it is irrelevant.It might actually be *more* interesting for virtio-gpu.. unless there is some other way to link guest and host fences to see what the potential latency of
guest->hostis
Quoted 5 lines
re: adding the tracepoint to other drivers, I'm fine with folks doing that as needed. Unless you have a better proposal about how to visualize init vs emit latency, I think it's fine to have an extra tracepoint even if it is redundant in some cases. The visualization tool is the customer here, we have to give it what it wants/needs.
As far as perfetto is concerned, I just use either _init or _emit on a per-timeline basis. We can drop this patch for msm, and do not need to change drivers whose latencies between _init/_emit are ignorable.
init vs emit latency is still interesting. I prefer keeping _init / _emit as generic events that tools can parse, rather than adding per-driver special cases that tools need to understand.
Quoted 17 lines
BR,
-RQuoted 13 lines
Regards,
Christian.Quoted 8 lines
Quoted 8 lines
So there isn't really any use case for the emit trace point, except when you want to figure out how much latency the scheduler introduce. Then you want to take a look at init and emit, but that isn't really that interesting for performance analyses.
Regards,
Christian.