drm/i915: track first and last processes that touch gem objects
Subsystems (2)
PATCH 1/1 drm/i915: track first and last processes that touch gem objects
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
To simplify post-mortem analysis, we also search for the processes names when gathering the i915_error_state and when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.
--- drivers/gpu/drm/i915/i915_debugfs.c | 45 ++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_drv.h | 5 +++ drivers/gpu/drm/i915/i915_gem_execbuffer.c | 6 ++++ drivers/gpu/drm/i915/i915_irq.c | 22 +++++++++++++ 4 files changed, 78 insertions(+), 0 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +45 -0
@@ -121,6 +121,7 @@ static const char *cache_level_str(int type)static void describe_obj (struct seq_file *m,struct drm_i915_gem_object *obj) { +int i; seq_printf(m, "%p: %s%s %8zdKiB %04x %04x %d %d%s%s%s", &obj->base, get_pin_flag(obj),
@@ -151,6 +152,28 @@ describe_obj(struct seq_file *m, struct drm_i915_gem_object *obj) }if (obj->ring !=NULL )seq_printf (m," (%s)" , obj->ring->name); + +/* Describe 1st and last process to touch the object */ +for (i=0 ; i <2 ; i++) { +struct pid *p; +struct task_struct *tsk =NULL ; + +/* Skip objects that have no associated pid */ +if (!obj->pid[i]) +continue ; + + p =find_get_pid (obj->pid[i]); +if (p) { + tsk =get_pid_task (p,PIDTYPE_PID ); +put_pid (p); + } + +seq_printf (m," (pid_%s: %5d [%s])" , + (i==0 ) ?"first" :"last" , + obj->pid[i], + (tsk) ? tsk->comm :"unknown" ); + + } }static int i915_gem_object_list_info (struct seq_file *m,void *data)
@@ -710,6 +733,7 @@ static void print_error_buffers(struct seq_file *m,struct drm_i915_error_buffer *err, int count) { + int i;seq_printf (m,"%s [%d]:\n" , name, count);while (count--) {
@@ -731,6 +755,27 @@ static void print_error_buffers(struct seq_file *m,if (err->fence_reg !=I915_FENCE_REG_NONE )seq_printf (m," (fence: %d)" , err->fence_reg); +/* Describe 1st and last process to touch the object */ +for (i=0 ; i <2 ; i++) { +struct pid *p; +struct task_struct *tsk =NULL ; + +/* Skip objects that have no associated pid */ +if (!err->pid[i]) +continue ; + + p =find_get_pid (err->pid[i]); +if (p) + tsk =get_pid_task (p,PIDTYPE_PID ); + +seq_printf (m," (pid_%s: %5d [%s])" , + (i==0 ) ?"first" :"last" , + err->pid[i], + (tsk) ? tsk->comm :"unknown" ); + +put_pid (p); + } +seq_printf (m,"\n" ); err++; }
drivers/gpu/drm/i915/i915_drv.h +5 -0
@@ -193,6 +193,8 @@ struct drm_i915_error_state { u32 purgeable:1; u32 ring:4; u32 cache_level:2; +u32 pid[2 ]; +char comm[2 ][TASK_COMM_LEN ]; } *active_bo, *pinned_bo;u32 active_bo_count, pinned_bo_count;struct intel_overlay_error_state *overlay;
@@ -891,6 +893,9 @@ struct drm_i915_gem_object {/** for phy allocated objects */ struct drm_i915_gem_phys_object *phys_obj; +/** pid of first and last process to touch the object */ +uint32_t pid[2 ]; +/** * Number of crtcs where this object is currently the fb, but * will be page flipped away on the next vblank. When it
drivers/gpu/drm/i915/i915_gem_execbuffer.c +6 -0
@@ -1140,6 +1140,12 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,list_add_tail (&obj->exec_list, &objects); obj->exec_handle = exec[i].handle; obj->exec_entry = &exec[i]; + +/* Discover pid of caller process */ +if (!obj->pid[0 ]) + obj->pid[0 ] = file->pid; + obj->pid[1 ] = file->pid; +eb_add_object (eb, obj); }
drivers/gpu/drm/i915/i915_irq.c +22 -0
@@ -789,8 +789,30 @@ static u32 capture_bo_list(struct drm_i915_error_buffer *err,int i =0 ;list_for_each_entry (obj, head, mm_list) { +struct pid *p =NULL ; +struct task_struct *tsk =NULL ; +int j; + err->size = obj->base.size; err->name = obj->base.name; + +/* Record processes which touched this object and collect their +* names to simplify further analysis. +*/ +for (j=0 ; j <2 ; j++) { + err->pid[j] = obj->pid[j]; + +if (err->pid[j]) + p =find_get_pid (err->pid[j]); +if (p) { + tsk =get_pid_task (p,PIDTYPE_PID ); +put_pid (p); + } + +snprintf (err->comm[j],TASK_COMM_LEN , + (tsk) ? tsk->comm :"unknown" ); + } + err->seqno = obj->last_rendering_seqno; err->gtt_offset = obj->gtt_offset; err->read_domains = obj->base.read_domains;
Signature · 1 line
1.7.8.1
Quoted 11 lines · Eugeni Dodonov
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
To simplify post-mortem analysis, we also search for the processes names when gathering the
i915_error_stateand when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.
May be this is too stupid question, but it seems you do not track neither writers using I915_GEM_PWRITE nor writers which modified buffer with mmaped access.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 1/1 drm/i915: track first and last processes that touch gem objects
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
To simplify post-mortem analysis, we also search for the processes names when gathering the i915_error_state and when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.
v2: also track objects accessed via mmap or pwrite.
--- drivers/gpu/drm/i915/i915_debugfs.c | 45 ++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_drv.h | 5 +++ drivers/gpu/drm/i915/i915_gem.c | 12 +++++++ drivers/gpu/drm/i915/i915_gem_execbuffer.c | 6 ++++ drivers/gpu/drm/i915/i915_irq.c | 22 +++++++++++++ 5 files changed, 90 insertions(+), 0 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +45 -0
@@ -121,6 +121,7 @@ static const char *cache_level_str(int type)static void describe_obj (struct seq_file *m,struct drm_i915_gem_object *obj) { +int i; seq_printf(m, "%p: %s%s %8zdKiB %04x %04x %d %d%s%s%s", &obj->base, get_pin_flag(obj),
@@ -151,6 +152,28 @@ describe_obj(struct seq_file *m, struct drm_i915_gem_object *obj) }if (obj->ring !=NULL )seq_printf (m," (%s)" , obj->ring->name); + +/* Describe 1st and last process to touch the object */ +for (i=0 ; i <2 ; i++) { +struct pid *p; +struct task_struct *tsk =NULL ; + +/* Skip objects that have no associated pid */ +if (!obj->pid[i]) +continue ; + + p =find_get_pid (obj->pid[i]); +if (p) { + tsk =get_pid_task (p,PIDTYPE_PID ); +put_pid (p); + } + +seq_printf (m," (pid_%s: %5d [%s])" , + (i==0 ) ?"first" :"last" , + obj->pid[i], + (tsk) ? tsk->comm :"unknown" ); + + } }static int i915_gem_object_list_info (struct seq_file *m,void *data)
@@ -710,6 +733,7 @@ static void print_error_buffers(struct seq_file *m,struct drm_i915_error_buffer *err, int count) { + int i;seq_printf (m,"%s [%d]:\n" , name, count);while (count--) {
@@ -731,6 +755,27 @@ static void print_error_buffers(struct seq_file *m,if (err->fence_reg !=I915_FENCE_REG_NONE )seq_printf (m," (fence: %d)" , err->fence_reg); +/* Describe 1st and last process to touch the object */ +for (i=0 ; i <2 ; i++) { +struct pid *p; +struct task_struct *tsk =NULL ; + +/* Skip objects that have no associated pid */ +if (!err->pid[i]) +continue ; + + p =find_get_pid (err->pid[i]); +if (p) + tsk =get_pid_task (p,PIDTYPE_PID ); + +seq_printf (m," (pid_%s: %5d [%s])" , + (i==0 ) ?"first" :"last" , + err->pid[i], + (tsk) ? tsk->comm :"unknown" ); + +put_pid (p); + } +seq_printf (m,"\n" ); err++; }
drivers/gpu/drm/i915/i915_drv.h +5 -0
@@ -193,6 +193,8 @@ struct drm_i915_error_state { u32 purgeable:1; u32 ring:4; u32 cache_level:2; +u32 pid[2 ]; +char comm[2 ][TASK_COMM_LEN ]; } *active_bo, *pinned_bo;u32 active_bo_count, pinned_bo_count;struct intel_overlay_error_state *overlay;
@@ -891,6 +893,9 @@ struct drm_i915_gem_object {/** for phy allocated objects */ struct drm_i915_gem_phys_object *phys_obj; +/** pid of first and last process to touch the object */ +uint32_t pid[2 ]; +/** * Number of crtcs where this object is currently the fb, but * will be page flipped away on the next vblank. When it
drivers/gpu/drm/i915/i915_gem.c +12 -0
@@ -988,6 +988,11 @@ i915_gem_pwrite_ioctl(struct drm_device *dev, void *data, goto out; } +/* Discover pid of caller process */ +if (!obj->pid[0 ]) + obj->pid[0 ] = file->pid; + obj->pid[1 ] = file->pid; +trace_i915_gem_object_pwrite (obj, args->offset, args->size);/* We can only do the GTT pwrite on untiled buffers, as otherwise
@@ -1144,6 +1149,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,struct drm_i915_private *dev_priv = dev->dev_private;struct drm_i915_gem_mmap *args = data;struct drm_gem_object *obj; +struct drm_i915_gem_object *i915_obj;unsigned long addr;if (!(dev->driver->driver_features &DRIVER_GEM ))
@@ -1158,6 +1164,12 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,return -E2BIG ; } +/* Discover pid of caller process */ + i915_obj =to_intel_bo (obj); +if (!i915_obj->pid[0 ]) + i915_obj->pid[0 ] = file->pid; + i915_obj->pid[1 ] = file->pid; +down_write (¤t->mm->mmap_sem); addr = do_mmap(obj->filp, 0, args->size, PROT_READ | PROT_WRITE, MAP_SHARED,
drivers/gpu/drm/i915/i915_gem_execbuffer.c +6 -0
@@ -1140,6 +1140,12 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,list_add_tail (&obj->exec_list, &objects); obj->exec_handle = exec[i].handle; obj->exec_entry = &exec[i]; + +/* Discover pid of caller process */ +if (!obj->pid[0 ]) + obj->pid[0 ] = file->pid; + obj->pid[1 ] = file->pid; +eb_add_object (eb, obj); }
drivers/gpu/drm/i915/i915_irq.c +22 -0
@@ -789,8 +789,30 @@ static u32 capture_bo_list(struct drm_i915_error_buffer *err,int i =0 ;list_for_each_entry (obj, head, mm_list) { +struct pid *p =NULL ; +struct task_struct *tsk =NULL ; +int j; + err->size = obj->base.size; err->name = obj->base.name; + +/* Record processes which touched this object and collect their +* names to simplify further analysis. +*/ +for (j=0 ; j <2 ; j++) { + err->pid[j] = obj->pid[j]; + +if (err->pid[j]) + p =find_get_pid (err->pid[j]); +if (p) { + tsk =get_pid_task (p,PIDTYPE_PID ); +put_pid (p); + } + +snprintf (err->comm[j],TASK_COMM_LEN , + (tsk) ? tsk->comm :"unknown" ); + } + err->seqno = obj->last_rendering_seqno; err->gtt_offset = obj->gtt_offset; err->read_domains = obj->base.read_domains;
Signature · 1 line
1.7.8.1
Quoted 5 lines · Eugeni Dodonov
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
Quoted 8 lines
To simplify post-mortem analysis, we also search for the processes names when gathering the
i915_error_stateand when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.v2: also track objects accessed via mmap or pwrite.
Only writes through GTT mappings are recorded, I think that writes when the object is in CPU domain are almost impossible to note with the driver (at least it is so on FreeBSD).
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 11 lines · Eugeni Dodonov
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
To simplify post-mortem analysis, we also search for the processes names when gathering the
i915_error_stateand when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.
I think you've slightly missed here. Tracking who created a buffer is interesting and who last used it, but you really need to also track on whose behalf the request (i.e. each batch) is executing.
For the goal of recording creator, you could just use:
obj->creator = current ? current->pid : 0;
in i915_gem_object_init with 0 as the special value for objects created by the driver outside of process context. And similarly for i915_add_request, though I'd associate those with the owner of the file_priv. The important point here is that a buffer may be associated with multiple batches submitted by one or more clients before a hang is detected, and so unless the dispatch pid is tracked you do not know who submitted the erroneous batch. (Even a batch may be submitted more than once by many clients, given sufficient pathology.) So adding the request queue to the i915_error_state would also be interesting, especially with the jiffie and ring->tail.
Also note that there is no direct link between i915_gem_fault() and usage of the object, the point at which you want to add the obj->last_used_by tracking to is domain management - which catches the usage of CPU mappings as well as move-to-active.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Quoted 32 lines · Chris Wilson
Quoted 11 lines · Eugeni Dodonov
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
To simplify post-mortem analysis, we also search for the processes names when gathering the
i915_error_stateand when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.I think you've slightly missed here. Tracking who created a buffer is interesting and who last used it, but you really need to also track on whose behalf the request (i.e. each batch) is executing.
For the goal of recording creator, you could just use:
obj->creator = current ? current->pid : 0;in
i915_gem_object_initwith 0 as the special value for objects created by the driver outside of process context. And similarly fori915_add_request, though I'd associate those with the owner of thefile_priv. The important point here is that a buffer may be associated with multiple batches submitted by one or more clients before a hang is detected, and so unless the dispatch pid is tracked you do not know who submitted the erroneous batch. (Even a batch may be submitted more than once by many clients, given sufficient pathology.) So adding the request queue to thei915_error_statewould also be interesting, especially with the jiffie andring->tail.
Quoted 4 lines
Also note that there is no direct link between
i915_gem_fault()and usage of the object, the point at which you want to add theobj->last_used_bytracking to is domain management - which catches the usage of CPU mappings as well as move-to-active.
I'll second Chris here - I think the interesting stuff is to add some kind of cheap ownership tracking, not who exactly created the buffer. The latter is imo only really interesting for resource accounting, and that would require it to be somewhat more solid. And we don't do any resource accounting atm anyway.
-Daniel
Signature · 3 lines
Daniel Vetter Mail: [email protected] Mobile: +41 (0)79 365 57 48
Quoted 38 lines · Daniel Vetter
Quoted 32 lines · Chris Wilson
Quoted 11 lines · Eugeni Dodonov
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
To simplify post-mortem analysis, we also search for the processes names when gathering the
i915_error_stateand when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.I think you've slightly missed here. Tracking who created a buffer is interesting and who last used it, but you really need to also track on whose behalf the request (i.e. each batch) is executing.
For the goal of recording creator, you could just use:
obj->creator = current ? current->pid : 0;in
i915_gem_object_initwith 0 as the special value for objects created by the driver outside of process context. And similarly fori915_add_request, though I'd associate those with the owner of thefile_priv. The important point here is that a buffer may be associated with multiple batches submitted by one or more clients before a hang is detected, and so unless the dispatch pid is tracked you do not know who submitted the erroneous batch. (Even a batch may be submitted more than once by many clients, given sufficient pathology.) So adding the request queue to thei915_error_statewould also be interesting, especially with the jiffie andring->tail.Quoted 4 lines
Also note that there is no direct link between
i915_gem_fault()and usage of the object, the point at which you want to add theobj->last_used_bytracking to is domain management - which catches the usage of CPU mappings as well as move-to-active.
Quoted 5 lines
I'll second Chris here - I think the interesting stuff is to add some kind of cheap ownership tracking, not who exactly created the buffer. The latter is imo only really interesting for resource accounting, and that would require it to be somewhat more solid. And we don't do any resource accounting atm anyway.
Having the creator associated with the buffer should be nice. I agree that for hang debugging, making the pid association part of the request struct makes more sense than tracking it per-object. With those two, I don't see much use for "last pwriter/executer" with the buffer.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 45 lines · Eric Anholt
Quoted 38 lines · Daniel Vetter
Quoted 32 lines · Chris Wilson
Quoted 11 lines · Eugeni Dodonov
This allows to hopefully find out who was responsible for the GPU death. We record the 1st and last process to touch each object, to keep track of the process which created the object originally and the last process to touch it.
To simplify post-mortem analysis, we also search for the processes names when gathering the
i915_error_stateand when peeking at the list of active gem objects in debugfs. This is not perfect for tracking all the processes, as they can quit or die before their batchbuffers got executed, but having to track them during the entire object lifetime would be excessively memcpy hungry.I think you've slightly missed here. Tracking who created a buffer is interesting and who last used it, but you really need to also track on whose behalf the request (i.e. each batch) is executing.
For the goal of recording creator, you could just use:
obj->creator = current ? current->pid : 0;in
i915_gem_object_initwith 0 as the special value for objects created by the driver outside of process context. And similarly fori915_add_request, though I'd associate those with the owner of thefile_priv. The important point here is that a buffer may be associated with multiple batches submitted by one or more clients before a hang is detected, and so unless the dispatch pid is tracked you do not know who submitted the erroneous batch. (Even a batch may be submitted more than once by many clients, given sufficient pathology.) So adding the request queue to thei915_error_statewould also be interesting, especially with the jiffie andring->tail.Quoted 4 lines
Also note that there is no direct link between
i915_gem_fault()and usage of the object, the point at which you want to add theobj->last_used_bytracking to is domain management - which catches the usage of CPU mappings as well as move-to-active.Quoted 5 lines
I'll second Chris here - I think the interesting stuff is to add some kind of cheap ownership tracking, not who exactly created the buffer. The latter is imo only really interesting for resource accounting, and that would require it to be somewhat more solid. And we don't do any resource accounting atm anyway.
Quoted 4 lines
Having the creator associated with the buffer should be nice. I agree that for hang debugging, making the pid association part of the request struct makes more sense than tracking it per-object. With those two, I don't see much use for "last pwriter/executer" with the buffer.
Could I recommend storing drm_file instead of the PID. That is what I have, and required for forced-throttling. You should be able to get to a pid from the file descriptor.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx