drm/msm: GPU crash state
Subsystems (4)
cover drm/msm: GPU crash state
This is a request for comment on code to store and dump a GPU state a hang with inspiration from the very good i915 GPU error state and the binary GPU snapshot in the downstream kernel.
The goal is to store and provide enough information to debug software and hardware issues on the Adreno hardware in a semi human-readable format that can also be parsed by scripts.
The goal for this request for comment is to get some consensus about the format and work through some of the technical issues.
The first patch is a bit of a standalone - it finally consolidates the 'show' and 'dump' code (where show goes to a seq_file and 'dump' goes to the kernel log) through judicious use of drm_printer.
The next patch captures the ringbuffer and register information the same way that 'show' does - most of the code is generic but eventually will get more platform specific as we add new chunks of data (especially for a5xx and beyond).
The third patch converts the existing show function to use the state instead of reading the data directly. This gets a little big ugly because due to the first patch, if we don't capture a state here then nothing gets printed out.
The final patch prints the crash state through its own debugfs node. Only one crash state is captured at a time and it persists until the state is cleared by a write to the debugfs node.
After we mostly agree on the behavior and design of the code the next step will be to pull over more state for 5xx including ringbuffer contents, active commands and buffers, shader caches and other debug information.
Jordan Crouse (4):
drm/msm: gpu: Use drm_printer to consolidate the show/dump code drm/msm: gpu: Capture the state of the GPU
drm/msm: gpu: Convert the GPU show functions to use the GPU state drm/msm: gpu: Capture the GPU state on a GPU hang
drivers/gpu/drm/msm/adreno/a3xx_gpu.c | 43 +++++----- drivers/gpu/drm/msm/adreno/a4xx_gpu.c | 39 ++++----- drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 44 +++++----- drivers/gpu/drm/msm/adreno/adreno_gpu.c | 138 +++++++++++++++++++------------- drivers/gpu/drm/msm/adreno/adreno_gpu.h | 10 ++- drivers/gpu/drm/msm/msm_debugfs.c | 78 ++++++++++++++++-- drivers/gpu/drm/msm/msm_gpu.h | 64 ++++++++++++++- 7 files changed, 296 insertions(+), 120 deletions(-)
Signature · 6 lines
1.9.1 _______________________________________________ Freedreno mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/freedreno
PATCH 1/4 drm/msm: gpu: Use drm_printer to consolidate the show/dump code
Use drm_printer to abstract the show/dump debug code so we can use the same code for both the kernel log and seq_file paths. Maintaining the existing dump format mandates that we break the single show function into two but thats a small price to pay for removing a bunch of code and consolidating the path.
--- drivers/gpu/drm/msm/adreno/a3xx_gpu.c | 18 +++------ drivers/gpu/drm/msm/adreno/a4xx_gpu.c | 15 ++----- drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 18 +++------ drivers/gpu/drm/msm/adreno/adreno_gpu.c | 70 +++++++++------------------------ drivers/gpu/drm/msm/adreno/adreno_gpu.h | 2 + 5 files changed, 34 insertions(+), 89 deletions(-)
drivers/gpu/drm/msm/adreno/a3xx_gpu.c +5 -13
@@ -39,7 +39,6 @@extern bool hang_debug; -static void a3xx_dump (struct msm_gpu *gpu);static bool a3xx_idle (struct msm_gpu *gpu);static bool a3xx_me_init (struct msm_gpu *gpu)
@@ -300,18 +299,18 @@ static int a3xx_hw_init(struct msm_gpu *gpu)static void a3xx_recover (struct msm_gpu *gpu) { +struct drm_printer p =drm_info_printer (gpu->dev->dev);int i; -adreno_dump_info (gpu); +adreno_show_info (gpu, &p); -for (i =0 ; i <8 ; i++) { -printk ("CP_SCRATCH_REG%d: %u\n" , i, +for (i =0 ; i <8 ; i++) +drm_printf (&p,"CP_SCRATCH_REG%d: %u\n" , i,gpu_read (gpu,REG_AXXX_CP_SCRATCH_REG0 + i)); - }/* dump registers before resetting gpu, if enabled: */ if (hang_debug) -a3xx_dump (gpu); +adreno_show_regs (gpu, &p);gpu_write (gpu,REG_A3XX_RBBM_SW_RESET_CMD ,1 );gpu_read (gpu,REG_A3XX_RBBM_SW_RESET_CMD );
@@ -419,13 +418,6 @@ static void a3xx_show(struct msm_gpu *gpu, struct seq_file *m) }#endif -/* would be nice to not have to duplicate the _show() stuff with printk(): */ -static void a3xx_dump (struct msm_gpu *gpu) -{ -printk ("status: %08x\n" , -gpu_read (gpu,REG_A3XX_RBBM_STATUS )); -adreno_dump (gpu); -}/* Register offset defines for A3XX */ static const unsigned int a3xx_register_offsets[REG_ADRENO_REGISTER_MAX ] = {REG_ADRENO_DEFINE (REG_ADRENO_CP_RB_BASE ,REG_AXXX_CP_RB_BASE ),
drivers/gpu/drm/msm/adreno/a4xx_gpu.c +4 -11
@@ -30,7 +30,6 @@ A4XX_INT0_UCHE_OOB_ACCESS) extern bool hang_debug; -static void a4xx_dump (struct msm_gpu *gpu);static bool a4xx_idle (struct msm_gpu *gpu);/*
@@ -298,18 +297,19 @@ static int a4xx_hw_init(struct msm_gpu *gpu)static void a4xx_recover (struct msm_gpu *gpu) { +struct drm_printer p =drm_info_printer (gpu->dev->dev);int i; -adreno_dump_info (gpu); +adreno_show_info (gpu, &p);for (i =0 ; i <8 ; i++) { -printk ("CP_SCRATCH_REG%d: %u\n" , i, +drm_printf (&p,"CP_SCRATCH_REG%d: %u\n" , i,gpu_read (gpu,REG_AXXX_CP_SCRATCH_REG0 + i)); }/* dump registers before resetting gpu, if enabled: */ if (hang_debug) -a4xx_dump (gpu); +adreno_show_regs (gpu, &p);gpu_write (gpu,REG_A4XX_RBBM_SW_RESET_CMD ,1 );gpu_read (gpu,REG_A4XX_RBBM_SW_RESET_CMD );
@@ -475,13 +475,6 @@ static void a4xx_show(struct msm_gpu *gpu, struct seq_file *m) REG_ADRENO_DEFINE(REG_ADRENO_CP_RB_CNTL, REG_A4XX_CP_RB_CNTL), }; -static void a4xx_dump (struct msm_gpu *gpu) -{ -printk ("status: %08x\n" , -gpu_read (gpu,REG_A4XX_RBBM_STATUS )); -adreno_dump (gpu); -} -static int a4xx_pm_resume (struct msm_gpu *gpu) {struct adreno_gpu *adreno_gpu =to_adreno_gpu (gpu);int ret;
drivers/gpu/drm/msm/adreno/a5xx_gpu.c +5 -13
@@ -22,7 +22,6 @@#include "a5xx_gpu.h" extern bool hang_debug; -static void a5xx_dump (struct msm_gpu *gpu);#define GPU_PAS_ID 13
@@ -755,17 +754,17 @@ static int a5xx_hw_init(struct msm_gpu *gpu)static void a5xx_recover (struct msm_gpu *gpu) { +struct drm_printer p =drm_info_printer (gpu->dev->dev);int i; -adreno_dump_info (gpu); +adreno_show_info (gpu, &p); -for (i =0 ; i <8 ; i++) { -printk ("CP_SCRATCH_REG%d: %u\n" , i, +for (i =0 ; i <8 ; i++) +drm_printf (&p,"CP_SCRATCH_REG%d: %u\n" , i,gpu_read (gpu,REG_A5XX_CP_SCRATCH_REG (i))); - }if (hang_debug) -a5xx_dump (gpu); +adreno_show_regs (gpu, &p);gpu_write (gpu,REG_A5XX_RBBM_SW_RESET_CMD ,1 );gpu_read (gpu,REG_A5XX_RBBM_SW_RESET_CMD );
@@ -1073,13 +1072,6 @@ static irqreturn_t a5xx_irq(struct msm_gpu *gpu) 0xB9A0, 0xB9BF, ~0 }; -static void a5xx_dump (struct msm_gpu *gpu) -{ -dev_info (gpu->dev->dev,"status: %08x\n" , -gpu_read (gpu,REG_A5XX_RBBM_STATUS )); -adreno_dump (gpu); -} -static int a5xx_pm_resume (struct msm_gpu *gpu) {int ret;
drivers/gpu/drm/msm/adreno/adreno_gpu.c +18 -52
107 lines not on this page · raw message
drivers/gpu/drm/msm/adreno/adreno_gpu.h +2 -0
@@ -223,6 +223,8 @@ int adreno_gpu_init(struct drm_device *drm, struct platform_device *pdev, int nr_rings);void adreno_gpu_cleanup (struct adreno_gpu *gpu); +void adreno_show_info (struct msm_gpu *gpu,struct drm_printer *p); +void adreno_show_regs (struct msm_gpu *gpu,struct drm_printer *p);/* ringbuffer helpers (the parts that are adreno specific) */
Signature · 1 line
1.9.1
PATCH 2/4 drm/msm: gpu: Capture the state of the GPU
In reply to Jordan CrouseAdd the infrastructure to capture the state current state of the GPU and store it in memory. This is useful for storing the state of a hung GPU so it can be dumped later.
For now grab the same basic ringbuffer information and registers that are provided by the debugfs 'gpu' node but obviously this can be extended to capture a much larger set of GPU information.
--- drivers/gpu/drm/msm/adreno/a3xx_gpu.c | 14 +++++++++ drivers/gpu/drm/msm/adreno/a4xx_gpu.c | 14 +++++++++ drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 22 ++++++++++++++ drivers/gpu/drm/msm/adreno/adreno_gpu.c | 54 +++++++++++++++++++++++++++++++++ drivers/gpu/drm/msm/adreno/adreno_gpu.h | 3 ++ drivers/gpu/drm/msm/msm_gpu.h | 19 ++++++++++++ 6 files changed, 126 insertions(+)
drivers/gpu/drm/msm/adreno/a3xx_gpu.c +14 -0
@@ -418,6 +418,18 @@ static void a3xx_show(struct msm_gpu *gpu, struct seq_file *m) }#endif +static struct msm_gpu_state *a3xx_gpu_state_get (struct msm_gpu *gpu) +{ +struct msm_gpu_state *state =adreno_gpu_state_get (gpu); + +if (IS_ERR (state)) +return state; + + state->rbbm_status =gpu_read (gpu,REG_A3XX_RBBM_STATUS ); + +return state; +} +/* Register offset defines for A3XX */ static const unsigned int a3xx_register_offsets[REG_ADRENO_REGISTER_MAX ] = {REG_ADRENO_DEFINE (REG_ADRENO_CP_RB_BASE ,REG_AXXX_CP_RB_BASE ),
@@ -444,6 +456,8 @@ static void a3xx_show(struct msm_gpu *gpu, struct seq_file *m)#ifdef CONFIG_DEBUG_FS .show = a3xx_show, #endif + .gpu_state_get = a3xx_gpu_state_get, + .gpu_state_put = adreno_gpu_state_put, }, };
drivers/gpu/drm/msm/adreno/a4xx_gpu.c +14 -0
@@ -464,6 +464,18 @@ static void a4xx_show(struct msm_gpu *gpu, struct seq_file *m) }#endif +static struct msm_gpu_state *a4xx_gpu_state_get (struct msm_gpu *gpu) +{ +struct msm_gpu_state *state =adreno_gpu_state_get (gpu); + +if (IS_ERR (state)) +return state; + + state->rbbm_status =gpu_read (gpu,REG_A4XX_RBBM_STATUS ); + +return state; +} +/* Register offset defines for A4XX, in order of enum adreno_regs */ static const unsigned int a4xx_register_offsets[REG_ADRENO_REGISTER_MAX ] = {REG_ADRENO_DEFINE (REG_ADRENO_CP_RB_BASE ,REG_A4XX_CP_RB_BASE ),
@@ -533,6 +545,8 @@ static int a4xx_get_timestamp(struct msm_gpu *gpu, uint64_t *value)#ifdef CONFIG_DEBUG_FS .show = a4xx_show, #endif + .gpu_state_get = a4xx_gpu_state_get, + .gpu_state_put = adreno_gpu_state_put, }, .get_timestamp = a4xx_get_timestamp, };
drivers/gpu/drm/msm/adreno/a5xx_gpu.c +22 -0
@@ -1133,6 +1133,26 @@ static int a5xx_get_timestamp(struct msm_gpu *gpu, uint64_t *value)return 0 ; } +static struct msm_gpu_state *a5xx_gpu_state_get (struct msm_gpu *gpu) +{ +struct msm_gpu_state *state; + +/* +* Temporarily disable hardware clock gating before going into +* adreno_show to avoid issues while reading the registers +*/ +a5xx_set_hwcg (gpu, false); + + state =adreno_gpu_state_get (gpu); + +if (!IS_ERR (state)) + state->rbbm_status =gpu_read (gpu,REG_A5XX_RBBM_STATUS ); + +a5xx_set_hwcg (gpu, true); + +return state; +} +#ifdef CONFIG_DEBUG_FS static void a5xx_show (struct msm_gpu *gpu,struct seq_file *m) {
@@ -1172,6 +1192,8 @@ static struct msm_ringbuffer *a5xx_active_ring(struct msm_gpu *gpu)#ifdef CONFIG_DEBUG_FS .show = a5xx_show, #endif + .gpu_state_get = a5xx_gpu_state_get, + .gpu_state_put = adreno_gpu_state_put, }, .get_timestamp = a5xx_get_timestamp, };
drivers/gpu/drm/msm/adreno/adreno_gpu.c +54 -0
60 lines not on this page · raw message
drivers/gpu/drm/msm/adreno/adreno_gpu.h +3 -0
@@ -226,6 +226,9 @@ int adreno_gpu_init(struct drm_device *drm, struct platform_device *pdev,void adreno_show_info (struct msm_gpu *gpu,struct drm_printer *p);void adreno_show_regs (struct msm_gpu *gpu,struct drm_printer *p); +struct msm_gpu_state *adreno_gpu_state_get (struct msm_gpu *gpu); +void adreno_gpu_state_put (struct msm_gpu_state *state); +/* ringbuffer helpers (the parts that are adreno specific) */ static inline void
drivers/gpu/drm/msm/msm_gpu.h +19 -0
@@ -27,6 +27,7 @@struct msm_gem_submit ;struct msm_gpu_perfcntr ; +struct msm_gpu_state ;struct msm_gpu_config {const char *ioname;
@@ -66,6 +67,8 @@ struct msm_gpu_funcs {/* show GPU status in debugfs: */ void (*show)(struct msm_gpu *gpu,struct seq_file *m);#endif +struct msm_gpu_state *(*gpu_state_get)(struct msm_gpu *gpu); +void (*gpu_state_put)(struct msm_gpu_state *state); };struct msm_gpu {
@@ -171,6 +174,22 @@ struct msm_gpu_submitqueue {struct kref ref; }; +struct msm_gpu_state { +struct timeval time; + +struct { +u32 fence; +u32 seqno; +u32 rptr; +u32 wptr; + } ring[MSM_GPU_MAX_RINGS ]; + +int nr_registers; +u32 *registers; + +u32 rbbm_status; +}; +static inline void gpu_write (struct msm_gpu *gpu,u32 reg,u32 data) {msm_writel (data, gpu->mmio + (reg <<2 ));
Signature · 6 lines
1.9.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
A message that is not in the archive (sent off-list or not yet received): 1515175221-5601-1-git-send-email-jcrouse-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org
PATCH 3/4 drm/msm: gpu: Convert the GPU show functions to use the GPU state
Convert the existing GPU show functions to use the GPU state to dump the information rather than reading it directly from the hardware. This will require an additional step to capture the state before dumping it for the existing nodes but it will greatly facilitate reusing the same code for dumping a previously captured state from a GPU hang.
--- drivers/gpu/drm/msm/adreno/a3xx_gpu.c | 20 ++++++--------- drivers/gpu/drm/msm/adreno/a4xx_gpu.c | 21 ++++++---------- drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 27 ++++++-------------- drivers/gpu/drm/msm/adreno/adreno_gpu.c | 44 ++++++++++++++++----------------- drivers/gpu/drm/msm/adreno/adreno_gpu.h | 9 ++++--- drivers/gpu/drm/msm/msm_debugfs.c | 21 +++++++++++----- drivers/gpu/drm/msm/msm_gpu.h | 3 ++- 7 files changed, 69 insertions(+), 76 deletions(-)
drivers/gpu/drm/msm/adreno/a3xx_gpu.c +8 -12
@@ -300,9 +300,12 @@ static int a3xx_hw_init(struct msm_gpu *gpu)static void a3xx_recover (struct msm_gpu *gpu) {struct drm_printer p =drm_info_printer (gpu->dev->dev); +struct msm_gpu_state *state;int i; -adreno_show_info (gpu, &p); + state = gpu->funcs->gpu_state_get (gpu); + +adreno_show_info (gpu, state, &p);for (i =0 ; i <8 ; i++) drm_printf(&p, "CP_SCRATCH_REG%d: %u\n", i,
@@ -310,12 +313,14 @@ static void a3xx_recover(struct msm_gpu *gpu)/* dump registers before resetting gpu, if enabled: */ if (hang_debug) -adreno_show_regs (gpu, &p); +adreno_show_regs (gpu, state, &p);gpu_write (gpu,REG_A3XX_RBBM_SW_RESET_CMD ,1 );gpu_read (gpu,REG_A3XX_RBBM_SW_RESET_CMD );gpu_write (gpu,REG_A3XX_RBBM_SW_RESET_CMD ,0 );adreno_recover (gpu); + + gpu->funcs->gpu_state_put (state); }static void a3xx_destroy (struct msm_gpu *gpu)
@@ -409,15 +414,6 @@ static irqreturn_t a3xx_irq(struct msm_gpu *gpu) ~0 /* sentinel */ }; -#ifdef CONFIG_DEBUG_FS -static void a3xx_show (struct msm_gpu *gpu,struct seq_file *m) -{ -seq_printf (m,"status: %08x\n" , -gpu_read (gpu,REG_A3XX_RBBM_STATUS )); -adreno_show (gpu, m); -} -#endif -static struct msm_gpu_state *a3xx_gpu_state_get (struct msm_gpu *gpu) {struct msm_gpu_state *state =adreno_gpu_state_get (gpu);
@@ -454,7 +450,7 @@ static struct msm_gpu_state *a3xx_gpu_state_get(struct msm_gpu *gpu) .irq = a3xx_irq, .destroy = a3xx_destroy, #ifdef CONFIG_DEBUG_FS - .show = a3xx_show, + .show = adreno_show, #endif .gpu_state_get = a3xx_gpu_state_get, .gpu_state_put = adreno_gpu_state_put,
drivers/gpu/drm/msm/adreno/a4xx_gpu.c +8 -13
@@ -298,9 +298,12 @@ static int a4xx_hw_init(struct msm_gpu *gpu)static void a4xx_recover (struct msm_gpu *gpu) {struct drm_printer p =drm_info_printer (gpu->dev->dev); +struct msm_gpu_state *state;int i; -adreno_show_info (gpu, &p); + state = gpu->funcs->gpu_state_get (gpu); + +adreno_show_info (gpu, state, &p);for (i =0 ; i <8 ; i++) { drm_printf(&p, "CP_SCRATCH_REG%d: %u\n", i,
@@ -309,12 +312,14 @@ static void a4xx_recover(struct msm_gpu *gpu)/* dump registers before resetting gpu, if enabled: */ if (hang_debug) -adreno_show_regs (gpu, &p); +adreno_show_regs (gpu, state, &p);gpu_write (gpu,REG_A4XX_RBBM_SW_RESET_CMD ,1 );gpu_read (gpu,REG_A4XX_RBBM_SW_RESET_CMD );gpu_write (gpu,REG_A4XX_RBBM_SW_RESET_CMD ,0 );adreno_recover (gpu); + + gpu->funcs->gpu_state_put (state); }static void a4xx_destroy (struct msm_gpu *gpu)
@@ -454,16 +459,6 @@ static irqreturn_t a4xx_irq(struct msm_gpu *gpu) ~0 /* sentinel */ }; -#ifdef CONFIG_DEBUG_FS -static void a4xx_show (struct msm_gpu *gpu,struct seq_file *m) -{ -seq_printf (m,"status: %08x\n" , -gpu_read (gpu,REG_A4XX_RBBM_STATUS )); -adreno_show (gpu, m); - -} -#endif -static struct msm_gpu_state *a4xx_gpu_state_get (struct msm_gpu *gpu) {struct msm_gpu_state *state =adreno_gpu_state_get (gpu);
@@ -543,7 +538,7 @@ static int a4xx_get_timestamp(struct msm_gpu *gpu, uint64_t *value) .irq = a4xx_irq, .destroy = a4xx_destroy, #ifdef CONFIG_DEBUG_FS - .show = a4xx_show, + .show = adreno_show, #endif .gpu_state_get = a4xx_gpu_state_get, .gpu_state_put = adreno_gpu_state_put,
drivers/gpu/drm/msm/adreno/a5xx_gpu.c +8 -19
@@ -755,21 +755,26 @@ static int a5xx_hw_init(struct msm_gpu *gpu)static void a5xx_recover (struct msm_gpu *gpu) {struct drm_printer p =drm_info_printer (gpu->dev->dev); +struct msm_gpu_state *state;int i; -adreno_show_info (gpu, &p); + state = gpu->funcs->gpu_state_get (gpu); + +adreno_show_info (gpu, state, &p);for (i =0 ; i <8 ; i++)drm_printf (&p,"CP_SCRATCH_REG%d: %u\n" , i,gpu_read (gpu,REG_A5XX_CP_SCRATCH_REG (i)));if (hang_debug) -adreno_show_regs (gpu, &p); +adreno_show_regs (gpu, state, &p);gpu_write (gpu,REG_A5XX_RBBM_SW_RESET_CMD ,1 );gpu_read (gpu,REG_A5XX_RBBM_SW_RESET_CMD );gpu_write (gpu,REG_A5XX_RBBM_SW_RESET_CMD ,0 );adreno_recover (gpu); + + gpu->funcs->gpu_state_put (state); }static void a5xx_destroy (struct msm_gpu *gpu)
@@ -1153,22 +1158,6 @@ static struct msm_gpu_state *a5xx_gpu_state_get(struct msm_gpu *gpu)return state; } -#ifdef CONFIG_DEBUG_FS -static void a5xx_show (struct msm_gpu *gpu,struct seq_file *m) -{ -seq_printf (m,"status: %08x\n" , -gpu_read (gpu,REG_A5XX_RBBM_STATUS )); - -/* -* Temporarily disable hardware clock gating before going into -* adreno_show to avoid issues while reading the registers -*/ -a5xx_set_hwcg (gpu, false); -adreno_show (gpu, m); -a5xx_set_hwcg (gpu, true); -} -#endif -static struct msm_ringbuffer *a5xx_active_ring (struct msm_gpu *gpu) {struct adreno_gpu *adreno_gpu =to_adreno_gpu (gpu);
@@ -1190,7 +1179,7 @@ static struct msm_ringbuffer *a5xx_active_ring(struct msm_gpu *gpu) .irq = a5xx_irq, .destroy = a5xx_destroy, #ifdef CONFIG_DEBUG_FS - .show = a5xx_show, + .show = adreno_show, #endif .gpu_state_get = a5xx_gpu_state_get, .gpu_state_put = adreno_gpu_state_put,
drivers/gpu/drm/msm/adreno/adreno_gpu.c +22 -22
76 lines not on this page · raw message
drivers/gpu/drm/msm/adreno/adreno_gpu.h +6 -3
@@ -211,7 +211,8 @@ void adreno_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit,void adreno_flush (struct msm_gpu *gpu,struct msm_ringbuffer *ring);bool adreno_idle (struct msm_gpu *gpu,struct msm_ringbuffer *ring);#ifdef CONFIG_DEBUG_FS -void adreno_show (struct msm_gpu *gpu,struct seq_file *m); +void adreno_show (struct msm_gpu *gpu,struct msm_gpu_state *state, +struct seq_file *m);#endif void adreno_dump_info (struct msm_gpu *gpu);void adreno_dump (struct msm_gpu *gpu);
@@ -223,8 +224,10 @@ int adreno_gpu_init(struct drm_device *drm, struct platform_device *pdev, int nr_rings);void adreno_gpu_cleanup (struct adreno_gpu *gpu); -void adreno_show_info (struct msm_gpu *gpu,struct drm_printer *p); -void adreno_show_regs (struct msm_gpu *gpu,struct drm_printer *p); +void adreno_show_info (struct msm_gpu *gpu,struct msm_gpu_state *state, +struct drm_printer *p); +void adreno_show_regs (struct msm_gpu *gpu,struct msm_gpu_state *state, +struct drm_printer *p);struct msm_gpu_state *adreno_gpu_state_get (struct msm_gpu *gpu);void adreno_gpu_state_put (struct msm_gpu_state *state);
drivers/gpu/drm/msm/msm_debugfs.c +15 -6
@@ -25,13 +25,22 @@ static int msm_gpu_show(struct drm_device *dev, struct seq_file *m) {struct msm_drm_private *priv = dev->dev_private;struct msm_gpu *gpu = priv->gpu; +struct msm_gpu_state *state; -if (gpu) { -seq_printf (m,"%s Status:\n" , gpu->name); -pm_runtime_get_sync (&gpu->pdev->dev); - gpu->funcs->show (gpu, m); -pm_runtime_put_sync (&gpu->pdev->dev); - } +if (!gpu) +return 0 ; + +pm_runtime_get_sync (&gpu->pdev->dev); + state = gpu->funcs->gpu_state_get (gpu); +pm_runtime_put_sync (&gpu->pdev->dev); + +if (IS_ERR (state)) +return PTR_ERR (state); + +seq_printf (m,"%s Status:\n" , gpu->name); + gpu->funcs->show (gpu, state, m); + + gpu->funcs->gpu_state_put (state);return 0 ; }
drivers/gpu/drm/msm/msm_gpu.h +2 -1
@@ -65,7 +65,8 @@ struct msm_gpu_funcs {void (*destroy)(struct msm_gpu *gpu);#ifdef CONFIG_DEBUG_FS /* show GPU status in debugfs: */ -void (*show)(struct msm_gpu *gpu,struct seq_file *m); +void (*show)(struct msm_gpu *gpu,struct msm_gpu_state *state, +struct seq_file *m);#endif struct msm_gpu_state *(*gpu_state_get)(struct msm_gpu *gpu);void (*gpu_state_put)(struct msm_gpu_state *state);
Signature · 6 lines
1.9.1 _______________________________________________ Freedreno mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/freedreno
PATCH 4/4 drm/msm: gpu: Capture the GPU state on a GPU hang
In reply to a message not in the archiveCapture the GPU state on a GPU hang and store it for later playback using the 'crash' node in the debugfs directory. Only one crash state is stored at a time on the assumption that the first hang is usually the most interesting. The existing crash state can be cleared by writing to the debugfs node and then a new one will be captured on the next hang.
--- drivers/gpu/drm/msm/adreno/a3xx_gpu.c | 1 + drivers/gpu/drm/msm/adreno/a4xx_gpu.c | 1 + drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 1 + drivers/gpu/drm/msm/adreno/adreno_gpu.c | 16 +++++++-- drivers/gpu/drm/msm/adreno/adreno_gpu.h | 2 +- drivers/gpu/drm/msm/msm_debugfs.c | 57 +++++++++++++++++++++++++++++++++ drivers/gpu/drm/msm/msm_gpu.h | 44 ++++++++++++++++++++++++- 7 files changed, 117 insertions(+), 5 deletions(-)
drivers/gpu/drm/msm/adreno/a3xx_gpu.c +1 -0
@@ -320,6 +320,7 @@ static void a3xx_recover(struct msm_gpu *gpu)gpu_write (gpu,REG_A3XX_RBBM_SW_RESET_CMD ,0 );adreno_recover (gpu); +msm_gpu_crashstate_set (gpu, state); gpu->funcs->gpu_state_put (state); }
drivers/gpu/drm/msm/adreno/a4xx_gpu.c +1 -0
@@ -319,6 +319,7 @@ static void a4xx_recover(struct msm_gpu *gpu)gpu_write (gpu,REG_A4XX_RBBM_SW_RESET_CMD ,0 );adreno_recover (gpu); +msm_gpu_crashstate_set (gpu, state); gpu->funcs->gpu_state_put (state); }
drivers/gpu/drm/msm/adreno/a5xx_gpu.c +1 -0
@@ -774,6 +774,7 @@ static void a5xx_recover(struct msm_gpu *gpu)gpu_write (gpu,REG_A5XX_RBBM_SW_RESET_CMD ,0 );adreno_recover (gpu); +msm_gpu_crashstate_set (gpu, state); gpu->funcs->gpu_state_put (state); }
drivers/gpu/drm/msm/adreno/adreno_gpu.c +13 -3
@@ -372,6 +372,8 @@ struct msm_gpu_state *adreno_gpu_state_get(struct msm_gpu *gpu)if (!state)return ERR_PTR (-ENOMEM ); +kref_init (&state->ref); +do_gettimeofday (&state->time);for (i =0 ; i < gpu->nr_rings; i++) {
@@ -407,15 +409,23 @@ struct msm_gpu_state *adreno_gpu_state_get(struct msm_gpu *gpu)return state; } -void adreno_gpu_state_put (struct msm_gpu_state *state) +static void adreno_gpu_state_destroy (struct kref *kref) { -if (IS_ERR_OR_NULL (state)) -return ; +struct msm_gpu_state *state =container_of (kref, + struct msm_gpu_state, ref);kfree (state->registers);kfree (state); } +int adreno_gpu_state_put (struct msm_gpu_state *state) +{ +if (IS_ERR_OR_NULL (state)) +return 1 ; + +return kref_put (&state->ref, adreno_gpu_state_destroy); +} +void adreno_show_info (struct msm_gpu *gpu,struct msm_gpu_state *state,struct drm_printer *p) {
drivers/gpu/drm/msm/adreno/adreno_gpu.h +1 -1
@@ -230,7 +230,7 @@ void adreno_show_regs(struct msm_gpu *gpu, struct msm_gpu_state *state, struct drm_printer *p);struct msm_gpu_state *adreno_gpu_state_get (struct msm_gpu *gpu); -void adreno_gpu_state_put (struct msm_gpu_state *state); +int adreno_gpu_state_put (struct msm_gpu_state *state);/* ringbuffer helpers (the parts that are adreno specific) */
drivers/gpu/drm/msm/msm_debugfs.c +57 -0
74 lines not on this page · raw message
drivers/gpu/drm/msm/msm_gpu.h +43 -1
66 lines not on this page · raw message
Signature · 6 lines
1.9.1 _______________________________________________ Freedreno mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/freedreno
Quoting Jordan Crouse (2018-01-05 18:00:17)
Quoted 10 lines
This is a request for comment on code to store and dump a GPU state a hang with inspiration from the very good i915 GPU error state and the binary GPU snapshot in the downstream kernel.
The goal is to store and provide enough information to debug software and hardware issues on the Adreno hardware in a semi human-readable format that can also be parsed by scripts.
The goal for this request for comment is to get some consensus about the format and work through some of the technical issues.
My biggest regret for i915/error is that we didn't adopt a sensible file format and organically grew it from dmesg-style logging. This is quite a hindrance when it comes to trying to improve the capture whilst maintaining compatibility with the existing tools. Switching to json/yaml at this point won't be too difficult to spot the change in format, just a large chunk of technical debt to pay off. So I would recommend you pick a an adaptable, human readable, file format for ease of tool development.
The second important feature for capturing error state is to include as much user information as possible. You want to be able to identify which library generated the hang in a post-mortem dump from a user in 6-12 months time, and just as importantly, why the library did what it did. I like the idea of userspace being able to attach buffers that are included in the error state (supplied as auxiliary information to the guilty command stream) to provide a flight-data-recorder from the user's pov. So design your interface with a view to extending to include blobs.
It would be interesting to have a common file format... While interpreting the data is going to highly specific to a gpu/driver, the data itself will be similar between drivers. If we had a common file format, we could extend something like mesa/intel/aubinator_error_decode and throw in a bunch of xml descriptors for the different gpus. Just a thought...
-Chris
_______________________________________________
Freedreno mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/freedreno
A message that is not in the archive (sent off-list or not yet received): 151517714292.6838.13295427873358659730-M6iVdVfohj6unts5RBS2dVaTQe2KTcn/@public.gmane.org
Quoted 12 lines · Chris Wilson
Quoting Jordan Crouse (2018-01-05 18:00:17)
Quoted 10 lines
This is a request for comment on code to store and dump a GPU state a hang with inspiration from the very good i915 GPU error state and the binary GPU snapshot in the downstream kernel.
The goal is to store and provide enough information to debug software and hardware issues on the Adreno hardware in a semi human-readable format that can also be parsed by scripts.
The goal for this request for comment is to get some consensus about the format and work through some of the technical issues.
Quoted 7 lines
My biggest regret for i915/error is that we didn't adopt a sensible file format and organically grew it from dmesg-style logging. This is quite a hindrance when it comes to trying to improve the capture whilst maintaining compatibility with the existing tools. Switching to json/yaml at this point won't be too difficult to spot the change in format, just a large chunk of technical debt to pay off. So I would recommend you pick a an adaptable, human readable, file format for ease of tool development.
This is a really great suggestion. The downstream qcom kernel uses a strictly binary format which is also problematic for other reasons. I like the idea of having something standard and extensible while remaining human readable without tools.
Quoted 8 lines
The second important feature for capturing error state is to include as much user information as possible. You want to be able to identify which library generated the hang in a post-mortem dump from a user in 6-12 months time, and just as importantly, why the library did what it did. I like the idea of userspace being able to attach buffers that are included in the error state (supplied as auxiliary information to the guilty command stream) to provide a flight-data-recorder from the user's pov. So design your interface with a view to extending to include blobs.
I love the ascii85 and compression stuff that i915 does and that would fit in well a nice file format as well.
Quoted 6 lines
It would be interesting to have a common file format... While interpreting the data is going to highly specific to a gpu/driver, the data itself will be similar between drivers. If we had a common file format, we could extend something like mesa/intel/aubinator_error_decode and throw in a bunch of xml descriptors for the different gpus. Just a thought...
I'm definitely open to this. There is never anything wrong with improved debugging for everybody.
Thanks,
Jordan
Signature · 6 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project _______________________________________________ Freedreno mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/freedreno
A message that is not in the archive (sent off-list or not yet received): 20180105221119.GA10299-9PYrDHPZ2Orvke4nUoYGnHL1okKdlPRT@public.gmane.org
Quoted 35 lines · Jordan Crouse
Quoted 12 lines · Chris Wilson
Quoting Jordan Crouse (2018-01-05 18:00:17)
Quoted 10 lines
This is a request for comment on code to store and dump a GPU state a hang with inspiration from the very good i915 GPU error state and the binary GPU snapshot in the downstream kernel.
The goal is to store and provide enough information to debug software and hardware issues on the Adreno hardware in a semi human-readable format that can also be parsed by scripts.
The goal for this request for comment is to get some consensus about the format and work through some of the technical issues.
Quoted 7 lines
My biggest regret for i915/error is that we didn't adopt a sensible file format and organically grew it from dmesg-style logging. This is quite a hindrance when it comes to trying to improve the capture whilst maintaining compatibility with the existing tools. Switching to json/yaml at this point won't be too difficult to spot the change in format, just a large chunk of technical debt to pay off. So I would recommend you pick a an adaptable, human readable, file format for ease of tool development.
This is a really great suggestion. The downstream qcom kernel uses a strictly binary format which is also problematic for other reasons. I like the idea of having something standard and extensible while remaining human readable without tools.
Quoted 8 lines
The second important feature for capturing error state is to include as much user information as possible. You want to be able to identify which library generated the hang in a post-mortem dump from a user in 6-12 months time, and just as importantly, why the library did what it did. I like the idea of userspace being able to attach buffers that are included in the error state (supplied as auxiliary information to the guilty command stream) to provide a flight-data-recorder from the user's pov. So design your interface with a view to extending to include blobs.
Quoted 2 lines
I love the ascii85 and compression stuff that i915 does and that would fit in well a nice file format as well.
I guess I should dig out from under a pile of snow and unread email and read your patches.. but if you don't already, including at least the full cmdline of the process that triggered the hang would be hugely useful. This was a big improvement when I added it to hangcheck dbg msgs and hangrd (ie. otherwise 90% of piglit crashes just showed up as "shader_runner" or something equally generic)..
maybe it is possible to capture more of /proc/$pid/* ? I guess that is a good area where code could be shared across drivers (ie. generic sections for /proc/$pid/cmdline and /proc/$pid/maps and whatever else, plus driver specific sections for various register/cmdstream/debug-state dumps)
Perhaps buffer-snapshots could be shared too (at least as far as file format).. I guess usefulness to other drivers depends on how much cmdstream has pointers to other useful stuff (like more cmdstream or shaders). This is probably the sorta thing that should be tunable since sometimes capturing every bo ref'd by the submit could be a bit much.
I'd definitely be open to extending my cmdstream parsing tools to a new format.. the "rd" format was pretty organically grown (although the type/length/value binary format gave enough extensibility that I didn't need to throw it away yet).. but I could definitely see the value in something more human readable yet still parsable to feed into cmdstream/register parsing.
BR,
-R
Quoted 13 lines
Quoted 6 lines
It would be interesting to have a common file format... While interpreting the data is going to highly specific to a gpu/driver, the data itself will be similar between drivers. If we had a common file format, we could extend something like mesa/intel/aubinator_error_decode and throw in a bunch of xml descriptors for the different gpus. Just a thought...
I'm definitely open to this. There is never anything wrong with improved debugging for everybody.
Thanks,
Jordan
Quoted 7 lines
--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
_______________________________________________
Freedreno mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/freedreno