Skip to contentSkip to conversation

Improve the dev coredump

PATCH In review [email protected] 2 participants 11 messages Aug 15, 2024 – Aug 19, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Improve the dev coredump

From: Trigger Huang <[email protected]>

The current dev coredump implementation sometimes cannot fully satisfy customer's requirements due to:
1, The enablement of dev coredump is under the control of gpu_recovery. Customer can not do dev coredump with gpu_recovery disabled
2, When job timeout happened, the dump GPU status will be happened after a lot of operations, like soft_reset. The concern here is that the status is not so close to the real GPU's error status.

So we introduced a new solution
1, A new parameter, gpu_coredump, is added to decouple the coredump and gpu reset 2, Do the coredump immediately after a job timeout

Trigger Huang (3):
drm/amdgpu: Add gpu_coredump parameter
drm/amdgpu: introduce new API for GPU core dump
drm/amdgpu: Change the timing of doing coredump

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  3 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 89 ++++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  8 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 10 +++
 4 files changed, 97 insertions(+), 13 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 drm/amdgpu: Add gpu_coredump parameter

From: Trigger Huang <[email protected]>

Add new separate parameter to control GPU coredump procedure. This can be used to decouple the coredump procedure from gpu recovery procedure

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h     | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 8 ++++++++
 2 files changed, 9 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 937de21a7142..4dd465ad14af 100644
@@ -201,6 +201,7 @@ extern uint amdgpu_force_long_training;
 extern int amdgpu_lbpw;
 extern int amdgpu_compute_multipipe;
 extern int amdgpu_gpu_recovery;
+extern int amdgpu_gpu_coredump;
 extern int amdgpu_emu_mode;
 extern uint amdgpu_smu_memory_pool_size;
 extern int amdgpu_smu_pptable_id;
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +8 -0
index b9529948f2b2..c5d357420236 100644
@@ -178,6 +178,7 @@ uint amdgpu_force_long_training;
 int amdgpu_lbpw = -1;
 int amdgpu_compute_multipipe = -1;
 int amdgpu_gpu_recovery = -1; /* auto */
+int amdgpu_gpu_coredump;
 int amdgpu_emu_mode;
 uint amdgpu_smu_memory_pool_size;
 int amdgpu_smu_pptable_id = -1;
@@ -556,6 +557,13 @@ module_param_named(compute_multipipe, amdgpu_compute_multipipe, int, 0444);
 MODULE_PARM_DESC(gpu_recovery, "Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto)");
 module_param_named(gpu_recovery, amdgpu_gpu_recovery, int, 0444);
 
+/**
+ * DOC: gpu_coredump (int)
+ * Set to enable GPU coredump mechanism (1 = enable, 0 = disable). The default is 0
+ */
+MODULE_PARM_DESC(gpu_coredump, "Enable GPU coredump mechanism, (1 = enable, 0 = disable (default))");
+module_param_named(gpu_coredump, amdgpu_gpu_coredump, int, 0444);
+
 /**
  * DOC: emu_mode (int)
  * Set value 1 to enable emulation mode. This is only needed when running on an emulator. The default is 0 (disabled).
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

Quoted 37 lines · On Thu, Aug 15, 2024 at 7:39 AM <[email protected]> wrote:

From: Trigger Huang <[email protected]>

Add new separate parameter to control GPU coredump procedure. This can be used to decouple the coredump procedure from gpu recovery procedure

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h     | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 8 ++++++++
 2 files changed, 9 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 937de21a7142..4dd465ad14af 100644
@@ -201,6 +201,7 @@ extern uint amdgpu_force_long_training;
 extern int amdgpu_lbpw;
 extern int amdgpu_compute_multipipe;
 extern int amdgpu_gpu_recovery;
+extern int amdgpu_gpu_coredump;
 extern int amdgpu_emu_mode;
 extern uint amdgpu_smu_memory_pool_size;
 extern int amdgpu_smu_pptable_id;
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1 -0
index b9529948f2b2..c5d357420236 100644
@@ -178,6 +178,7 @@ uint amdgpu_force_long_training;
 int amdgpu_lbpw = -1;
 int amdgpu_compute_multipipe = -1;
 int amdgpu_gpu_recovery = -1; /* auto */
+int amdgpu_gpu_coredump;
 int amdgpu_emu_mode;
 uint amdgpu_smu_memory_pool_size;
 int amdgpu_smu_pptable_id = -1;

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(compute_multipipe, amdgpu_compute_multipipe, int, 0444); · line 565

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +6 -0
@@ -556,6 +557,13 @@ module_param_named(compute_multipipe, amdgpu_compute_multipipe, int, 0444);
 MODULE_PARM_DESC(gpu_recovery, "Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto)");
 module_param_named(gpu_recovery, amdgpu_gpu_recovery, int, 0444);
 
+/**
+ * DOC: gpu_coredump (int)
+ * Set to enable GPU coredump mechanism (1 = enable, 0 = disable). The default is 0
+ */
+MODULE_PARM_DESC(gpu_coredump, "Enable GPU coredump mechanism, (1 = enable, 0 = disable (default))");
+module_param_named(gpu_coredump, amdgpu_gpu_coredump, int, 0444);

I don't think we need a separate parameter for this, although if we do, this would need to be enabled by default. If it needs to be decoupled from reset, that's fine, but I don't see the need for a separate knob.

Alex

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1 -1
+
 /**
  * DOC: emu_mode (int)
  * Set value 1 to enable emulation mode. This is only needed when running on an emulator. The default is 0 (disabled).
--

2.34.1

#3 lore raw

RE: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

[AMD Official Use Only - AMD Internal Distribution Only]

Quoted 61 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Friday, August 16, 2024 12:02 AM
To: Huang, Trigger <[email protected]>
Cc: [email protected]; Khatri, Sunil <[email protected]>; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c uint amdgpu_force_long_training; int

Quoted 30 lines · On Thu, Aug 15, 2024 at 7:39 AM <[email protected]> wrote:

From: Trigger Huang <[email protected]>

Add new separate parameter to control GPU coredump procedure. This can be used to decouple the coredump procedure from gpu recovery procedure

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h     | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 8 ++++++++
 2 files changed, 9 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index 937de21a7142..4dd465ad14af 100644
@@ -201,6 +201,7 @@ extern uint amdgpu_force_long_training;  extern

int amdgpu_lbpw; extern int amdgpu_compute_multipipe; extern int amdgpu_gpu_recovery;
+extern int amdgpu_gpu_coredump;
extern int amdgpu_emu_mode;
extern uint amdgpu_smu_memory_pool_size; extern int
amdgpu_smu_pptable_id; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index b9529948f2b2..c5d357420236 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -178,6 +178,7 @@ uint amdgpu_force_long_training;  int

amdgpu_lbpw

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(compute_multipipe, · line 557

Quoted 8 lines

= -1; int amdgpu_compute_multipipe = -1; int amdgpu_gpu_recovery = -1; /* auto */

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1 -0
+int amdgpu_gpu_coredump;
 int amdgpu_emu_mode;
 uint amdgpu_smu_memory_pool_size;
 int amdgpu_smu_pptable_id = -1;
@@ -556,6 +557,13 @@ module_param_named(compute_multipipe,
 amdgpu_compute_multipipe, int, 0444);

MODULE_PARM_DESC(gpu_recovery,

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c

Quoted 10 lines

"Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto)"); module_param_named(gpu_recovery, amdgpu_gpu_recovery, int, 0444);

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +6 -0
+/**
+ * DOC: gpu_coredump (int)
+ * Set to enable GPU coredump mechanism (1 = enable, 0 = disable).
+The default is 0  */ MODULE_PARM_DESC(gpu_coredump, "Enable GPU
+coredump mechanism, (1 = enable, 0 = disable (default))");
+module_param_named(gpu_coredump, amdgpu_gpu_coredump, int,

0444);

Quoted 5 lines

I don't think we need a separate parameter for this, although if we do, this would need to be enabled by default. If it needs to be decoupled from reset, that's fine, but I don't see the need for a separate knob.

Alex

Hi Alex,
It is fine to enable it by default
There are several application scenarios that I can think of.
1, Customer may need to do the core dump with gpu_recovery disabled. This can be used for GPU hang debug
2, Customer may need to disable the core dump with gpu_recovery enabled. This can be used for quick GPU recovery, especially for some lightweight hangs that can be processed by soft recovery or per ring reset. 3, Customer may need to enable the core dump with gpu_recovery enabled. This can be used for GPU recovery but record the core dump for further check in stress test or system health check.
It seems not easy to support all the scenarios by only using amdgpu_gpu_coredump, right?

Regards,
Trigger

Quoted 9 lines
Quoted 5 lines

+
/**
* DOC: emu_mode (int)
* Set value 1 to enable emulation mode. This is only needed when running

on an emulator. The default is 0 (disabled).

Quoted 3 lines

--
2.34.1

#4 lore raw

RE: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

[AMD Official Use Only - AMD Internal Distribution Only]

Quoted 84 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Huang, Trigger
Sent: Friday, August 16, 2024 2:36 PM
To: Alex Deucher <[email protected]>
Cc: [email protected]; Khatri, Sunil <[email protected]>; Deucher, Alexander <[email protected]>
Subject: RE: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

[AMD Official Use Only - AMD Internal Distribution Only]

[AMD Official Use Only - AMD Internal Distribution Only]

Quoted 6 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Friday, August 16, 2024 12:02 AM
To: Huang, Trigger <[email protected]>
Cc: [email protected]; Khatri, Sunil
<[email protected]>; Deucher, Alexander

<[email protected]>

Quoted 35 lines

Subject: Re: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c uint amdgpu_force_long_training; int

Quoted 31 lines · On Thu, Aug 15, 2024 at 7:39 AM <[email protected]> wrote:

From: Trigger Huang <[email protected]>

Add new separate parameter to control GPU coredump procedure. This can be used to decouple the coredump procedure from gpu recovery procedure

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h     | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 8 ++++++++
 2 files changed, 9 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index 937de21a7142..4dd465ad14af 100644
@@ -201,6 +201,7 @@ extern uint amdgpu_force_long_training;  extern

int amdgpu_lbpw; extern int amdgpu_compute_multipipe; extern int amdgpu_gpu_recovery;
+extern int amdgpu_gpu_coredump;
extern int amdgpu_emu_mode;
extern uint amdgpu_smu_memory_pool_size; extern int
amdgpu_smu_pptable_id; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index b9529948f2b2..c5d357420236 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -178,6 +178,7 @@ uint amdgpu_force_long_training;  int

amdgpu_lbpw

Quoted 11 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(compute_multipipe, · line 557

Quoted 8 lines

= -1; int amdgpu_compute_multipipe = -1; int amdgpu_gpu_recovery = -1; /* auto */

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1 -0
+int amdgpu_gpu_coredump;
 int amdgpu_emu_mode;
 uint amdgpu_smu_memory_pool_size;
 int amdgpu_smu_pptable_id = -1;
@@ -556,6 +557,13 @@ module_param_named(compute_multipipe,
 amdgpu_compute_multipipe, int, 0444);

MODULE_PARM_DESC(gpu_recovery,

Quoted 2 lines

"Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto)"); module_param_named(gpu_recovery, amdgpu_gpu_recovery,

int,

Quoted 10 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c

Quoted 8 lines

0444);

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +6 -0
+/**
+ * DOC: gpu_coredump (int)
+ * Set to enable GPU coredump mechanism (1 = enable, 0 = disable).
+The default is 0  */ MODULE_PARM_DESC(gpu_coredump, "Enable GPU
+coredump mechanism, (1 = enable, 0 = disable (default))");
+module_param_named(gpu_coredump, amdgpu_gpu_coredump, int,

0444);

Quoted 3 lines

I don't think we need a separate parameter for this, although if we do, this would need to be enabled by default. If it needs to be decoupled from reset, that's fine, but I don't see the need for a separate

knob.

Quoted 2 lines

Alex

Quoted 13 lines

Hi Alex,
It is fine to enable it by default
There are several application scenarios that I can think of.
1, Customer may need to do the core dump with gpu_recovery disabled.
This can be used for GPU hang debug
2, Customer may need to disable the core dump with gpu_recovery
enabled. This can be used for quick GPU recovery, especially for some lightweight hangs that can be processed by soft recovery or per ring reset.
3, Customer may need to enable the core dump with gpu_recovery
enabled. This can be used for GPU recovery but record the core dump for further check in stress test or system health check.
It seems not easy to support all the scenarios by only using amdgpu_gpu_coredump, right?

Sorry, this is a typo, here I mean amdgpu_gpu_recovery, while not amdgpu_gpu_coredump.

Quoted 13 lines

Regards,
Trigger

Quoted 10 lines
Quoted 6 lines

+
/**
* DOC: emu_mode (int)
* Set value 1 to enable emulation mode. This is only needed when
running

on an emulator. The default is 0 (disabled).

Quoted 3 lines

--
2.34.1

#5 lore raw

Re: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

In reply to Huang, Trigger
Quoted 70 lines · Trigger

[AMD Official Use Only - AMD Internal Distribution Only]

Quoted 61 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Friday, August 16, 2024 12:02 AM
To: Huang, Trigger <[email protected]>
Cc: [email protected]; Khatri, Sunil <[email protected]>; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c uint amdgpu_force_long_training; int

Quoted 30 lines · On Thu, Aug 15, 2024 at 7:39 AM <[email protected]> wrote:

From: Trigger Huang <[email protected]>

Add new separate parameter to control GPU coredump procedure. This can be used to decouple the coredump procedure from gpu recovery procedure

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h     | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 8 ++++++++
 2 files changed, 9 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index 937de21a7142..4dd465ad14af 100644
@@ -201,6 +201,7 @@ extern uint amdgpu_force_long_training;  extern

int amdgpu_lbpw; extern int amdgpu_compute_multipipe; extern int amdgpu_gpu_recovery;
+extern int amdgpu_gpu_coredump;
extern int amdgpu_emu_mode;
extern uint amdgpu_smu_memory_pool_size; extern int
amdgpu_smu_pptable_id; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index b9529948f2b2..c5d357420236 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -178,6 +178,7 @@ uint amdgpu_force_long_training;  int

amdgpu_lbpw

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(compute_multipipe, · line 557

Quoted 8 lines

= -1; int amdgpu_compute_multipipe = -1; int amdgpu_gpu_recovery = -1; /* auto */

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1 -0
+int amdgpu_gpu_coredump;
 int amdgpu_emu_mode;
 uint amdgpu_smu_memory_pool_size;
 int amdgpu_smu_pptable_id = -1;
@@ -556,6 +557,13 @@ module_param_named(compute_multipipe,
 amdgpu_compute_multipipe, int, 0444);

MODULE_PARM_DESC(gpu_recovery,

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c

Quoted 10 lines

"Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto)"); module_param_named(gpu_recovery, amdgpu_gpu_recovery, int, 0444);

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +6 -0
+/**
+ * DOC: gpu_coredump (int)
+ * Set to enable GPU coredump mechanism (1 = enable, 0 = disable).
+The default is 0  */ MODULE_PARM_DESC(gpu_coredump, "Enable GPU
+coredump mechanism, (1 = enable, 0 = disable (default))");
+module_param_named(gpu_coredump, amdgpu_gpu_coredump, int,

0444);

Quoted 5 lines

I don't think we need a separate parameter for this, although if we do, this would need to be enabled by default. If it needs to be decoupled from reset, that's fine, but I don't see the need for a separate knob.

Alex

Quoted 7 lines

Hi Alex,
It is fine to enable it by default
There are several application scenarios that I can think of.
1, Customer may need to do the core dump with gpu_recovery disabled. This can be used for GPU hang debug
2, Customer may need to disable the core dump with gpu_recovery enabled. This can be used for quick GPU recovery, especially for some lightweight hangs that can be processed by soft recovery or per ring reset. 3, Customer may need to enable the core dump with gpu_recovery enabled. This can be used for GPU recovery but record the core dump for further check in stress test or system health check.
It seems not easy to support all the scenarios by only using amdgpu_gpu_coredump, right?

We always want devcoredump enabled by default for full adapter resets, otherwise it kind of defeats the purpose of the feature. Do we want devcoredumps for cases where we can recover via soft recovery or per queue reset? If we mainly care about full adapter reset then we can do something like:

1. in amdgpu_job_timedout(), we can do:
if (!amdgpu_gpu_recovery)
amdgpu_dev_coredump()
between per ring reset and full adapter reset. That way it won't get called for soft recovery or per queue reset.

2. leave the current dev_coredump code in place for the case when recovery is enabled.

If we want it for both soft-recovery and queue reset and full adapter reset, then we just just unconditionally call it in amdgpu_job_timedout(). If the finer grained resets don't work, we'll get two dumps, but I think that's probably ok.

Alex

Quoted 12 lines

Regards,
Trigger

Quoted 9 lines
Quoted 5 lines

+
/**
* DOC: emu_mode (int)
* Set value 1 to enable emulation mode. This is only needed when running

on an emulator. The default is 0 (disabled).

Quoted 3 lines

--
2.34.1

#6 lore raw

RE: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

[AMD Official Use Only - AMD Internal Distribution Only]

Quoted 109 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Friday, August 16, 2024 9:43 PM
To: Huang, Trigger <[email protected]>
Cc: [email protected]; Khatri, Sunil <[email protected]>; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

Quoted 51 lines · Trigger

[AMD Official Use Only - AMD Internal Distribution Only]

Quoted 48 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Friday, August 16, 2024 12:02 AM
To: Huang, Trigger <[email protected]>
Cc: [email protected]; Khatri, Sunil <[email protected]>; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH 1/3] drm/amdgpu: Add gpu_coredump parameter

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c uint amdgpu_force_long_training; int

Quoted 31 lines · On Thu, Aug 15, 2024 at 7:39 AM <[email protected]> wrote:

From: Trigger Huang <[email protected]>

Add new separate parameter to control GPU coredump procedure. This can be used to decouple the coredump procedure from gpu recovery procedure

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h     | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 8 ++++++++
 2 files changed, 9 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 937de21a7142..4dd465ad14af 100644
@@ -201,6 +201,7 @@ extern uint amdgpu_force_long_training;
 extern int amdgpu_lbpw;  extern int amdgpu_compute_multipipe;
 extern int amdgpu_gpu_recovery;
+extern int amdgpu_gpu_coredump;
 extern int amdgpu_emu_mode;
 extern uint amdgpu_smu_memory_pool_size;  extern int

amdgpu_smu_pptable_id; diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index b9529948f2b2..c5d357420236 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -178,6 +178,7 @@ uint amdgpu_force_long_training;  int

amdgpu_lbpw

Quoted 6 lines

= -1; int amdgpu_compute_multipipe = -1; int amdgpu_gpu_recovery = -1; /* auto */
+int amdgpu_gpu_coredump;
int amdgpu_emu_mode;
uint amdgpu_smu_memory_pool_size; int amdgpu_smu_pptable_id =
-1; @@ -556,6 +557,13 @@

module_param_named(compute_multipipe,

Quoted 14 lines
Quoted 14 lines
Quoted 1 line

amdgpu_compute_multipipe, int, 0444);

MODULE_PARM_DESC(gpu_recovery,

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c

Quoted 10 lines

"Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto)"); module_param_named(gpu_recovery, amdgpu_gpu_recovery, int, 0444);

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +6 -0
+/**
+ * DOC: gpu_coredump (int)
+ * Set to enable GPU coredump mechanism (1 = enable, 0 = disable).
+The default is 0  */ MODULE_PARM_DESC(gpu_coredump, "Enable GPU
+coredump mechanism, (1 = enable, 0 = disable (default))");
+module_param_named(gpu_coredump, amdgpu_gpu_coredump, int,

0444);

Quoted 3 lines
Quoted 3 lines

I don't think we need a separate parameter for this, although if we do, this would need to be enabled by default. If it needs to be decoupled from reset, that's fine, but I don't see the need for a separate

knob.

Quoted 7 lines
Quoted 2 lines

Alex

Hi Alex,
It is fine to enable it by default
There are several application scenarios that I can think of.
1, Customer may need to do the core dump with gpu_recovery disabled.

This can be used for GPU hang debug

Quoted 1 line

2, Customer may need to disable the core dump with gpu_recovery

enabled. This can be used for quick GPU recovery, especially for some lightweight hangs that can be processed by soft recovery or per ring reset.

Quoted 1 line

3, Customer may need to enable the core dump with gpu_recovery

enabled. This can be used for GPU recovery but record the core dump for further check in stress test or system health check.

Quoted 1 line

It seems not easy to support all the scenarios by only using

amdgpu_gpu_coredump, right?

We always want devcoredump enabled by default for full adapter resets, otherwise it kind of defeats the purpose of the feature. Do we want devcoredumps for cases where we can recover via soft recovery or per queue reset? If we mainly care about full adapter reset then we can do something like:

1. in amdgpu_job_timedout(), we can do:
if (!amdgpu_gpu_recovery)
amdgpu_dev_coredump()
between per ring reset and full adapter reset. That way it won't get called for soft recovery or per queue reset.

Quoted 6 lines

2. leave the current dev_coredump code in place for the case when recovery is enabled.

If we want it for both soft-recovery and queue reset and full adapter reset, then we just just unconditionally call it in amdgpu_job_timedout(). If the finer grained resets don't work, we'll get two dumps, but I think that's probably ok.

OK, that sounds reasonable. Let me drop the new parameter in the new patch.
And I think we want it for all(soft recovery+ queue reset+ full adapter reset) in job timeout scenario , and my only concern is that we need to do the coredump immediately after a job timeout to get a closer representation of GPU's error status. For other cases/scenarios, I will leave the current logic unchanged in the new patch

Thanks,
Trigger

Quoted 16 lines

Alex

Quoted 13 lines

Regards,
Trigger

Quoted 10 lines
Quoted 6 lines

+
/**
* DOC: emu_mode (int)
* Set value 1 to enable emulation mode. This is only needed when
running

on an emulator. The default is 0 (disabled).

Quoted 3 lines

--
2.34.1

#7 lore raw

PATCH 2/3 drm/amdgpu: introduce new API for GPU core dump

In reply to [email protected]

From: Trigger Huang <[email protected]>

Put ip dump and register to dev_coredumpm together

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 73 ++++++++++++++++++++++
 2 files changed, 75 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -0
index 4dd465ad14af..b64284324961 100644
@@ -1401,6 +1401,8 @@ bool amdgpu_device_should_recover_gpu(struct amdgpu_device *adev);
 int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
 			      struct amdgpu_job *job,
 			      struct amdgpu_reset_context *reset_context);
+void amdgpu_device_gpu_coredump(struct amdgpu_device *adev,
+				struct amdgpu_job *job);
 void amdgpu_device_pci_config_reset(struct amdgpu_device *adev);
 int amdgpu_device_pci_reset(struct amdgpu_device *adev);
 bool amdgpu_device_need_post(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +73 -0

85 lines not on this page · raw message

Signature · 1 line
2.34.1
#8 lore raw

PATCH 3/3 drm/amdgpu: Change the timing of doing coredump

In reply to [email protected]

From: Trigger Huang <[email protected]>

Do the coredump immediately after a job timeout to get a closer representation of GPU's error status. For other code paths that need to do the coredump, keep the original logic unchanged, except:
1,All the coredump operations will be under the control of parameter amdgpu_gpu_coredump
2, Do the ip dump and register to dev codedump system together.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 16 +++-------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 10 ++++++++++
 2 files changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3 -13
index 69d6a5b7ca34..a8eb76d82921 100644
@@ -5341,15 +5341,9 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
 			}
 		}
 
-		if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)) {
-			dev_info(tmp_adev->dev, "Dumping IP State\n");
-			/* Trigger ip dump before we reset the asic */
-			for (i = 0; i < tmp_adev->num_ip_blocks; i++)
-				if (tmp_adev->ip_blocks[i].version->funcs->dump_ip_state)
-					tmp_adev->ip_blocks[i].version->funcs
-						->dump_ip_state((void *)tmp_adev);
-			dev_info(tmp_adev->dev, "Dumping IP State Completed\n");
-		}
+		if (amdgpu_gpu_coredump &&
+		    (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)))
+			amdgpu_device_gpu_coredump_single(tmp_adev, job);
 
 		if (need_full_reset)
 			r = amdgpu_device_ip_suspend(adev);
@@ -5444,10 +5438,6 @@ int amdgpu_do_asic_reset(struct list_head *device_list_handle,
 					goto out;
 
 				vram_lost = amdgpu_device_check_vram_lost(tmp_adev);
-
-				if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags))
-					amdgpu_coredump(tmp_adev, vram_lost, reset_context);
-
 				if (vram_lost) {
 					DRM_INFO("VRAM is lost due to GPU reset!\n");
 					amdgpu_inc_vram_lost(tmp_adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +10 -0
index c6a1783fc9ef..63869820c334 100644
@@ -48,6 +48,12 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
 		return DRM_GPU_SCHED_STAT_ENODEV;
 	}
 
+	/*
+	 * Do the coredump immediately after a job timeout to get a closer
+	 * representation of GPU's error status.
+	 */
+	if (amdgpu_gpu_coredump)
+		amdgpu_device_gpu_coredump(adev, job);
 
 	adev->job_hang = true;
 
@@ -101,6 +107,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
 		reset_context.src = AMDGPU_RESET_SRC_JOB;
 		clear_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags);
 
+		/* To avoid a double coredump, as we have already done it */
+		if (amdgpu_gpu_coredump)
+			set_bit(AMDGPU_SKIP_COREDUMP, &reset_context.flags);
+
 		r = amdgpu_device_gpu_recover(ring->adev, job, &reset_context);
 		if (r)
 			dev_err(adev->dev, "GPU Recovery Failed: %d\n", r);
Signature · 1 line
2.34.1
#9 lore raw

Re: [PATCH 3/3] drm/amdgpu: Change the timing of doing coredump

Quoted 40 lines · On Thu, Aug 15, 2024 at 7:50 AM <[email protected]> wrote:

From: Trigger Huang <[email protected]>

Do the coredump immediately after a job timeout to get a closer representation of GPU's error status. For other code paths that need to do the coredump, keep the original logic unchanged, except:
1,All the coredump operations will be under the control of parameter amdgpu_gpu_coredump
2, Do the ip dump and register to dev codedump system together.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 16 +++-------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 10 ++++++++++
 2 files changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3 -9
index 69d6a5b7ca34..a8eb76d82921 100644
@@ -5341,15 +5341,9 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
                        }
                }
 
-               if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)) {
-                       dev_info(tmp_adev->dev, "Dumping IP State\n");
-                       /* Trigger ip dump before we reset the asic */
-                       for (i = 0; i < tmp_adev->num_ip_blocks; i++)
-                               if (tmp_adev->ip_blocks[i].version->funcs->dump_ip_state)
-                                       tmp_adev->ip_blocks[i].version->funcs
-                                               ->dump_ip_state((void *)tmp_adev);
-                       dev_info(tmp_adev->dev, "Dumping IP State Completed\n");
-               }
+               if (amdgpu_gpu_coredump &&
+                   (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)))
+                       amdgpu_device_gpu_coredump_single(tmp_adev, job);
 
                if (need_full_reset)
                        r = amdgpu_device_ip_suspend(adev);

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_do_asic_reset(struct list_head *device_list_handle, · line 5449

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -3
@@ -5444,10 +5438,6 @@ int amdgpu_do_asic_reset(struct list_head *device_list_handle,
                                        goto out;
 
                                vram_lost = amdgpu_device_check_vram_lost(tmp_adev);
-
-                               if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags))
-                                       amdgpu_coredump(tmp_adev, vram_lost, reset_context);

This needs to stay here, otherwise, we won't know the status of vram_lost.

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -1
-
                                if (vram_lost) {
                                        DRM_INFO("VRAM is lost due to GPU reset!\n");
                                        amdgpu_inc_vram_lost(tmp_adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +0 -0
index c6a1783fc9ef..63869820c334 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) · line 56

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +6 -0
@@ -48,6 +48,12 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
                return DRM_GPU_SCHED_STAT_ENODEV;
        }
 
+       /*
+        * Do the coredump immediately after a job timeout to get a closer
+        * representation of GPU's error status.
+        */
+       if (amdgpu_gpu_coredump)
+               amdgpu_device_gpu_coredump(adev, job);

The problem with doing this here is that we miss core dumps for GPU resets that happen for reasons besides a user job timeout. E.g., resets from KFD or a hang due to bad programming in a kernel context.

If you want to keep this here, I'd suggest something like:

if (!amdgpu_gpu_recovery)

     amdgu_core_dump();

That way you'll get a dump in both cases. Maybe add a flag to skip printing vram_lost in this case since it happens before reset so it's irrelevant.

Alex

Quoted 17 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +4 -1
        adev->job_hang = true;
 
@@ -101,6 +107,10 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
                reset_context.src = AMDGPU_RESET_SRC_JOB;
                clear_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags);
 
+               /* To avoid a double coredump, as we have already done it */
+               if (amdgpu_gpu_coredump)
+                       set_bit(AMDGPU_SKIP_COREDUMP, &reset_context.flags);
+
                r = amdgpu_device_gpu_recover(ring->adev, job, &reset_context);
                if (r)
                        dev_err(adev->dev, "GPU Recovery Failed: %d\n", r);
--

2.34.1

#10 lore raw

RE: [PATCH 3/3] drm/amdgpu: Change the timing of doing coredump

[AMD Official Use Only - AMD Internal Distribution Only]

Quoted 61 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Friday, August 16, 2024 12:09 AM
To: Huang, Trigger <[email protected]>
Cc: [email protected]; Khatri, Sunil <[email protected]>; Deucher, Alexander <[email protected]>
Subject: Re: [PATCH 3/3] drm/amdgpu: Change the timing of doing coredump

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_device_pre_asic_reset(struct

Quoted 22 lines · On Thu, Aug 15, 2024 at 7:50 AM <[email protected]> wrote:

From: Trigger Huang <[email protected]>

Do the coredump immediately after a job timeout to get a closer representation of GPU's error status. For other code paths that need to do the coredump, keep the original logic unchanged, except:
1,All the coredump operations will be under the control of parameter amdgpu_gpu_coredump 2, Do the ip dump and register to dev codedump system together.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 16 +++-------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    | 10 ++++++++++
 2 files changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 69d6a5b7ca34..a8eb76d82921 100644
@@ -5341,15 +5341,9 @@ int amdgpu_device_pre_asic_reset(struct

amdgpu_device *adev,

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -7
                        }
                }
 
-               if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)) {
-                       dev_info(tmp_adev->dev, "Dumping IP State\n");
-                       /* Trigger ip dump before we reset the asic */
-                       for (i = 0; i < tmp_adev->num_ip_blocks; i++)
-                               if (tmp_adev->ip_blocks[i].version->funcs->dump_ip_state)
-                                       tmp_adev->ip_blocks[i].version->funcs
-                                               ->dump_ip_state((void *)tmp_adev);

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -3
-                       dev_info(tmp_adev->dev, "Dumping IP State Completed\n");
-               }
+               if (amdgpu_gpu_coredump &&
+                   (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)))
+                       amdgpu_device_gpu_coredump_single(tmp_adev,
+ job);
 
                if (need_full_reset)
                        r = amdgpu_device_ip_suspend(adev); @@
-5444,10 +5438,6 @@ int amdgpu_do_asic_reset(struct list_head

*device_list_handle,

Quoted 8 lines
                                        goto out;

                                vram_lost =

amdgpu_device_check_vram_lost(tmp_adev);

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -2
-
-                               if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context-

flags))
- amdgpu_coredump(tmp_adev, vram_lost,

Quoted 4 lines

reset_context);

This needs to stay here, otherwise, we won't know the status of vram_lost.

Ok, understood, seems vram_lost is the only dependence between core dump and GPU rest. Will not change the logic here.

Quoted 22 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c static enum drm_gpu_sched_stat

Quoted 11 lines

-

                                if (vram_lost) {
                                        DRM_INFO("VRAM is lost due to GPU reset!\n");

amdgpu_inc_vram_lost(tmp_adev); diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index c6a1783fc9ef..63869820c334 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +0 -0
@@ -48,6 +48,12 @@ static enum drm_gpu_sched_stat

amdgpu_job_timedout(struct drm_sched_job *s_job)

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) · line 56

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +6 -0
                return DRM_GPU_SCHED_STAT_ENODEV;
        }
 
+       /*
+        * Do the coredump immediately after a job timeout to get a closer
+        * representation of GPU's error status.
+        */
+       if (amdgpu_gpu_coredump)
+               amdgpu_device_gpu_coredump(adev, job);
Quoted 3 lines

The problem with doing this here is that we miss core dumps for GPU resets that happen for reasons besides a user job timeout. E.g., resets from KFD or a hang due to bad programming in a kernel context.

Ok, I will not change the logic in the code path of other reasons, but I want to add a control of the master switch, amdgpu_gpu_coredump, when doing dump in other code paths.

Quoted 10 lines

If you want to keep this here, I'd suggest something like:

if (!amdgpu_gpu_recovery)

     amdgu_core_dump();

That way you'll get a dump in both cases. Maybe add a flag to skip printing vram_lost in this case since it happens before reset so it's irrelevant.

Alex

Good idea, let me add a flag to skip printing vram_lost in job timeout scenario, but still printing vram_lost in other code patch, like KFD hang. And as described before, there are several application scenarios, it seems only using amdgpu_gpu_recovery can not support all the scenarios

Regards,
Trigger

Quoted 22 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c static enum drm_gpu_sched_stat

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +0 -0
        adev->job_hang = true;
 
@@ -101,6 +107,10 @@ static enum drm_gpu_sched_stat

amdgpu_job_timedout(struct drm_sched_job *s_job)

Quoted 16 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +5 -0
                reset_context.src = AMDGPU_RESET_SRC_JOB;
                clear_bit(AMDGPU_NEED_FULL_RESET,
 &reset_context.flags);
 
+               /* To avoid a double coredump, as we have already done it */
+               if (amdgpu_gpu_coredump)
+                       set_bit(AMDGPU_SKIP_COREDUMP,
+ &reset_context.flags);
+
                r = amdgpu_device_gpu_recover(ring->adev, job, &reset_context);
                if (r)
                        dev_err(adev->dev, "GPU Recovery Failed:

%d\n", r);
--
2.34.1