Skip to contentSkip to conversation

drm/amdgpu: Add autodump debugfs node for gpu reset

PATCH In review [email protected] 5 participants 12 messages Apr 23, 2020 – Apr 24, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: Add autodump debugfs node for gpu reset

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
 4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +9 -0
index bc1e0fd71a09..a505b547f242 100644
@@ -724,6 +724,13 @@ struct amd_powerplay {
 	const struct amd_pm_funcs *pp_funcs;
 };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
 #define AMDGPU_RESET_MAGIC_NUM 64
 #define AMDGPU_MAX_DF_PERFMONS 4
 struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
 	char				product_number[16];
 	char				product_name[32];
 	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +85 -0

99 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
 int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
 int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
 int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
 	int i, r = 0;
 	bool need_full_reset  = *need_full_reset_arg;
 
+	amdgpu_debugfs_wait_dump(adev);
+
 	/* block all schedulers and reset given job's ring */
 	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 		struct amdgpu_ring *ring = adev->rings[i];
Signature · 6 lines
2.20.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

Hi Christian

Do you think we need to kill the daemon app if we do KMD unloading ? usually user need to close the app first and then the KMD could be unloaded

If we don't want to manually shutdown the daemon app we can do a "KILL" signal send to that process, or we can implement "read" and let app call "read()" to fetch information like:
1) xxx process hang
2) kmd unloading

And daemon can close() the node if it receives "kmd unloading" instead of doing the dump

Thanks

_____________________________________
Monk Liu|GPU Virtualization Team |AMD

-----Original Message-----
From: Zhao, Jiange <[email protected]>
Sent: Thursday, April 23, 2020 3:20 PM
To: [email protected]
Cc: Koenig, Christian <[email protected]>; Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed.
On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
 4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +9 -0
index bc1e0fd71a09..a505b547f242 100644
@@ -724,6 +724,13 @@ struct amd_powerplay {
 	const struct amd_pm_funcs *pp_funcs;
 };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
 #define AMDGPU_RESET_MAGIC_NUM 64
 #define AMDGPU_MAX_DF_PERFMONS 4
 struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
 	char				product_number[16];
 	char				product_name[32];
 	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c

index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +82 -0

96 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,  int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);  int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);  int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
 	int i, r = 0;
 	bool need_full_reset  = *need_full_reset_arg;
 
+	amdgpu_debugfs_wait_dump(adev);
+
 	/* block all schedulers and reset given job's ring */
 	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 		struct amdgpu_ring *ring = adev->rings[i];

--
2.20.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#2 lore raw

Oh, looks if the daemon is opening the node KMD don't have a chance to enter the path of shutdown/unload driver, thus no chance to return "kmd unloading" to the app...

_____________________________________
Monk Liu|GPU Virtualization Team |AMD

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Liu, Monk
Sent: Thursday, April 23, 2020 3:52 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Deucher, Alexander <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Kuehling, Felix <[email protected]>; Koenig, Christian <[email protected]>; Zhang, Hawking <[email protected]>
Subject: RE: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Hi Christian

Do you think we need to kill the daemon app if we do KMD unloading ? usually user need to close the app first and then the KMD could be unloaded

If we don't want to manually shutdown the daemon app we can do a "KILL" signal send to that process, or we can implement "read" and let app call "read()" to fetch information like:
1) xxx process hang
2) kmd unloading

And daemon can close() the node if it receives "kmd unloading" instead of doing the dump

Thanks

_____________________________________
Monk Liu|GPU Virtualization Team |AMD

-----Original Message-----
From: Zhao, Jiange <[email protected]>
Sent: Thursday, April 23, 2020 3:20 PM
To: [email protected]
Cc: Koenig, Christian <[email protected]>; Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed.
On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
 4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +9 -0
index bc1e0fd71a09..a505b547f242 100644
@@ -724,6 +724,13 @@ struct amd_powerplay {
 	const struct amd_pm_funcs *pp_funcs;
 };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
 #define AMDGPU_RESET_MAGIC_NUM 64
 #define AMDGPU_MAX_DF_PERFMONS 4
 struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
 	char				product_number[16];
 	char				product_name[32];
 	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c

index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +82 -0

96 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,  int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);  int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);  int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
 	int i, r = 0;
 	bool need_full_reset  = *need_full_reset_arg;
 
+	amdgpu_debugfs_wait_dump(adev);
+
 	/* block all schedulers and reset given job's ring */
 	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 		struct amdgpu_ring *ring = adev->rings[i];

--
2.20.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Cmonk.liu%40amd.com%7C2d5beed35028403ebe1708d7e75b4353%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637232251473250650&amp;sdata=SmXKMH9LgbD5K2gkm6Vqysu%2FgvtHLtLoJpJGcLGFd%2F4%3D&amp;reserved=0
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#3 lore raw

Yes, an open file descriptor holds a reference to the driver module.

So it shouldn't be possible to unload the driver while it is open.

Christian.

Am 23.04.20 um 09:54 schrieb Liu, Monk:

Quoted 203 lines

Oh, looks if the daemon is opening the node KMD don't have a chance to enter the path of shutdown/unload driver, thus no chance to return "kmd unloading" to the app...

_____________________________________
Monk Liu|GPU Virtualization Team |AMD

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Liu, Monk
Sent: Thursday, April 23, 2020 3:52 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Deucher, Alexander <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Kuehling, Felix <[email protected]>; Koenig, Christian <[email protected]>; Zhang, Hawking <[email protected]>
Subject: RE: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Hi Christian

Do you think we need to kill the daemon app if we do KMD unloading ? usually user need to close the app first and then the KMD could be unloaded

If we don't want to manually shutdown the daemon app we can do a "KILL" signal send to that process, or we can implement "read" and let app call "read()" to fetch information like:
1) xxx process hang
2) kmd unloading

And daemon can close() the node if it receives "kmd unloading" instead of doing the dump

Thanks

_____________________________________
Monk Liu|GPU Virtualization Team |AMD

-----Original Message-----
From: Zhao, Jiange <[email protected]>
Sent: Thursday, April 23, 2020 3:20 PM
To: [email protected]
Cc: Koenig, Christian <[email protected]>; Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed.
On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
  4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +9 -0
index bc1e0fd71a09..a505b547f242 100644
@@ -724,6 +724,13 @@ struct amd_powerplay {
  	const struct amd_pm_funcs *pp_funcs;
  };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
  #define AMDGPU_RESET_MAGIC_NUM 64
  #define AMDGPU_MAX_DF_PERFMONS 4
  struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
  	char				product_number[16];
  	char				product_name[32];
  	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
  };
 
  static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c

index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +82 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  	return 0;
  }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if
+defined(CONFIG_DEBUG_FS)
+	int ret;
+	unsigned long tmo = 600*HZ;
+
+	if (!adev->autodump.registered)
+		return 0;
+
+	wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+	ret = wait_for_completion_interruptible_timeout(&adev->autodump.completed, tmo);
+	if (ret == 0) { /* time out and dump tool still not finish its dump*/
+		pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+		return -ETIMEDOUT;
+	}
+#endif
+	return 0;
+}
+
  #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct
+file *file) {
+	int ret;
+	struct amdgpu_device *adev;
+
+	ret = simple_open(inode, file);
+	if (ret)
+		return ret;
+
+	adev = file->private_data;
+	if (adev->autodump.registered == true)
+		return -EINVAL;
+
+	adev->autodump.registered = true;
+
+	return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode, struct
+file *file) {
+	struct amdgpu_device *adev = file->private_data;
+
+	complete(&adev->autodump.completed);
+	adev->autodump.registered = false;
+
+	return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct
+poll_table_struct *poll_table) {
+	struct amdgpu_device *adev = file->private_data;
+
+	poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+	if (adev->in_gpu_reset)
+		return POLLIN | POLLRDNORM | POLLWRNORM;
+
+	return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+	.owner = THIS_MODULE,
+	.open = amdgpu_debugfs_autodump_open,
+	.poll = amdgpu_debugfs_autodump_poll,
+	.release = amdgpu_debugfs_autodump_release, };
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev) {
+	struct dentry *entry;
+
+	init_completion(&adev->autodump.completed);
+	init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+	adev->autodump.registered = false;
+
+	entry = debugfs_create_file("autodump", 0600,
+			adev->ddev->primary->debugfs_root,
+			adev, &autodump_debug_fops);
+	adev->autodump.dentry = entry;
+
+	return 0;
+}
+
  /**
   * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
   *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev)
 
  	amdgpu_ras_debugfs_create_all(adev);
 
+	amdgpu_debugfs_autodump_init(adev);
+
  	return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
  					ARRAY_SIZE(amdgpu_debugfs_list));
  }
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,  int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);  int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);  int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -1
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
  	int i, r = 0;
  	bool need_full_reset  = *need_full_reset_arg;
 
+	amdgpu_debugfs_wait_dump(adev);
+
  	/* block all schedulers and reset given job's ring */
  	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
  		struct amdgpu_ring *ring = adev->rings[i];
--

2.20.1

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Cmonk.liu%40amd.com%7C2d5beed35028403ebe1708d7e75b4353%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637232251473250650&amp;sdata=SmXKMH9LgbD5K2gkm6Vqysu%2FgvtHLtLoJpJGcLGFd%2F4%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#4 lore raw
In reply to [email protected]

Am 23.04.20 um 09:19 schrieb [email protected]:

Quoted 31 lines

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
  4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index bc1e0fd71a09..a505b547f242 100644

drivers/gpu/drm/amd/amdgpu/amdgpu.h struct amd_powerplay { · line 729

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
@@ -724,6 +724,13 @@ struct amd_powerplay {
  	const struct amd_pm_funcs *pp_funcs;
  };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;

Registered and completed seems to have the same meaning.

Quoted 21 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +6 -0
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
  #define AMDGPU_RESET_MAGIC_NUM 64
  #define AMDGPU_MAX_DF_PERFMONS 4
  struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
  	char				product_number[16];
  	char				product_name[32];
  	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
  };
 
  static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +0 -0
index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 81

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +5 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  	return 0;
  }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev)
+{
+#if defined(CONFIG_DEBUG_FS)
+	int ret;
+	unsigned long tmo = 600*HZ;

In general please declare constant lines first and variable like "i" or "r" last.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 88

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +7 -0
+
+	if (!adev->autodump.registered)
+		return 0;
+
+	wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+	ret = wait_for_completion_interruptible_timeout(&adev->autodump.completed, tmo);

This is racy, in other words it can happen that a new client opens up the debugfs file without being signaled but blocks the reset here.

You could use two completion structures to avoid that.

Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	if (ret == 0) { /* time out and dump tool still not finish its dump*/
+		pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+		return -ETIMEDOUT;
+	}
+#endif
+	return 0;
+}
+
  #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct file *file)
+{

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 110

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	int ret;
+	struct amdgpu_device *adev;
+
+	ret = simple_open(inode, file);
+	if (ret)
+		return ret;
+
+	adev = file->private_data;
+	if (adev->autodump.registered == true)
+		return -EINVAL;

Probably better to return -EBUSY here. And this is racy, and might need a lock e.g. multiple clients could open the file at the same time.

If we use a struct completion for registered we could use the spinlock of that one for protection here.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 112

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2 -0
+
+	adev->autodump.registered = true;

You also need to reset the completion structure here otherwise only the first GPU reset would work with this.

Quoted 75 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +51 -0
+
+	return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode, struct file *file)
+{
+	struct amdgpu_device *adev = file->private_data;
+
+	complete(&adev->autodump.completed);
+	adev->autodump.registered = false;
+
+	return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct poll_table_struct *poll_table)
+{
+	struct amdgpu_device *adev = file->private_data;
+
+	poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+	if (adev->in_gpu_reset)
+		return POLLIN | POLLRDNORM | POLLWRNORM;
+
+	return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+	.owner = THIS_MODULE,
+	.open = amdgpu_debugfs_autodump_open,
+	.poll = amdgpu_debugfs_autodump_poll,
+	.release = amdgpu_debugfs_autodump_release,
+};
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev)
+{
+	struct dentry *entry;
+
+	init_completion(&adev->autodump.completed);
+	init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+	adev->autodump.registered = false;
+
+	entry = debugfs_create_file("autodump", 0600,
+			adev->ddev->primary->debugfs_root,
+			adev, &autodump_debug_fops);
+	adev->autodump.dentry = entry;
+
+	return 0;
+}
+
  /**
   * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
   *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev)
 
  	amdgpu_ras_debugfs_create_all(adev);
 
+	amdgpu_debugfs_autodump_init(adev);
+
  	return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
  					ARRAY_SIZE(amdgpu_debugfs_list));
  }
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3922

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
  	int i, r = 0;
  	bool need_full_reset  = *need_full_reset_arg;
 
+	amdgpu_debugfs_wait_dump(adev);
+
  	/* block all schedulers and reset given job's ring */
  	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
  		struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#5 lore raw

[AMD Official Use Only - Internal Distribution Only]

Hi Christian,

Here are some explanations,
(1) registered means that an app is listening to this node, completion means that this app has finished a dump.
(2) after a dump, listening app would close this node. If it wants to get another reset hang, it needs to open the node again. In this way, release() function can wrap up and make it ready for another dump.
(3) At the same time, considering the use case of this node, I believe that only the first GPU reset is worthy of a dump.
(4) I didn't implement race condition guard because I believe that this node caters for a cautious super-user and a single client is enough to do all the work. I can add the logic if you think it is necessary.

Jiange

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 23, 2020 4:53 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Am 23.04.20 um 09:19 schrieb [email protected]:

Quoted 32 lines

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
  4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index bc1e0fd71a09..a505b547f242 100644

drivers/gpu/drm/amd/amdgpu/amdgpu.h struct amd_powerplay { · line 729

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
@@ -724,6 +724,13 @@ struct amd_powerplay {
  	const struct amd_pm_funcs *pp_funcs;
  };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;

Registered and completed seems to have the same meaning.

Quoted 23 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +6 -0
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
  #define AMDGPU_RESET_MAGIC_NUM 64
  #define AMDGPU_MAX_DF_PERFMONS 4
  struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
  	char				product_number[16];
  	char				product_name[32];
  	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
  };
 
  static inline struct amdgpu_device *amdgpu_ttm_adev(struct 

ttm_bo_device *bdev) diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 80

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +4 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  	return 0;
  }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if 
+defined(CONFIG_DEBUG_FS)
+	int ret;
+	unsigned long tmo = 600*HZ;

In general please declare constant lines first and variable like "i" or "r" last.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev,

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +9 -0
+
+	if (!adev->autodump.registered)
+		return 0;
+
+	wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+	ret = 
+wait_for_completion_interruptible_timeout(&adev->autodump.completed, 
+tmo);

This is racy, in other words it can happen that a new client opens up the debugfs file without being signaled but blocks the reset here.

You could use two completion structures to avoid that.

Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	if (ret == 0) { /* time out and dump tool still not finish its dump*/
+		pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+		return -ETIMEDOUT;
+	}
+#endif
+	return 0;
+}
+
  #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct 
+file *file) {

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 110

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	int ret;
+	struct amdgpu_device *adev;
+
+	ret = simple_open(inode, file);
+	if (ret)
+		return ret;
+
+	adev = file->private_data;
+	if (adev->autodump.registered == true)
+		return -EINVAL;

Probably better to return -EBUSY here. And this is racy, and might need a lock e.g. multiple clients could open the file at the same time.

If we use a struct completion for registered we could use the spinlock of that one for protection here.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 113

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2 -0
+
+	adev->autodump.registered = true;

You also need to reset the completion structure here otherwise only the first GPU reset would work with this.

Quoted 76 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +49 -0
+
+	return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode, 
+struct file *file) {
+	struct amdgpu_device *adev = file->private_data;
+
+	complete(&adev->autodump.completed);
+	adev->autodump.registered = false;
+
+	return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct 
+poll_table_struct *poll_table) {
+	struct amdgpu_device *adev = file->private_data;
+
+	poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+	if (adev->in_gpu_reset)
+		return POLLIN | POLLRDNORM | POLLWRNORM;
+
+	return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+	.owner = THIS_MODULE,
+	.open = amdgpu_debugfs_autodump_open,
+	.poll = amdgpu_debugfs_autodump_poll,
+	.release = amdgpu_debugfs_autodump_release, };
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev) {
+	struct dentry *entry;
+
+	init_completion(&adev->autodump.completed);
+	init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+	adev->autodump.registered = false;
+
+	entry = debugfs_create_file("autodump", 0600,
+			adev->ddev->primary->debugfs_root,
+			adev, &autodump_debug_fops);
+	adev->autodump.dentry = entry;
+
+	return 0;
+}
+
  /**
   * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
   *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device 
 *adev)
 
  	amdgpu_ras_debugfs_create_all(adev);
 
+	amdgpu_debugfs_autodump_init(adev);
+
  	return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
  					ARRAY_SIZE(amdgpu_debugfs_list));
  }
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3922

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
  	int i, r = 0;
  	bool need_full_reset  = *need_full_reset_arg;
 
+	amdgpu_debugfs_wait_dump(adev);
+
  	/* block all schedulers and reset given job's ring */
  	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
  		struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#6 lore raw

Am 23.04.20 um 12:44 schrieb Zhao, Jiange:

Quoted 6 lines

[AMD Official Use Only - Internal Distribution Only]

Hi Christian,

Here are some explanations,
(1) registered means that an app is listening to this node, completion means that this app has finished a dump.

Yeah and both are the same thing. E.g. when an app is listening the reset code needs to wait and when no app is listening it doesn't.

We could rename the member to something like app_listening if that makes things more clearer.

Quoted 1 line

(2) after a dump, listening app would close this node. If it wants to get another reset hang, it needs to open the node again. In this way, release() function can wrap up and make it ready for another dump.

Yeah, but you are forgetting to reset the completion structure. With the current implementation the second timeout will reset the GPU immediately without waiting for the app to do the core dump.

Quoted 1 line

(3) At the same time, considering the use case of this node, I believe that only the first GPU reset is worthy of a dump.

That is probably not such a bad idea, but we should leave this decision to the end user which can either restart or app or leave it like this.

Quoted 1 line

(4) I didn't implement race condition guard because I believe that this node caters for a cautious super-user and a single client is enough to do all the work. I can add the logic if you think it is necessary.

That is a very dangerous assumption you made here. All kernel code must in general be free of such things.

Regards,
Christian.

Quoted 205 lines

Jiange

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 23, 2020 4:53 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Am 23.04.20 um 09:19 schrieb [email protected]:

Quoted 32 lines

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
   4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index bc1e0fd71a09..a505b547f242 100644

drivers/gpu/drm/amd/amdgpu/amdgpu.h struct amd_powerplay { · line 729

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
@@ -724,6 +724,13 @@ struct amd_powerplay {
   	const struct amd_pm_funcs *pp_funcs;
   };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;

Registered and completed seems to have the same meaning.

Quoted 23 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +6 -0
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
   #define AMDGPU_RESET_MAGIC_NUM 64
   #define AMDGPU_MAX_DF_PERFMONS 4
   struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
   	char				product_number[16];
   	char				product_name[32];
   	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
   };
 
   static inline struct amdgpu_device *amdgpu_ttm_adev(struct

ttm_bo_device *bdev) diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 80

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +4 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
   	return 0;
   }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if
+defined(CONFIG_DEBUG_FS)
+	int ret;
+	unsigned long tmo = 600*HZ;

In general please declare constant lines first and variable like "i" or "r" last.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev,

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +9 -0
+
+	if (!adev->autodump.registered)
+		return 0;
+
+	wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+	ret =
+wait_for_completion_interruptible_timeout(&adev->autodump.completed,
+tmo);

This is racy, in other words it can happen that a new client opens up the debugfs file without being signaled but blocks the reset here.

You could use two completion structures to avoid that.

Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	if (ret == 0) { /* time out and dump tool still not finish its dump*/
+		pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+		return -ETIMEDOUT;
+	}
+#endif
+	return 0;
+}
+
   #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct
+file *file) {

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 110

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	int ret;
+	struct amdgpu_device *adev;
+
+	ret = simple_open(inode, file);
+	if (ret)
+		return ret;
+
+	adev = file->private_data;
+	if (adev->autodump.registered == true)
+		return -EINVAL;

Probably better to return -EBUSY here. And this is racy, and might need a lock e.g. multiple clients could open the file at the same time.

If we use a struct completion for registered we could use the spinlock of that one for protection here.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 113

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2 -0
+
+	adev->autodump.registered = true;

You also need to reset the completion structure here otherwise only the first GPU reset would work with this.

Quoted 79 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +49 -0
+
+	return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode,
+struct file *file) {
+	struct amdgpu_device *adev = file->private_data;
+
+	complete(&adev->autodump.completed);
+	adev->autodump.registered = false;
+
+	return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct
+poll_table_struct *poll_table) {
+	struct amdgpu_device *adev = file->private_data;
+
+	poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+	if (adev->in_gpu_reset)
+		return POLLIN | POLLRDNORM | POLLWRNORM;
+
+	return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+	.owner = THIS_MODULE,
+	.open = amdgpu_debugfs_autodump_open,
+	.poll = amdgpu_debugfs_autodump_poll,
+	.release = amdgpu_debugfs_autodump_release, };
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev) {
+	struct dentry *entry;
+
+	init_completion(&adev->autodump.completed);
+	init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+	adev->autodump.registered = false;
+
+	entry = debugfs_create_file("autodump", 0600,
+			adev->ddev->primary->debugfs_root,
+			adev, &autodump_debug_fops);
+	adev->autodump.dentry = entry;
+
+	return 0;
+}
+
   /**
    * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
    *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device
 *adev)
 
   	amdgpu_ras_debugfs_create_all(adev);
 
+	amdgpu_debugfs_autodump_init(adev);
+
   	return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
   					ARRAY_SIZE(amdgpu_debugfs_list));
   }
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
   int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
   int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
   int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
   	int i, r = 0;
   	bool need_full_reset  = *need_full_reset_arg;
 
Quoted 5 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3922

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
+	amdgpu_debugfs_wait_dump(adev);
+
   	/* block all schedulers and reset given job's ring */
   	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
   		struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#7 lore raw
In reply to Zhao, Jiange

Hi Jiange,

FYI I'm working on adding a new "--autodump" command to umr that uses this feature.
This is not yet merged but you can find the code here: https://gitlab.freedesktop.org/pepp/umr/-/tree/autodump

Quoted 1 line

(3) At the same time, considering the use case of this node, I believe that only the first GPU reset is worthy of a dump.

If it's possible I'd like to be able to do multiple dumps instead of limiting ourselves to only the first one.

Thanks!
Pierre-Eric

Quoted 211 lines

(4) I didn't implement race condition guard because I believe that this node caters for a cautious super-user and a single client is enough to do all the work. I can add the logic if you think it is necessary.

Jiange

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 23, 2020 4:53 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Am 23.04.20 um 09:19 schrieb [email protected]:

Quoted 32 lines

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
  4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index bc1e0fd71a09..a505b547f242 100644

drivers/gpu/drm/amd/amdgpu/amdgpu.h struct amd_powerplay { · line 729

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
@@ -724,6 +724,13 @@ struct amd_powerplay {
  	const struct amd_pm_funcs *pp_funcs;
  };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;

Registered and completed seems to have the same meaning.

Quoted 23 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +6 -0
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
  #define AMDGPU_RESET_MAGIC_NUM 64
  #define AMDGPU_MAX_DF_PERFMONS 4
  struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
  	char				product_number[16];
  	char				product_name[32];
  	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
  };
 
  static inline struct amdgpu_device *amdgpu_ttm_adev(struct 

ttm_bo_device *bdev) diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 80

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +4 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  	return 0;
  }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if 
+defined(CONFIG_DEBUG_FS)
+	int ret;
+	unsigned long tmo = 600*HZ;

In general please declare constant lines first and variable like "i" or "r" last.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev,

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +9 -0
+
+	if (!adev->autodump.registered)
+		return 0;
+
+	wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+	ret = 
+wait_for_completion_interruptible_timeout(&adev->autodump.completed, 
+tmo);

This is racy, in other words it can happen that a new client opens up the debugfs file without being signaled but blocks the reset here.

You could use two completion structures to avoid that.

Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	if (ret == 0) { /* time out and dump tool still not finish its dump*/
+		pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+		return -ETIMEDOUT;
+	}
+#endif
+	return 0;
+}
+
  #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct 
+file *file) {

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 110

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+	int ret;
+	struct amdgpu_device *adev;
+
+	ret = simple_open(inode, file);
+	if (ret)
+		return ret;
+
+	adev = file->private_data;
+	if (adev->autodump.registered == true)
+		return -EINVAL;

Probably better to return -EBUSY here. And this is racy, and might need a lock e.g. multiple clients could open the file at the same time.

If we use a struct completion for registered we could use the spinlock of that one for protection here.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 113

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2 -0
+
+	adev->autodump.registered = true;

You also need to reset the completion structure here otherwise only the first GPU reset would work with this.

Quoted 79 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +49 -0
+
+	return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode, 
+struct file *file) {
+	struct amdgpu_device *adev = file->private_data;
+
+	complete(&adev->autodump.completed);
+	adev->autodump.registered = false;
+
+	return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct 
+poll_table_struct *poll_table) {
+	struct amdgpu_device *adev = file->private_data;
+
+	poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+	if (adev->in_gpu_reset)
+		return POLLIN | POLLRDNORM | POLLWRNORM;
+
+	return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+	.owner = THIS_MODULE,
+	.open = amdgpu_debugfs_autodump_open,
+	.poll = amdgpu_debugfs_autodump_poll,
+	.release = amdgpu_debugfs_autodump_release, };
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev) {
+	struct dentry *entry;
+
+	init_completion(&adev->autodump.completed);
+	init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+	adev->autodump.registered = false;
+
+	entry = debugfs_create_file("autodump", 0600,
+			adev->ddev->primary->debugfs_root,
+			adev, &autodump_debug_fops);
+	adev->autodump.dentry = entry;
+
+	return 0;
+}
+
  /**
   * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
   *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device 
 *adev)
 
  	amdgpu_ras_debugfs_create_all(adev);
 
+	amdgpu_debugfs_autodump_init(adev);
+
  	return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
  					ARRAY_SIZE(amdgpu_debugfs_list));
  }
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
  	int i, r = 0;
  	bool need_full_reset  = *need_full_reset_arg;
 
Quoted 10 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3922

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
+	amdgpu_debugfs_wait_dump(adev);
+
  	/* block all schedulers and reset given job's ring */
  	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
  		struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#8 lore raw

[AMD Official Use Only - Internal Distribution Only]

Hi,

Of course, considering all the suggestions, I will implement a write callback for usermode app to notify KMD that a dump is finished by sending "Done".

In this way, usermode app can do multiple dumps without closing the node,

Jiange
________________________________
From: Pelloux-prayer, Pierre-eric <[email protected]>
Sent: Friday, April 24, 2020 3:46 PM
To: Zhao, Jiange <[email protected]>; Koenig, Christian <[email protected]>; [email protected] <[email protected]>
Cc: Deucher, Alexander <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Kuehling, Felix <[email protected]>; Liu, Monk <[email protected]>; Zhang, Hawking <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Hi Jiange,

FYI I'm working on adding a new "--autodump" command to umr that uses this feature.
This is not yet merged but you can find the code here: https://gitlab.freedesktop.org/pepp/umr/-/tree/autodump

Quoted 1 line

(3) At the same time, considering the use case of this node, I believe that only the first GPU reset is worthy of a dump.

If it's possible I'd like to be able to do multiple dumps instead of limiting ourselves to only the first one.

Thanks!
Pierre-Eric

Quoted 211 lines

(4) I didn't implement race condition guard because I believe that this node caters for a cautious super-user and a single client is enough to do all the work. I can add the logic if you think it is necessary.

Jiange

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 23, 2020 4:53 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Am 23.04.20 um 09:19 schrieb [email protected]:

Quoted 32 lines

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
  4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index bc1e0fd71a09..a505b547f242 100644

drivers/gpu/drm/amd/amdgpu/amdgpu.h struct amd_powerplay { · line 729

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
@@ -724,6 +724,13 @@ struct amd_powerplay {
      const struct amd_pm_funcs *pp_funcs;
  };
 
+struct amdgpu_autodump {
+    bool                            registered;
+    struct completion               completed;

Registered and completed seems to have the same meaning.

Quoted 23 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +6 -0
+    struct dentry                   *dentry;
+    struct wait_queue_head          gpu_hang_wait;
+};
+
  #define AMDGPU_RESET_MAGIC_NUM 64
  #define AMDGPU_MAX_DF_PERFMONS 4
  struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
      char                            product_number[16];
      char                            product_name[32];
      char                            serial[16];
+
+    struct amdgpu_autodump          autodump;
  };
 
  static inline struct amdgpu_device *amdgpu_ttm_adev(struct

ttm_bo_device *bdev) diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 80

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +4 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
      return 0;
  }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if
+defined(CONFIG_DEBUG_FS)
+    int ret;
+    unsigned long tmo = 600*HZ;

In general please declare constant lines first and variable like "i" or "r" last.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev,

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +9 -0
+
+    if (!adev->autodump.registered)
+            return 0;
+
+    wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+    ret =
+wait_for_completion_interruptible_timeout(&adev->autodump.completed,
+tmo);

This is racy, in other words it can happen that a new client opens up the debugfs file without being signaled but blocks the reset here.

You could use two completion structures to avoid that.

Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+    if (ret == 0) { /* time out and dump tool still not finish its dump*/
+            pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+            return -ETIMEDOUT;
+    }
+#endif
+    return 0;
+}
+
  #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct
+file *file) {

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 110

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+    int ret;
+    struct amdgpu_device *adev;
+
+    ret = simple_open(inode, file);
+    if (ret)
+            return ret;
+
+    adev = file->private_data;
+    if (adev->autodump.registered == true)
+            return -EINVAL;

Probably better to return -EBUSY here. And this is racy, and might need a lock e.g. multiple clients could open the file at the same time.

If we use a struct completion for registered we could use the spinlock of that one for protection here.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 113

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2 -0
+
+    adev->autodump.registered = true;

You also need to reset the completion structure here otherwise only the first GPU reset would work with this.

Quoted 79 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +49 -0
+
+    return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode,
+struct file *file) {
+    struct amdgpu_device *adev = file->private_data;
+
+    complete(&adev->autodump.completed);
+    adev->autodump.registered = false;
+
+    return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct
+poll_table_struct *poll_table) {
+    struct amdgpu_device *adev = file->private_data;
+
+    poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+    if (adev->in_gpu_reset)
+            return POLLIN | POLLRDNORM | POLLWRNORM;
+
+    return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+    .owner = THIS_MODULE,
+    .open = amdgpu_debugfs_autodump_open,
+    .poll = amdgpu_debugfs_autodump_poll,
+    .release = amdgpu_debugfs_autodump_release, };
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev) {
+    struct dentry *entry;
+
+    init_completion(&adev->autodump.completed);
+    init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+    adev->autodump.registered = false;
+
+    entry = debugfs_create_file("autodump", 0600,
+                    adev->ddev->primary->debugfs_root,
+                    adev, &autodump_debug_fops);
+    adev->autodump.dentry = entry;
+
+    return 0;
+}
+
  /**
   * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
   *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device
 *adev)
 
      amdgpu_ras_debugfs_create_all(adev);
 
+    amdgpu_debugfs_autodump_init(adev);
+
      return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
                                      ARRAY_SIZE(amdgpu_debugfs_list));
  }
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
  int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
  int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
      int i, r = 0;
      bool need_full_reset  = *need_full_reset_arg;
 
Quoted 10 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3922

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
+    amdgpu_debugfs_wait_dump(adev);
+
      /* block all schedulers and reset given job's ring */
      for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
              struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#9 lore raw

Running "umr --autodump" currently does the following:
- open the debugfs autodump file
- wait for a hang (using poll)
- dump gfx rings
- close the file and exit

I don't think adding support for a "Done" message is necessary, but I'd expect to be able to restart "umr --autodump" and be able to perform another dump.

Pierre-Eric

Quoted 253 lines · Jiange

[AMD Official Use Only - Internal Distribution Only]

Hi,

Of course, considering all the suggestions, I will implement a write callback for usermode app to notify KMD that a dump is finished by sending "Done".

In this way, usermode app can do multiple dumps without closing the node,

Jiange
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*From:* Pelloux-prayer, Pierre-eric <[email protected]>
*Sent:* Friday, April 24, 2020 3:46 PM
*To:* Zhao, Jiange <[email protected]>; Koenig, Christian <[email protected]>; [email protected] <[email protected]>
*Cc:* Deucher, Alexander <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Kuehling, Felix <[email protected]>; Liu, Monk <[email protected]>; Zhang, Hawking <[email protected]> *Subject:* Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Hi Jiange,

FYI I'm working on adding a new "--autodump" command to umr that uses this feature.
This is not yet merged but you can find the code here: https://gitlab.freedesktop.org/pepp/umr/-/tree/autodump

Quoted 1 line

(3) At the same time, considering the use case of this node, I believe that only the first GPU reset is worthy of a dump.

If it's possible I'd like to be able to do multiple dumps instead of limiting ourselves to only the first one.

Thanks!
Pierre-Eric

Quoted 212 lines

(4) I didn't implement race condition guard because I believe that this node caters for a cautious super-user and a single client is enough to do all the work. I can add the logic if you think it is necessary.

Jiange

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 23, 2020 4:53 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Am 23.04.20 um 09:19 schrieb [email protected]:

Quoted 34 lines

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++ drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +

   4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index bc1e0fd71a09..a505b547f242 100644
@@ -724,6 +724,13 @@ struct amd_powerplay {
        const struct amd_pm_funcs *pp_funcs;
    };
Quoted 4 lines


+struct amdgpu_autodump {
+    bool                            registered;
+    struct completion               completed;

Registered and completed seems to have the same meaning.

Quoted 26 lines

+    struct dentry                   *dentry; +    struct wait_queue_head          gpu_hang_wait;

drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -0
+};
+

   #define AMDGPU_RESET_MAGIC_NUM 64
   #define AMDGPU_MAX_DF_PERFMONS 4
   struct amdgpu_device {

drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -0
@@ -990,6 +997,8 @@ struct amdgpu_device {
        char                            product_number[16];
        char                            product_name[32];
        char                            serial[16];
+
+    struct amdgpu_autodump          autodump;
    };


   static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +0 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
        return 0;
    }

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 81

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +4 -0
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if 
+defined(CONFIG_DEBUG_FS)
+    int ret;
+    unsigned long tmo = 600*HZ;

In general please declare constant lines first and variable like "i" or "r" last.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev,

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +9 -0
+
+    if (!adev->autodump.registered)
+            return 0;
+
+    wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+    ret = 
+wait_for_completion_interruptible_timeout(&adev->autodump.completed, 
+tmo);

This is racy, in other words it can happen that a new client opens up the debugfs file without being signaled but blocks the reset here.

You could use two completion structures to avoid that.

Quoted 12 lines

+    if (ret == 0) { /* time out and dump tool still not finish its dump*/ +            pr_err("autodump: timeout before dump finished, move on to gpu recovery\n"); +            return -ETIMEDOUT;
+    }

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +4 -0
+#endif
+    return 0;
+}
+

   #if defined(CONFIG_DEBUG_FS)

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2 -0
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct 
+file *file) {

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 110

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+    int ret;
+    struct amdgpu_device *adev;
+
+    ret = simple_open(inode, file);
+    if (ret)
+            return ret;
+
+    adev = file->private_data;
+    if (adev->autodump.registered == true)
+            return -EINVAL;

Probably better to return -EBUSY here. And this is racy, and might need a lock e.g. multiple clients could open the file at the same time.

If we use a struct completion for registered we could use the spinlock of that one for protection here.

Quoted 2 lines

+
+    adev->autodump.registered = true;

You also need to reset the completion structure here otherwise only the first GPU reset would work with this.

Quoted 80 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +47 -0
+
+    return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode, 
+struct file *file) {
+    struct amdgpu_device *adev = file->private_data;
+
+    complete(&adev->autodump.completed);
+    adev->autodump.registered = false;
+
+    return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct 
+poll_table_struct *poll_table) {
+    struct amdgpu_device *adev = file->private_data;
+
+    poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+    if (adev->in_gpu_reset)
+            return POLLIN | POLLRDNORM | POLLWRNORM;
+
+    return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+    .owner = THIS_MODULE,
+    .open = amdgpu_debugfs_autodump_open,
+    .poll = amdgpu_debugfs_autodump_poll,
+    .release = amdgpu_debugfs_autodump_release, };
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev) {
+    struct dentry *entry;
+
+    init_completion(&adev->autodump.completed);
+    init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+    adev->autodump.registered = false;
+
+    entry = debugfs_create_file("autodump", 0600,
+                    adev->ddev->primary->debugfs_root,
+                    adev, &autodump_debug_fops);
+    adev->autodump.dentry = entry;
+
+    return 0;
+}
+

   /**
    * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes *

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +0 -0
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device 
 *adev)


       amdgpu_ras_debugfs_create_all(adev);

+    amdgpu_debugfs_autodump_init(adev);
+
       return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list, ARRAY_SIZE(amdgpu_debugfs_list)); }

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
    int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
    int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
    int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
        int i, r = 0;
        bool need_full_reset  = *need_full_reset_arg;


+    amdgpu_debugfs_wait_dump(adev);

Quoted 9 lines
Quoted 4 lines

+
       /* block all schedulers and reset given job's ring */ for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 5 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#10 lore raw

I agree. Closing the node is fine, we just need to make sure that reopening it gives us another try again.

Shouldn't be to much of a problem, except for a single line the last version already looked like it should have worked for this.

Christian.

Am 24.04.20 um 10:30 schrieb Pierre-Eric Pelloux-Prayer:

Quoted 6 lines

Running "umr --autodump" currently does the following:
- open the debugfs autodump file
- wait for a hang (using poll)
- dump gfx rings
- close the file and exit

Quoted 257 lines

I don't think adding support for a "Done" message is necessary, but I'd expect to be able to restart "umr --autodump" and be able to perform another dump.

Pierre-Eric

Quoted 251 lines · Jiange

[AMD Official Use Only - Internal Distribution Only]

Hi,

Of course, considering all the suggestions, I will implement a write callback for usermode app to notify KMD that a dump is finished by sending "Done".

In this way, usermode app can do multiple dumps without closing the node,

Jiange
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*From:* Pelloux-prayer, Pierre-eric <[email protected]>
*Sent:* Friday, April 24, 2020 3:46 PM
*To:* Zhao, Jiange <[email protected]>; Koenig, Christian <[email protected]>; [email protected] <[email protected]>
*Cc:* Deucher, Alexander <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Kuehling, Felix <[email protected]>; Liu, Monk <[email protected]>; Zhang, Hawking <[email protected]> *Subject:* Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Hi Jiange,

FYI I'm working on adding a new "--autodump" command to umr that uses this feature.
This is not yet merged but you can find the code here: https://gitlab.freedesktop.org/pepp/umr/-/tree/autodump

Quoted 1 line

(3) At the same time, considering the use case of this node, I believe that only the first GPU reset is worthy of a dump.

If it's possible I'd like to be able to do multiple dumps instead of limiting ourselves to only the first one.

Thanks!
Pierre-Eric

Quoted 206 lines

(4) I didn't implement race condition guard because I believe that this node caters for a cautious super-user and a single client is enough to do all the work. I can add the logic if you think it is necessary.

Jiange

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Thursday, April 23, 2020 4:53 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset

Am 23.04.20 um 09:19 schrieb [email protected]:

Quoted 32 lines

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
    drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
    drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
    drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
    4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index bc1e0fd71a09..a505b547f242 100644

drivers/gpu/drm/amd/amdgpu/amdgpu.h struct amd_powerplay { · line 729

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
@@ -724,6 +724,13 @@ struct amd_powerplay {
        const struct amd_pm_funcs *pp_funcs;
    };
 
+struct amdgpu_autodump {
+    bool                            registered;
+    struct completion               completed;

Registered and completed seems to have the same meaning.

Quoted 23 lines
drivers/gpu/drm/amd/amdgpu/amdgpu.h +6 -0
+    struct dentry                   *dentry;
+    struct wait_queue_head          gpu_hang_wait;
+};
+
    #define AMDGPU_RESET_MAGIC_NUM 64
    #define AMDGPU_MAX_DF_PERFMONS 4
    struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
        char                            product_number[16];
        char                            product_name[32];
        char                            serial[16];
+
+    struct amdgpu_autodump          autodump;
    };
 
    static inline struct amdgpu_device *amdgpu_ttm_adev(struct

ttm_bo_device *bdev) diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894fa3693..cdd4bf00adee 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 80

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +4 -0
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
        return 0;
    }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if
+defined(CONFIG_DEBUG_FS)
+    int ret;
+    unsigned long tmo = 600*HZ;

In general please declare constant lines first and variable like "i" or "r" last.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev,

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +9 -0
+
+    if (!adev->autodump.registered)
+            return 0;
+
+    wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+    ret =
+wait_for_completion_interruptible_timeout(&adev->autodump.completed,
+tmo);

This is racy, in other words it can happen that a new client opens up the debugfs file without being signaled but blocks the reset here.

You could use two completion structures to avoid that.

Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+    if (ret == 0) { /* time out and dump tool still not finish its dump*/
+            pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+            return -ETIMEDOUT;
+    }
+#endif
+    return 0;
+}
+
    #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct
+file *file) {

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 110

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10 -0
+    int ret;
+    struct amdgpu_device *adev;
+
+    ret = simple_open(inode, file);
+    if (ret)
+            return ret;
+
+    adev = file->private_data;
+    if (adev->autodump.registered == true)
+            return -EINVAL;

Probably better to return -EBUSY here. And this is racy, and might need a lock e.g. multiple clients could open the file at the same time.

If we use a struct completion for registered we could use the spinlock of that one for protection here.

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 113

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2 -0
+
+    adev->autodump.registered = true;

You also need to reset the completion structure here otherwise only the first GPU reset would work with this.

Quoted 79 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +49 -0
+
+    return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode,
+struct file *file) {
+    struct amdgpu_device *adev = file->private_data;
+
+    complete(&adev->autodump.completed);
+    adev->autodump.registered = false;
+
+    return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct
+poll_table_struct *poll_table) {
+    struct amdgpu_device *adev = file->private_data;
+
+    poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+    if (adev->in_gpu_reset)
+            return POLLIN | POLLRDNORM | POLLWRNORM;
+
+    return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+    .owner = THIS_MODULE,
+    .open = amdgpu_debugfs_autodump_open,
+    .poll = amdgpu_debugfs_autodump_poll,
+    .release = amdgpu_debugfs_autodump_release, };
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev) {
+    struct dentry *entry;
+
+    init_completion(&adev->autodump.completed);
+    init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+    adev->autodump.registered = false;
+
+    entry = debugfs_create_file("autodump", 0600,
+                    adev->ddev->primary->debugfs_root,
+                    adev, &autodump_debug_fops);
+    adev->autodump.dentry = entry;
+
+    return 0;
+}
+
    /**
     * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
     *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device
 *adev)
 
        amdgpu_ras_debugfs_create_all(adev);
 
+    amdgpu_debugfs_autodump_init(adev);
+
        return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
                                        ARRAY_SIZE(amdgpu_debugfs_list));

   }

drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
    int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
    int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
    int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
        int i, r = 0;
        bool need_full_reset  = *need_full_reset_arg;
 
Quoted 10 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3922

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
+    amdgpu_debugfs_wait_dump(adev);
+
        /* block all schedulers and reset given job's ring */
        for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
                struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#11 lore raw
In reply to [email protected]

Hi,

The build fails for me with this patch applied (poll_wait, POLLIN, POLLRDNORM and POLLWRNORM are undeclared).

Adding "#include <linux/poll.h>" to amdgpu_debugfs.c fixes it.

Pierre-Eric

Quoted 173 lines · On 23/04/2020 09:19, [email protected] wrote:

From: Jiange Zhao <[email protected]>

When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.

A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.

After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().

There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 85 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  2 +
 4 files changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +9 -0
index bc1e0fd71a09..a505b547f242 100644
@@ -724,6 +724,13 @@ struct amd_powerplay {
 	const struct amd_pm_funcs *pp_funcs;
 };
 
+struct amdgpu_autodump {
+	bool				registered;
+	struct completion		completed;
+	struct dentry			*dentry;
+	struct wait_queue_head		gpu_hang_wait;
+};
+
 #define AMDGPU_RESET_MAGIC_NUM 64
 #define AMDGPU_MAX_DF_PERFMONS 4
 struct amdgpu_device {
@@ -990,6 +997,8 @@ struct amdgpu_device {
 	char				product_number[16];
 	char				product_name[32];
 	char				serial[16];
+
+	struct amdgpu_autodump		autodump;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +85 -0
index 1a4894fa3693..cdd4bf00adee 100644
@@ -74,8 +74,91 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
 	return 0;
 }
 
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev)
+{
+#if defined(CONFIG_DEBUG_FS)
+	int ret;
+	unsigned long tmo = 600*HZ;
+
+	if (!adev->autodump.registered)
+		return 0;
+
+	wake_up_interruptible(&adev->autodump.gpu_hang_wait);
+
+	ret = wait_for_completion_interruptible_timeout(&adev->autodump.completed, tmo);
+	if (ret == 0) { /* time out and dump tool still not finish its dump*/
+		pr_err("autodump: timeout before dump finished, move on to gpu recovery\n");
+		return -ETIMEDOUT;
+	}
+#endif
+	return 0;
+}
+
 #if defined(CONFIG_DEBUG_FS)
 
+static int amdgpu_debugfs_autodump_open(struct inode *inode, struct file *file)
+{
+	int ret;
+	struct amdgpu_device *adev;
+
+	ret = simple_open(inode, file);
+	if (ret)
+		return ret;
+
+	adev = file->private_data;
+	if (adev->autodump.registered == true)
+		return -EINVAL;
+
+	adev->autodump.registered = true;
+
+	return 0;
+}
+
+static int amdgpu_debugfs_autodump_release(struct inode *inode, struct file *file)
+{
+	struct amdgpu_device *adev = file->private_data;
+
+	complete(&adev->autodump.completed);
+	adev->autodump.registered = false;
+
+	return 0;
+}
+
+unsigned int amdgpu_debugfs_autodump_poll(struct file *file, struct poll_table_struct *poll_table)
+{
+	struct amdgpu_device *adev = file->private_data;
+
+	poll_wait(file, &adev->autodump.gpu_hang_wait, poll_table);
+
+	if (adev->in_gpu_reset)
+		return POLLIN | POLLRDNORM | POLLWRNORM;
+
+	return 0;
+}
+
+static const struct file_operations autodump_debug_fops = {
+	.owner = THIS_MODULE,
+	.open = amdgpu_debugfs_autodump_open,
+	.poll = amdgpu_debugfs_autodump_poll,
+	.release = amdgpu_debugfs_autodump_release,
+};
+
+static int amdgpu_debugfs_autodump_init(struct amdgpu_device *adev)
+{
+	struct dentry *entry;
+
+	init_completion(&adev->autodump.completed);
+	init_waitqueue_head(&adev->autodump.gpu_hang_wait);
+	adev->autodump.registered = false;
+
+	entry = debugfs_create_file("autodump", 0600,
+			adev->ddev->primary->debugfs_root,
+			adev, &autodump_debug_fops);
+	adev->autodump.dentry = entry;
+
+	return 0;
+}
+
 /**
  * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes
  *
@@ -1434,6 +1517,8 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev)
 
 	amdgpu_ras_debugfs_create_all(adev);
 
+	amdgpu_debugfs_autodump_init(adev);
+
 	return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,
 					ARRAY_SIZE(amdgpu_debugfs_list));
 }
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1 -0
index de12d1101526..9428940a696d 100644
@@ -40,3 +40,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,
 int amdgpu_debugfs_fence_init(struct amdgpu_device *adev);
 int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev);
 int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);
+int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 3d601d5dd5af..44e54ea7af0f 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3922

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
@@ -3915,6 +3915,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
 	int i, r = 0;
 	bool need_full_reset  = *need_full_reset_arg;
 
+	amdgpu_debugfs_wait_dump(adev);
+
 	/* block all schedulers and reset given job's ring */
 	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 		struct amdgpu_ring *ring = adev->rings[i];

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx