drm/amdgpu: Add autodump debugfs node for gpu reset v4
Subsystems (2)
PATCH 1/1 drm/amdgpu: Add autodump debugfs node for gpu reset v4
From: Jiange Zhao <[email protected]>
When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.
A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.
After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().
There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.
v2: (1) changed 'registered' to 'app_listening'
(2) add a mutex in open() to prevent race condition
v3 (chk): grab the reset lock to avoid race in autodump_open,
rename debugfs file to amdgpu_autodump, provide autodump_read as well,
style and code cleanups
v4: add 'bool app_listening' to differentiate situations, so that
the node can be reopened; also, there is no need to wait for completion when no app is waiting for a dump.
v5: change 'bool app_listening' to 'enum amdgpu_autodump_state'
add 'app_state_mutex' for race conditions:
(1)Only 1 user can open this file node (2)wait_dump() can only take effect after poll() executed. (3)eliminated the race condition between release() and
wait_dump()
v6: removed 'enum amdgpu_autodump_state' and 'app_state_mutex'
removed state checking in amdgpu_debugfs_wait_dump Improve on top of version 3 so that the node can be reopened.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 78 ++++++++++++++++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h | 6 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 + 4 files changed, 87 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -0
@@ -992,6 +992,8 @@ struct amdgpu_device {char product_number[16 ];char product_name[32 ];char serial[16 ]; + +struct amdgpu_autodump autodump; };static inline struct amdgpu_device *amdgpu_ttm_adev (struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +77 -1
98 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +6 -0
@@ -31,6 +31,11 @@ struct amdgpu_debugfs {unsigned num_files; }; +struct amdgpu_autodump { +struct completion dumping; +struct wait_queue_head gpu_hang; +}; +int amdgpu_debugfs_regs_init (struct amdgpu_device *adev);int amdgpu_debugfs_init (struct amdgpu_device *adev);void amdgpu_debugfs_fini (struct amdgpu_device *adev);
@@ -40,3 +45,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,int amdgpu_debugfs_fence_init (struct amdgpu_device *adev);int amdgpu_debugfs_firmware_init (struct amdgpu_device *adev);int amdgpu_debugfs_gem_init (struct amdgpu_device *adev); +int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
@@ -3927,6 +3927,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,int i, r =0 ;bool need_full_reset = *need_full_reset_arg; +amdgpu_debugfs_wait_dump (adev); +/* block all schedulers and reset given job's ring */ for (i =0 ; i <AMDGPU_MAX_RINGS ; ++i) {struct amdgpu_ring *ring = adev->rings[i];
Signature · 6 lines
2.20.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
[AMD Official Use Only - Internal Distribution Only]
Hi @Koenig, Christian<mailto:[email protected]>,
I made some changes on top of version 3 and tested it. Can you help review?
Jiange
________________________________
From: Zhao, Jiange <[email protected]>
Sent: Saturday, May 9, 2020 5:45 PM
To: [email protected] <[email protected]>
Cc: Koenig, Christian <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]>
Subject: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset v4
From: Jiange Zhao <[email protected]>
When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.
A usermode app would open 'autodump' node under debugfs system and poll() for readable/writable. When a GPU reset is due, amdgpu would notify usermode app through wait_queue_head and give it 10 minutes to dump info.
After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in release().
There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.
v2: (1) changed 'registered' to 'app_listening'
(2) add a mutex in open() to prevent race condition
v3 (chk): grab the reset lock to avoid race in autodump_open,
rename debugfs file to amdgpu_autodump, provide autodump_read as well,
style and code cleanups
v4: add 'bool app_listening' to differentiate situations, so that
the node can be reopened; also, there is no need to wait for completion when no app is waiting for a dump.
v5: change 'bool app_listening' to 'enum amdgpu_autodump_state'
add 'app_state_mutex' for race conditions:
(1)Only 1 user can open this file node (2)wait_dump() can only take effect after poll() executed. (3)eliminated the race condition between release() and
wait_dump()
v6: removed 'enum amdgpu_autodump_state' and 'app_state_mutex'
removed state checking in amdgpu_debugfs_wait_dump Improve on top of version 3 so that the node can be reopened.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 78 ++++++++++++++++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h | 6 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 + 4 files changed, 87 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -0
@@ -992,6 +992,8 @@ struct amdgpu_device {char product_number[16 ];char product_name[32 ];char serial[16 ]; + +struct amdgpu_autodump autodump; };static inline struct amdgpu_device *amdgpu_ttm_adev (struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +77 -1
98 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +6 -0
@@ -31,6 +31,11 @@ struct amdgpu_debugfs {unsigned num_files; }; +struct amdgpu_autodump { +struct completion dumping; +struct wait_queue_head gpu_hang; +}; +int amdgpu_debugfs_regs_init (struct amdgpu_device *adev);int amdgpu_debugfs_init (struct amdgpu_device *adev);void amdgpu_debugfs_fini (struct amdgpu_device *adev);
@@ -40,3 +45,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,int amdgpu_debugfs_fence_init (struct amdgpu_device *adev);int amdgpu_debugfs_firmware_init (struct amdgpu_device *adev);int amdgpu_debugfs_gem_init (struct amdgpu_device *adev); +int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
@@ -3927,6 +3927,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,int i, r =0 ;bool need_full_reset = *need_full_reset_arg; +amdgpu_debugfs_wait_dump (adev); +/* block all schedulers and reset given job's ring */ for (i =0 ; i <AMDGPU_MAX_RINGS ; ++i) {struct amdgpu_ring *ring = adev->rings[i];
--
2.20.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Christian König
maintainerThanks for the reminder, had to much todo yesterday and just forgot about it.
Christian.
Am 13.05.20 um 10:16 schrieb Zhao, Jiange:
Quoted 221 lines
[AMD Official Use Only - Internal Distribution Only]
Hi @Koenig, Christian <mailto:[email protected]>,
I made some changes on top of version 3 and tested it. Can you help review?
Jiange
------------------------------------------------------------------------
*From:* Zhao, Jiange <[email protected]>
*Sent:* Saturday, May 9, 2020 5:45 PM
*To:* [email protected] <[email protected]>
*Cc:* Koenig, Christian <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Deucher, Alexander <[email protected]>; Liu, Monk <[email protected]>; Zhao, Jiange <[email protected]> *Subject:* [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset v4
From: Jiange Zhao <[email protected]>When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.
A usermode app would open 'autodump' node under debugfs system and
poll()for readable/writable. When a GPU reset is due, amdgpu would notify usermode app throughwait_queue_headand give it 10 minutes to dump info.After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in
release().There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.
v2: (1) changed 'registered' to '
app_listening'
(2) add a mutex inopen()to prevent race conditionv3 (chk): grab the reset lock to avoid race in
autodump_open, rename debugfs file toamdgpu_autodump, provideautodump_readas well,
style and code cleanupsv4: add 'bool
app_listening' to differentiate situations, so that the node can be reopened; also, there is no need to wait for completion when no app is waiting for a dump.v5: change 'bool
app_listening' to 'enumamdgpu_autodump_state' add 'app_state_mutex' for race conditions:
(1)Only 1 user can open this file node (2)wait_dump() can only take effect afterpoll()executed. (3)eliminated the race condition betweenrelease()andwait_dump()v6: removed 'enum
amdgpu_autodump_state' and 'app_state_mutex' removed state checking inamdgpu_debugfs_wait_dumpImprove on top of version 3 so that the node can be reopened.Signed-off-by Jiange Zhao <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 +drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c| 78 ++++++++++++++++++++-drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h| 6 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +4 files changed, 87 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +2
-0@@ -992,6 +992,8 @@ struct amdgpu_device { char product_number[16]; char product_name[32]; char serial[16]; + + struct amdgpu_autodump autodump; }; static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +37
-1@@ -27,7 +27,7 @@ #include <linux/pci.h> #include <linux/uaccess.h> #include <linux/pm_runtime.h> - +#include <linux/poll.h> #include <drm/drm_debugfs.h> #include "amdgpu.h"@@ -74,8 +74,82 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev, return 0; } +int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev) +{ +#if defined(CONFIG_DEBUG_FS ) + unsigned long timeout = 600 * HZ; + int ret; + +wake_up_interruptible (&adev->autodump.gpu_hang); + + ret =wait_for_completion_interruptible_timeout (&adev->autodump.dumping, timeout); + complete_all(&adev->autodump.dumping); + if (ret == 0) { + pr_err("autodump: timeout, move on to gpu recovery\n"); + return -ETIMEDOUT; + } +#endif + return 0; +} + #if defined(CONFIG_DEBUG_FS) +static int amdgpu_debugfs_autodump_open (struct inode *inode,struct file *file) +{ + struct amdgpu_device *adev = inode->i_private; + int ret; + + file->private_data = adev; + + mutex_lock(&adev->lock_reset); + if (adev->autodump.dumping.done) + ret = 0; + else + ret = -EBUSY; + mutex_unlock(&adev->lock_reset); + + return ret; +} + +static int amdgpu_debugfs_autodump_release(struct inode *inode,struct file *file)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +8
-0+{ + struct amdgpu_device *adev = file->private_data; + + complete_all(&adev->autodump.dumping); + return 0; +} + +static unsigned int amdgpu_debugfs_autodump_poll(struct file *file,struct
poll_table_struct*poll_table)drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +30
-0+{ + struct amdgpu_device *adev = file->private_data; + +reinit_completion (&adev->autodump.dumping); + poll_wait(file, &adev->autodump.gpu_hang, poll_table); + + if (adev->in_gpu_reset) + return POLLIN | POLLRDNORM | POLLWRNORM; + + return 0; +} + +static const struct file_operations autodump_debug_fops = { + .owner = THIS_MODULE, + .open = amdgpu_debugfs_autodump_open, + .poll = amdgpu_debugfs_autodump_poll, + .release = amdgpu_debugfs_autodump_release, +}; + +static void amdgpu_debugfs_autodump_init (struct amdgpu_device *adev) +{ + init_completion(&adev->autodump.dumping); + complete_all(&adev->autodump.dumping); +init_waitqueue_head (&adev->autodump.gpu_hang); + + debugfs_create_file("amdgpu_autodump", 0600, + adev->ddev->primary->debugfs_root, + adev, &autodump_debug_fops); +} +/**
*amdgpu_debugfs_process_reg_op- Handle MMIO register reads/writes *drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +2
-0@@ -1434,6 +1508,8 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev) amdgpu_ras_debugfs_create_all(adev); + amdgpu_debugfs_autodump_init(adev); + return amdgpu_debugfs_add_files(adev, amdgpu_debugfs_list,ARRAY_SIZE (amdgpu_debugfs_list)); }drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +6
-0@@ -31,6 +31,11 @@ struct amdgpu_debugfs { unsigned num_files; }; +struct amdgpu_autodump { + struct completion dumping; + struct wait_queue_head gpu_hang; +}; + int amdgpu_debugfs_regs_init(struct amdgpu_device *adev); int amdgpu_debugfs_init(struct amdgpu_device *adev); void amdgpu_debugfs_fini(struct amdgpu_device *adev);@@ -40,3 +45,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev, int amdgpu_debugfs_fence_init(struct amdgpu_device *adev); int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev); int amdgpu_debugfs_gem_init(struct amdgpu_device *adev); +int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2
-0@@ -3927,6 +3927,8 @@ static int amdgpu_device_pre_asic_reset(structamdgpu_device *adev, int i, r = 0; bool need_full_reset = *need_full_reset_arg; + amdgpu_debugfs_wait_dump(adev); + /* block all schedulers and reset given job's ring */ for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
Quoted 4 lines
struct
amdgpu_ring*ring = adev->rings[i];Signature · 1 line
2.20.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Am 09.05.20 um 11:45 schrieb [email protected]:
Quoted 119 lines
From: Jiange Zhao <[email protected]>
When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.
A usermode app would open 'autodump' node under debugfs system and
poll()for readable/writable. When a GPU reset is due, amdgpu would notify usermode app throughwait_queue_headand give it 10 minutes to dump info.After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in
release().There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.
v2: (1) changed 'registered' to '
app_listening'
(2) add a mutex inopen()to prevent race conditionv3 (chk): grab the reset lock to avoid race in
autodump_open,
rename debugfs file toamdgpu_autodump, provideautodump_readas well,
style and code cleanupsv4: add 'bool
app_listening' to differentiate situations, so that
the node can be reopened; also, there is no need to wait for completion when no app is waiting for a dump.v5: change 'bool
app_listening' to 'enumamdgpu_autodump_state'
add 'app_state_mutex' for race conditions:
(1)Only 1 user can open this file node (2)wait_dump() can only take effect afterpoll()executed. (3)eliminated the race condition betweenrelease()and
wait_dump()v6: removed 'enum
amdgpu_autodump_state' and 'app_state_mutex'
removed state checking inamdgpu_debugfs_wait_dumpImprove on top of version 3 so that the node can be reopened.Signed-off-by Jiange Zhao <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 78 ++++++++++++++++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h | 6 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 + 4 files changed, 87 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +2
-0@@ -992,6 +992,8 @@ struct amdgpu_device {char product_number[16 ];char product_name[32 ];char serial[16 ]; + +struct amdgpu_autodump autodump; };static inline struct amdgpu_device *amdgpu_ttm_adev (struct ttm_bo_device *bdev)drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +39
-1@@ -27,7 +27,7 @@#include <linux/pci.h> #include <linux/uaccess.h> #include <linux/pm_runtime.h> - +#include <linux/poll.h> #include <drm/drm_debugfs.h> #include "amdgpu.h" @@ -74,8 +74,82 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,return 0 ; } +int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev) +{ +#if defined(CONFIG_DEBUG_FS ) +unsigned long timeout =600 *HZ ; +int ret; + +wake_up_interruptible (&adev->autodump.gpu_hang); + + ret =wait_for_completion_interruptible_timeout (&adev->autodump.dumping, timeout); +complete_all (&adev->autodump.dumping); +if (ret ==0 ) { +pr_err ("autodump: timeout, move on to gpu recovery\n" ); +return -ETIMEDOUT ; + } +#endif +return 0 ; +} +#if defined(CONFIG_DEBUG_FS ) +static int amdgpu_debugfs_autodump_open (struct inode *inode,struct file *file) +{ +struct amdgpu_device *adev = inode->i_private; +int ret; + + file->private_data = adev; + +mutex_lock (&adev->lock_reset); +if (adev->autodump.dumping.done) + ret =0 ; +else + ret = -EBUSY ; +mutex_unlock (&adev->lock_reset); + +return ret; +} + +static int amdgpu_debugfs_autodump_release (struct inode *inode,struct file *file) +{ +struct amdgpu_device *adev = file->private_data;
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 126
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10
-0+ +complete_all (&adev->autodump.dumping); +return 0 ; +} + +static unsigned int amdgpu_debugfs_autodump_poll (struct file *file,struct poll_table_struct *poll_table) +{ +struct amdgpu_device *adev = file->private_data; + +reinit_completion (&adev->autodump.dumping);
Why do you have the reinit_completion here and not in open callback?
Apart from that looks good to me.
Regards,
Christian.
Quoted 64 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +28
-0+poll_wait (file, &adev->autodump.gpu_hang, poll_table); + +if (adev->in_gpu_reset) +return POLLIN |POLLRDNORM |POLLWRNORM ; + +return 0 ; +} + +static const struct file_operations autodump_debug_fops = { + .owner =THIS_MODULE , + .open = amdgpu_debugfs_autodump_open, + .poll = amdgpu_debugfs_autodump_poll, + .release = amdgpu_debugfs_autodump_release, +}; + +static void amdgpu_debugfs_autodump_init (struct amdgpu_device *adev) +{ +init_completion (&adev->autodump.dumping); +complete_all (&adev->autodump.dumping); +init_waitqueue_head (&adev->autodump.gpu_hang); + +debugfs_create_file ("amdgpu_autodump" ,0600 , + adev->ddev->primary->debugfs_root, + adev, &autodump_debug_fops); +} +/** * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes * @@ -1434,6 +1508,8 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev)amdgpu_ras_debugfs_create_all (adev); +amdgpu_debugfs_autodump_init (adev); +return amdgpu_debugfs_add_files (adev, amdgpu_debugfs_list,ARRAY_SIZE (amdgpu_debugfs_list)); }drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +6
-0@@ -31,6 +31,11 @@ struct amdgpu_debugfs {unsigned num_files; }; +struct amdgpu_autodump { +struct completion dumping; +struct wait_queue_head gpu_hang; +}; +int amdgpu_debugfs_regs_init (struct amdgpu_device *adev);int amdgpu_debugfs_init (struct amdgpu_device *adev);void amdgpu_debugfs_fini (struct amdgpu_device *adev);@@ -40,3 +45,4 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev,int amdgpu_debugfs_fence_init (struct amdgpu_device *adev);int amdgpu_debugfs_firmware_init (struct amdgpu_device *adev);int amdgpu_debugfs_gem_init (struct amdgpu_device *adev); +int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3934
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2
-0@@ -3927,6 +3927,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,int i, r =0 ;bool need_full_reset = *need_full_reset_arg; +amdgpu_debugfs_wait_dump (adev); +/* block all schedulers and reset given job's ring */ for (i =0 ; i <AMDGPU_MAX_RINGS ; ++i) {struct amdgpu_ring *ring = adev->rings[i];
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
[AMD Official Use Only - Internal Distribution Only]
Hi Christian,
Since amdgpu_debugfs_wait_dump() would need 'audodump.dumping.done==0' to actually stop and wait for user mode app to dump.
Since usermode app might open a file , do nothing and close it. I believe a poll() function would be a better indicator that the usermode app actually wants to do a dump.
Also, a reset might happen between open() and poll(). The worst case would be wait_dump() would wait until timeout and usermode poll would always fail.
Jiange
-----Original Message-----
From: Christian König <[email protected]>
Sent: Wednesday, May 13, 2020 4:20 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Deucher, Alexander <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Zhao, Jiange <[email protected]>; Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset v4
Am 09.05.20 um 11:45 schrieb [email protected]:
Quoted 121 lines
From: Jiange Zhao <[email protected]>
When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.
A usermode app would open 'autodump' node under debugfs system and
poll()for readable/writable. When a GPU reset is due, amdgpu would notify usermode app throughwait_queue_headand give it 10 minutes to dump info.After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in
release().There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.
v2: (1) changed 'registered' to '
app_listening'
(2) add a mutex inopen()to prevent race conditionv3 (chk): grab the reset lock to avoid race in
autodump_open,
rename debugfs file toamdgpu_autodump, provideautodump_readas well,
style and code cleanupsv4: add 'bool
app_listening' to differentiate situations, so that
the node can be reopened; also, there is no need to wait for completion when no app is waiting for a dump.v5: change 'bool
app_listening' to 'enumamdgpu_autodump_state'
add 'app_state_mutex' for race conditions:
(1)Only 1 user can open this file node (2)wait_dump() can only take effect afterpoll()executed. (3)eliminated the race condition betweenrelease()and
wait_dump()v6: removed 'enum
amdgpu_autodump_state' and 'app_state_mutex'
removed state checking inamdgpu_debugfs_wait_dumpImprove on top of version 3 so that the node can be reopened.Signed-off-by Jiange Zhao <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 78 ++++++++++++++++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h | 6 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 + 4 files changed, 87 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +2
-0@@ -992,6 +992,8 @@ struct amdgpu_device {char product_number[16 ];char product_name[32 ];char serial[16 ]; + +struct amdgpu_autodump autodump; }; static inline struct amdgpu_device *amdgpu_ttm_adev(struct
ttm_bo_device*bdev) diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index1a4894fa3693..261b67ece7fb100644drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +38
-1@@ -27,7 +27,7 @@#include <linux/pci.h> #include <linux/uaccess.h> #include <linux/pm_runtime.h> - +#include <linux/poll.h> #include <drm/drm_debugfs.h> #include "amdgpu.h" @@ -74,8 +74,82 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev, return 0; } +int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if +defined(CONFIG_DEBUG_FS) + unsigned long timeout = 600 * HZ; + int ret; + + wake_up_interruptible(&adev->autodump.gpu_hang); + + ret = wait_for_completion_interruptible_timeout(&adev->autodump.dumping, timeout); + complete_all(&adev->autodump.dumping); + if (ret == 0) { + pr_err("autodump: timeout, move on to gpu recovery\n"); + return -ETIMEDOUT; + } +#endif + return 0; +} + #if defined(CONFIG_DEBUG_FS) +static int amdgpu_debugfs_autodump_open(struct inode *inode, struct +file *file) { + struct amdgpu_device *adev = inode->i_private; + int ret; + + file->private_data = adev; + + mutex_lock(&adev->lock_reset); + if (adev->autodump.dumping.done) + ret = 0; + else + ret = -EBUSY; + mutex_unlock(&adev->lock_reset); + + return ret; +} + +static int amdgpu_debugfs_autodump_release(struct inode *inode, +struct file *file) { + struct amdgpu_device *adev = file->private_data;
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 125
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10
-0+ +complete_all (&adev->autodump.dumping); +return 0 ; +} + +static unsigned int amdgpu_debugfs_autodump_poll (struct file *file, +struct poll_table_struct *poll_table) { +struct amdgpu_device *adev = file->private_data; + +reinit_completion (&adev->autodump.dumping);
Why do you have the reinit_completion here and not in open callback?
Apart from that looks good to me.
Regards,
Christian.
Quoted 66 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +27
-0+poll_wait (file, &adev->autodump.gpu_hang, poll_table); + +if (adev->in_gpu_reset) +return POLLIN |POLLRDNORM |POLLWRNORM ; + +return 0 ; +} + +static const struct file_operations autodump_debug_fops = { + .owner =THIS_MODULE , + .open = amdgpu_debugfs_autodump_open, + .poll = amdgpu_debugfs_autodump_poll, + .release = amdgpu_debugfs_autodump_release, }; + +static void amdgpu_debugfs_autodump_init (struct amdgpu_device *adev) +{ +init_completion (&adev->autodump.dumping); +complete_all (&adev->autodump.dumping); +init_waitqueue_head (&adev->autodump.gpu_hang); + +debugfs_create_file ("amdgpu_autodump" ,0600 , + adev->ddev->primary->debugfs_root, + adev, &autodump_debug_fops); +} +/** * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes * @@ -1434,6 +1508,8 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev)amdgpu_ras_debugfs_create_all (adev); +amdgpu_debugfs_autodump_init (adev); +return amdgpu_debugfs_add_files (adev, amdgpu_debugfs_list,ARRAY_SIZE (amdgpu_debugfs_list));}drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +5
-0@@ -31,6 +31,11 @@ struct amdgpu_debugfs {unsigned num_files; }; +struct amdgpu_autodump { +struct completion dumping; +struct wait_queue_head gpu_hang; +}; +int amdgpu_debugfs_regs_init (struct amdgpu_device *adev);int amdgpu_debugfs_init (struct amdgpu_device *adev); void amdgpu_debugfs_fini(struct amdgpu_device *adev); @@ -40,3 +45,4@@ int amdgpu_debugfs_add_files(struct
amdgpu_device*adev,
intamdgpu_debugfs_fence_init(struct amdgpu_device *adev); intamdgpu_debugfs_firmware_init(struct amdgpu_device *adev); intamdgpu_debugfs_gem_init(struct amdgpu_device *adev);drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1
-0+int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3934
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2
-0@@ -3927,6 +3927,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,int i, r =0 ;bool need_full_reset = *need_full_reset_arg; +amdgpu_debugfs_wait_dump (adev); +/* block all schedulers and reset given job's ring */ for (i =0 ; i <AMDGPU_MAX_RINGS ; ++i) {struct amdgpu_ring *ring = adev->rings[i];
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Christian König
maintainerQuoted 1 line
Since usermode app might open a file , do nothing and close it.
That case is unproblematic since closing the debugfs file sets the state of the struct completion to completed again no matter if we waited or not.
But when you don't reset in the open() callback we open a small windows between open and poll where userspace could open the debugfs file twice.
Regards,
Christian.
Am 13.05.20 um 11:37 schrieb Zhao, Jiange:
Quoted 227 lines
[AMD Official Use Only - Internal Distribution Only]
Hi Christian,
Since
amdgpu_debugfs_wait_dump()would need 'audodump.dumping.done==0' to actually stop and wait for user mode app to dump.Since usermode app might open a file , do nothing and close it. I believe a
poll()function would be a better indicator that the usermode app actually wants to do a dump.Also, a reset might happen between
open()andpoll(). The worst case would bewait_dump()would wait until timeout and usermode poll would always fail.Jiange
-----Original Message-----
From: Christian König <[email protected]>
Sent: Wednesday, May 13, 2020 4:20 PM
To: Zhao, Jiange <[email protected]>; [email protected]
Cc: Deucher, Alexander <[email protected]>; Pelloux-prayer, Pierre-eric <[email protected]>; Zhao, Jiange <[email protected]>; Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Add autodump debugfs node for gpu reset v4Am 09.05.20 um 11:45 schrieb [email protected]:
Quoted 121 lines
From: Jiange Zhao <[email protected]>
When GPU got timeout, it would notify an interested part of an opportunity to dump info before actual GPU reset.
A usermode app would open 'autodump' node under debugfs system and
poll()for readable/writable. When a GPU reset is due, amdgpu would notify usermode app throughwait_queue_headand give it 10 minutes to dump info.After usermode app has done its work, this 'autodump' node is closed. On node closure, amdgpu gets to know the dump is done through the completion that is triggered in
release().There is no write or read callback because necessary info can be obtained through dmesg and umr. Messages back and forth between usermode app and amdgpu are unnecessary.
v2: (1) changed 'registered' to '
app_listening'
(2) add a mutex inopen()to prevent race conditionv3 (chk): grab the reset lock to avoid race in
autodump_open,
rename debugfs file toamdgpu_autodump, provideautodump_readas well,
style and code cleanupsv4: add 'bool
app_listening' to differentiate situations, so that
the node can be reopened; also, there is no need to wait for completion when no app is waiting for a dump.v5: change 'bool
app_listening' to 'enumamdgpu_autodump_state'
add 'app_state_mutex' for race conditions:
(1)Only 1 user can open this file node (2)wait_dump() can only take effect afterpoll()executed. (3)eliminated the race condition betweenrelease()and
wait_dump()v6: removed 'enum
amdgpu_autodump_state' and 'app_state_mutex'
removed state checking inamdgpu_debugfs_wait_dumpImprove on top of version 3 so that the node can be reopened.Signed-off-by Jiange Zhao <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 78 ++++++++++++++++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h | 6 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 + 4 files changed, 87 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +2
-0@@ -992,6 +992,8 @@ struct amdgpu_device {char product_number[16 ];char product_name[32 ];char serial[16 ]; + +struct amdgpu_autodump autodump; }; static inline struct amdgpu_device *amdgpu_ttm_adev(struct
ttm_bo_device*bdev) diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index1a4894fa3693..261b67ece7fb100644drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +38
-1@@ -27,7 +27,7 @@#include <linux/pci.h> #include <linux/uaccess.h> #include <linux/pm_runtime.h> - +#include <linux/poll.h> #include <drm/drm_debugfs.h> #include "amdgpu.h" @@ -74,8 +74,82 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev, return 0; } +int amdgpu_debugfs_wait_dump(struct amdgpu_device *adev) { #if +defined(CONFIG_DEBUG_FS) + unsigned long timeout = 600 * HZ; + int ret; + + wake_up_interruptible(&adev->autodump.gpu_hang); + + ret = wait_for_completion_interruptible_timeout(&adev->autodump.dumping, timeout); + complete_all(&adev->autodump.dumping); + if (ret == 0) { + pr_err("autodump: timeout, move on to gpu recovery\n"); + return -ETIMEDOUT; + } +#endif + return 0; +} + #if defined(CONFIG_DEBUG_FS) +static int amdgpu_debugfs_autodump_open(struct inode *inode, struct +file *file) { + struct amdgpu_device *adev = inode->i_private; + int ret; + + file->private_data = adev; + + mutex_lock(&adev->lock_reset); + if (adev->autodump.dumping.done) + ret = 0; + else + ret = -EBUSY; + mutex_unlock(&adev->lock_reset); + + return ret; +} + +static int amdgpu_debugfs_autodump_release(struct inode *inode, +struct file *file) { + struct amdgpu_device *adev = file->private_data;drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c int amdgpu_debugfs_add_files(struct amdgpu_device *adev, · line 125
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +10
-0+ +complete_all (&adev->autodump.dumping); +return 0 ; +} + +static unsigned int amdgpu_debugfs_autodump_poll (struct file *file, +struct poll_table_struct *poll_table) { +struct amdgpu_device *adev = file->private_data; + +reinit_completion (&adev->autodump.dumping);Why do you have the
reinit_completionhere and not in open callback?Apart from that looks good to me.
Regards,
Christian.Quoted 69 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +27
-0+poll_wait (file, &adev->autodump.gpu_hang, poll_table); + +if (adev->in_gpu_reset) +return POLLIN |POLLRDNORM |POLLWRNORM ; + +return 0 ; +} + +static const struct file_operations autodump_debug_fops = { + .owner =THIS_MODULE , + .open = amdgpu_debugfs_autodump_open, + .poll = amdgpu_debugfs_autodump_poll, + .release = amdgpu_debugfs_autodump_release, }; + +static void amdgpu_debugfs_autodump_init (struct amdgpu_device *adev) +{ +init_completion (&adev->autodump.dumping); +complete_all (&adev->autodump.dumping); +init_waitqueue_head (&adev->autodump.gpu_hang); + +debugfs_create_file ("amdgpu_autodump" ,0600 , + adev->ddev->primary->debugfs_root, + adev, &autodump_debug_fops); +} +/** * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes * @@ -1434,6 +1508,8 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev)amdgpu_ras_debugfs_create_all (adev); +amdgpu_debugfs_autodump_init (adev); +return amdgpu_debugfs_add_files (adev, amdgpu_debugfs_list,ARRAY_SIZE (amdgpu_debugfs_list));}drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +5
-0@@ -31,6 +31,11 @@ struct amdgpu_debugfs {unsigned num_files; }; +struct amdgpu_autodump { +struct completion dumping; +struct wait_queue_head gpu_hang; +}; +int amdgpu_debugfs_regs_init (struct amdgpu_device *adev);int amdgpu_debugfs_init (struct amdgpu_device *adev); void amdgpu_debugfs_fini(struct amdgpu_device *adev); @@ -40,3 +45,4@@ int amdgpu_debugfs_add_files(struct
amdgpu_device*adev,int amdgpu_debugfs_fence_init(struct amdgpu_device *adev); int amdgpu_debugfs_firmware_init(struct amdgpu_device *adev); int amdgpu_debugfs_gem_init(struct amdgpu_device *adev);drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.h +1
-0+int amdgpu_debugfs_wait_dump (struct amdgpu_device *adev);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -3927,6 +3927,8 @@ static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,int i, r =0 ;bool need_full_reset = *need_full_reset_arg;
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · line 3934
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2
-0+amdgpu_debugfs_wait_dump (adev); +/* block all schedulers and reset given job's ring */ for (i =0 ; i <AMDGPU_MAX_RINGS ; ++i) {struct amdgpu_ring *ring = adev->rings[i];
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx