Skip to contentSkip to conversation

drm/amdgpu: bail on INFO IOCTL if the GPU is in reset

PATCH In review Alex Deucher 3 participants 6 messages Jan 29, 2024 – Feb 15, 2024

Subsystems (2)
lore
Share & tools
lore

Alex Deucher

author maintainer
#0 lore raw

PATCH 1/1 drm/amdgpu: bail on INFO IOCTL if the GPU is in reset

This avoids queries to read registers or query the SMU for telemetry data while the GPU is in reset. This mirrors what we already do for sysfs.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
index a2df3025a754..d522e99c6f81 100644
@@ -607,6 +607,9 @@ int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
 	int i, found, ret;
 	int ui32_size = sizeof(ui32);
 
+	if (amdgpu_in_reset(adev))
+		return -EPERM;
+
 	if (!info->return_size || !info->return_pointer)
 		return -EINVAL;
 
Signature · 1 line
2.42.0
author maintainer
#1 lore raw

[AMD Official Use Only - General]

Ping?

Quoted 33 lines

-----Original Message-----
From: Deucher, Alexander <[email protected]>
Sent: Monday, January 29, 2024 10:56 AM
To: [email protected]
Cc: Deucher, Alexander <[email protected]>
Subject: [PATCH] drm/amdgpu: bail on INFO IOCTL if the GPU is in reset

This avoids queries to read registers or query the SMU for telemetry data while the GPU is in reset. This mirrors what we already do for sysfs.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -1
index a2df3025a754..d522e99c6f81 100644
@@ -607,6 +607,9 @@ int amdgpu_info_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
      int i, found, ret;
      int ui32_size = sizeof(ui32);
 
+     if (amdgpu_in_reset(adev))
+             return -EPERM;
+
      if (!info->return_size || !info->return_pointer)
              return -EINVAL;
 
--

2.42.0

#2 lore raw

Well using this is in sysfs is a bug to begin with. This would prevent starting new applications and crashing applications which don't expect to get an -EPERM in return here.

If we need to make operations mutual exclusive with resets then we need to take the appropriate locks and *not* work around by abusing amdgpu_in_reset().

The functionality of amdgpu_in_reset() is just to check in lower level functions if we are inside the higher level reset thread and *not* protect anybody from concurrent access.

I think we should probably completely nuke the underlying flag and using the thread owner of the lock to prevent such an abuse.

Regards,
Christian.

Am 12.02.24 um 21:56 schrieb Deucher, Alexander:

Quoted 37 lines

[AMD Official Use Only - General]

Ping?

Quoted 33 lines

-----Original Message-----
From: Deucher, Alexander <[email protected]>
Sent: Monday, January 29, 2024 10:56 AM
To: [email protected]
Cc: Deucher, Alexander <[email protected]>
Subject: [PATCH] drm/amdgpu: bail on INFO IOCTL if the GPU is in reset

This avoids queries to read registers or query the SMU for telemetry data while the GPU is in reset. This mirrors what we already do for sysfs.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -1
index a2df3025a754..d522e99c6f81 100644
@@ -607,6 +607,9 @@ int amdgpu_info_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
       int i, found, ret;
       int ui32_size = sizeof(ui32);
 
+     if (amdgpu_in_reset(adev))
+             return -EPERM;
+
       if (!info->return_size || !info->return_pointer)
               return -EINVAL;
 
--

2.42.0

#3 lore raw
Quoted 9 lines · Christian König

Well using this is in sysfs is a bug to begin with. This would prevent starting new applications and crashing applications which don't expect to get an -EPERM in return here.

If we need to make operations mutual exclusive with resets then we need to take the appropriate locks and *not* work around by abusing amdgpu_in_reset().

Quoted 6 lines

The functionality of amdgpu_in_reset() is just to check in lower level functions if we are inside the higher level reset thread and *not* protect anybody from concurrent access.

I think we should probably completely nuke the underlying flag and using the thread owner of the lock to prevent such an abuse.

Can we land some variant of this for now? It fixes known issues and it's the same behavior we have in sysfs and debugfs already. It's not clear to me how this should best be handled. We basically want to block any access to the GPU (registers, firmwares, etc.) while the GPU is going through a reset. Just locking the reset domain doesn't seem like the right solution.

Alex

Quoted 43 lines

Regards,
Christian.

Am 12.02.24 um 21:56 schrieb Deucher, Alexander:

Quoted 37 lines

[AMD Official Use Only - General]

Ping?

Quoted 33 lines

-----Original Message-----
From: Deucher, Alexander <[email protected]>
Sent: Monday, January 29, 2024 10:56 AM
To: [email protected]
Cc: Deucher, Alexander <[email protected]>
Subject: [PATCH] drm/amdgpu: bail on INFO IOCTL if the GPU is in reset

This avoids queries to read registers or query the SMU for telemetry data while the GPU is in reset. This mirrors what we already do for sysfs.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -1
index a2df3025a754..d522e99c6f81 100644
@@ -607,6 +607,9 @@ int amdgpu_info_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
       int i, found, ret;
       int ui32_size = sizeof(ui32);
 
+     if (amdgpu_in_reset(adev))
+             return -EPERM;
+
       if (!info->return_size || !info->return_pointer)
               return -EINVAL;
 
--

2.42.0

#4 lore raw

Am 15.02.24 um 15:36 schrieb Alex Deucher:

Quoted 10 lines
Quoted 8 lines · Christian König

Well using this is in sysfs is a bug to begin with. This would prevent starting new applications and crashing applications which don't expect to get an -EPERM in return here.

If we need to make operations mutual exclusive with resets then we need to take the appropriate locks and *not* work around by abusing amdgpu_in_reset().

Quoted 7 lines
Quoted 6 lines

The functionality of amdgpu_in_reset() is just to check in lower level functions if we are inside the higher level reset thread and *not* protect anybody from concurrent access.

I think we should probably completely nuke the underlying flag and using the thread owner of the lock to prevent such an abuse.

Can we land some variant of this for now?

I don't think so, it most likely will break existing use cases.

What we might be able to do is to frame this with amdgpu_device_lock_reset_domain() / amdgpu_device_unlock_reset_domain().

Quoted 1 line

It fixes known issues and it's the same behavior we have in sysfs and debugfs already.

Yeah, as I said that is broken to begin with. It's just that for sysfs and debugfs nobody notices.

Regards,
Christian.

Quoted 49 lines

It's not
clear to me how this should best be handled. We basically want to block any access to the GPU (registers, firmwares, etc.) while the GPU is going through a reset. Just locking the reset domain doesn't seem like the right solution.

Alex

Quoted 41 lines

Regards,
Christian.

Am 12.02.24 um 21:56 schrieb Deucher, Alexander:

Quoted 37 lines

[AMD Official Use Only - General]

Ping?

Quoted 33 lines

-----Original Message-----
From: Deucher, Alexander <[email protected]>
Sent: Monday, January 29, 2024 10:56 AM
To: [email protected]
Cc: Deucher, Alexander <[email protected]>
Subject: [PATCH] drm/amdgpu: bail on INFO IOCTL if the GPU is in reset

This avoids queries to read registers or query the SMU for telemetry data while the GPU is in reset. This mirrors what we already do for sysfs.

---
   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
   1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -1
index a2df3025a754..d522e99c6f81 100644
@@ -607,6 +607,9 @@ int amdgpu_info_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
        int i, found, ret;
        int ui32_size = sizeof(ui32);
 
+     if (amdgpu_in_reset(adev))
+             return -EPERM;
+
        if (!info->return_size || !info->return_pointer)
                return -EINVAL;
 
--

2.42.0

#5 lore raw

Am 15.02.24 um 15:43 schrieb Christian König:

Quoted 19 lines

Am 15.02.24 um 15:36 schrieb Alex Deucher:

Quoted 10 lines
Quoted 8 lines · Christian König

Well using this is in sysfs is a bug to begin with. This would prevent starting new applications and crashing applications which don't expect to get an -EPERM in return here.

If we need to make operations mutual exclusive with resets then we need to take the appropriate locks and *not* work around by abusing amdgpu_in_reset().

Quoted 7 lines
Quoted 6 lines

The functionality of amdgpu_in_reset() is just to check in lower level functions if we are inside the higher level reset thread and *not* protect anybody from concurrent access.

I think we should probably completely nuke the underlying flag and using the thread owner of the lock to prevent such an abuse.

Can we land some variant of this for now?

Quoted 7 lines

I don't think so, it most likely will break existing use cases.

What we might be able to do is to frame this with amdgpu_device_lock_reset_domain() / amdgpu_device_unlock_reset_domain().

Quoted 1 line

It fixes known issues and it's the same behavior we have in sysfs and debugfs already.

Yeah, as I said that is broken to begin with. It's just that for sysfs and debugfs nobody notices.

Wait a second, debugfs is actually doing the right thing in some functions:

        /* Avoid accidently unparking the sched thread during GPU reset */ r = down_read_killable(&adev->reset_domain->sem);
        if (r)
                goto pro_end;
...

        up_read(&adev->reset_domain->sem);

This needs to replace amdgpu_in_reset() in pretty much all debugfs and sysfs function.

Probably best to wrap that in some inline amdgpu_reset_* functions and document why those needs to be used.

Regards,
Christian.

Quoted 55 lines

Regards,
Christian.

Quoted 50 lines

   It's not
clear to me how this should best be handled.  We basically want to block any access to the GPU (registers, firmwares, etc.) while the GPU is going through a reset.  Just locking the reset domain doesn't seem like the right solution.

Alex

Quoted 42 lines

Regards,
Christian.

Am 12.02.24 um 21:56 schrieb Deucher, Alexander:

Quoted 38 lines

[AMD Official Use Only - General]

Ping?

Quoted 34 lines

-----Original Message-----
From: Deucher, Alexander <[email protected]>
Sent: Monday, January 29, 2024 10:56 AM
To: [email protected]
Cc: Deucher, Alexander <[email protected]>
Subject: [PATCH] drm/amdgpu: bail on INFO IOCTL if the GPU is in reset

This avoids queries to read registers or query the SMU for telemetry data while the GPU is in reset. This mirrors what we already do for sysfs.

---
   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++

   1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index a2df3025a754..d522e99c6f81 100644
@@ -607,6 +607,9 @@ int amdgpu_info_ioctl(struct drm_device *dev, 

void
*data, struct drm_file *filp)
        int i, found, ret;
        int ui32_size = sizeof(ui32);

+     if (amdgpu_in_reset(adev))
+             return -EPERM;
+
        if (!info->return_size || !info->return_pointer) return -EINVAL;

Signature · 1 line
2.42.0