amdgpu: ratelimit on vm faults to avoid spaming console
Subsystems (2)
These are rather minor however should help stop some folks machines grinding to a halt when a userspace application somehow gets the GPU into some horrible state causing the console to fill very quickly. Applies on top of master.
Please kindly review,
Edward O'Callaghan (2):
[PATCH v2 1/2] amdgpu: Use dev_err() over vanilla printk() in [PATCH v2 2/2] amdgpu: Wrap dev_err() calls on vm faults with
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1478493309-30328-1-git-send-email-funfunctor-dczkZgxz+BNUPWh3PAxdjQ@public.gmane.org
PATCH v2 1/2 amdgpu: Use dev_err() over vanilla printk() in vm_decode_fault()
--- drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 2 +- drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +1 -1
@@ -711,7 +711,7 @@ static void gmc_v7_0_vm_decode_fault(struct amdgpu_device *adev, mc_id =REG_GET_FIELD (status,VM_CONTEXT1_PROTECTION_FAULT_STATUS ,MEMORY_CLIENT_ID ); - printk("VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n", + dev_err(adev->dev, "VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n", protections, vmid, addr, REG_GET_FIELD(status, VM_CONTEXT1_PROTECTION_FAULT_STATUS, MEMORY_CLIENT_RW) ?
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +1 -1
@@ -837,7 +837,7 @@ static void gmc_v8_0_vm_decode_fault(struct amdgpu_device *adev, mc_id =REG_GET_FIELD (status,VM_CONTEXT1_PROTECTION_FAULT_STATUS ,MEMORY_CLIENT_ID ); - printk("VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n", + dev_err(adev->dev, "VM fault (0x%02x, vmid %d) at page %u, %s from '%s' (0x%08x) (%d)\n", protections, vmid, addr, REG_GET_FIELD(status, VM_CONTEXT1_PROTECTION_FAULT_STATUS, MEMORY_CLIENT_RW) ?
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH v2 2/2 amdgpu: Wrap dev_err() calls on vm faults with printk_ratelimit()
In reply to a message not in the archiveIt can be the case that upon GPU page faults we start trashing the logs, and so let us ratelimit here to avoid that.
V2. Fix issue where calling dev_err_ratelimited separately for
each line means that some lines corresponding to a single VM fault may or may not appear depending on the rate.
- Michel Dänzer.
--- drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c | 16 +++++++++------- drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 16 +++++++++------- drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 16 +++++++++------- 3 files changed, 27 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c +9 -7
@@ -1007,13 +1007,15 @@ static int gmc_v6_0_process_interrupt(struct amdgpu_device *adev,if (amdgpu_vm_fault_stop ==AMDGPU_VM_FAULT_STOP_FIRST )gmc_v6_0_set_fault_enable_default (adev, false); -dev_err (adev->dev,"GPU fault detected: %d 0x%08x\n" , - entry->src_id, entry->src_data); -dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n" , - addr); -dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n" , - status); -gmc_v6_0_vm_decode_fault (adev, status, addr,0 ); +if (printk_ratelimit ()) { +dev_err (adev->dev,"GPU fault detected: %d 0x%08x\n" , + entry->src_id, entry->src_data); +dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n" , + addr); +dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n" , + status); +gmc_v6_0_vm_decode_fault (adev, status, addr,0 ); + }return 0 ; }
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +9 -7
@@ -1198,13 +1198,15 @@ static int gmc_v7_0_process_interrupt(struct amdgpu_device *adev,if (amdgpu_vm_fault_stop ==AMDGPU_VM_FAULT_STOP_FIRST )gmc_v7_0_set_fault_enable_default (adev, false); -dev_err (adev->dev,"GPU fault detected: %d 0x%08x\n" , - entry->src_id, entry->src_data); -dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n" , - addr); -dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n" , - status); -gmc_v7_0_vm_decode_fault (adev, status, addr, mc_client); +if (printk_ratelimit ()) { +dev_err (adev->dev,"GPU fault detected: %d 0x%08x\n" , + entry->src_id, entry->src_data); +dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n" , + addr); +dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n" , + status); +gmc_v7_0_vm_decode_fault (adev, status, addr, mc_client); + }return 0 ; }
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +9 -7
@@ -1242,13 +1242,15 @@ static int gmc_v8_0_process_interrupt(struct amdgpu_device *adev,if (amdgpu_vm_fault_stop ==AMDGPU_VM_FAULT_STOP_FIRST )gmc_v8_0_set_fault_enable_default (adev, false); -dev_err (adev->dev,"GPU fault detected: %d 0x%08x\n" , - entry->src_id, entry->src_data); -dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n" , - addr); -dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n" , - status); -gmc_v8_0_vm_decode_fault (adev, status, addr, mc_client); +if (printk_ratelimit ()) { +dev_err (adev->dev,"GPU fault detected: %d 0x%08x\n" , + entry->src_id, entry->src_data); +dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n" , + addr); +dev_err (adev->dev," VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n" , + status); +gmc_v8_0_vm_decode_fault (adev, status, addr, mc_client); + }return 0 ; }
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Am 07.11.2016 um 05:35 schrieb Edward O'Callaghan:
Quoted 6 lines
These are rather minor however should help stop some folks machines grinding to a halt when a userspace application somehow gets the GPU into some horrible state causing the console to fill very quickly. Applies on top of master.
Please kindly review,
Clearly looks like a good idea to me. Patch are Reviewed-by: Christian König <[email protected]>.
Regards,
Christian.
Quoted 8 lines
Edward O'Callaghan (2):
[PATCH v2 1/2] amdgpu: Usedev_err()over vanillaprintk()in [PATCH v2 2/2] amdgpu: Wrapdev_err()calls on vm faults with
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines · Edward O'Callaghan
These are rather minor however should help stop some folks machines grinding to a halt when a userspace application somehow gets the GPU into some horrible state causing the console to fill very quickly. Applies on top of master.
I'm generally ok with the idea. My only concern would be if this would adversely affect the radeon GPUVM debugging options in mesa and piglit. Nicolai? Marek?
Alex
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CADnq5_NG7GcFJxA8mq0-fEE51Jw7TCjYPn68wSMA846tFB8fNQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Could we provide fault information through a ring buffer and a debugfs or drm ioctl interface?
Tom
________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of Alex Deucher <[email protected]>
Sent: Monday, November 7, 2016 13:35
To: Edward O'Callaghan; Nicolai Hähnle; Marek Olsak
Cc: Michel Dänzer; amd-gfx list
Subject: Re: amdgpu: ratelimit on vm faults to avoid spaming console
Quoted 4 lines · Edward O'Callaghan
These are rather minor however should help stop some folks machines grinding to a halt when a userspace application somehow gets the GPU into some horrible state causing the console to fill very quickly. Applies on top of master.
I'm generally ok with the idea. My only concern would be if this would adversely affect the radeon GPUVM debugging options in mesa and piglit. Nicolai? Marek?
Alex
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
amd-gfx Info Page - lists.freedesktop.org<https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives. Using amd-gfx: To post a message to all the list members, send email ...
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CY4PR12MB1768E71EFBD0472E7E170B72F7A70-rpdhrqHFk06yjjPBNVDk/[email protected]
I would rather say that we need to add this to the per context lockup informations.
Marek had a rather good start for this, but from the VM fault handler it is actually rather tricky to figure out reliable who caused it.
Because of this we didn't followed this approach to the end.
Christian.
Am 07.11.2016 um 19:42 schrieb StDenis, Tom:
Quoted 38 lines
Could we provide fault information through a ring buffer and a debugfs or drm ioctl interface?
Tom
------------------------------------------------------------------------
*From:* amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of Alex Deucher <[email protected]> *Sent:* Monday, November 7, 2016 13:35
*To:* Edward O'Callaghan; Nicolai Hähnle; Marek Olsak
*Cc:* Michel Dänzer; amd-gfx list
*Subject:* Re: amdgpu: ratelimit on vm faults to avoid spaming consoleQuoted 4 lines · Edward O'Callaghan
These are rather minor however should help stop some folks machines grinding to a halt when a userspace application somehow gets the GPU into some horrible state causing the console to fill very quickly. Applies on top of master.
I'm generally ok with the idea. My only concern would be if this would adversely affect the radeon GPUVM debugging options in mesa and piglit. Nicolai? Marek?
Alex
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
amd-gfx Info Page - lists.freedesktop.org <https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives. Using amd-gfx: To post a message to all the list members, send email ...
Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 02ac0302-1977-feab-575a-96ad69c43c7f-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
I had a mini discussion about this issue the other day on the IRC channel. Essentially yes - we would want to map back from the offending context back to the pid and I suppose kill it to restore a 'good state'. I don't know yet how to go about implementing such a mechanism so any suggestions would be really helpful to me!
I think, when the driver does winds up in these kinds of state, for us to have the ability to provide something meaningful for the user to do to file a bug report much like how the intel driver does would be a nice addition.
In any case, this point of those patches were just to give some breathing room and allow for machines to stay responsive enough to allow for further debug and maintain access to a working console for the user.
Cheers,
Edward.
Quoted 63 lines · Christian König
I would rather say that we need to add this to the per context lockup informations.
Marek had a rather good start for this, but from the VM fault handler it is actually rather tricky to figure out reliable who caused it.
Because of this we didn't followed this approach to the end.
Christian.
Am 07.11.2016 um 19:42 schrieb StDenis, Tom:
Quoted 44 lines
Could we provide fault information through a ring buffer and a debugfs or drm ioctl interface?
Tom
------------------------------------------------------------------------
*From:* amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of Alex Deucher <[email protected]>
*Sent:* Monday, November 7, 2016 13:35
*To:* Edward O'Callaghan; Nicolai Hähnle; Marek Olsak
*Cc:* Michel Dänzer; amd-gfx list
*Subject:* Re: amdgpu: ratelimit on vm faults to avoid spaming consoleQuoted 4 lines · Edward O'Callaghan
These are rather minor however should help stop some folks machines grinding to a halt when a userspace application somehow gets the GPU into some horrible state causing the console to fill very quickly. Applies on top of master.
I'm generally ok with the idea. My only concern would be if this would adversely affect the radeon GPUVM debugging options in mesa and piglit. Nicolai? Marek?
Alex
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
amd-gfx Info Page - lists.freedesktop.org
<https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives. Using amd-gfx: To post a message to all the list members, send email ...Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 5 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx