Skip to contentSkip to conversation

radeon 0000:02:00.0: GPU lockup CP stall for more than 10000msec

DISCUSSION Borislav Petkov 2 participants 7 messages Jan 10, 2013 – Jan 15, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

[ deliberately breaking the thread because it got too long]

Quoted 7 lines · Borislav Petkov

Hi Alex,

got the sickest bug on 3.8-rc1, see below. The GPU locks up somewhere down radeon_fence_wait_seq, judging by the error messages.

And this doesn't happen with 3.7, of course.

Quoted 7 lines

Let me know if you need any more info, thanks.

[16273.668350] radeon 0000:02:00.0: GPU lockup CP stall for more than 10000msec
[16273.668361] radeon 0000:02:00.0: GPU lockup (waiting for 0x000000000000002b last fence id 0x000000000000002a)
[16273.882550] plugin-containe[11435]: segfault at 7f1f0a66cc08 ip 00007f1f13289bdb sp 00007f1f0a2fe9e0 error 4 in libflashplayer.so[7f1f130c5000+117b000]
[16274.502807] ------------[ cut here ]------------
[16274.502845] WARNING: at lib/list_debug.c:53 __list_del_entry+0x63/0xd0()

Ok, this got fixed by 909d9eb67f1e4e39f2ea88e96bde03d560cde3eb which is upstream now. And I'm testing -rc2+ which contains this patch already
+ tip/master + another fix from Alan which reworks fb console locking (should be unrelated) and the machine gets unresponsive for a couple of seconds and then it is fine again.

See dmesg below, the GPU gets the same lockup CP stall without the list corruption so it recovers fine. But I didn't have those stalls before so it has to be something which came up with 3.8 merge window.

[44730.749380] radeon 0000:02:00.0: GPU lockup CP stall for more than 10000msec
[44730.749391] radeon 0000:02:00.0: GPU lockup (waiting for 0x0000000000305211 last fence id 0x0000000000305210)
[44730.750596] radeon 0000:02:00.0: Saved 25 dwords of commands on ring 0.
[44730.750612] radeon 0000:02:00.0: GPU softreset: 0x00000007
[44730.768865] radeon 0000:02:00.0:   R_008010_GRBM_STATUS      = 0xA0003030
[44730.768874] radeon 0000:02:00.0:   R_008014_GRBM_STATUS2     = 0x00000003
[44730.768880] radeon 0000:02:00.0:   R_000E50_SRBM_STATUS      = 0x200000C0
[44730.768885] radeon 0000:02:00.0:   R_008674_CP_STALLED_STAT1 = 0x00000000
[44730.768889] radeon 0000:02:00.0:   R_008678_CP_STALLED_STAT2 = 0x00000000
[44730.768894] radeon 0000:02:00.0:   R_00867C_CP_BUSY_STAT     = 0x00020184
[44730.768898] radeon 0000:02:00.0:   R_008680_CP_STAT          = 0x80028645
[44730.768903] radeon 0000:02:00.0:   R_008020_GRBM_SOFT_RESET=0x00007FEE
[44730.783898] radeon 0000:02:00.0: R_008020_GRBM_SOFT_RESET=0x00000001
[44730.798893] radeon 0000:02:00.0:   R_008010_GRBM_STATUS      = 0xA0003030
[44730.798896] radeon 0000:02:00.0:   R_008014_GRBM_STATUS2     = 0x00000003
[44730.798899] radeon 0000:02:00.0:   R_000E50_SRBM_STATUS      = 0x200080C0
[44730.798901] radeon 0000:02:00.0:   R_008674_CP_STALLED_STAT1 = 0x00000000
[44730.798904] radeon 0000:02:00.0:   R_008678_CP_STALLED_STAT2 = 0x00000000
[44730.798907] radeon 0000:02:00.0:   R_00867C_CP_BUSY_STAT     = 0x00000000
[44730.798909] radeon 0000:02:00.0:   R_008680_CP_STAT          = 0x80100000
[44730.819926] radeon 0000:02:00.0: GPU reset succeeded, trying to resume
[44730.836763] [drm] probing gen 2 caps for device 10de:377 = 1/0
[44730.839732] [drm] PCIE GART of 512M enabled (table at 0x0000000000040000).
[44730.839826] radeon 0000:02:00.0: WB enabled
[44730.839831] radeon 0000:02:00.0: fence driver on ring 0 use gpu addr 0x0000000020000c00 and cpu addr 0xffff880220223c00
[44730.839834] radeon 0000:02:00.0: fence driver on ring 3 use gpu addr 0x0000000020000c0c and cpu addr 0xffff880220223c0c
[44730.871080] [drm] ring test on 0 succeeded in 0 usecs
[44730.871140] [drm] ring test on 3 succeeded in 1 usecs
[44730.871187] [drm] ib test on ring 0 succeeded in 0 usecs
[44730.871206] [drm] ib test on ring 3 succeeded in 1 usecs

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#1 lore raw
Quoted 24 lines · Borislav Petkov

[ deliberately breaking the thread because it got too long]

Quoted 7 lines · Borislav Petkov

Hi Alex,

got the sickest bug on 3.8-rc1, see below. The GPU locks up somewhere down radeon_fence_wait_seq, judging by the error messages.

And this doesn't happen with 3.7, of course.

Quoted 7 lines

Let me know if you need any more info, thanks.

[16273.668350] radeon 0000:02:00.0: GPU lockup CP stall for more than 10000msec
[16273.668361] radeon 0000:02:00.0: GPU lockup (waiting for 0x000000000000002b last fence id 0x000000000000002a)
[16273.882550] plugin-containe[11435]: segfault at 7f1f0a66cc08 ip 00007f1f13289bdb sp 00007f1f0a2fe9e0 error 4 in libflashplayer.so[7f1f130c5000+117b000]
[16274.502807] ------------[ cut here ]------------
[16274.502845] WARNING: at lib/list_debug.c:53 __list_del_entry+0x63/0xd0()

Ok, this got fixed by 909d9eb67f1e4e39f2ea88e96bde03d560cde3eb which is upstream now. And I'm testing -rc2+ which contains this patch already
+ tip/master + another fix from Alan which reworks fb console locking (should be unrelated) and the machine gets unresponsive for a couple of seconds and then it is fine again.

Quoted 3 lines

See dmesg below, the GPU gets the same lockup CP stall without the list corruption so it recovers fine. But I didn't have those stalls before so it has to be something which came up with 3.8 merge window.

I'm assuming you didn't also update your userspace gfx stack? Does disabling the new DMA ring for ttm bo moves avoid the issue?

Alex

#2 lore raw
Quoted 1 line · Alex Deucher

I'm assuming you didn't also update your userspace gfx stack?

By that you mean x.org etc, right? Or GPU microcode too? In any case, I haven't touched any of those deliberately, AFAICR at least.

Quoted 1 line

Does disabling the new DMA ring for ttm bo moves avoid the issue?

How do I do that?

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#3 lore raw
Quoted 5 lines · Borislav Petkov
Quoted 1 line · Alex Deucher

I'm assuming you didn't also update your userspace gfx stack?

By that you mean x.org etc, right? Or GPU microcode too? In any case, I haven't touched any of those deliberately, AFAICR at least.

Right. Xorg drivers or mesa drivers.

Quoted 4 lines
Quoted 2 lines

Does disabling the new DMA ring for ttm bo moves avoid the issue?

How do I do that?

(quoted diff) +0 -0

b/drivers/gpu/drm/radeon/radeon_asic.c
index 9056faf..b0cc46d 100644

drivers/gpu/drm/radeon/radeon_asic.c +4 -4
@@ -974,8 +974,8 @@ static struct radeon_asic r600_asic = {
                .blit_ring_index = RADEON_RING_TYPE_GFX_INDEX,
                .dma = &r600_copy_dma,
                .dma_ring_index = R600_RING_TYPE_DMA_INDEX,
-               .copy = &r600_copy_dma,
-               .copy_ring_index = R600_RING_TYPE_DMA_INDEX,
+               .copy = &r600_copy_blit,
+               .copy_ring_index = RADEON_RING_TYPE_GFX_INDEX,
        },
        .surface = {
                .set_reg = r600_set_surface_reg,
@@ -1058,8 +1058,8 @@ static struct radeon_asic rs780_asic = {
                .blit_ring_index = RADEON_RING_TYPE_GFX_INDEX,
                .dma = &r600_copy_dma,
                .dma_ring_index = R600_RING_TYPE_DMA_INDEX,
-               .copy = &r600_copy_dma,
-               .copy_ring_index = R600_RING_TYPE_DMA_INDEX,
+               .copy = &r600_copy_blit,
+               .copy_ring_index = RADEON_RING_TYPE_GFX_INDEX,
        },
        .surface = {
                .set_reg = r600_set_surface_reg,
Quoted 9 lines

Thanks.

--
Regards/Gruss,
Boris.

Sent from a fat crate under my desk. Formatting is fine. --

#4 lore raw

drivers/gpu/drm/radeon/radeon_asic.c

Quoted 7 lines · Alex Deucher
Quoted 3 lines
Quoted 1 line

Does disabling the new DMA ring for ttm bo moves avoid the issue?

How do I do that?

drivers/gpu/drm/radeon/radeon_asic.c +0 -0
index 9056faf..b0cc46d 100644

[ … ]

Ok, I'm running -rc3 with this and will watch it for any changes in behavior.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#5 lore raw
Quoted 2 lines · Borislav Petkov

Ok, I'm running -rc3 with this and will watch it for any changes in behavior.

AFAICT, this fixes the CP stalls, for I haven't seen any of them in dmesg for the last couple of days after applying your revert.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#6 lore raw
Quoted 6 lines · Borislav Petkov
Quoted 2 lines · Borislav Petkov

Ok, I'm running -rc3 with this and will watch it for any changes in behavior.

AFAICT, this fixes the CP stalls, for I haven't seen any of them in dmesg for the last couple of days after applying your revert.

Can you remove that revert and try the attached patch as an alternative?

Thanks,

Alex

Quoted 9 lines

Thanks.

--
Regards/Gruss,
Boris.

Sent from a fat crate under my desk. Formatting is fine. --

From cb5206aa1c6c52b1b459eaf93797c593cff0c8db Mon Sep 17 00:00:00 2001
From: Alex Deucher <[email protected]>
Date: Mon, 14 Jan 2013 11:04:39 -0500
Subject: [PATCH] drm/radeon: clear reset flags if engines are idle

Fixes a regression in the gpu reset code after the rework for DMA support.

---
 drivers/gpu/drm/radeon/evergreen.c |    6 ++++++
 drivers/gpu/drm/radeon/ni.c        |    6 ++++++
 drivers/gpu/drm/radeon/r600.c      |    6 ++++++
 drivers/gpu/drm/radeon/si.c        |    6 ++++++
 4 files changed, 24 insertions(+), 0 deletions(-)
drivers/gpu/drm/radeon/evergreen.c +6 -0
index 061fa0a..4d0e60a 100644
@@ -2401,6 +2401,12 @@ static int evergreen_gpu_soft_reset(struct radeon_device *rdev, u32 reset_mask)
 {
 	struct evergreen_mc_save save;
 
+	if (!(RREG32(GRBM_STATUS) & GUI_ACTIVE))
+		reset_mask &= ~(RADEON_RESET_GFX | RADEON_RESET_COMPUTE);
+
+	if (RREG32(DMA_STATUS_REG) & DMA_IDLE)
+		reset_mask &= ~RADEON_RESET_DMA;
+
 	if (reset_mask == 0)
 		return 0;
 
drivers/gpu/drm/radeon/ni.c +6 -0
index 896f1cb..59acabb 100644
@@ -1409,6 +1409,12 @@ static int cayman_gpu_soft_reset(struct radeon_device *rdev, u32 reset_mask)
 {
 	struct evergreen_mc_save save;
 
+	if (!(RREG32(GRBM_STATUS) & GUI_ACTIVE))
+		reset_mask &= ~(RADEON_RESET_GFX | RADEON_RESET_COMPUTE);
+
+	if (RREG32(DMA_STATUS_REG) & DMA_IDLE)
+		reset_mask &= ~RADEON_RESET_DMA;
+
 	if (reset_mask == 0)
 		return 0;
 
drivers/gpu/drm/radeon/r600.c +6 -0
index 537e259..3cb9d60 100644
@@ -1378,6 +1378,12 @@ static int r600_gpu_soft_reset(struct radeon_device *rdev, u32 reset_mask)
 {
 	struct rv515_mc_save save;
 
+	if (!(RREG32(GRBM_STATUS) & GUI_ACTIVE))
+		reset_mask &= ~(RADEON_RESET_GFX | RADEON_RESET_COMPUTE);
+
+	if (RREG32(DMA_STATUS_REG) & DMA_IDLE)
+		reset_mask &= ~RADEON_RESET_DMA;
+
 	if (reset_mask == 0)
 		return 0;
 
drivers/gpu/drm/radeon/si.c +6 -0
index 3240a3d..ae8b482 100644
@@ -2215,6 +2215,12 @@ static int si_gpu_soft_reset(struct radeon_device *rdev, u32 reset_mask)
 {
 	struct evergreen_mc_save save;
 
+	if (!(RREG32(GRBM_STATUS) & GUI_ACTIVE))
+		reset_mask &= ~(RADEON_RESET_GFX | RADEON_RESET_COMPUTE);
+
+	if (RREG32(DMA_STATUS_REG) & DMA_IDLE)
+		reset_mask &= ~RADEON_RESET_DMA;
+
 	if (reset_mask == 0)
 		return 0;
 
Signature · 1 line
1.7.7.5