Question about how to troubleshoot sandybridge kernel opps and subsequest GPU lockup
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello,
I'm running wow in wine on 64 bit fedora rawhide on a dell vostro 3550 (i5 with integrated GPU).
I'm reliably able to produce 2 types of crashes:
- wow freezes, but I can get to text console, in this case I'm able to
grab a kernel stack trace (below) prior to seeing the normal
[drm:i915_wait_request] *ERROR* i915_wait_request returns -11 (awaiting 452684 at 452608, next 452686)
- the other is a complete freeze of the system, hard reset required, nothing logged to /var/log/messages
Is there any value in me creating a bug report for this, it seems to be a pretty common issue. Is there any use in my trying different kernel command line optios for the i915 driver or config options to the xorg intel driver?
I have the various git trees pulled out (I was looking for recent changes that might be related to this issue). I'm capable of building and installing from these git trees if there are specific bits that I should test.
Oct 22 20:52:59 localhost kernel: [ 939.830806] ------------[ cut here ]------------ Oct 22 20:52:59 localhost kernel: [ 939.830814] WARNING: at drivers/gpu/drm/i915/i915_drv.c:372 gen6_gt_force_wake_put+0x29/0x51 [i915]() Oct 22 20:52:59 localhost kernel: [ 939.830816] Hardware name: Vostro 3550 Oct 22 20:52:59 localhost kernel: [ 939.830818] Modules linked in: snd_seq_dummy fuse ip6table_filter ip6_tables ebtable_nat ebtables xt_state xt_CHECKSUM iptable_mangle ppdev parport_pc lp parport vboxpci vboxnetadp vboxnetflt vboxdrv bridge stp llc tun rfcomm bnep ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi snd_hda_codec_idt uvcvideo videodev btusb media bluetooth v4l2_compat_ioctl32 arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm iwlagn microcode mac80211 dell_laptop iTCO_wdt r8169 i2c_i801 snd_timer cfg80211 snd mii iTCO_vendor_support dcdbas dell_wmi sparse_keymap soundcore rfkill snd_page_alloc virtio_net kvm_intel kvm binfmt_misc wmi i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] Oct 22 20:52:59 localhost kernel: [ 939.830926] Pid: 0, comm: swapper Tainted: G WC 3.1.0-0.rc10.git0.1.fc17.x86_64 #1 Oct 22 20:52:59 localhost kernel: [ 939.830928] Call Trace: Oct 22 20:52:59 localhost kernel: [ 939.830930] <IRQ [<ffffffff8105c3a0>] warn_slowpath_common+0x83/0x9b Oct 22 20:52:59 localhost kernel: [ 939.830941] [<ffffffff8105c3d2>] warn_slowpath_null+0x1a/0x1c Oct 22 20:52:59 localhost kernel: [ 939.830952] [<ffffffffa006b624>] gen6_gt_force_wake_put+0x29/0x51 [i915] Oct 22 20:52:59 localhost kernel: [ 939.830963] [<ffffffffa006f45f>] i915_read32+0x44/0x6b [i915] Oct 22 20:52:59 localhost kernel: [ 939.830975] [<ffffffffa00724a9>] i915_hangcheck_elapsed+0xe8/0x1f8 [i915] Oct 22 20:52:59 localhost kernel: [ 939.831027] [<ffffffff81062ddd>] irq_exit+0x5d/0xcf Oct 22 20:52:59 localhost kernel: [ 939.831032] [<ffffffff8150de91>] smp_apic_timer_interrupt+0x7c/0x8a Oct 22 20:52:59 localhost kernel: [ 939.831036] [<ffffffff8150bd73>] apic_timer_interrupt+0x73/0x80 Oct 22 20:52:59 localhost kernel: [ 939.831038] <EOI [<ffffffff81014ded>] ? paravirt_read_tsc+0x9/0xd Oct 22 20:52:59 localhost kernel: [ 939.831046] [<ffffffff81297075>] ? intel_idle+0xe5/0x10c Oct 22 20:52:59 localhost kernel: [ 939.831050] [<ffffffff81297071>] ? intel_idle+0xe1/0x10c Oct 22 20:52:59 localhost kernel: [ 939.831054] [<ffffffff813e14fe>] cpuidle_idle_call+0x11c/0x1fe Oct 22 20:52:59 localhost kernel: [ 939.831059] [<ffffffff8100e2ef>] cpu_idle+0xab/0x101 Oct 22 20:52:59 localhost kernel: [ 939.831063] [<ffffffff814df673>] rest_init+0xd7/0xde Oct 22 20:52:59 localhost kernel: [ 939.831067] [<ffffffff814df59c>] ? csum_partial_copy_generic+0x16c/0x16c Oct 22 20:52:59 localhost kernel: [ 939.831072] [<ffffffff81d53bb0>] start_kernel+0x3dd/0x3ea Oct 22 20:52:59 localhost kernel: [ 939.831076] [<ffffffff81d532c4>] x86_64_start_reservations+0xaf/0xb3 Oct 22 20:52:59 localhost kernel: [ 939.831081] [<ffffffff81d53140>] ? early_idt_handlers+0x140/0x140 Oct 22 20:52:59 localhost kernel: [ 939.831085] [<ffffffff81d533ca>] x86_64_start_kernel+0x102/0x111 Oct 22 20:52:59 localhost kernel: [ 939.831088] ---[ end trace f5cba358bac6b7e5 ]---
Signature · 2 lines
James R. Leu [email protected]
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 7 lines · James R. Leu
I'm running wow in wine on 64 bit fedora rawhide on a dell vostro 3550 (i5 with integrated GPU).
I'm reliably able to produce 2 types of crashes:
- wow freezes, but I can get to text console, in this case I'm able to
grab a kernel stack trace (below) prior to seeing the normal
[drm:i915_wait_request] *ERROR*i915_wait_requestreturns -11 (awaiting 452684 at 452608, next 452686)
I'm pretty sure that below that line there's a gpu hang report. If that's the case, the please grab everything in /sys/kernel/debug/dri, put it into a tar.gz and attach it (you need to do this _after_ the machine is hung, the kernel will write a gpu crash dump into i915_error_state).
The userspace parts of the i915 driver are very important for gpu hangs, so please attach the version of mesa, libdrm and xf86-video-intel you've installed.
Also please attach all your i915.ko module options as listed in /sys/module/i915/parameters
Quoted 1 line
- the other is a complete freeze of the system, hard reset required, nothing logged to
/var/log/messages
It's rather likely that this is the same issue as above. Depending upon exact circumstances the gpu can take down the entire system.
Quoted 3 lines
Is there any value in me creating a bug report for this, it seems to be a pretty common issue. Is there any use in my trying different kernel command line optios for
the i915 driver or config options to the xorg intel driver?
Yes, gpu hangs are one of the more common issues, but until you've submitted the error_state there's no way to diagnose the issue and tell whether we have got a report already.
Quoted 22 lines
I have the various git trees pulled out (I was looking for recent changes that might be related to this issue). I'm capable of building and installing from these git trees if there are specific bits that I should test.
[ 939.830806] ------------[ cut here ]------------ [ 939.830814] WARNING: at drivers/gpu/drm/i915/i915_drv.c:372 gen6_gt_force_wake_put+0x29/0x51 [i915]() [ 939.830816] Hardware name: Vostro 3550 [ 939.830818] Modules linked in: snd_seq_dummy fuse ip6table_filter ip6_tables ebtable_nat ebtables xt_state xt_CHECKSUM iptable_mangle ppdev parport_pc lp parport vboxpci vboxnetadp vboxnetflt vboxdrv bridge stp llc tun rfcomm bnep ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi snd_hda_codec_idt uvcvideo videodev btusb media bluetooth v4l2_compat_ioctl32 arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm iwlagn microcode mac80211 dell_laptop iTCO_wdt r8169 i2c_i801 snd_timer cfg80211 snd mii iTCO_vendor_support dcdbas dell_wmi sparse_keymap soundcore rfkill snd_page_alloc virtio_net kvm_intel kvm binfmt_misc wmi i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 939.830926] Pid: 0, comm: swapper Tainted: G WC 3.1.0-0.rc10.git0.1.fc17.x86_64 #1 [ 939.830928] Call Trace: [ 939.830930] <IRQ [<ffffffff8105c3a0>] warn_slowpath_common+0x83/0x9b [ 939.830941] [<ffffffff8105c3d2>] warn_slowpath_null+0x1a/0x1c [ 939.830952] [<ffffffffa006b624>] gen6_gt_force_wake_put+0x29/0x51 [i915] [ 939.830963] [<ffffffffa006f45f>] i915_read32+0x44/0x6b [i915] [ 939.830975] [<ffffffffa00724a9>] i915_hangcheck_elapsed+0xe8/0x1f8 [i915] [ 939.831027] [<ffffffff81062ddd>] irq_exit+0x5d/0xcf [ 939.831032] [<ffffffff8150de91>] smp_apic_timer_interrupt+0x7c/0x8a [ 939.831036] [<ffffffff8150bd73>] apic_timer_interrupt+0x73/0x80 [ 939.831038] <EOI [<ffffffff81014ded>] ? paravirt_read_tsc+0x9/0xd [ 939.831046] [<ffffffff81297075>] ? intel_idle+0xe5/0x10c [ 939.831050] [<ffffffff81297071>] ? intel_idle+0xe1/0x10c [ 939.831054] [<ffffffff813e14fe>] cpuidle_idle_call+0x11c/0x1fe
Quoted 8 lines
[ 939.831059] [<ffffffff8100e2ef>] cpu_idle+0xab/0x101 [ 939.831063] [<ffffffff814df673>] rest_init+0xd7/0xde [ 939.831067] [<ffffffff814df59c>] ? csum_partial_copy_generic+0x16c/0x16c [ 939.831072] [<ffffffff81d53bb0>] start_kernel+0x3dd/0x3ea [ 939.831076] [<ffffffff81d532c4>] x86_64_start_reservations+0xaf/0xb3 [ 939.831081] [<ffffffff81d53140>] ? early_idt_handlers+0x140/0x140 [ 939.831085] [<ffffffff81d533ca>] x86_64_start_kernel+0x102/0x111 [ 939.831088] ---[ end trace f5cba358bac6b7e5 ]---
This WARN here is a possible sideeffect of a dying gpu. Independant, but rather harmless bug. Unfortunately no easy solution, hence no patch atm.
Yours, Daniel
Signature · 3 lines
Daniel Vetter Mail: [email protected] Mobile: +41 (0)79 365 57 48
Debug output attached
Quoted 73 lines · Daniel Vetter
Quoted 7 lines · James R. Leu
I'm running wow in wine on 64 bit fedora rawhide on a dell vostro 3550 (i5 with integrated GPU).
I'm reliably able to produce 2 types of crashes:
- wow freezes, but I can get to text console, in this case I'm able to
grab a kernel stack trace (below) prior to seeing the normal
[drm:i915_wait_request] *ERROR*i915_wait_requestreturns -11 (awaiting 452684 at 452608, next 452686)I'm pretty sure that below that line there's a gpu hang report. If that's the case, the please grab everything in
/sys/kernel/debug/dri, put it into a tar.gz and attach it (you need to do this _after_ the machine is hung, the kernel will write a gpu crash dump intoi915_error_state).The userspace parts of the i915 driver are very important for gpu hangs, so please attach the version of mesa, libdrm and xf86-video-intel you've installed.
Also please attach all your i915.ko module options as listed in
/sys/module/i915/parametersQuoted 1 line
- the other is a complete freeze of the system, hard reset required, nothing logged to
/var/log/messagesIt's rather likely that this is the same issue as above. Depending upon exact circumstances the gpu can take down the entire system.
Quoted 3 lines
Is there any value in me creating a bug report for this, it seems to be a pretty common issue. Is there any use in my trying different kernel command line optios for
the i915 driver or config options to the xorg intel driver?Yes, gpu hangs are one of the more common issues, but until you've submitted the
error_statethere's no way to diagnose the issue and tell whether we have got a report already.Quoted 22 lines
I have the various git trees pulled out (I was looking for recent changes that might be related to this issue). I'm capable of building and installing from these git trees if there are specific bits that I should test.
[ 939.830806] ------------[ cut here ]------------ [ 939.830814] WARNING: at drivers/gpu/drm/i915/i915_drv.c:372 gen6_gt_force_wake_put+0x29/0x51 [i915]() [ 939.830816] Hardware name: Vostro 3550 [ 939.830818] Modules linked in: snd_seq_dummy fuse ip6table_filter ip6_tables ebtable_nat ebtables xt_state xt_CHECKSUM iptable_mangle ppdev parport_pc lp parport vboxpci vboxnetadp vboxnetflt vboxdrv bridge stp llc tun rfcomm bnep ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi snd_hda_codec_idt uvcvideo videodev btusb media bluetooth v4l2_compat_ioctl32 arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm iwlagn microcode mac80211 dell_laptop iTCO_wdt r8169 i2c_i801 snd_timer cfg80211 snd mii iTCO_vendor_support dcdbas dell_wmi sparse_keymap soundcore rfkill snd_page_alloc virtio_net kvm_intel kvm binfmt_misc wmi i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 939.830926] Pid: 0, comm: swapper Tainted: G WC 3.1.0-0.rc10.git0.1.fc17.x86_64 #1 [ 939.830928] Call Trace: [ 939.830930] <IRQ [<ffffffff8105c3a0>] warn_slowpath_common+0x83/0x9b [ 939.830941] [<ffffffff8105c3d2>] warn_slowpath_null+0x1a/0x1c [ 939.830952] [<ffffffffa006b624>] gen6_gt_force_wake_put+0x29/0x51 [i915] [ 939.830963] [<ffffffffa006f45f>] i915_read32+0x44/0x6b [i915] [ 939.830975] [<ffffffffa00724a9>] i915_hangcheck_elapsed+0xe8/0x1f8 [i915] [ 939.831027] [<ffffffff81062ddd>] irq_exit+0x5d/0xcf [ 939.831032] [<ffffffff8150de91>] smp_apic_timer_interrupt+0x7c/0x8a [ 939.831036] [<ffffffff8150bd73>] apic_timer_interrupt+0x73/0x80 [ 939.831038] <EOI [<ffffffff81014ded>] ? paravirt_read_tsc+0x9/0xd [ 939.831046] [<ffffffff81297075>] ? intel_idle+0xe5/0x10c [ 939.831050] [<ffffffff81297071>] ? intel_idle+0xe1/0x10c [ 939.831054] [<ffffffff813e14fe>] cpuidle_idle_call+0x11c/0x1feQuoted 8 lines
[ 939.831059] [<ffffffff8100e2ef>] cpu_idle+0xab/0x101 [ 939.831063] [<ffffffff814df673>] rest_init+0xd7/0xde [ 939.831067] [<ffffffff814df59c>] ? csum_partial_copy_generic+0x16c/0x16c [ 939.831072] [<ffffffff81d53bb0>] start_kernel+0x3dd/0x3ea [ 939.831076] [<ffffffff81d532c4>] x86_64_start_reservations+0xaf/0xb3 [ 939.831081] [<ffffffff81d53140>] ? early_idt_handlers+0x140/0x140 [ 939.831085] [<ffffffff81d533ca>] x86_64_start_kernel+0x102/0x111 [ 939.831088] ---[ end trace f5cba358bac6b7e5 ]---This WARN here is a possible sideeffect of a dying gpu. Independant, but rather harmless bug. Unfortunately no easy solution, hence no patch atm.
Yours, Daniel
Signature · 3 lines
Daniel Vetter Mail: [email protected] Mobile: +41 (0)79 365 57 48
Signature · 2 lines
James R. Leu [email protected]
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 1 line · James R. Leu
Debug output attached
You're in luck! I fixed this GPU hang today in Mesa master.
This commit fixes the hang:
commit 3cc0a7be23ab603ed40d602595f673a44e079885
Author: Kenneth Graunke <[email protected]> Date: Fri Oct 21 01:03:37 2011 -0700
i965: Apply post-sync non-zero workaround to homebrew workaround.
In commit 3e5d3626, Eric added a homebrew workaround to fix GPU hangs in the Mesa "engine" demo and oglc's api-texcoord test.
Unfortunately, his PIPE_CONTROL contains a Depth Stall, which necessitates the post-sync non-zero workaround,
Fixes GPU hangs in Civilization 4, PlaneShift, and 3DMMES. Hopefully Heroes of Newerth as well, though I haven't tested that.
NOTE: This is candidate for the 7.11 branch.
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=40324
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=41096
I'm planning on cherry-picking it to the 7.11 branch in the next few days, so it ought to make the upcoming 7.11.1 release.
Quoted 24 lines · On Mon, 24 Oct 2011 19:43:44 -0700 Kenneth Graunke <[email protected]>…
Quoted 1 line · James R. Leu
Debug output attached
You're in luck! I fixed this GPU hang today in Mesa master.
This commit fixes the hang:
commit
3cc0a7be23ab603ed40d602595f673a44e079885Author: Kenneth Graunke <[email protected]> Date: Fri Oct 21 01:03:37 2011 -0700i965: Apply post-sync non-zero workaround to homebrew workaround.
In commit
3e5d3626, Eric added a homebrew workaround to fix GPU
hangs in the Mesa "engine" demo and oglc's api-texcoord test.Unfortunately, his
PIPE_CONTROLcontains a Depth Stall, which necessitates the post-sync non-zero workaround,Fixes GPU hangs in Civilization 4, PlaneShift, and 3DMMES. Hopefully Heroes of Newerth as well, though I haven't tested that.
NOTE: This is candidate for the 7.11 branch.
Quoted 7 lines
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=40324
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=41096Signed-off-by Kenneth Graunke <[email protected]>Reviewed-and-tested-by Eric Anholt <[email protected]>I'm planning on cherry-picking it to the 7.11 branch in the next few days, so it ought to make the upcoming 7.11.1 release.
It's good that we have so many ways and opportunities to test our GPU reset reliability.
Gordon, can you make sure our regular QA covers GPU hang detect and reset using a few different methods (e.g. the ones above but without the fix from Ken in Mesa)? It's important that reset work really well and ideally w/o even being noticed by the user, so the more ways we have to wedge things, the better we can test the reset path's invisibility.
Thanks,
Jesse
Quoted 38 lines · Jesse Barnes
Quoted 24 lines · On Mon, 24 Oct 2011 19:43:44 -0700 Kenneth Graunke <[email protected]>…
Quoted 1 line · James R. Leu
Debug output attached
You're in luck! I fixed this GPU hang today in Mesa master.
This commit fixes the hang:
commit
3cc0a7be23ab603ed40d602595f673a44e079885Author: Kenneth Graunke <[email protected]> Date: Fri Oct 21 01:03:37 2011 -0700i965: Apply post-sync non-zero workaround to homebrew workaround.
In commit
3e5d3626, Eric added a homebrew workaround to fix GPU
hangs in the Mesa "engine" demo and oglc's api-texcoord test.Unfortunately, his
PIPE_CONTROLcontains a Depth Stall, which necessitates the post-sync non-zero workaround,Fixes GPU hangs in Civilization 4, PlaneShift, and 3DMMES. Hopefully Heroes of Newerth as well, though I haven't tested that.
NOTE: This is candidate for the 7.11 branch.
Quoted 7 lines
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=40324
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=41096Signed-off-by Kenneth Graunke <[email protected]>Reviewed-and-tested-by Eric Anholt <[email protected]>I'm planning on cherry-picking it to the 7.11 branch in the next few days, so it ought to make the upcoming 7.11.1 release.
It's good that we have so many ways and opportunities to test our GPU reset reliability.
Quoted 6 lines
Gordon, can you make sure our regular QA covers GPU hang detect and reset using a few different methods (e.g. the ones above but without the fix from Ken in Mesa)? It's important that reset work really well and ideally w/o even being noticed by the user, so the more ways we have to wedge things, the better we can test the reset path's invisibility.
I'm thinking about adding a debugfs file that stops ringbuffer tail writes on the specified ring to simulate a gpu hang. This way we can really stress-test the hangcheck and error_state capture code. And by throwing random workloads at the gpu while we "hang" it we hopefully can decently exercise the gpu reset code and see whether it properly resets the gpu (or just takes down the entire system).
-Daniel
Signature · 3 lines
Daniel Vetter Mail: [email protected] Mobile: +41 (0)79 365 57 48