r200 dri driver deadlocks
I'm currently using an r200 (specifically, an agp 'ATI Technologies Inc Radeon R200 QM [Radeon 9100]') on a uniproc Pentium 3 board equipped with an intel 440bx/piix4 type chipset (the agp controller is identified as 'Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 03)'
All of this was tested with a virgin 2.6.8.1 (with debug info and frame pointers enabled) and Debian's XFree86 4.3.0.1, using DarkPlaces and Twilight (both popular quakeGL engine forks) as test apps, unless otherwise noted.
Thanks to wli (who I owe at least one beer for this, may be an entire case), we've been able to figure exactly whats going on. The driver is turning off interrupts, then deadlocking. (No sysrq, no sshing in, capslock's light doesn't work.)
Turning the NMI watchdog on, it 'fixes' the deadlock (and thanks to the watchdog, ssh and sysrq now work, but capslock's light still doesn't work), but the app and X are still dead, but now I can ssh in and kill -9 them both, however, and quite obviously, I can't start another X, but I can reboot cleanly.
Things already tested that don't effect bug:
Turning SMP on or off
Turning 4k stacks on or off
Using new radeon fbcon, using old radeon fbcon, using no fbcon Turning Local APIC for uniproc and/or IO-APIC for uniproc on or off Turning preempt on or off
Using mem=nopentium
Waving a dead chicken over the box
Things already tested for:
Kernels as far back as 2.6.0 have this bug, haven't tested any earlier
Thanks to netconsole (who I recommend to anyone that can't setup serial console stuff), I was able to capture the entire kernel output, including sysrq-t output right after my test app crashes.
I'm including both the netconsole output and the .config. vmlinux and radeon.ko (and anything else you need) are available upon request.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Can you insmod the radeon drm module with drm_opts=debug do the test and send on the trace, it may be getting wedged somewhere unexpected...
Dave.
Quoted 37 lines
All of this was tested with a virgin 2.6.8.1 (with debug info and frame pointers enabled) and Debian's XFree86 4.3.0.1, using DarkPlaces and Twilight (both popular quakeGL engine forks) as test apps, unless otherwise noted.
Thanks to wli (who I owe at least one beer for this, may be an entire case), we've been able to figure exactly whats going on. The driver is turning off interrupts, then deadlocking. (No sysrq, no sshing in, capslock's light doesn't work.)
Turning the NMI watchdog on, it 'fixes' the deadlock (and thanks to the watchdog, ssh and sysrq now work, but capslock's light still doesn't work), but the app and X are still dead, but now I can ssh in and kill -9 them both, however, and quite obviously, I can't start another X, but I can reboot cleanly.
Things already tested that don't effect bug:
Turning SMP on or off
Turning 4k stacks on or off
Using new radeon fbcon, using old radeon fbcon, using no fbcon Turning Local APIC for uniproc and/or IO-APIC for uniproc on or off Turning preempt on or off
Using mem=nopentium
Waving a dead chicken over the boxThings already tested for:
Kernels as far back as 2.6.0 have this bug, haven't tested any earlierThanks to netconsole (who I recommend to anyone that can't setup serial console stuff), I was able to capture the entire kernel output, including sysrq-t output right after my test app crashes.
I'm including both the netconsole output and the .config. vmlinux and radeon.ko (and anything else you need) are available upon request.
Signature · 3 lines
David Airlie, Software Engineer http://www.skynet.ie/~airlied / airlied at skynet.ie pam_smb / Linux DECstation / Linux VAX / ILUG person
Quoted 3 lines · Patrick McFarland
All of this was tested with a virgin 2.6.8.1 (with debug info and frame pointers enabled) and Debian's XFree86 4.3.0.1, [...]
What version of the DRI driver?
Signature · 2 lines
Earthling Michel Dänzer | Debian (powerpc), X and DRI developer Libre software enthusiast | http://svcs.affero.net/rm.php?r=daenzer
Quoted 1 line · Michel Dänzer
What version of the DRI driver?
Where do I look for that?
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 4 lines · Patrick McFarland
Quoted 1 line · Michel Dänzer
What version of the DRI driver?
Where do I look for that?
Where did you get r200_dri.so from?
Signature · 2 lines
Earthling Michel Dänzer | Debian (powerpc), X and DRI developer Libre software enthusiast | http://svcs.affero.net/rm.php?r=daenzer
Quoted 9 lines · Michel Dänzer
Quoted 4 lines · Patrick McFarland
Quoted 1 line · Michel Dänzer
What version of the DRI driver?
Where do I look for that?
Where did you get r200_dri.so from?
From the one that comes with the Deb X I mentioned above.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 3 lines · Dave Airlie
Can you insmod the radeon drm module with drm_opts=debug do the test and send on the trace, it may be getting wedged somewhere unexpected...
Here you go, but it doesn't look like it has output anything interesting.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 5 lines · Patrick McFarland
Quoted 2 lines · Michel Dänzer
Where did you get r200_dri.so from?
From the one that comes with the Deb X I mentioned above.
Please try something newer, e.g. my xlibmesa-gl1-dri-trunk or a binary snapshot from dri.sf.net.
Signature · 2 lines
Earthling Michel Dänzer | Debian (powerpc), X and DRI developer Libre software enthusiast | http://svcs.affero.net/rm.php?r=daenzer
Quoted 9 lines · Michel Dänzer
Quoted 5 lines · Patrick McFarland
Quoted 2 lines · Michel Dänzer
Where did you get r200_dri.so from?
From the one that comes with the Deb X I mentioned above.
Please try something newer, e.g. my xlibmesa-gl1-dri-trunk or a binary snapshot from dri.sf.net.
That shouldn't matter, should it? The userland stuff should never lock the machine up.
I'll test it anyhow, though.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 6 lines · Patrick McFarland
Quoted 5 lines · Michel Dänzer
Quoted 4 lines · Patrick McFarland
Quoted 2 lines · Michel Dänzer
Where did you get r200_dri.so from?
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
From the one that comes with the Deb X I mentioned above.
Please try something newer, e.g. my xlibmesa-gl1-dri-trunk or a binary snapshot from dri.sf.net.
That shouldn't matter, should it? The userland stuff should never lock the machine up.
In an ideal world... Feel free to track down the cause and add code to the DRM to prevent it.
Signature · 2 lines
Earthling Michel Dänzer | Debian (powerpc), X and DRI developer Libre software enthusiast | http://svcs.affero.net/rm.php?r=daenzer
Quoted 6 lines · Michel Dänzer
Quoted 2 lines · Patrick McFarland
That shouldn't matter, should it? The userland stuff should never lock the machine up.
In an ideal world... Feel free to track down the cause and add code to the DRM to prevent it.
I would, except, as many have noted before, even looking at the r200 driver requires years
of therapy to get rid of the nightmares.
So, yeah, I'll check to see if today's dri cvs snapshot works. If it doesn't, I'm not sure what to do.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 6 lines · Patrick McFarland
Quoted 5 lines · Michel Dänzer
Quoted 4 lines · Patrick McFarland
Quoted 2 lines · Michel Dänzer
Where did you get r200_dri.so from?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
From the one that comes with the Deb X I mentioned above.
Please try something newer, e.g. my xlibmesa-gl1-dri-trunk or a binary snapshot from dri.sf.net.
That shouldn't matter, should it? The userland stuff should never lock the machine up.
I'll test it anyhow, though.
No, it shouldn't. Anything that directly accesses hardware belongs in the kernel. How to fix this is a pretty hot topic now.
Lee
Quoted 1 line · On Sun, 05 Sep 2004 20:14:43 -0400 Lee Revell <[email protected]> wrote:
On Sun, 2004-09-05 at 16:18, Patrick McFarland wrote:
[snip]
Quoted 7 lines
Quoted 4 lines
That shouldn't matter, should it? The userland stuff should never lock the machine up.
I'll test it anyhow, though.No, it shouldn't. Anything that directly accesses hardware belongs in the kernel. How to fix this is a pretty hot topic now.
That's not the whole truth. There are just too many ways to lock up those 3D chips. For instance I fixed a lockup in the r100 driver where the order in which state changing commands were sent to the hardware would cause a lockup. Each individual state changing command is perfectly valid. Finding all permutations that trigger a lockup would have been too much of a hassle and may not even have been true for all supported hardware out there. So we made the user-space driver emit state changing commands in a fixed order, which seems to work everywhere.
Regars,
Felix
Quoted 3 lines
Lee
| Felix Kühling <[email protected]> http://fxk.de.vu | | PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 |
Quoted 1 line · Lee Revell
How to fix this is a pretty hot topic now.
Yow, I didn't mean to cause such an upset. ;)
Currently, the dri cvs snapshot for 20040905 doesn't compile with 2.6.8.1 for me (I've sent
a bug report to the dri-devel mailing list about this) so Lee and Michel, you'll have to wait
until tomorrow (or maybe even the day after that) to see how the test goes.
I'm hoping it does work, this bug is pretty nasty imho. Who knew Quake could take an entire box out in under 10 seconds. ;)
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 10 lines · Patrick McFarland
Quoted 1 line · Lee Revell
How to fix this is a pretty hot topic now.
Yow, I didn't mean to cause such an upset. ;)
Currently, the dri cvs snapshot for 20040905 doesn't compile with 2.6.8.1 for me (I've sent
a bug report to the dri-devel mailing list about this) so Lee and Michel, you'll have to wait
until tomorrow (or maybe even the day after that) to see how the test goes.
You can test the r200_dri.so from the snapshot with the DRM from the kernel...
Signature · 2 lines
Earthling Michel Dänzer | Debian (powerpc), X and DRI developer Libre software enthusiast | http://svcs.affero.net/rm.php?r=daenzer
Quoted 14 lines · --- Felix Kühling
Quoted 1 line · On Sun, 05 Sep 2004 20:14:43 -0400 Lee Revell <[email protected]> wrote:
On Sun, 2004-09-05 at 16:18, Patrick McFarland wrote:
[snip]
Quoted 2 lines
Quoted 2 lines
That shouldn't matter, should it? The userland stuff should never
lock
Quoted 5 lines
Quoted 2 lines
the machine up.
I'll test it anyhow, though.No, it shouldn't. Anything that directly accesses hardware belongs in the kernel. How to fix this is a pretty hot topic now.
Quoted 10 lines
That's not the whole truth. There are just too many ways to lock up those 3D chips. For instance I fixed a lockup in the r100 driver where the order in which state changing commands were sent to the hardware would cause a lockup. Each individual state changing command is perfectly valid. Finding all permutations that trigger a lockup would have been too much of a hassle and may not even have been true for all supported hardware out there. So we made the user-space driver emit state changing commands in a fixed order, which seems to work everywhere.
Dose the DRM varify that the cmds are in this order? Why not just have the DRM 'sort' the cmds? A simple bouble sort would have no more overhead then the check for correct order, but it would fix missordered cmd streams.
Once this is done the statement holds true, userland stuff should never...
Quoted 11 lines
Regars,
FelixQuoted 3 lines
Lee
| Felix Kühling <[email protected]> http://fxk.de.vu | | PGP Fingerprint: 6A3C 9566 5B30 DDED 73C3 B152 151C 5CC1 D888 E595 |
Quoted 11 lines
-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_idP47&alloc_id808&op=click
--
_______________________________________________
Dri-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/dri-devel
__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - Send 10MB messages!
http://promotions.yahoo.com/new_mail
Quoted 7 lines
Dose the DRM varify that the cmds are in this order? Why not just have the DRM 'sort' the cmds? A simple bouble sort would have no more overhead then the check for correct order, but it would fix missordered cmd streams.
Once this is done the statement holds true, userland stuff should never...
Feel free to implement it and profile it, but there are so many ways to lock up a radeon chip it is scary, the above was just one example, some days if you look at it funny it can lockup :-), it is accepted that userland can crap out 3D chips, the Intel ones are fairly easy to hangup also..
Dave.
Most IMPORTANT is that some-one some-where there is a list of ALL of these. These are best in the form of code comments so the the respective places in the code can be changed.
Quoted 8 lines · --- Dave Airlie
Quoted 1 line
Dose the DRM varify that the cmds are in this order? Why not just
have
Quoted 1 line
the DRM 'sort' the cmds? A simple bouble sort would have no more
overhead
Quoted 4 lines
then the check for correct order, but it would fix missordered cmd streams.
Once this is done the statement holds true, userland stuff should
Quoted 9 lines
never...
Quoted 1 line
Feel free to implement it and profile it, but there are so many ways to lock up a radeon chip it is scary, the above was just one example, some days if you look at it funny it can lockup :-), it is accepted that userland can crap out 3D chips, the Intel ones are fairly easy to hangup also..
I'd love to, where do I start? The problem he is that I have no-idea...
1. What values I'd neet to test for and sort.
2. The order of the sorting(probly documented in DRI-client code).
3. Where in the DRM I can proform the needed test and sort.
I would also love a list of ALL of these so I can fix them one by one. A good project for a new DRI developer, no.
Quoted 2 lines
Dave.
__________________________________
Do you Yahoo!?
Yahoo! Mail Address AutoComplete - You start. We finish.
http://promotions.yahoo.com/new_mail
Quoted 2 lines · Michel Dänzer
You can test the r200_dri.so from the snapshot with the DRM from the kernel...
And drum roll please...
The dri cvs snapshot works fine on both it's own kernel module, and the one that comes
with 2.6.8.1. So now what? (And does this mean it isn't a kernel bug?)
<rant>
Also, what happens to r200 users who happen to use Debian? Using dri cvs snapshots
obviously isn't an option for everyone (though I don't mind at all) and upgrading to Xorg
(when Xorg gets this fix if it doesn't already) is even less of an option. The official word
from the Debian X Strike Force is not to switch to Xorg until debriX (modular X) gets
somewhere.
</rant>
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 1 line · Patrick McFarland
Lots of badly formatted text.
I do apologize for anyone who had to read that.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 29 lines · Mike Mestnik
Most IMPORTANT is that some-one some-where there is a list of ALL of these. These are best in the form of code comments so the the respective places in the code can be changed.
Quoted 13 lines · --- Dave Airlie
Quoted 1 line
Dose the DRM varify that the cmds are in this order? Why not just
have
Quoted 1 line
the DRM 'sort' the cmds? A simple bouble sort would have no more
overhead
Quoted 4 lines
then the check for correct order, but it would fix missordered cmd streams.
Once this is done the statement holds true, userland stuff should
Quoted 8 lines
never...
Feel free to implement it and profile it, but there are so many ways to lock up a radeon chip it is scary, the above was just one example, some days if you look at it funny it can lockup :-), it is accepted that userland can crap out 3D chips, the Intel ones are fairly easy to hangup also..
Quoted 7 lines
I'd love to, where do I start? The problem he is that I have no-idea...
1. What values I'd neet to test for and sort.
2. The order of the sorting(probly documented in DRI-client code).
3. Where in the DRM I can proform the needed test and sort.I would also love a list of ALL of these so I can fix them one by one. A good project for a new DRI developer, no.
I seriously doubt this is doable. Unless you put the whole driver in the kernel, which of course nobody wants. I frequently caused gpu lockups by experimental driver changes (for instance, wrong vertex setup). I think the consensus was that it's ok for the driver to lock up the gpu, but it should not lock up the kernel. It might be possible to prevent lockups by a watchdog, resetting the gpu if a lockup is detected. This is how ATI deals with lockups in windows (dubbed "VPU Recover"), and there is a patch floating around for DRI too (though it is not exactly for that, and doesn't always work).
Roland
Quoted 2 lines · Patrick McFarland
Also, what happens to r200 users who happen to use Debian? Using dri cvs snapshots
If Debian is currently shipping a buggy driver then Debian needs to ship a working driver. Same as anyone else. You'll also need the newest dri driver for Radeon IGP (most ATI chipset laptops) and the newer R2xx hardware.
Alan
Quoted 7 lines · Patrick McFarland
Quoted 2 lines · Michel Dänzer
You can test the r200_dri.so from the snapshot with the DRM from the kernel...
And drum roll please...
Too early :(
Quoted 3 lines
The dri cvs snapshot works fine on both it's own kernel module, and the one that comes
with 2.6.8.1. So now what? (And does this mean it isn't a kernel bug?)
I have compiled both the kernel module and the replacement X server from the yesterday's CVS checkout of DRI (but according to "outdated" instructions in the Wiki). It made a difference against XFree86 4.4.0 + in-kernel radeon.ko.
The difference is that, instead of just hanging, after several minutes of run time applications (e.g. "really slick screensavers") print:
drmRadeonIrqWait: -16
and exit with status 1.
After that, 2D works, but the _next_ fullscreen OpenGL application hangs the system immediately on start.
Signature · 1 line
Alexander E. Patrakov