Hang problem on Tyan K7 Thunder resolved -- SB Live! heads-up

The evidence for this is indirect but strong. The hang happened in every pre-2.4.8 configuration we tested if there was an SB Live! actually in the machine. It never happened if either 2.4.8 was running or the SB Live! was removed. This theory also accounts for our failure to observe hangs during

Quoted 76 lines

moderate to intense X GUI activity -- that traffic was going over the AGP bus, and we had enough memory in the box that it never swapped.

Now that we seem to be out of the woods, I can cop to why I'm doing qualification tests on bleeding-edge PCs. I'm writing an article for Linux Journal on building the ultimate Linux box. I won't spoil the surprise by telling you what else is in the machine, but I will tell you that it is jaw-droppingly fast and sexy hardware and that you'll get to read all about it before the end of the year.

In the meantime, here is my draft writeup on the hang problem:

<sect1 id='horror_story'><title>The Inevitable Horror Story</title>

<para>Sadly, life got much less pleasant for quite a while after that. We started seeing mysterious hangs -- the machine would lock up hard and random intervals, usually during disk I/O operations. This is almost the worst kind of problem to troubleshoot, as it leaves no clues other than the bare fact of the machine's catatonia -- you get no oops message, and all the state you might have used to post-mortem disappears when the machine is reset. The only kind of problem that's worse is one that adds irreproducibility to the catatonia. But fortunately, we found that doing <command>make clean</command> or <command>make world</command> on an X source tree produced the hang pretty reliably.</para>

<para>Approximately thirty hours of troubleshooting (interrupted by far too little sleep) ensued as Gary and I tried to track down the problem. We formed and discarded lots of theories based on where we had not yet seen the hang. For a while we thought the problem only bit in console mode, not in X mode. For another while we thought it happened only under SMP kernels. For a third while we thought we could avoid it by compiling kernels for the Pentium II rather than the Athlon. All these beliefs were eventually falsified amidst much wailing and gnashing of teeth.</para>

<para>Once it became clear that there was a problem at or near the hardware level, we still had a lot of hypotheses to choose from -- with all of them having pretty unpleasant ramifications for our chances of qualifying this box before I had to fly home. Quite possibly the motherboard was bad. Or we might have been seeing thermal flakeouts due to insufficient cooling of the motherboard chips or memory.</para>

<para>About eighteen hours in, just before we both crashed in exhaustion, we posted the problem to the <email>linux-kernel</email> mailing list. We got a rather larger number of responses than we expected (nearly twenty) within a few hours. Several were quite helpful. And the breakthrough came when a couple of linux-kernel people confirmed that the SB Live! is a frequent source of hangs and lockups on other fast PCI machines. With a few more hours of testing (during which our X source tree probably got cleaned and rebuilt more times than is allowed by law) we satisfied ourselves that the lockups stop happening when the SB Live! has been summarily yanked from the machine.</para>

<para>The most helpful advice we got came from one Daniel T. Chen, who reported that he had nailed some similar lockups to the SB Live! running over a Via chipset -- and that they stopped when he upgraded to 2.4.8 and the newest version of the emu10k1 driver. So while Gary took a much-needed break (and his wife and kids to a David Byrne concert), I built 2.4.8 (with emu10k1.o hard-compiled in) and ran our torture test -- first with the SB Live! omitted, and then with it in the machine. No hang. Victory!</para>

<para>Perhaps it's belaboring the obvious, but the way this problem got resolved was yet another testimony to the power of open-source development and the community that has evolved around it. Once again, our technology and our social machine complemented each other and delivered the goods.</para>

Signature · 9 lines

		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

What is a magician but a practicing theorist?
	-- Obi-Wan Kenobi, 'Return of the Jedi'
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

I've used ALSA for quite awhile for EMU10k1 (E-MU APS, not SBLive) and have had no problems. I noticed that the EMU10K1 driver was updated in 2.4.8 so I tried it. I had a lockup four times during audio playback, so I switched back to ALSA and now everything is stable once again.

The in kernel one seemed fine. The 2.4.8 update one is definitely broken on
SMP boxes

Quoted 3 lines

The in kernel one seemed fine. The 2.4.8 update one is definitely broken on
SMP boxes

I'm getting 2.4.8 Oopsen that seem to be in emu10k1 code on UP - see my message "2.4.8 oops in ksoftirqd_CPU0"...

Quoted 3 lines

so I think Linus should do the only sane thing - back it out. I'm backing it out of -ac. Of my three boxes, one spews noise, one locks up smp and one works.

But later there was a patch from maintainer (Rui Sousa). Besides, there were no updates from maintainers for over a year, so I think "non maintainer" did a good thing - at least maintainers started to > send patches finally. Instead of backing out I suggest for maintainers to send more patches...

so I think Linus should do the only sane thing - back it out. I'm backing it out of -ac. Of my three boxes, one spews noise, one locks up smp and one works.

The problem with backing it out is that apparently nobody has tried to really maintain it for a year, and if it gets backed out nobody will even bother to try to fix it. So I'll let it be for a while, at least.

Quoted 3 lines

The problem with backing it out is that apparently nobody has tried to really maintain it for a year, and if it gets backed out nobody will even bother to try to fix it. So I'll let it be for a while, at least.

I thought this was a stable kernel tree not 2.5 ?

Alan

Quoted 3 lines

The problem with backing it out is that apparently nobody has tried to really maintain it for a year, and if it gets backed out nobody will even bother to try to fix it. So I'll let it be for a while, at least.

I thought this was a stable kernel tree not 2.5 ?

Quoted 3 lines

The problem with backing it out is that apparently nobody has tried to really maintain it for a year, and if it gets backed out nobody will even bother to try to fix it. So I'll let it be for a while, at least.

I thought this was a stable kernel tree not 2.5 ?

Oops while *using* the driver seem to be much less severe than random lockups when *NOT* using driver. According to ESR, 2.4.7 is doing the latter.

Quoted 7 lines · On 2001.08.12 05:04 Alan Cox wrote:

Quoted 3 lines

The in kernel one seemed fine. The 2.4.8 update one is definitely broken on
SMP boxes

I'm getting 2.4.8 Oopsen that seem to be in emu10k1 code on UP - see my message "2.4.8 oops in ksoftirqd_CPU0"...

Yep. So far the new driver that Linus took from a non maintaier breaks

SMP
Some mixers
Uniprocessor with some cards
Surround sound (spews noise on cards)

so I think Linus should do the only sane thing - back it out. I'm backing it out of -ac. Of my three boxes, one spews noise, one locks up smp and one works.

Alan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 2 lines · On 2001.08.12 05:04 Alan Cox wrote:

Quoted 1 line

The in kernel one seemed fine. The 2.4.8 update one is definitely

broken

Quoted 5 lines

Quoted 2 lines

on
SMP boxes

I'm getting 2.4.8 Oopsen that seem to be in emu10k1 code on UP - see my message "2.4.8 oops in ksoftirqd_CPU0"...

Yep. So far the new driver that Linus took from a non maintaier breaks

SMP
Some mixers
Uniprocessor with some cards
Surround sound (spews noise on cards)

so I think Linus should do the only sane thing - back it out. I'm backing it out of -ac. Of my three boxes, one spews noise, one locks up smp and one works.

Quoted 7 lines · On 2001.08.12 13:15 Alan Cox wrote:

Yep. So far the new driver that Linus took from a non maintaier breaks

SMP
Some mixers
Uniprocessor with some cards
Surround sound (spews noise on cards)

Quoted 1 line

so I think Linus should do the only sane thing - back it out. I'm

backing

Quoted 2 lines

it out of -ac. Of my three boxes, one spews noise, one locks up smp and one works.

Does the CVS driver work? If it does I can try that one instead of the in-kernel one.

Quoted 1 line

EIP; d796940f <[emu10k1]emu10k1_waveout_bh+f/70> <=====

I've answered that one on my own - I installed today's CVS emu10k1 and got another Oops:

In article <[email protected]> you write:

Quoted 2 lines

I've answered that one on my own - I installed today's CVS emu10k1 and

got

Quoted 1 line

another Oops:

The oops seems to be due to "wave_dev->woinst" being NULL.

Can you try just adding the line

	if (!woinst)
		return;

to the top of the function (just before the "spin_lock_irqsave()"). Does that fix it for you?

Linus

Quoted 6 lines

Can you try just adding the line

	if (!woinst)
		return;

Quoted 4 lines

to the top of the function (just before the "spin_lock_irqsave()"). Does that fix it for you?

Linus

So far so good - however I don't have a consistent way to reproduce this. I'll just keep running and see if the Oops happens again.

Mind trying an alternate approach: remove the "if (!woinst)" thing, and instead move the line that initializes the tasklets down two lines (there's two places, they look something like

                tasklet_init(&wiinst->timer.tasklet, emu10k1_wavein_bh, 
(unsigned long) wave_dev);
                wave_dev->wiinst = wiinst;
                emu10k1_wavein_setformat(wave_dev, &wiinst->format);

and they _should_ do the "tasklet_init()" _after_ the other initializations, ie move that line down a bit, like so:

                wave_dev->wiinst = wiinst;
                emu10k1_wavein_setformat(wave_dev, &wiinst->format);
                tasklet_init(&wiinst->timer.tasklet, emu10k1_wavein_bh,

(unsigned long) wave_dev);

Does that also fix it?

And sure, I realize that you want to run it for a while..

Call me dumb, but what was wrong with the SB Live! code in the 2.4.7 trees? Mine works fine and has since I first installed RH 7.1 on this system. The only problem I had was when I compiled it into the kernel (instead of compiling as a module), sound would not work and I could not configure it with sndconfig.

Call me dumb, but what was wrong with the SB Live! code in the 2.4.7 trees? Mine works fine and has since I first installed RH 7.1 on this system. The only problem I had was when I compiled it into the kernel (instead of compiling as a module), sound would not work and I could not configure it with sndconfig.

In article <[email protected]> you write:

Quoted 5 lines

Call me dumb, but what was wrong with the SB Live! code in the 2.4.7 trees? Mine works fine and has since I first installed RH 7.1 on this system. The only problem I had was when I compiled it into the kernel (instead of compiling as a module), sound would not work and I could not configure it with sndconfig.

Well, the fact that it didn't work when compiled into the kernel means (for me), that it doesn't work at all.

Also, if you followed the other thread on the Tyan Thunder lockup, you'll have noticed that it locked up under heavy PCI loads. At least on that machine it stopped with the 2.4.8 driver.

Does the new driver not work for you? There seems to be a bug at close() time, in that the driver uses "tasklet_unlock_wait()" instead of "tasklet_kill()" to kill the tasklets, and that wouldn't work reliably.

Anything else you can find?

Call me dumb, but what was wrong with the SB Live! code in the 2.4.7 trees? Mine works fine and has since I first installed RH 7.1 on this system. The only problem I had was when I compiled it into the kernel (instead of compiling as a module), sound would not work and I could not configure it with sndconfig.

PGA

Signature · 9 lines

Paul G. Allen
UNIX Admin II/Network Security
Akamai Technologies, Inc.
www.akamai.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Linus Torvalds wrote:

Quoted 3 lines

Also, if you followed the other thread on the Tyan Thunder lockup, you'll have noticed that it locked up under heavy PCI loads. At least on that machine it stopped with the 2.4.8 driver.

I hadn't read the entire thread, but I did see that heavy loads seemed to be a problem. I am running a GeForce 3 and when I play games or do other graphics related things - I have a few OpenGL projects I've been working on - I try sqeak every bit of detail and piece of "eye candy" I can out of the system, as well as the best available sound. I'd say running Quake 3 at 1600x1200x32 in multiplayer should be a heavy load on everything, including PCI.