Skip to contentSkip to conversation

NForce2 pseudoscience stability testing (2.6.0-test11)

DISCUSSION Craig Bradney 5 participants 6 messages Dec 3, 2003 2 messages not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Jump down for my replies..

Quoted 7 lines · On Wed, 2003-12-03 at 00:36, [email protected] wrote:

Regardless of BIOS version, doesn’t the kernel get to fix values if things are "wrong?" I've seen this message on several boards where various tables and ACPI and APIC things are "fixed up." The only reason I want to find out so bad is that board survives unusual amounts of abuse on Windows 2000. (I havent been able to hang Windows as Prakash had.)

Quoted 8 lines

Right now I can't even afford to test 2.6.0.test11 in terms of time but very similar problems exist in 2.4, suggesting something fundamental?

About the IDE, it seems to be the easiest way to promote the problem but time seems to be the biggest factor. Some have suggested wrt this NFORCE2 problem that idle time makes it worse, but I've seen the hang under both conditions.

Not idle time.. my PC idles (with KDE/xchat/evolution running , if that is idling :)) while i am out or sleeping.. thats at least 4 hours non stop per day...

Quoted 4 lines

I have a very minimal set of things running on this board, I shut off the USB1.x and 2.0 controller, jumpered the Si SATA off and turned of Audio and the NVIDIA Lan. (Prakash suggested the Si SATA was the evil but I have it jumpered off.)

Audio on and used, USB 2.0, 1.x on but nothing plugged, NVIDIA lan off, 3com lan on and connected at 100Mbit.

Quoted 8 lines

For me the formula to reproduce this problem is this simple:
- Do anything (including passing the noapic nolapix noapixio etc) in a UP-kernel with APIC compiled in, get hang.
- To avoid the problem, recompile the UP-kernel with APIC turned off.

What probably isnt the problem:
- Motherboard flavor (This problem has cropped up on all nforce2 boards)

a7n8x deluxe v2 with standard clocked athlon xp 2600+

Quoted 2 lines

- BIOS (This problem can change nature with a BIOS rev, but I havent seen it go away and I tried several).

bios 1007.

Quoted 1 line

- SATA - problem appears even if SATA is shut off via jumper, as does Craig.

SATA is shut off on jumper. no SATA in kernel

Quoted 2 lines

- Memory - I swapped memory in a test to see. I also tried the dual channel and the single channel mode. No change.

dual channel 2x256mb

Quoted 3 lines

- SMP / UP / passed flags Julien said: "I run strictly non-SMP kernels and they always crash if APIC (or local APIC?) is enabled." - I see the same things.

smp off, preempt off. lapic on, apic on, acpi on

Quoted 2 lines

(Julien also suggest a quick way to see if the system is stable: type hdparm -t /dev/hd<someharddrive> several times).

as my uptime below suggests.. that didnt hurt a thing. I havent rebooted since that test.

Quoted 10 lines

- ASUS. I have a rev2.0 PCB, and several rev 1.x PCB, and they all suffer. Others also report boards from different manufacturers hanging with the NFORCE2. (Prakash: Abit NF7-S V2.0), Lenar (Epox 8RDA+ , 8RDA3+ )
- Kernel 2.4 or 2.6 specifically. This problem occurs both in later 2.6 builds and 2.4.23 (and below)
- IDE (NFORCE/AMD IDE driver in kernel) It seems the problem is easily exacerbated by IDE, but I think that this is not the root cause. In NO-APIC mode the IDE behaves reliably.

Quoted 2 lines

Craig said he ran it for 18 hours with abuse, including Juliens hdparm test.

Uptime is 4 days 1:55 and counting...

Things I can remember running in that time:
apache, mysql, proftpd, kde, evolution, xchat2, mozilla, konsole, scribus, any updates including gcc 3.2.3 from 3.2.2 that came through gentoo's portage system, gcc recompilation of scribus quite often (cvs updates and own coding), gdb for about 3 hours finding one scribus segfault testing over and over again with many crashes (fixed now btw for those scribus users out there :))

Quoted 3 lines

-> Craig, which kernel are you using? Distro (RedHat Taroon's kernel has LAPIC turned off)? PCB Rev of motherboard? Bios revision? Whats the lspci, cat /proc/interrupts look like? dmesg?

I'm using Gentoo's gentoo-dev-sources 2.6.0_beta11 which is 2.6 test11 plus the patches to be found at:
http://dev.gentoo.org/~brad_mssw/kernel_patches/2.6.0/genpatches-0.6/

They have released a 2.6_beta11-r1 that I ahvent upgraded to which uses
http://dev.gentoo.org/~brad_mssw/kernel_patches/2.6.0/genpatches-0.7/

and i notice there is a 0.8 directory.

So.. sorry if my comments have misled.. its not exactly standard 2.6 now that I do check it. There is one nvidia nforce network patch.

As above, PCB is 2.0, BIOS is standard 1007 from ASUS.

lspci and proc/interrupts follow. dmesg has been wiped out by some packet command errors with drive seek errors on hdc (dvdrw) from when i was playing music with that drive.

-----
lspci:
00:00.0 Host bridge: nVidia Corporation nForce2 AGP (different version?) (rev c1)
00:00.1 RAM memory: nVidia Corporation nForce2 Memory Controller 1 (rev c1)
00:00.2 RAM memory: nVidia Corporation nForce2 Memory Controller 4 (rev c1)
00:00.3 RAM memory: nVidia Corporation nForce2 Memory Controller 3 (rev c1)
00:00.4 RAM memory: nVidia Corporation nForce2 Memory Controller 2 (rev c1)
00:00.5 RAM memory: nVidia Corporation nForce2 Memory Controller 5 (rev c1)
00:01.0 ISA bridge: nVidia Corporation nForce2 ISA Bridge (rev a4) 00:01.1 SMBus: nVidia Corporation nForce2 SMBus (MCP) (rev a2) 00:02.0 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
00:02.1 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
00:02.2 USB Controller: nVidia Corporation nForce2 USB Controller (rev a4)
00:05.0 Multimedia audio controller: nVidia Corporation nForce MultiMedia audio [Via VT82C686B] (rev a2)
00:06.0 Multimedia audio controller: nVidia Corporation nForce2 AC97 Audio Controler (MCP) (rev a1)
00:08.0 PCI bridge: nVidia Corporation nForce2 External PCI Bridge (rev a3)
00:09.0 IDE interface: nVidia Corporation nForce2 IDE (rev a2) 00:0c.0 PCI bridge: nVidia Corporation nForce2 PCI Bridge (rev a3) 00:0d.0 FireWire (IEEE 1394): nVidia Corporation nForce2 FireWire (IEEE 1394) Controller (rev a3)
00:1e.0 PCI bridge: nVidia Corporation nForce2 AGP (rev c1) 02:01.0 Ethernet controller: 3Com Corporation 3C920B-EMB Integrated Fast Ethernet Controller (rev 40)
03:00.0 VGA compatible controller: ATI Technologies Inc Radeon R250 If [Radeon 9000] (rev 01)
03:00.1 Display controller: ATI Technologies Inc Radeon R250 [Radeon 9000] (Secondary) (rev 01)
-----
cat /proc/interrupts
CPU0

  0:  350520478          XT-PIC  timer
  1:     346366    IO-APIC-edge  i8042
  2:          0          XT-PIC  cascade
  8:          2    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 12:    3568875    IO-APIC-edge  i8042
 14:    2286009    IO-APIC-edge  ide0
 15:     153023    IO-APIC-edge  ide1
 19:   26315864   IO-APIC-level  radeon@PCI:3:0:0
 21:     855007   IO-APIC-level  ehci_hcd, NVidia nForce2, eth0
 22:          3   IO-APIC-level  ohci1394
NMI:          0
LOC:  350520402
ERR:          0
MIS:          0

-----

Quoted 3 lines

-> Josh, Craig & anyone else who gets this working on 2.6.test11 or some other fork of 2.6 can they please try 2.4.23-release as well and see if the machine hangs as well?

I was running gentoo's 2.4.23 pre 8 before this.. its still on the machine. I get no hangs with it.

This machine has never run Win* (its only 3 weeks old). Wouldnt go near an Uber BIOS on a still warranted motherboard.

Hope some of it helps.

Craig

Quoted 133 lines

The Linux APIC code generically works on most other hardware. Something specific to the NFORCE2 chips and its interaction with Linux's APIC code causes the hard hangs. The Windows 2000's APIC code was made before the NFORCE2 existed, and it seems to run fine there.

- About that Uber BIOS bios for the Asus Deluxe board, Anyone running
this: a) do you really want to run a hacked bios when other OS run fine on the unhacked BIOS b) do you believe that any of the un-hidden settings the uber bios or settings you may have changed helps solve this problem?

These bugs on the LK Bugzilla seem related:
http://bugme.osdl.org/show_bug.cgi?id=1203

Loosely related:
http://bugme.osdl.org/show_bug.cgi?id=1530
http://bugme.osdl.org/show_bug.cgi?id=1440
http://bugme.osdl.org/show_bug.cgi?id=1269

Does anyone know which developers would be interested in looking at this? I think it would be better if a specific
patch fixed this problem than try a kernel from bitkeeper on a daily basis and wait for the problem to go away without ever knowing what caused it.

Quoted 102 lines

To me the strangest thing is that when I first got this board a month or so ago it would hang with APIC or LAPIC enabled. Now it works fine without disabling APIC. All I did was update the BIOS and use it for a while with APIC disabled. 2.6.0-test9-mm through 2.6.0-test11 all work just fine. Still at the same time some people are reporting that it works, some are reporting that it doesn't. I probably wouldn't think to much of this except I was one of the ones that said APIC causes crashes with IDE load, but now it doesn't?

On approximately Tue, Dec 02, 2003 at 10:13:46AM +0000, ross.alexander wrote: Alistair,

I upgraded the BIOS about a week ago to 1007. I personally found it to be less stable than 1006. I don't believe it is a problem with my hardware combination since it has been stable for long periods of time. I was running the SMP kernel simply because I (wrongly) presumed a) you needed it to get the IO-APIC working, and b) it didn't do any harm.

It is clear that the UP kernel is considerable more stable than the SMP kernel. This is a very useful fact since it suggests that it is not a problem with the IDE device driver per se. The whole purpose of my testing is to try to determine which options increased the stability and hence highlight where the problem could be.

One of the reasons I don't like ACPI is the huge amount of additional complexity it adds and the amount of stuff it could screw up. Now I have not heard that any of the VIA KTxxx based motherboards have any problems. If this is true then the problem does not lie with the LAPIC, since that is in the processor, not the MB. The fact that it seems to only occur with the NForce2 chipset means it could well be some interrupt coming into the LAPIC from Interrupt Bus. However I certainly don't claim to be an expert on this so I could well be talking complete crap.

Conclusion: More testing required.

Cheers,

Ross

Alistair John Strachan <s0348365 28/11/2003
04:46 p.m.

To: ross.alexander
<brendan cc: linux-kernel
Subject: Re: NForce2 pseudoscience stability testing (2.6.0-test11)

On Friday 28 November 2003 15:13,
ross.alexander

The conclusion to this is the problem is in Local APIC with SMP. I'm not saying this is actually true only that is what the data suggests. If anybody wants me to try some other stuff feel free to suggest ideas.

Cheers,

Ross

It's evidently a configuration problem, albeit BIOS, mainboard revision, memory quality, etc. because I and many others like me are able to run Linux 2.4/2.6 with all the options you tested and still achieve absolute stability, on the nForce 2 platform.

My system is an EPOX 8RDA+, with an Athlon 2500+ (Barton) overclocked to 2.2Ghz, and 2x256MB TwinMOS PC3200 dimms. FSB is at 400Mhz, and the ram timings are 4,2,2,2. One might expect such a configuration to be unstable,

but it is not.

I'm currently running 2.6.0-test10-mm1 with full ACPI (+ routing), APIC and local APIC, no preempt, UP, and everything has been rock-solid, despite the machine being under constant 100% CPU load and fairly active IO load.

Also, many others have found that just disabling local apic (and the MPS setting in the BIOS) as well as ACPI solves their problem, so I'm skeptical that SMP really causes *nForce 2 specific* instability.

-- Cheers, Alistair.

personal: alistair()devzero!co!uk university:
s0348365()sms!ed!ac!uk student: CS/AI Undergraduate
contact: 7/10 Darroch Court, University of Edinburgh.

- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/

#7 lore raw
In reply to a message not in the archive
Quoted 3 lines · On Wed, 2003-12-03 at 00:36, [email protected] wrote:

Right now I can't even afford to test 2.6.0.test11 in terms of time but very similar problems exist in 2.4, suggesting something fundamental?

2.6.0-test11 deadlocked after less than a hour here.

Quoted 4 lines

About the IDE, it seems to be the easiest way to promote the problem but time seems to be the biggest factor. Some have suggested wrt this NFORCE2 problem that idle time makes it worse, but I've seen the hang under both conditions.

Well, IDE is what i'd blame. My original experience about lost interrupts leads me to ide. Since i never loose interrupts without io-apic.

Also, i tried to get it running with the ide cable that came with the board but that caused my 60 gig Quantum and my 80 gig Seagate disks to show up as 8mb disks with garbled names so i skipped that =) (2.6 also mentions something about cable info bits not set properly)

I use USB 2.0 and 1.1. I have also disabled audio and lan. A7N8X-X Bios 1007 (Nforce2-400).

Quoted 4 lines

The Linux APIC code generically works on most other hardware. Something specific to the NFORCE2 chips and its interaction with Linux's APIC code causes the hard hangs. The Windows 2000's APIC code was made before the NFORCE2 existed, and it seems to run fine there.

Well, IO-APIC without the amd/nvidia driver works, although a lost interrupt stalls the machine for a annoying-ammount-of-time.

Quoted 5 lines

- About that Uber BIOS bios for the Asus Deluxe board, Anyone running
this: a) do you really want to run a hacked bios when other OS run fine on the unhacked BIOS b) do you believe that any of the un-hidden settings the uber bios or settings you may have changed helps solve this problem?

Yeah, anyone that has checked the changes?

Signature · 1 line
Ian Kumlien <[email protected]>
#3 lore raw
Quoted 4 lines · Prakash K. Cheemplavam
Quoted 1 line

smp off, preempt off. lapic on, apic on, acpi on

Why haven't you enabled preempt? Does it lock with preempt on?

Having been more a lurker on the list before this issue, I had read that there were some preempt issues with 2.6.. so I turned it off. No idea if it locks with preempt on. Next time I do a recompile Ill try it out.

Craig

Quoted 8 lines

Prakash

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw
In reply to Prakash K. Cheemplavam

In article <[email protected]>,
Prakash K. Cheemplavam <[email protected]> wrote:

| > smp off, preempt off. lapic on, apic on, acpi on
| 
| Why haven't you enabled preempt? Does it lock with preempt on?

I haven't found preempt to cause problems with my applications, but neither have I seen any significant gain in throughput or responsiveness in 2.6. I did see small gains in 2.4, but not enough to motivate me to add the feature as a patch or even, in 2.6, enable it as an option.

I don't know about the original poster, but the reports of problems discourage exploration unless the application is likely to benefit from slight gains in response and to have multiple tasks of different priority running.

Have you seen a benefit from preempt? And if so, what application and how much? I may not be doing anything which benefits from preempt enough to notice (news, mail and dns servers, desktops), but other than dns on 2.4 I haven't seen improvement.

enlightenmet, please?

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#6 lore raw
In reply to a message not in the archive
Quoted 5 lines · bill davidsen

In article <[email protected]>,
Prakash K. Cheemplavam <[email protected]> wrote:
| > smp off, preempt off. lapic on, apic on, acpi on
| | Why haven't you enabled preempt? Does it lock with preempt on?

[snip]

Quoted 6 lines

Have you seen a benefit from preempt? And if so, what application and how much? I may not be doing anything which benefits from preempt enough to notice (news, mail and dns servers, desktops), but other than dns on 2.4 I haven't seen improvement.

enlightenmet, please?

Well, I am fairly new to Linux. Only since a month or so I am regurlarly using Linux and only since 2.6 kernel. I always used preempt and had no problems. The only problem I have, is with APIC. 2.6 is very smooth for me. I don't know whether it is because of preempt or not. I just was curious, why Craig didn't use it and whether the is a connection.

Prakash