Skip to contentSkip to conversation

Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

DISCUSSION Ross Dickson 5 participants 15 messages Dec 13, 2003 – Dec 15, 2003

lore
Share & tools
lore
#0 lore raw

Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.

These patches may not be required if your bios is sorted out - mine are not yet.

The apic ack delay no longer adds a small delay to every timer int so I believe its performance hit is barely detectable.

In fact the busier the system gets with irqs- the more likelyhood of the delay time expiring naturally and the less the impact of the patch.

It now only delays the ones that would be too quick and most likely cause a hard lockup. It also reports its existence on boot if used. e.g.

..APIC TIMER ack delay, reload:16701, safe:16691

And if

#define APIC_DEBUG 0

is set to 1 in

/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h

Then you can report at boot ten pre delay apic timer times as well to get a feel as to whether the delay had to kick in or not. Note that ten is a very small sample size but it gives an idea of the timing numbers.

The apic timer counts down from the reload value and interrupts at zero. The reload value corresponds to the time between HZ at the rate the counter counts. Nothing new so far.
e.g. My system is running 1000Hz so the time is 1ms.
The reload value is 16701 so that represents 1ms.

The safe count it calculates for my system is 16691 so any number smaller than that is expected to be when it is safe to ack the local apic after an apic timer interrupt. If you want to experiment with the delay time in nano seconds just change the 600UL. If I set mine to 400UL then my hard lockups return. Interestingly I can read the local apic timer safely right after the interrupt - I just can't safely ack it then. It is as if the C1 disconnect logic within the processor only screws with the ACK irq cancellation logic and then only for a short time after an irq.

The io-apic edge patch has been cleaned up a little. If the above debug is on then it will display the 8259a xtpic interrupt mask. I get hex fb and hex ff indicating that the only int enabled on the 8259A xtpic which handles irq 0-7 is the cascade irq 2 which is OK because on the other 8259A irq 8-15 are masked. This masked xtpic should always be the case. We want the 8259A off during our test to see if the 8254 timer is connected directly to pin 0 of the ioapic. The other messages are as per previous version.

Lastly I need sleep (4 am) so they are not yet done as patch files. I have put them into two text files and bzipped them as an attachment. Anyhow they are small and insert in the same places as their previous versions.

Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat the favour with these please?

Again they are still experimental.... so I am hoping Ian, Jesse and others will put them to the test.

Thanks
Ross Dickson.

#1 lore raw
Quoted 4 lines · Ross Dickson

Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.

<snip>

Quoted 10 lines

Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat the favour with these please?

Again they are still experimental.... so I am hoping Ian, Jesse and others will put them to the test.

Thanks
Ross Dickson.

I am not Jesse but I thought I would diff them for 2.6 myself. Thanks Ross and others for putting so much time and effort into this. I haven't had time to test them, but I soon will. Uptime is:
15:19:45 up 2 days, 3:53, 7 users, load average: 0.05, 0.28, 0.37
with the original v1 patches.

Patches attached.

Signature · 6 lines
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin
arch/i386/kernel/apic.c +26 -0
diff -urN linux-2.6.0-test11/arch/i386/kernel/apic.c linux-2.6.0-test11-nf2/arch/i386/kernel/apic.c
@@ -1089,6 +1089,32 @@
 	 */
 	irq_stat[cpu].apic_timer_irqs++;
 
+#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
+	/*
+	 * on 2200XP & nforce2 chipset we need 600ns?
+	 * from timer irq start to apic irq ack to prevent
+	 * hard lockups, use apic timer itself.
+	 * C1 disconnect bit related.  Ross Dickson.
+	 */
+	{
+	  static unsigned int passno, safecnt;
+	  if(!passno {/*calculate timing*/
+	    safecnt = apic_read(APIC_TMICT) -
+	      ( (600UL * apic_read(APIC_TMICT) ) /
+		(1000000000UL/HZ) );
+	    printk("..APIC TIMER ack delay, reload:%u, safe:%u\n",
+		   apic_read(APIC_TMICT), safecnt);
+	    passno++;
+	  }
+#if APIC_DEBUG
+	     if(passno<12) {
+	       unsigned int at1 = apic_read(APIC_TMCCT);
+	       if(passno>1)
+		 Dprintk("..APIC TIMER ack delay, predelay count: %u \n", at1 );
+	       passno++;
+	     }
+#endif
+
 	/*
 	 * NOTE! We'd better ACK the irq immediately,
 	 * because timer handling can be slow.
arch/i386/kernel/io_apic.c +48 -0

54 lines not on this page · raw message

#2 lore raw
In reply to Ross Dickson
Quoted 3 lines · Ross Dickson

Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.

I have 3days of uptime here... It's low due to me wanting to test dual channel memory...

Quoted 2 lines

These patches may not be required if your bios is sorted out - mine are not yet.

Neither is mine =P

Quoted 2 lines

The apic ack delay no longer adds a small delay to every timer int so I believe its performance hit is barely detectable.

Niiice

Quoted 8 lines

In fact the busier the system gets with irqs- the more likelyhood of the delay time expiring naturally and the less the impact of the patch.

It now only delays the ones that would be too quick and most likely cause a hard lockup. It also reports its existence on boot if used. e.g.

..APIC TIMER ack delay, reload:16701, safe:16691

Quoted 7 lines

And if

#define APIC_DEBUG 0

is set to 1 in

/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h

Right now i'm compiling and hope that the patch doesn't need more workups. (Josh McKinney patches missed some things, more on that further down)

Quoted 10 lines

Then you can report at boot ten pre delay apic timer times as well to get a feel as to whether the delay had to kick in or not. Note that ten is a very small sample size but it gives an idea of the timing numbers.

The apic timer counts down from the reload value and interrupts at zero. The reload value corresponds to the time between HZ at the rate the counter counts. Nothing new so far.
e.g. My system is running 1000Hz so the time is 1ms.
The reload value is 16701 so that represents 1ms.

Quoted 8 lines

The safe count it calculates for my system is 16691 so any number smaller than that is expected to be when it is safe to ack the local apic after an apic timer interrupt. If you want to experiment with the delay time in nano seconds just change the 600UL. If I set mine to 400UL then my hard lockups return. Interestingly I can read the local apic timer safely right after the interrupt - I just can't safely ack it then. It is as if the C1 disconnect logic within the processor only screws with the ACK irq cancellation logic and then only for a short time after an irq.

Did you see the 240usecond delay that happens after a disconnect? But according to the amd northbridge spec, the pci bus will be disabled or something at that time...

Quoted 13 lines

The io-apic edge patch has been cleaned up a little. If the above debug is on then it will display the 8259a xtpic interrupt mask. I get hex fb and hex ff indicating that the only int enabled on the 8259A xtpic which handles irq 0-7 is the cascade irq 2 which is OK because on the other 8259A irq 8-15 are masked. This masked xtpic should always be the case. We want the 8259A off during our test to see if the 8254 timer is connected directly to pin 0 of the ioapic. The other messages are as per previous version.

Lastly I need sleep (4 am) so they are not yet done as patch files. I have put them into two text files and bzipped them as an attachment. Anyhow they are small and insert in the same places as their previous versions.

Quoted 5 lines

Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat the favour with these please?

Again they are still experimental.... so I am hoping Ian, Jesse and others will put them to the test.

Compiling 2.6.0 atm, since i want to test that somemore on this machine...

Quoted 5 lines · Josh McKinney

I am not Jesse but I thought I would diff them for 2.6 myself. Thanks Ross and others for putting so much time and effort into this. I haven't had time to test them, but I soon will. Uptime is:
15:19:45 up 2 days, 3:53, 7 users, load average: 0.05, 0.28, 0.37
with the original v1 patches.

apic part is missing a closing ) and the last part of the #ifdef =) So, either rediff or let the users fix that up =)

Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
#4 lore raw
In reply to Ross Dickson
Quoted 1 line · Ross Dickson

..APIC TIMER ack delay, reload:16701, safe:16691

calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757
---

Survived my greptest which no non patched kernel has ever done on this machine.

Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?
(I mean, is this like the non updated gnome-terminal in mdk 9.1 that deadlocks on 2.6 if you run some ncurces apps in a "larger than usual" window?)

Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
#5 lore raw
Quoted 18 lines · you
Quoted 1 line · Ross Dickson

..APIC TIMER ack delay, reload:16701, safe:16691

calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757

Thanks Ian.

Quoted 1 line

From this we see your local apic is indeed counting 1.2 times faster than mine

ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time. Given 20791 is 1ms on your system then your 12 counts is 577ns
But more importantly from the ack delay theory as your machine like mine is prone to lockups then a lockup could likely have occured at count:20786 having only 240ns time expired. Next worst case was less likely to lockup at count:20780.

The only ones any delay would have been added to by the patch would be the count:20786 and count:20780 and it would have been just enough to wait until the counter got below the safe:20779 so the patch contributes little overhead.

Quoted 8 lines

---

Survived my greptest which no non patched kernel has ever done on this machine.

Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?

This may be what you have already tried:
I am not sure where it is in the 2.6 config or indeed if it is different but it is CONFIG_LOG_BUF_SHIFT under kernel hacking on 2.4.23 maybe try 16 for 64K. To match dmesg output try

dmesg -s65536

(unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)

Quoted 7 lines

(I mean, is this like the non updated gnome-terminal in mdk 9.1 that deadlocks on 2.6 if you run some ncurces apps in a "larger than usual" window?)

Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
#6 lore raw
Quoted 20 lines · Ross Dickson
Quoted 18 lines · you
Quoted 1 line · Ross Dickson

..APIC TIMER ack delay, reload:16701, safe:16691

calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757

Quoted 7 lines

Thanks Ian.
From this we see your local apic is indeed counting 1.2 times faster than mine ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time. Given 20791 is 1ms on your system then your 12 counts is 577ns
But more importantly from the ack delay theory as your machine like mine is prone to lockups then a lockup could likely have occured at count:20786 having only 240ns time expired. Next worst case was less likely to lockup at count:20780.

I just had a lockup running with preempt, now trying with preempt disabled. This is a clean 2.6.0-test11 with just io-apic and apic v2 patches.

Quoted 11 lines

The only ones any delay would have been added to by the patch would be the count:20786 and count:20780 and it would have been just enough to wait until the counter got below the safe:20779 so the patch contributes little overhead.

Quoted 6 lines

Survived my greptest which no non patched kernel has ever done on this machine.

Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?

Quoted 8 lines

This may be what you have already tried:
I am not sure where it is in the 2.6 config or indeed if it is different but it is CONFIG_LOG_BUF_SHIFT under kernel hacking on 2.4.23 maybe try 16 for 64K. To match dmesg output try

dmesg -s65536

(unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)

Ahhh great!, no, it doesn't auto detect it... Maybe there is a newer version, i hate mdk for being so nice to new versions and ignoring the old.

Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
#13 lore raw
In reply to Ross Dickson

Hi again,

Unfortunately that kernel deadlocked as well... Both times while downloading, is there some patches that i have missed for 2.6?

Please send em all if you have been successful in running it past the 'crash deadline'.

Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
#7 lore raw
In reply to Ian Kumlien
Quoted 29 lines · you
Quoted 20 lines · Ross Dickson
Quoted 18 lines · you
Quoted 1 line · Ross Dickson

..APIC TIMER ack delay, reload:16701, safe:16691

calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757

Quoted 7 lines

Thanks Ian.
From this we see your local apic is indeed counting 1.2 times faster than mine ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time. Given 20791 is 1ms on your system then your 12 counts is 577ns
But more importantly from the ack delay theory as your machine like mine is prone to lockups then a lockup could likely have occured at count:20786 having only 240ns time expired. Next worst case was less likely to lockup at count:20780.

Quoted 3 lines

I just had a lockup running with preempt, now trying with preempt disabled. This is a clean 2.6.0-test11 with just io-apic and apic v2 patches.

Thanks for testing, thats not good news.
Try rebooting with kernel args to get back in.

acpi=off noapic

My 600ns is likely too small - try putting 800UL or 1000UL in place of the 600UL

The v1 patch ((cpu_khz >> 12)+200) gave 700ns additional delay to your existing code path so I think I was being too optimistic in the initial v2 timing. I made the initial timing CPU freq dependent on the assumption that a faster cpu would get to the delay point quicker.

On my system the v1 patch had 13 counts of total delay whereas my initial the v2 gave only 10 so I think 800UL is the smallest value we should use.

If that does not fix it then it could also be that reading the timer so early may also be dangerous? or something else? and so then we would go back to v1 apic patch.

Quoted 17 lines
Quoted 17 lines

The only ones any delay would have been added to by the patch would be the count:20786 and count:20780 and it would have been just enough to wait until the counter got below the safe:20779 so the patch contributes little overhead.

Quoted 6 lines

Survived my greptest which no non patched kernel has ever done on this machine.

Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?

This may be what you have already tried:
I am not sure where it is in the 2.6 config or indeed if it is different but it is CONFIG_LOG_BUF_SHIFT under kernel hacking on 2.4.23 maybe try 16 for 64K. To match dmesg output try

Quoted 7 lines
Quoted 3 lines

dmesg -s65536

(unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)

Ahhh great!, no, it doesn't auto detect it... Maybe there is a newer version, i hate mdk for being so nice to new versions and ignoring the old.

My wishful thinking, I don't think any version autodetects but it would be a nice feature. Ross.

Quoted 4 lines
Signature · 2 lines
-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
#8 lore raw
Quoted 5 lines · Ross Dickson

The v1 patch ((cpu_khz >> 12)+200) gave 700ns additional delay to your existing code path so I think I was being too optimistic in the initial v2 timing. I made the initial timing CPU freq dependent on the assumption that a faster cpu would get to the delay point quicker.

Maybe it scales with bus speed, not CPU internal speed?

-- Jamie

#3 lore raw
In reply to Ian Kumlien
Quoted 5 lines · Ian Kumlien
Quoted 3 lines · Ross Dickson

Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.

<snip>

Quoted 5 lines

Right now i'm compiling and hope that the patch doesn't need more workups. (Josh McKinney patches missed some things, more on that further down)

<snip>

Quoted 6 lines

apic part is missing a closing ) and the last part of the #ifdef =) So, either rediff or let the users fix that up =)

Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

Oops. I was playing kernel hacker while I should have been doing other things. Patch rediffed and both attached. Thanks for catching my mess up.

Signature · 6 lines
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin
arch/i386/kernel/io_apic.c +48 -0
diff -urN linux-2.6.0-test11/arch/i386/kernel/io_apic.c linux-2.6.0-test11-nf2/arch/i386/kernel/io_apic.c
@@ -2128,6 +2128,54 @@
 		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
 	}
 
+#ifdef CONFIG_ACPI_BOOT && CONFIG_X86_UP_IOAPIC
+        /* for nforce2 try vector 0 on pin0
+         * Note 8259a is already masked, also by default
+         * the io_apic_set_pci_routing call disables the 8259 irq 0
+         * so we must be connected directly to the 8254 timer if this works
+         * Note2: this violates the above comment re Subtle but works!
+         */
+        printk(KERN_INFO "..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...\n");
+        if (pin1 != -1) {
+                extern spinlock_t i8259A_lock;
+                unsigned long flags;
+                int tok, saved_timer_ack = timer_ack;
+                /*
+                 * Ok, does IRQ0 through the IOAPIC work?
+                 */
+                io_apic_set_pci_routing ( 0, 0, 0, 0, 0); /* connect pin */
+                unmask_IO_APIC_irq(0);
+                timer_ack = 0;
+
+                /*
+
+
+
+                 * Ok, does IRQ0 through the IOAPIC work?
+                 */
+                spin_lock_irqsave(&i8259A_lock, flags);
+                Dprintk("..TIMER check 8259 ints disabled, imr1:%02x, imr2:%02x\n", inb(0x21), inb(0xA1));
+                tok = timer_irq_works();
+                spin_unlock_irqrestore(&i8259A_lock, flags);
+                if (tok) {
+                        if (nmi_watchdog == NMI_IO_APIC) {
+                                disable_8259A_irq(0);
+                                setup_nmi();
+                                enable_8259A_irq(0);
+                                check_nmi_watchdog();
+                        }
+                        printk(KERN_INFO "..TIMER: works OK on apic pin0 irq0\n" );
+                        return;
+                }
+                /* failed */
+                timer_ack = saved_timer_ack;
+                clear_IO_APIC_pin(0, 0);
+                io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
+                printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC Pin 0\n");
+        }
+/* end new stuff for nforce2 */
+#endif
+
 	printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
 	if (pin2 != -1) {
 		printk("\n..... (found pin %d) ...", pin2);
arch/i386/kernel/apic.c +31 -0
diff -urN linux-2.6.0-test11/arch/i386/kernel/apic.c linux-2.6.0-test11-nf2/arch/i386/kernel/apic.c
@@ -1089,6 +1089,37 @@
 	 */
 	irq_stat[cpu].apic_timer_irqs++;
 
+#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
+        /*
+         * on 2200XP & nforce2 chipset we need 600ns?
+         * from timer irq start to apic irq ack to prevent
+         * hard lockups, use apic timer itself.
+         * C1 disconnect bit related.  Ross Dickson.
+         */
+        {
+                static unsigned int passno, safecnt;
+                if(!passno) { /* calculate timing */
+                        safecnt = apic_read(APIC_TMICT) -
+                                ( (600UL * apic_read(APIC_TMICT) ) /
+                                (1000000000UL/HZ) );
+                        printk("..APIC TIMER ack delay, reload:%u, safe:%u\n",
+                                apic_read(APIC_TMICT), safecnt);
+                        passno++;
+                }
+#if APIC_DEBUG
+                if(passno<12) {
+                        unsigned int at1 = apic_read(APIC_TMCCT);
+                        if( passno > 1 )
+                                Dprintk("..APIC TIMER ack delay, predelay count:%u \n", at1 );
+                        passno++;
+                }
+# endif
+                /* delay only if required */
+                while( apic_read(APIC_TMCCT) > safecnt )
+                        ndelay(100);
+        }
+#endif
+
 	/*
 	 * NOTE! We'd better ACK the irq immediately,
 	 * because timer handling can be slow.
#9 lore raw
In reply to Jamie Lokier
Quoted 8 lines · Jamie Lokier
Quoted 5 lines · Ross Dickson

The v1 patch ((cpu_khz >> 12)+200) gave 700ns additional delay to your existing code path so I think I was being too optimistic in the initial v2 timing. I made the initial timing CPU freq dependent on the assumption that a faster cpu would get to the delay point quicker.

Maybe it scales with bus speed, not CPU internal speed?

Very good point.

Hmmm It may well be the settling time of the cpu PLL (phase lock loop) changing from low power disconnect speed up to bus speed - higher speed bus on the same type of silicon may take longer? as also it may take more power to run faster so the internal bus bit drivers may also take more time to settle? These sorts of things are mentioned in earlier model athlon errata. Especially if there is perhaps a marginal northbridge timing in the area Ian has thought about?

The nforce2 ram controller is also unique and kind of predicts and caches reads so maybe the apic timer irq code is ready to run a lot sooner than other chipsets so the normal delays wrt loading from memory are not there? Are we compensating for that? We don't really know unfortunately.

Quoted 5 lines

-- Jamie

Josh - Thanks very much for the patches.

The v2 io-apic mods seems OK so far.

I am not yet sure the v2 apic patch is going to be stable enough for everyone. I suspect it is the apic reads.

Ian had problems at 600ns so now has been trying 800ns. I have yet to hear how his machine is.

I have also gone from 600ns to 800ns and had my first hard lockup after about 6 hrs on 800ns delay timeout.

I am using a heavily patched 2.4.23 kern so it could be a coincidence but I am suspicious that it may not be that safe to be reading the apic registers at all during the delay timeout as the v2 apic patch does.

I am now going to try increasing the wait loop delay from 100ns to 400ns in case the apic does not like being hammered repetitively during the delay time. - It could be that the bus between the cpu core and the local apic is marginal on either timing (PLL) or current and if we hammer it we may be asking for incorrect reads?

I am about to recompile with the following values that differ from my original v2 posting.
( 800UL and the ndelay(400) )

#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
	/*
	 * on AMDXP & nforce2 chipset we need about 800ns?
	 * from timer irq start to apic irq ack to prevent
	 * hard lockups, use apic timer itself.
	 * C1 disconnect bit related.  Ross Dickson.
	 */
	{
		static unsigned int passno, safecnt;
		if(!passno) { /* calculate timing */
			safecnt = apic_read(APIC_TMICT) -
				( (800UL * apic_read(APIC_TMICT) ) /
				(1000000000UL/HZ) );
			printk("..APIC TIMER ack delay, reload:%u, safe:%u\n",
				apic_read(APIC_TMICT), safecnt);
			passno++;
		}
#if APIC_DEBUG
		if(passno<12) {
			unsigned int at1 = apic_read(APIC_TMCCT);
			if( passno > 1 )
				Dprintk("..APIC TIMER ack delay, predelay count:%u \n", at1 );
			passno++;
		}
# endif
		/* delay only if required */
		while( apic_read(APIC_TMCCT) > safecnt )
			ndelay(400);
	}
#endif

Regards
Ross.

#10 lore raw

On Sunday 14 December 2003 21:24, Ross Dickson wrote:
<snip>

Quoted 22 lines

The v2 io-apic mods seems OK so far.

I am not yet sure the v2 apic patch is going to be stable enough for everyone. I suspect it is the apic reads.

Ian had problems at 600ns so now has been trying 800ns. I have yet to hear how his machine is.

I have also gone from 600ns to 800ns and had my first hard lockup after about 6 hrs on 800ns delay timeout.

I am using a heavily patched 2.4.23 kern so it could be a coincidence but I am suspicious that it may not be that safe to be reading the apic registers at all during the delay timeout as the v2 apic patch does.

I am now going to try increasing the wait loop delay from 100ns to 400ns in case the apic does not like being hammered repetitively during the delay time. - It could be that the bus between the cpu core and the local apic is marginal on either timing (PLL) or current and if we hammer it we may be asking for incorrect reads?

Quoted 3 lines

I am about to recompile with the following values that differ from my original v2 posting.
( 800UL and the ndelay(400) )

<snip>
I had a lockup on a boot so I am trying a bit more conservative with

1000UL and ndelay(400)

I don't think anyone should try any less than this but hey?

This gives my system a safety margin of 16 apic counts. The v1 patch on my system typically gave a safety delay of 13 counts.

The performance hit with the v2 is still less than with the v1. With v2 additional delay would only have been present on 2 out of 10 instead of 10 out of 10 with v1.

..APIC TIMER ack delay, reload:16701, safe:16685
..APIC TIMER ack delay, predelay count:16657
..APIC TIMER ack delay, predelay count:16664
..APIC TIMER ack delay, predelay count:16677
..APIC TIMER ack delay, predelay count:16691
..APIC TIMER ack delay, predelay count:16636
..APIC TIMER ack delay, predelay count:16649
..APIC TIMER ack delay, predelay count:16660
..APIC TIMER ack delay, predelay count:16671
..APIC TIMER ack delay, predelay count:16686
..APIC TIMER ack delay, predelay count:16635

Although as previously mentioned on my system it takes 28 apic counts to write an io byte to the south bridge such as an ack in xtpic mode so v1 was not what you would call slow.

If 28 counts was a benchmark then I could still try v2 up to about 1750UL and still be quicker than the xtpic.

Ross.

#11 lore raw

Resend due to no CC...

Quoted 4 lines · Ross Dickson

I had a lockup on a boot so I am trying a bit more conservative with

1000UL and ndelay(400)

Quoted 8 lines

I don't think anyone should try any less than this but hey?

This gives my system a safety margin of 16 apic counts. The v1 patch on my system typically gave a safety delay of 13 counts.

The performance hit with the v2 is still less than with the v1. With v2 additional delay would only have been present on 2 out of 10 instead of 10 out of 10 with v1.

13h 18min here now, with 800UL 100ns

Works like a charm. It seems like we are just avoiding a timer race with some values... ie more delay might not solve the problem.

Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
#12 lore raw
In reply to Ross Dickson
Quoted 7 lines · Ross Dickson

Hmmm It may well be the settling time of the cpu PLL (phase lock loop) changing from low power disconnect speed up to bus speed - higher speed bus on the same type of silicon may take longer? as also it may take more power to run faster so the internal bus bit drivers may also take more time to settle? These sorts of things are mentioned in earlier model athlon errata. Especially if there is perhaps a marginal northbridge timing in the area Ian has thought about?

If you're waiting for a PLL to settle, it could vary tremendously.

Is there some way to sense the disconnect state and put in a proper delay e.g. 10 microseconds when it is sensed? I'd guess the disconnect state isn't encounted much under system load, which is the only time you really care about fast timer interrupts.

Quoted 5 lines

I am now going to try increasing the wait loop delay from 100ns to 400ns in case the apic does not like being hammered repetitively during the delay time. - It could be that the bus between the cpu core and the local apic is marginal on either timing (PLL) or current and if we hammer it we may be asking for incorrect reads?

It's possible that hammering it will consume more bus power and push a stabilising PLL and/or power supply a bit far.

If it's purely marginal timing, then doing fewer reads just means you'll hit the marginal state less often, not avoid it completely.

-- Jamie

#14 lore raw
In reply to Ross Dickson
Quoted 4 lines · Ross Dickson

It also reports its existence on boot if used. e.g.

..APIC TIMER ack delay, reload:16701, safe:16691

Quoted 9 lines

And if

#define APIC_DEBUG 0

is set to 1 in

/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h

I'm trying the two new patches with 2.6 and my Award bios which fixes the crash problem itself but didn't turn on edge timer without a patch, and didn't turn on nmi_watchdog=1 with the first patch, only =2.

nmi_watchdog gets a lot more ticks with =1, too.

bob@where cat /proc/interrupts
CPU0

 0:     924534    IO-APIC-edge  timer
 1:       1070    IO-APIC-edge  i8042
 2:          0          XT-PIC  cascade
 8:          1    IO-APIC-edge  rtc
 9:          0   IO-APIC-level  acpi
12:      14077    IO-APIC-edge  i8042
14:         10    IO-APIC-edge  ide0
15:         10    IO-APIC-edge  ide1
16:      44086   IO-APIC-level  ide2, ide3, eth0
17:          0   IO-APIC-level  yenta, yenta
19:      35570   IO-APIC-level  ide4, ide5
21:          0   IO-APIC-level  NVidia nForce2
NMI:     924555
LOC:     924432
ERR:          0
MIS:         22

I stayed with 600UL 100ndelay just to see if anything breaks with amd XP3000+ and patches with a bios that doesn't crash with nforce2 but needs help from patches on other points(to get edge timer on and to use nmi_watchdog=1 rather than =2). Also hope we get a clue about what Award bios update does that Phoenix does not do so far.

/usr/src/kernel-source-2.6.0/include/asm-i386/apic.h #define APIC_DEBUG 1

...but I don't see any

calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769

etc

in the area of "NET: Registered" I see the following--

Dec 15 05:34:30 where kernel: PCI: Setting latency timer of device 0000:00:06.0 to 64
Dec 15 05:34:30 where kernel: intel8x0: clocking to 47414
Dec 15 05:34:30 where kernel: ALSA device list:
Dec 15 05:34:30 where kernel:   #0: Dummy 1
Dec 15 05:34:30 where kernel:   #1: Virtual MIDI Card 1
Dec 15 05:34:30 where kernel:   #2: NVidia nForce2 at 0xe5080000, irq 21
Dec 15 05:34:30 where kernel: NET: Registered protocol family 2
Dec 15 05:34:30 where kernel: IP: routing cache hash table of 8192 buckets, 64Kbytes
Dec 15 05:34:30 where kernel: TCP: Hash tables configured (established 262144 bind 65536)
Dec 15 05:34:30 where kernel: ip_conntrack version 2.1 (8191 buckets, 65528 max) - 296 bytes per conntrack
Dec 15 05:34:30 where kernel: ip_tables: (C) 2000-2002 Netfilter core team
Dec 15 05:34:30 where kernel: ipt_recent v0.3.1: Stephen Frost <[email protected]>.  http://snowman.net/projects/ipt_rece
nt/
Dec 15 05:34:30 where kernel: NET: Registered protocol family 1
Dec 15 05:34:30 where kernel: NET: Registered protocol family 17
Dec 15 05:34:30 where kernel: BIOS EDD facility v0.10 2003-Oct-11, 4 devices found
Dec 15 05:34:30 where kernel: Please report your BIOS at http://domsch.com/linux/edd30/results.html
Dec 15 05:34:30 where kernel: md: Autodetecting RAID arrays.
Dec 15 05:34:30 where kernel: NFORCE2: chipset revision 162

log starts with this acpi stuff--

Dec 15 05:34:30 where kernel: klogd 1.4.1#10, log source = /proc/kmsg started.
Dec 15 05:34:30 where kernel: Inspecting /boot/System.map-2.6.0.nf2
Dec 15 05:34:30 where kernel: Loaded 33440 symbols from /boot/System.map-2.6.0.nf2.
Dec 15 05:34:30 where kernel: Symbols match kernel version 2.6.0.
Dec 15 05:34:30 where kernel: No module symbols loaded - kernel modules not enabled.
Dec 15 05:34:30 where kernel: IRQs 3 4 *5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [LFIR] (IRQs 3 4 5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [L3CM] (IRQs 3 4 5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [LIDE] (IRQs 3 4 5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC1] (IRQs *16)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC2] (IRQs *17)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC3] (IRQs *18)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC4] (IRQs *19)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC5] (IRQs 16)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCF] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCG] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCH] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCI] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCJ] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCK] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCS] (IRQs *23)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCL] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCM] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [AP3C] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCZ] (IRQs 20 21 22)
Dec 15 05:34:30 where kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
Dec 15 05:34:30 where kernel: pnp: the driver 'system' has been registered
Dec 15 05:34:30 where kernel: PnPBIOS: Scanning system for PnP BIOS support...
Dec 15 05:34:30 where kernel: PnPBIOS: Found PnP BIOS installation structure at 0xc00fc660
Dec 15 05:34:30 where kernel: PnPBIOS: PnP BIOS version 1.0, entry 0xf0000:0xc690, dseg 0xf0000
Dec 15 05:34:30 where kernel: pnp: match found with the PnP device '00:07' and the driver 'system'
Dec 15 05:34:30 where kernel: pnp: match found with the PnP device '00:08' and the driver 'system'
Dec 15 05:34:30 where kernel: PnPBIOS: 13 nodes reported by PnP BIOS; 13 recorded by driver
Dec 15 05:34:30 where kernel: SCSI subsystem initialized
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCS] enabled at IRQ 23
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-23 -> 0xa9 -> IRQ 23 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:01[A] -> 2-23 -> IRQ 23
Dec 15 05:34:30 where kernel: Pin 2-23 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCF] enabled at IRQ 20
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-20 -> 0xb1 -> IRQ 20 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:02[A] -> 2-20 -> IRQ 20
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCG] enabled at IRQ 22
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-22 -> 0xb9 -> IRQ 22 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:02[B] -> 2-22 -> IRQ 22
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCL] enabled at IRQ 21
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-21 -> 0xc1 -> IRQ 21 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:02[C] -> 2-21 -> IRQ 21
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCH] enabled at IRQ 20
Dec 15 05:34:30 where kernel: Pin 2-20 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCI] enabled at IRQ 22
Dec 15 05:34:30 where kernel: Pin 2-22 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCJ] enabled at IRQ 21
Dec 15 05:34:30 where kernel: Pin 2-21 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCK] enabled at IRQ 20
Dec 15 05:34:30 where kernel: Pin 2-20 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCM] enabled at IRQ 22
Dec 15 05:34:30 where kernel: Pin 2-22 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [AP3C] enabled at IRQ 21
Dec 15 05:34:30 where kernel: Pin 2-21 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCZ] enabled at IRQ 20
Dec 15 05:34:30 where kernel: Pin 2-20 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC4] enabled at IRQ 19
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-19 -> 0xc9 -> IRQ 19 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[A] -> 2-19 -> IRQ 19
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC1] enabled at IRQ 16
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-16 -> 0xd1 -> IRQ 16 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[B] -> 2-16 -> IRQ 16
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC2] enabled at IRQ 17
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-17 -> 0xd9 -> IRQ 17 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[C] -> 2-17 -> IRQ 17
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC3] enabled at IRQ 18
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-18 -> 0xe1 -> IRQ 18 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[D] -> 2-18 -> IRQ 18
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where last message repeated 3 times
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: PCI: Using ACPI for IRQ routing