Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered
Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.
These patches may not be required if your bios is sorted out - mine are not yet.
The apic ack delay no longer adds a small delay to every timer int so I believe its performance hit is barely detectable.
In fact the busier the system gets with irqs- the more likelyhood of the delay time expiring naturally and the less the impact of the patch.
It now only delays the ones that would be too quick and most likely cause a hard lockup. It also reports its existence on boot if used. e.g.
..APIC TIMER ack delay, reload:16701, safe:16691
And if
#define APIC_DEBUG 0
is set to 1 in
/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h
Then you can report at boot ten pre delay apic timer times as well to get a feel as to whether the delay had to kick in or not. Note that ten is a very small sample size but it gives an idea of the timing numbers.
The apic timer counts down from the reload value and interrupts at zero. The reload value corresponds to the time between HZ at the rate the counter counts. Nothing new so far.
e.g. My system is running 1000Hz so the time is 1ms.
The reload value is 16701 so that represents 1ms.
The safe count it calculates for my system is 16691 so any number smaller than that is expected to be when it is safe to ack the local apic after an apic timer interrupt. If you want to experiment with the delay time in nano seconds just change the 600UL. If I set mine to 400UL then my hard lockups return. Interestingly I can read the local apic timer safely right after the interrupt - I just can't safely ack it then. It is as if the C1 disconnect logic within the processor only screws with the ACK irq cancellation logic and then only for a short time after an irq.
The io-apic edge patch has been cleaned up a little. If the above debug is on then it will display the 8259a xtpic interrupt mask. I get hex fb and hex ff indicating that the only int enabled on the 8259A xtpic which handles irq 0-7 is the cascade irq 2 which is OK because on the other 8259A irq 8-15 are masked. This masked xtpic should always be the case. We want the 8259A off during our test to see if the 8254 timer is connected directly to pin 0 of the ioapic. The other messages are as per previous version.
Lastly I need sleep (4 am) so they are not yet done as patch files. I have put them into two text files and bzipped them as an attachment. Anyhow they are small and insert in the same places as their previous versions.
Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat the favour with these please?
Again they are still experimental.... so I am hoping Ian, Jesse and others will put them to the test.
Thanks
Ross Dickson.
Quoted 4 lines · Ross Dickson
Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.
<snip>
Quoted 10 lines
Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat the favour with these please?
Again they are still experimental.... so I am hoping Ian, Jesse and others will put them to the test.
Thanks
Ross Dickson.
I am not Jesse but I thought I would diff them for 2.6 myself. Thanks Ross and others for putting so much time and effort into this. I haven't had time to test them, but I soon will. Uptime is:
15:19:45 up 2 days, 3:53, 7 users, load average: 0.05, 0.28, 0.37
with the original v1 patches.
Patches attached.
Signature · 6 lines
Josh McKinney | Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
| They that can give up essential liberty
Linux, the choice -o) | to obtain a little temporary safety deserve
of the GNU generation /\ | neither liberty or safety.
_\_v | -Benjamin Franklinarch/i386/kernel/apic.c +26 -0
@@ -1089,6 +1089,32 @@*/ irq_stat[cpu].apic_timer_irqs++; +#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX +/* +* on 2200XP & nforce2 chipset we need 600ns? +* from timer irq start to apic irq ack to prevent +* hard lockups, use apic timer itself. +* C1 disconnect bit related. Ross Dickson. +*/ + { +static unsigned int passno, safecnt; + if(!passno {/*calculate timing*/ + safecnt =apic_read (APIC_TMICT ) - + ( (600UL *apic_read (APIC_TMICT ) ) / + (1000000000UL /HZ ) ); +printk ("..APIC TIMER ack delay, reload:%u, safe:%u\n" , +apic_read (APIC_TMICT ), safecnt); + passno++; + } +#if APIC_DEBUG +if (passno<12 ) { +unsigned int at1 =apic_read (APIC_TMCCT ); +if (passno>1 ) +Dprintk ("..APIC TIMER ack delay, predelay count: %u \n" , at1 ); + passno++; + } +#endif +/* * NOTE! We'd better ACK the irq immediately, * because timer handling can be slow.
arch/i386/kernel/io_apic.c +48 -0
54 lines not on this page · raw message
Quoted 3 lines · Ross Dickson
Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.
I have 3days of uptime here... It's low due to me wanting to test dual channel memory...
Quoted 2 lines
These patches may not be required if your bios is sorted out - mine are not yet.
Neither is mine =P
Quoted 2 lines
The apic ack delay no longer adds a small delay to every timer int so I believe its performance hit is barely detectable.
Niiice
Quoted 8 lines
In fact the busier the system gets with irqs- the more likelyhood of the delay time expiring naturally and the less the impact of the patch.
It now only delays the ones that would be too quick and most likely cause a hard lockup. It also reports its existence on boot if used. e.g.
..APIC TIMER ack delay, reload:16701, safe:16691
Quoted 7 lines
And if
#define
APIC_DEBUG0is set to 1 in
/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h
Right now i'm compiling and hope that the patch doesn't need more workups. (Josh McKinney patches missed some things, more on that further down)
Quoted 10 lines
Then you can report at boot ten pre delay apic timer times as well to get a feel as to whether the delay had to kick in or not. Note that ten is a very small sample size but it gives an idea of the timing numbers.
The apic timer counts down from the reload value and interrupts at zero. The reload value corresponds to the time between HZ at the rate the counter counts. Nothing new so far.
e.g. My system is running 1000Hz so the time is 1ms.
The reload value is 16701 so that represents 1ms.
Quoted 8 lines
The safe count it calculates for my system is 16691 so any number smaller than that is expected to be when it is safe to ack the local apic after an apic timer interrupt. If you want to experiment with the delay time in nano seconds just change the 600UL. If I set mine to 400UL then my hard lockups return. Interestingly I can read the local apic timer safely right after the interrupt - I just can't safely ack it then. It is as if the C1 disconnect logic within the processor only screws with the ACK irq cancellation logic and then only for a short time after an irq.
Did you see the 240usecond delay that happens after a disconnect? But according to the amd northbridge spec, the pci bus will be disabled or something at that time...
Quoted 13 lines
The io-apic edge patch has been cleaned up a little. If the above debug is on then it will display the 8259a xtpic interrupt mask. I get hex fb and hex ff indicating that the only int enabled on the 8259A xtpic which handles irq 0-7 is the cascade irq 2 which is OK because on the other 8259A irq 8-15 are masked. This masked xtpic should always be the case. We want the 8259A off during our test to see if the 8254 timer is connected directly to pin 0 of the ioapic. The other messages are as per previous version.
Lastly I need sleep (4 am) so they are not yet done as patch files. I have put them into two text files and bzipped them as an attachment. Anyhow they are small and insert in the same places as their previous versions.
Quoted 5 lines
Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat the favour with these please?
Again they are still experimental.... so I am hoping Ian, Jesse and others will put them to the test.
Compiling 2.6.0 atm, since i want to test that somemore on this machine...
Quoted 5 lines · Josh McKinney
I am not Jesse but I thought I would diff them for 2.6 myself. Thanks Ross and others for putting so much time and effort into this. I haven't had time to test them, but I soon will. Uptime is:
15:19:45 up 2 days, 3:53, 7 users, load average: 0.05, 0.28, 0.37
with the original v1 patches.
apic part is missing a closing ) and the last part of the #ifdef =) So, either rediff or let the users fix that up =)
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 5 lines · Ian Kumlien
Quoted 3 lines · Ross Dickson
Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch although I do not think anyone had any problems with the first release.
<snip>
Quoted 5 lines
Right now i'm compiling and hope that the patch doesn't need more workups. (Josh McKinney patches missed some things, more on that further down)
<snip>
Quoted 6 lines
apic part is missing a closing ) and the last part of the #ifdef =) So, either rediff or let the users fix that up =)
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Oops. I was playing kernel hacker while I should have been doing other things. Patch rediffed and both attached. Thanks for catching my mess up.
Signature · 6 lines
Josh McKinney | Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
| They that can give up essential liberty
Linux, the choice -o) | to obtain a little temporary safety deserve
of the GNU generation /\ | neither liberty or safety.
_\_v | -Benjamin Franklinarch/i386/kernel/io_apic.c +48 -0
@@ -2128,6 +2128,54 @@printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); } +#ifdef CONFIG_ACPI_BOOT &&CONFIG_X86_UP_IOAPIC +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +* Note2: this violates the above comment re Subtle but works! + */ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok, saved_timer_ack = timer_ack; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + timer_ack =0 ; + +/* + + + +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER check 8259 ints disabled, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +check_nmi_watchdog (); + } +printk (KERN_INFO "..TIMER: works OK on apic pin0 irq0\n" ); +return ; + } +/* failed */ + timer_ack = saved_timer_ack; +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC Pin 0\n" ); + } +/* end new stuff for nforce2 */ +#endif +printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);
arch/i386/kernel/apic.c +31 -0
@@ -1089,6 +1089,37 @@*/ irq_stat[cpu].apic_timer_irqs++; +#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX +/* +* on 2200XP & nforce2 chipset we need 600ns? +* from timer irq start to apic irq ack to prevent +* hard lockups, use apic timer itself. +* C1 disconnect bit related. Ross Dickson. +*/ + { +static unsigned int passno, safecnt; +if (!passno) {/* calculate timing */ + safecnt =apic_read (APIC_TMICT ) - + ( (600UL *apic_read (APIC_TMICT ) ) / + (1000000000UL /HZ ) ); +printk ("..APIC TIMER ack delay, reload:%u, safe:%u\n" , +apic_read (APIC_TMICT ), safecnt); + passno++; + } +#if APIC_DEBUG +if (passno<12 ) { +unsigned int at1 =apic_read (APIC_TMCCT ); +if ( passno >1 ) +Dprintk ("..APIC TIMER ack delay, predelay count:%u \n" , at1 ); + passno++; + } +# endif +/* delay only if required */ +while (apic_read (APIC_TMCCT ) > safecnt ) +ndelay (100 ); + } +#endif +/* * NOTE! We'd better ACK the irq immediately, * because timer handling can be slow.
Quoted 1 line · Ross Dickson
..APIC TIMER ack delay, reload:16701, safe:16691
calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757
---
Survived my greptest which no non patched kernel has ever done on this machine.
Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?
(I mean, is this like the non updated gnome-terminal in mdk 9.1 that deadlocks on 2.6 if you run some ncurces apps in a "larger than usual" window?)
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 18 lines · you
Quoted 1 line · Ross Dickson
..APIC TIMER ack delay, reload:16701, safe:16691
calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757
Thanks Ian.
Quoted 1 line
From this we see your local apic is indeed counting 1.2 times faster than mine
ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time. Given 20791 is 1ms on your system then your 12 counts is 577ns
But more importantly from the ack delay theory as your machine like mine is prone to lockups then a lockup could likely have occured at count:20786 having only 240ns time expired. Next worst case was less likely to lockup at count:20780.
The only ones any delay would have been added to by the patch would be the count:20786 and count:20780 and it would have been just enough to wait until the counter got below the safe:20779 so the patch contributes little overhead.
Quoted 8 lines
---
Survived my greptest which no non patched kernel has ever done on this machine.
Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?
This may be what you have already tried:
I am not sure where it is in the 2.6 config or indeed if it is different but it is CONFIG_LOG_BUF_SHIFT under kernel hacking on 2.4.23 maybe try 16 for 64K. To match dmesg output try
dmesg -s65536
(unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)
Quoted 7 lines
(I mean, is this like the non updated gnome-terminal in mdk 9.1 that deadlocks on 2.6 if you run some ncurces apps in a "larger than usual" window?)
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 20 lines · Ross Dickson
Quoted 18 lines · you
Quoted 1 line · Ross Dickson
..APIC TIMER ack delay, reload:16701, safe:16691
calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757
Quoted 7 lines
Thanks Ian.
From this we see your local apic is indeed counting 1.2 times faster than mine ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time. Given 20791 is 1ms on your system then your 12 counts is 577ns
But more importantly from the ack delay theory as your machine like mine is prone to lockups then a lockup could likely have occured at count:20786 having only 240ns time expired. Next worst case was less likely to lockup at count:20780.
I just had a lockup running with preempt, now trying with preempt disabled. This is a clean 2.6.0-test11 with just io-apic and apic v2 patches.
Quoted 11 lines
The only ones any delay would have been added to by the patch would be the count:20786 and count:20780 and it would have been just enough to wait until the counter got below the safe:20779 so the patch contributes little overhead.
Quoted 6 lines
Survived my greptest which no non patched kernel has ever done on this machine.
Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?
Quoted 8 lines
This may be what you have already tried:
I am not sure where it is in the 2.6 config or indeed if it is different but it isCONFIG_LOG_BUF_SHIFTunder kernel hacking on 2.4.23 maybe try 16 for 64K. To match dmesg output trydmesg -s65536
(unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)
Ahhh great!, no, it doesn't auto detect it... Maybe there is a newer version, i hate mdk for being so nice to new versions and ignoring the old.
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 29 lines · you
Quoted 20 lines · Ross Dickson
Quoted 18 lines · you
Quoted 1 line · Ross Dickson
..APIC TIMER ack delay, reload:16701, safe:16691
calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757Quoted 7 lines
Thanks Ian.
From this we see your local apic is indeed counting 1.2 times faster than mine ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time. Given 20791 is 1ms on your system then your 12 counts is 577ns
But more importantly from the ack delay theory as your machine like mine is prone to lockups then a lockup could likely have occured at count:20786 having only 240ns time expired. Next worst case was less likely to lockup at count:20780.
Quoted 3 lines
I just had a lockup running with preempt, now trying with preempt disabled. This is a clean 2.6.0-test11 with just io-apic and apic v2 patches.
Thanks for testing, thats not good news.
Try rebooting with kernel args to get back in.
acpi=off noapic
My 600ns is likely too small - try putting 800UL or 1000UL in place of the 600UL
The v1 patch ((cpu_khz >> 12)+200) gave 700ns additional delay to your existing code path so I think I was being too optimistic in the initial v2 timing. I made the initial timing CPU freq dependent on the assumption that a faster cpu would get to the delay point quicker.
On my system the v1 patch had 13 counts of total delay whereas my initial the v2 gave only 10 so I think 800UL is the smallest value we should use.
If that does not fix it then it could also be that reading the timer so early may also be dangerous? or something else? and so then we would go back to v1 apic patch.
Quoted 17 lines
Quoted 17 lines
The only ones any delay would have been added to by the patch would be the count:20786 and count:20780 and it would have been just enough to wait until the counter got below the safe:20779 so the patch contributes little overhead.
Quoted 6 lines
Survived my greptest which no non patched kernel has ever done on this machine.
Has anyone got that extended ringbuffer to work? I haven't been able to get a complete "boot" dmesg in ages because of all the output all the drivers make... Does it need a updated dmesg?
This may be what you have already tried:
I am not sure where it is in the 2.6 config or indeed if it is different but it isCONFIG_LOG_BUF_SHIFTunder kernel hacking on 2.4.23 maybe try 16 for 64K. To match dmesg output try
Quoted 7 lines
Quoted 3 lines
dmesg -s65536
(unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)
Ahhh great!, no, it doesn't auto detect it... Maybe there is a newer version, i hate mdk for being so nice to new versions and ignoring the old.
My wishful thinking, I don't think any version autodetects but it would be a nice feature. Ross.
Quoted 4 lines
Signature · 2 lines
-- Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 5 lines · Ross Dickson
The v1 patch ((
cpu_khz>> 12)+200) gave 700ns additional delay to your existing code path so I think I was being too optimistic in the initial v2 timing. I made the initial timing CPU freq dependent on the assumption that a faster cpu would get to the delay point quicker.
Maybe it scales with bus speed, not CPU internal speed?
-- Jamie
Quoted 8 lines · Jamie Lokier
Quoted 5 lines · Ross Dickson
The v1 patch ((
cpu_khz>> 12)+200) gave 700ns additional delay to your existing code path so I think I was being too optimistic in the initial v2 timing. I made the initial timing CPU freq dependent on the assumption that a faster cpu would get to the delay point quicker.Maybe it scales with bus speed, not CPU internal speed?
Very good point.
Hmmm It may well be the settling time of the cpu PLL (phase lock loop) changing from low power disconnect speed up to bus speed - higher speed bus on the same type of silicon may take longer? as also it may take more power to run faster so the internal bus bit drivers may also take more time to settle? These sorts of things are mentioned in earlier model athlon errata. Especially if there is perhaps a marginal northbridge timing in the area Ian has thought about?
The nforce2 ram controller is also unique and kind of predicts and caches reads so maybe the apic timer irq code is ready to run a lot sooner than other chipsets so the normal delays wrt loading from memory are not there? Are we compensating for that? We don't really know unfortunately.
Quoted 5 lines
-- Jamie
Josh - Thanks very much for the patches.
The v2 io-apic mods seems OK so far.
I am not yet sure the v2 apic patch is going to be stable enough for everyone. I suspect it is the apic reads.
Ian had problems at 600ns so now has been trying 800ns. I have yet to hear how his machine is.
I have also gone from 600ns to 800ns and had my first hard lockup after about 6 hrs on 800ns delay timeout.
I am using a heavily patched 2.4.23 kern so it could be a coincidence but I am suspicious that it may not be that safe to be reading the apic registers at all during the delay timeout as the v2 apic patch does.
I am now going to try increasing the wait loop delay from 100ns to 400ns in case the apic does not like being hammered repetitively during the delay time. - It could be that the bus between the cpu core and the local apic is marginal on either timing (PLL) or current and if we hammer it we may be asking for incorrect reads?
I am about to recompile with the following values that differ from my original v2 posting.
( 800UL and the ndelay(400) )
#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
/*
* on AMDXP & nforce2 chipset we need about 800ns?
* from timer irq start to apic irq ack to prevent
* hard lockups, use apic timer itself.
* C1 disconnect bit related. Ross Dickson.
*/
{
static unsigned int passno, safecnt;
if(!passno) { /* calculate timing */
safecnt = apic_read(APIC_TMICT) -
( (800UL * apic_read(APIC_TMICT) ) /
(1000000000UL/HZ) );
printk("..APIC TIMER ack delay, reload:%u, safe:%u\n",
apic_read(APIC_TMICT), safecnt);
passno++;
}
#if APIC_DEBUG
if(passno<12) {
unsigned int at1 = apic_read(APIC_TMCCT);
if( passno > 1 )
Dprintk("..APIC TIMER ack delay, predelay count:%u \n", at1 );
passno++;
}
# endif
/* delay only if required */
while( apic_read(APIC_TMCCT) > safecnt )
ndelay(400);
}
#endifRegards
Ross.
On Sunday 14 December 2003 21:24, Ross Dickson wrote:
<snip>
Quoted 22 lines
The v2 io-apic mods seems OK so far.
I am not yet sure the v2 apic patch is going to be stable enough for everyone. I suspect it is the apic reads.
Ian had problems at 600ns so now has been trying 800ns. I have yet to hear how his machine is.
I have also gone from 600ns to 800ns and had my first hard lockup after about 6 hrs on 800ns delay timeout.
I am using a heavily patched 2.4.23 kern so it could be a coincidence but I am suspicious that it may not be that safe to be reading the apic registers at all during the delay timeout as the v2 apic patch does.
I am now going to try increasing the wait loop delay from 100ns to 400ns in case the apic does not like being hammered repetitively during the delay time. - It could be that the bus between the cpu core and the local apic is marginal on either timing (PLL) or current and if we hammer it we may be asking for incorrect reads?
Quoted 3 lines
I am about to recompile with the following values that differ from my original v2 posting.
( 800UL and the ndelay(400) )
<snip>
I had a lockup on a boot so I am trying a bit more conservative with
1000UL and ndelay(400)
I don't think anyone should try any less than this but hey?
This gives my system a safety margin of 16 apic counts. The v1 patch on my system typically gave a safety delay of 13 counts.
The performance hit with the v2 is still less than with the v1. With v2 additional delay would only have been present on 2 out of 10 instead of 10 out of 10 with v1.
..APIC TIMER ack delay, reload:16701, safe:16685
..APIC TIMER ack delay, predelay count:16657
..APIC TIMER ack delay, predelay count:16664
..APIC TIMER ack delay, predelay count:16677
..APIC TIMER ack delay, predelay count:16691
..APIC TIMER ack delay, predelay count:16636
..APIC TIMER ack delay, predelay count:16649
..APIC TIMER ack delay, predelay count:16660
..APIC TIMER ack delay, predelay count:16671
..APIC TIMER ack delay, predelay count:16686
..APIC TIMER ack delay, predelay count:16635
Although as previously mentioned on my system it takes 28 apic counts to write an io byte to the south bridge such as an ack in xtpic mode so v1 was not what you would call slow.
If 28 counts was a benchmark then I could still try v2 up to about 1750UL and still be quicker than the xtpic.
Ross.
Resend due to no CC...
Quoted 4 lines · Ross Dickson
I had a lockup on a boot so I am trying a bit more conservative with
1000UL and ndelay(400)
Quoted 8 lines
I don't think anyone should try any less than this but hey?
This gives my system a safety margin of 16 apic counts. The v1 patch on my system typically gave a safety delay of 13 counts.
The performance hit with the v2 is still less than with the v1. With v2 additional delay would only have been present on 2 out of 10 instead of 10 out of 10 with v1.
13h 18min here now, with 800UL 100ns
Works like a charm. It seems like we are just avoiding a timer race with some values... ie more delay might not solve the problem.
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 7 lines · Ross Dickson
Hmmm It may well be the settling time of the cpu PLL (phase lock loop) changing from low power disconnect speed up to bus speed - higher speed bus on the same type of silicon may take longer? as also it may take more power to run faster so the internal bus bit drivers may also take more time to settle? These sorts of things are mentioned in earlier model athlon errata. Especially if there is perhaps a marginal northbridge timing in the area Ian has thought about?
If you're waiting for a PLL to settle, it could vary tremendously.
Is there some way to sense the disconnect state and put in a proper delay e.g. 10 microseconds when it is sensed? I'd guess the disconnect state isn't encounted much under system load, which is the only time you really care about fast timer interrupts.
Quoted 5 lines
I am now going to try increasing the wait loop delay from 100ns to 400ns in case the apic does not like being hammered repetitively during the delay time. - It could be that the bus between the cpu core and the local apic is marginal on either timing (PLL) or current and if we hammer it we may be asking for incorrect reads?
It's possible that hammering it will consume more bus power and push a stabilising PLL and/or power supply a bit far.
If it's purely marginal timing, then doing fewer reads just means you'll hit the marginal state less often, not avoid it completely.
-- Jamie
Hi again,
Unfortunately that kernel deadlocked as well... Both times while downloading, is there some patches that i have missed for 2.6?
Please send em all if you have been successful in running it past the 'crash deadline'.
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 4 lines · Ross Dickson
It also reports its existence on boot if used. e.g.
..APIC TIMER ack delay, reload:16701, safe:16691
Quoted 9 lines
And if
#define
APIC_DEBUG0is set to 1 in
/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h
I'm trying the two new patches with 2.6 and my Award bios which fixes the crash problem itself but didn't turn on edge timer without a patch, and didn't turn on nmi_watchdog=1 with the first patch, only =2.
nmi_watchdog gets a lot more ticks with =1, too.
bob@where cat /proc/interrupts
CPU0
0: 924534 IO-APIC-edge timer 1: 1070 IO-APIC-edge i8042 2: 0 XT-PIC cascade 8: 1 IO-APIC-edge rtc 9: 0 IO-APIC-level acpi 12: 14077 IO-APIC-edge i8042 14: 10 IO-APIC-edge ide0 15: 10 IO-APIC-edge ide1 16: 44086 IO-APIC-level ide2, ide3, eth0 17: 0 IO-APIC-level yenta, yenta 19: 35570 IO-APIC-level ide4, ide5 21: 0 IO-APIC-level NVidia nForce2 NMI: 924555 LOC: 924432 ERR: 0 MIS: 22
I stayed with 600UL 100ndelay just to see if anything breaks with amd XP3000+ and patches with a bios that doesn't crash with nforce2 but needs help from patches on other points(to get edge timer on and to use nmi_watchdog=1 rather than =2). Also hope we get a clue about what Award bios update does that Phoenix does not do so far.
/usr/src/kernel-source-2.6.0/include/asm-i386/apic.h #define APIC_DEBUG 1
...but I don't see any
calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
etc
in the area of "NET: Registered" I see the following--
Dec 15 05:34:30 where kernel: PCI: Setting latency timer of device 0000:00:06.0 to 64 Dec 15 05:34:30 where kernel: intel8x0: clocking to 47414 Dec 15 05:34:30 where kernel: ALSA device list: Dec 15 05:34:30 where kernel: #0: Dummy 1 Dec 15 05:34:30 where kernel: #1: Virtual MIDI Card 1 Dec 15 05:34:30 where kernel: #2: NVidia nForce2 at 0xe5080000, irq 21 Dec 15 05:34:30 where kernel: NET: Registered protocol family 2 Dec 15 05:34:30 where kernel: IP: routing cache hash table of 8192 buckets, 64Kbytes Dec 15 05:34:30 where kernel: TCP: Hash tables configured (established 262144 bind 65536) Dec 15 05:34:30 where kernel: ip_conntrack version 2.1 (8191 buckets, 65528 max) - 296 bytes per conntrack Dec 15 05:34:30 where kernel: ip_tables: (C) 2000-2002 Netfilter core team Dec 15 05:34:30 where kernel: ipt_recent v0.3.1: Stephen Frost <[email protected]>. http://snowman.net/projects/ipt_rece nt/ Dec 15 05:34:30 where kernel: NET: Registered protocol family 1 Dec 15 05:34:30 where kernel: NET: Registered protocol family 17 Dec 15 05:34:30 where kernel: BIOS EDD facility v0.10 2003-Oct-11, 4 devices found Dec 15 05:34:30 where kernel: Please report your BIOS at http://domsch.com/linux/edd30/results.html Dec 15 05:34:30 where kernel: md: Autodetecting RAID arrays.
Dec 15 05:34:30 where kernel: NFORCE2: chipset revision 162
log starts with this acpi stuff--
Dec 15 05:34:30 where kernel: klogd 1.4.1#10, log source = /proc/kmsg started. Dec 15 05:34:30 where kernel: Inspecting /boot/System.map-2.6.0.nf2 Dec 15 05:34:30 where kernel: Loaded 33440 symbols from /boot/System.map-2.6.0.nf2. Dec 15 05:34:30 where kernel: Symbols match kernel version 2.6.0. Dec 15 05:34:30 where kernel: No module symbols loaded - kernel modules not enabled. Dec 15 05:34:30 where kernel: IRQs 3 4 *5 6 7 10 11 12 14 15) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [LFIR] (IRQs 3 4 5 6 7 10 11 12 14 15) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [L3CM] (IRQs 3 4 5 6 7 10 11 12 14 15) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [LIDE] (IRQs 3 4 5 6 7 10 11 12 14 15) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC1] (IRQs *16) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC2] (IRQs *17) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC3] (IRQs *18) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC4] (IRQs *19) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC5] (IRQs 16) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCF] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCG] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCH] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCI] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCJ] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCK] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCS] (IRQs *23) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCL] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCM] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [AP3C] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCZ] (IRQs 20 21 22) Dec 15 05:34:30 where kernel: Linux Plug and Play Support v0.97 (c) Adam Belay Dec 15 05:34:30 where kernel: pnp: the driver 'system' has been registered Dec 15 05:34:30 where kernel: PnPBIOS: Scanning system for PnP BIOS support... Dec 15 05:34:30 where kernel: PnPBIOS: Found PnP BIOS installation structure at 0xc00fc660 Dec 15 05:34:30 where kernel: PnPBIOS: PnP BIOS version 1.0, entry 0xf0000:0xc690, dseg 0xf0000 Dec 15 05:34:30 where kernel: pnp: match found with the PnP device '00:07' and the driver 'system' Dec 15 05:34:30 where kernel: pnp: match found with the PnP device '00:08' and the driver 'system' Dec 15 05:34:30 where kernel: PnPBIOS: 13 nodes reported by PnP BIOS; 13 recorded by driver Dec 15 05:34:30 where kernel: SCSI subsystem initialized Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCS] enabled at IRQ 23 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-23 -> 0xa9 -> IRQ 23 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:00:01[A] -> 2-23 -> IRQ 23 Dec 15 05:34:30 where kernel: Pin 2-23 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCF] enabled at IRQ 20 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-20 -> 0xb1 -> IRQ 20 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:00:02[A] -> 2-20 -> IRQ 20 Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCG] enabled at IRQ 22 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-22 -> 0xb9 -> IRQ 22 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:00:02[B] -> 2-22 -> IRQ 22 Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCL] enabled at IRQ 21 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-21 -> 0xc1 -> IRQ 21 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:00:02[C] -> 2-21 -> IRQ 21 Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCH] enabled at IRQ 20 Dec 15 05:34:30 where kernel: Pin 2-20 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCI] enabled at IRQ 22 Dec 15 05:34:30 where kernel: Pin 2-22 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCJ] enabled at IRQ 21 Dec 15 05:34:30 where kernel: Pin 2-21 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCK] enabled at IRQ 20 Dec 15 05:34:30 where kernel: Pin 2-20 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCM] enabled at IRQ 22 Dec 15 05:34:30 where kernel: Pin 2-22 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [AP3C] enabled at IRQ 21 Dec 15 05:34:30 where kernel: Pin 2-21 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCZ] enabled at IRQ 20 Dec 15 05:34:30 where kernel: Pin 2-20 already programmed Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC4] enabled at IRQ 19 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-19 -> 0xc9 -> IRQ 19 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:01:06[A] -> 2-19 -> IRQ 19 Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC1] enabled at IRQ 16 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-16 -> 0xd1 -> IRQ 16 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:01:06[B] -> 2-16 -> IRQ 16 Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC2] enabled at IRQ 17 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-17 -> 0xd9 -> IRQ 17 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:01:06[C] -> 2-17 -> IRQ 17 Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC3] enabled at IRQ 18 Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-18 -> 0xe1 -> IRQ 18 Mode:1 Active:0) Dec 15 05:34:30 where kernel: 00:01:06[D] -> 2-18 -> IRQ 18 Dec 15 05:34:30 where kernel: Pin 2-16 already programmed Dec 15 05:34:30 where kernel: Pin 2-17 already programmed Dec 15 05:34:30 where kernel: Pin 2-18 already programmed Dec 15 05:34:30 where kernel: Pin 2-19 already programmed Dec 15 05:34:30 where kernel: Pin 2-17 already programmed Dec 15 05:34:30 where kernel: Pin 2-18 already programmed Dec 15 05:34:30 where kernel: Pin 2-19 already programmed Dec 15 05:34:30 where kernel: Pin 2-16 already programmed Dec 15 05:34:30 where kernel: Pin 2-19 already programmed Dec 15 05:34:30 where kernel: Pin 2-16 already programmed Dec 15 05:34:30 where kernel: Pin 2-17 already programmed Dec 15 05:34:30 where kernel: Pin 2-18 already programmed Dec 15 05:34:30 where kernel: Pin 2-16 already programmed Dec 15 05:34:30 where kernel: Pin 2-17 already programmed Dec 15 05:34:30 where kernel: Pin 2-18 already programmed Dec 15 05:34:30 where kernel: Pin 2-19 already programmed Dec 15 05:34:30 where kernel: Pin 2-18 already programmed Dec 15 05:34:30 where last message repeated 3 times Dec 15 05:34:30 where kernel: Pin 2-19 already programmed Dec 15 05:34:30 where kernel: PCI: Using ACPI for IRQ routing