2.6, 2.4, Nforce2, Experimental idle halt workaround instead of apic ack delay.
PATCH 1/1 2.6, 2.4, Nforce2, Experimental idle halt workaround instead of apic ack delay.
Greetings,
Inspired by Jamie Lokier posting re halt and idle task here is the result of my attempts to move the hard lockup compensation from the "wake up from C1 disconnect" to the "going into C1 disconnect".
I surmise that between the Athlon and Nforce2 there is an issue surrounding bus drive or pll clock recovery ...etc.... on C1 disconnect and reconnect. Not sure how much of each is responsible for the problems. I have no detailed docs or test gear to dive in and get to the bottom of it. Given the lockups appear in multitude when you change the timer tick rate on a 2.4 kernel from 100Hz to 1000Hz then I think it is reasonable to assume some part does not like doing C1 disconnect cycles that frequently. Also given that a small delay put into the apic timer interrupt routine that typically wakes up the cpu from disconnect can give stability, then we appear to have a second issue of minimum cycle time for back to back C1 disconnect -reconnect cycles.
Approaching from this perspective the following patch implements a new idle thread. One which does not go into C1 disconnect (hlt) if less than 1.6% of the apic timer interval is left to execute. When you think about it, why do we disconnect if we are about to reconnect? It also has a small timing delay to help with back to back disconnect cycles ( SMI might put us into one? ). The result should be a slightly faster system (then with my apic ack delay patch) when busy but still with disconnect functioning to save power and lower heat with typical loads.
This is EXPERIMENTAL - I have only tested it for a few hours on my EPOX8RGA+ system. At this point it is kernel arg driven although in the future it could be a pci quirk. If it works well then we would ditch the apic ack delay patch, i.e use these mods of process.c in place of my earlier mods to apic.c. For testing this you do not have to back out of the apic.c changes, just drop the "apic_tack=x" kernel arg during the test boots and replace it with the following.
The KERNEL ARG to invoke it is "idle=C1halt".
Please share how/if it works..... -performance -cpu temperature -side effects? Files are also in attached tarball if whitespace problems.
Finally, this is seperate to my "io-apic edge for nmi debug" patch. You still need that patch for "nmi_debug=1". Some have reported time of day clock skew when routing the 8254 timer through the io-apic with that patch. My system has no skew problems with that patch and a modified 2.4.24 kernel.
A correct fix has since been authored by Maciej Rozycki and included with the 2.6.3-rc1-mm1 kernel tree. It does not give functioning "nmi_debug=1" with my system as it correctly attempts to route the 8254 timer to where the bios instructs, fails because the bios claims io-apic pin 2 instead of 0 and then falls back to a valid virtual wire mode. I have not yet attempted to combine my io-apic patch (io-apic pin 2 fails so try pin 0) with these later fixes - nor do I know if I should because have no nforce2 tech docs.
Jesse Allen's posting on that patch and his Nforce2
http://linux.derkeiler.com/Mailing-Lists/Kernel/2004-02/2544.html
Regards
Ross Dickson
patch for 2.6.3-rc1-mm1
---CUT HERE---
arch/i386/kernel/process.c +28 -9
@@ -50,10 +50,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> #include <asm/tlbflush.h>
@@ -92,11 +93,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();
@@ -104,10 +105,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (500 );/* helps nforce2 but adds 0.5us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )
@@ -195,20 +215,18 @@ static inline void check_cpu_quiescent(v* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle;check_cpu_quiescent (); irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule ();
@@ -260,16 +278,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---
patch for 2.4.24:
---CUT HERE---
arch/i386/kernel/process.c +27 -7
@@ -78,20 +78,38 @@ void enable_hlt(void)/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (current_cpu_data.hlt_works_ok && !hlt_counter) {__cli (); -if (!current->need_resched) +if (!current->need_resched)safe_halt ();else __sti (); } } +/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (current_cpu_data.hlt_works_ok && !hlt_counter) { +__cli (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!current->need_resched) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (500 );/* helps nforce2 but adds 0.5us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +__sti (); + } +}/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution.
@@ -121,21 +139,22 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void ); +void cpu_idle (void ) {/* endless idle loop with no priority at all */ init_idle (); current->nice =20 ; current->counter =-100 ;while (1 ) { -void (*idle)(void ) = pm_idle;if (!idle) - idle = default_idle; + idle = pm_idle ? pm_idle : default_idle;while (!current->need_resched)idle ();schedule ();check_pgt_cache (); }
@@ -143,13 +162,15 @@ void cpu_idle (void)static int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---
Quoted 10 lines · Ross Dickson
Greetings,
Approaching from this perspective the following patch implements a new idle thread. One which does not go into C1 disconnect (hlt) if less than 1.6% of the apic timer interval is left to execute. When you think about it, why do we disconnect if we are about to reconnect? It also has a small timing delay to help with back to back disconnect cycles ( SMI might put us into one? ). The result should be a slightly faster system (then with my apic ack delay patch) when busy but still with disconnect functioning to save power and lower heat with typical loads.
Is there a measurable performance loss over not having the patch at all? Some nforce2 systems work just fine. Is there a way to distinguish between systems that need it and those that don't?
(if anyone's running a betting pool, my money's on nforce2+cpu with half frequency multiplier ;)
Quoted 3 lines · Derek Foreman
Is there a measurable performance loss over not having the patch at all? Some nforce2 systems work just fine. Is there a way to distinguish between systems that need it and those that don't?
Do you have one of those systems to hand? My betting is on that when you enable APIC/IOAPIC you will see crashes very frequently. This isn't enabled in the default kernel config..
PS, Ross: Again, great work, thanks. I am running the patches you posted in the thread starter (without the previous ones) on 2.6.3-rc2-mm1 without problem.
Daniel
Re: [PATCH] 2.6, 2.4, Nforce2, Experimental idle halt workaroundinstead of apic ack delay.
Quoted 5 lines
Do you have one of those systems to hand? My betting is on that when you enable APIC/IOAPIC you will see crashes very frequently. This isn't enabled in
the default kernel config..
well... i'm runnig an Asus A7N8X with the nForce2 chipset and APIC and IO-APIC enabled. I don't have any problems at all, my machine is on 24/7. i'm running kernel 2.6.2-gentoo, but none of the paches applyed to the kernel is about APIC.
Quoted 8 lines · Daniel Drake
Quoted 3 lines · Derek Foreman
Is there a measurable performance loss over not having the patch at all? Some nforce2 systems work just fine. Is there a way to distinguish between systems that need it and those that don't?
Do you have one of those systems to hand? My betting is on that when you enable APIC/IOAPIC you will see crashes very frequently. This isn't enabled in the default kernel config..
APIC, ACPI, IOAPIC, enabled or disabled.. lots of different kernels soltek frn2 nforce2 based board, barton XP 2500+, 512mb ddr Bios doesn't allow disabling of stop grant, but I've tried both states with athcool.
It's really entering the stop grant state, because when completely idle it makes a measurable heat difference. (though when playing mp3s, almost all of the cooling benefit is lost)
This machine hasn't crashed yet.
If there's something I can do to crash it so I don't feel special anymore, or if there's some way I can provide useful information, let me know.
Quoted 2 lines
PS, Ross: Again, great work, thanks. I am running the patches you posted in the thread starter (without the previous ones) on 2.6.3-rc2-mm1 without problem.
I hope my email wasn't interpreted as an attack on the patch. I think getting linux running on flakey hardware without any support at all from the manufacturer is incredibly cool.
Quoted 3 lines · Derek Foreman
Some nforce2 systems work just fine. Is there a way to distinguish between systems that need it and those that don't?
Some nforce2 systems are fixed in certain bioses. The problem is we don't know where/what it is in the bios. C1 disconnect is a clue.
Quoted 2 lines
(if anyone's running a betting pool, my money's on nforce2+cpu with half frequency multiplier ;)
I don't know what your talking about. My Shuttle AN35N nforce2 board can run vanilla kernels with the 12-5-2003 dated bios version and not lock up. The frequencies I run are all the default/standard ones.
Jesse
Quoted 7 lines · Jesse Allen
Quoted 3 lines · Derek Foreman
Some nforce2 systems work just fine. Is there a way to distinguish between systems that need it and those that don't?
Some nforce2 systems are fixed in certain bioses. The problem is we don't know where/what it is in the bios. C1 disconnect is a clue.
So a machine that locks up with stop grant enabled under one bios revision might run just fine with stop grant enabled on another?
Quoted 6 lines
Quoted 2 lines
(if anyone's running a betting pool, my money's on nforce2+cpu with half frequency multiplier ;)
I don't know what your talking about. My Shuttle AN35N nforce2 board can run vanilla kernels with the 12-5-2003 dated bios version and not lock up. The frequencies I run are all the default/standard ones.
Some old (model 4, I think) athlons had a problem with disconnect, but only in the half multiplier versions.
Carlos' athlon has a 12.5 multiplier, so my theory's bogus
Quoted 7 lines · Derek Foreman
Quoted 5 lines · Jesse Allen
Quoted 3 lines · Derek Foreman
Some nforce2 systems work just fine. Is there a way to distinguish between systems that need it and those that don't?
Quoted 5 lines
Quoted 2 lines
Some nforce2 systems are fixed in certain bioses. The problem is we don't know where/what it is in the bios. C1 disconnect is a clue.
So a machine that locks up with stop grant enabled under one bios revision might run just fine with stop grant enabled on another?
Surely someone can decode the BIOS of an nforce user that is now ok after a BIOS update?.
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
(if anyone's running a betting pool, my money's on nforce2+cpu with half frequency multiplier ;)
I don't know what your talking about. My Shuttle AN35N nforce2 board can run vanilla kernels with the 12-5-2003 dated bios version and not lock up. The frequencies I run are all the default/standard ones.
Some old (model 4, I think) athlons had a problem with disconnect, but only in the half multiplier versions.
Carlos' athlon has a 12.5 multiplier, so my theory's bogus
Whenthis thread first(?) started way back when in Nov or Dec last year I was pretty happy.. no lockups until the 5th day. so what does "My Shuttle AN35N nforce2 board can run vanilla kernels with the 12-5-2003 dated bios version and not lock up." mean?
Ross's v3 patches for 2.6/2.61 are still running well here on 2.6.1. I'm waiting for work to settle down before trying his new ones.
Craig
Quoted 4 lines · Craig Bradney
so what does "My
Shuttle AN35N nforce2 board can run vanilla kernels with the 12-5-2003 dated bios version and not lock up." mean?
vanilla kernels = 2.6.0-test11 through 2.6.3-rc2 and no patches. APIC is on.
12-5-2003 BIOS:
http://marc.theaimsgroup.com/?l=linux-kernel&m=107124823504332&w=2
not lock up:
I could reproduce the lockup consistantly. With the 12-5-2003 bios, I cannot. Two months have passed since the original report.
Quoted 2 lines
Whenthis thread first(?) started way back when in Nov or Dec last year I was pretty happy.. no lockups until the 5th day.
The different nforce boards react differently because of different hardware an manufacterers. But they all do have a common symptom.
I don't know how to identify a fix from my bioses. If someone has any clue, I will help out.
Jesse
Quoted 8 lines · Jesse Allen
Quoted 4 lines · Craig Bradney
so what does "My
Shuttle AN35N nforce2 board can run vanilla kernels with the 12-5-2003 dated bios version and not lock up." mean?vanilla kernels = 2.6.0-test11 through 2.6.3-rc2 and no patches. APIC is on.
and local APIC and ACPI?
Quoted 5 lines
12-5-2003 BIOS:
http://marc.theaimsgroup.com/?l=linux-kernel&m=107124823504332&w=2not lock up:
I could reproduce the lockup consistantly. With the 12-5-2003 bios, I cannot. Two months have passed since the original report.
If thats the case you are a lucky person!
Quoted 9 lines
Quoted 3 lines
Whenthis thread first(?) started way back when in Nov or Dec last year I was pretty happy.. no lockups until the 5th day.
The different nforce boards react differently because of different hardware an manufacterers. But they all do have a common symptom.
I don't know how to identify a fix from my bioses. If someone has any clue, I will help out.
No idea either.. but the "uber bios"
(http://homepage.ntlworld.com/michael.mcclay/)
guy might be able to help some of us out if the changes were found... if you trust someone other than your motherboard manufacturer writing BIOSes.. but I guess thats the same as any OS project in some ways.
I notice ASUS have updated theri A7V8X BIOS to 1008.. maybe they will release updates for their nForce2 boards soon too. One can only hope!
Craig
Quoted 5 lines · Craig Bradney
Quoted 2 lines · Jesse Allen
vanilla kernels = 2.6.0-test11 through 2.6.3-rc2 and no patches. APIC is on.
and local APIC and ACPI?
Yes.
Quoted 8 lines
Quoted 6 lines
12-5-2003 BIOS:
http://marc.theaimsgroup.com/?l=linux-kernel&m=107124823504332&w=2not lock up:
I could reproduce the lockup consistantly. With the 12-5-2003 bios, I cannot. Two months have passed since the original report.If thats the case you are a lucky person!
I know =)
Quoted 9 lines
Quoted 2 lines
I don't know how to identify a fix from my bioses. If someone has any clue, I will help out.
No idea either.. but the "uber bios"
(http://homepage.ntlworld.com/michael.mcclay/)
guy might be able to help some of us out if the changes were found... if you trust someone other than your motherboard manufacturer writing BIOSes.. but I guess thats the same as any OS project in some ways.
What we need is a kernel patch, because it will be difficult to produce bios fixes for each bios iteration. If he can identify what the changes are in my bios and identify a bug, then a sufficiently talented kernel hacker can produce the work-around for other buggy bioses.
Jesse
Quoted 30 lines · Jesse Allen
Quoted 5 lines · Craig Bradney
Quoted 2 lines · Jesse Allen
vanilla kernels = 2.6.0-test11 through 2.6.3-rc2 and no patches. APIC is on.
and local APIC and ACPI?
Yes.
Quoted 8 lines
Quoted 6 lines
12-5-2003 BIOS:
http://marc.theaimsgroup.com/?l=linux-kernel&m=107124823504332&w=2not lock up:
I could reproduce the lockup consistantly. With the 12-5-2003 bios, I cannot. Two months have passed since the original report.If thats the case you are a lucky person!
I know =)
Quoted 9 lines
Quoted 2 lines
I don't know how to identify a fix from my bioses. If someone has any clue, I will help out.
No idea either.. but the "uber bios"
(http://homepage.ntlworld.com/michael.mcclay/)
guy might be able to help some of us out if the changes were found... if you trust someone other than your motherboard manufacturer writing BIOSes.. but I guess thats the same as any OS project in some ways.
Quoted 4 lines
What we need is a kernel patch, because it will be difficult to produce bios fixes for each bios iteration. If he can identify what the changes are in my bios and identify a bug, then a sufficiently talented kernel hacker can produce the work-around for other buggy bioses.
I dont know the guy myself, but maybe as you can hand him two BIOS files, he might be able to work out the difference for us if you sent him an email?
Craig
Quoted 18 lines · Jesse Allen
Quoted 4 lines · Craig Bradney
so what does "My
Shuttle AN35N nforce2 board can run vanilla kernels with the 12-5-2003 dated bios version and not lock up." mean?vanilla kernels = 2.6.0-test11 through 2.6.3-rc2 and no patches. APIC is on.
12-5-2003 BIOS:
http://marc.theaimsgroup.com/?l=linux-kernel&m=107124823504332&w=2not lock up:
I could reproduce the lockup consistantly. With the 12-5-2003 bios, I cannot. Two months have passed since the original report.
Quoted 8 lines
Quoted 2 lines
Whenthis thread first(?) started way back when in Nov or Dec last year I was pretty happy.. no lockups until the 5th day.
The different nforce boards react differently because of different hardware an manufacterers. But they all do have a common symptom.
I don't know how to identify a fix from my bioses. If someone has any clue, I will help out.
FWIW, my Biostar M7NCDPro with the latest (12-08-2003) BIOS locks up consistently unless I disable APIC (either in the kernel or the BIOS) or apply some sort of patch (like the APIC and disconnect-quirk patches that briefly made the rounds in the -mm tree).
-Roberto Sanchez
Hi,
I am just testing this patch with latest 2.6.3-rc2-mm1. It works in that sense, that my machine doesn't lock up of APIC issue. (If it locks up - hasn't done yet - then because of something else, I am currently discssing it in another thread...)
But it doesn't work in the sense of cooling my machine down. Though athcool reports disconnect is activated it behaves like it is not, ie, turning disconnect off makes no difference in temperatures. Your old tack patch in conjunction with 2.6.2-rc1 (linus) works like a charm, ie no lock-ups and less temp.
Any idea? I haven't taken out the apic_tack line, but I have added the idle=... line. Should that be a problem? I mean the apic_tack should safely be ignored, isn't it? Since I swap kernels quite often, I am too lazy to edit the boot line every time...
Prakash
Quoted 7 lines · Prakash K. Cheemplavam
Hi,
I am just testing this patch with latest 2.6.3-rc2-mm1. It works in that sense, that my machine doesn't lock up of APIC issue. (If it locks up - hasn't done yet - then because of something else, I am currently discssing it in another thread...)
Quoted 6 lines
But it doesn't work in the sense of cooling my machine down. Though athcool reports disconnect is activated it behaves like it is not, ie, turning disconnect off makes no difference in temperatures. Your old tack patch in conjunction with 2.6.2-rc1 (linus) works like a charm, ie no lock-ups and less temp.
Thanks Prakash for testing it and spotting thermal problem.
Here are some temperatures from my machine read from the bios on reboot. I gave it minimal activity for the minutes prior to reboot.
Win98, 47C
XPHome, 42C
Patched Linux 2.4.24 (1000Hz), 40C
Patched Linux 2.6.3-rc1-mm1, 53C OUCH!
Sorry, I will have to go through my latest patch and see why the temp differs so much between 2.4 and 2.6. I currently use patched 2.4.24 with Suse 8.2 for convenience. When it stopped the lockups on 2.6 I thought the 2.6 was working the same way.
Daniel - I think you patched your 2.6 too. I assume it is also hot?
Quoted 4 lines
Any idea? I haven't taken out the
apic_tackline, but I have added the idle=... line. Should that be a problem? I mean theapic_tackshould safely be ignored, isn't it? Since I swap kernels quite often, I am too lazy to edit the boot line every time...
Correct, apic_tack will be ignored if the apic.c is not patched with my apic ack delay patch.
Ross.
Quoted 5 lines
Prakash
Nforce2, APIC, CPU Disconnect and setup_boot_APIC_clock()
Hello all,
I once noticed there was a drift between the 8254 timer and the APIC timer. With vanilla and -mm kernels, APIC timer is running faster. With Ross' patches I noticed it was running slower, and the crashes go away. I dig in that direction and found that with a APIC timer directly programmed slower than calibrated, adding ~20 more bus cycles to the counter, hangs disappear, at least during the tests (for a few minutes, maybe they are made much rarer). I eventually disabled altogether the APIC timer with the patchlet attached, and did an entire test (dumping the whole hard drive to /dev/null with very high network and soundcard activity), and it survived. I will continue to test, just to make sure...
At least there is a way to get nforce2 + APIC + CPU disconnect and actually have a cooler idle CPU. Side effects are no more LOC rising counter in /proc/interrupts, no more nmi_watchdog=2 and if you work around the bios acpi apic source override to get the timer on pin #0, no more nmi_watchdog=1. This is of course not the best solution.
Now the experts may look why commenting out setup_boot_APIC_clock() in APIC_init_uniprocessor() of arch/i386/kernel/apic.c works, and find a better fix if any.
Mathieu
i386/kernel/apic.c renamed from i386/kernel/apic.c.old +1 -1
@@ -1198,7 +1198,7 @@if (!skip_ioapic_setup && nr_ioapics)setup_IO_APIC ();#endif -setup_boot_APIC_clock (); +/*setup_boot_APIC_clock();*/ return 0 ; }
Quoted 25 lines · Ross Dickson
Quoted 7 lines · Prakash K. Cheemplavam
Hi,
I am just testing this patch with latest 2.6.3-rc2-mm1. It works in that sense, that my machine doesn't lock up of APIC issue. (If it locks up - hasn't done yet - then because of something else, I am currently discssing it in another thread...)
Quoted 6 lines
But it doesn't work in the sense of cooling my machine down. Though athcool reports disconnect is activated it behaves like it is not, ie, turning disconnect off makes no difference in temperatures. Your old tack patch in conjunction with 2.6.2-rc1 (linus) works like a charm, ie no lock-ups and less temp.
Thanks Prakash for testing it and spotting thermal problem.
Here are some temperatures from my machine read from the bios on reboot. I gave it minimal activity for the minutes prior to reboot.
Win98, 47C
XPHome, 42C
Patched Linux 2.4.24 (1000Hz), 40C
Patched Linux 2.6.3-rc1-mm1, 53C OUCH!
Quoted 5 lines
Sorry, I will have to go through my latest patch and see why the temp differs so much between 2.4 and 2.6. I currently use patched 2.4.24 with Suse 8.2 for convenience. When it stopped the lockups on 2.6 I thought the 2.6 was working the same way.
Found the problem for 2.6
After fixing it the 2.6 temperature is
Patched Linux 2.6.3-rc1-mm1, 38C
Ambient today is 1C cooler also.
The fix is to put the brackets back on "!need_resched()" so that we call the function and test its return value - not just test the function pointer!
Should be as follows:
static void c1halt_idle(void)
{
if (!hlt_counter && current_cpu_data.hlt_works_ok) {
local_irq_disable();
/* only hlt disconnect if more than 1.6% of apic interval remains */
if( (!need_resched() ) &&
(apic_read(APIC_TMCCT) > (apic_read(APIC_TMICT)>>6))) {
ndelay(500); /* helps nforce2 but adds 0.5us hard int latency */
safe_halt(); /* nothing better to do until we wake up */
}
else
local_irq_enable();
}
}Again attached in tarball, 2.4 patch was not changed.
Revision1 patch for 2.6.3-rc1-mm1:
---CUTHERE---
arch/i386/kernel/process.c +28 -9
@@ -50,10 +50,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> #include <asm/tlbflush.h>
@@ -92,11 +93,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();
@@ -104,10 +105,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (500 );/* helps nforce2 but adds 0.5us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )
@@ -195,20 +215,18 @@ static inline void check_cpu_quiescent(v* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle;check_cpu_quiescent (); irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule ();
@@ -260,16 +278,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUTHERE---
arch/i386/kernel/process.c +28 -8
@@ -50,10 +50,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> #include <asm/tlbflush.h>
@@ -92,11 +93,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();
@@ -104,10 +105,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (500 );/* helps nforce2 but adds 0.5us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )
@@ -195,20 +215,18 @@ static inline void check_cpu_quiescent(v* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle;check_cpu_quiescent (); irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule ();
@@ -260,16 +278,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup);
GCC feature request: warn on "if (function_name)"
Quoted 2 lines · Ross Dickson
The fix is to put the brackets back on "!need_resched()" so that we call the function and test its return value - not just test the function pointer!
[ Ross' bug was writing "if (!need_resched)" instead of "if (!need_resched())" ]
I'm very surprised GCC doesn't warn about that. A quick test confirms GCC 3.2.2 at least doesn't.
So, this is a feature request:
- Warn when a function name is tested in a boolean context. (A function pointer variable or expression should not be warned for).
By boolean context I mean any place where a function name is used as a value and tested against zero. Some examples:
if (function_name)
if (function_name && ( <some other expression> )) if (function_name != 0)
if (function_name == 0)
if (!function_name)
x = function_name ? a : b;
- Don't warn if there are two levels of parantheses.
I know it's occasionally useful to test the NULL-ness of a functin name, of weak symbols. In most cases, though, it's a bug. If you really want to check a weak symbol, just write "if ((symbol))". That syntax is already well known for testing the result of an assignment, as in "if ((x = 1))" does not yield a warning but "if (x = 1)" does.
Perhaps a later GCC than 3.2.2 already has this test; if someone is able to check, that would be nice.
Thanks muchly :)
-- Jamie
Re: GCC feature request: warn on "if (function_name)"
Quoted 3 lines · Jamie Lokier
Perhaps a later GCC than 3.2.2 already has this test; if someone is able to check, that would be nice.
Yes, 3.4.0 does have this check.
t.c:5: warning: the address of i', will always evaluate as true'
Thanks,
Andrew Pinski
Quoted 20 lines
Quoted 13 lines
Quoted 2 lines
But it doesn't work in the sense of cooling my machine down. Though athcool reports disconnect is activated it behaves like it is not, ie, turning disconnect off makes no difference in temperatures. Your old tack patch in conjunction with 2.6.2-rc1 (linus) works like a charm, ie no lock-ups and less temp.
Thanks Prakash for testing it and spotting thermal problem.
Here are some temperatures from my machine read from the bios on reboot. I gave it minimal activity for the minutes prior to reboot.
Win98, 47C
XPHome, 42C
Patched Linux 2.4.24 (1000Hz), 40C
Patched Linux 2.6.3-rc1-mm1, 53C OUCH!Quoted 5 lines
Sorry, I will have to go through my latest patch and see why the temp differs so much between 2.4 and 2.6. I currently use patched 2.4.24 with Suse 8.2 for convenience. When it stopped the lockups on 2.6 I thought the 2.6 was working the same way.
Quoted 5 lines
Found the problem for 2.6
After fixing it the 2.6 temperature is
Patched Linux 2.6.3-rc1-mm1, 38C
Ambient today is 1C cooler also.
Yes, I am just trying your new patch, and it works! Furthermore it seems to have less ipact on system performance than the tack one, as now hdparm reports the same figures as without using APIC. Well done!
Have you read the post from Mathieu about his finding of APIC and 8254 timer not being sync, which causes lock-ups? Maybe there should be the correct way of fixing it. Furthermore I saw this in latest ACPI update:
[ACPI] nforce2 timer lockup from Maciej W. Rozycki
Is this the fix or something else?
Cheers,
Prakash
Quoted 6 lines
Quoted 5 lines
Found the problem for 2.6
After fixing it the 2.6 temperature is
Patched Linux 2.6.3-rc1-mm1, 38C
Ambient today is 1C cooler also.
Quoted 6 lines
Yes, I am just trying your new patch, and it works! Furthermore it seems to have less ipact on system performance than the tack one, as now hdparm reports the same figures as without using APIC. Well done!
Have you read the post from Mathieu about his finding of APIC and 8254 timer not being sync, which causes lock-ups?
I have seen the posting from Mathieu. It is very interesting work. With an embedded system you would not waste the resources to setup and run two timers at almost the same 1ms interval as is done in i386 linux unless you had very good reason. I think for the PC it has to do with the 8254 historically having a more accurate oscillator source?
I am thinking that maybe nforce2 is locking up when the 8254 interrupt and the apic interrupt trigger almost co-incident with each other for a while - yielding back to back cycles of coming out of C1 disconnect and then being thrown back into it by the idle thread. This could cause all sorts of problems within the chips such as hot tristate drivers or power rail fluctuations if the capacitors and regulators cannot compensate for such a frequency. An analogy is the low frequency beat while tuning a musical instrument. Also if the oscillators are from different crystals then the beat frequency will also vary with temperature.
By changing the timing differential between the apic timer and the 8254 timer Mathieu forces the interrupts come into phase with each other for a briefer period of time so then not so many back to back C1 cycles occur? And eliminating the apic timer removes this source of interrupt co-incidence.
Maybe the Athlon nforce2 system can withstand the C1 disconnect dance of the two timers but then throwing in the repetitive block read interrupts of an IDE hard drive triggers failure. Twos company - three's a crowd...??? I am just guessing.
Quoted 5 lines
Maybe there should be the
correct way of fixing it. Furthermore I saw this in latest ACPI update:
[ACPI] nforce2 timer lockup from Maciej W. RozyckiIs this the fix or something else?
Partly, its related to my io-apic edge patch regarding timer ints. Maciej & I corresponded about the ioapic & 8254 timers in some detail in December. Jesse spotted Maciej's fix called nforce-irq-setup-fix.patch first appearing in 2.6.3-rc1-mm1 and is referred to in this thread starter.
It gets another mention in 2.6.3-rc2-mm1 changelog as now being included in the ACPI patch so I am pretty sure that is it. "-nforce-irq-setup-fix.patch"
" This is in the ACPI patch"
Regards
Ross.
Quoted 4 lines
Cheers,
Prakash
Quoted 9 lines · Ross Dickson
On Saturday 14 February 2004 00:41, Ross Dickson wrote:
Found the problem for 2.6After fixing it the 2.6 temperature is
Patched Linux 2.6.3-rc1-mm1, 38C
Ambient today is 1C cooler also.The fix is to put the brackets back on "!need_resched()" so that we call the function and test its return value - not just test the function pointer!
Seems to work just great here... I also did some testing before i applied it.
I tested Uberbios for my mb, but it didn't help at all, actually it seemed quite odd... (Strange values etc)
Now i'm back to normal bios and running Ross' quality work again! =)
Keep up the good work guys, seems like the trade off is getting smaller and smaller =). Btw, uptime before i decided that i should update kernel was 17 days (haven't had any nforce in quite some time when running ross' quality work, so the uptime more reflects my eagerness to update kernels and stuff)
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 28 lines · Ross Dickson
Quoted 19 lines · Ross Dickson
Quoted 6 lines · Prakash K. Cheemplavam
Hi,
I am just testing this patch with latest 2.6.3-rc2-mm1. It works in that sense, that my machine doesn't lock up of APIC issue. (If it locks up - hasn't done yet - then because of something else, I am currently discssing it in another thread...)
But it doesn't work in the sense of cooling my machine down. Though athcool reports disconnect is activated it behaves like it is not, ie, turning disconnect off makes no difference in temperatures. Your old tack patch in conjunction with 2.6.2-rc1 (linus) works like a charm, ie no lock-ups and less temp.
Thanks Prakash for testing it and spotting thermal problem.
Here are some temperatures from my machine read from the bios on reboot. I gave it minimal activity for the minutes prior to reboot.
Win98, 47C
XPHome, 42C
Patched Linux 2.4.24 (1000Hz), 40C
Patched Linux 2.6.3-rc1-mm1, 53C OUCH!Quoted 5 lines
Sorry, I will have to go through my latest patch and see why the temp differs so much between 2.4 and 2.6. I currently use patched 2.4.24 with Suse 8.2 for convenience. When it stopped the lockups on 2.6 I thought the 2.6 was working the same way.
Quoted 5 lines
Found the problem for 2.6
After fixing it the 2.6 temperature is
Patched Linux 2.6.3-rc1-mm1, 38C
Ambient today is 1C cooler also.
Well, I hate to say it, but it seems, it doesn't work, or at least not so well, (running hot, but stability seems to be there) with 2.6.3-mm2. Like I had 52°C mostly idle with your patch and APIC just a few moments ago. Now back to PIC within a few minutes I am back to 45°C...7°C is too much of a difference for me.
To be honest, I can't remenber how it was with the older kernel and I haven't tested temp of apic_tack patch with 2.6.3-mm2, but for the moment I am back to PIC. (My CPU is running at 11x200MHz, 1,7vcore btw.)
Cheers,
Prakash
Quoted 14 lines · Prakash K. Cheemplavam
Quoted 8 lines
Quoted 7 lines
Here are some temperatures from my machine read from the bios on reboot. I gave it minimal activity for the minutes prior to reboot.
Win98, 47C
XPHome, 42C
Patched Linux 2.4.24 (1000Hz), 40C
Patched Linux 2.6.3-rc1-mm1, 53C OUCH!Quoted 5 lines
Found the problem for 2.6
After fixing it the 2.6 temperature is
Patched Linux 2.6.3-rc1-mm1, 38C
Ambient today is 1C cooler also.
Quoted 6 lines
Well, I hate to say it, but it seems, it doesn't work, or at least not so well, (running hot, but stability seems to be there) with 2.6.3-mm2. Like I had 52?C mostly idle with your patch and APIC just a few moments ago. Now back to PIC within a few minutes I am back to 45?C...7?C is too much of a difference for me.
While on the subject of tempertures, I found something a bit weird. In linux with C1 disconnects enabled, the system temperature was 36 C. I rebooted to BIOS setup and watched the temperatures there. For some weird reason, the system temperature rose from 36 C, to about 41 C. And I also watched the CPU temp rise from about 41 C to about 51 C ! I boot back into linux, and I watched the system temperature _drop_ to 36 C again. What is C1 disconnects disabled during POST now on my BIOS? I have done this before when I first got it and never noticed something like this.
Jesse
Oh before I forget, I had to resolve a reject by hand, but I *think* I did it right. (And yes, I used your corrected versionof the patch.) Well, maybe you take a look over the patch and rediff. :-)
Prakash
Quoted 8 lines · Prakash K. Cheemplavam
Oh before I forget, I had to resolve a reject by hand, but I *think* I did it right. (And yes, I used your corrected versionof the patch.) Well, maybe you take a look over the patch and rediff. :-)
Prakash
Hi Prakash,
Patches attached rediffed for 2.6.3 and 2.6.3-mm3.
Late comers see start of this thread for more detail.
Also Arjen's page at http://atlas.et.tudelft.nl/verwei90/nforce2/index.html
Note that the ioapic patch is not essential for 2.6.3-mm3, only if you want to use nmi_debug=1.
2.6.3-mm3 sets up 8254 timer OK in a virtual wire mode without this patch.
It will be interesting to see if there is any clock skew associated with these patches, I get no skew on 2.4.24.
The kernel arg to invoke the idle halt workaround is still "idle=C1halt". I have increased slightly the ndelay time from 500ns to 600ns - I had some lockups on startup and shutdown with certain removable IDE drives, this seemed to help. Feel free to adjust the value if you want to.
Also if you want snappier performance you can try different values instead of 6 in this line:
+ (apic_read(APIC_TMCCT) > (apic_read(APIC_TMICT)>>6))) {
The 6 yields a divide by 2^6 = 64 for the 1.6% (1/64 = 1.6%)
If you try say 5 for 2^5 = 32 for 3.1% or 4 for 6% or 3 for 12.5% then if less than this percentage of time remains in the slice the system won't go into hlt disconnect thus avoiding the reconnect - recovery time associated with that cycle that would have been.
I have found a need for less ndelay time with the more the percentage (lower number)- there appears to be a tradeoff between the two.
3 for 12.5% or 2 for 25% might even work with no ndelay time i.e. the ndelay line removed or commented from the code, for some.
Of course expect a bit more heat under moderate loads with an increase in the percentage but some may prefer it for their application.
Happy hacking,
Ross.
For 2.6.3 ioapic
---CUT HERE---
arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +45 -1
@@ -2188,12 +2188,56 @@ static inline void check_timer(void)check_nmi_watchdog (); }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +* Note2: this violates the above comment re Subtle but works! +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok, saved_timer_ack = timer_ack; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + timer_ack =0 ; + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +check_nmi_watchdog (); + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ + timer_ack = saved_timer_ack; +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/*
---CUT HERE---
For 2.6.3 idleC1halt
---CUT HERE---
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28 -8
@@ -48,10 +48,11 @@#include <asm/irq.h> #include <asm/desc.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");
@@ -87,11 +88,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();
@@ -99,10 +100,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )
@@ -136,20 +156,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }
@@ -200,16 +218,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup);
---CUT HERE---
For 2.6.3-mm3 ioapic
---CUT HERE---
arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +43 -1
55 lines not on this page · raw message
---CUT HERE---
For 2.6.3-mm3 idleC1halt
---CUT HERE---
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28 -8
@@ -49,10 +49,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");
@@ -88,11 +89,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();
@@ -100,10 +101,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )
@@ -137,20 +157,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }
@@ -201,16 +219,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup);
---CUT HERE---
Attached patches also in tarballs if whitespace problems.
Quoted 4 lines · Ross Dickson
Quoted 2 lines · Prakash K. Cheemplavam
Oh before I forget, I had to resolve a reject by hand, but I *think* I did it right. (And yes, I used your corrected versionof the patch.) Well, maybe you take a look over the patch and rediff. :-)
Quoted 8 lines
Quoted 4 lines
Prakash
Hi Prakash,
Patches attached rediffed for 2.6.3 and 2.6.3-mm3.
Hi Ross,
so I am running this patch now since you released it (5:21h) and so far stable. Temp is better, bt not as goos as pic mode (about 3-4°C higher). Nevertheless, nice work!
Thanks,
Prakash
Lately I am trying to find out why my CPU runs so much hotter in W2K (50ish Celsius) than it does with Linux (36 Celsius) when idle. I am wondering if Microsoft secretly disables the disconnect :)
On the linux front I am playing with 2.6.3-mm patched aditionally with nforce2-timer and apic-tack. There is no need for Maciej's patch, because Andrew was nice enough to include this in -mm.
Personally, I would like to figure out a way of keeping the C1 disconnect, because I like a silent system and there isn't much cooling fannoise required when my cpu is at 36 deg. than when it is at 50+ idle.
I gather that not everyone has these big differences, so it may not be interesting to relentlessly pursue C1 like I am :P
Something else that strikes me as odd, is that 2.6.3-mm like I built it, immediately locks up solid if you omit apic_tack=2, but 2.4 with acpi and apic enabled, without Ross' patches would not react this vigorously.
Regards,
Arjen
Quoted 393 lines · Ross Dickson
Quoted 8 lines · Prakash K. Cheemplavam
Oh before I forget, I had to resolve a reject by hand, but I *think* I did it right. (And yes, I used your corrected versionof the patch.) Well, maybe you take a look over the patch and rediff. :-)
Prakash
Hi Prakash,
Patches attached rediffed for 2.6.3 and 2.6.3-mm3.Late comers see start of this thread for more detail.
Also Arjen's page at http://atlas.et.tudelft.nl/verwei90/nforce2/index.htmlNote that the ioapic patch is not essential for 2.6.3-mm3, only if you want to use nmi_debug=1.
2.6.3-mm3 sets up 8254 timer OK in a virtual wire mode without this patch.It will be interesting to see if there is any clock skew associated with these patches, I get no skew on 2.4.24.
The kernel arg to invoke the idle halt workaround is still "idle=C1halt". I have increased slightly the ndelay time from 500ns to 600ns - I had some lockups on startup and shutdown with certain removable IDE drives, this seemed to help. Feel free to adjust the value if you want to.
Also if you want snappier performance you can try different values instead of 6 in this line:
(quoted diff) +1
-0+ (apic_read(APIC_TMCCT) > (apic_read(APIC_TMICT)>>6))) {The 6 yields a divide by 2^6 = 64 for the 1.6% (1/64 = 1.6%)
If you try say 5 for 2^5 = 32 for 3.1% or 4 for 6% or 3 for 12.5% then if less than this percentage of time remains in the slice the system won't go into hlt disconnect thus avoiding the reconnect - recovery time associated with that cycle that would have been.
I have found a need for less ndelay time with the more the percentage (lower number)- there appears to be a tradeoff between the two.
3 for 12.5% or 2 for 25% might even work with no ndelay time i.e. the ndelay line removed or commented from the code, for some.
Of course expect a bit more heat under moderate loads with an increase in the percentage but some may prefer it for their application.
Happy hacking,
Ross.For 2.6.3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +45
-2@@ -2188,12 +2188,56 @@ static inline void check_timer(void)check_nmi_watchdog (); }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +* Note2: this violates the above comment re Subtle but works! +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok, saved_timer_ack = timer_ack; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + timer_ack =0 ; + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +check_nmi_watchdog (); + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ + timer_ack = saved_timer_ack; +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-9@@ -48,10 +48,11 @@#include <asm/irq.h> #include <asm/desc.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -87,11 +88,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -99,10 +100,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -136,20 +156,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -200,16 +218,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---For 2.6.3-mm3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +43
-2@@ -2206,12 +2206,54 @@ static inline void check_timer(void) timer_ack = !cpu_has_tsc; }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +if (check_nmi_watchdog () <0 ); + timer_ack = !cpu_has_tsc; + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3-mm3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-9@@ -49,10 +49,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -88,11 +89,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -100,10 +101,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -137,20 +157,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -201,16 +219,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---Attached patches also in tarballs if whitespace problems.
Quoted 3 lines · Arjen Verweij
Lately I am trying to find out why my CPU runs so much hotter in W2K (50ish Celsius) than it does with Linux (36 Celsius) when idle. I am wondering if Microsoft secretly disables the disconnect :)
Sounds like it, or maybe your bios power management function is called and it busy idles instead of disconnect.
Jesse's posting talks of temperature rise in bios setup- could be similar.
http://linux.derkeiler.com/Mailing-Lists/Kernel/2004-02/5943.html
Quoted 4 lines
On the linux front I am playing with 2.6.3-mm patched aditionally with nforce2-timer and apic-tack. There is no need for Maciej's patch, because Andrew was nice enough to include this in -mm.
Yes Maciej's patch fixed up routing of the timer interrupts in bios instructed way. Works well but timer ends up routed into local apic in Virtual wire mode which is incompatible with "nmi_debug=1". Alone it does not prevent nforce2 lockups. Likewise my ioapic patch alone cannot prevent lockups.
Quoted 4 lines
Personally, I would like to figure out a way of keeping the C1 disconnect, because I like a silent system and there isn't much cooling fannoise required when my cpu is at 36 deg. than when it is at 50+ idle.
This patch (idleC1halt) keeps the C1 disconnect and is now my preferred replacement for the apic ack delay patch. They both achieve the same goal of keeping C1 disconnect functioning on buggy nforce2 systems.
Quoted 3 lines
I gather that not everyone has these big differences, so it may not be interesting to relentlessly pursue C1 like I am :P
Already persued in my patches, but I would prefer a better way such as a bios fix in all mobo brands -if that is enough. Power management stuff should be invisible to OS.
Quoted 4 lines
Something else that strikes me as odd, is that 2.6.3-mm like I built it, immediately locks up solid if you omit apic_tack=2, but 2.4 with acpi and apic enabled, without Ross' patches would not react this vigorously.
I have found it to be the 1000Hz timer ticks by default in 2.6 whereas in 2.4 you only have 100Hz ticks. By default you can have 10 times the quantity of disconnects and reconnects per second with 2.6 as compared with 2.4. The -CPU? -Chipset? -memory? -power supply? -track layout? -bios timings? etc. combo the way it is set up on a lot of Nforce2 boards cannot hack it.
Quoted 406 lines
Regards,
Arjen
Quoted 393 lines · Ross Dickson
Quoted 8 lines · Prakash K. Cheemplavam
Oh before I forget, I had to resolve a reject by hand, but I *think* I did it right. (And yes, I used your corrected versionof the patch.) Well, maybe you take a look over the patch and rediff. :-)
Prakash
Hi Prakash,
Patches attached rediffed for 2.6.3 and 2.6.3-mm3.Late comers see start of this thread for more detail.
Also Arjen's page at http://atlas.et.tudelft.nl/verwei90/nforce2/index.htmlNote that the ioapic patch is not essential for 2.6.3-mm3, only if you want to use nmi_debug=1.
2.6.3-mm3 sets up 8254 timer OK in a virtual wire mode without this patch.It will be interesting to see if there is any clock skew associated with these patches, I get no skew on 2.4.24.
The kernel arg to invoke the idle halt workaround is still "idle=C1halt". I have increased slightly the ndelay time from 500ns to 600ns - I had some lockups on startup and shutdown with certain removable IDE drives, this seemed to help. Feel free to adjust the value if you want to.
Also if you want snappier performance you can try different values instead of 6 in this line:
(quoted diff) +1
-0+ (apic_read(APIC_TMCCT) > (apic_read(APIC_TMICT)>>6))) {The 6 yields a divide by 2^6 = 64 for the 1.6% (1/64 = 1.6%)
If you try say 5 for 2^5 = 32 for 3.1% or 4 for 6% or 3 for 12.5% then if less than this percentage of time remains in the slice the system won't go into hlt disconnect thus avoiding the reconnect - recovery time associated with that cycle that would have been.
I have found a need for less ndelay time with the more the percentage (lower number)- there appears to be a tradeoff between the two.
3 for 12.5% or 2 for 25% might even work with no ndelay time i.e. the ndelay line removed or commented from the code, for some.
Of course expect a bit more heat under moderate loads with an increase in the percentage but some may prefer it for their application.
Happy hacking,
Ross.For 2.6.3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +45
-2@@ -2188,12 +2188,56 @@ static inline void check_timer(void)check_nmi_watchdog (); }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +* Note2: this violates the above comment re Subtle but works! +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok, saved_timer_ack = timer_ack; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + timer_ack =0 ; + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +check_nmi_watchdog (); + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ + timer_ack = saved_timer_ack; +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-9@@ -48,10 +48,11 @@#include <asm/irq.h> #include <asm/desc.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -87,11 +88,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -99,10 +100,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -136,20 +156,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -200,16 +218,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---For 2.6.3-mm3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +43
-2@@ -2206,12 +2206,54 @@ static inline void check_timer(void) timer_ack = !cpu_has_tsc; }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +if (check_nmi_watchdog () <0 ); + timer_ack = !cpu_has_tsc; + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3-mm3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-9@@ -49,10 +49,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -88,11 +89,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -100,10 +101,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -137,20 +157,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -201,16 +219,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---Attached patches also in tarballs if whitespace problems.
Quoted 9 lines · Ross Dickson
Quoted 3 lines
Something else that strikes me as odd, is that 2.6.3-mm like I built it, immediately locks up solid if you omit apic_tack=2, but 2.4 with acpi and apic enabled, without Ross' patches would not react this vigorously.
I have found it to be the 1000Hz timer ticks by default in 2.6 whereas in 2.4 you only have 100Hz ticks. By default you can have 10 times the quantity of disconnects and reconnects per second with 2.6 as compared with 2.4. The -CPU? -Chipset? -memory? -power supply? -track layout? -bios timings? etc. combo the way it is set up on a lot of Nforce2 boards cannot hack it.
One option is to reduce HZ, by e.g. applying
http://www.csd.uu.se/~mikpe/linux/patches/2.6/patch-config-hz-2.6.3
and selecting CONFIG_HZ_100.
I did this originally to prevent clock slowdown on a 486 during heavy disk I/O, but at least one person reported that it helped his PIII as well: that could very well have been a chipset or #SMM BIOS issue.
Hi Ross,
Finally got around to getting 2.6.3 up and running.. and your latest patches installed. Nothing adverse to report so far. 43 mins uptime with basically not very much happening in that time. Temperature on the mobo and cpu seem about the same, possibly 1 degree higher. 2.6.3 seems smoother. I didnt modify your patches at all, left the 6 in there as default.
Still no BIOS update from ASUS.. grumble grumble.
thanks for your work! (and to the rest of course!)
Craig
Quoted 461 lines · Ross Dickson
Quoted 3 lines · Arjen Verweij
Lately I am trying to find out why my CPU runs so much hotter in W2K (50ish Celsius) than it does with Linux (36 Celsius) when idle. I am wondering if Microsoft secretly disables the disconnect :)
Sounds like it, or maybe your bios power management function is called and it busy idles instead of disconnect.
Jesse's posting talks of temperature rise in bios setup- could be similar.
http://linux.derkeiler.com/Mailing-Lists/Kernel/2004-02/5943.htmlQuoted 4 lines
On the linux front I am playing with 2.6.3-mm patched aditionally with nforce2-timer and apic-tack. There is no need for Maciej's patch, because Andrew was nice enough to include this in -mm.
Yes Maciej's patch fixed up routing of the timer interrupts in bios instructed way. Works well but timer ends up routed into local apic in Virtual wire mode which is incompatible with "nmi_debug=1". Alone it does not prevent nforce2 lockups. Likewise my ioapic patch alone cannot prevent lockups.
Quoted 4 lines
Personally, I would like to figure out a way of keeping the C1 disconnect, because I like a silent system and there isn't much cooling fannoise required when my cpu is at 36 deg. than when it is at 50+ idle.
This patch (idleC1halt) keeps the C1 disconnect and is now my preferred replacement for the apic ack delay patch. They both achieve the same goal of keeping C1 disconnect functioning on buggy nforce2 systems.
Quoted 3 lines
I gather that not everyone has these big differences, so it may not be interesting to relentlessly pursue C1 like I am :P
Already persued in my patches, but I would prefer a better way such as a bios fix in all mobo brands -if that is enough. Power management stuff should be invisible to OS.
Quoted 4 lines
Something else that strikes me as odd, is that 2.6.3-mm like I built it, immediately locks up solid if you omit apic_tack=2, but 2.4 with acpi and apic enabled, without Ross' patches would not react this vigorously.
I have found it to be the 1000Hz timer ticks by default in 2.6 whereas in 2.4 you only have 100Hz ticks. By default you can have 10 times the quantity of disconnects and reconnects per second with 2.6 as compared with 2.4. The -CPU? -Chipset? -memory? -power supply? -track layout? -bios timings? etc. combo the way it is set up on a lot of Nforce2 boards cannot hack it.
Quoted 396 lines
Regards,
Arjen
Quoted 387 lines · Ross Dickson
Quoted 8 lines · Prakash K. Cheemplavam
Oh before I forget, I had to resolve a reject by hand, but I *think* I did it right. (And yes, I used your corrected versionof the patch.) Well, maybe you take a look over the patch and rediff. :-)
Prakash
Hi Prakash,
Patches attached rediffed for 2.6.3 and 2.6.3-mm3.Late comers see start of this thread for more detail.
Also Arjen's page at http://atlas.et.tudelft.nl/verwei90/nforce2/index.htmlNote that the ioapic patch is not essential for 2.6.3-mm3, only if you want to use nmi_debug=1.
2.6.3-mm3 sets up 8254 timer OK in a virtual wire mode without this patch.It will be interesting to see if there is any clock skew associated with these patches, I get no skew on 2.4.24.
The kernel arg to invoke the idle halt workaround is still "idle=C1halt". I have increased slightly the ndelay time from 500ns to 600ns - I had some lockups on startup and shutdown with certain removable IDE drives, this seemed to help. Feel free to adjust the value if you want to.
Also if you want snappier performance you can try different values instead of 6 in this line:
(quoted diff) +1
-0+ (apic_read(APIC_TMCCT) > (apic_read(APIC_TMICT)>>6))) {The 6 yields a divide by 2^6 = 64 for the 1.6% (1/64 = 1.6%)
If you try say 5 for 2^5 = 32 for 3.1% or 4 for 6% or 3 for 12.5% then if less than this percentage of time remains in the slice the system won't go into hlt disconnect thus avoiding the reconnect - recovery time associated with that cycle that would have been.
I have found a need for less ndelay time with the more the percentage (lower number)- there appears to be a tradeoff between the two.
3 for 12.5% or 2 for 25% might even work with no ndelay time i.e. the ndelay line removed or commented from the code, for some.
Of course expect a bit more heat under moderate loads with an increase in the percentage but some may prefer it for their application.
Happy hacking,
Ross.For 2.6.3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +45
-2@@ -2188,12 +2188,56 @@ static inline void check_timer(void)check_nmi_watchdog (); }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +* Note2: this violates the above comment re Subtle but works! +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok, saved_timer_ack = timer_ack; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + timer_ack =0 ; + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +check_nmi_watchdog (); + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ + timer_ack = saved_timer_ack; +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-9@@ -48,10 +48,11 @@#include <asm/irq.h> #include <asm/desc.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -87,11 +88,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -99,10 +100,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -136,20 +156,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -200,16 +218,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---For 2.6.3-mm3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +43
-2@@ -2206,12 +2206,54 @@ static inline void check_timer(void) timer_ack = !cpu_has_tsc; }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +if (check_nmi_watchdog () <0 ); + timer_ack = !cpu_has_tsc; + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3-mm3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-8@@ -49,10 +49,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -88,11 +89,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -100,10 +101,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -137,20 +157,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -201,16 +219,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }Quoted 10 lines
Quoted 6 lines
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +0
-1__setup ("idle=" , idle_setup); ---CUT HERE---Attached patches also in tarballs if whitespace problems.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Ross et al.,
I finally got around to trying a newer kernel than 2.6.0-test11 without candy. A few weeks ago I bought a SATA disc, so I wanted to migrate my dual boot system. Due to serious headache caused by an OS from an unnamed company from Redmond this took considerably longer than expected.
Never mind that stuff though, lets cut to the chase. I am currently running a 2.6.3-mm kernel, with a slightly modified forcedeth.c, that is I ripped out DEV_NEED_TIMERIRQ from the pci_device_id struct, so the irq event debugging stuff doesn't get in the way of my Wake-on-Lan hack.
Carl-Daniel Hailfinger suggested this and he is probably removing that code from a future revision altogether. It is feasible that forcedeth will one day support WOL natively.
I patched 2.6.3 vanilla with -mm and the apictack patch under the Craig Bradney section on my little page. Booting with:
append="apic_tack=2 hdg=noprobe"
because of no second sata drive on my siimage chip yields a nice and workable system pretty much. So far I have only had hard lockups trying to dd two partitions simultaneously. It doesn't seem to be realiably reproducable, and it happened maybe two times.
The other quirk is probably APIC-siimage related, but I didn't bother to report it to lkml, since my kernel is tainted anyway:
barton kernel: Disabling IRQ #18hu Mar 4 23:27:20 2004 ...
irq 18: nobody cared!
Call Trace: [<c010ca8a>] __report_bad_irq+0x2a/0x90 [<c010cb7c>] note_interrupt+0x6c/0xa0 [<c010ce51>] do_IRQ+0x121/0x130 [<c0107000>] _stext+0x0/0x60 [<c03f9044>] common_interrupt+0x18/0x20 [<c0107000>] _stext+0x0/0x60 [<c0109053>] default_idle+0x23/0x30 [<c01090bc>] cpu_idle+0x2c/0x40 [<c050a77d>] start_kernel+0x18d/0x1d0 [<c050a480>] unknown_bootoption+0x0/0x120
handlers:
[<c02bf5a0>] (ide_intr+0x0/0x190)
Disabling IRQ #18
hde: sata_error = 0x00080000, watchdog = 1, siimage_mmio_ide_dma_test_irq
So far I have seen this only once, and I don't know what causes it.
Ross, I prefer using your old patch because it keeps my temperature within reasonable bounds when the case is closed. Sorry.
Regards,
Arjen
Quoted 393 lines · Ross Dickson
Quoted 8 lines · Prakash K. Cheemplavam
Oh before I forget, I had to resolve a reject by hand, but I *think* I did it right. (And yes, I used your corrected versionof the patch.) Well, maybe you take a look over the patch and rediff. :-)
Prakash
Hi Prakash,
Patches attached rediffed for 2.6.3 and 2.6.3-mm3.Late comers see start of this thread for more detail.
Also Arjen's page at http://atlas.et.tudelft.nl/verwei90/nforce2/index.htmlNote that the ioapic patch is not essential for 2.6.3-mm3, only if you want to use nmi_debug=1.
2.6.3-mm3 sets up 8254 timer OK in a virtual wire mode without this patch.It will be interesting to see if there is any clock skew associated with these patches, I get no skew on 2.4.24.
The kernel arg to invoke the idle halt workaround is still "idle=C1halt". I have increased slightly the ndelay time from 500ns to 600ns - I had some lockups on startup and shutdown with certain removable IDE drives, this seemed to help. Feel free to adjust the value if you want to.
Also if you want snappier performance you can try different values instead of 6 in this line:
(quoted diff) +1
-0+ (apic_read(APIC_TMCCT) > (apic_read(APIC_TMICT)>>6))) {The 6 yields a divide by 2^6 = 64 for the 1.6% (1/64 = 1.6%)
If you try say 5 for 2^5 = 32 for 3.1% or 4 for 6% or 3 for 12.5% then if less than this percentage of time remains in the slice the system won't go into hlt disconnect thus avoiding the reconnect - recovery time associated with that cycle that would have been.
I have found a need for less ndelay time with the more the percentage (lower number)- there appears to be a tradeoff between the two.
3 for 12.5% or 2 for 25% might even work with no ndelay time i.e. the ndelay line removed or commented from the code, for some.
Of course expect a bit more heat under moderate loads with an increase in the percentage but some may prefer it for their application.
Happy hacking,
Ross.For 2.6.3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +45
-2@@ -2188,12 +2188,56 @@ static inline void check_timer(void)check_nmi_watchdog (); }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +* Note2: this violates the above comment re Subtle but works! +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok, saved_timer_ack = timer_ack; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + timer_ack =0 ; + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +check_nmi_watchdog (); + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ + timer_ack = saved_timer_ack; +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-9@@ -48,10 +48,11 @@#include <asm/irq.h> #include <asm/desc.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -87,11 +88,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -99,10 +100,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -136,20 +156,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -200,16 +218,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---For 2.6.3-mm3 ioapic
---CUT HERE---arch/i386/kernel/io_apic.c renamed from arch/i386/kernel/io_apic.c.original +43
-2@@ -2206,12 +2206,54 @@ static inline void check_timer(void) timer_ack = !cpu_has_tsc; }return ; }clear_IO_APIC_pin (0 , pin1); -printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n" ); +printk (KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n" ,pin1); + } + +#if defined(CONFIG_ACPI_BOOT ) && defined(CONFIG_X86_UP_IOAPIC ) +/* for nforce2 try vector 0 on pin0 +* Note 8259a is already masked, also by default +* the io_apic_set_pci_routing call disables the 8259 irq 0 +* so we must be connected directly to the 8254 timer if this works +*/ +printk (KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n" ); +if (pin1 !=-1 ) { +extern spinlock_t i8259A_lock; +unsigned long flags; +int tok; +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +io_apic_set_pci_routing (0 ,0 ,0 ,0 ,0 );/* connect pin */ +unmask_IO_APIC_irq (0 ); + +/* +* Ok, does IRQ0 through the IOAPIC work? +*/ +spin_lock_irqsave (&i8259A_lock, flags); +Dprintk ("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n" ,inb (0x21 ),inb (0xA1 )); + tok =timer_irq_works (); +spin_unlock_irqrestore (&i8259A_lock, flags); +if (tok) { +if (nmi_watchdog ==NMI_IO_APIC ) { +disable_8259A_irq (0 ); +setup_nmi (); +enable_8259A_irq (0 ); +if (check_nmi_watchdog () <0 ); + timer_ack = !cpu_has_tsc; + } +printk (KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" ); +return ; + } +/* failed */ +clear_IO_APIC_pin (0 ,0 ); +io_apic_set_pci_routing (0 , pin1,0 ,0 ,0 ); +printk (KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n" ); } +#endif printk (KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... " );if (pin2 !=-1 ) {printk ("\n..... (found pin %d) ..." , pin2);/* ---CUT HERE--- For 2.6.3-mm3 idleC1halt
---CUT HERE---arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c.original +28
-9@@ -49,10 +49,11 @@#include <asm/desc.h> #include <asm/atomic_kmap.h> #ifdef CONFIG_MATH_EMULATION #include <asm/math_emu.h> #endif +#include <asm/apic.h> #include <linux/irq.h> #include <linux/err.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");@@ -88,11 +89,11 @@ EXPORT_SYMBOL(enable_hlt);/* * We use this if we don't have any better * idle routine.. */ -void default_idle (void ) +static void default_idle (void ) {if (!hlt_counter && current_cpu_data.hlt_works_ok) {local_irq_disable ();if (!need_resched ())safe_halt ();@@ -100,10 +101,29 @@ void default_idle(void)local_irq_enable (); } }/* +* We use this to avoid nforce2 lockups +* Reduces frequency of C1 disconnects +*/ +static void c1halt_idle (void ) +{ +if (!hlt_counter && current_cpu_data.hlt_works_ok) { +local_irq_disable (); +/* only hlt disconnect if more than 1.6% of apic interval remains */ +if ( (!need_resched ()) && + (apic_read (APIC_TMCCT ) > (apic_read (APIC_TMICT )>>6 ))) { +ndelay (600 );/* helps nforce2 but adds 0.6us hard int latency */ +safe_halt ();/* nothing better to do until we wake up */ + } +else +local_irq_enable (); + } +} + +/* * On SMP it's slightly faster (but much more power-consuming!) * to poll the ->work.need_resched flag instead of waiting for the * cross-CPU IPI to arrive. Use this option with caution. */ static void poll_idle (void )@@ -137,20 +157,18 @@ static void poll_idle (void)* The idle thread. There's no useful work to be * done, so just try to conserve power and have a * low exit latency (ie sit in a loop waiting for * somebody to say that they'd like to reschedule) */ +static void (*idle)(void );void cpu_idle (void ) {/* endless idle loop with no priority at all */ while (1 ) {while (!need_resched ()) { -void (*idle)(void ) = pm_idle; -if (!idle) - idle = default_idle; - + idle = pm_idle ? pm_idle : default_idle; irq_stat[smp_processor_id ()].idle_timestamp = jiffies;idle (); }schedule (); }@@ -201,16 +219,18 @@ void __init select_idle_routine(const ststatic int __init idle_setup (char *str) {if (!strncmp (str,"poll" ,4 )) {printk ("using polling idle threads.\n" ); - pm_idle = poll_idle; + idle = poll_idle; }else if (!strncmp (str,"halt" ,4 )) {printk ("using halt in idle threads.\n" ); - pm_idle = default_idle; + idle = default_idle; + }else if (!strncmp (str,"C1halt" ,6 )) { +printk ("using C1 halt disconnect friendly idle threads.\n" ); + idle = c1halt_idle; } -return 1 ; }__setup ("idle=" , idle_setup); ---CUT HERE---Attached patches also in tarballs if whitespace problems.
Hi Arjen
<snip>
Quoted 5 lines
So far I have seen this only once, and I don't know what causes it.
Ross, I prefer using your old patch because it keeps my temperature within reasonable bounds when the case is closed. Sorry.
<snip>
I have put the idle=C1halt patch that Ross released a little while back now into Gentoo-dev-sources-2.6.3 as I reported to the list yesterday. I no longer use the old apic_tack=2 patch. I have been playing silly buggers with hardware, but so far the PC has made it to 11 hours and now 7 hours with no issues.
After those 11 hours I decided I'd change the PC setup in here and disconnected a fan and a hard drive and moved my server s/w (apache etc) to this PC so I only have one in here fpr now.
Right now, CPU is at 34C which is only 1-3C higher than with the other patch, including having one less fan sucking air out the back of the box. Motherboard is actually lower (27C) than before (29C). Ambient room temp is normal.
After those 11 hours, I am quite sure that on normal use (ie not compiling) the motherboard and cpu was 1-2C lower than with apic_tack=2.
regards
Craig
Yes, but for me the temp diff in Celsius between idle and load for the CPU is almost 20 degrees. This has a dramatic impact on the case temperature when it is closed because I haven't added fans in the top of the case yet.
So you see I value the disconnect quite a bit. Maybe when I get better cooling I will disable it altogether in the BIOS so this headache will disappear forever ;)
Arjen
Quoted 33 lines · Craig Bradney
Hi Arjen
<snip>
Quoted 5 lines
So far I have seen this only once, and I don't know what causes it.
Ross, I prefer using your old patch because it keeps my temperature within reasonable bounds when the case is closed. Sorry.
<snip>
I have put the idle=C1halt patch that Ross released a little while back now into Gentoo-dev-sources-2.6.3 as I reported to the list yesterday. I no longer use the old apic_tack=2 patch. I have been playing silly buggers with hardware, but so far the PC has made it to 11 hours and now 7 hours with no issues.
After those 11 hours I decided I'd change the PC setup in here and disconnected a fan and a hard drive and moved my server s/w (apache etc) to this PC so I only have one in here fpr now.
Right now, CPU is at 34C which is only 1-3C higher than with the other patch, including having one less fan sucking air out the back of the box. Motherboard is actually lower (27C) than before (29C). Ambient room temp is normal.
After those 11 hours, I am quite sure that on normal use (ie not compiling) the motherboard and cpu was 1-2C lower than with apic_tack=2.
regards
Craig
Hi,
I added a section about Wake-on-LAN on my humble website. Most of you never seem to turn your boxes off, but maybe it could proof useful in the future.
http://atlas.et.tudelft.nl/verwei90/nforce2/wol.html
Regards,
Arjen
Quoted 47 lines · Arjen Verweij
Yes, but for me the temp diff in Celsius between idle and load for the CPU is almost 20 degrees. This has a dramatic impact on the case temperature when it is closed because I haven't added fans in the top of the case yet.
So you see I value the disconnect quite a bit. Maybe when I get better cooling I will disable it altogether in the BIOS so this headache will disappear forever ;)
Arjen
Quoted 33 lines · Craig Bradney
Hi Arjen
<snip>
Quoted 5 lines
So far I have seen this only once, and I don't know what causes it.
Ross, I prefer using your old patch because it keeps my temperature within reasonable bounds when the case is closed. Sorry.
<snip>
I have put the idle=C1halt patch that Ross released a little while back now into Gentoo-dev-sources-2.6.3 as I reported to the list yesterday. I no longer use the old apic_tack=2 patch. I have been playing silly buggers with hardware, but so far the PC has made it to 11 hours and now 7 hours with no issues.
After those 11 hours I decided I'd change the PC setup in here and disconnected a fan and a hard drive and moved my server s/w (apache etc) to this PC so I only have one in here fpr now.
Right now, CPU is at 34C which is only 1-3C higher than with the other patch, including having one less fan sucking air out the back of the box. Motherboard is actually lower (27C) than before (29C). Ambient room temp is normal.
After those 11 hours, I am quite sure that on normal use (ie not compiling) the motherboard and cpu was 1-2C lower than with apic_tack=2.
regards
Craig
Quoted 9 lines · Craig Bradney
<snip>
I have put the idle=C1halt patch that Ross released a little while back now into Gentoo-dev-sources-2.6.3 as I reported to the list yesterday. I no longer use the old apic_tack=2 patch. I have been playing silly buggers with hardware, but so far the PC has made it to 11 hours and now 7 hours with no issues.
Just thought I would let everyone know that my A7N8X deluxe is up to:
13:34:38 up 18 days, 18:50, 7 users, load average: 1.00, 1.03, 1.02
with 2.6.3-mm1 and no other patches. I did put Ross's latest idle=C1halt patch in their but forgot to pass the command-line arg. I do have disconnect turned off here, with it on my speakers make a high pitched buzzing noise. Of course, acpi, apic, local apic etc is turned on. So does anyone have a sure fire way to hang this thing yet?
Signature · 6 lines
Josh McKinney | Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
| They that can give up essential liberty
Linux, the choice -o) | to obtain a little temporary safety deserve
of the GNU generation /\ | neither liberty or safety.
_\_v | -Benjamin FranklinQuoted 1 line · Ross Dickson
Patches attached rediffed for 2.6.3 and 2.6.3-mm3.
I've been running the idleC1halt patch against 2.6.3 and 2.6.4 for some time and have had no crashes and no clock skew. A7N8X Delux mobo, AMD Athlon XP 3000+.
Thanks for your work.
-- Edd