HT apparently not detected properly on 2.4.23
Hi,
With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:CONFIG_ACPI=yCONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:CONFIG_NR_CPUS=4
Also, I believe this has been reported before, but the system only seems to interrupt on CPU0 with either kernel, unless I apply a patch I found here:
http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
or run the `irqbalance' program.. which I don't care to do. This problem _seems_ to be isolated to the Supermicro, as HT does seem properly detected on several other SMP systems I have at work (compaq ML370/ML380) with 2.4.23
-Ethan
Quoted 9 lines · Ethan Weinstein
Hi,
With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=yCONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:CONFIG_NR_CPUS=4
This may be the known problem about CONFIG_NR_CPU's not working properly in all cases. Try up'ing it to 32.
Quoted 11 lines · Andre Tomt
Quoted 9 lines · Ethan Weinstein
Hi,
With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=yCONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:CONFIG_NR_CPUS=4
Quoted 2 lines
This may be the known problem about CONFIG_NR_CPU's not working properly in all cases. Try up'ing it to 32.
Depending on the logical addressing of your processors, CONFIG_NR_CPUS=8 may work in this case too.
Some processors/motherboards logically address the CPUs other that 0-3 if there are 4 processors.
Currently CONFIG_NR_CPUS needs to be big enough to hold the largest logical processor number.
Quoted 8 lines · Mike Fedyk
Depending on the logical addressing of your processors,
CONFIG_NR_CPUS=8may work in this case too.Some processors/motherboards logically address the CPUs other that 0-3 if there are 4 processors.
Currently
CONFIG_NR_CPUSneeds to be big enough to hold the largest logical processor number.
Thanks. This rings a bell, I seem to remember what I thought to be "impossible numbering" on the CPUs such as "CPU7" or somesuch in a dmesg.. Will have to look through logs. I'll raise CONFIG_NR_CPUS and see what I get. Now, the question is, why do we only interrupt on CPU0 on this pasrticular box|chipset with a vanilla kernel? ACPI problem? My SMP G4 ppc has an option in menuconfig: "distribute interrupts on all CPUs by default", the x86 has no such option.
Ethan
Quoted 7 lines · Ethan Weinstein
With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=yCONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:CONFIG_NR_CPUS=4
Quoted 8 lines
Also, I believe this has been reported before, but the system only seems to interrupt on CPU0 with either kernel, unless I apply a patch I found here:
http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
or run the `irqbalance' program.. which I don't care to do. This problem _seems_ to be isolated to the Supermicro, as HT does seem properly detected on several other SMP systems I have at work (compaq ML370/ML380) with 2.4.23
You probably have sparse physical APIC ID's.
-- wli
Quoted 11 lines · William Lee Irwin III
Quoted 8 lines · Ethan Weinstein
With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=yCONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:CONFIG_NR_CPUS=4
Also, I believe this has been reported before, but the system only seems to interrupt on CPU0 with either kernel, unless I apply a patch I found here:
http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
or run the `irqbalance' program.. which I don't care to do. This problem _seems_ to be isolated to the Supermicro, as HT does seem properly detected on several other SMP systems I have at work (compaq ML370/ML380) with 2.4.23
Quoted 2 lines
You probably have sparse physical APIC ID's.
Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like it was the numbering scheme:
Dec 3 18:32:28 spicymeatball kernel: CPU 0 (0x0000) enabled Dec 3 18:32:28 spicymeatball kernel: Processor #0 Pentium 4(tm) XEON(tm) APIC version 16 Dec 3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0001] id[0x6] enabled[1]) Dec 3 18:32:28 spicymeatball kernel: CPU 1 (0x0600) enabled Dec 3 18:32:28 spicymeatball kernel: Processor #6 Pentium 4(tm) XEON(tm) APIC version 16 Dec 3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0002] id[0x1] enabled[1]) Dec 3 18:32:28 spicymeatball kernel: CPU 2 (0x0100) enabled Dec 3 18:32:28 spicymeatball kernel: Processor #1 Pentium 4(tm) XEON(tm) APIC version 16 Dec 3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0003] id[0x7] enabled[1]) Dec 3 18:32:28 spicymeatball kernel: CPU 3 (0x0700) enabled Dec 3 18:32:28 spicymeatball kernel: Processor #7 Pentium 4(tm) XEON(tm) APIC version 16
But we're still only interrupting on CPU0 with this kernel.
-Ethan
Quoted 1 line · William Lee Irwin III
Quoted 1 line
You probably have sparse physical APIC ID's.
Quoted 2 lines · Ethan Weinstein
Ok, setting
CONFIG_NR_CPUS=8does indeed solve the HT issue, looks like it was the numbering scheme:
It's easy to fix this; just terminate the loop when the count of cpus kicked hits NR_CPUS or the bit would overflow the map instead of the bit hitting NR_CPUS. Compare smpboot.c in 2.4 vs. 2.6. No need for the extra CONFIG_NR_CPUS bloat.
Quoted 1 line · Ethan Weinstein
But we're still only interrupting on CPU0 with this kernel.
That's P-IV retardation. Not really fixable, but irqbalance daemons etc. can help mitigate the severity of the erratum. I'm not sure if 2.4 has the interfaces for the userspace solutions to work or the kernel solutions either.
-- wli
Quoted 11 lines · William Lee Irwin III
Quoted 1 line · Ethan Weinstein
But we're still only interrupting on CPU0 with this kernel.
That's P-IV retardation. Not really fixable, but irqbalance daemons etc. can help mitigate the severity of the erratum. I'm not sure if 2.4 has the interfaces for the userspace solutions to work or the kernel solutions either.
I somehow suspected that, and appreciate the clarification. I'll see if I can't get that patch I found to apply to 2.4.23, it seems to solve this a bit more elegantly than the irqbalance program.
CPU0 CPU1 CPU2 CPU3
0: 67962 69066 68010 68954 IO-APIC-edge timer 1: 1 0 0 1 IO-APIC-edge keyboard 2: 0 0 0 0 XT-PIC cascade 3: 10027 10944 9904 11063 IO-APIC-edge serial 8: 2 0 0 0 IO-APIC-edge rtc 16: 9 7 7 18 IO-APIC-level sym53c8xx 22: 6842 7336 7166 7429 IO-APIC-level eth0 48: 6429 6370 6202 6268 IO-APIC-level aic79xx 49: 1939 2018 2067 1949 IO-APIC-level aic79xx 54: 340 345 340 352 IO-APIC-level eth1 NMI: 0 0 0 0 LOC: 273824 273846 273844 273845 ERR: 0 MIS: 0
-Ethan
Quoted 2 lines · Ethan Weinstein
Ok, setting
CONFIG_NR_CPUS=8does indeed solve the HT issue, looks like it was the numbering scheme:
Something like this might do the trick. NR_CPUS is already checked indirectly via max_cpus.
-- wli
===== arch/i386/kernel/smpboot.c 1.17 vs edited =====
arch/i386/kernel/smpboot.c +1 -1
@@ -1106,7 +1106,7 @@*/ Dprintk ("CPU present map: %lx\n" , phys_cpu_present_map); -for (bit =0 ; bit <NR_CPUS ; bit++) { +for (bit =0 ; bit <BITS_PER_LONG ; bit++) { apicid =cpu_present_to_apicid (bit);/* don't try to boot BAD_APICID */
Quoted 13 lines · William Lee Irwin III
Quoted 1 line · William Lee Irwin III
Quoted 1 line
You probably have sparse physical APIC ID's.
Quoted 2 lines · Ethan Weinstein
Ok, setting
CONFIG_NR_CPUS=8does indeed solve the HT issue, looks like it was the numbering scheme:It's easy to fix this; just terminate the loop when the count of cpus kicked hits
NR_CPUSor the bit would overflow the map instead of the bit hittingNR_CPUS. Comparesmpboot.cin 2.4 vs. 2.6. No need for the extraCONFIG_NR_CPUSbloat.
Quoted 7 lines
Quoted 1 line · Ethan Weinstein
But we're still only interrupting on CPU0 with this kernel.
That's P-IV retardation. Not really fixable, but irqbalance daemons etc. can help mitigate the severity of the erratum. I'm not sure if 2.4 has the interfaces for the userspace solutions to work or the kernel solutions either.
I think I will just remove CONFIG_NR_CPUS instead... it seems to bring more trouble than advantages.
Quoted 2 lines · Marcelo Tosatti
I think I will just remove
CONFIG_NR_CPUSinstead... it seems to bring more trouble than advantages.
I've gotten a success report from this patch (forwarded separately), but that's okay.
To handle sparse physical APIC ID's within 2.4 limitations without this patch, NR_CPUS == BITS_PER_LONG whenever CONFIG_SMP=y is an effective requirement.
-- wli
Hmmmm.....
SO someone finally realized that HT is not being detected properly dispite all the boot time parameters in the lilo/grub with APIC turned on? It just seems to be a major problem since 2.4.21 version and I was hoping it was resolved. :)
This patch you mentioned, can you send a copy to me as well? Just want to get my clients off my back. :)
Signature · 3 lines
Best regards, Russell "Elik" Rademacher Freelance Remote System Adminstrator/Tech Support
Hello Ethan,
Seems that patch of yours finally did the job of reporting it right this time around. I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot. And they all finally reports the number of processors correctly as it should be.
Thanks Ethan. I going to put it on more servers that got those and see how they all go as well. As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters. Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.
Let see if the Dell servers also report the same as well when I get though with them tonight.
Friday, December 5, 2003, 11:24:37 AM, you wrote:
Quoted 11 lines · Ethan Weinstein> Russell "Elik" Rademacher wrote:
Quoted 11 lines
Hmmmm.....
SO someone finally realized that HT is not being detected properly
dispite all the boot time parameters in the lilo/grub with APIC turned on? It just seems to be a major problem since 2.4.21 version and I was hoping it was resolved. :)This patch you mentioned, can you send a copy to me as well? Just
want to get my clients off my back. :)
Ethan Weinstein> heheh. Welcome to linux-kernel. Try running Ethan Weinstein> an arch other than x86 if
Ethan Weinstein> you _really_ like having your problems ignored =)
Ethan Weinstein> ===== arch/i386/kernel/smpboot.c 1.17 vs edited =====
Ethan Weinstein> --- 1.17/arch/i386/kernel/smpboot.c Mon Nov 3 05:48:33 2003
Ethan Weinstein> +++ edited/arch/i386/kernel/smpboot.c Wed Dec 3 16:45:27 2003
Ethan Weinstein> @@ -1106,7 +1106,7 @@
Ethan Weinstein> */
Ethan Weinstein> Dprintk("CPU present map: %lx\n", phys_cpu_present_map);
Ethan Weinstein> - for (bit = 0; bit < NR_CPUS; bit++) {
Ethan Weinstein> + for (bit = 0; bit < BITS_PER_LONG; bit++) {
Ethan Weinstein> apicid = cpu_present_to_apicid(bit);
Ethan Weinstein> /* don't try to boot BAD_APICID */
Signature · 3 lines
Best regards, Russell "Elik" Rademacher Freelance Remote System Adminstrator/Tech Support
Quoted 8 lines · Russell "Elik" Rademacher
Hello Ethan,
Seems that patch of yours finally did the job of reporting it right this time around. I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot. And they all finally reports the number of processors correctly as it should be.
Thanks Ethan. I going to put it on more servers that got those and see how they all go as well. As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters. Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.
Let see if the Dell servers also report the same as well when I get though with them tonight.
Glad to help, but don't credit me with that patch, it was courtesy of William Irwin.
-E
Hello Ethan,
So noted. Thanks William for this. Now I have to resume patching the rest of 284 servers from various clients this weekend and have it all brought up to speed.
Friday, December 5, 2003, 4:28:37 PM, you wrote:
Quoted 15 lines · Ethan Weinstein> Russell "Elik" Rademacher wrote:
Quoted 15 lines
Hello Ethan,
Seems that patch of yours finally did the job of reporting it right
this time around. I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot. And they all finally reports the number of processors correctly as it should be.Thanks Ethan. I going to put it on more servers that got those and
see how they all go as well. As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters. Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.
Quoted 2 lines
Quoted 2 lines
Let see if the Dell servers also report the same as well when I get though with them tonight.
Ethan Weinstein> Glad to help, but don't credit me with that Ethan Weinstein> patch, it was courtesy of
Ethan Weinstein> William Irwin.
Ethan Weinstein> -E
Signature · 3 lines
Best regards, Russell "Elik" Rademacher Freelance Remote System Adminstrator/Tech Support
Quoted 3 lines · Russell Elik Rademacher
So noted. Thanks William for this. Now I have to resume patching
the rest of 284 servers from various clients this weekend and have it all brought up to speed.
Good to hear it's resolved the issue.
Thanks for testing.
-- wli