Skip to contentSkip to conversation

HT apparently not detected properly on 2.4.23

DISCUSSION Ethan Weinstein 6 participants 16 messages Dec 3, 2003 – Dec 5, 2003 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi,

With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=y
CONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:
CONFIG_NR_CPUS=4

Also, I believe this has been reported before, but the system only seems to interrupt on CPU0 with either kernel, unless I apply a patch I found here:
http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
or run the `irqbalance' program.. which I don't care to do. This problem _seems_ to be isolated to the Supermicro, as HT does seem properly detected on several other SMP systems I have at work (compaq ML370/ML380) with 2.4.23

-Ethan

#1 lore raw
Quoted 9 lines · Ethan Weinstein

Hi,

With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=y
CONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:
CONFIG_NR_CPUS=4

This may be the known problem about CONFIG_NR_CPU's not working properly in all cases. Try up'ing it to 32.

#2 lore raw
Quoted 11 lines · Andre Tomt
Quoted 9 lines · Ethan Weinstein

Hi,

With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=y
CONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:
CONFIG_NR_CPUS=4

Quoted 2 lines

This may be the known problem about CONFIG_NR_CPU's not working properly in all cases. Try up'ing it to 32.

Depending on the logical addressing of your processors, CONFIG_NR_CPUS=8 may work in this case too.

Some processors/motherboards logically address the CPUs other that 0-3 if there are 4 processors.

Currently CONFIG_NR_CPUS needs to be big enough to hold the largest logical processor number.

#3 lore raw
Quoted 8 lines · Mike Fedyk

Depending on the logical addressing of your processors, CONFIG_NR_CPUS=8 may work in this case too.

Some processors/motherboards logically address the CPUs other that 0-3 if there are 4 processors.

Currently CONFIG_NR_CPUS needs to be big enough to hold the largest logical processor number.

Thanks. This rings a bell, I seem to remember what I thought to be "impossible numbering" on the CPUs such as "CPU7" or somesuch in a dmesg.. Will have to look through logs. I'll raise CONFIG_NR_CPUS and see what I get. Now, the question is, why do we only interrupt on CPU0 on this pasrticular box|chipset with a vanilla kernel? ACPI problem? My SMP G4 ppc has an option in menuconfig: "distribute interrupts on all CPUs by default", the x86 has no such option.

Ethan

#4 lore raw
In reply to Ethan Weinstein
Quoted 7 lines · Ethan Weinstein

With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=y
CONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:
CONFIG_NR_CPUS=4

Quoted 8 lines

Also, I believe this has been reported before, but the system only seems to interrupt on CPU0 with either kernel, unless I apply a patch I found here:
http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
or run the `irqbalance' program.. which I don't care to do. This problem _seems_ to be isolated to the Supermicro, as HT does seem properly detected on several other SMP systems I have at work (compaq ML370/ML380) with 2.4.23

You probably have sparse physical APIC ID's.

-- wli

#5 lore raw
Quoted 11 lines · William Lee Irwin III
Quoted 8 lines · Ethan Weinstein

With 2.4.22, my Supermicro X5DPL-iGM-O (E7501 chipset) with 2 [email protected] and hypertherading enabled shows 4 cpu's in /proc/cpuinfo|proc/interrupts, with:
CONFIG_ACPI=y
CONFIG_ACPI_HT_ONLY=y
The same config with 2.4.23 only shows 2 cpus, even with:
CONFIG_NR_CPUS=4
Also, I believe this has been reported before, but the system only seems to interrupt on CPU0 with either kernel, unless I apply a patch I found here:
http://www.hardrock.org/kernel/2.4.22/irqbalance-2.4.22-MRC.patch
or run the `irqbalance' program.. which I don't care to do. This problem _seems_ to be isolated to the Supermicro, as HT does seem properly detected on several other SMP systems I have at work (compaq ML370/ML380) with 2.4.23

Quoted 2 lines

You probably have sparse physical APIC ID's.

Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like it was the numbering scheme:

Dec  3 18:32:28 spicymeatball kernel: CPU 0 (0x0000) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #0 Pentium 4(tm) XEON(tm) APIC version 16
Dec  3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0001] id[0x6] enabled[1])
Dec  3 18:32:28 spicymeatball kernel: CPU 1 (0x0600) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #6 Pentium 4(tm) XEON(tm) APIC version 16
Dec  3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0002] id[0x1] enabled[1])
Dec  3 18:32:28 spicymeatball kernel: CPU 2 (0x0100) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #1 Pentium 4(tm) XEON(tm) APIC version 16
Dec  3 18:32:28 spicymeatball kernel: LAPIC (acpi_id[0x0003] id[0x7] enabled[1])
Dec  3 18:32:28 spicymeatball kernel: CPU 3 (0x0700) enabled
Dec  3 18:32:28 spicymeatball kernel: Processor #7 Pentium 4(tm) XEON(tm) APIC version 16

But we're still only interrupting on CPU0 with this kernel.

-Ethan

#6 lore raw
Quoted 1 line · William Lee Irwin III
Quoted 1 line

You probably have sparse physical APIC ID's.

Quoted 2 lines · Ethan Weinstein

Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like it was the numbering scheme:

It's easy to fix this; just terminate the loop when the count of cpus kicked hits NR_CPUS or the bit would overflow the map instead of the bit hitting NR_CPUS. Compare smpboot.c in 2.4 vs. 2.6. No need for the extra CONFIG_NR_CPUS bloat.

Quoted 1 line · Ethan Weinstein

But we're still only interrupting on CPU0 with this kernel.

That's P-IV retardation. Not really fixable, but irqbalance daemons etc. can help mitigate the severity of the erratum. I'm not sure if 2.4 has the interfaces for the userspace solutions to work or the kernel solutions either.

-- wli

#7 lore raw
Quoted 11 lines · William Lee Irwin III
Quoted 1 line · Ethan Weinstein

But we're still only interrupting on CPU0 with this kernel.

That's P-IV retardation. Not really fixable, but irqbalance daemons etc. can help mitigate the severity of the erratum. I'm not sure if 2.4 has the interfaces for the userspace solutions to work or the kernel solutions either.

I somehow suspected that, and appreciate the clarification. I'll see if I can't get that patch I found to apply to 2.4.23, it seems to solve this a bit more elegantly than the irqbalance program.

CPU0 CPU1 CPU2 CPU3

  0:      67962      69066      68010      68954    IO-APIC-edge  timer
  1:          1          0          0          1    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  3:      10027      10944       9904      11063    IO-APIC-edge  serial
  8:          2          0          0          0    IO-APIC-edge  rtc
 16:          9          7          7         18   IO-APIC-level  sym53c8xx
 22:       6842       7336       7166       7429   IO-APIC-level  eth0
 48:       6429       6370       6202       6268   IO-APIC-level  aic79xx
 49:       1939       2018       2067       1949   IO-APIC-level  aic79xx
 54:        340        345        340        352   IO-APIC-level  eth1
NMI:          0          0          0          0
LOC:     273824     273846     273844     273845
ERR:          0
MIS:          0

-Ethan

#16 lore raw
In reply to Ethan Weinstein
Quoted 2 lines · Ethan Weinstein

Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like it was the numbering scheme:

Something like this might do the trick. NR_CPUS is already checked indirectly via max_cpus.

-- wli

===== arch/i386/kernel/smpboot.c 1.17 vs edited =====

arch/i386/kernel/smpboot.c +1 -1
@@ -1106,7 +1106,7 @@
 	 */
 	Dprintk("CPU present map: %lx\n", phys_cpu_present_map);
 
-	for (bit = 0; bit < NR_CPUS; bit++) {
+	for (bit = 0; bit < BITS_PER_LONG; bit++) {
 		apicid = cpu_present_to_apicid(bit);
 		
 		/* don't try to boot BAD_APICID */
#8 lore raw
In reply to William Lee Irwin III
Quoted 13 lines · William Lee Irwin III
Quoted 1 line · William Lee Irwin III
Quoted 1 line

You probably have sparse physical APIC ID's.

Quoted 2 lines · Ethan Weinstein

Ok, setting CONFIG_NR_CPUS=8 does indeed solve the HT issue, looks like it was the numbering scheme:

It's easy to fix this; just terminate the loop when the count of cpus kicked hits NR_CPUS or the bit would overflow the map instead of the bit hitting NR_CPUS. Compare smpboot.c in 2.4 vs. 2.6. No need for the extra CONFIG_NR_CPUS bloat.

Quoted 7 lines
Quoted 1 line · Ethan Weinstein

But we're still only interrupting on CPU0 with this kernel.

That's P-IV retardation. Not really fixable, but irqbalance daemons etc. can help mitigate the severity of the erratum. I'm not sure if 2.4 has the interfaces for the userspace solutions to work or the kernel solutions either.

I think I will just remove CONFIG_NR_CPUS instead... it seems to bring more trouble than advantages.

#9 lore raw
Quoted 2 lines · Marcelo Tosatti

I think I will just remove CONFIG_NR_CPUS instead... it seems to bring more trouble than advantages.

I've gotten a success report from this patch (forwarded separately), but that's okay.

To handle sparse physical APIC ID's within 2.4 limitations without this patch, NR_CPUS == BITS_PER_LONG whenever CONFIG_SMP=y is an effective requirement.

-- wli

#10 lore raw

Hmmmm.....

SO someone finally realized that HT is not being detected properly dispite all the boot time parameters in the lilo/grub with APIC turned on? It just seems to be a major problem since 2.4.21 version and I was hoping it was resolved. :)

This patch you mentioned, can you send a copy to me as well? Just want to get my clients off my back. :)

Signature · 3 lines
Best regards,
Russell "Elik" Rademacher
Freelance Remote System Adminstrator/Tech Support
#12 lore raw
In reply to a message not in the archive

Hello Ethan,

Seems that patch of yours finally did the job of reporting it right this time around. I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot. And they all finally reports the number of processors correctly as it should be.

Thanks Ethan. I going to put it on more servers that got those and see how they all go as well. As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters. Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.

Let see if the Dell servers also report the same as well when I get though with them tonight.

Friday, December 5, 2003, 11:24:37 AM, you wrote:

Quoted 11 lines · Ethan Weinstein> Russell "Elik" Rademacher wrote:
Quoted 11 lines

Hmmmm.....

SO someone finally realized that HT is not being detected properly
dispite all the boot time parameters in the lilo/grub with APIC turned on? It just seems to be a major problem since 2.4.21 version and I was hoping it was resolved. :)

This patch you mentioned, can you send a copy to me as well? Just
want to get my clients off my back. :)

Ethan Weinstein> heheh. Welcome to linux-kernel. Try running Ethan Weinstein> an arch other than x86 if
Ethan Weinstein> you _really_ like having your problems ignored =)

Ethan Weinstein> ===== arch/i386/kernel/smpboot.c 1.17 vs edited =====
Ethan Weinstein> --- 1.17/arch/i386/kernel/smpboot.c Mon Nov 3 05:48:33 2003
Ethan Weinstein> +++ edited/arch/i386/kernel/smpboot.c Wed Dec 3 16:45:27 2003
Ethan Weinstein> @@ -1106,7 +1106,7 @@
Ethan Weinstein> */
Ethan Weinstein> Dprintk("CPU present map: %lx\n", phys_cpu_present_map);

Ethan Weinstein> - for (bit = 0; bit < NR_CPUS; bit++) {
Ethan Weinstein> + for (bit = 0; bit < BITS_PER_LONG; bit++) {
Ethan Weinstein> apicid = cpu_present_to_apicid(bit);
Ethan Weinstein> /* don't try to boot BAD_APICID */

Signature · 3 lines
Best regards,
Russell "Elik" Rademacher
Freelance Remote System Adminstrator/Tech Support
#13 lore raw
Quoted 8 lines · Russell "Elik" Rademacher

Hello Ethan,

Seems that patch of yours finally did the job of reporting it right this time around. I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot. And they all finally reports the number of processors correctly as it should be.

Thanks Ethan. I going to put it on more servers that got those and see how they all go as well. As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters. Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.

Let see if the Dell servers also report the same as well when I get though with them tonight.

Glad to help, but don't credit me with that patch, it was courtesy of William Irwin.

-E

#14 lore raw

Hello Ethan,

So noted. Thanks William for this. Now I have to resume patching the rest of 284 servers from various clients this weekend and have it all brought up to speed.

Friday, December 5, 2003, 4:28:37 PM, you wrote:

Quoted 15 lines · Ethan Weinstein> Russell "Elik" Rademacher wrote:
Quoted 15 lines

Hello Ethan,

Seems that patch of yours finally did the job of reporting it right
this time around. I had to gank a few servers of different versions and giving the webhosting customers some little inconvience of downtime of 3 to 10 minutes for the linux recompile and reboot. And they all finally reports the number of processors correctly as it should be.

Thanks Ethan. I going to put it on more servers that got those and
see how they all go as well. As for the servers, they are all usually ServerWorks, Intel Chipsets of various types with Intel E1000, E100, Realtek network ports and with 3Ware or SCSI Adaptec adapters. Got about 80 of them in various types to go though, but 5 of them I that tested it on, it all reports correctly.

Quoted 2 lines
Quoted 2 lines

Let see if the Dell servers also report the same as well when I get though with them tonight.

Ethan Weinstein> Glad to help, but don't credit me with that Ethan Weinstein> patch, it was courtesy of
Ethan Weinstein> William Irwin.

Ethan Weinstein> -E

Signature · 3 lines
Best regards,
Russell "Elik" Rademacher
Freelance Remote System Adminstrator/Tech Support
#15 lore raw
Quoted 3 lines · Russell Elik Rademacher

So noted. Thanks William for this. Now I have to resume patching
the rest of 284 servers from various clients this weekend and have it all brought up to speed.

Good to hear it's resolved the issue.

Thanks for testing.

-- wli