APIC version and 8-bit APIC IDs
Quoted 5 lines · Martin Wilck
Hi William, hello everyone,
The
MP_valid_apicid()function [arch/i386/kernel/mpparse.c] checks whether the APIC version field is >=20 in order to determine whether the CPU supports 8-bit physical APIC ids.
Yes, it's broken. In fact I removed it in my physflat32 patch which is needed for 16 core AMD systems. I don't think there is a generic way to fix it because the XAPIC check breaks
on AMD systems and there is no good way to decide early
on subarchitectures before doing this check. Also it's only a sanity check for broken BIOS, and in this case it causes more problems than it solves.
ftp://ftp.firstfloor.org/pub/ak/x86_64/x86_64-2.6.13rc3-1/patches/physflat32
Will hopefully be fixed in 2.6.14.
-Andi
Quoted 4 lines · Andi Kleen
Yes, it's broken. In fact I removed it in my physflat32 patch which is needed for 16 core AMD systems. I don't think there is a generic way to fix it because the XAPIC check breaks on AMD systems
on the Intel Xeon MP systems, too,
Quoted 3 lines
and there is no good way to decide early on subarchitectures before doing this check. Also it's only a sanity check for broken BIOS, and in this case it causes more problems
than it solves.
agreed.
Quoted 1 line
ftp://ftp.firstfloor.org/pub/ak/x86_64/x86_64-2.6.13rc3-1/patches/physflat32
That is a beautiful patch, thank you.
Only one small point: I wonder whether it is correct to use the number of CPUs as criterion for this architecture. AFAICS, the Specs allow
having only 4 CPUS, but giving them APIC IDs e.g. 16,17,18,19. In this case, physflat32 should be used as well (in particular, the APIC ID broadcast and mask must be set to 0xff).
Quoted 1 line
Will hopefully be fixed in 2.6.14.
Great,
Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Quoted 6 lines · Martin Wilck
Quoted 4 lines
Yes, it's broken. In fact I removed it in my physflat32 patch which is needed for 16 core AMD systems. I don't think there is a generic way to fix it because the XAPIC check breaks on AMD systems
on the Intel Xeon MP systems, too,
How so? The XAPIC version check should work there.
The problem on AMD happens because it reports an old APIC version.
Quoted 11 lines
Quoted 5 lines
and there is no good way to decide early
on subarchitectures before doing this check. Also it's only a sanity check for broken BIOS, and in this case it causes more problems than it solves.agreed.
Quoted 1 line
ftp://ftp.firstfloor.org/pub/ak/x86_64/x86_64-2.6.13rc3-1/patches/physflat32
That is a beautiful patch, thank you.
It'll probably be revamped somewhat before inclusion. In particular it has been suggested to merge it with bigsmp because the setup should work on Intel too.
Quoted 6 lines
Only one small point: I wonder whether it is correct to use the number of CPUs as criterion for this architecture. AFAICS, the Specs allow having only 4 CPUS, but giving them APIC IDs e.g. 16,17,18,19. In this case, physflat32 should be used as well (in particular, the APIC ID broadcast and mask must be set to
0xff).
I fixed that in the 64bit version already, but not in 32bit yet. Yes, the value of the APIC IDs must be used as indicator.
-Andi
Quoted 3 lines · Andi Kleen
Quoted 1 line
on the Intel Xeon MP systems, too,
How so? The XAPIC version check should work there.
ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16
This is a boot message excerpt from one of the "Potomac" systems we have. x86_64 kernel without xapic check, therefore the CPUs are activated, but they wouldn't be on recent i386.
Quoted 2 lines
I fixed that in the 64bit version already, but not in 32bit yet. Yes, the value of the APIC IDs must be used as indicator.
Great.
Regards, Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Quoted 6 lines · I wrote:
Quoted 1 line
How so? The XAPIC version check should work there.
ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16
Forget it. I have fallen prey to this line:
processor.mpc_apicver = 0x10; /* TBD: lapic version */
in arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)
Cheers
Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Quoted 9 lines · Martin Wilck
Quoted 6 lines · I wrote:
Quoted 1 line
How so? The XAPIC version check should work there.
ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16
Quoted 6 lines
Forget it. I have fallen prey to this line:
processor.mpc_apicver = 0x10; /* TBD: lapic version */in
arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)
Heh. Should probably fix that. Can you file a bug with the ACPI people on http://bugzilla.kernel.org ? (or do a patch?)
-Andi
So MPTABLE do not have problem with it, only acpi related...?
YH
Quoted 27 lines · Andi Kleen
Quoted 9 lines · Martin Wilck
Quoted 6 lines · I wrote:
Quoted 1 line
How so? The XAPIC version check should work there.
ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16Quoted 6 lines
Forget it. I have fallen prey to this line:
processor.mpc_apicver = 0x10; /* TBD: lapic version */in
arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)Heh. Should probably fix that. Can you file a bug with the ACPI people on http://bugzilla.kernel.org ? (or do a patch?)
(quoted diff) +0
-2-Andi -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · yhlu
So MPTABLE do not have problem with it, only acpi related...?
It's only a cosmetic problem I think with the printk being wrong. The actual decision in the code should all use the true value.
Another way would be to just remove the printk output.
-Andi
why matrin need to make apic id to be greater than 0x10 when system is only 2way?
too much io-apic in system?
YH
Quoted 11 lines · Andi Kleen
Quoted 1 line · yhlu
So MPTABLE do not have problem with it, only acpi related...?
It's only a cosmetic problem I think with the printk being wrong. The actual decision in the code should all use the true value.
Another way would be to just remove the printk output.
-Andi
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Martin Wilck
Quoted 6 lines · I wrote:
Quoted 1 line
How so? The XAPIC version check should work there.
ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16Forget it. I have fallen prey to this line:
Quoted 7 lines
Quoted 4 lines
processor.mpc_apicver = 0x10; /* TBD: lapic version */in
arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)Heh. Should probably fix that. Can you file a bug with the ACPI people on http://bugzilla.kernel.org ? (or do a patch?)
Here is the patch.
--
Fix the apic version that gets printed during boot.
arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +1 -1
@@ -707,7 +707,7 @@ processor.mpc_type =MP_PROCESSOR ; processor.mpc_apicid = id; - processor.mpc_apicver =0x10 ;/* TBD: lapic version */ + processor.mpc_apicver =GET_APIC_VERSION (apic_read (APIC_LVR )); processor.mpc_cpuflag = (enabled ?CPU_ENABLED :0 ); processor.mpc_cpuflag |= (boot_cpu ?CPU_BOOTPROCESSOR :0 ); processor.mpc_cpufeature = (boot_cpu_data.x86 << 8) |
arch/x86_64/kernel/mpparse.c line 710
Quoted 8 lines
arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +1
-1@@ -707,7 +707,7 @@ processor.mpc_type =MP_PROCESSOR ; processor.mpc_apicid = id; - processor.mpc_apicver =0x10 ;/* TBD: lapic version */ + processor.mpc_apicver =GET_APIC_VERSION (apic_read (APIC_LVR ));
That code executes on the BP. What happens when different CPUs have different APIC versions? Not sure if that can happen, but it still looks a bit like "cheating.
-Andi
Quoted 4 lines · Andi Kleen
Quoted 4 lines
arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +0
-0@@ -707,7 +707,7 @@
Quoted 8 lines
arch/x86_64/kernel/mpparse.c line 710
Quoted 4 lines
arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +1
-1processor.mpc_type =MP_PROCESSOR ; processor.mpc_apicid = id; - processor.mpc_apicver =0x10 ;/* TBD: lapic version */ + processor.mpc_apicver =GET_APIC_VERSION (apic_read (APIC_LVR ));That code executes on the BP. What happens when different CPUs have different APIC versions? Not sure if that can happen, but it still looks a bit like "cheating.
Not sure if we ever support CPUs with different APIC versions. That will probably require some ACPI SPEC changes too..
I would say, for now just follow the i386 code.
thanks,
suresh
Quoted 4 lines
Not sure if we ever support CPUs with different APIC versions. That will probably require some ACPI SPEC changes too..
I would say, for now just follow the i386 code.
Ok I applied the patch. Thanks.
-Andi
Quoted 2 lines · yhlu
why matrin need to make apic id to be greater than
0x10when system is only 2way?
It's a scalable system where multiple boards may be combined. Anyway, I see nothing in the specs that says you must start counting CPUs from zero.
Regards
Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Quoted 2 lines · Martin Wilck
It's a scalable system where multiple boards may be combined. Anyway, I see nothing in the specs that says you must start counting CPUs from zero.
Well, Intel's "Multiprocessor Specification" mandates that (see section
3.6.1 and also the compliance list in Appendix C). I does not mandate local APIC IDs to be consecutive though.
Maciej
Quoted 2 lines · Maciej W. Rozycki
Well, Intel's "Multiprocessor Specification" mandates that (see section
3.6.1 and also the compliance list in Appendix C). I does not mandate local APIC IDs to be consecutive though.
Unless I am mistaken, the MP spec does not say that _CPUs_ must start from 0. We had an IO-APIC at 0. The MP spec says that the IDs must be unique (I am told this isn't true any more because an IO APIC and a CPU may have the same ID) and _need not_ be consecutive.
We tried different setups; one had IO APICs at 0,1,2 and CPUs starting at 16. I can't see that this is forbidden (the reason is that the IO-APICs have only 4-bit APIC ID registers). Anyway we changed it now to have both IO-APICs and CPUs start at 0.
Regards
Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Quoted 4 lines · Martin Wilck
Unless I am mistaken, the MP spec does not say that _CPUs_ must start from 0. We had an IO-APIC at 0. The MP spec says that the IDs must be unique (I am told this isn't true any more because an IO APIC and a CPU may have the same ID) and
_neednot_be consecutive.
You are unfortunately mistaken -- the spec is explicit about *local* APIC
IDs having to start at 0. There are at least two places in the spec that refer to that.
Quoted 4 lines
We tried different setups; one had IO APICs at 0,1,2 and CPUs starting at 16. I can't see that this is forbidden (the reason is that the IO-APICs have only 4-bit APIC ID registers). Anyway we changed it now to have both IO-APICs and CPUs start at 0.
You can always assign 0, 16, 17, etc. to local APICs and then 1, 2, 3,
etc. for I/O APICs. Frankly I don't know what the actual justification behind the requirement is. Note that the ID of 0 need not necessarily belong to the BSP.
Maciej
Quoted 2 lines · Maciej W. Rozycki
You are unfortunately mistaken -- the spec is explicit about *local* APIC
IDs having to start at 0. There are at least two places in the spec that refer to that.
I see. Thanks for the clarification.
Quoted 2 lines
You can always assign 0, 16, 17, etc. to local APICs and then 1, 2, 3,
etc. for I/O APICs.
Actually, that's what we did (I messed thinks up in my posting). I think that's an ugly configuration, though, and totally misleading in terms of system topology.
Quoted 1 line
Frankly I don't know what the actual justification behind the requirement is. Note that the ID of 0 need not necessarily belong to the BSP.
Starting both local APICs and IO-APICs at 0 isa working and aesthetically pleasing solution. We're pursuing that now.
Thanks again, Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Hi Andi, hi everyone,
Quoted 7 lines
Quoted 3 lines
The
MP_valid_apicid()function [arch/i386/kernel/mpparse.c] checks whether the APIC version field is >=20 in order to determine whether the CPU supports 8-bit physical APIC ids.Yes, it's broken. ... . Also it's only
a sanity check for broken BIOS, and in this case it causes more problems than it solves.
We have another issue with the APIC IDs: the GET_APIC_ID and APIC_ID_MASK macros from the subarch code. In all subarchs except summit and es7000, these macros use a hard-coded mask 0x0F for the APIC ID. Thus, any APIC ID >15 will be truncated, leading to obscure errors.
We are wondering why these masks are there in the subarch code at all. After all, whether or not 8-bit APIC IDs are supported depends mainly on the CPU type used. Why wouldn't it possible to have a "default" architecture with APIC IDs > 15, if the CPUs allow it?
In other words: What would be broken if we just used an APIC ID mask of 0xFF everywhere?
The current situation with MP_valid_apicid() on the one hand (masking the APIC ID as a function of local APIC version) and APIC_ID_MASK (masking the APIC as a function of subarch) on the other hand is inconsistent. A correct approach must take both CPU and architecture constraints into account, and use a CPU-type-dependent variable mask in the subarch code.
Regards
Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Quoted 4 lines · Martin Wilck
We are wondering why these masks are there in the subarch code at all. After all, whether or not 8-bit APIC IDs are supported depends mainly on the CPU type used. Why wouldn't it possible to have a "default" architecture with APIC IDs > 15, if the CPUs allow it?
It actually depends on the APIC type, rather than the CPU. E.g. with
Pentium systems the width of the ID is either 4 bits or 8 bits, depending on whether the integrated or an external 82489DX APIC is used. This should be able to be determined by the APIC version; for v <= 0xf the ID is 8-bit and for v >= 0x10 it used to be 4-bit. Now you only need to determine what is the value of v above 0x10 that makes the ID 8-bit again.
Maciej
Hi Maciej,
Quoted 7 lines
It actually depends on the APIC type, rather than the CPU. E.g. with Pentium systems the width of the ID is either 4 bits or 8 bits, depending on whether the integrated or an external 82489DX APIC is used. This should be able to be determined by the APIC version; for v <=
0xfthe ID is 8-bit and for v >=0x10it used to be 4-bit. Now you only need to determine what is the value of v above0x10that makes the ID 8-bit again.
That would be v>=0x14 for Intel. But that is wrong for AMD CPUs. The actual Dual-Core Athlon CPUs we have report an APIC version of 0x10. Please refer to the start of this thread.
Anyway, I understand that you agree this does not belong into the subarch code?
Regards
Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy
Quoted 2 lines · Martin Wilck
In other words: What would be broken if we just used an APIC ID mask of
0xFFeverywhere?
Nothing I think. It's more historical reasons. The physflat subarchitecture patch essentially removed it, but it needs some rework and merging with bigsmp now.
Quoted 6 lines
The current situation with
MP_valid_apicid()on the one hand (masking the APIC ID as a function of local APIC version) andAPIC_ID_MASK(masking the APIC as a function of subarch) on the other hand is inconsistent. A correct approach must take both CPU and architecture constraints into account, and use a CPU-type-dependent variable mask in the subarch code.
Yes, it's broken right now.
-Andi
[ Putting everyone cc again and leaving maciej's reply intact for reference ]
Quoted 18 lines · Maciej W. Rozycki
Quoted 1 line
The ID is
0x14for Intel. But that is wrong for AMD CPUs. The actual Dual-CoreWhy can't hw designers ever get such things right? Sigh...
Quoted 2 lines
Athlon CPUs we have report an APIC version of
0x10. Please refer to the start of this thread.Frankly, such a change implies a change to the inter-APIC communication
protocol, so the major revision should have been bumped, like it happened for the I/O APIC (0x20); I hope they do not worry of changing the design so many times they run out of numbers! Thus none of the implementers has done their job properly, but for Intel there is at least some change, while for AMD there seems no generic way of determining that --0x10implies a "traditional" integrated APIC as implemented in Pentium in 1995, using a three-wire serial bus for communication...Perhaps a variable should be added holding the "architectural revision"
of the APIC subsystem, set up by the early CPU/APIC initialization code and used from there on instead of direct references to the version register as implemented in the hardware. It does not have to be based on what hardware uses, e.g.:- 0: no APIC,
Quoted 8 lines
- 1: 82489DX -- communication using a five-wire inter-APIC bus, 8-bit physical ID, 32-bit logical ID, etc.,
- 2: Pentium-style -- communication using a three-wire inter-APIC bus, 4-bit physical ID, 8-bit logical ID, etc.,
- 3: P4-style -- communication using the system bus, 8-bit physical ID, 8-bit logical ID, etc.
AMD has their HyperTransport bus, which is yet a different thing.
According to what AMD told us, all K8 CPUs support 8-bit APIC IDs ("extended APIC IDs" in their speak), but they are only enabled "when both ApicExtId and ApicExtBrdCst in the HyperTransport Transaction Control Register are set".
As we cannot read PCI config space at APIC configuration time, my suggestion would be something like
unsigned char phys_apic_id_mask(int apic_version)
if (apic_version < 0x10 || apic_version >= 0x14 ||
(boot_cpu_data.vendor == X86_VENDOR_AMD &&
boot_cpu_data.family >= 0x0f))
return 0xff;
else
return 0x0f;
}That would assume that no BIOS is so stupid to set APIC IDs >0xf and at the same time forget to enable the respective bits in the HyperTransport Transaction Control Register.
I do not oversee whether boot_cpu_data is initialized when the APIC IDs need to be tested/masked, but somebody else certainly does.
Quoted 10 lines
Quoted 4 lines
Anyway, I understand that you agree this does not belong into the subarch code?
That's CPU/chipset specific indeed. It shouldn't really depend on the
platform these are used in. Exceptions, if any, can be handled on a case by case basis.Maciej
Regards
Martin
Signature · 4 lines
Martin Wilck Phone: +49 5251 8 15113 Fujitsu Siemens Computers Fax: +49 5251 8 20409 Heinz-Nixdorf-Ring 1 mailto:[email protected] D-33106 Paderborn http://www.fujitsu-siemens.com/primergy