Skip to contentSkip to conversation

APIC version and 8-bit APIC IDs

DISCUSSION Andi Kleen 5 participants 23 messages Aug 12, 2005 – Aug 31, 2005 3 messages not in the archive

lore
Share & tools
lore

Andi Kleen

author
#1 lore raw
In reply to a message not in the archive
Quoted 5 lines · Martin Wilck

Hi William, hello everyone,

The MP_valid_apicid() function [arch/i386/kernel/mpparse.c] checks whether the APIC version field is >=20 in order to determine whether the CPU supports 8-bit physical APIC ids.

Yes, it's broken. In fact I removed it in my physflat32 patch which is needed for 16 core AMD systems. I don't think there is a generic way to fix it because the XAPIC check breaks
on AMD systems and there is no good way to decide early
on subarchitectures before doing this check. Also it's only a sanity check for broken BIOS, and in this case it causes more problems than it solves.

ftp://ftp.firstfloor.org/pub/ak/x86_64/x86_64-2.6.13rc3-1/patches/physflat32

Will hopefully be fixed in 2.6.14.

-Andi

#2 lore raw
Quoted 4 lines · Andi Kleen

Yes, it's broken. In fact I removed it in my physflat32 patch which is needed for 16 core AMD systems. I don't think there is a generic way to fix it because the XAPIC check breaks on AMD systems

on the Intel Xeon MP systems, too,

Quoted 3 lines

and there is no good way to decide early on subarchitectures before doing this check. Also it's only a sanity check for broken BIOS, and in this case it causes more problems
than it solves.

agreed.

Quoted 1 line

ftp://ftp.firstfloor.org/pub/ak/x86_64/x86_64-2.6.13rc3-1/patches/physflat32

That is a beautiful patch, thank you.

Only one small point: I wonder whether it is correct to use the number of CPUs as criterion for this architecture. AFAICS, the Specs allow
having only 4 CPUS, but giving them APIC IDs e.g. 16,17,18,19. In this case, physflat32 should be used as well (in particular, the APIC ID broadcast and mask must be set to 0xff).

Quoted 1 line

Will hopefully be fixed in 2.6.14.

Great,

Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
author
#3 lore raw
Quoted 6 lines · Martin Wilck
Quoted 4 lines

Yes, it's broken. In fact I removed it in my physflat32 patch which is needed for 16 core AMD systems. I don't think there is a generic way to fix it because the XAPIC check breaks on AMD systems

on the Intel Xeon MP systems, too,

How so? The XAPIC version check should work there.

The problem on AMD happens because it reports an old APIC version.

Quoted 11 lines
Quoted 5 lines

and there is no good way to decide early
on subarchitectures before doing this check. Also it's only a sanity check for broken BIOS, and in this case it causes more problems than it solves.

agreed.

Quoted 1 line

ftp://ftp.firstfloor.org/pub/ak/x86_64/x86_64-2.6.13rc3-1/patches/physflat32

That is a beautiful patch, thank you.

It'll probably be revamped somewhat before inclusion. In particular it has been suggested to merge it with bigsmp because the setup should work on Intel too.

Quoted 6 lines

Only one small point: I wonder whether it is correct to use the number of CPUs as criterion for this architecture. AFAICS, the Specs allow having only 4 CPUS, but giving them APIC IDs e.g. 16,17,18,19. In this case, physflat32 should be used as well (in particular, the APIC ID broadcast and mask must be set to 0xff).

I fixed that in the 64bit version already, but not in 32bit yet. Yes, the value of the APIC IDs must be used as indicator.

-Andi

#4 lore raw
Quoted 3 lines · Andi Kleen
Quoted 1 line

on the Intel Xeon MP systems, too,

How so? The XAPIC version check should work there.

ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16

This is a boot message excerpt from one of the "Potomac" systems we have. x86_64 kernel without xapic check, therefore the CPUs are activated, but they wouldn't be on recent i386.

Quoted 2 lines

I fixed that in the 64bit version already, but not in 32bit yet. Yes, the value of the APIC IDs must be used as indicator.

Great.
Regards, Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#5 lore raw
Quoted 6 lines · I wrote:
Quoted 1 line

How so? The XAPIC version check should work there.

ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16

Forget it. I have fallen prey to this line:

	processor.mpc_apicver = 0x10; /* TBD: lapic version */

in arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)

Cheers
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
author
#6 lore raw
Quoted 9 lines · Martin Wilck
Quoted 6 lines · I wrote:
Quoted 1 line

How so? The XAPIC version check should work there.

ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16

Quoted 6 lines

Forget it. I have fallen prey to this line:

	processor.mpc_apicver = 0x10; /* TBD: lapic version */

in arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)

Heh. Should probably fix that. Can you file a bug with the ACPI people on http://bugzilla.kernel.org ? (or do a patch?)

-Andi

#7 lore raw

So MPTABLE do not have problem with it, only acpi related...?

YH

Quoted 27 lines · Andi Kleen
Quoted 9 lines · Martin Wilck
Quoted 6 lines · I wrote:
Quoted 1 line

How so? The XAPIC version check should work there.

ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16

Quoted 6 lines

Forget it. I have fallen prey to this line:

      processor.mpc_apicver = 0x10; /* TBD: lapic version */

in arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)

Heh. Should probably fix that. Can you file a bug with the ACPI people on http://bugzilla.kernel.org ? (or do a patch?)

(quoted diff) +0 -2
-Andi
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#8 lore raw
Quoted 1 line · yhlu

So MPTABLE do not have problem with it, only acpi related...?

It's only a cosmetic problem I think with the printk being wrong. The actual decision in the code should all use the true value.

Another way would be to just remove the printk output.

-Andi

#9 lore raw

why matrin need to make apic id to be greater than 0x10 when system is only 2way?

too much io-apic in system?

YH

Quoted 11 lines · Andi Kleen
Quoted 1 line · yhlu

So MPTABLE do not have problem with it, only acpi related...?

It's only a cosmetic problem I think with the printk being wrong. The actual decision in the code should all use the true value.

Another way would be to just remove the printk output.

-Andi

#15 lore raw
In reply to Andi Kleen
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Martin Wilck
Quoted 6 lines · I wrote:
Quoted 1 line

How so? The XAPIC version check should work there.

ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] enabled) Processor #33 15:4 APIC version 16
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x26] enabled) Processor #38 15:4 APIC version 16

Forget it. I have fallen prey to this line:

Quoted 7 lines
Quoted 4 lines
	processor.mpc_apicver = 0x10; /* TBD: lapic version */

in arch/x86_64/kernel/mpparse.c.
I am used to get correct answers from Linux :-)

Heh. Should probably fix that. Can you file a bug with the ACPI people on http://bugzilla.kernel.org ? (or do a patch?)

Here is the patch.

--
Fix the apic version that gets printed during boot.

arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +1 -1
@@ -707,7 +707,7 @@
 
 	processor.mpc_type = MP_PROCESSOR;
 	processor.mpc_apicid = id;
-	processor.mpc_apicver = 0x10; /* TBD: lapic version */
+	processor.mpc_apicver = GET_APIC_VERSION(apic_read(APIC_LVR));
 	processor.mpc_cpuflag = (enabled ? CPU_ENABLED : 0);
 	processor.mpc_cpuflag |= (boot_cpu ? CPU_BOOTPROCESSOR : 0);
 	processor.mpc_cpufeature = (boot_cpu_data.x86 << 8) | 
author
#16 lore raw

arch/x86_64/kernel/mpparse.c line 710

Quoted 8 lines
arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +1 -1
@@ -707,7 +707,7 @@
 
 	processor.mpc_type = MP_PROCESSOR;
 	processor.mpc_apicid = id;
-	processor.mpc_apicver = 0x10; /* TBD: lapic version */
+	processor.mpc_apicver = GET_APIC_VERSION(apic_read(APIC_LVR));

That code executes on the BP. What happens when different CPUs have different APIC versions? Not sure if that can happen, but it still looks a bit like "cheating.

-Andi

#17 lore raw
Quoted 4 lines · Andi Kleen
Quoted 4 lines
arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +0 -0
@@ -707,7 +707,7 @@
 
Quoted 8 lines

arch/x86_64/kernel/mpparse.c line 710

Quoted 4 lines
arch/x86_64/kernel/mpparse.c renamed from arch/x86_64/kernel/mpparse.c~ +1 -1
 	processor.mpc_type = MP_PROCESSOR;
 	processor.mpc_apicid = id;
-	processor.mpc_apicver = 0x10; /* TBD: lapic version */
+	processor.mpc_apicver = GET_APIC_VERSION(apic_read(APIC_LVR));

That code executes on the BP. What happens when different CPUs have different APIC versions? Not sure if that can happen, but it still looks a bit like "cheating.

Not sure if we ever support CPUs with different APIC versions. That will probably require some ACPI SPEC changes too..

I would say, for now just follow the i386 code.

thanks,
suresh

author
#18 lore raw
Quoted 4 lines

Not sure if we ever support CPUs with different APIC versions. That will probably require some ACPI SPEC changes too..

I would say, for now just follow the i386 code.

Ok I applied the patch. Thanks.

-Andi

#10 lore raw
In reply to yhlu
Quoted 2 lines · yhlu

why matrin need to make apic id to be greater than 0x10 when system is only 2way?

It's a scalable system where multiple boards may be combined. Anyway, I see nothing in the specs that says you must start counting CPUs from zero.

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#11 lore raw
Quoted 2 lines · Martin Wilck

It's a scalable system where multiple boards may be combined. Anyway, I see nothing in the specs that says you must start counting CPUs from zero.

Well, Intel's "Multiprocessor Specification" mandates that (see section
3.6.1 and also the compliance list in Appendix C). I does not mandate local APIC IDs to be consecutive though.

Maciej

#12 lore raw
Quoted 2 lines · Maciej W. Rozycki

Well, Intel's "Multiprocessor Specification" mandates that (see section
3.6.1 and also the compliance list in Appendix C). I does not mandate local APIC IDs to be consecutive though.

Unless I am mistaken, the MP spec does not say that _CPUs_ must start from 0. We had an IO-APIC at 0. The MP spec says that the IDs must be unique (I am told this isn't true any more because an IO APIC and a CPU may have the same ID) and _need not_ be consecutive.

We tried different setups; one had IO APICs at 0,1,2 and CPUs starting at 16. I can't see that this is forbidden (the reason is that the IO-APICs have only 4-bit APIC ID registers). Anyway we changed it now to have both IO-APICs and CPUs start at 0.

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#13 lore raw
Quoted 4 lines · Martin Wilck

Unless I am mistaken, the MP spec does not say that _CPUs_ must start from 0. We had an IO-APIC at 0. The MP spec says that the IDs must be unique (I am told this isn't true any more because an IO APIC and a CPU may have the same ID) and _need not_ be consecutive.

You are unfortunately mistaken -- the spec is explicit about *local* APIC
IDs having to start at 0. There are at least two places in the spec that refer to that.

Quoted 4 lines

We tried different setups; one had IO APICs at 0,1,2 and CPUs starting at 16. I can't see that this is forbidden (the reason is that the IO-APICs have only 4-bit APIC ID registers). Anyway we changed it now to have both IO-APICs and CPUs start at 0.

You can always assign 0, 16, 17, etc. to local APICs and then 1, 2, 3,
etc. for I/O APICs. Frankly I don't know what the actual justification behind the requirement is. Note that the ID of 0 need not necessarily belong to the BSP.

Maciej

#14 lore raw
Quoted 2 lines · Maciej W. Rozycki

You are unfortunately mistaken -- the spec is explicit about *local* APIC
IDs having to start at 0. There are at least two places in the spec that refer to that.

I see. Thanks for the clarification.

Quoted 2 lines

You can always assign 0, 16, 17, etc. to local APICs and then 1, 2, 3,
etc. for I/O APICs.

Actually, that's what we did (I messed thinks up in my posting). I think that's an ugly configuration, though, and totally misleading in terms of system topology.

Quoted 1 line

Frankly I don't know what the actual justification behind the requirement is. Note that the ID of 0 need not necessarily belong to the BSP.

Starting both local APICs and IO-APICs at 0 isa working and aesthetically pleasing solution. We're pursuing that now.

Thanks again, Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#19 lore raw
In reply to Andi Kleen

Hi Andi, hi everyone,

Quoted 7 lines
Quoted 3 lines

The MP_valid_apicid() function [arch/i386/kernel/mpparse.c] checks whether the APIC version field is >=20 in order to determine whether the CPU supports 8-bit physical APIC ids.

Yes, it's broken. ... . Also it's only
a sanity check for broken BIOS, and in this case it causes more problems than it solves.

We have another issue with the APIC IDs: the GET_APIC_ID and APIC_ID_MASK macros from the subarch code. In all subarchs except summit and es7000, these macros use a hard-coded mask 0x0F for the APIC ID. Thus, any APIC ID >15 will be truncated, leading to obscure errors.

We are wondering why these masks are there in the subarch code at all. After all, whether or not 8-bit APIC IDs are supported depends mainly on the CPU type used. Why wouldn't it possible to have a "default" architecture with APIC IDs > 15, if the CPUs allow it?

In other words: What would be broken if we just used an APIC ID mask of 0xFF everywhere?

The current situation with MP_valid_apicid() on the one hand (masking the APIC ID as a function of local APIC version) and APIC_ID_MASK (masking the APIC as a function of subarch) on the other hand is inconsistent. A correct approach must take both CPU and architecture constraints into account, and use a CPU-type-dependent variable mask in the subarch code.

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
#20 lore raw
Quoted 4 lines · Martin Wilck

We are wondering why these masks are there in the subarch code at all. After all, whether or not 8-bit APIC IDs are supported depends mainly on the CPU type used. Why wouldn't it possible to have a "default" architecture with APIC IDs > 15, if the CPUs allow it?

It actually depends on the APIC type, rather than the CPU. E.g. with
Pentium systems the width of the ID is either 4 bits or 8 bits, depending on whether the integrated or an external 82489DX APIC is used. This should be able to be determined by the APIC version; for v <= 0xf the ID is 8-bit and for v >= 0x10 it used to be 4-bit. Now you only need to determine what is the value of v above 0x10 that makes the ID 8-bit again.

Maciej

#21 lore raw

Hi Maciej,

Quoted 7 lines

It actually depends on the APIC type, rather than the CPU. E.g. with Pentium systems the width of the ID is either 4 bits or 8 bits, depending on whether the integrated or an external 82489DX APIC is used. This should be able to be determined by the APIC version; for v <= 0xf the ID is 8-bit and for v >= 0x10 it used to be 4-bit. Now you only need to determine what is the value of v above 0x10 that makes the ID 8-bit again.

That would be v>=0x14 for Intel. But that is wrong for AMD CPUs. The actual Dual-Core Athlon CPUs we have report an APIC version of 0x10. Please refer to the start of this thread.

Anyway, I understand that you agree this does not belong into the subarch code?

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy
author
#25 lore raw
In reply to Martin Wilck
Quoted 2 lines · Martin Wilck

In other words: What would be broken if we just used an APIC ID mask of 0xFF everywhere?

Nothing I think. It's more historical reasons. The physflat subarchitecture patch essentially removed it, but it needs some rework and merging with bigsmp now.

Quoted 6 lines

The current situation with MP_valid_apicid() on the one hand (masking the APIC ID as a function of local APIC version) and APIC_ID_MASK (masking the APIC as a function of subarch) on the other hand is inconsistent. A correct approach must take both CPU and architecture constraints into account, and use a CPU-type-dependent variable mask in the subarch code.

Yes, it's broken right now.

-Andi

#24 lore raw
In reply to a message not in the archive

[ Putting everyone cc again and leaving maciej's reply intact for reference ]

Quoted 18 lines · Maciej W. Rozycki
Quoted 1 line

The ID is 0x14 for Intel. But that is wrong for AMD CPUs. The actual Dual-Core

Why can't hw designers ever get such things right? Sigh...

Quoted 2 lines

Athlon CPUs we have report an APIC version of 0x10. Please refer to the start of this thread.

Frankly, such a change implies a change to the inter-APIC communication
protocol, so the major revision should have been bumped, like it happened for the I/O APIC (0x20); I hope they do not worry of changing the design so many times they run out of numbers! Thus none of the implementers has done their job properly, but for Intel there is at least some change, while for AMD there seems no generic way of determining that -- 0x10 implies a "traditional" integrated APIC as implemented in Pentium in 1995, using a three-wire serial bus for communication...

Perhaps a variable should be added holding the "architectural revision"
of the APIC subsystem, set up by the early CPU/APIC initialization code and used from there on instead of direct references to the version register as implemented in the hardware. It does not have to be based on what hardware uses, e.g.:

- 0: no APIC,

Quoted 8 lines

- 1: 82489DX -- communication using a five-wire inter-APIC bus, 8-bit physical ID, 32-bit logical ID, etc.,

- 2: Pentium-style -- communication using a three-wire inter-APIC bus, 4-bit physical ID, 8-bit logical ID, etc.,

- 3: P4-style -- communication using the system bus, 8-bit physical ID, 8-bit logical ID, etc.

AMD has their HyperTransport bus, which is yet a different thing.

According to what AMD told us, all K8 CPUs support 8-bit APIC IDs ("extended APIC IDs" in their speak), but they are only enabled "when both ApicExtId and ApicExtBrdCst in the HyperTransport Transaction Control Register are set".

As we cannot read PCI config space at APIC configuration time, my suggestion would be something like

unsigned char phys_apic_id_mask(int apic_version)

        if (apic_version < 0x10 || apic_version >= 0x14 ||
		(boot_cpu_data.vendor == X86_VENDOR_AMD &&
         	 boot_cpu_data.family >= 0x0f))
		return 0xff;
	else
		return 0x0f;
}

That would assume that no BIOS is so stupid to set APIC IDs >0xf and at the same time forget to enable the respective bits in the HyperTransport Transaction Control Register.

I do not oversee whether boot_cpu_data is initialized when the APIC IDs need to be tested/masked, but somebody else certainly does.

Quoted 10 lines
Quoted 4 lines

Anyway, I understand that you agree this does not belong into the subarch code?

That's CPU/chipset specific indeed. It shouldn't really depend on the
platform these are used in. Exceptions, if any, can be handled on a case by case basis.

Maciej

Regards
Martin

Signature · 4 lines
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1        mailto:[email protected]
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy