Skip to contentSkip to conversation

x86_64: MAX_LOCAL_APIC way too big?

DISCUSSION Denys Vlasenko 3 participants 4 messages Sep 25, 2015 – Sep 26, 2015

lore
Share & tools
lore
#0 lore raw

For 64-bit kernels, MAX_LOCAL_APIC is 32k:

#ifdef CONFIG_X86_32
...
#else

# define MAX_IO_APICS 128
# define MAX_LOCAL_APIC 32768
#endif

(It seems to be a bit of a misnomer, it's not a maximum number of APICs we support, it's the highest APIC _id_ we support.)

32 thousand APICs? That's a lot. Especially considering that event with CONFIG_MAXSMP=y, NR_CPUS is "only" 8096.

After a quick glance through code, it looks like such a big value causes several data arrays to be quite oversized:

phys_cpu_present_map is 4 kbytes (one bit per apicid) __apicid_to_node is 64 kbytes
apic_version is 128 kbytes (!!!)

This is where they are defined:

include/asm/mpspec.h

	#define PHYSID_ARRAY_SIZE       BITS_TO_LONGS(MAX_LOCAL_APIC)
	struct physid_mask {
	        unsigned long mask[PHYSID_ARRAY_SIZE];
	};
	typedef struct physid_mask physid_mask_t;
apic/apic.c
	physid_mask_t phys_cpu_present_map;

mm/numa.c:

	s16 __apicid_to_node[MAX_LOCAL_APIC] = {
	        [0 ... MAX_LOCAL_APIC-1] = NUMA_NO_NODE
	};

apic/apic.c:

	int apic_version[MAX_LOCAL_APIC];

Maybe we can reduce MAX_LOCAL_APIC?
Why it has to be this big in the first place?

IIRC: APIC id at first was just a 8-bit quantity, then x2apic mode it was extended to 32 bits.

On "usual" systems, apic ids simply go from zero to maximum logical CPU number, mirroring CPU ids.

I imagine there are broken/unusual systems in the wild where apic ids are non-contiguous for some reason.
IOW: even if NR_CPUS is low (say, 4), setting MAX_LOCAL_APIC ot 4 is not a good idea.

I propose the following:

/* For small SMP, allow non-contiguous APIC IDs.
* We expect that big SMP machines have
* non-broken APIC id numbering.
*/

# define MAX_LOCAL_APIC (NR_CPUS > 256 ? NR_CPUS : 256)

If this looks too risky, then how about this?

# define MAX_LOCAL_APIC (NR_CPUS > 256/2 ? NR_CPUS*2 : 256)

Why 256? We have this code in two places:

                apic_version[new_apicid] =
                         GET_APIC_VERSION(apic_read(APIC_LVR));
and
	#define GET_APIC_VERSION(x)     ((x) & 0xFFu)

IOW: apic_version[x] needs to be valid for any 8-bit x.

In one place, we need to add a check that index is not out-of-bounds:

	int generic_processor_info(int apicid, int version)
	{
		...
	        apic_version[apicid] = version;

MAX_IO_APICS is a bit big too, this ioapics[] array uses 9216 bytes:

apic/io_apic.c

	static struct ioapic {
	        /*
	         * # of IRQ routing registers
	         */
	        int nr_registers;
	        /*
	         * Saved state during suspend/resume, or while enabling intr-remap.
	         */
	        struct IO_APIC_route_entry *saved_registers;
	        /* I/O APIC config */
	        struct mpc_ioapic mp_config;
	        /* IO APIC gsi routing info */
	        struct mp_ioapic_gsi  gsi_config;
	        struct ioapic_domain_cfg irqdomain_cfg;
	        struct irq_domain *irqdomain;
	        struct resource *iomem_res;
	} ioapics[MAX_IO_APICS];

Do we really expect to have machines with 128 IOAPICS? We use this value even if NR_CPUS is, say, only 4.

I propose:

/* Minimum is 8.
* For largish NR_CPUS, we expect to have fewer IOAPICS than CPUs.
* No matter how big NR_CPUS is, cap IOAPICs count at 128. */

# define MAX_IO_APICS (NR_CPUS < 8 ? 8 : (NR_CPUS < 128 ? NR_CPUS : 128))

FYI: MAX_LOCAL_APIC was changed by this commit:

commit a65d1d644c2b65bfb99e766e7160d764b8b2bfa4

Author: Jack Steiner <[email protected]>
Date:   Fri Mar 28 14:12:08 2008 -0500

x86: increase size of APICID

Increase the number of bits in an apicid from 8 to 32.

By default, MP_processor_info() gets the APICID from the mpc_config_processor structure. However, this structure limits the size of APICID to 8 bits. This patch allows the caller of MP_processor_info() to optionally pass a larger APICID that will be used instead of the one in the mpc_config_processor struct.
...

#else

 # define MAX_IO_APICS 128
-# define MAX_LOCAL_APIC 256
+# define MAX_LOCAL_APIC 32768
 #endif
#1 lore raw
Quoted 45 lines · Denys Vlasenko

For 64-bit kernels, MAX_LOCAL_APIC is 32k:

#ifdef CONFIG_X86_32
...
#else

# define MAX_IO_APICS 128
# define MAX_LOCAL_APIC 32768
#endif

(It seems to be a bit of a misnomer, it's not a maximum number of APICs we support, it's the highest APIC _id_ we support.)

32 thousand APICs? That's a lot. Especially considering that event with CONFIG_MAXSMP=y, NR_CPUS is "only" 8096.

After a quick glance through code, it looks like such a big value causes several data arrays to be quite oversized:

phys_cpu_present_map is 4 kbytes (one bit per apicid) __apicid_to_node is 64 kbytes
apic_version is 128 kbytes (!!!)

This is where they are defined:

include/asm/mpspec.h

	#define PHYSID_ARRAY_SIZE       BITS_TO_LONGS(MAX_LOCAL_APIC)
	struct physid_mask {
	        unsigned long mask[PHYSID_ARRAY_SIZE];
	};
	typedef struct physid_mask physid_mask_t;
apic/apic.c
	physid_mask_t phys_cpu_present_map;

mm/numa.c:

	s16 __apicid_to_node[MAX_LOCAL_APIC] = {
	        [0 ... MAX_LOCAL_APIC-1] = NUMA_NO_NODE
	};

apic/apic.c:

	int apic_version[MAX_LOCAL_APIC];
Quoted 8 lines

Maybe we can reduce MAX_LOCAL_APIC?
Why it has to be this big in the first place?

IIRC: APIC id at first was just a 8-bit quantity, then x2apic mode it was extended to 32 bits.

On "usual" systems, apic ids simply go from zero to maximum logical CPU number, mirroring CPU ids.

Hi Denys,
The above assumption is risky with modern
x86 platforms. APIC ids are assigned by firmware,
and may be discrete.

Quoted 19 lines

I imagine there are broken/unusual systems in the wild where apic ids are non-contiguous for some reason.
IOW: even if NR_CPUS is low (say, 4), setting MAX_LOCAL_APIC ot 4 is not a good idea.

I propose the following:

/* For small SMP, allow non-contiguous APIC IDs.
* We expect that big SMP machines have
* non-broken APIC id numbering.
*/

# define MAX_LOCAL_APIC (NR_CPUS > 256 ? NR_CPUS : 256)

If this looks too risky, then how about this?

# define MAX_LOCAL_APIC (NR_CPUS > 256/2 ? NR_CPUS*2 : 256)
Quoted 6 lines

Why 256? We have this code in two places:

                apic_version[new_apicid] =
                         GET_APIC_VERSION(apic_read(APIC_LVR));
and
	#define GET_APIC_VERSION(x)     ((x) & 0xFFu)

IOW: apic_version[x] needs to be valid for any 8-bit x.

The above code just assume that the value stored in apic_version[] are 8-bit. It has no explicit relationship with apic id. Thanks!
Gerry

Quoted 66 lines

In one place, we need to add a check that index is not out-of-bounds:

	int generic_processor_info(int apicid, int version)
	{
		...
	        apic_version[apicid] = version;

MAX_IO_APICS is a bit big too, this ioapics[] array uses 9216 bytes:

apic/io_apic.c

	static struct ioapic {
	        /*
	         * # of IRQ routing registers
	         */
	        int nr_registers;
	        /*
	         * Saved state during suspend/resume, or while enabling intr-remap.
	         */
	        struct IO_APIC_route_entry *saved_registers;
	        /* I/O APIC config */
	        struct mpc_ioapic mp_config;
	        /* IO APIC gsi routing info */
	        struct mp_ioapic_gsi  gsi_config;
	        struct ioapic_domain_cfg irqdomain_cfg;
	        struct irq_domain *irqdomain;
	        struct resource *iomem_res;
	} ioapics[MAX_IO_APICS];

Do we really expect to have machines with 128 IOAPICS? We use this value even if NR_CPUS is, say, only 4.

I propose:

/* Minimum is 8.
* For largish NR_CPUS, we expect to have fewer IOAPICS than CPUs.
* No matter how big NR_CPUS is, cap IOAPICs count at 128. */

# define MAX_IO_APICS (NR_CPUS < 8 ? 8 : (NR_CPUS < 128 ? NR_CPUS : 128))

FYI: MAX_LOCAL_APIC was changed by this commit:

commit a65d1d644c2b65bfb99e766e7160d764b8b2bfa4

Author: Jack Steiner <[email protected]>
Date:   Fri Mar 28 14:12:08 2008 -0500

x86: increase size of APICID

Increase the number of bits in an apicid from 8 to 32.

By default, MP_processor_info() gets the APICID from the mpc_config_processor structure. However, this structure limits the size of APICID to 8 bits. This patch allows the caller of MP_processor_info() to optionally pass a larger APICID that will be used instead of the one in the mpc_config_processor struct.
...

#else

 # define MAX_IO_APICS 128
(quoted diff) +1 -1
-# define MAX_LOCAL_APIC 256
+# define MAX_LOCAL_APIC 32768
 #endif
#2 lore raw
Quoted 13 lines · Jiang Liu
Quoted 12 lines · Denys Vlasenko

32 thousand APICs? That's a lot. Especially considering that event with CONFIG_MAXSMP=y, NR_CPUS is "only" 8096.

After a quick glance through code, it looks like such a big value causes several data arrays to be quite oversized:

phys_cpu_present_map is 4 kbytes (one bit per apicid) __apicid_to_node is 64 kbytes
apic_version is 128 kbytes (!!!)

...

Quoted 6 lines
Quoted 6 lines

Maybe we can reduce MAX_LOCAL_APIC?
Why it has to be this big in the first place?

IIRC: APIC id at first was just a 8-bit quantity, then x2apic mode it was extended to 32 bits.

Quoted 8 lines
Quoted 2 lines

On "usual" systems, apic ids simply go from zero to maximum logical CPU number, mirroring CPU ids.

Hi Denys,
The above assumption is risky with modern
x86 platforms. APIC ids are assigned by firmware,
and may be discrete.

I guess it means a new CONFIG option will be needed then, for such machines. I'll send an RFC patch now.

#3 lore raw
In reply to Denys Vlasenko
Quoted 5 lines · Denys Vlasenko

For 64-bit kernels, MAX_LOCAL_APIC is 32k:

#ifdef CONFIG_X86_32
...
#else

Quoted 7 lines
# define MAX_IO_APICS 128
# define MAX_LOCAL_APIC 32768
#endif

(It seems to be a bit of a misnomer, it's not a maximum number of APICs we support, it's the highest APIC _id_ we support.)

Gah, that's a pretty serious misnomer - so it should be fixed regardless of the outcome of the sizing discussion.

Thanks,

Ingo