Skip to contentSkip to conversation

x86/irq: switch x2APIC default destination mode

PATCH Superseded Roger Pau Monne 2 participants 10 messages Jun 23, 2022 – Jun 29, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover x86/irq: switch x2APIC default destination mode

Hello,

The following series aims to change the default x2APIC Destination mode from Logical to Physical. This is done in order to cope with boxes that don't have a huge amount of CPUs, but do have a non trivial amount of PCI devices using MSI(-X).

The default x2APIC destination mode can now be set from Kconfig, and will default to phys in order to reliable boot on all boxes.

Further patches are a bit of cleanup related to the interrupt limits reported at boot, and making those values more realistic.

Thanks, Roger.

Roger Pau Monne (6):
x86/Kconfig: add selection of default x2APIC destination mode x86/x2apic: use physical destination mode by default x86/setup: init nr_irqs after having detected x2APIC support x86/irq: fix setting irq limits
x86/irq: print nr_irqs as limit on the number of MSI(-X) interrupts x86/irq: do not set nr_irqs based on nr_irqs_gsi in APIC mode

 docs/misc/xen-command-line.pandoc |  5 ++---
 xen/arch/x86/Kconfig              | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/genapic/x2apic.c     |  6 ++++--
 xen/arch/x86/include/asm/apic.h   |  2 ++
 xen/arch/x86/io_apic.c            | 10 ----------
 xen/arch/x86/irq.c                | 15 +++++++++++++++
 xen/arch/x86/mpparse.c            |  5 +++++
 7 files changed, 57 insertions(+), 15 deletions(-)
Signature · 1 line
2.36.1
#1 lore raw

PATCH 1/6 x86/Kconfig: add selection of default x2APIC destination mode

Allow selecting the default x2APIC destination mode from Kconfig. Note the default destination mode is still Logical (Cluster) mode.

---
 xen/arch/x86/Kconfig          | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/genapic/x2apic.c |  6 ++++--
 2 files changed, 33 insertions(+), 2 deletions(-)
xen/arch/x86/Kconfig +29 -0
index 1e31edc99f..f560dc13f4 100644
@@ -226,6 +226,35 @@ config XEN_ALIGN_2M
 
 endchoice
 
+choice
+	prompt "x2APIC default destination mode"
+	default X2APIC_LOGICAL
+	---help---
+	  Specify default destination mode for x2APIC.
+
+	  If unsure, choose "Logical".
+
+config X2APIC_LOGICAL
+	bool "Logical mode"
+	---help---
+	  Use Logical Destination mode.
+
+	  When using this mode APICs are addressed using the Logical
+	  Destination mode, which allows for optimized IPI sending,
+	  but also reduces the amount of vectors available for external
+	  interrupts when compared to physical mode.
+
+config X2APIC_PHYS
+	bool "Physical mode"
+	---help---
+	  Use Physical Destination mode.
+
+	  When using this mode APICs are addressed using the Physical
+	  Destination mode, which allows using all dynamic vectors on
+	  each CPU independently.
+
+endchoice
+
 config GUEST
 	bool
 
xen/arch/x86/genapic/x2apic.c +4 -2
index de5032f202..4b9bbe2f3e 100644
@@ -228,7 +228,7 @@ static struct notifier_block x2apic_cpu_nfb = {
    .notifier_call = update_clusterinfo
 };
 
-static s8 __initdata x2apic_phys = -1; /* By default we use logical cluster mode. */
+static int8_t __initdata x2apic_phys = -1;
 boolean_param("x2apic_phys", x2apic_phys);
 
 const struct genapic *__init apic_x2apic_probe(void)
@@ -241,7 +241,9 @@ const struct genapic *__init apic_x2apic_probe(void)
          * the usage of the high 16 bits to hold the cluster ID.
          */
         x2apic_phys = !iommu_intremap ||
-                      (acpi_gbl_FADT.flags & ACPI_FADT_APIC_PHYSICAL);
+                      (acpi_gbl_FADT.flags & ACPI_FADT_APIC_PHYSICAL) ||
+                      (IS_ENABLED(CONFIG_X2APIC_PHYS) &&
+                       !(acpi_gbl_FADT.flags & ACPI_FADT_APIC_CLUSTER));
     }
     else if ( !x2apic_phys )
         switch ( iommu_intremap )
Signature · 1 line
2.36.1
#2 lore raw

Re: [PATCH 1/6] x86/Kconfig: add selection of default x2APIC destination mode

Quoted 14 lines · Roger Pau Monne

Allow selecting the default x2APIC destination mode from Kconfig. Note the default destination mode is still Logical (Cluster) mode.

---
 xen/arch/x86/Kconfig          | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/genapic/x2apic.c |  6 ++++--
 2 files changed, 33 insertions(+), 2 deletions(-)
xen/arch/x86/Kconfig +0 -0
index 1e31edc99f..f560dc13f4 100644

xen/arch/x86/Kconfig config XEN_ALIGN_2M · line 230

Quoted 5 lines
xen/arch/x86/Kconfig +2 -0
@@ -226,6 +226,35 @@ config XEN_ALIGN_2M
 
 endchoice
 
+choice
+	prompt "x2APIC default destination mode"

What's the point of using "choice" here, and not a single "bool"?

xen/arch/x86/Kconfig config XEN_ALIGN_2M · line 232

Quoted 2 lines
xen/arch/x86/Kconfig +2 -0
+	default X2APIC_LOGICAL
+	---help---

Nit: Please don't use ---help--- anymore - we're trying to phase out its use as Linux has dropped it altogether (and hence once we update our Kconfig, we'd like to change as few places as possible), leaving just "help".

One downside of "choice" (iirc) is that the individual sub-options' help text is inaccessible from at least the command line version of kconfig.

Quoted 5 lines
xen/arch/x86/Kconfig +5 -0
+	  Specify default destination mode for x2APIC.
+
+	  If unsure, choose "Logical".
+
+config X2APIC_LOGICAL

xen/arch/x86/Kconfig config XEN_ALIGN_2M · line 247

Quoted 10 lines
xen/arch/x86/Kconfig +10 -0
+	bool "Logical mode"
+	---help---
+	  Use Logical Destination mode.
+
+	  When using this mode APICs are addressed using the Logical
+	  Destination mode, which allows for optimized IPI sending,
+	  but also reduces the amount of vectors available for external
+	  interrupts when compared to physical mode.
+
+config X2APIC_PHYS

X2APIC_PHYSICAL (to be in line with X2APIC_LOGICAL)?

Jan

#3 lore raw

Re: [PATCH 1/6] x86/Kconfig: add selection of default x2APIC destination mode

Quoted 16 lines · Jan Beulich
Quoted 15 lines · Roger Pau Monne

Allow selecting the default x2APIC destination mode from Kconfig. Note the default destination mode is still Logical (Cluster) mode.

---
 xen/arch/x86/Kconfig          | 29 +++++++++++++++++++++++++++++
 xen/arch/x86/genapic/x2apic.c |  6 ++++--
 2 files changed, 33 insertions(+), 2 deletions(-)
xen/arch/x86/Kconfig +0 -0
index 1e31edc99f..f560dc13f4 100644
@@ -226,6 +226,35 @@ config XEN_ALIGN_2M
 
Quoted 6 lines

xen/arch/x86/Kconfig config XEN_ALIGN_2M · line 230

Quoted 4 lines
xen/arch/x86/Kconfig +2 -0
 endchoice
 
+choice
+	prompt "x2APIC default destination mode"

What's the point of using "choice" here, and not a single "bool"?

I think choice better reflects the purpose of the option, it's selecting between two different modes. It could be expressed with a bool, but I think it's less clear.

Quoted 10 lines

xen/arch/x86/Kconfig config XEN_ALIGN_2M · line 232

Quoted 2 lines
xen/arch/x86/Kconfig +2 -0
+	default X2APIC_LOGICAL
+	---help---

Nit: Please don't use ---help--- anymore - we're trying to phase out its use as Linux has dropped it altogether (and hence once we update our Kconfig, we'd like to change as few places as possible), leaving just "help".

One downside of "choice" (iirc) is that the individual sub-options' help text is inaccessible from at least the command line version of kconfig.

Hm, I usually use menuconfig when wanting to poke at options help.

I guess I could introduce a single X2APIC_PHYSICAL bool that starts with default false and notes that otherwise the destination mode is logical.

Quoted 9 lines
Quoted 9 lines
xen/arch/x86/Kconfig +9 -0
+	  Specify default destination mode for x2APIC.
+
+	  If unsure, choose "Logical".
+
+config X2APIC_LOGICAL
+	bool "Logical mode"
+	---help---
+	  Use Logical Destination mode.
+
Quoted 8 lines

xen/arch/x86/Kconfig config XEN_ALIGN_2M · line 247

Quoted 6 lines
xen/arch/x86/Kconfig +6 -0
+	  When using this mode APICs are addressed using the Logical
+	  Destination mode, which allows for optimized IPI sending,
+	  but also reduces the amount of vectors available for external
+	  interrupts when compared to physical mode.
+
+config X2APIC_PHYS

X2APIC_PHYSICAL (to be in line with X2APIC_LOGICAL)?

Right, was about to expand it but did consider PHYS to be clear enough (opposed to using LOG or LOGIC), will expand in next version.

Thanks, Roger.

#4 lore raw

PATCH 2/6 x86/x2apic: use physical destination mode by default

In reply to Roger Pau Monne

Using cluster mode by default greatly limits the amount of vectors available, as then vector space is shared amongst all the CPUs in the logical cluster.

This can lead to vector shortage issues on boxes with not a huge amount of CPUs but with a non-trivial amount of devices, there are reports of boxes with 32 CPUs (2 logical clusters, and thus only 414 dynamic vectors) that run out of vectors and fail to setup interrupts for dom0.

This could be considered as a regression when switching from xAPIC mode, as when using xAPIC only physical mode is supported.

Switch default Kconfig selection to use x2APIC physical mode.

---
 docs/misc/xen-command-line.pandoc | 5 ++---
 xen/arch/x86/Kconfig              | 4 ++--
 2 files changed, 4 insertions(+), 5 deletions(-)
docs/misc/xen-command-line.pandoc +2 -3
index a92b7d228c..952874c4f4 100644
@@ -2646,11 +2646,10 @@ Permit use of x2apic setup for SMP environments.
 ### x2apic_phys (x86)
 > `= <boolean>`
 
-> Default: `true` if **FADT** mandates physical mode or if interrupt remapping
->          is not available, `false` otherwise.
+> Default: `false` if **FADT** mandates cluster mode, `true` otherwise.
 
 In the case that x2apic is in use, this option switches between physical and
-clustered mode.  The default, given no hint from the **FADT**, is cluster
+clustered mode.  The default, given no hint from the **FADT**, is physical
 mode.
 
 ### xenheap_megabytes (arm32)
xen/arch/x86/Kconfig +2 -2
index f560dc13f4..74bfb37db4 100644
@@ -228,11 +228,11 @@ endchoice
 
 choice
 	prompt "x2APIC default destination mode"
-	default X2APIC_LOGICAL
+	default X2APIC_PHYS
 	---help---
 	  Specify default destination mode for x2APIC.
 
-	  If unsure, choose "Logical".
+	  If unsure, choose "Physical".
 
 config X2APIC_LOGICAL
 	bool "Logical mode"
Signature · 1 line
2.36.1
#5 lore raw

Re: [PATCH 2/6] x86/x2apic: use physical destination mode by default

Quoted 4 lines · Roger Pau Monne

Using cluster mode by default greatly limits the amount of vectors available, as then vector space is shared amongst all the CPUs in the logical cluster.

Quoted 8 lines

This can lead to vector shortage issues on boxes with not a huge amount of CPUs but with a non-trivial amount of devices, there are reports of boxes with 32 CPUs (2 logical clusters, and thus only 414 dynamic vectors) that run out of vectors and fail to setup interrupts for dom0.

This could be considered as a regression when switching from xAPIC mode, as when using xAPIC only physical mode is supported.

When using more than 8 CPUs.

You also don't mention the downside (higher IPI send effort) at all.

Jan

#6 lore raw

PATCH 3/6 x86/setup: init nr_irqs after having detected x2APIC support

In reply to Roger Pau Monne

Logic in ioapic_init() that sets the number of available vectors for external interrupts requires knowing the x2APIC Destination Mode. As such move the call after x2APIC BSP setup.

Do it as part of init_irq_data(), which is called just after x2APIC BSP init and also makes use of nr_irqs itself.

No functional change intended.

---
 xen/arch/x86/io_apic.c | 10 ----------
 xen/arch/x86/irq.c     | 10 ++++++++++
 2 files changed, 10 insertions(+), 10 deletions(-)
xen/arch/x86/io_apic.c +0 -10
index c086f40f63..8d4923ba9a 100644
@@ -2653,16 +2653,6 @@ void __init ioapic_init(void)
                max_gsi_irqs, nr_irqs_gsi);
         nr_irqs_gsi = max_gsi_irqs;
     }
-
-    if ( nr_irqs == 0 )
-        nr_irqs = cpu_has_apic ?
-                  max(0U + num_present_cpus() * NR_DYNAMIC_VECTORS,
-                      8 * nr_irqs_gsi) :
-                  nr_irqs_gsi;
-    else if ( nr_irqs < 16 )
-        nr_irqs = 16;
-    printk(XENLOG_INFO "IRQ limits: %u GSI, %u MSI/MSI-X\n",
-           nr_irqs_gsi, nr_irqs - nr_irqs_gsi);
 }
 
 unsigned int arch_hwdom_irqs(domid_t domid)
xen/arch/x86/irq.c +10 -0
index de30ee7779..b51e25f696 100644
@@ -420,6 +420,16 @@ int __init init_irq_data(void)
     struct irq_desc *desc;
     int irq, vector;
 
+    if ( nr_irqs == 0 )
+        nr_irqs = cpu_has_apic ? max(0U + num_present_cpus() *
+                                     NR_DYNAMIC_VECTORS, 8 * nr_irqs_gsi)
+                               : nr_irqs_gsi;
+    else if ( nr_irqs < 16 )
+        nr_irqs = 16;
+
+    printk(XENLOG_INFO "IRQ limits: %u GSI, %u MSI/MSI-X\n",
+           nr_irqs_gsi, nr_irqs - nr_irqs_gsi);
+
     for ( vector = 0; vector < X86_NR_VECTORS; ++vector )
         this_cpu(vector_irq)[vector] = INT_MIN;
 
Signature · 1 line
2.36.1
#7 lore raw

PATCH 4/6 x86/irq: fix setting irq limits

In reply to Roger Pau Monne

Current code to calculate nr_irqs assumes the APIC destination mode to be physical, so all vectors on each possible CPU is available for use by a different interrupt source. This is not true when using Logical (Cluster) destination mode, where CPUs in the same cluster share the vector space.

Fix by calculating the maximum Cluster ID and use it to derive the number of clusters in the system. Note the code assumes Cluster IDs to be contiguous, or else we will set nr_irqs to a number higher than the real amount of vectors (still not fatal).

The number of clusters is then used instead of the number of present CPUs when calculating the value of nr_irqs.

---
 xen/arch/x86/genapic/x2apic.c   |  2 +-
 xen/arch/x86/include/asm/apic.h |  2 ++
 xen/arch/x86/irq.c              | 10 ++++++++--
 xen/arch/x86/mpparse.c          |  5 +++++
 4 files changed, 16 insertions(+), 3 deletions(-)
xen/arch/x86/genapic/x2apic.c +1 -1
index 4b9bbe2f3e..cd1f55cad8 100644
@@ -228,7 +228,7 @@ static struct notifier_block x2apic_cpu_nfb = {
    .notifier_call = update_clusterinfo
 };
 
-static int8_t __initdata x2apic_phys = -1;
+int8_t __initdata x2apic_phys = -1;
 boolean_param("x2apic_phys", x2apic_phys);
 
 const struct genapic *__init apic_x2apic_probe(void)
xen/arch/x86/include/asm/apic.h +2 -0
index 7625c0ecd6..6060628836 100644
@@ -27,6 +27,8 @@ enum apic_mode {
 extern bool iommu_x2apic_enabled;
 extern u8 apic_verbosity;
 extern bool directed_eoi_enabled;
+extern uint16_t x2apic_max_cluster_id;
+extern int8_t x2apic_phys;
 
 void check_x2apic_preenabled(void);
 void x2apic_bsp_setup(void);
xen/arch/x86/irq.c +8 -2
index b51e25f696..b64d18c450 100644
@@ -421,9 +421,15 @@ int __init init_irq_data(void)
     int irq, vector;
 
     if ( nr_irqs == 0 )
-        nr_irqs = cpu_has_apic ? max(0U + num_present_cpus() *
-                                     NR_DYNAMIC_VECTORS, 8 * nr_irqs_gsi)
+    {
+        unsigned int vec_spaces =
+            (x2apic_enabled && !x2apic_phys) ? x2apic_max_cluster_id + 1
+                                             : num_present_cpus();
+
+        nr_irqs = cpu_has_apic ? max(vec_spaces * NR_DYNAMIC_VECTORS,
+                                     8 * nr_irqs_gsi)
                                : nr_irqs_gsi;
+    }
     else if ( nr_irqs < 16 )
         nr_irqs = 16;
 
xen/arch/x86/mpparse.c +5 -0
index d8ccab2449..dc112bffc7 100644
@@ -131,6 +131,8 @@ static int __init mpf_checksum(unsigned char *mp, int len)
 	return sum & 0xFF;
 }
 
+uint16_t __initdata x2apic_max_cluster_id;
+
 /* Return xen's logical cpu_id of the new added cpu or <0 if error */
 static int MP_processor_info_x(struct mpc_config_processor *m,
 			       u32 apicid, bool hotplug)
@@ -199,6 +201,9 @@ static int MP_processor_info_x(struct mpc_config_processor *m,
 		def_to_bigsmp = true;
 	}
 
+	x2apic_max_cluster_id = max(x2apic_max_cluster_id,
+				    (uint16_t)(apicid >> 4));
+
 	return cpu;
 }
 
Signature · 1 line
2.36.1
#8 lore raw

PATCH 5/6 x86/irq: print nr_irqs as limit on the number of MSI(-X) interrupts

In reply to Roger Pau Monne

Using nr_irqs minus nr_irqs_gsi is misleading, as GSI interrupts are not allocated unless requested by the hardware domain, so a hardware domain could not use any GSI (or just one for the ACPI SCI), and hence (almost) all nr_irqs will be available for MSI(-X) usage.

No functional difference, just affects the printed message.

---
 xen/arch/x86/irq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/x86/irq.c +1 -1
index b64d18c450..7f75ec8bcc 100644
@@ -434,7 +434,7 @@ int __init init_irq_data(void)
         nr_irqs = 16;
 
     printk(XENLOG_INFO "IRQ limits: %u GSI, %u MSI/MSI-X\n",
-           nr_irqs_gsi, nr_irqs - nr_irqs_gsi);
+           nr_irqs_gsi, nr_irqs);
 
     for ( vector = 0; vector < X86_NR_VECTORS; ++vector )
         this_cpu(vector_irq)[vector] = INT_MIN;
Signature · 1 line
2.36.1
#9 lore raw

PATCH 6/6 x86/irq: do not set nr_irqs based on nr_irqs_gsi in APIC mode

In reply to Roger Pau Monne

When using an APIC do not set nr_irqs based on a factor of nr_irqs_gsi (currently x8), and instead do so exclusively based on the amount of available vectors on the system.

There's no point in setting nr_irqs to a value higher than the available set of vectors, as vector allocation will fail anyway.

Fixes e99d45da8a ('x86/x2apic: properly implement cluster mode')
---
 xen/arch/x86/irq.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
xen/arch/x86/irq.c +1 -2
index 7f75ec8bcc..e3b0bee527 100644
@@ -426,8 +426,7 @@ int __init init_irq_data(void)
             (x2apic_enabled && !x2apic_phys) ? x2apic_max_cluster_id + 1
                                              : num_present_cpus();
 
-        nr_irqs = cpu_has_apic ? max(vec_spaces * NR_DYNAMIC_VECTORS,
-                                     8 * nr_irqs_gsi)
+        nr_irqs = cpu_has_apic ? vec_spaces * NR_DYNAMIC_VECTORS
                                : nr_irqs_gsi;
     }
     else if ( nr_irqs < 16 )
Signature · 1 line
2.36.1