Skip to contentSkip to conversation

x86/apic: improvements to disconnect_bsp_APIC

PATCH In review Roger Pau Monne 3 participants 8 messages Jan 23, 2020 – Jan 27, 2020

lore
Share & tools
lore
#0 lore raw

v2 cover x86/apic: improvements to disconnect_bsp_APIC

Hello,

The series contain some improvements to disconnect_bsp_APIC, which started as a way to fix the "APIC error on CPU0: 40(00)" error printed by some Intel boxes on reboot or shutdown. First patch is the fix for the error, second patch is a cleanup.

Roger Pau Monne (2):
x86/apic: fix disabling LVT0 in disconnect_bsp_APIC x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}

 xen/arch/x86/apic.c | 21 ++++-----------------
 1 file changed, 4 insertions(+), 17 deletions(-)
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v2 1/2 x86/apic: fix disabling LVT0 in disconnect_bsp_APIC

The Intel SDM states:

"When an illegal vector value (0 to 15) is written to a LVT entry and the delivery mode is Fixed (bits 8-11 equal 0), the APIC may signal an illegal vector error, without regard to whether the mask bit is set or whether an interrupt is actually seen on the input."

And that's exactly what's currently done in disconnect_bsp_APIC when virt_wire_setup is true and LVT LINT0 is being masked. By writing only APIC_LVT_MASKED Xen is actually setting the vector to 0 and the delivery mode to Fixed (0), and hence it triggers an APIC error even when the LVT entry is masked.

This would usually manifest when Xen is being shut down, as that's where disconnect_bsp_APIC is called:

(XEN) APIC error on CPU0: 40(00)

Fix this by calling clear_local_APIC prior to setting the LVT LINT registers which already clear LVT LINT0, and hence the troublesome write can be avoided as the register is already cleared.

---
Changes since v1:
- Use clear_local_APIC in order to clear LINT0.

---
 xen/arch/x86/apic.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
xen/arch/x86/apic.c +2 -4
index a6a7754d77..508b1586f2 100644
@@ -262,6 +262,8 @@ void disconnect_bsp_APIC(int virt_wire_setup)
         /* Go back to Virtual Wire compatibility mode */
         unsigned long value;
 
+        clear_local_APIC();
+
         /* For the spurious interrupt use vector F, and enable it */
         value = apic_read(APIC_SPIV);
         value &= ~APIC_VECTOR_MASK;
@@ -279,10 +281,6 @@ void disconnect_bsp_APIC(int virt_wire_setup)
             value = SET_APIC_DELIVERY_MODE(value, APIC_MODE_EXTINT);
             apic_write(APIC_LVT0, value);
         }
-        else {
-            /* Disable LVT0 */
-            apic_write(APIC_LVT0, APIC_LVT_MASKED);
-        }
 
         /* For LVT1 make it edge triggered, active high, nmi and enabled */
         value = apic_read(APIC_LVT1);
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v2 1/2
#2 lore raw

Re: [Xen-devel] [PATCH v2 1/2] x86/apic: fix disabling LVT0 in disconnect_bsp_APIC

Quoted 16 lines · Roger Pau Monne

The Intel SDM states:

"When an illegal vector value (0 to 15) is written to a LVT entry and the delivery mode is Fixed (bits 8-11 equal 0), the APIC may signal an illegal vector error, without regard to whether the mask bit is set or whether an interrupt is actually seen on the input."

And that's exactly what's currently done in disconnect_bsp_APIC when virt_wire_setup is true and LVT LINT0 is being masked. By writing only APIC_LVT_MASKED Xen is actually setting the vector to 0 and the delivery mode to Fixed (0), and hence it triggers an APIC error even when the LVT entry is masked.

This would usually manifest when Xen is being shut down, as that's where disconnect_bsp_APIC is called:

Quoted 8 lines

(XEN) APIC error on CPU0: 40(00)

Fix this by calling clear_local_APIC prior to setting the LVT LINT registers which already clear LVT LINT0, and hence the troublesome write can be avoided as the register is already cleared.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

PATCH v2 2/2 x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}

In reply to Roger Pau Monne

There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.

---
 xen/arch/x86/apic.c | 15 ++-------------
 1 file changed, 2 insertions(+), 13 deletions(-)
xen/arch/x86/apic.c +2 -13
index 508b1586f2..c18314c1a3 100644
@@ -273,23 +273,12 @@ void disconnect_bsp_APIC(int virt_wire_setup)
 
         if (!virt_wire_setup) {
             /* For LVT0 make it edge triggered, active high, external and enabled */
-            value = apic_read(APIC_LVT0);
-            value &= ~(APIC_MODE_MASK | APIC_SEND_PENDING |
-                       APIC_INPUT_POLARITY | APIC_LVT_REMOTE_IRR |
-                       APIC_LVT_LEVEL_TRIGGER | APIC_LVT_MASKED );
-            value |= APIC_LVT_REMOTE_IRR | APIC_SEND_PENDING;
-            value = SET_APIC_DELIVERY_MODE(value, APIC_MODE_EXTINT);
+            value = APIC_LVT_REMOTE_IRR | APIC_SEND_PENDING | APIC_DM_EXTINT;
             apic_write(APIC_LVT0, value);
         }
 
         /* For LVT1 make it edge triggered, active high, nmi and enabled */
-        value = apic_read(APIC_LVT1);
-        value &= ~(
-            APIC_MODE_MASK | APIC_SEND_PENDING |
-            APIC_INPUT_POLARITY | APIC_LVT_REMOTE_IRR |
-            APIC_LVT_LEVEL_TRIGGER | APIC_LVT_MASKED);
-        value |= APIC_LVT_REMOTE_IRR | APIC_SEND_PENDING;
-        value = SET_APIC_DELIVERY_MODE(value, APIC_MODE_NMI);
+        value = APIC_LVT_REMOTE_IRR | APIC_SEND_PENDING | APIC_DM_NMI;
         apic_write(APIC_LVT1, value);
     }
 }
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}

Quoted 4 lines · Roger Pau Monne

There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.

The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}

Quoted 11 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monne

There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.

The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?

I can tell you what the Linux feedback will be.

This is not a fastpath.  Always do RMW, because life is too short to keep on unbreaking hardware which makes such assumptions.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}

Quoted 13 lines · Andrew Cooper
Quoted 11 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monne

There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.

The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?

Quoted 4 lines

I can tell you what the Linux feedback will be.

This is not a fastpath.  Always do RMW, because life is too short to keep on unbreaking hardware which makes such assumptions.

We already do such kinds of direct writes to LVT{0/1}, see clear_local_APIC where Xen clears the registers by directly writing APIC_LVT_MASKED to them (no RMW). As the modified code follows a call to clear_local_APIC there's nothing to preserve at this point.

Note that init_bsp_APIC and smp_callin also call clear_local_APIC, so there's no data in those registers that could survive (regardless of my added call to clear_local_APIC in the previous patch).

I'm not arguing this is correct (not sure it's incorrect either), but given how things are handled ATM it seems quite dumb to do a RMW in
disconnect_bsp_APIC: it gives the false impression Xen is preserving something, when the register has already been wiped at startup.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}

Quoted 22 lines · Roger Pau Monné
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monne

There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.

The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?

Quoted 4 lines

I can tell you what the Linux feedback will be.

This is not a fastpath.  Always do RMW, because life is too short to keep on unbreaking hardware which makes such assumptions.

We already do such kinds of direct writes to LVT{0/1}, see clear_local_APIC where Xen clears the registers by directly writing APIC_LVT_MASKED to them (no RMW). As the modified code follows a call to clear_local_APIC there's nothing to preserve at this point.

Quoted 8 lines

Note that init_bsp_APIC and smp_callin also call clear_local_APIC, so there's no data in those registers that could survive (regardless of my added call to clear_local_APIC in the previous patch).

I'm not arguing this is correct (not sure it's incorrect either), but given how things are handled ATM it seems quite dumb to do a RMW in
disconnect_bsp_APIC: it gives the false impression Xen is preserving something, when the register has already been wiped at startup.

The problem isn't with the currently defined fields (when we're trying to clear them).  It is with implementations which depend on set reserved bits not changing, and/or new fields defined at some future point.

If we already have a mix, then some more probably isn't a problem, but you did ask what the Linux feedback would be...

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel