x86/apic: improvements to disconnect_bsp_APIC
v2 cover x86/apic: improvements to disconnect_bsp_APIC
Hello,
The series contain some improvements to disconnect_bsp_APIC, which started as a way to fix the "APIC error on CPU0: 40(00)" error printed by some Intel boxes on reboot or shutdown. First patch is the fix for the error, second patch is a cleanup.
Roger Pau Monne (2):
x86/apic: fix disabling LVT0 in disconnect_bsp_APIC x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}
xen/arch/x86/apic.c | 21 ++++----------------- 1 file changed, 4 insertions(+), 17 deletions(-)
Signature · 7 lines
2.25.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 1/2 x86/apic: fix disabling LVT0 in disconnect_bsp_APIC
The Intel SDM states:
"When an illegal vector value (0 to 15) is written to a LVT entry and the delivery mode is Fixed (bits 8-11 equal 0), the APIC may signal an illegal vector error, without regard to whether the mask bit is set or whether an interrupt is actually seen on the input."
And that's exactly what's currently done in disconnect_bsp_APIC when virt_wire_setup is true and LVT LINT0 is being masked. By writing only APIC_LVT_MASKED Xen is actually setting the vector to 0 and the delivery mode to Fixed (0), and hence it triggers an APIC error even when the LVT entry is masked.
This would usually manifest when Xen is being shut down, as that's where disconnect_bsp_APIC is called:
(XEN) APIC error on CPU0: 40(00)
Fix this by calling clear_local_APIC prior to setting the LVT LINT registers which already clear LVT LINT0, and hence the troublesome write can be avoided as the register is already cleared.
---
Changes since v1:
- Use clear_local_APIC in order to clear LINT0.
--- xen/arch/x86/apic.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
xen/arch/x86/apic.c +2 -4
@@ -262,6 +262,8 @@ void disconnect_bsp_APIC(int virt_wire_setup)/* Go back to Virtual Wire compatibility mode */ unsigned long value; +clear_local_APIC (); +/* For the spurious interrupt use vector F, and enable it */ value =apic_read (APIC_SPIV ); value &= ~APIC_VECTOR_MASK ;
@@ -279,10 +281,6 @@ void disconnect_bsp_APIC(int virt_wire_setup) value =SET_APIC_DELIVERY_MODE (value,APIC_MODE_EXTINT );apic_write (APIC_LVT0 , value); } -else { -/* Disable LVT0 */ -apic_write (APIC_LVT0 ,APIC_LVT_MASKED ); - }/* For LVT1 make it edge triggered, active high, nmi and enabled */ value =apic_read (APIC_LVT1 );
Signature · 7 lines
2.25.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v2 1/2] x86/apic: fix disabling LVT0 in disconnect_bsp_APIC
Quoted 16 lines · Roger Pau Monne
The Intel SDM states:
"When an illegal vector value (0 to 15) is written to a LVT entry and the delivery mode is Fixed (bits 8-11 equal 0), the APIC may signal an illegal vector error, without regard to whether the mask bit is set or whether an interrupt is actually seen on the input."
And that's exactly what's currently done in
disconnect_bsp_APICwhenvirt_wire_setupis true and LVT LINT0 is being masked. By writing onlyAPIC_LVT_MASKEDXen is actually setting the vector to 0 and the delivery mode to Fixed (0), and hence it triggers an APIC error even when the LVT entry is masked.This would usually manifest when Xen is being shut down, as that's where
disconnect_bsp_APICis called:
Quoted 8 lines
(XEN) APIC error on CPU0: 40(00)
Fix this by calling
clear_local_APICprior to setting the LVT LINT registers which already clear LVT LINT0, and hence the troublesome write can be avoided as the register is already cleared.Reported-by Andrew Cooper <[email protected]>Signed-off-by Roger Pau Monné <[email protected]>
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 2/2 x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}
In reply to Roger Pau MonneThere's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.
--- xen/arch/x86/apic.c | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-)
xen/arch/x86/apic.c +2 -13
@@ -273,23 +273,12 @@ void disconnect_bsp_APIC(int virt_wire_setup)if (!virt_wire_setup) {/* For LVT0 make it edge triggered, active high, external and enabled */ - value =apic_read (APIC_LVT0 ); - value &= ~(APIC_MODE_MASK |APIC_SEND_PENDING | -APIC_INPUT_POLARITY |APIC_LVT_REMOTE_IRR | -APIC_LVT_LEVEL_TRIGGER |APIC_LVT_MASKED ); - value |=APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING ; - value =SET_APIC_DELIVERY_MODE (value,APIC_MODE_EXTINT ); + value =APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING |APIC_DM_EXTINT ;apic_write (APIC_LVT0 , value); }/* For LVT1 make it edge triggered, active high, nmi and enabled */ - value =apic_read (APIC_LVT1 ); - value &= ~( -APIC_MODE_MASK |APIC_SEND_PENDING | -APIC_INPUT_POLARITY |APIC_LVT_REMOTE_IRR | -APIC_LVT_LEVEL_TRIGGER |APIC_LVT_MASKED ); - value |=APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING ; - value =SET_APIC_DELIVERY_MODE (value,APIC_MODE_NMI ); + value =APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING |APIC_DM_NMI ;apic_write (APIC_LVT1 , value); } }
Signature · 7 lines
2.25.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}
Quoted 4 lines · Roger Pau Monne
There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.
The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}
Quoted 11 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monne
There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.
The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?
I can tell you what the Linux feedback will be.
This is not a fastpath. Always do RMW, because life is too short to keep on unbreaking hardware which makes such assumptions.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}
Quoted 13 lines · Andrew Cooper
Quoted 11 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monne
There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.
The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?
Quoted 4 lines
I can tell you what the Linux feedback will be.
This is not a fastpath. Always do RMW, because life is too short to keep on unbreaking hardware which makes such assumptions.
We already do such kinds of direct writes to LVT{0/1}, see clear_local_APIC where Xen clears the registers by directly writing APIC_LVT_MASKED to them (no RMW). As the modified code follows a call to clear_local_APIC there's nothing to preserve at this point.
Note that init_bsp_APIC and smp_callin also call clear_local_APIC, so there's no data in those registers that could survive (regardless of my added call to clear_local_APIC in the previous patch).
I'm not arguing this is correct (not sure it's incorrect either), but given how things are handled ATM it seems quite dumb to do a RMW indisconnect_bsp_APIC: it gives the false impression Xen is preserving something, when the register has already been wiped at startup.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v2 2/2] x86/apic: simplify disconnect_bsp_APIC setup of LVT{0/1}
Quoted 22 lines · Roger Pau Monné
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monne
There's no need to read the current values of LVT{0/1} for the purposes of the function, which seem to be to save the currently selected vector: in the destination modes used (ExtINT and NMI) the vector field is ignored and hence can be set to 0.
The question is - is there firmware putting data in these fields that it expects to survive unmodified? Such behavior would be highly suspect (to me at least), but you never know. There ought to be a reason it's been done this way not just in Xen, but also in Linux. IOW may I ask that you at least make an attempt to submit the same change for Linux, to see what the feedback is?
Quoted 4 lines
I can tell you what the Linux feedback will be.
This is not a fastpath. Always do RMW, because life is too short to keep on unbreaking hardware which makes such assumptions.
We already do such kinds of direct writes to LVT{0/1}, see
clear_local_APICwhere Xen clears the registers by directly writingAPIC_LVT_MASKEDto them (no RMW). As the modified code follows a call toclear_local_APICthere's nothing to preserve at this point.
Quoted 8 lines
Note that
init_bsp_APICandsmp_callinalso callclear_local_APIC, so there's no data in those registers that could survive (regardless of my added call toclear_local_APICin the previous patch).I'm not arguing this is correct (not sure it's incorrect either), but given how things are handled ATM it seems quite dumb to do a RMW in
disconnect_bsp_APIC: it gives the false impression Xen is preserving something, when the register has already been wiped at startup.
The problem isn't with the currently defined fields (when we're trying to clear them). It is with implementations which depend on set reserved bits not changing, and/or new fields defined at some future point.
If we already have a mix, then some more probably isn't a problem, but you did ask what the Linux feedback would be...
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel