ACPI Code First ECR: Support for resetting the Multiprocessor Wakeup Mailbox
Subsystems (3)
RFC 1/1 ACPI Code First ECR: Support for resetting the Multiprocessor Wakeup Mailbox
From: Rafael J. Wysocki <[email protected]>
The Multiprocessor Wakeup Structure in MADT is used for starting application processors (APs) on systems without a hardware mechanism suitable for this purpose. It describes a mailbox (Multiprocessor Wakeup Mailbox) allowing the boot CPU to send commands to APs and, in particular, to cause them to jump to a wakeup vector provided by the OS.
As currently defined, after using the Multiprocessor Wakeup Mailbox to make a given AP jump to the wakeup vector, it cannot be used any more with that AP, which is problematic in some use cases, like kexec() or hibernation, because it makes transferring control of APs from one OS instance to another difficult.
For this reason, the definition of Multiprocessor Wakeup Structure is changed so as to allow the Multiprocessor Wakeup Mailbox to be reset in order to be used once again with a given AP.
# Title: Add support for resetting the Multiprocessor Wakeup Mailbox
# Status: Draft
# Document: ACPI Specification 6.6
# License
SPDX-License Identifier: CC-BY-4.0
# Submitter:
* Sponsor: Rafael J. Wysocki, Intel
* Creator/Contributor: Kirill Shutemov, Intel
# Summary of the Change
Add a ResetVector field to Multiprocessor Wakeup Structure in order to provide the OS with a physical address of a Multiprocessor Wakeup Mailbox entry point for application processors that need to use the mailbox again after having used it once already.
# Benefits of the Change
It will allow a new OS instance to be started without going through a full system reset, for example after carrying out an OS update.
There are valid use cases in which the new OS started this way is different from the OS that has been running so far and in the absence of a hardware mechanism for resetting application processors, the only way to pass control of them from one OS instance to the other is through the platform firmware.
# Impact of the Change
OS and platform firmware will need to recognize version 1 of the Multiprocessor Wakeup Mailbox.
# Detailed Description of the Change
ACPI specification source changes needed to implement this proposal are as follows:
--- source/05_ACPI_Software_Programming_Model.rst | 40 +++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 4 deletions(-)
source/05_ACPI_Software_Programming_Model.rst +36 -4
@@ -4488,7 +4488,27 @@ The OS section can only be written by OS The OS section contains command, flags, APIC ID, and a wakeup address. After the OS detects the processor number from the MADT table, the OS may prepare the wakeup routine, fill the wakeup address field in the mailbox, indicate which processor need to be wakeup in the APID ID field, and send wakeup command. Once an application processor detects the wakeup command and its own APIC ID, the application processor will jump to the OS-provided wakeup address. The application processor will ignore the command if the APIC ID does not match its own. -For each application processor, the mailbox can be used only once for the wakeup command. After the application process takes the action according to the command, this mailbox will no longer be checked by this application processor. Other processors can continue using the mailbox for the next command. +For each application processor, the mailbox can be used only once for the wakeup command, unless +the MailBoxVersion field value is greater than 0 and the ResetVector field contains a nonzero value. + +After the application process takes the action according to the command, this mailbox will no longer +be checked by this application processor until the mailbox is reset for it as described below. Other +processors can continue using the mailbox for the next command. + +In case the mailbox needs to be used once again, for example in order to start a new version of +the OS without carrying out a full system reset, the ResetVector field value can be used for +making the given application processor enter a state to check the mailbox. For this purpose, +the OS needs to set up the mailbox reset environment, as per the ResetVector field description, +for the application processor in question and make that application processor jump to the +firmware-provided mailbox reset address retrieved from the ResetVector field. This needs to be +done for each application processor individually and doing it for one application processor does +not affect the other application processors, so they can continue to operate undisturbed. However, +if the ResetVector field value is 0, the mailbox cannot be reset and so it can be used only once. + +After an application processor has jumped to the reset address, the OS is required to verify that +the mailbox responds to commands by sending the test command to it. When it responds by changing +the command to noop, the OS is not required to maintain the mailbox reset environment for the given +application processor any more. .. list-table:: **Multiprocessor Wakeup Structure** :name: multiprocessor-wakeup-structure-table
@@ -4507,11 +4527,11 @@ For each application processor, the mail * - Length - 1 - 1 - - 16 + - 24 * - MailBoxVersion - 2 - 2 - - Version of the mailbox. 0 for this version of the spec. + - Version of the mailbox. 1 for this version of the spec. * - *Reserved* - 4 - 4
@@ -4520,6 +4540,17 @@ For each application processor, the mail - 8 - 8 - Physical address of the mailbox. It must be in ACPINvs. It must also be 4K bytes aligned. See :numref:`multiprocessor-wakeup-mailbox-structure` for the Mailbox definition. + * - ResetVector + - 8 + - 16 + - | The mailbox reset vector address for application processor(s). + | For Intel processors, the mailbox reset environment is: + | Interrupts must be disabled. + | RFLAGES.IF set to 0. + | Long mode enabled. + | Paging mode is enabled and physical memory for reset vector is identity mapped (virtual address equals physical address). + | Reset vector must be contained within one physical page. + | Selectors are set to flat and otherwise not used. .. list-table:: **Multiprocessor Wakeup Mailbox Structure** :name: multiprocessor-wakeup-mailbox-structure
@@ -4536,7 +4567,8 @@ For each application processor, the mail - 0 - | 0: Noop - no operation. | 1: Wakeup – jump to the wakeup vector. - | 2-0xFFFF: Reserved + | 2: Test - respond by changing the command to Noop. + | 3-0xFFFF: Reserved. * - *Reserved* - 2 - 2
Rafael J. Wysocki
maintainerQuoted 126 lines · Rafael J. Wysocki
From: Rafael J. Wysocki <[email protected]>
The Multiprocessor Wakeup Structure in MADT is used for starting application processors (APs) on systems without a hardware mechanism suitable for this purpose. It describes a mailbox (Multiprocessor Wakeup Mailbox) allowing the boot CPU to send commands to APs and, in particular, to cause them to jump to a wakeup vector provided by the OS.
As currently defined, after using the Multiprocessor Wakeup Mailbox to make a given AP jump to the wakeup vector, it cannot be used any more with that AP, which is problematic in some use cases, like
kexec()or hibernation, because it makes transferring control of APs from one OS instance to another difficult.For this reason, the definition of Multiprocessor Wakeup Structure is changed so as to allow the Multiprocessor Wakeup Mailbox to be reset in order to be used once again with a given AP.
# Title: Add support for resetting the Multiprocessor Wakeup Mailbox
# Status: Draft
# Document: ACPI Specification 6.6
# License
SPDX-License Identifier: CC-BY-4.0# Submitter:
* Sponsor: Rafael J. Wysocki, Intel
* Creator/Contributor: Kirill Shutemov, Intel# Summary of the Change
Add a ResetVector field to Multiprocessor Wakeup Structure in order to provide the OS with a physical address of a Multiprocessor Wakeup Mailbox entry point for application processors that need to use the mailbox again after having used it once already.# Benefits of the Change
It will allow a new OS instance to be started without going through a full system reset, for example after carrying out an OS update.There are valid use cases in which the new OS started this way is different from the OS that has been running so far and in the absence of a hardware mechanism for resetting application processors, the only way to pass control of them from one OS instance to the other is through the platform firmware.
# Impact of the Change
OS and platform firmware will need to recognize version 1 of the Multiprocessor Wakeup Mailbox.# Detailed Description of the Change
ACPI specification source changes needed to implement this proposal are as follows:--- source/05_ACPI_Software_Programming_Model.rst | 40 +++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 4 deletions(-)Index:
ACPI_spec/source/05_ACPI_Software_Programming_Model.rst
===================================================================source/05_ACPI_Software_Programming_Model.rst +34
-3@@ -4488,7 +4488,27 @@ The OS section can only be written by OS The OS section contains command, flags, APIC ID, and a wakeup address. After the OS detects the processor number from the MADT table, the OS may prepare the wakeup routine, fill the wakeup address field in the mailbox, indicate which processor need to be wakeup in the APID ID field, and send wakeup command. Once an application processor detects the wakeup command and its own APIC ID, the application processor will jump to the OS-provided wakeup address. The application processor will ignore the command if the APIC ID does not match its own. -For each application processor, the mailbox can be used only once for the wakeup command. After the application process takes the action according to the command, this mailbox will no longer be checked by this application processor. Other processors can continue using the mailbox for the next command. +For each application processor, the mailbox can be used only once for the wakeup command, unless +the MailBoxVersion field value is greater than 0 and the ResetVector field contains a nonzero value. + +After the application process takes the action according to the command, this mailbox will no longer +be checked by this application processor until the mailbox is reset for it as described below. Other +processors can continue using the mailbox for the next command. + +In case the mailbox needs to be used once again, for example in order to start a new version of +the OS without carrying out a full system reset, the ResetVector field value can be used for +making the given application processor enter a state to check the mailbox. For this purpose, +the OS needs to set up the mailbox reset environment, as per the ResetVector field description, +for the application processor in question and make that application processor jump to the +firmware-provided mailbox reset address retrieved from the ResetVector field. This needs to be +done for each application processor individually and doing it for one application processor does +not affect the other application processors, so they can continue to operate undisturbed. However, +if the ResetVector field value is 0, the mailbox cannot be reset and so it can be used only once. + +After an application processor has jumped to the reset address, the OS is required to verify that +the mailbox responds to commands by sending the test command to it. When it responds by changing +the command to noop, the OS is not required to maintain the mailbox reset environment for the given +application processor any more. .. list-table:: **Multiprocessor Wakeup Structure** :name: multiprocessor-wakeup-structure-table@@ -4507,11 +4527,11 @@ For each application processor, the mail * - Length - 1 - 1 - - 16 + - 24 * - MailBoxVersion - 2 - 2 - - Version of the mailbox. 0 for this version of the spec. + - Version of the mailbox. 1 for this version of the spec. * - *Reserved* - 4 - 4@@ -4520,6 +4540,17 @@ For each application processor, the mail - 8 - 8 - Physical address of the mailbox. It must be in ACPINvs. It must also be 4K bytes aligned. See :numref:`multiprocessor-wakeup-mailbox-structure` for the Mailbox definition. + * - ResetVector + - 8 + - 16 + - | The mailbox reset vector address for application processor(s). + | For Intel processors, the mailbox reset environment is: + | Interrupts must be disabled. + | RFLAGES.IF set to 0. + | Long mode enabled. + | Paging mode is enabled and physical memory for reset vector is identity mapped (virtual address equals physical address). + | Reset vector must be contained within one physical page. + | Selectors are set to flat and otherwise not used. .. list-table:: **Multiprocessor Wakeup Mailbox Structure** :name: multiprocessor-wakeup-mailbox-structure
source/05_ACPI_Software_Programming_Model.rst For each application processor, the mail · line 4574
Quoted 9 lines
source/05_ACPI_Software_Programming_Model.rst +2
-1@@ -4536,7 +4567,8 @@ For each application processor, the mail - 0 - | 0: Noop - no operation. | 1: Wakeup – jump to the wakeup vector. - | 2-0xFFFF: Reserved + | 2: Test - respond by changing the command to Noop. + | 3-0xFFFF: Reserved. * - *Reserved* - 2 - 2
This ECR has been approved as submitted.
[tip: x86/cc] x86/acpi: Add support for CPU offlining for ACPI MADT wakeup method
In reply to Rafael J. WysockiThe following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: 1ceebe2e46720de02af4cf626dc847ecc4a263fd Gitweb: https://git.kernel.org/tip/1ceebe2e46720de02af4cf626dc847ecc4a263fd Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 14 Jun 2024 12:59:03 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Mon, 17 Jun 2024 17:46:25 +02:00
x86/acpi: Add support for CPU offlining for ACPI MADT wakeup method
MADT Multiprocessor Wakeup structure version 1 brings support for CPU offlining:
BIOS provides a reset vector where the CPU has to jump to for offlining itself. The new TEST mailbox command can be used to test whether the CPU offlined itself which means the BIOS has control over the CPU and can online it again via the ACPI MADT wakeup method.
Add CPU offlining support for the ACPI MADT wakeup method by implementing custom cpu_die(), play_dead() and stop_this_cpu() SMP operations.
CPU offlining makes it possible to hand over secondary CPUs over kexec, not limiting the second kernel to a single CPU.
The change conforms to the approved ACPI spec change proposal. See the Link.
--- arch/x86/include/asm/acpi.h | 2 +- arch/x86/kernel/acpi/Makefile | 2 +- arch/x86/kernel/acpi/madt_playdead.S | 28 ++++- arch/x86/kernel/acpi/madt_wakeup.c | 184 +++++++++++++++++++++++++- include/acpi/actbl2.h | 15 +- 5 files changed, 227 insertions(+), 4 deletions(-) create mode 100644 arch/x86/kernel/acpi/madt_playdead.S
arch/x86/include/asm/acpi.h +2 -0
@@ -83,6 +83,8 @@ union acpi_subtable_headers;int __init acpi_parse_mp_wake (union acpi_subtable_headers *header,const unsigned long end); +void asm_acpi_mp_play_dead (u64 reset_vector,u64 pgd_pa); +/* * Check if the CPU can handle C2 and deeper */
arch/x86/kernel/acpi/Makefile +1 -1
@@ -4,7 +4,7 @@ obj-$(CONFIG_ACPI) += boot.o obj-$(CONFIG_ACPI_SLEEP) += sleep.o wakeup_$(BITS).o obj-$(CONFIG_ACPI_APEI) += apei.o obj-$(CONFIG_ACPI_CPPC_LIB) += cppc.o -obj-$(CONFIG_ACPI_MADT_WAKEUP) += madt_wakeup.o +obj-$(CONFIG_ACPI_MADT_WAKEUP) += madt_wakeup.o madt_playdead.o ifneq ($(CONFIG_ACPI_PROCESSOR),) obj-y += cstate.o
arch/x86/kernel/acpi/madt_playdead.S new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/linkage.h> +#include <asm/nospec-branch.h> +#include <asm/page_types.h> +#include <asm/processor-flags.h> + + .text + .align PAGE_SIZE + +/* + * asm_acpi_mp_play_dead() - Hand over control of the CPU to the BIOS + * + * rdi: Address of the ACPI MADT MPWK ResetVector + * rsi: PGD of the identity mapping + */ +SYM_FUNC_START(asm_acpi_mp_play_dead) + /* Turn off global entries. Following CR3 write will flush them. */ + movq %cr4, %rdx + andq $~(X86_CR4_PGE), %rdx + movq %rdx, %cr4 + + /* Switch to identity mapping */ + movq %rsi, %cr3 + + /* Jump to reset vector */ + ANNOTATE_RETPOLINE_SAFE + jmp *%rdi +SYM_FUNC_END(asm_acpi_mp_play_dead)
arch/x86/kernel/acpi/madt_wakeup.c +182 -2
212 lines not on this page · raw message
include/acpi/actbl2.h +14 -1
@@ -1197,8 +1197,20 @@ struct acpi_madt_multiproc_wakeup {u16 version;u32 reserved;/* reserved - must be zero */ u64 mailbox_address; +u64 reset_vector; }; +/* Values for Version field above */ + +enum acpi_madt_multiproc_wakeup_version { +ACPI_MADT_MP_WAKEUP_VERSION_NONE =0 , +ACPI_MADT_MP_WAKEUP_VERSION_V1 =1 , +ACPI_MADT_MP_WAKEUP_VERSION_RESERVED =2 ,/* 2 and greater are reserved */ +}; + +#define ACPI_MADT_MP_WAKEUP_SIZE_V0 16 +#define ACPI_MADT_MP_WAKEUP_SIZE_V1 24 +#define ACPI_MULTIPROC_WAKEUP_MB_OS_SIZE 2032#define ACPI_MULTIPROC_WAKEUP_MB_FIRMWARE_SIZE 2048
@@ -1211,7 +1223,8 @@ struct acpi_madt_multiproc_wakeup_mailbox {u8 reserved_firmware[ACPI_MULTIPROC_WAKEUP_MB_FIRMWARE_SIZE ];/* reserved for firmware use */ }; -#define ACPI_MP_WAKE_COMMAND_WAKEUP 1 +#define ACPI_MP_WAKE_COMMAND_WAKEUP 1 +#define ACPI_MP_WAKE_COMMAND_TEST 2/* 17: CPU Core Interrupt Controller (ACPI 6.5) */