Skip to contentSkip to conversation

x86/hyperv: Support wakeup mailbox for VTL2 TDX guest

PATCH In review Yunhong Jiang 6 participants 34 messages Aug 23, 2024 – Mar 20, 2025

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v2 cover x86/hyperv: Support wakeup mailbox for VTL2 TDX guest

This set of patches add ACPI multiprocessor wakeup support to VTL2 TDX VMs booting with device tree instead of ACPI.

Historically, x86 platforms have booted secondary processors (APs) using INIT followed by the start up IPI (SIPI) messages. However, TDX VMs can't use this protocol because this protocol requires assistance from VMMs while VMMs are not trusted by TDX guest.

ACPI specification version 6.4 introduced a new wakeup mailbox model to address this issue. A "Multiprocessor Wakeup Structure" has been added to an existing ACPI table (MADT). This structure provides the physical of a "Multiprocessor Wakeup Mailbox Structure". Message written to the mailbox structure steers the APs to the boot code.

With this new wakeup model, TDX VMs with ACPI support boot the APs securely. However, TDX VMs with the device tree, like VTL2 TDX hyperv guests, have no ACPI support and still face the challenge.

To fix this challenge, either a new mechanism from scratch is introduced, or the TDX VMs with device tree can utilize the ACPI wakeup model.

By reusing the ACPI wakeup mailbox model, the Multiprocessor Wakeup Mailbox Structure will be kept and the message mechanism will be the same as ACPI. This will reduce maintenance effort in the long term.

The first patch moves the madt wakeup implementation to generic code. The patches 2-3 add the wakeup mailbox support to the device tree. The patches 4-5 add the wakeup mailbox support to the hyperv guest. The patches 6-8 update the real mode memory reservation.
The last patch applies the wakeup mailbox support to the VTL2 TDX guest.

v2:
- Fix the cover letter's summary phrase.
- Fix the DT binding document to pass validation.
- Change the DT binding document to be ACPI independent.
- Move ACPI-only functions into the #ifdef CONFIG_ACPI.
- Change dtb_parse_mp_wake() to return mailbox physical address.
- Rework the hv_is_private_mmio_tdx().
- Remove unrelated real mode change from the patch that marks mailbox page private.
- Check hv_isolation_type_tdx() instead of wakeup_mailbox_addr in hv_vtl_init_platform() because wakeup_mailbox_addr is not parsed yet.
- Add memory range support to reserve_real_mode.
- Remove realmode_reserve callback and use the memory range.
- Move setting the real_mode_header to hv_vtl_init_platform.
- Update comments and commit messages.
- Minor style changes.

Yunhong Jiang (9):
x86/acpi: Move ACPI MADT wakeup to generic code
dt-bindings: x86: Add a binding for x86 wakeup mailbox x86/dt: Support the ACPI multiprocessor wakeup for device tree x86/hyperv: Parse the ACPI wakeup mailbox
x86/hyperv: Mark ACPI wakeup mailbox page as private x86/realmode: Add memory range support to reserve_real_mode x86/hyperv: Move setting the real_mode_header to hv_vtl_init_platform x86/hyperv: Set realmode_limit to 4G for VTL2 TDX guest x86/hyperv: Use wakeup mailbox for VTL2 guests if available

 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 MAINTAINERS                                   |  3 +
 arch/x86/Kconfig                              |  2 +-
 arch/x86/hyperv/hv_vtl.c                      | 29 +++++++--
 arch/x86/include/asm/acpi.h                   |  1 -
 arch/x86/include/asm/madt_wakeup.h            | 16 +++++
 arch/x86/include/asm/mshyperv.h               |  3 +
 arch/x86/include/asm/x86_init.h               |  6 ++
 arch/x86/kernel/Makefile                      |  1 +
 arch/x86/kernel/acpi/Makefile                 |  1 -
 arch/x86/kernel/cpu/mshyperv.c                |  2 +
 arch/x86/kernel/{acpi => }/madt_playdead.S    |  0
 arch/x86/kernel/{acpi => }/madt_wakeup.c      | 38 +++++++++++
 arch/x86/kernel/x86_init.c                    |  3 +
 arch/x86/realmode/init.c                      | 14 ++--
 drivers/hv/hv_common.c                        |  8 +++
 16 files changed, 176 insertions(+), 15 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
 create mode 100644 arch/x86/include/asm/madt_wakeup.h
 rename arch/x86/kernel/{acpi => }/madt_playdead.S (100%)
 rename arch/x86/kernel/{acpi => }/madt_wakeup.c (90%)

base-commit: fedb9ddeb1445f85d8f691b20f3faaa6dab8dd3f

Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 1/9 x86/acpi: Move ACPI MADT wakeup to generic code

In order to support the ACPI mailbox wakeup in device tree, move the MADT wakeup code out of the acpi directory, so that both ACPI and device tree can use it.

---
 MAINTAINERS                                | 2 ++
 arch/x86/kernel/Makefile                   | 1 +
 arch/x86/kernel/acpi/Makefile              | 1 -
 arch/x86/kernel/{acpi => }/madt_playdead.S | 0
 arch/x86/kernel/{acpi => }/madt_wakeup.c   | 0
 5 files changed, 3 insertions(+), 1 deletion(-)
 rename arch/x86/kernel/{acpi => }/madt_playdead.S (100%)
 rename arch/x86/kernel/{acpi => }/madt_wakeup.c (100%)
MAINTAINERS +2 -0
index 969eb9c6e759..5555a3bbac5f 100644
@@ -289,6 +289,8 @@ F:	Documentation/ABI/testing/configfs-acpi
 F:	Documentation/ABI/testing/sysfs-bus-acpi
 F:	Documentation/firmware-guide/acpi/
 F:	arch/x86/kernel/acpi/
+F:	arch/x86/kernel/madt_playdead.S
+F:	arch/x86/kernel/madt_wakeup.c
 F:	arch/x86/pci/acpi.c
 F:	drivers/acpi/
 F:	drivers/pci/*/*acpi*
arch/x86/kernel/Makefile +1 -0
index f7918980667a..0823e5ba3aea 100644
@@ -163,4 +163,5 @@ ifeq ($(CONFIG_X86_64),y)
 
 	obj-$(CONFIG_MMCONF_FAM10H)	+= mmconf-fam10h_64.o
 	obj-y				+= vsmp_64.o
+	obj-$(CONFIG_ACPI_MADT_WAKEUP)	+= madt_wakeup.o madt_playdead.o
 endif
arch/x86/kernel/acpi/Makefile +0 -1
index 842a5f449404..fc17b3f136fe 100644
@@ -4,7 +4,6 @@ obj-$(CONFIG_ACPI)		+= boot.o
 obj-$(CONFIG_ACPI_SLEEP)	+= sleep.o wakeup_$(BITS).o
 obj-$(CONFIG_ACPI_APEI)		+= apei.o
 obj-$(CONFIG_ACPI_CPPC_LIB)	+= cppc.o
-obj-$(CONFIG_ACPI_MADT_WAKEUP)	+= madt_wakeup.o madt_playdead.o
 
 ifneq ($(CONFIG_ACPI_PROCESSOR),)
 obj-y				+= cstate.o
arch/x86/kernel/madt_playdead.S renamed from arch/x86/kernel/acpi/madt_playdead.S +0 -0
similarity index 100%
rename from arch/x86/kernel/acpi/madt_playdead.S
rename to arch/x86/kernel/madt_playdead.S
arch/x86/kernel/madt_wakeup.c renamed from arch/x86/kernel/acpi/madt_wakeup.c +0 -0
similarity index 100%
rename from arch/x86/kernel/acpi/madt_wakeup.c
rename to arch/x86/kernel/madt_wakeup.c
Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 2/9 dt-bindings: x86: Add a binding for x86 wakeup mailbox

In reply to Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +64 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false
+
+properties:
+  wakeup-mailbox-addr:
+    $ref: /schemas/types.yaml#/definitions/uint64
+    description: |
+      The physical address of the wakeup mailbox data structure. The address
+      must be 4K bytes aligned 4k-size memory and it should be in the reserved
+      memory.
+
+      This requires the "enable-method" property in the cpus node is set to
+      "acpi-wakeup-mailbox".
+
+required:
+  - wakeup-mailbox-addr
+
+additionalProperties: false
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 52 lines · Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +38 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.

Documentation/devicetree/bindings/x86/wakeup.yaml line 48

Quoted 10 lines
Documentation/devicetree/bindings/x86/wakeup.yaml new file +10 -0
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false

This schema is still a no-op because of this false.

What is the point of defining one property if it is not placed anywhere? Every device node can have it? Seems wrong...

You need to come with proper schema. Lack of an example is another thing
- this cannot be even validated by the tools.

Best regards,
Krzysztof

#4 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 66 lines · Krzysztof Kozlowski
Quoted 52 lines · Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +38 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.

Documentation/devicetree/bindings/x86/wakeup.yaml line 48

Quoted 10 lines
Documentation/devicetree/bindings/x86/wakeup.yaml new file +10 -0
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false

This schema is still a no-op because of this false.

Quoted 8 lines

What is the point of defining one property if it is not placed anywhere? Every device node can have it? Seems wrong...

You need to come with proper schema. Lack of an example is another thing
- this cannot be even validated by the tools.

Best regards,
Krzysztof

Thank you for the feedback. Will update the schema file on next round submission.

Thanks
--jyh

Quoted 1 line
#5 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 67 lines · Yunhong Jiang
Quoted 66 lines · Krzysztof Kozlowski
Quoted 52 lines · Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +38 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.

Documentation/devicetree/bindings/x86/wakeup.yaml line 48

Quoted 10 lines
Documentation/devicetree/bindings/x86/wakeup.yaml new file +10 -0
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false

This schema is still a no-op because of this false.

Quoted 8 lines
Quoted 8 lines

What is the point of defining one property if it is not placed anywhere? Every device node can have it? Seems wrong...

You need to come with proper schema. Lack of an example is another thing
- this cannot be even validated by the tools.

Best regards,
Krzysztof

Hi, Krzysztof, I'm working to address your comments and have some questions. Hope to get help/guide from your side.

For the select, the writing-schema.rst describes it as "A json-schema used to match nodes for applying the schema" but I'm a bit confused. In my case, should it be "cpus" node? Is there any code/tools that uses this property, so that I can have a better understanding?

For your "validated by the tools", can you please share the tools you used to validate the schema? I used "make dt_binding_check" per the submitting-patches.rst but I think your comments is about another tool.

Sorry for the bothering. I read the DT spec and the Documentation/devicetree/bindings documents and still not sure.

Than you
--jyh

Quoted 9 lines

Thank you for the feedback. Will update the schema file on next round submission.

Thanks
--jyh

Quoted 1 line
#6 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 77 lines · Yunhong Jiang
Quoted 67 lines · Yunhong Jiang
Quoted 66 lines · Krzysztof Kozlowski
Quoted 52 lines · Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +38 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.

Documentation/devicetree/bindings/x86/wakeup.yaml line 48

Quoted 10 lines
Documentation/devicetree/bindings/x86/wakeup.yaml new file +10 -0
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false

This schema is still a no-op because of this false.

Quoted 8 lines
Quoted 8 lines

What is the point of defining one property if it is not placed anywhere? Every device node can have it? Seems wrong...

You need to come with proper schema. Lack of an example is another thing
- this cannot be even validated by the tools.

Best regards,
Krzysztof

Quoted 7 lines

Hi, Krzysztof, I'm working to address your comments and have some questions. Hope to get help/guide from your side.

For the select, the writing-schema.rst describes it as "A json-schema used to match nodes for applying the schema" but I'm a bit confused. In my case, should it be "cpus" node? Is there any code/tools that uses this property, so that I can have a better understanding?

Usually we expect matching by compatible, but it does not seem suitable here because it is not related to any specific device, right? That is the problem with all this DT-reuse-for-virtual-stuff work. It just does not follow usual expectations and guidelines - you do not describe a device.

You can still match by nodes. See all top-level bindings.

Quoted 4 lines

For your "validated by the tools", can you please share the tools you used to validate the schema? I used "make dt_binding_check" per the submitting-patches.rst but I think your comments is about another tool.

See writing-schema document.

Best regards,
Krzysztof

#7 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 86 lines · Krzysztof Kozlowski
Quoted 77 lines · Yunhong Jiang
Quoted 67 lines · Yunhong Jiang
Quoted 66 lines · Krzysztof Kozlowski
Quoted 52 lines · Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +38 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.

Documentation/devicetree/bindings/x86/wakeup.yaml line 48

Quoted 10 lines
Documentation/devicetree/bindings/x86/wakeup.yaml new file +10 -0
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false

This schema is still a no-op because of this false.

Quoted 8 lines
Quoted 8 lines

What is the point of defining one property if it is not placed anywhere? Every device node can have it? Seems wrong...

You need to come with proper schema. Lack of an example is another thing
- this cannot be even validated by the tools.

Best regards,
Krzysztof

Quoted 7 lines

Hi, Krzysztof, I'm working to address your comments and have some questions. Hope to get help/guide from your side.

For the select, the writing-schema.rst describes it as "A json-schema used to match nodes for applying the schema" but I'm a bit confused. In my case, should it be "cpus" node? Is there any code/tools that uses this property, so that I can have a better understanding?

Quoted 4 lines

Usually we expect matching by compatible, but it does not seem suitable here because it is not related to any specific device, right? That is the problem with all this DT-reuse-for-virtual-stuff work. It just does not follow usual expectations and guidelines - you do not describe a device.

Thank you for the reply.

I'm a bit confused on your "do not describe a device".
I think VM is also a device, it's just a virtual device, but I don't see much difference of the virtual and physical device from DT point of view, possibly I missed some point.

Quoted 2 lines

You can still match by nodes. See all top-level bindings.

After checking the code at
https://github.com/devicetree-org/dt-schema/blob/main/dtschema/validator.py,
seems the 'select' is translated to 'if'/'then'.

Do you have any example of "top-level bindings"? I tried to check binding for enable-methods like arm/cpu-enable-method/nuvoton,npcm750-smp or cpu/idle-states.yaml, but they are either not schema file, or quite different.

I have been struggling on this device binding document for a while. I reconsidered what this binding is for. This binding means, if the cpus node has "enable-method" as "acpi-wakeup-mailbox", then the device should have property "wakeup-mailbox-addr" with uint64 type.

In that case, I'm considering to set the "select" to be true so that it will apply to any potential device, and add if/then keyword to check the enable-method. But seems it does not work and I'm still trying to figure out the reason (I'm new to the json/json schema and is still learning).

I received followed error:
cpus: '#address-cells', '#size-cells', 'cpu@0', 'enable-method' do not match any of the regexes: 'pinctrl-[0-9]+'
from schema $id: http://devicetree.org/schemas/x86/wakeup.yaml#
cpu@0: 'device_type', 'reg' do not match any of the regexes: 'pinctrl-[0-9]+'
from schema $id: http://devicetree.org/schemas/x86/wakeup.yaml#

With the followed yaml file (I delete some description).

$ cat Documentation/devicetree/bindings/x86/wakeup.yaml
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
# Copyright (C) 2024 Intel Corporation
%YAML 1.2
---
$id: http://devicetree.org/schemas/x86/wakeup.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#

title: x86 mailbox wakeup
maintainers:
- Yunhong Jiang <[email protected]>

description: |
......
Removed to save space.

properties:
wakeup-mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint32
description: |
......
Removed to save space.

select: true

if:
properties:
enable-method:
contains:
const: acpi-wakeup-mailbox
required:
- enable-method

then:
required:
- wakeup-mailbox-addr

additionalProperties: false

examples:
- |

    cpus {
      #address-cells = <1>;
      #size-cells = <0>;
      enable-method = "acpi-wakeup-mailbox";
      wakeup-mailbox-addr = <0x1c000500>;
      cpu@0 {
        device_type = "cpu";
        reg = <0x1>;
      };
    };

...

Quoted 7 lines
Quoted 5 lines

For your "validated by the tools", can you please share the tools you used to validate the schema? I used "make dt_binding_check" per the submitting-patches.rst but I think your comments is about another tool.

See writing-schema document.

Yes, I figured out in the end that the validate tools means the dt-schema tools.

Thank you
--jyh

Quoted 6 lines

Best regards,
Krzysztof

#8 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 187 lines · Yunhong Jiang
Quoted 86 lines · Krzysztof Kozlowski
Quoted 77 lines · Yunhong Jiang
Quoted 67 lines · Yunhong Jiang
Quoted 66 lines · Krzysztof Kozlowski
Quoted 52 lines · Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +38 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.

Documentation/devicetree/bindings/x86/wakeup.yaml line 48

Quoted 10 lines
Documentation/devicetree/bindings/x86/wakeup.yaml new file +10 -0
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false

This schema is still a no-op because of this false.

Quoted 8 lines
Quoted 8 lines

What is the point of defining one property if it is not placed anywhere? Every device node can have it? Seems wrong...

You need to come with proper schema. Lack of an example is another thing
- this cannot be even validated by the tools.

Best regards,
Krzysztof

Quoted 7 lines

Hi, Krzysztof, I'm working to address your comments and have some questions. Hope to get help/guide from your side.

For the select, the writing-schema.rst describes it as "A json-schema used to match nodes for applying the schema" but I'm a bit confused. In my case, should it be "cpus" node? Is there any code/tools that uses this property, so that I can have a better understanding?

Quoted 4 lines

Usually we expect matching by compatible, but it does not seem suitable here because it is not related to any specific device, right? That is the problem with all this DT-reuse-for-virtual-stuff work. It just does not follow usual expectations and guidelines - you do not describe a device.

Thank you for the reply.

I'm a bit confused on your "do not describe a device".
I think VM is also a device, it's just a virtual device, but I don't see much difference of the virtual and physical device from DT point of view, possibly I missed some point.

Quoted 2 lines

You can still match by nodes. See all top-level bindings.

After checking the code at
https://github.com/devicetree-org/dt-schema/blob/main/dtschema/validator.py,
seems the 'select' is translated to 'if'/'then'.

Do you have any example of "top-level bindings"? I tried to check binding for enable-methods like arm/cpu-enable-method/nuvoton,npcm750-smp or cpu/idle-states.yaml, but they are either not schema file, or quite different.

I have been struggling on this device binding document for a while. I reconsidered what this binding is for. This binding means, if the cpus node has "enable-method" as "acpi-wakeup-mailbox", then the device should have property "wakeup-mailbox-addr" with uint64 type.

In that case, I'm considering to set the "select" to be true so that it will apply to any potential device, and add if/then keyword to check the enable-method. But seems it does not work and I'm still trying to figure out the reason (I'm new to the json/json schema and is still learning).

I received followed error:
cpus: '#address-cells', '#size-cells', 'cpu@0', 'enable-method' do not match any of the regexes: 'pinctrl-[0-9]+'
from schema $id: http://devicetree.org/schemas/x86/wakeup.yaml#
cpu@0: 'device_type', 'reg' do not match any of the regexes: 'pinctrl-[0-9]+'
from schema $id: http://devicetree.org/schemas/x86/wakeup.yaml#

With the followed yaml file (I delete some description).

$ cat Documentation/devicetree/bindings/x86/wakeup.yaml
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
# Copyright (C) 2024 Intel Corporation
%YAML 1.2
---
$id: http://devicetree.org/schemas/x86/wakeup.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#

title: x86 mailbox wakeup
maintainers:
- Yunhong Jiang <[email protected]>

description: |
......
Removed to save space.

properties:
wakeup-mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint32
description: |
......
Removed to save space.

select: true

if:
properties:
enable-method:
contains:
const: acpi-wakeup-mailbox
required:
- enable-method

then:
required:
- wakeup-mailbox-addr

additionalProperties: false

examples:
- |

    cpus {
      #address-cells = <1>;
      #size-cells = <0>;
      enable-method = "acpi-wakeup-mailbox";
      wakeup-mailbox-addr = <0x1c000500>;
      cpu@0 {
        device_type = "cpu";
        reg = <0x1>;
      };
    };

...

Quoted 6 lines
Quoted 5 lines

For your "validated by the tools", can you please share the tools you used to validate the schema? I used "make dt_binding_check" per the submitting-patches.rst but I think your comments is about another tool.

Quoted 5 lines
Quoted 1 line

See writing-schema document.

Yes, I figured out in the end that the validate tools means the dt-schema tools.

Thank you
--jyh

One thing just come to my mind. The Documentation/devicetree/bindings/x86/wakeup.yaml only adds one property, wakeup-mailbox-addr, to the cpus node, and it does not specify any new object. Per my understanding, the json schema file normally specifies a new object. Is my change a supported scenario in current Documentation/devicetree?

I checked similar scenario like arm/cpu-enable-method/nuvoton and npcm750-smp/cpu-enable-method/al,alpine-smp, they don't present as a json schema file. Wonder if it's because the same reason.

Quoted 8 lines
Quoted 7 lines

Best regards,
Krzysztof

#9 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 9 lines · Yunhong Jiang
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines

For your "validated by the tools", can you please share the tools you used to validate the schema? I used "make dt_binding_check" per the submitting-patches.rst but I think your comments is about another tool.

See writing-schema document.

Yes, I figured out in the end that the validate tools means the dt-schema tools.

Quoted 8 lines
Quoted 2 lines

Thank you
--jyh

One thing just come to my mind. The Documentation/devicetree/bindings/x86/wakeup.yaml only adds one property, wakeup-mailbox-addr, to the cpus node, and it does not specify any new object. Per my understanding, the json schema file normally specifies a new object. Is my change a supported scenario in current Documentation/devicetree?

I think that's not related. You can add properties per specific devices, but (as I said) you do not have a compatible. Compatible represents a specific device. You do not have it. Look at the cpus.yaml schema.

Quoted 4 lines

I checked similar scenario like arm/cpu-enable-method/nuvoton and npcm750-smp/cpu-enable-method/al,alpine-smp, they don't present as a json schema file. Wonder if it's because the same reason.

Rather no one cares about that hardware.

Best regards,
Krzysztof

#10 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

In reply to Yunhong Jiang
Quoted 92 lines · Yunhong Jiang
Quoted 86 lines · Krzysztof Kozlowski
Quoted 77 lines · Yunhong Jiang
Quoted 67 lines · Yunhong Jiang
Quoted 66 lines · Krzysztof Kozlowski
Quoted 52 lines · Yunhong Jiang

Add the binding to use mailbox wakeup mechanism to bringup APs.

---
 .../devicetree/bindings/x86/wakeup.yaml       | 64 +++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/x86/wakeup.yaml
Documentation/devicetree/bindings/x86/wakeup.yaml new file +38 -0
new file mode 100644
index 000000000000..cb84e2756bca
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2024 Intel Corporation
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/x86/wakeup.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: x86 mailbox wakeup
+maintainers:
+  - Yunhong Jiang <[email protected]>
+
+description: |
+  The x86 mailbox wakeup mechanism defines a mechanism to let the bootstrap
+  processor (BSP) to wake up application processors (APs) through a wakeup
+  mailbox.
+
+  The "wakeup-mailbox-addr" property specifies the wakeup mailbox address. The
+  wakeup mailbox is a 4K-aligned 4K-size memory block allocated in the reserved
+  memory.
+
+  The wakeup mailbox structure is defined as follows.
+
+    uint16_t command;
+    uint16_t reserved;
+    uint32_t apic_id;
+    uint64_t wakeup_vector;
+    uint8_t  reservedForOs[2032];
+
+  The memory after reservedForOs field is reserved and OS should not touch it.
+
+  To wakes up a AP, the BSP prepares the wakeup routine, fills the wakeup
+  routine's address into the wakeup_vector field, fill the apic_id field with
+  the target AP's APIC_ID, and write 1 to the command field. After receiving the
+  wakeup command, the target AP will jump to the wakeup routine.
+
+  For each AP, the mailbox can be used only once for the wakeup command. After
+  the AP jumps to the wakeup routine, the mailbox will no longer be checked by
+  this AP.

Documentation/devicetree/bindings/x86/wakeup.yaml line 48

Quoted 10 lines
Documentation/devicetree/bindings/x86/wakeup.yaml new file +10 -0
+
+  The wakeup mailbox structure and the wakeup process is the same as
+  the Multiprocessor Wakeup Mailbox Structure defined in ACPI spec version 6.5,
+  section 5.2.12.19 [1].
+
+  References:
+
+  [1] https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html
+
+select: false

This schema is still a no-op because of this false.

Quoted 8 lines
Quoted 8 lines

What is the point of defining one property if it is not placed anywhere? Every device node can have it? Seems wrong...

You need to come with proper schema. Lack of an example is another thing
- this cannot be even validated by the tools.

Best regards,
Krzysztof

Quoted 7 lines

Hi, Krzysztof, I'm working to address your comments and have some questions. Hope to get help/guide from your side.

For the select, the writing-schema.rst describes it as "A json-schema used to match nodes for applying the schema" but I'm a bit confused. In my case, should it be "cpus" node? Is there any code/tools that uses this property, so that I can have a better understanding?

Quoted 4 lines

Usually we expect matching by compatible, but it does not seem suitable here because it is not related to any specific device, right? That is the problem with all this DT-reuse-for-virtual-stuff work. It just does not follow usual expectations and guidelines - you do not describe a device.

Quoted 6 lines

Thank you for the reply.

I'm a bit confused on your "do not describe a device".
I think VM is also a device, it's just a virtual device, but I don't see much difference of the virtual and physical device from DT point of view, possibly I missed some point.

VM is purely a software construct, so it is not a device. But regardless of terminology, you did not describe here VM or its part, either.

Quoted 11 lines
Quoted 3 lines

You can still match by nodes. See all top-level bindings.

After checking the code at
https://github.com/devicetree-org/dt-schema/blob/main/dtschema/validator.py,
seems the 'select' is translated to 'if'/'then'.

Do you have any example of "top-level bindings"? I tried to check binding for enable-methods like arm/cpu-enable-method/nuvoton,npcm750-smp or cpu/idle-states.yaml, but they are either not schema file, or quite different.

Most of board schemas, so for example in arm directory.

Quoted 44 lines

I have been struggling on this device binding document for a while. I reconsidered what this binding is for. This binding means, if the cpus node has "enable-method" as "acpi-wakeup-mailbox", then the device should have property "wakeup-mailbox-addr" with uint64 type.

In that case, I'm considering to set the "select" to be true so that it will apply to any potential device, and add if/then keyword to check the enable-method. But seems it does not work and I'm still trying to figure out the reason (I'm new to the json/json schema and is still learning).

I received followed error:
cpus: '#address-cells', '#size-cells', 'cpu@0', 'enable-method' do not match any of the regexes: 'pinctrl-[0-9]+'
from schema $id: http://devicetree.org/schemas/x86/wakeup.yaml#
cpu@0: 'device_type', 'reg' do not match any of the regexes: 'pinctrl-[0-9]+'
from schema $id: http://devicetree.org/schemas/x86/wakeup.yaml#

With the followed yaml file (I delete some description).

$ cat Documentation/devicetree/bindings/x86/wakeup.yaml
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
# Copyright (C) 2024 Intel Corporation
%YAML 1.2
---
$id: http://devicetree.org/schemas/x86/wakeup.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#

title: x86 mailbox wakeup
maintainers:
- Yunhong Jiang <[email protected]>

description: |
......
Removed to save space.

properties:
wakeup-mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint32
description: |
......
Removed to save space.

select: true

Quoted 7 lines

if:
properties:
enable-method:
contains:
const: acpi-wakeup-mailbox
required:
- enable-method

enable-method is part of CPUs, so you probably should match the CPUs... I am not sure, I don't have the big picture here.

Maybe if companies want to push more of bindings for purely virtual systems, then they should first get involved more, instead of relying on us. Provide reviews for your virtual stuff, provide guidance. There is resistance in accepting bindings for such cases for a reason - I don't even know what exactly is this and judging/reviewing based on my practices will no be accurate.

Quoted 27 lines

then:
required:
- wakeup-mailbox-addr

additionalProperties: false

examples:
- |

    cpus {
      #address-cells = <1>;
      #size-cells = <0>;
      enable-method = "acpi-wakeup-mailbox";
      wakeup-mailbox-addr = <0x1c000500>;
      cpu@0 {
        device_type = "cpu";
        reg = <0x1>;
      };
    };

...

Quoted 6 lines
Quoted 5 lines

For your "validated by the tools", can you please share the tools you used to validate the schema? I used "make dt_binding_check" per the submitting-patches.rst but I think your comments is about another tool.

Quoted 12 lines
Quoted 1 line

See writing-schema document.

Yes, I figured out in the end that the validate tools means the dt-schema tools.

Thank you
--jyh

Quoted 6 lines

Best regards,
Krzysztof

Best regards,
Krzysztof

#11 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

On Fri, Sep 20, 2024 at 01:15:41PM +0200, Krzysztof Kozlowski wrote:

[...]

Quoted 9 lines

enable-method is part of CPUs, so you probably should match the CPUs... I am not sure, I don't have the big picture here.

Maybe if companies want to push more of bindings for purely virtual systems, then they should first get involved more, instead of relying on us. Provide reviews for your virtual stuff, provide guidance. There is resistance in accepting bindings for such cases for a reason - I don't even know what exactly is this and judging/reviewing based on my practices will no be accurate.

Hi Krzysztof,

I am taking over this work from Yunhong.

First of all, I apologize for the late reply. I will make sure communications are timely in the future.

Our goal is to describe in the device tree a mechanism or artifact to boot secondary CPUs.

In our setup, the firmware puts secondary CPUs to monitor a memory location (i.e., the wakeup mailbox) while spinning. From the boot CPU, the OS writes in the mailbox the wakeup vector and the ID of the secondary CPU it wants to boot. When a secondary CPU sees its own ID it will jump to the wakeup vector.

This is similar to the spin-table described in the Device Tree specification. The key difference is that with the spin-table CPUs spin until a non-zero value is written in cpu-release-addr. The wakeup mailbox uses CPU IDs.

You raised the issue of the lack of a compatible property, and the fact that we are not describing an actual device.

I took your suggestion of matching by node and I came up with the binding below. I see these advantages in this approach:

* I define a new node with a compatible property.
* There is precedent: the psci node. In the cpus node, each cpu@n has an enable-method property that specify psci.
* The mailbox is a device as it is located in a reserved memory region. This true regardless of the device tree describing bare-metal or virtualized machines.

Thanks in advance for your feedback!

Best,
Ricardo

(only the relevant sections of the binding are shown for brevity)

properties:
$nodename:
const: wakeup-mailbox

compatible:
const: x86,wakeup-mailbox

mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint64

required:
- compatible
- mailbox-addr

additionalProperties: false

examples:
- |

    wakeup-mailbox {
      compatible = "x86,wakeup-mailbox";
      mailbox-addr = <0 0x1c000500>;
    };

    cpus {
        #address-cells = <1>;
        #size-cells = <0>;

        cpu@0 {
            device_type = "cpu";
            reg = <0x00>;
            enable-method = "wakeup-mailbox";
        };
    };

...

#12 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 35 lines · Ricardo Neri

On Fri, Sep 20, 2024 at 01:15:41PM +0200, Krzysztof Kozlowski wrote:

[...]

Quoted 9 lines

enable-method is part of CPUs, so you probably should match the CPUs... I am not sure, I don't have the big picture here.

Maybe if companies want to push more of bindings for purely virtual systems, then they should first get involved more, instead of relying on us. Provide reviews for your virtual stuff, provide guidance. There is resistance in accepting bindings for such cases for a reason - I don't even know what exactly is this and judging/reviewing based on my practices will no be accurate.

Hi Krzysztof,

I am taking over this work from Yunhong.

First of all, I apologize for the late reply. I will make sure communications are timely in the future.

Our goal is to describe in the device tree a mechanism or artifact to boot secondary CPUs.

In our setup, the firmware puts secondary CPUs to monitor a memory location (i.e., the wakeup mailbox) while spinning. From the boot CPU, the OS writes in the mailbox the wakeup vector and the ID of the secondary CPU it wants to boot. When a secondary CPU sees its own ID it will jump to the wakeup vector.

This is similar to the spin-table described in the Device Tree specification. The key difference is that with the spin-table CPUs spin until a non-zero value is written in cpu-release-addr. The wakeup mailbox uses CPU IDs.

Quoted 8 lines

You raised the issue of the lack of a compatible property, and the fact that we are not describing an actual device.

I took your suggestion of matching by node and I came up with the binding below. I see these advantages in this approach:

* I define a new node with a compatible property.
* There is precedent: the psci node. In the cpus node, each cpu@n has

psci is a standard. If you are documenting here a standard, clearly express it and provide reference to the specification.

Quoted 10 lines

an enable-method property that specify psci.
* The mailbox is a device as it is located in a reserved memory region. This true regardless of the device tree describing bare-metal or virtualized machines.

Thanks in advance for your feedback!

Best,
Ricardo

Quoted 8 lines

(only the relevant sections of the binding are shown for brevity)

properties:
$nodename:
const: wakeup-mailbox

compatible:
const: x86,wakeup-mailbox

You need vendor prefix for this particular device. If I pointed out lack of device and specific compatible, then adding random compatible does not solve it. I understand it solves for you, but not from the bindings point of view.

Quoted 3 lines

mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint64

So is this some sort of reserved memory? Mailbox needs mbox-cells, so maybe that's not mailbox.

Best regards,
Krzysztof

#13 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 36 lines · Krzysztof Kozlowski
Quoted 35 lines · Ricardo Neri

On Fri, Sep 20, 2024 at 01:15:41PM +0200, Krzysztof Kozlowski wrote:

[...]

Quoted 9 lines

enable-method is part of CPUs, so you probably should match the CPUs... I am not sure, I don't have the big picture here.

Maybe if companies want to push more of bindings for purely virtual systems, then they should first get involved more, instead of relying on us. Provide reviews for your virtual stuff, provide guidance. There is resistance in accepting bindings for such cases for a reason - I don't even know what exactly is this and judging/reviewing based on my practices will no be accurate.

Hi Krzysztof,

I am taking over this work from Yunhong.

First of all, I apologize for the late reply. I will make sure communications are timely in the future.

Our goal is to describe in the device tree a mechanism or artifact to boot secondary CPUs.

In our setup, the firmware puts secondary CPUs to monitor a memory location (i.e., the wakeup mailbox) while spinning. From the boot CPU, the OS writes in the mailbox the wakeup vector and the ID of the secondary CPU it wants to boot. When a secondary CPU sees its own ID it will jump to the wakeup vector.

This is similar to the spin-table described in the Device Tree specification. The key difference is that with the spin-table CPUs spin until a non-zero value is written in cpu-release-addr. The wakeup mailbox uses CPU IDs.

Quoted 9 lines
Quoted 8 lines

You raised the issue of the lack of a compatible property, and the fact that we are not describing an actual device.

I took your suggestion of matching by node and I came up with the binding below. I see these advantages in this approach:

* I define a new node with a compatible property.
* There is precedent: the psci node. In the cpus node, each cpu@n has

Thanks for your feedback!

Quoted 2 lines

psci is a standard. If you are documenting here a standard, clearly express it and provide reference to the specification.

It is not really a standard, but this mailbox behaves indentically to the wakeup mailbox described in the ACPI spec [1]. I am happy reference the spec in the documentation of the binding... or describe in full the mechanism of mailbox without referring to ACPI. You had indicated you don't care about what ACPI does [2].

In a nutshell, the wakeup mailbox is similar to the spin table used in ARM
boards: it is reserved memory region that secondary CPUs monitor while spinning.

Quoted 18 lines
Quoted 18 lines

an enable-method property that specify psci.
* The mailbox is a device as it is located in a reserved memory region. This true regardless of the device tree describing bare-metal or virtualized machines.

Thanks in advance for your feedback!

Best,
Ricardo

(only the relevant sections of the binding are shown for brevity)

properties:
$nodename:
const: wakeup-mailbox

Quoted 7 lines
Quoted 2 lines

compatible:
const: x86,wakeup-mailbox

You need vendor prefix for this particular device. If I pointed out lack of device and specific compatible, then adding random compatible does not solve it. I understand it solves for you, but not from the bindings point of view.

I see. Platform firmware will implement the mailbox. It would not be any specific hardware from Intel. Perhaps intel,wakeup-mailbox?

Quoted 6 lines
Quoted 4 lines

mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint64

So is this some sort of reserved memory?

Yes, the mailbox is located in reserved memory.

Quoted 2 lines

Mailbox needs mbox-cells, so
maybe that's not mailbox.

Your comment got me to look under Documentation/devicetree/bindings/mailbox. Maybe I am ovethinking this and I can describe the mailbox as others vendors do (see below).

Thanks and BR,
Ricardo

[1]. https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/05_ACPI_Software_Programming_Model/ACPI_Software_Programming_Model.html#multiprocessor-wakeup-structure
if that is OK.
[2]. https://lore.kernel.org/lkml/[email protected]

description: |
[Removed for brevity]

properties:
compatible:
const: intel,wakeup-mailbox

reg:
maxItems: 1

'#mbox-cells':
const: 1

required:
- compatible
- reg

additionalProperties: false

examples:
- |

    wakeupmailbox: mailbox@1c000500 {
      compatible = "intel,wakeup-mailbox";
      reg = <0x1c000500 0x100>;
      #mbox-cells = <1>;
    };

    cpus {
        #address-cells = <1>;
        #size-cells = <0>;

        cpu@0 {
            device_type = "cpu";
            reg = <0x00>;
            enable-method = "wakeup-mailbox";
        };
    };

...

#14 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 49 lines · Ricardo Neri
Quoted 36 lines · Krzysztof Kozlowski
Quoted 35 lines · Ricardo Neri

On Fri, Sep 20, 2024 at 01:15:41PM +0200, Krzysztof Kozlowski wrote:

[...]

Quoted 9 lines

enable-method is part of CPUs, so you probably should match the CPUs... I am not sure, I don't have the big picture here.

Maybe if companies want to push more of bindings for purely virtual systems, then they should first get involved more, instead of relying on us. Provide reviews for your virtual stuff, provide guidance. There is resistance in accepting bindings for such cases for a reason - I don't even know what exactly is this and judging/reviewing based on my practices will no be accurate.

Hi Krzysztof,

I am taking over this work from Yunhong.

First of all, I apologize for the late reply. I will make sure communications are timely in the future.

Our goal is to describe in the device tree a mechanism or artifact to boot secondary CPUs.

In our setup, the firmware puts secondary CPUs to monitor a memory location (i.e., the wakeup mailbox) while spinning. From the boot CPU, the OS writes in the mailbox the wakeup vector and the ID of the secondary CPU it wants to boot. When a secondary CPU sees its own ID it will jump to the wakeup vector.

This is similar to the spin-table described in the Device Tree specification. The key difference is that with the spin-table CPUs spin until a non-zero value is written in cpu-release-addr. The wakeup mailbox uses CPU IDs.

Quoted 9 lines
Quoted 8 lines

You raised the issue of the lack of a compatible property, and the fact that we are not describing an actual device.

I took your suggestion of matching by node and I came up with the binding below. I see these advantages in this approach:

* I define a new node with a compatible property.
* There is precedent: the psci node. In the cpus node, each cpu@n has

Thanks for your feedback!

Quoted 8 lines
Quoted 2 lines

psci is a standard. If you are documenting here a standard, clearly express it and provide reference to the specification.

It is not really a standard, but this mailbox behaves indentically to the wakeup mailbox described in the ACPI spec [1]. I am happy reference the spec in the documentation of the binding... or describe in full the mechanism of mailbox without referring to ACPI. You had indicated you don't care about what ACPI does [2].

Behaving like ACPI and implementing a spec are two different things. The question is whether you need to implement it like that and I believe answer is: no.

Quoted 36 lines

In a nutshell, the wakeup mailbox is similar to the spin table used in ARM
boards: it is reserved memory region that secondary CPUs monitor while spinning.

Quoted 18 lines
Quoted 18 lines

an enable-method property that specify psci.
* The mailbox is a device as it is located in a reserved memory region. This true regardless of the device tree describing bare-metal or virtualized machines.

Thanks in advance for your feedback!

Best,
Ricardo

(only the relevant sections of the binding are shown for brevity)

properties:
$nodename:
const: wakeup-mailbox

Quoted 7 lines
Quoted 2 lines

compatible:
const: x86,wakeup-mailbox

You need vendor prefix for this particular device. If I pointed out lack of device and specific compatible, then adding random compatible does not solve it. I understand it solves for you, but not from the bindings point of view.

I see. Platform firmware will implement the mailbox. It would not be any specific hardware from Intel. Perhaps intel,wakeup-mailbox?

Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint64

So is this some sort of reserved memory?

Yes, the mailbox is located in reserved memory.

Then why reserved memory bindings are not working?

Anyway this was half a year ago. None of the emails are in my inbox. None of the context is in my head.

It's the second or third email this month someone responds to my email from 2024.

Frankly, that's a neat trick. I won't remember anything, but it would be impolite to say just "no" without arguments. So now you will resend the same code leading to the same discussions we had half a year ago. Or ignoring that discussions.

I don't understand why this should be reviewers problem, so no, that's just unfair.

Best regards,
Krzysztof

#15 lore raw

Re: [PATCH v2 2/9] dt-bindings: x86: Add a binding for x86 wakeup mailbox

Quoted 115 lines · Krzysztof Kozlowski
Quoted 49 lines · Ricardo Neri
Quoted 36 lines · Krzysztof Kozlowski
Quoted 35 lines · Ricardo Neri

On Fri, Sep 20, 2024 at 01:15:41PM +0200, Krzysztof Kozlowski wrote:

[...]

Quoted 9 lines

enable-method is part of CPUs, so you probably should match the CPUs... I am not sure, I don't have the big picture here.

Maybe if companies want to push more of bindings for purely virtual systems, then they should first get involved more, instead of relying on us. Provide reviews for your virtual stuff, provide guidance. There is resistance in accepting bindings for such cases for a reason - I don't even know what exactly is this and judging/reviewing based on my practices will no be accurate.

Hi Krzysztof,

I am taking over this work from Yunhong.

First of all, I apologize for the late reply. I will make sure communications are timely in the future.

Our goal is to describe in the device tree a mechanism or artifact to boot secondary CPUs.

In our setup, the firmware puts secondary CPUs to monitor a memory location (i.e., the wakeup mailbox) while spinning. From the boot CPU, the OS writes in the mailbox the wakeup vector and the ID of the secondary CPU it wants to boot. When a secondary CPU sees its own ID it will jump to the wakeup vector.

This is similar to the spin-table described in the Device Tree specification. The key difference is that with the spin-table CPUs spin until a non-zero value is written in cpu-release-addr. The wakeup mailbox uses CPU IDs.

Quoted 9 lines
Quoted 8 lines

You raised the issue of the lack of a compatible property, and the fact that we are not describing an actual device.

I took your suggestion of matching by node and I came up with the binding below. I see these advantages in this approach:

* I define a new node with a compatible property.
* There is precedent: the psci node. In the cpus node, each cpu@n has

Thanks for your feedback!

Quoted 8 lines
Quoted 2 lines

psci is a standard. If you are documenting here a standard, clearly express it and provide reference to the specification.

It is not really a standard, but this mailbox behaves indentically to the wakeup mailbox described in the ACPI spec [1]. I am happy reference the spec in the documentation of the binding... or describe in full the mechanism of mailbox without referring to ACPI. You had indicated you don't care about what ACPI does [2].

Behaving like ACPI and implementing a spec are two different things. The question is whether you need to implement it like that and I believe answer is: no.

Quoted 36 lines

In a nutshell, the wakeup mailbox is similar to the spin table used in ARM
boards: it is reserved memory region that secondary CPUs monitor while spinning.

Quoted 18 lines
Quoted 18 lines

an enable-method property that specify psci.
* The mailbox is a device as it is located in a reserved memory region. This true regardless of the device tree describing bare-metal or virtualized machines.

Thanks in advance for your feedback!

Best,
Ricardo

(only the relevant sections of the binding are shown for brevity)

properties:
$nodename:
const: wakeup-mailbox

Quoted 7 lines
Quoted 2 lines

compatible:
const: x86,wakeup-mailbox

You need vendor prefix for this particular device. If I pointed out lack of device and specific compatible, then adding random compatible does not solve it. I understand it solves for you, but not from the bindings point of view.

I see. Platform firmware will implement the mailbox. It would not be any specific hardware from Intel. Perhaps intel,wakeup-mailbox?

Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

mailbox-addr:
$ref: /schemas/types.yaml#/definitions/uint64

So is this some sort of reserved memory?

Yes, the mailbox is located in reserved memory.

Then why reserved memory bindings are not working?

Anyway this was half a year ago. None of the emails are in my inbox. None of the context is in my head.

It's the second or third email this month someone responds to my email from 2024.

Quoted 8 lines

Frankly, that's a neat trick. I won't remember anything, but it would be impolite to say just "no" without arguments. So now you will resend the same code leading to the same discussions we had half a year ago. Or ignoring that discussions.

I don't understand why this should be reviewers problem, so no, that's just unfair.

Thank you again for your feedback. I will send a third version of the patchset. This will give the full context. I will incorporate your feedback in such version.

Thanks and BR,
Ricardo

#16 lore raw

PATCH v2 3/9 x86/dt: Support the ACPI multiprocessor wakeup for device tree

In reply to Yunhong Jiang

When a TDX guest boots with the device tree instead of ACPI, it can reuse the ACPI multiprocessor wakeup mechanism to wake up application processors(AP), without introducing a new mechanism from scrach.

In the ACPI spec, two structures are defined to wake up the APs: the multiprocessor wakeup structure and the multiprocessor wakeup mailbox structure. The multiprocessor wakeup structure is passed to OS through a Multiple APIC Description Table(MADT), one field specifying the physical address of the multiprocessor wakeup mailbox structure. The OS sends a message to firmware through the multiprocessor wakeup mailbox structure, to bring up the APs.

In device tree environment, the multiprocessor wakeup structure is not used, to reduce the dependency on the generic ACPI table. The information defined in this structure is defined in the properties of cpus node in the device tree. The "wakeup-mailbox-addr" property specifies the physical address of the multiprocessor wakeup mailbox structure. The OS will follow the ACPI spec to send the message to the firmware to bring up the APs.

---
 MAINTAINERS                        |  1 +
 arch/x86/Kconfig                   |  2 +-
 arch/x86/include/asm/acpi.h        |  1 -
 arch/x86/include/asm/madt_wakeup.h | 16 +++++++++++++
 arch/x86/kernel/madt_wakeup.c      | 38 ++++++++++++++++++++++++++++++
 5 files changed, 56 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/madt_wakeup.h
MAINTAINERS +1 -0
index 5555a3bbac5f..900de6501eba 100644
@@ -288,6 +288,7 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
 F:	Documentation/ABI/testing/configfs-acpi
 F:	Documentation/ABI/testing/sysfs-bus-acpi
 F:	Documentation/firmware-guide/acpi/
+F:	arch/x86/include/asm/madt_wakeup.h
 F:	arch/x86/kernel/acpi/
 F:	arch/x86/kernel/madt_playdead.S
 F:	arch/x86/kernel/madt_wakeup.c
arch/x86/Kconfig +1 -1
index d422247b2882..dba46dd30049 100644
@@ -1123,7 +1123,7 @@ config X86_LOCAL_APIC
 config ACPI_MADT_WAKEUP
 	def_bool y
 	depends on X86_64
-	depends on ACPI
+	depends on ACPI || OF
 	depends on SMP
 	depends on X86_LOCAL_APIC
 
arch/x86/include/asm/acpi.h +0 -1
index 21bc53f5ed0c..0e082303ca26 100644
@@ -83,7 +83,6 @@ union acpi_subtable_headers;
 int __init acpi_parse_mp_wake(union acpi_subtable_headers *header,
 			      const unsigned long end);
 
-void asm_acpi_mp_play_dead(u64 reset_vector, u64 pgd_pa);
 
 /*
  * Check if the CPU can handle C2 and deeper
arch/x86/include/asm/madt_wakeup.h new file +16 -0
new file mode 100644
index 000000000000..a8cd50af581a
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_X86_MADT_WAKEUP_H
+#define __ASM_X86_MADT_WAKEUP_H
+
+void asm_acpi_mp_play_dead(u64 reset_vector, u64 pgd_pa);
+
+#if defined(CONFIG_OF) && defined(CONFIG_ACPI_MADT_WAKEUP)
+u64 dtb_parse_mp_wake(void);
+#else
+static inline u64 dtb_parse_mp_wake(void)
+{
+	return 0;
+}
+#endif
+
+#endif /* __ASM_X86_MADT_WAKEUP_H */
arch/x86/kernel/madt_wakeup.c +38 -0
index d5ef6215583b..7257e7484569 100644
@@ -14,6 +14,8 @@
 #include <asm/nmi.h>
 #include <asm/processor.h>
 #include <asm/reboot.h>
+#include <asm/madt_wakeup.h>
+#include <asm/prom.h>
 
 /* Physical address of the Multiprocessor Wakeup Structure mailbox */
 static u64 acpi_mp_wake_mailbox_paddr __ro_after_init;
@@ -122,6 +124,7 @@ static int __init init_transition_pgtable(pgd_t *pgd)
 	return 0;
 }
 
+#ifdef CONFIG_ACPI
 static int __init acpi_mp_setup_reset(u64 reset_vector)
 {
 	struct x86_mapping_info info = {
@@ -168,6 +171,7 @@ static int __init acpi_mp_setup_reset(u64 reset_vector)
 
 	return 0;
 }
+#endif
 
 static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip)
 {
@@ -226,6 +230,7 @@ static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip)
 	return 0;
 }
 
+#ifdef CONFIG_ACPI
 static void acpi_mp_disable_offlining(struct acpi_madt_multiproc_wakeup *mp_wake)
 {
 	cpu_hotplug_disable_offlining();
@@ -290,3 +295,36 @@ int __init acpi_parse_mp_wake(union acpi_subtable_headers *header,
 
 	return 0;
 }
+#endif /* CONFIG_ACPI */
+
+#ifdef CONFIG_OF
+u64 __init dtb_parse_mp_wake(void)
+{
+	struct device_node *node;
+	u64 mailaddr = 0;
+
+	node = of_find_node_by_path("/cpus");
+	if (!node)
+		return 0;
+
+	if (of_property_match_string(node, "enable-method", "acpi-wakeup-mailbox") < 0) {
+		pr_err("No acpi wakeup mailbox enable-method\n");
+		goto done;
+	}
+
+	if (of_property_read_u64(node, "wakeup-mailbox-addr", &mailaddr)) {
+		pr_err("Invalid wakeup mailbox addr\n");
+		goto done;
+	}
+	acpi_mp_wake_mailbox_paddr = mailaddr;
+	pr_info("dt wakeup-mailbox: addr 0x%llx\n", mailaddr);
+
+	/* No support for the MADT reset vector yet */
+	cpu_hotplug_disable_offlining();
+	apic_update_callback(wakeup_secondary_cpu_64, acpi_wakeup_cpu);
+
+done:
+	of_node_put(node);
+	return mailaddr;
+}
+#endif /* CONFIG_OF */
Signature · 1 line
2.25.1
#17 lore raw

Re: [PATCH v2 3/9] x86/dt: Support the ACPI multiprocessor wakeup for device tree

kernel test robot noticed the following build warnings:

Hi Yunhong,

kernel test robot noticed the following build warnings:

[auto build test WARNING on fedb9ddeb1445f85d8f691b20f3faaa6dab8dd3f]

url:    https://github.com/intel-lab-lkp/linux/commits/Yunhong-Jiang/x86-acpi-Move-ACPI-MADT-wakeup-to-generic-code/20240826-135414
base:   fedb9ddeb1445f85d8f691b20f3faaa6dab8dd3f
patch link:    https://lore.kernel.org/r/20240823232327.2408869-4-yunhong.jiang%40linux.intel.com
patch subject: [PATCH v2 3/9] x86/dt: Support the ACPI multiprocessor wakeup for device tree
config: x86_64-buildonly-randconfig-006-20240831 (https://download.01.org/0day-ci/archive/20240831/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240831/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> arch/x86/kernel/madt_wakeup.c:81:19: warning: 'init_transition_pgtable' defined but not used [-Wunused-function]
      81 | static int __init init_transition_pgtable(pgd_t *pgd)
         |                   ^~~~~~~~~~~~~~~~~~~~~~~
>> arch/x86/kernel/madt_wakeup.c:70:20: warning: 'free_pgt_page' defined but not used [-Wunused-function]
      70 | static void __init free_pgt_page(void *pgt, void *dummy)
         |                    ^~~~~~~~~~~~~
>> arch/x86/kernel/madt_wakeup.c:40:13: warning: 'acpi_mp_cpu_die' defined but not used [-Wunused-function]
      40 | static void acpi_mp_cpu_die(unsigned int cpu)
         |             ^~~~~~~~~~~~~~~
>> arch/x86/kernel/madt_wakeup.c:34:13: warning: 'acpi_mp_play_dead' defined but not used [-Wunused-function]
      34 | static void acpi_mp_play_dead(void)
         |             ^~~~~~~~~~~~~~~~~
>> arch/x86/kernel/madt_wakeup.c:29:13: warning: 'acpi_mp_stop_this_cpu' defined but not used [-Wunused-function]

29 | static void acpi_mp_stop_this_cpu(void)
| ^~~~~~~~~~~~~~~~~~~~~

vim +/init_transition_pgtable +81 arch/x86/kernel/madt_wakeup.c

1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14 28

1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  @29  static void acpi_mp_stop_this_cpu(void)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   30  {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   31  	asm_acpi_mp_play_dead(acpi_mp_reset_vector_paddr, acpi_mp_pgd);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   32  }
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   33  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  @34  static void acpi_mp_play_dead(void)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   35  {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   36  	play_dead_common();
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   37  	asm_acpi_mp_play_dead(acpi_mp_reset_vector_paddr, acpi_mp_pgd);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   38  }
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   39  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  @40  static void acpi_mp_cpu_die(unsigned int cpu)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   41  {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   42  	u32 apicid = per_cpu(x86_cpu_to_apicid, cpu);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   43  	unsigned long timeout;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   44  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   45  	/*
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   46  	 * Use TEST mailbox command to prove that BIOS got control over
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   47  	 * the CPU before declaring it dead.
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   48  	 *
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   49  	 * BIOS has to clear 'command' field of the mailbox.
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   50  	 */
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   51  	acpi_mp_wake_mailbox->apic_id = apicid;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   52  	smp_store_release(&acpi_mp_wake_mailbox->command,
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   53  			  ACPI_MP_WAKE_COMMAND_TEST);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   54  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   55  	/* Don't wait longer than a second. */
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   56  	timeout = USEC_PER_SEC;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   57  	while (READ_ONCE(acpi_mp_wake_mailbox->command) && --timeout)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   58  		udelay(1);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   59  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   60  	if (!timeout)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   61  		pr_err("Failed to hand over CPU %d to BIOS\n", cpu);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   62  }
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   63  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   64  /* The argument is required to match type of x86_mapping_info::alloc_pgt_page */
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   65  static void __init *alloc_pgt_page(void *dummy)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   66  {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   67  	return memblock_alloc(PAGE_SIZE, PAGE_SIZE);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   68  }
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   69  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  @70  static void __init free_pgt_page(void *pgt, void *dummy)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   71  {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   72  	return memblock_free(pgt, PAGE_SIZE);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   73  }
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   74  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   75  /*
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   76   * Make sure asm_acpi_mp_play_dead() is present in the identity mapping at
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   77   * the same place as in the kernel page tables. asm_acpi_mp_play_dead() switches
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   78   * to the identity mapping and the function has be present at the same spot in
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   79   * the virtual address space before and after switching page tables.
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   80   */
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  @81  static int __init init_transition_pgtable(pgd_t *pgd)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   82  {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   83  	pgprot_t prot = PAGE_KERNEL_EXEC_NOENC;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   84  	unsigned long vaddr, paddr;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   85  	p4d_t *p4d;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   86  	pud_t *pud;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   87  	pmd_t *pmd;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   88  	pte_t *pte;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   89  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   90  	vaddr = (unsigned long)asm_acpi_mp_play_dead;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   91  	pgd += pgd_index(vaddr);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   92  	if (!pgd_present(*pgd)) {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   93  		p4d = (p4d_t *)alloc_pgt_page(NULL);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   94  		if (!p4d)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   95  			return -ENOMEM;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   96  		set_pgd(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE));
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   97  	}
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   98  	p4d = p4d_offset(pgd, vaddr);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14   99  	if (!p4d_present(*p4d)) {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  100  		pud = (pud_t *)alloc_pgt_page(NULL);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  101  		if (!pud)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  102  			return -ENOMEM;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  103  		set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE));
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  104  	}
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  105  	pud = pud_offset(p4d, vaddr);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  106  	if (!pud_present(*pud)) {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  107  		pmd = (pmd_t *)alloc_pgt_page(NULL);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  108  		if (!pmd)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  109  			return -ENOMEM;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  110  		set_pud(pud, __pud(__pa(pmd) | _KERNPG_TABLE));
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  111  	}
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  112  	pmd = pmd_offset(pud, vaddr);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  113  	if (!pmd_present(*pmd)) {
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  114  		pte = (pte_t *)alloc_pgt_page(NULL);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  115  		if (!pte)
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  116  			return -ENOMEM;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  117  		set_pmd(pmd, __pmd(__pa(pte) | _KERNPG_TABLE));
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  118  	}
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  119  	pte = pte_offset_kernel(pmd, vaddr);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  120  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  121  	paddr = __pa(vaddr);
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  122  	set_pte(pte, pfn_pte(paddr >> PAGE_SHIFT, prot));
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  123  
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  124  	return 0;
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  125  }
1ceebe2e46720d arch/x86/kernel/acpi/madt_wakeup.c Kirill A. Shutemov 2024-06-14  126  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#18 lore raw

RE: [PATCH v2 3/9] x86/dt: Support the ACPI multiprocessor wakeup for device tree

In reply to Yunhong Jiang

From: Yunhong Jiang <[email protected]> Sent: Friday, August 23, 2024 4:23 PM

Quoted 114 lines

When a TDX guest boots with the device tree instead of ACPI, it can reuse the ACPI multiprocessor wakeup mechanism to wake up application processors(AP), without introducing a new mechanism from scrach.

In the ACPI spec, two structures are defined to wake up the APs: the multiprocessor wakeup structure and the multiprocessor wakeup mailbox structure. The multiprocessor wakeup structure is passed to OS through a Multiple APIC Description Table(MADT), one field specifying the physical address of the multiprocessor wakeup mailbox structure. The OS sends a message to firmware through the multiprocessor wakeup mailbox structure, to bring up the APs.

In device tree environment, the multiprocessor wakeup structure is not used, to reduce the dependency on the generic ACPI table. The information defined in this structure is defined in the properties of cpus node in the device tree. The "wakeup-mailbox-addr" property specifies the physical address of the multiprocessor wakeup mailbox structure. The OS will follow the ACPI spec to send the message to the firmware to bring up the APs.

---
 MAINTAINERS                        |  1 +
 arch/x86/Kconfig                   |  2 +-
 arch/x86/include/asm/acpi.h        |  1 -
 arch/x86/include/asm/madt_wakeup.h | 16 +++++++++++++
 arch/x86/kernel/madt_wakeup.c      | 38 ++++++++++++++++++++++++++++++
 5 files changed, 56 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/madt_wakeup.h
MAINTAINERS +0 -0
index 5555a3bbac5f..900de6501eba 100644
@@ -288,6 +288,7 @@ T:	git

git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
F: Documentation/ABI/testing/configfs-acpi
F: Documentation/ABI/testing/sysfs-bus-acpi
F: Documentation/firmware-guide/acpi/

MAINTAINERS +1 -0
+F:	arch/x86/include/asm/madt_wakeup.h
 F:	arch/x86/kernel/acpi/
 F:	arch/x86/kernel/madt_playdead.S
 F:	arch/x86/kernel/madt_wakeup.c
arch/x86/Kconfig +1 -1
index d422247b2882..dba46dd30049 100644
@@ -1123,7 +1123,7 @@ config X86_LOCAL_APIC
 config ACPI_MADT_WAKEUP
 	def_bool y
 	depends on X86_64
-	depends on ACPI
+	depends on ACPI || OF
 	depends on SMP
 	depends on X86_LOCAL_APIC
 
arch/x86/include/asm/acpi.h +0 -1
index 21bc53f5ed0c..0e082303ca26 100644
@@ -83,7 +83,6 @@ union acpi_subtable_headers;
 int __init acpi_parse_mp_wake(union acpi_subtable_headers *header,
 			      const unsigned long end);
 
-void asm_acpi_mp_play_dead(u64 reset_vector, u64 pgd_pa);
 
 /*
  * Check if the CPU can handle C2 and deeper
arch/x86/include/asm/madt_wakeup.h new file +16 -0
new file mode 100644
index 000000000000..a8cd50af581a
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_X86_MADT_WAKEUP_H
+#define __ASM_X86_MADT_WAKEUP_H
+
+void asm_acpi_mp_play_dead(u64 reset_vector, u64 pgd_pa);
+
+#if defined(CONFIG_OF) && defined(CONFIG_ACPI_MADT_WAKEUP)
+u64 dtb_parse_mp_wake(void);
+#else
+static inline u64 dtb_parse_mp_wake(void)
+{
+	return 0;
+}
+#endif
+
+#endif /* __ASM_X86_MADT_WAKEUP_H */
arch/x86/kernel/madt_wakeup.c +3 -0
index d5ef6215583b..7257e7484569 100644
@@ -14,6 +14,8 @@
 #include <asm/nmi.h>
 #include <asm/processor.h>
 #include <asm/reboot.h>
+#include <asm/madt_wakeup.h>
+#include <asm/prom.h>
 
 /* Physical address of the Multiprocessor Wakeup Structure mailbox */
 static u64 acpi_mp_wake_mailbox_paddr __ro_after_init;
@@ -122,6 +124,7 @@ static int __init init_transition_pgtable(pgd_t *pgd)
 	return 0;
 }
 
+#ifdef CONFIG_ACPI
 static int __init acpi_mp_setup_reset(u64 reset_vector)
 {
 	struct x86_mapping_info info = {

arch/x86/kernel/madt_wakeup.c static int __init acpi_mp_setup_reset(u64 reset_vector) · line 174

Quoted 4 lines
arch/x86/kernel/madt_wakeup.c +1 -0
@@ -168,6 +171,7 @@ static int __init acpi_mp_setup_reset(u64 reset_vector)
 
 	return 0;
 }
+#endif

When acpi_mp_setup_reset() is #ifdef'ed out because of CONFIG_ACPI not being set, doesn't this generate compile warnings about init_transition_pgtable(), alloc_pgt_page(), free_pgt_page(), etc. being unused?

It appears that the only code in madt_wakeup.c that is shared between the ACPI and OF cases is acpi_wakeup_cpu()? Is that correct?

Quoted 20 lines
arch/x86/kernel/madt_wakeup.c +6 -0
 static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip)
 {
@@ -226,6 +230,7 @@ static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip)
 	return 0;
 }
 
+#ifdef CONFIG_ACPI
 static void acpi_mp_disable_offlining(struct acpi_madt_multiproc_wakeup *mp_wake)
 {
 	cpu_hotplug_disable_offlining();
@@ -290,3 +295,36 @@ int __init acpi_parse_mp_wake(union acpi_subtable_headers *header,
 
 	return 0;
 }
+#endif /* CONFIG_ACPI */
+
+#ifdef CONFIG_OF
+u64 __init dtb_parse_mp_wake(void)
+{

arch/x86/kernel/madt_wakeup.c int __init acpi_parse_mp_wake(union acpi_subtable_headers *header, · line 312

Quoted 10 lines
arch/x86/kernel/madt_wakeup.c +10 -0
+	struct device_node *node;
+	u64 mailaddr = 0;
+
+	node = of_find_node_by_path("/cpus");
+	if (!node)
+		return 0;
+
+	if (of_property_match_string(node, "enable-method", "acpi-wakeup-mailbox") < 0) {
+		pr_err("No acpi wakeup mailbox enable-method\n");
+		goto done;

Patch 4 of this series unconditionally calls dtb_parse_mp_wake(), so it will be called in normal VMs, as a well as SEV-SNP and TDX kernels built for VTL 2 (assuming CONFIG_OF is set). Normal VMs presumably won't be using DT and won't have the "/cpus" node, so this function will silently do nothing, which is fine. But do you expect the DT "/cpus" node to be present in an SEV-SNP VTL 2 environment? If so, this function will either output some spurious error messages, or SEV-SNP will use the ACPI wakeup mailbox method, which is probably not what you want.

Michael

Quoted 21 lines
arch/x86/kernel/madt_wakeup.c +18 -1
+	}
+
+	if (of_property_read_u64(node, "wakeup-mailbox-addr", &mailaddr)) {
+		pr_err("Invalid wakeup mailbox addr\n");
+		goto done;
+	}
+	acpi_mp_wake_mailbox_paddr = mailaddr;
+	pr_info("dt wakeup-mailbox: addr 0x%llx\n", mailaddr);
+
+	/* No support for the MADT reset vector yet */
+	cpu_hotplug_disable_offlining();
+	apic_update_callback(wakeup_secondary_cpu_64, acpi_wakeup_cpu);
+
+done:
+	of_node_put(node);
+	return mailaddr;
+}
+#endif /* CONFIG_OF */
--

2.25.1

#19 lore raw

Re: [PATCH v2 3/9] x86/dt: Support the ACPI multiprocessor wakeup for device tree

Quoted 125 lines · Michael Kelley

From: Yunhong Jiang <[email protected]> Sent: Friday, August 23, 2024 4:23 PM

Quoted 114 lines

When a TDX guest boots with the device tree instead of ACPI, it can reuse the ACPI multiprocessor wakeup mechanism to wake up application processors(AP), without introducing a new mechanism from scrach.

In the ACPI spec, two structures are defined to wake up the APs: the multiprocessor wakeup structure and the multiprocessor wakeup mailbox structure. The multiprocessor wakeup structure is passed to OS through a Multiple APIC Description Table(MADT), one field specifying the physical address of the multiprocessor wakeup mailbox structure. The OS sends a message to firmware through the multiprocessor wakeup mailbox structure, to bring up the APs.

In device tree environment, the multiprocessor wakeup structure is not used, to reduce the dependency on the generic ACPI table. The information defined in this structure is defined in the properties of cpus node in the device tree. The "wakeup-mailbox-addr" property specifies the physical address of the multiprocessor wakeup mailbox structure. The OS will follow the ACPI spec to send the message to the firmware to bring up the APs.

---
 MAINTAINERS                        |  1 +
 arch/x86/Kconfig                   |  2 +-
 arch/x86/include/asm/acpi.h        |  1 -
 arch/x86/include/asm/madt_wakeup.h | 16 +++++++++++++
 arch/x86/kernel/madt_wakeup.c      | 38 ++++++++++++++++++++++++++++++
 5 files changed, 56 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/madt_wakeup.h
MAINTAINERS +0 -0
index 5555a3bbac5f..900de6501eba 100644
@@ -288,6 +288,7 @@ T:	git

git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
F: Documentation/ABI/testing/configfs-acpi
F: Documentation/ABI/testing/sysfs-bus-acpi
F: Documentation/firmware-guide/acpi/

MAINTAINERS +1 -0
+F:	arch/x86/include/asm/madt_wakeup.h
 F:	arch/x86/kernel/acpi/
 F:	arch/x86/kernel/madt_playdead.S
 F:	arch/x86/kernel/madt_wakeup.c
arch/x86/Kconfig +1 -1
index d422247b2882..dba46dd30049 100644
@@ -1123,7 +1123,7 @@ config X86_LOCAL_APIC
 config ACPI_MADT_WAKEUP
 	def_bool y
 	depends on X86_64
-	depends on ACPI
+	depends on ACPI || OF
 	depends on SMP
 	depends on X86_LOCAL_APIC
 
arch/x86/include/asm/acpi.h +0 -1
index 21bc53f5ed0c..0e082303ca26 100644
@@ -83,7 +83,6 @@ union acpi_subtable_headers;
 int __init acpi_parse_mp_wake(union acpi_subtable_headers *header,
 			      const unsigned long end);
 
-void asm_acpi_mp_play_dead(u64 reset_vector, u64 pgd_pa);
 
 /*
  * Check if the CPU can handle C2 and deeper
arch/x86/include/asm/madt_wakeup.h new file +16 -0
new file mode 100644
index 000000000000..a8cd50af581a
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_X86_MADT_WAKEUP_H
+#define __ASM_X86_MADT_WAKEUP_H
+
+void asm_acpi_mp_play_dead(u64 reset_vector, u64 pgd_pa);
+
+#if defined(CONFIG_OF) && defined(CONFIG_ACPI_MADT_WAKEUP)
+u64 dtb_parse_mp_wake(void);
+#else
+static inline u64 dtb_parse_mp_wake(void)
+{
+	return 0;
+}
+#endif
+
+#endif /* __ASM_X86_MADT_WAKEUP_H */
arch/x86/kernel/madt_wakeup.c +3 -0
index d5ef6215583b..7257e7484569 100644
@@ -14,6 +14,8 @@
 #include <asm/nmi.h>
 #include <asm/processor.h>
 #include <asm/reboot.h>
+#include <asm/madt_wakeup.h>
+#include <asm/prom.h>
 
 /* Physical address of the Multiprocessor Wakeup Structure mailbox */
 static u64 acpi_mp_wake_mailbox_paddr __ro_after_init;
@@ -122,6 +124,7 @@ static int __init init_transition_pgtable(pgd_t *pgd)
 	return 0;
 }
 
+#ifdef CONFIG_ACPI
 static int __init acpi_mp_setup_reset(u64 reset_vector)
 {
 	struct x86_mapping_info info = {

arch/x86/kernel/madt_wakeup.c static int __init acpi_mp_setup_reset(u64 reset_vector) · line 174

Quoted 4 lines
arch/x86/kernel/madt_wakeup.c +1 -0
@@ -168,6 +171,7 @@ static int __init acpi_mp_setup_reset(u64 reset_vector)
 
 	return 0;
 }
+#endif

When acpi_mp_setup_reset() is #ifdef'ed out because of CONFIG_ACPI not being set, doesn't this generate compile warnings about init_transition_pgtable(), alloc_pgt_page(), free_pgt_page(), etc. being unused?

Quoted 2 lines

It appears that the only code in madt_wakeup.c that is shared between the ACPI and OF cases is acpi_wakeup_cpu()? Is that correct?

Yes, the acpi_wakeup_cpu() is the only code. Interestingly that I don't see compilation warning for the functions you listed like alloc_pgt_page()/free_pgt_page() etc when built with CONFIG_ACPI disabled. Will check in deep and figure out the reason.

Quoted 42 lines
Quoted 21 lines
arch/x86/kernel/madt_wakeup.c +6 -0
 static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip)
 {
@@ -226,6 +230,7 @@ static int acpi_wakeup_cpu(u32 apicid, unsigned long start_ip)
 	return 0;
 }
 
+#ifdef CONFIG_ACPI
 static void acpi_mp_disable_offlining(struct acpi_madt_multiproc_wakeup *mp_wake)
 {
 	cpu_hotplug_disable_offlining();
@@ -290,3 +295,36 @@ int __init acpi_parse_mp_wake(union acpi_subtable_headers *header,
 
 	return 0;
 }
+#endif /* CONFIG_ACPI */
+
+#ifdef CONFIG_OF
+u64 __init dtb_parse_mp_wake(void)
+{

arch/x86/kernel/madt_wakeup.c int __init acpi_parse_mp_wake(union acpi_subtable_headers *header, · line 312

Quoted 10 lines
arch/x86/kernel/madt_wakeup.c +10 -0
+	struct device_node *node;
+	u64 mailaddr = 0;
+
+	node = of_find_node_by_path("/cpus");
+	if (!node)
+		return 0;
+
+	if (of_property_match_string(node, "enable-method", "acpi-wakeup-mailbox") < 0) {
+		pr_err("No acpi wakeup mailbox enable-method\n");
+		goto done;

Patch 4 of this series unconditionally calls dtb_parse_mp_wake(), so it will be called in normal VMs, as a well as SEV-SNP and TDX kernels built for VTL 2 (assuming CONFIG_OF is set). Normal VMs presumably won't be using DT and won't have the "/cpus" node, so this function will silently do nothing, which is fine. But do you expect the DT "/cpus" node to be present in an SEV-SNP VTL 2 environment? If so, this function will either output some spurious error messages, or SEV-SNP will use the ACPI wakeup mailbox method, which is probably not what you want.

Quoted 1 line

Michael

Yes, will fix the spurios error messages. Thank you for pointing out this.

Thanks
--jyh

Quoted 23 lines
Quoted 22 lines
arch/x86/kernel/madt_wakeup.c +18 -1
+	}
+
+	if (of_property_read_u64(node, "wakeup-mailbox-addr", &mailaddr)) {
+		pr_err("Invalid wakeup mailbox addr\n");
+		goto done;
+	}
+	acpi_mp_wake_mailbox_paddr = mailaddr;
+	pr_info("dt wakeup-mailbox: addr 0x%llx\n", mailaddr);
+
+	/* No support for the MADT reset vector yet */
+	cpu_hotplug_disable_offlining();
+	apic_update_callback(wakeup_secondary_cpu_64, acpi_wakeup_cpu);
+
+done:
+	of_node_put(node);
+	return mailaddr;
+}
+#endif /* CONFIG_OF */
--

2.25.1

#20 lore raw

PATCH v2 4/9 x86/hyperv: Parse the ACPI wakeup mailbox

In reply to Yunhong Jiang

Parse the wakeup mailbox VTL2 TDX guest. Put it to the guest_late_init, so that it will be invoked before hyperv_init() where the mailbox address is checked.

---
 arch/x86/include/asm/mshyperv.h | 3 +++
 arch/x86/kernel/cpu/mshyperv.c  | 2 ++
 drivers/hv/hv_common.c          | 8 ++++++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/mshyperv.h +3 -0
index 390c4d13956d..5178b96c7fc9 100644
@@ -10,6 +10,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/paravirt.h>
 #include <asm/mshyperv.h>
+#include <asm/madt_wakeup.h>
 
 /*
  * Hyper-V always provides a single IO-APIC at this MMIO address.
@@ -49,6 +50,8 @@ extern u64 hv_current_partition_id;
 
 extern union hv_ghcb * __percpu *hv_ghcb_pg;
 
+extern u64 wakeup_mailbox_addr;
+
 bool hv_isolation_type_snp(void);
 bool hv_isolation_type_tdx(void);
 u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2);
arch/x86/kernel/cpu/mshyperv.c +2 -0
index 3d4237f27569..f6b727b4bd0b 100644
@@ -43,6 +43,8 @@ struct ms_hyperv_info ms_hyperv;
 bool hyperv_paravisor_present __ro_after_init;
 EXPORT_SYMBOL_GPL(hyperv_paravisor_present);
 
+u64 wakeup_mailbox_addr;
+
 #if IS_ENABLED(CONFIG_HYPERV)
 static inline unsigned int hv_get_nested_msr(unsigned int reg)
 {
drivers/hv/hv_common.c +8 -0
index 9c452bfbd571..14b005b6270f 100644
@@ -365,6 +365,14 @@ void __init ms_hyperv_late_init(void)
 	u8 *randomdata;
 	u32 length, i;
 
+	/*
+	 * Parse the ACPI wakeup structure information from device tree.
+	 * Currently VTL2 TDX guest only.
+	 */
+#ifdef CONFIG_X86_64
+	wakeup_mailbox_addr = dtb_parse_mp_wake();
+#endif
+
 	/*
 	 * Seed the Linux random number generator with entropy provided by
 	 * the Hyper-V host in ACPI table OEM0.
Signature · 1 line
2.25.1
#21 lore raw

RE: [PATCH v2 4/9] x86/hyperv: Parse the ACPI wakeup mailbox

From: Yunhong Jiang <[email protected]>

Quoted 4 lines

Parse the wakeup mailbox VTL2 TDX guest. Put it to the guest_late_init, so that it will be invoked before hyperv_init() where the mailbox address is checked.

Could you elaborate on the choice to set the wakeup_mailbox_address in ms_hyperv_late_init()? The code in hv_common.c is intended to be code that is architecture neutral (see the comment at the top of the module), so it's a red flag to see #ifdef CONFIG_X86_64. Couldn't the wakeup_mailbox_address be set in the x86 version of hyperv_init() before it is needed?

Quoted 34 lines
---
 arch/x86/include/asm/mshyperv.h | 3 +++
 arch/x86/kernel/cpu/mshyperv.c  | 2 ++
 drivers/hv/hv_common.c          | 8 ++++++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/mshyperv.h +3 -0
index 390c4d13956d..5178b96c7fc9 100644
@@ -10,6 +10,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/paravirt.h>
 #include <asm/mshyperv.h>
+#include <asm/madt_wakeup.h>
 
 /*
  * Hyper-V always provides a single IO-APIC at this MMIO address.
@@ -49,6 +50,8 @@ extern u64 hv_current_partition_id;
 
 extern union hv_ghcb * __percpu *hv_ghcb_pg;
 
+extern u64 wakeup_mailbox_addr;
+
 bool hv_isolation_type_snp(void);
 bool hv_isolation_type_tdx(void);
 u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2);
arch/x86/kernel/cpu/mshyperv.c +0 -0
index 3d4237f27569..f6b727b4bd0b 100644

arch/x86/kernel/cpu/mshyperv.c struct ms_hyperv_info ms_hyperv; · line 46

Quoted 4 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
@@ -43,6 +43,8 @@ struct ms_hyperv_info ms_hyperv;
 bool hyperv_paravisor_present __ro_after_init;
 EXPORT_SYMBOL_GPL(hyperv_paravisor_present);
 
+u64 wakeup_mailbox_addr;

This value duplicates acpi_mp_wake_mailbox_paddr in madt_wakeup.c. It looks like the duplicate value is used for two things:

1) In hv_is_private_mmio_tdx() to control the encrypted vs. decrypted mapping (Patch 5 of this series)

2) As a boolean in hv_vtl_early_init() to avoid overwriting the wakeup_secondary_cpu_64 value when dtb_parse_mp_wake() has set it to acpi_wakeup_cpu(). (Patch 9 of this series).

Having a duplicate value is messy, and I'm wondering if it can be avoided. For (1), hv_private_mmio_tdx() could call into a function added to madt_wakeup.c to make the check. For (2), the check should probably be based on hv_isolation_type_tdx() instead of whether the wakeup mailbox address is set. I'll note that Patch 5 of this series is using hv_isolation_type_tdx(), so there's a bit of an inconsistency in testing the wakeup_mailbox_addr in Patch 9.

This is just a suggestion, as I haven't worked out all the details. If you think it ends up being messier than the duplicate value, then I'm OK with it.

Michael

Quoted 26 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
+
 #if IS_ENABLED(CONFIG_HYPERV)
 static inline unsigned int hv_get_nested_msr(unsigned int reg)
 {
drivers/hv/hv_common.c +8 -1
index 9c452bfbd571..14b005b6270f 100644
@@ -365,6 +365,14 @@ void __init ms_hyperv_late_init(void)
 	u8 *randomdata;
 	u32 length, i;
 
+	/*
+	 * Parse the ACPI wakeup structure information from device tree.
+	 * Currently VTL2 TDX guest only.
+	 */
+#ifdef CONFIG_X86_64
+	wakeup_mailbox_addr = dtb_parse_mp_wake();
+#endif
+
 	/*
 	 * Seed the Linux random number generator with entropy provided by
 	 * the Hyper-V host in ACPI table OEM0.
--

2.25.1

#22 lore raw

Re: [PATCH v2 4/9] x86/hyperv: Parse the ACPI wakeup mailbox

Quoted 6 lines · Michael Kelley

From: Yunhong Jiang <[email protected]>

Quoted 4 lines

Parse the wakeup mailbox VTL2 TDX guest. Put it to the guest_late_init, so that it will be invoked before hyperv_init() where the mailbox address is checked.

Quoted 6 lines

Could you elaborate on the choice to set the wakeup_mailbox_address in ms_hyperv_late_init()? The code in hv_common.c is intended to be code that is architecture neutral (see the comment at the top of the module), so it's a red flag to see #ifdef CONFIG_X86_64. Couldn't the wakeup_mailbox_address be set in the x86 version of hyperv_init() before it is needed?

Sure, will try to put it in hyperv_init() before it's needed.

Quoted 52 lines
Quoted 35 lines
---
 arch/x86/include/asm/mshyperv.h | 3 +++
 arch/x86/kernel/cpu/mshyperv.c  | 2 ++
 drivers/hv/hv_common.c          | 8 ++++++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/mshyperv.h +3 -0
index 390c4d13956d..5178b96c7fc9 100644
@@ -10,6 +10,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/paravirt.h>
 #include <asm/mshyperv.h>
+#include <asm/madt_wakeup.h>
 
 /*
  * Hyper-V always provides a single IO-APIC at this MMIO address.
@@ -49,6 +50,8 @@ extern u64 hv_current_partition_id;
 
 extern union hv_ghcb * __percpu *hv_ghcb_pg;
 
+extern u64 wakeup_mailbox_addr;
+
 bool hv_isolation_type_snp(void);
 bool hv_isolation_type_tdx(void);
 u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2);
arch/x86/kernel/cpu/mshyperv.c +0 -0
index 3d4237f27569..f6b727b4bd0b 100644

arch/x86/kernel/cpu/mshyperv.c struct ms_hyperv_info ms_hyperv; · line 46

Quoted 4 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
@@ -43,6 +43,8 @@ struct ms_hyperv_info ms_hyperv;
 bool hyperv_paravisor_present __ro_after_init;
 EXPORT_SYMBOL_GPL(hyperv_paravisor_present);
 
+u64 wakeup_mailbox_addr;

This value duplicates acpi_mp_wake_mailbox_paddr in madt_wakeup.c. It looks like the duplicate value is used for two things:

1) In hv_is_private_mmio_tdx() to control the encrypted vs. decrypted mapping (Patch 5 of this series)

2) As a boolean in hv_vtl_early_init() to avoid overwriting the wakeup_secondary_cpu_64 value when dtb_parse_mp_wake() has set it to acpi_wakeup_cpu(). (Patch 9 of this series).

Quoted 9 lines

Having a duplicate value is messy, and I'm wondering if it can be avoided. For (1), hv_private_mmio_tdx() could call into a function added to madt_wakeup.c to make the check. For (2), the check should probably be based on hv_isolation_type_tdx() instead of whether the wakeup mailbox address is set. I'll note that Patch 5 of this series is using hv_isolation_type_tdx(), so there's a bit of an inconsistency in testing the wakeup_mailbox_addr in Patch 9.

I think your comment includes two points, the duplicated variables and the incosistency in the testing.

Thank you for pointing out the duplication of wakeup_mailbox_addr with acpi_mp_wake_mailbox_paddr. I didn't realize it. Yes, such duplication should be avoided and will fix it in next submission.

Agree the inconsistency in testing wakeup_mailbox_addr and hv_isolation_type_tdx() is not good. IMHO, the wakeup_mailbox_addr (or the new function you proposed) is better than hv_isolation_type_tdx(), since the wakeup_mailbox_addr is more directly related. But hv_vtl_init_platform() happens before DT parse, thus I have to use the hv_isolation_type_tdx() in it. I don't have a good idea on how to fix this.

Thanks
--jyh

Quoted 34 lines

This is just a suggestion, as I haven't worked out all the details. If you think it ends up being messier than the duplicate value, then I'm OK with it.

Michael

Quoted 26 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
+
 #if IS_ENABLED(CONFIG_HYPERV)
 static inline unsigned int hv_get_nested_msr(unsigned int reg)
 {
drivers/hv/hv_common.c +8 -1
index 9c452bfbd571..14b005b6270f 100644
@@ -365,6 +365,14 @@ void __init ms_hyperv_late_init(void)
 	u8 *randomdata;
 	u32 length, i;
 
+	/*
+	 * Parse the ACPI wakeup structure information from device tree.
+	 * Currently VTL2 TDX guest only.
+	 */
+#ifdef CONFIG_X86_64
+	wakeup_mailbox_addr = dtb_parse_mp_wake();
+#endif
+
 	/*
 	 * Seed the Linux random number generator with entropy provided by
 	 * the Hyper-V host in ACPI table OEM0.
--

2.25.1

#23 lore raw

RE: [PATCH v2 4/9] x86/hyperv: Parse the ACPI wakeup mailbox

From: Yunhong Jiang <[email protected]> Sent: Tuesday, September 3, 2024 1:19 PM

Quoted 92 lines · Michael Kelley
Quoted 6 lines

From: Yunhong Jiang <[email protected]>

Quoted 4 lines

Parse the wakeup mailbox VTL2 TDX guest. Put it to the guest_late_init, so that it will be invoked before hyperv_init() where the mailbox address is checked.

Quoted 6 lines

Could you elaborate on the choice to set the wakeup_mailbox_address in ms_hyperv_late_init()? The code in hv_common.c is intended to be code that is architecture neutral (see the comment at the top of the module), so it's a red flag to see #ifdef CONFIG_X86_64. Couldn't the wakeup_mailbox_address be set in the x86 version of hyperv_init() before it is needed?

Sure, will try to put it in hyperv_init() before it's needed.

Quoted 52 lines
Quoted 35 lines
---
 arch/x86/include/asm/mshyperv.h | 3 +++
 arch/x86/kernel/cpu/mshyperv.c  | 2 ++
 drivers/hv/hv_common.c          | 8 ++++++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/mshyperv.h +3 -0
index 390c4d13956d..5178b96c7fc9 100644
@@ -10,6 +10,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/paravirt.h>
 #include <asm/mshyperv.h>
+#include <asm/madt_wakeup.h>
 
 /*
  * Hyper-V always provides a single IO-APIC at this MMIO address.
@@ -49,6 +50,8 @@ extern u64 hv_current_partition_id;
 
 extern union hv_ghcb * __percpu *hv_ghcb_pg;
 
+extern u64 wakeup_mailbox_addr;
+
 bool hv_isolation_type_snp(void);
 bool hv_isolation_type_tdx(void);
 u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2);
arch/x86/kernel/cpu/mshyperv.c +0 -0
index 3d4237f27569..f6b727b4bd0b 100644

arch/x86/kernel/cpu/mshyperv.c struct ms_hyperv_info ms_hyperv; · line 46

Quoted 4 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
@@ -43,6 +43,8 @@ struct ms_hyperv_info ms_hyperv;
 bool hyperv_paravisor_present __ro_after_init;
 EXPORT_SYMBOL_GPL(hyperv_paravisor_present);
 
+u64 wakeup_mailbox_addr;

This value duplicates acpi_mp_wake_mailbox_paddr in madt_wakeup.c. It looks like the duplicate value is used for two things:

1) In hv_is_private_mmio_tdx() to control the encrypted vs. decrypted mapping (Patch 5 of this series)

2) As a boolean in hv_vtl_early_init() to avoid overwriting the wakeup_secondary_cpu_64 value when dtb_parse_mp_wake() has set it to acpi_wakeup_cpu(). (Patch 9 of this series).

Quoted 9 lines

Having a duplicate value is messy, and I'm wondering if it can be avoided. For (1), hv_private_mmio_tdx() could call into a function added to madt_wakeup.c to make the check. For (2), the check should probably be based on hv_isolation_type_tdx() instead of whether the wakeup mailbox address is set. I'll note that Patch 5 of this series is using hv_isolation_type_tdx(), so there's a bit of an inconsistency in testing the wakeup_mailbox_addr in Patch 9.

I think your comment includes two points, the duplicated variables and the incosistency in the testing.

Thank you for pointing out the duplication of wakeup_mailbox_addr with acpi_mp_wake_mailbox_paddr. I didn't realize it. Yes, such duplication should be avoided and will fix it in next submission.

Agree the inconsistency in testing wakeup_mailbox_addr and hv_isolation_type_tdx() is not good. IMHO, the wakeup_mailbox_addr (or the new function you proposed) is better than hv_isolation_type_tdx(), since the wakeup_mailbox_addr is more directly related. But hv_vtl_init_platform() happens before DT parse, thus I have to use the hv_isolation_type_tdx() in it. I don't have a good idea on how to fix this.

Quoted 3 lines

Thanks
--jyh

I don't think there's a requirement to set the "is_private_mmio" function in hv_vtl_init_platform(). It just needs to be set before acpi_wakeup_cpu() is called, which does the memremap() that will invoke the "is_private_mmio" function to decide whether to map as encrypted or decrypted.

So maybe setting the "is_private_mmio" function could be done after dtb_parse_mp_wake() is called in its new location, and you know you have a valid wake mailbox address? Again, I haven't worked out all the details, so this approach might be just as messy, but in a different way. Use your judgment ... :-)

Michael

#24 lore raw

Re: [PATCH v2 4/9] x86/hyperv: Parse the ACPI wakeup mailbox

Quoted 103 lines · Michael Kelley

From: Yunhong Jiang <[email protected]> Sent: Tuesday, September 3, 2024 1:19 PM

Quoted 92 lines · Michael Kelley
Quoted 6 lines

From: Yunhong Jiang <[email protected]>

Quoted 4 lines

Parse the wakeup mailbox VTL2 TDX guest. Put it to the guest_late_init, so that it will be invoked before hyperv_init() where the mailbox address is checked.

Quoted 6 lines

Could you elaborate on the choice to set the wakeup_mailbox_address in ms_hyperv_late_init()? The code in hv_common.c is intended to be code that is architecture neutral (see the comment at the top of the module), so it's a red flag to see #ifdef CONFIG_X86_64. Couldn't the wakeup_mailbox_address be set in the x86 version of hyperv_init() before it is needed?

Sure, will try to put it in hyperv_init() before it's needed.

Quoted 52 lines
Quoted 35 lines
---
 arch/x86/include/asm/mshyperv.h | 3 +++
 arch/x86/kernel/cpu/mshyperv.c  | 2 ++
 drivers/hv/hv_common.c          | 8 ++++++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/mshyperv.h +3 -0
index 390c4d13956d..5178b96c7fc9 100644
@@ -10,6 +10,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/paravirt.h>
 #include <asm/mshyperv.h>
+#include <asm/madt_wakeup.h>
 
 /*
  * Hyper-V always provides a single IO-APIC at this MMIO address.
@@ -49,6 +50,8 @@ extern u64 hv_current_partition_id;
 
 extern union hv_ghcb * __percpu *hv_ghcb_pg;
 
+extern u64 wakeup_mailbox_addr;
+
 bool hv_isolation_type_snp(void);
 bool hv_isolation_type_tdx(void);
 u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2);
arch/x86/kernel/cpu/mshyperv.c +0 -0
index 3d4237f27569..f6b727b4bd0b 100644

arch/x86/kernel/cpu/mshyperv.c struct ms_hyperv_info ms_hyperv; · line 46

Quoted 4 lines
arch/x86/kernel/cpu/mshyperv.c +1 -0
@@ -43,6 +43,8 @@ struct ms_hyperv_info ms_hyperv;
 bool hyperv_paravisor_present __ro_after_init;
 EXPORT_SYMBOL_GPL(hyperv_paravisor_present);
 
+u64 wakeup_mailbox_addr;

This value duplicates acpi_mp_wake_mailbox_paddr in madt_wakeup.c. It looks like the duplicate value is used for two things:

1) In hv_is_private_mmio_tdx() to control the encrypted vs. decrypted mapping (Patch 5 of this series)

2) As a boolean in hv_vtl_early_init() to avoid overwriting the wakeup_secondary_cpu_64 value when dtb_parse_mp_wake() has set it to acpi_wakeup_cpu(). (Patch 9 of this series).

Quoted 9 lines

Having a duplicate value is messy, and I'm wondering if it can be avoided. For (1), hv_private_mmio_tdx() could call into a function added to madt_wakeup.c to make the check. For (2), the check should probably be based on hv_isolation_type_tdx() instead of whether the wakeup mailbox address is set. I'll note that Patch 5 of this series is using hv_isolation_type_tdx(), so there's a bit of an inconsistency in testing the wakeup_mailbox_addr in Patch 9.

I think your comment includes two points, the duplicated variables and the incosistency in the testing.

Thank you for pointing out the duplication of wakeup_mailbox_addr with acpi_mp_wake_mailbox_paddr. I didn't realize it. Yes, such duplication should be avoided and will fix it in next submission.

Agree the inconsistency in testing wakeup_mailbox_addr and hv_isolation_type_tdx() is not good. IMHO, the wakeup_mailbox_addr (or the new function you proposed) is better than hv_isolation_type_tdx(), since the wakeup_mailbox_addr is more directly related. But hv_vtl_init_platform() happens before DT parse, thus I have to use the hv_isolation_type_tdx() in it. I don't have a good idea on how to fix this.

Quoted 3 lines

Thanks
--jyh

I don't think there's a requirement to set the "is_private_mmio" function in hv_vtl_init_platform(). It just needs to be set before acpi_wakeup_cpu() is called, which does the memremap() that will invoke the "is_private_mmio" function to decide whether to map as encrypted or decrypted.

Quoted 5 lines

So maybe setting the "is_private_mmio" function could be done after dtb_parse_mp_wake() is called in its new location, and you know you have a valid wake mailbox address? Again, I haven't worked out all the details, so this approach might be just as messy, but in a different way. Use your judgment ... :-)

Sorry that I didn't explain clearly. The testing in hv_vtl_init_platform() is not only for the is_private_mmio, but also for the realmode_reserve(), which happens before the DT parse.

BTW, I don't know why the trampoline_64.S is put into the real mode blob. I don't find any specific requirement in the code, but I'm not sure if I missed anything. If this dependency is removed, all the TDX guest will benefit.

Thank you
--jyh

Quoted 2 lines

Michael

#25 lore raw

PATCH v2 5/9 x86/hyperv: Mark ACPI wakeup mailbox page as private

In reply to Yunhong Jiang

Current code maps MMIO devices as shared (decrypted) by default in a confidential computing VM. However, the wakeup mailbox must be accessed as private (encrypted) because it's accessed by the OS and the firmware, both are in the guest's context and encrypted. Set the wakeup mailbox range as private explicitly.

---
 arch/x86/hyperv/hv_vtl.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/x86/hyperv/hv_vtl.c +16 -0
index 04775346369c..987a6a1200b0 100644
@@ -22,10 +22,26 @@ static bool __init hv_vtl_msi_ext_dest_id(void)
 	return true;
 }
 
+static inline bool within_page(u64 addr, u64 start)
+{
+	return addr >= start && addr < (start + PAGE_SIZE);
+}
+
+/*
+ * The ACPI wakeup mailbox are accessed by the OS and the BIOS, both are in the
+ * guest's context, instead of the hypervisor/VMM context.
+ */
+static bool hv_is_private_mmio_tdx(u64 addr)
+{
+	return wakeup_mailbox_addr && within_page(addr, wakeup_mailbox_addr);
+}
+
 void __init hv_vtl_init_platform(void)
 {
 	pr_info("Linux runs in Hyper-V Virtual Trust Level\n");
 
+	if (hv_isolation_type_tdx())
+		x86_platform.hyper.is_private_mmio = hv_is_private_mmio_tdx;
 	x86_platform.realmode_reserve = x86_init_noop;
 	x86_platform.realmode_init = x86_init_noop;
 	x86_init.irqs.pre_vector_init = x86_init_noop;
Signature · 1 line
2.25.1
#26 lore raw

RE: [PATCH v2 5/9] x86/hyperv: Mark ACPI wakeup mailbox page as private

From: Yunhong Jiang <[email protected]> Sent: Friday, August 23, 2024 4:23 PM

Quoted 30 lines

Current code maps MMIO devices as shared (decrypted) by default in a confidential computing VM. However, the wakeup mailbox must be accessed as private (encrypted) because it's accessed by the OS and the firmware, both are in the guest's context and encrypted. Set the wakeup mailbox range as private explicitly.

---
 arch/x86/hyperv/hv_vtl.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/x86/hyperv/hv_vtl.c +10 -0
index 04775346369c..987a6a1200b0 100644
@@ -22,10 +22,26 @@ static bool __init hv_vtl_msi_ext_dest_id(void)
 	return true;
 }
 
+static inline bool within_page(u64 addr, u64 start)
+{
+	return addr >= start && addr < (start + PAGE_SIZE);
+}
+
+/*
+ * The ACPI wakeup mailbox are accessed by the OS and the BIOS, both are in the
+ * guest's context, instead of the hypervisor/VMM context.
+ */
+static bool hv_is_private_mmio_tdx(u64 addr)

arch/x86/hyperv/hv_vtl.c static bool __init hv_vtl_msi_ext_dest_id(void) · line 44

Quoted 10 lines
arch/x86/hyperv/hv_vtl.c +6 -0
+{
+	return wakeup_mailbox_addr && within_page(addr, wakeup_mailbox_addr);
+}
+
 void __init hv_vtl_init_platform(void)
 {
 	pr_info("Linux runs in Hyper-V Virtual Trust Level\n");
 
+	if (hv_isolation_type_tdx())
+		x86_platform.hyper.is_private_mmio = hv_is_private_mmio_tdx;

hv_vtl_init_platform() is unconditionally called in ms_hyperv_init_platform(). So in the case of a normal TDX guest running with a paravisor on Hyper-V, the above code will overwrite the is_private_mmio function that was set in hv_vtom_init(). Then the mapping of the emulated IOAPIC and TPM provided by the paravisor won't be correct.

Michael

Quoted 6 lines
arch/x86/hyperv/hv_vtl.c +0 -1
 	x86_platform.realmode_reserve = x86_init_noop;
 	x86_platform.realmode_init = x86_init_noop;
 	x86_init.irqs.pre_vector_init = x86_init_noop;
--

2.25.1

#27 lore raw

Re: [PATCH v2 5/9] x86/hyperv: Mark ACPI wakeup mailbox page as private

Quoted 49 lines · Michael Kelley

From: Yunhong Jiang <[email protected]> Sent: Friday, August 23, 2024 4:23 PM

Quoted 30 lines

Current code maps MMIO devices as shared (decrypted) by default in a confidential computing VM. However, the wakeup mailbox must be accessed as private (encrypted) because it's accessed by the OS and the firmware, both are in the guest's context and encrypted. Set the wakeup mailbox range as private explicitly.

---
 arch/x86/hyperv/hv_vtl.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/x86/hyperv/hv_vtl.c +10 -0
index 04775346369c..987a6a1200b0 100644
@@ -22,10 +22,26 @@ static bool __init hv_vtl_msi_ext_dest_id(void)
 	return true;
 }
 
+static inline bool within_page(u64 addr, u64 start)
+{
+	return addr >= start && addr < (start + PAGE_SIZE);
+}
+
+/*
+ * The ACPI wakeup mailbox are accessed by the OS and the BIOS, both are in the
+ * guest's context, instead of the hypervisor/VMM context.
+ */
+static bool hv_is_private_mmio_tdx(u64 addr)

arch/x86/hyperv/hv_vtl.c static bool __init hv_vtl_msi_ext_dest_id(void) · line 44

Quoted 10 lines
arch/x86/hyperv/hv_vtl.c +6 -0
+{
+	return wakeup_mailbox_addr && within_page(addr, wakeup_mailbox_addr);
+}
+
 void __init hv_vtl_init_platform(void)
 {
 	pr_info("Linux runs in Hyper-V Virtual Trust Level\n");
 
+	if (hv_isolation_type_tdx())
+		x86_platform.hyper.is_private_mmio = hv_is_private_mmio_tdx;

hv_vtl_init_platform() is unconditionally called in ms_hyperv_init_platform(). So in the case of a normal TDX guest running with a paravisor on Hyper-V, the above code will overwrite the is_private_mmio function that was set in hv_vtom_init(). Then the mapping of the emulated IOAPIC and TPM provided by the paravisor won't be correct.

Quoted 1 line

Michael

non-VTL Hyper-V platforms are expected to disable CONFIG_HYPERV_VTL_MODE, that means for a normal TDX guest hv_vtl_init_platform will be an empty stub. Have I missed anything ?

- Saurabh

#28 lore raw

RE: [PATCH v2 5/9] x86/hyperv: Mark ACPI wakeup mailbox page as private

From: Saurabh Singh Sengar <[email protected]> Sent: Monday, September 2, 2024 11:39 AM

Quoted 52 lines · Michael Kelley
Quoted 1 line

From: Yunhong Jiang <[email protected]> Sent: Friday, August 23, 2024

4:23 PM

Quoted 48 lines
Quoted 30 lines

Current code maps MMIO devices as shared (decrypted) by default in a confidential computing VM. However, the wakeup mailbox must be accessed as private (encrypted) because it's accessed by the OS and the firmware, both are in the guest's context and encrypted. Set the wakeup mailbox range as private explicitly.

---
 arch/x86/hyperv/hv_vtl.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/x86/hyperv/hv_vtl.c +10 -0
index 04775346369c..987a6a1200b0 100644
@@ -22,10 +22,26 @@ static bool __init hv_vtl_msi_ext_dest_id(void)
 	return true;
 }
 
+static inline bool within_page(u64 addr, u64 start)
+{
+	return addr >= start && addr < (start + PAGE_SIZE);
+}
+
+/*
+ * The ACPI wakeup mailbox are accessed by the OS and the BIOS, both are in the
+ * guest's context, instead of the hypervisor/VMM context.
+ */
+static bool hv_is_private_mmio_tdx(u64 addr)

arch/x86/hyperv/hv_vtl.c static bool __init hv_vtl_msi_ext_dest_id(void) · line 44

Quoted 10 lines
arch/x86/hyperv/hv_vtl.c +6 -0
+{
+	return wakeup_mailbox_addr && within_page(addr, wakeup_mailbox_addr);
+}
+
 void __init hv_vtl_init_platform(void)
 {
 	pr_info("Linux runs in Hyper-V Virtual Trust Level\n");
 
+	if (hv_isolation_type_tdx())
+		x86_platform.hyper.is_private_mmio = hv_is_private_mmio_tdx;

hv_vtl_init_platform() is unconditionally called in ms_hyperv_init_platform(). So in the case of a normal TDX guest running with a paravisor on Hyper-V, the above code will overwrite the is_private_mmio function that was set in hv_vtom_init(). Then the mapping of the emulated IOAPIC and TPM provided by the paravisor won't be correct.

Quoted 6 lines
Quoted 1 line

Michael

non-VTL Hyper-V platforms are expected to disable CONFIG_HYPERV_VTL_MODE, that means for a normal TDX guest hv_vtl_init_platform will be an empty stub. Have I missed anything ?

Ah, you are right. My concern is misplaced and can be ignored. :-)

Michael

#29 lore raw

PATCH v2 6/9 x86/realmode: Add memory range support to reserve_real_mode

In reply to Yunhong Jiang

Currently the reserve_real_mode() always allocates memory from below 1M range, although some real mode blob code can execute above 1M.

The VTL2 TDX hyperv guest may have no memory available below 1M, but it needs to invoke some real mode blob code that can execute above 1M memory.

Instead of providing a platform specific realmode_reserve callback, the reserve_real_mode() is updated to support flexible memory range to meet this requirement.

---
 arch/x86/include/asm/x86_init.h |  6 ++++++
 arch/x86/kernel/x86_init.c      |  3 +++
 arch/x86/realmode/init.c        | 14 ++++++--------
 3 files changed, 15 insertions(+), 8 deletions(-)
arch/x86/include/asm/x86_init.h +6 -0
index 213cf5379a5a..9e3198bfe56e 100644
@@ -31,12 +31,18 @@ struct x86_init_mpparse {
  *				platform
  * @memory_setup:		platform specific memory setup
  * @dmi_setup:			platform specific DMI setup
+ * @realmode_limit:		platform specific address limit for the realmode trampoline
+ *				(default 1M)
+ * @reserve_bios:		platform specific address limit for reserving the BIOS area
+ *				(default 1M)
  */
 struct x86_init_resources {
 	void (*probe_roms)(void);
 	void (*reserve_resources)(void);
 	char *(*memory_setup)(void);
 	void (*dmi_setup)(void);
+	unsigned long realmode_limit;
+	unsigned long reserve_bios;
 };
 
 /**
arch/x86/kernel/x86_init.c +3 -0
index 82b128d3f309..f3226aa77bfb 100644
@@ -8,6 +8,7 @@
 #include <linux/ioport.h>
 #include <linux/export.h>
 #include <linux/pci.h>
+#include <linux/sizes.h>
 
 #include <asm/acpi.h>
 #include <asm/bios_ebda.h>
@@ -68,6 +69,8 @@ struct x86_init_ops x86_init __initdata = {
 		.reserve_resources	= reserve_standard_io_resources,
 		.memory_setup		= e820__memory_setup_default,
 		.dmi_setup		= dmi_setup,
+		.realmode_limit		= SZ_1M,
+		.reserve_bios		= SZ_1M,
 	},
 
 	.mpparse = {
arch/x86/realmode/init.c +6 -8
index f9bc444a3064..6d658ad8c0f4 100644
@@ -45,7 +45,7 @@ void load_trampoline_pgtable(void)
 
 void __init reserve_real_mode(void)
 {
-	phys_addr_t mem;
+	phys_addr_t mem, limit = x86_init.resources.realmode_limit;
 	size_t size = real_mode_size_needed();
 
 	if (!size)
@@ -54,17 +54,15 @@ void __init reserve_real_mode(void)
 	WARN_ON(slab_is_available());
 
 	/* Has to be under 1M so we can execute real-mode AP code. */
-	mem = memblock_phys_alloc_range(size, PAGE_SIZE, 0, 1<<20);
+	mem = memblock_phys_alloc_range(size, PAGE_SIZE, 0, limit);
 	if (!mem)
-		pr_info("No sub-1M memory is available for the trampoline\n");
+		pr_info("No memory below %lluM for the real-mode trampoline\n", limit >> 20);
 	else
 		set_real_mode_mem(mem);
 
-	/*
-	 * Unconditionally reserve the entire first 1M, see comment in
-	 * setup_arch().
-	 */
-	memblock_reserve(0, SZ_1M);
+	/* Reserve the entire first 1M, if enabled. See comment in setup_arch(). */
+	if (x86_init.resources.reserve_bios)
+		memblock_reserve(0, x86_init.resources.reserve_bios);
 }
 
 static void __init sme_sev_setup_real_mode(struct trampoline_header *th)
Signature · 1 line
2.25.1
#30 lore raw

PATCH v2 7/9 x86/hyperv: Move setting the real_mode_header to hv_vtl_init_platform

In reply to Yunhong Jiang

For the VTL2 hyperv guest, currently the hv_vtl_init_platform() clears x86_platform.realmode_reserve/init while the hv_vtl_early_init() sets the real_mode_header.

Set the real_mode_header together with x86_platform.realmode_reserve/init in hv_vtl_init_platform(). This is ok because x86_platform.realmode_init() is invoked from an early initcall while hv_vtl_init_platform() is called during early boot.

---
 arch/x86/hyperv/hv_vtl.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/hyperv/hv_vtl.c +1 -1
index 987a6a1200b0..e5aa2688cdd0 100644
@@ -44,6 +44,7 @@ void __init hv_vtl_init_platform(void)
 		x86_platform.hyper.is_private_mmio = hv_is_private_mmio_tdx;
 	x86_platform.realmode_reserve = x86_init_noop;
 	x86_platform.realmode_init = x86_init_noop;
+	real_mode_header = &hv_vtl_real_mode_header;
 	x86_init.irqs.pre_vector_init = x86_init_noop;
 	x86_init.timers.timer_init = x86_init_noop;
 
@@ -259,7 +260,6 @@ int __init hv_vtl_early_init(void)
 		panic("XSAVE has to be disabled as it is not supported by this module.\n"
 			  "Please add 'noxsave' to the kernel command line.\n");
 
-	real_mode_header = &hv_vtl_real_mode_header;
 	apic_update_callback(wakeup_secondary_cpu_64, hv_vtl_wakeup_secondary_cpu);
 
 	return 0;
Signature · 1 line
2.25.1
#31 lore raw

PATCH v2 8/9 x86/hyperv: Set realmode_limit to 4G for VTL2 TDX guest

In reply to Yunhong Jiang

The VTL2 TDX guest may have no sub-1M memory available, but it needs to invoke trampoline_start64 to wake up the APs through the wakeup mailbox mechanism. Set realmode_limit to 4G for the VTL2 TDX guest, so that reserve_real_mode allocae memory under 4G.

---
 arch/x86/hyperv/hv_vtl.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
arch/x86/hyperv/hv_vtl.c +8 -4
index e5aa2688cdd0..5829aac74f80 100644
@@ -40,11 +40,15 @@ void __init hv_vtl_init_platform(void)
 {
 	pr_info("Linux runs in Hyper-V Virtual Trust Level\n");
 
-	if (hv_isolation_type_tdx())
+	if (hv_isolation_type_tdx()) {
 		x86_platform.hyper.is_private_mmio = hv_is_private_mmio_tdx;
-	x86_platform.realmode_reserve = x86_init_noop;
-	x86_platform.realmode_init = x86_init_noop;
-	real_mode_header = &hv_vtl_real_mode_header;
+		x86_init.resources.realmode_limit = SZ_4G;
+		x86_init.resources.reserve_bios = 0;
+	} else {
+		x86_platform.realmode_reserve = x86_init_noop;
+		x86_platform.realmode_init = x86_init_noop;
+		real_mode_header = &hv_vtl_real_mode_header;
+	}
 	x86_init.irqs.pre_vector_init = x86_init_noop;
 	x86_init.timers.timer_init = x86_init_noop;
 
Signature · 1 line
2.25.1
#32 lore raw

RE: [PATCH v2 8/9] x86/hyperv: Set realmode_limit to 4G for VTL2 TDX guest

From: Yunhong Jiang <[email protected]>

Quoted 5 lines

The VTL2 TDX guest may have no sub-1M memory available, but it needs to invoke trampoline_start64 to wake up the APs through the wakeup mailbox mechanism. Set realmode_limit to 4G for the VTL2 TDX guest, so that reserve_real_mode allocae memory under 4G.

s/allocate/allocate/

Michael

Quoted 33 lines
---
 arch/x86/hyperv/hv_vtl.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
arch/x86/hyperv/hv_vtl.c +8 -5
index e5aa2688cdd0..5829aac74f80 100644
@@ -40,11 +40,15 @@ void __init hv_vtl_init_platform(void)
 {
 	pr_info("Linux runs in Hyper-V Virtual Trust Level\n");
 
-	if (hv_isolation_type_tdx())
+	if (hv_isolation_type_tdx()) {
 		x86_platform.hyper.is_private_mmio = hv_is_private_mmio_tdx;
-	x86_platform.realmode_reserve = x86_init_noop;
-	x86_platform.realmode_init = x86_init_noop;
-	real_mode_header = &hv_vtl_real_mode_header;
+		x86_init.resources.realmode_limit = SZ_4G;
+		x86_init.resources.reserve_bios = 0;
+	} else {
+		x86_platform.realmode_reserve = x86_init_noop;
+		x86_platform.realmode_init = x86_init_noop;
+		real_mode_header = &hv_vtl_real_mode_header;
+	}
 	x86_init.irqs.pre_vector_init = x86_init_noop;
 	x86_init.timers.timer_init = x86_init_noop;
 
--

2.25.1

#33 lore raw

PATCH v2 9/9 x86/hyperv: Use wakeup mailbox for VTL2 guests if available

In reply to Yunhong Jiang

For VTL2 hyperv guest with wakeup mailbox in device tree, don't overwrite wakeup_secondary_cpu_64 so that the acpi_wakeup_cpu will be used to bring up the APs.

---
 arch/x86/hyperv/hv_vtl.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/hyperv/hv_vtl.c +2 -1
index 5829aac74f80..09a7410200ba 100644
@@ -264,7 +264,8 @@ int __init hv_vtl_early_init(void)
 		panic("XSAVE has to be disabled as it is not supported by this module.\n"
 			  "Please add 'noxsave' to the kernel command line.\n");
 
-	apic_update_callback(wakeup_secondary_cpu_64, hv_vtl_wakeup_secondary_cpu);
+	if (!wakeup_mailbox_addr)
+		apic_update_callback(wakeup_secondary_cpu_64, hv_vtl_wakeup_secondary_cpu);
 
 	return 0;
 }
Signature · 1 line
2.25.1