Skip to contentSkip to conversation

Documentation/x86: Improve the AMX documentation

PATCH Superseded Chang S. Bae 1 participant 10 messages Sep 22, 2022 – Jan 21, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v5 cover Documentation/x86: Improve the AMX documentation

Hi all,

The change from the last version [1]:
* Update the purpose part (patch1) by rewriting the altstack story and having it as the only reason.

Thanks Dave Hansen for the feedback.

=== Cover Letter ===

When the AMX feature was supported in Linux, the dynamic feature enabling model was introduced. The following documentation changes were considered to help users to adopt this new enabling model:

(1) The AMX-enabling code example is expected to clarify the steps.
(2) Along with that, a couple of ABI constants may be useful for users.
(3) Also, describing the motiviation will provide the context of this.
(4) Lastly, the description of new guest options are added as missing.

If they are acceptable, the arch_prctl(2) manual page [2] will be followed up separately with something similar to the kernel documentation.

These patches can be also found in this repository:
git://github.com/intel/amx-linux.git amx-doc

And the compiled preview is available here:
https://htmlpreview.github.io/?https://github.com/intel/amx-linux/doc-web/x86/xstate.html

Thanks,
Chang

[1] https://lore.kernel.org/lkml/[email protected]/
[2] arch_prctl(2): https://man7.org/linux/man-pages/man2/arch_prctl.2.html

Chang S. Bae (4):
Documentation/x86: Explain the purpose for dynamic features x86/arch_prctl: Add AMX feature numbers as ABI constants Documentation/x86: Add the AMX enabling example Documentation/x86: Explain the state component permission for guests

 Documentation/x86/xstate.rst      | 100 ++++++++++++++++++++++++++++++
 arch/x86/include/uapi/asm/prctl.h |   3 +
 2 files changed, 103 insertions(+)

base-commit: 08ed00508bc1fa0a0bc6dd2420f982b55051de23

Signature · 1 line
2.17.1
#1 lore raw

PATCH v5 1/4 Documentation/x86: Explain the purpose for dynamic features

This summary will help to guide the proper use of the enabling model.

---
Changes from v4:
* Re-write about the sigaltstack sizing (Dave Hansen).
* Drop the second point as the case is not clear yet.

Changes from v3:
* Add as a new patch (Tony Luck).

---
 Documentation/x86/xstate.rst | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
Documentation/x86/xstate.rst +16 -0
index 5cec7fb558d6..e954e79af4ce 100644
@@ -11,6 +11,22 @@ are enabled by XCR0 as well, but the first use of related instruction is
 trapped by the kernel because by default the required large XSTATE buffers
 are not allocated automatically.
 
+The purpose for dynamic features
+--------------------------------
+
+Legacy userspace libraries often have hard-coded, static sizes for
+alternate signal stacks, often using MINSIGSTKSZ which is typically 2KB.
+That stack must be able to store at *least* the signal frame that the
+kernel sets up before jumping into the signal handler. That signal frame
+must include an XSAVE buffer defined by the CPU.
+
+However, that means that the size of signal stacks is dynamic, not static,
+because different CPUs have differently-sized XSAVE buffers. A compiled-in
+size of 2KB with existing applications is too small for new CPU features
+like AMX. Instead of universally requiring larger stack, with the dynamic
+enabling, the kernel can enforce userspace applications to have
+properly-sized altstacks.
+
 Using dynamically enabled XSTATE features in user space applications
 --------------------------------------------------------------------
 
Signature · 1 line
2.17.1
#2 lore raw

PATCH v5 2/4 x86/arch_prctl: Add AMX feature numbers as ABI constants

In reply to Chang S. Bae

AMX state is dynamically enabled by the architecture-specific prctl(). Expose the state components as ABI constants. They become handy not to be looked up from the architecture specification.

---
Changes from v2:
* Add as a new patch (Tony Luck).

---
 arch/x86/include/uapi/asm/prctl.h | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/include/uapi/asm/prctl.h +3 -0
index 500b96e71f18..f298c778f856 100644
@@ -16,6 +16,9 @@
 #define ARCH_GET_XCOMP_GUEST_PERM	0x1024
 #define ARCH_REQ_XCOMP_GUEST_PERM	0x1025
 
+#define ARCH_XCOMP_TILECFG		17
+#define ARCH_XCOMP_TILEDATA		18
+
 #define ARCH_MAP_VDSO_X32		0x2001
 #define ARCH_MAP_VDSO_32		0x2002
 #define ARCH_MAP_VDSO_64		0x2003
Signature · 1 line
2.17.1
#3 lore raw

PATCH v5 3/4 Documentation/x86: Add the AMX enabling example

In reply to Chang S. Bae

Explain steps to enable the dynamic feature with a code example.

---
Changes from v1:
* Update the description without mentioning CPUID & XGETBV (Dave Hansen).

Changes from v2:
* Massage sentences (Bagas Sanjaya).
* Adjust the example with the (future) prctl.h.

---
 Documentation/x86/xstate.rst | 55 ++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
Documentation/x86/xstate.rst +55 -0
index e954e79af4ce..23b1c9f3efb2 100644
@@ -80,6 +80,61 @@ the handler allocates a larger xstate buffer for the task so the large
 state can be context switched. In the unlikely cases that the allocation
 fails, the kernel sends SIGSEGV.
 
+AMX TILE_DATA enabling example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Below is the example of how userspace applications enable
+TILE_DATA dynamically:
+
+  1. The application first needs to query the kernel for AMX
+     support::
+
+        #include <asm/prctl.h>
+        #include <sys/syscall.h>
+        #include <stdio.h>
+        #include <unistd.h>
+
+        #ifndef ARCH_GET_XCOMP_SUPP
+        #define ARCH_GET_XCOMP_SUPP  0x1021
+        #endif
+
+        #ifndef ARCH_XCOMP_TILECFG
+        #define ARCH_XCOMP_TILECFG   17
+        #endif
+
+        #ifndef ARCH_XCOMP_TILEDATA
+        #define ARCH_XCOMP_TILEDATA  18
+        #endif
+
+        #define MASK_XCOMP_TILE      ((1 << ARCH_XCOMP_TILECFG) | \
+                                      (1 << ARCH_XCOMP_TILEDATA))
+
+        unsigned long features;
+        long rc;
+
+        ...
+
+        rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features);
+
+        if (!rc && (features & MASK_XCOMP_TILE) == MASK_XCOMP_TILE)
+            printf("AMX is available.\n");
+
+  2. After that, determining support for AMX, an application must
+     explicitly ask permission to use it::
+
+        #ifndef ARCH_REQ_XCOMP_PERM
+        #define ARCH_REQ_XCOMP_PERM  0x1023
+        #endif
+
+        ...
+
+        rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, ARCH_XCOMP_TILEDATA);
+
+        if (!rc)
+            printf("AMX is ready for use.\n");
+
+Note this example does not include the sigaltstack preparation.
+
 Dynamic features in signal frames
 ---------------------------------
 
Signature · 1 line
2.17.1
#4 lore raw

PATCH v5 4/4 Documentation/x86: Explain the state component permission for guests

In reply to Chang S. Bae

Commit 980fe2fddcff ("x86/fpu: Extend fpu_xstate_prctl() with guest permissions") extends a couple of arch_prctl(2) options for VCPU threads. Add description for them.

---
Changes from v1:
* Add the reason for the guest options (Dave Hansen).
* Add a note to allude some VMM policy, i.e. KVM_X86_XCOMP_GUEST_SUPP.
* Move it in the separate section.

Note the correspondent attributes were also proposed for the KVM API. But, it was seen as inessential:

    https://lore.kernel.org/lkml/[email protected]/
---
 Documentation/x86/xstate.rst | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
Documentation/x86/xstate.rst +29 -0
index 23b1c9f3efb2..ae5c69e48b11 100644
@@ -143,3 +143,32 @@ entry if the feature is in its initial configuration.  This differs from
 non-dynamic features which are always written regardless of their
 configuration.  Signal handlers can examine the XSAVE buffer's XSTATE_BV
 field to determine if a features was written.
+
+Dynamic features for virtual machines
+-------------------------------------
+
+The permission for the guest state component needs to be managed separately
+from the host, as they are exclusive to each other. A coupled of options
+are extended to control the guest permission:
+
+-ARCH_GET_XCOMP_GUEST_PERM
+
+ arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features);
+
+ ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it
+ provides the same semantics and functionality but for the guest
+ components.
+
+-ARCH_REQ_XCOMP_GUEST_PERM
+
+ arch_prctl(ARCH_REQ_XCOMP_GUEST_PERM, feature_nr);
+
+ ARCH_REQ_XCOMP_GUEST_PERM is a variant of ARCH_REQ_XCOMP_PERM. It has the
+ same semantics for the guest permission. While providing a similar
+ functionality, this comes with a constraint. Permission is frozen when the
+ first VCPU is created. Any attempt to change permission after that point
+ is going to be rejected. So, the permission has to be requested before the
+ first VCPU creation.
+
+Note that some VMMs may have already established a set of supported state
+components. These options are not presumed to support any particular VMM.
Signature · 1 line
2.17.1
#5 lore raw

v5 cover Documentation/x86: Improve the AMX documentation

In reply to Chang S. Bae

Hi,

This is a resend of the previous posting [1]. They are rebased on the current tip master.

Looks like the below changes are still helpful for users as I had to reference the pending series to some folks:

(1) The AMX-enabling code example is expected to clarify the steps.
(2) Along with that, a couple of ABI constants may be useful for users.
(3) Also, describing the motivation will provide the context of this.
(4) Lastly, the description of new guest options is added as missing.

If they are acceptable, I will follow up on the arch_prctl(2) documentation
[2] to update based on this.

These patches can be also found in this repository:
git://github.com/intel/amx-linux.git amx-doc

And the compiled preview is available here:
https://htmlpreview.github.io/?https://github.com/intel/amx-linux/doc-web/x86/xstate.html

Thanks,
Chang

[1] https://lore.kernel.org/lkml/[email protected]/
[2] arch_prctl(2): https://man7.org/linux/man-pages/man2/arch_prctl.2.html

Chang S. Bae (4):
Documentation/x86: Explain the purpose for dynamic features x86/arch_prctl: Add AMX feature numbers as ABI constants Documentation/x86: Add the AMX enabling example Documentation/x86: Explain the state component permission for guests

 Documentation/x86/xstate.rst      | 100 ++++++++++++++++++++++++++++++
 arch/x86/include/uapi/asm/prctl.h |   3 +
 2 files changed, 103 insertions(+)

base-commit: 1642cea9d9ab82b596857eea14bf6e04806a6c5a

Signature · 1 line
2.17.1
#6 lore raw

PATCH RESEND v5 1/4 Documentation/x86: Explain the purpose for dynamic features

This summary will help to guide the proper use of the enabling model.

---
Changes from v4:
* Re-write about the sigaltstack sizing (Dave Hansen).
* Drop the second point as the case is not clear yet.

Changes from v3:
* Add as a new patch (Tony Luck).

---
 Documentation/x86/xstate.rst | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
Documentation/x86/xstate.rst +16 -0
index 5cec7fb558d6..e954e79af4ce 100644
@@ -11,6 +11,22 @@ are enabled by XCR0 as well, but the first use of related instruction is
 trapped by the kernel because by default the required large XSTATE buffers
 are not allocated automatically.
 
+The purpose for dynamic features
+--------------------------------
+
+Legacy userspace libraries often have hard-coded, static sizes for
+alternate signal stacks, often using MINSIGSTKSZ which is typically 2KB.
+That stack must be able to store at *least* the signal frame that the
+kernel sets up before jumping into the signal handler. That signal frame
+must include an XSAVE buffer defined by the CPU.
+
+However, that means that the size of signal stacks is dynamic, not static,
+because different CPUs have differently-sized XSAVE buffers. A compiled-in
+size of 2KB with existing applications is too small for new CPU features
+like AMX. Instead of universally requiring larger stack, with the dynamic
+enabling, the kernel can enforce userspace applications to have
+properly-sized altstacks.
+
 Using dynamically enabled XSTATE features in user space applications
 --------------------------------------------------------------------
 
Signature · 1 line
2.17.1
#7 lore raw

PATCH RESEND v5 2/4 x86/arch_prctl: Add AMX feature numbers as ABI constants

In reply to Chang S. Bae

AMX state is dynamically enabled by the architecture-specific prctl(). Expose the state components as ABI constants. They become handy not to be looked up from the architecture specification.

---
Changes from v2:
* Add as a new patch (Tony Luck).

---
 arch/x86/include/uapi/asm/prctl.h | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/include/uapi/asm/prctl.h +3 -0
index 500b96e71f18..f298c778f856 100644
@@ -16,6 +16,9 @@
 #define ARCH_GET_XCOMP_GUEST_PERM	0x1024
 #define ARCH_REQ_XCOMP_GUEST_PERM	0x1025
 
+#define ARCH_XCOMP_TILECFG		17
+#define ARCH_XCOMP_TILEDATA		18
+
 #define ARCH_MAP_VDSO_X32		0x2001
 #define ARCH_MAP_VDSO_32		0x2002
 #define ARCH_MAP_VDSO_64		0x2003
Signature · 1 line
2.17.1
#8 lore raw

PATCH RESEND v5 3/4 Documentation/x86: Add the AMX enabling example

In reply to Chang S. Bae

Explain steps to enable the dynamic feature with a code example.

---
Changes from v1:
* Update the description without mentioning CPUID & XGETBV (Dave Hansen).

Changes from v2:
* Massage sentences (Bagas Sanjaya).
* Adjust the example with the (future) prctl.h.

---
 Documentation/x86/xstate.rst | 55 ++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
Documentation/x86/xstate.rst +55 -0
index e954e79af4ce..23b1c9f3efb2 100644
@@ -80,6 +80,61 @@ the handler allocates a larger xstate buffer for the task so the large
 state can be context switched. In the unlikely cases that the allocation
 fails, the kernel sends SIGSEGV.
 
+AMX TILE_DATA enabling example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Below is the example of how userspace applications enable
+TILE_DATA dynamically:
+
+  1. The application first needs to query the kernel for AMX
+     support::
+
+        #include <asm/prctl.h>
+        #include <sys/syscall.h>
+        #include <stdio.h>
+        #include <unistd.h>
+
+        #ifndef ARCH_GET_XCOMP_SUPP
+        #define ARCH_GET_XCOMP_SUPP  0x1021
+        #endif
+
+        #ifndef ARCH_XCOMP_TILECFG
+        #define ARCH_XCOMP_TILECFG   17
+        #endif
+
+        #ifndef ARCH_XCOMP_TILEDATA
+        #define ARCH_XCOMP_TILEDATA  18
+        #endif
+
+        #define MASK_XCOMP_TILE      ((1 << ARCH_XCOMP_TILECFG) | \
+                                      (1 << ARCH_XCOMP_TILEDATA))
+
+        unsigned long features;
+        long rc;
+
+        ...
+
+        rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features);
+
+        if (!rc && (features & MASK_XCOMP_TILE) == MASK_XCOMP_TILE)
+            printf("AMX is available.\n");
+
+  2. After that, determining support for AMX, an application must
+     explicitly ask permission to use it::
+
+        #ifndef ARCH_REQ_XCOMP_PERM
+        #define ARCH_REQ_XCOMP_PERM  0x1023
+        #endif
+
+        ...
+
+        rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, ARCH_XCOMP_TILEDATA);
+
+        if (!rc)
+            printf("AMX is ready for use.\n");
+
+Note this example does not include the sigaltstack preparation.
+
 Dynamic features in signal frames
 ---------------------------------
 
Signature · 1 line
2.17.1
#9 lore raw

PATCH RESEND v5 4/4 Documentation/x86: Explain the state component permission for guests

In reply to Chang S. Bae

Commit 980fe2fddcff ("x86/fpu: Extend fpu_xstate_prctl() with guest permissions") extends a couple of arch_prctl(2) options for VCPU threads. Add description for them.

---
Changes from v1:
* Add the reason for the guest options (Dave Hansen).
* Add a note to allude some VMM policy, i.e. KVM_X86_XCOMP_GUEST_SUPP.
* Move it in the separate section.

Note the correspondent attributes were also proposed for the KVM API. But, it was seen as inessential:

    https://lore.kernel.org/lkml/[email protected]/
---
 Documentation/x86/xstate.rst | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
Documentation/x86/xstate.rst +29 -0
index 23b1c9f3efb2..ae5c69e48b11 100644
@@ -143,3 +143,32 @@ entry if the feature is in its initial configuration.  This differs from
 non-dynamic features which are always written regardless of their
 configuration.  Signal handlers can examine the XSAVE buffer's XSTATE_BV
 field to determine if a features was written.
+
+Dynamic features for virtual machines
+-------------------------------------
+
+The permission for the guest state component needs to be managed separately
+from the host, as they are exclusive to each other. A coupled of options
+are extended to control the guest permission:
+
+-ARCH_GET_XCOMP_GUEST_PERM
+
+ arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features);
+
+ ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it
+ provides the same semantics and functionality but for the guest
+ components.
+
+-ARCH_REQ_XCOMP_GUEST_PERM
+
+ arch_prctl(ARCH_REQ_XCOMP_GUEST_PERM, feature_nr);
+
+ ARCH_REQ_XCOMP_GUEST_PERM is a variant of ARCH_REQ_XCOMP_PERM. It has the
+ same semantics for the guest permission. While providing a similar
+ functionality, this comes with a constraint. Permission is frozen when the
+ first VCPU is created. Any attempt to change permission after that point
+ is going to be rejected. So, the permission has to be requested before the
+ first VCPU creation.
+
+Note that some VMMs may have already established a set of supported state
+components. These options are not presumed to support any particular VMM.
Signature · 1 line
2.17.1